Here is how I make my case work,
I create a directory with the same path the configuration reports error, and copy the same source directory from my build machine over there.
Then I follow the guidance from DietmarRe: Build debian Linux distribution for Galileo board with debootstrap
CC="gcc -m32 -fno-stack-protector -march=i586" ./configure
and some other configuration options needed for my package. This time configuration finished ok.
During make, it report a missing library, which also can be found in the work folder of my image built machine. So I move that library to /usr/lib folder of my Galileo board, and make again. This time it passed.
After make install, I have a working expect.
Still, would like to learn how other people can do it in a less messy way.