Qt was not designed to be cross-compiled, so this is going to be a bit more difficult and time consuming than most other projects. That being said, I think it would be worth the effort, so here's how to do it.
First, download and extract the source package.
Run the configure script and build everything for the host machine. Something like this should do it:
./configure -prefix /usr -release -verbose make
Now it is necessary to backup the Makefiles for the utilities so they don't get rebuilt for the AVR32.