I prefer iTerm and luckily the solutions to switch to using it instead of the xterm launched by X11 is pretty easy to implement.
The first option is to make a user specific copy of xinitrc and modify the line that launches xterm to rather launch iTerm. If you don't have a user specific xinitrc (~/.xinitrc) you can grab a copy from /etc/X11/xinit/xinitrc or /usr/X11R6/lib/X11/xinit/xinitrc.
Near the bottom of the xinitrc you'll see a few lines that reference xterm that you can comment our and substitute with:
If you access remote X applications very infrequently it may simply be easier to ensure that X11 is running and then export your DISPLAY variable manually in iTerm:
$ export DISPLAY=:0
The final solution is to make all terms aware of your DISPLAY variable by editing your local environment plist (~/.MacOSX/environment.plist):
$ open ~/.MacOSX/environment.plist
This will open the plist in the Property List Editor where you can add a new child (DISPLAY) under the root node with a value of ':0'. For this to take effect you need to log out/in. You could also have dropped 'export DISPLAY=:0' into your local ~/.bash_profile to achieve the same results.