In CotVNC, you do this via the Connection -> Connection Profiles menu option, then click on the Colors tab. You may see another speed boost if you tell your VNC client to only display 256 colors. There used to be quite a few Mac clients, but the only one I'm aware of with any recent activity is Chicken of the VNC (CotVNC). Redstone Software, though, makes an OS X VNC server called, simply enough, OSXvnc. RealVNC is the official home of the software, but they don't offer a Mac server (just PC/Unix servers and clients). [ robg adds: For those who don't know, VNC is a way to control one computer's GUI from another location.

Like I said, a very simple and obvious hint.

Last night, I switched the background to a flat colour only, and the increase has been almost astronomical. I had all manner of fancy images used as desktop backgrounds on my Mac. The speed isn't too bad, but I've found a very simple (and once you know, completely obvious) way to increase the connection speed. I use VNC on an XP box at work to connect to my Mac at home.