If you need a custom display size (say you're trying to emulate an eeePC 901 display) open up a terminal and navigate to VirtualBox's directory, on Windows that's:
And run the following, making sure to change the VM name and display resolution:
VBoxManage.exe setextradata "[My Virtual Display Name]" CustomVideoMode1 1024x600x32
In this case a display that is 1024 by 600 and 32bits of color depth.
Then inside you're VirtualMachine you should see a new display resolution detected that you can use.
With "Guest Additions" installed you can also force a maximum resolution:
VBoxManage.exe setextradata "[My Virtual Display Name]" GUI/MaxGuestResolution 1024,600