If you need a custom display size (say you're trying to emulate an eeePC 901 display) open up a terminal and navigate to VirtualBox's directory, on Windows that's:

C:\Program Files\Oracle\VirtualBox  

And run the following, making sure to change the VM name and display resolution:

VBoxManage.exe setextradata "[My Virtual Display Name]" CustomVideoMode1 1024x600x32  

In this case a display that is 1024 by 600 and 32bits of color depth.

Then inside you're VirtualMachine you should see a new display resolution detected that you can use.

With "Guest Additions" installed you can also force a maximum resolution:

VBoxManage.exe setextradata "[My Virtual Display Name]" GUI/MaxGuestResolution 1024,600