I was getting 10 second delays on ssh connections to my Virtual Box Machine. In my case it was due to ssh trying to do ipv6 negotiation, which apparently fails after a timeout. The cure was to set the following value in ~/.ssh/config:

AddressFamily inet