vBulletin 3.8.7 and all other vBulletin versions up to (but not including) 4.2.0 sends the mails on each page load. If you set the emails sent per load too high, you will significantly slow down the browsing speed for all users who use your forum until the mailqueue has been cleared.
vBulletin 4.2.0 sends the same number of emails, however it spreads it out every 10 minutes. If you were to upgrade, you could take advantage of that to ensure the forum only sends X mails per hour.
Unless your hosting provider imposes any form of arbitrary restrictions on the number of mails per hour (or you yourself do so via WHM or any equivalent hosting control panel), the number of mails you send won't impact the likelihood of your mails getting flagged as spam.
All vBMail emails comply with most spam policies - our mails include an 1-click Unsubscribe link that does not require the user in question to log in or fill out their email, and will prevent any further communication from being received.
As such, any user who marks emails from your forum as Spam are being rather silly
vBMail itself cannot limit the amount of emails per hour, as it uses vBulletin's email queue functionality.