Speaking of Boy Bunnies

Now that I'm discussing male rabbits, I have a sneaking suspicion that the bunny currently amusing himself hopping around our back yard is a guy bunny. I've never seen him with another bunny or with any little baby bunnies, which I think would be around if he was female. So I'm really suspicious that he's a boy. Honestly, what is up with all of these guy bunnies sneaking into the house, and hopping into the yard? With the number of girl bunny sisters I have, surely that counts as some sort of safety hazard!