There is no way my 12 year old would be staying out that late under those circumstances. You are right about the possibilities of what could happen and your son is too young to realize what he could be in for. He may buck you outwardly, but have a sense of security and feeling loved underneath.
I would say that you know he doesn't do those things, but other people around him might be and you don't want him exposed to that. It simply isn't appropriate for him to be staying out that late unsupervised at his age. It's because you love him and want him to be safe, not because you are trying to be the hard guy.
The reality is developmentally, he's not ready to make complex adult decisions which is the reason we don't let kids under the age of 14 stay out after dark unsupervised when the temptations will likely be there.
Personally I don't let my 12 year old go anywhere at anytime without some sort of adult supervision, but she's a girl and doesn't have the best common sense, bless her heart.
Kudos for being a caring parent who actually sets boundaries. I think we are a dying breed sadly enough.