C.,
this is such a good question. I know what you mean about 'time standing still' when they are outside of your view.
I think you should follow your instincts, if it doesn't feel right to let him outside of your view, don't do it.
My daughter just turned 5 and the other girls on our cauldesac are all older - like 9, 10 etc.. I allow her to go out front and play as long as I can see her. This means looking out the window or walking out front every few minutes, but it means the world to her. I try to do things to get the kids to play in my yard or in my house so I don't have to look too far :-)
I do not let her out of my sight, but that's just me and she is still just 5. Overprotective? maybe but I don't care, I don't think we can protect our precious children too much.
Can you ride bikes along with your son and friends? - far enough away to let him feel some freedom, but close enough for your comfort?
best of luck to you.
W.