I haven't read the responses, yet, but I do agree with your SWH.
Our situation is different because we live on a small farm, and our entire property is fenced and gated. There is a large creek at the back of the property, and fields all around. Our dangers come from snakes, coyotes, and a stock pond. We don't have neighbors close enough to walk to, well, not until the kids are teenagers.
When our first son was about 4, if I remember correctly, I let him play out back by himself. He couldn't go to the front yard because that's where the pond is. He is my rule follower, so I knew I could trust him to stay out back, and I checked on him from the window, too.
Our boys are 6.5 years apart, and our oldest is extremely responsible, so he has always helped with watching over his brother. They also get along well, so they like hanging out together. Because of that, our youngest was allowed to go outside at an earlier age, but he had his brother with him.
Now, at ages 8 and 14, they roam freely and explore all over the property, and I ring a large cowbell whenever I need them to return home.
If we lived in a more traditional neighborhood, we would have different rules.