Personally, I would say they need you out there always. My two are 12 and 9 (will be 13 and 10 in June). They are both great swimmers and love the water, and I still stay outside by the pool with them most of the time. I read a book or magazine, or just watch them. Which often involves reminding them of some safety rule. I do get up and walk inside to get a refreshed glass of ice water, or stand inside to fold a load of towels. But our floor plan layout gives me a visual of the pool from inside through all glass french doors directly onto the patio/pool deck and from the bay breakfast window (full glass also) that is directly onto the patio/pool deck. I never stay inside more than 60-90 seconds. I keep a phone outside with me, so that I don't get distracted by a ringing phone or caught up in a conversation and forget they are out there. Plus, it is right there for 911 if it ever were necessary.
I do NOT allow them to be outside in the pool area at ALL if there are ANY guests, without an adult supervising them. My own kids know the rules of the pool and pool area, and are not overly aggressive or into dunking people or rough housing at all. They like to dive for dive toys and race each other from one end to the other. No diving board exists and no diving is allowed as we don't have the depth to permit it (6 ft at the deepest point, which is in the middle... shallow on both ends). But I know that other kids can often not remember the rules or think they won't be enforced. Some rules are going to be tested by kids no matter whose house they're at and all kids think that some rules don't apply to them. And I will admit, that often I will let rules slide once in a while b/c it is just easier (no running in the house, but someone ran through the family room, etc). But not with the pool. All rules are enforced. If I see a guest break one, they get ONE warning. If they break it again they have to get out and sit for a period of time. Then I remind them of the rule, give them another chance, and watch them like a hawk. I give no quarter for pool rule breakers and I am not particularly nice about it. Although otherwise, I am. :) They know I mean business regarding pool safety. We have been here for 5 years and our kids have a healthy respect for the pool. When the pool isn't "open" for the warm months, they are not even allowed out on the patio or around the pool when they are outside playing (it is screened in)... and they have never tested that rule. They just play in the yard and never try to even short-cut into the house through the patio... they use the other doors. They don't "hide" there when playing hide and seek. They just are not allowed out there. Period.
But, we only have 2 kids. Even that 60-90 seconds that I might pop inside for a new cup of ice in my water, or to retrieve the basket of clothes from the dryer to bring to the family room where I can see outside, would not happen if there were 3. Three is just a bad number when it comes to pool safety. Two could be playing together and not notice something going on with number 3. Our two play very interactively and would tell on the other if they were to do something they shouldn't, lol. And no one would be overlooked.
They know to NEVER pretend to be "in trouble" in the water, too. And never to pull on another swimmer when playing. These things took several summers to ingrain in them. And visitors never seem to know these rules. The first month or so of each summer I never leave them alone (even for that 60-90 seconds to refresh my water) until I see how much they remember the rules.
While your kids' personalities do play a part... I still wouldn't just let them go outside to swim and you only look out to check on them every 10 minutes or so. It needs to be that you are keeping track of how long you are away (by the seconds) not how long you have before you have to check on them.
I do notice, that the more kids in the water, the more risk taking goes on. At all age levels. Definitely wait a few more summers before you let them have any unsupervised time. If you can see them from where you are inside the house, then you can sit inside and stare out the window. But understand, especially since they are going to be new to understanding pool safety, that you need to be out there where you can correct dangerous behavior right from the start. It will take awhile (if not a LONG while, depending on your kids) before they remember what is NOT okay. And they won't think they are doing anything wrong, until you point it out. From that perspective, I would say your 12 yr old probably needs at least 2 full summers to ingrain the rules of safety around the pool. And that is assuming that he is a good swimmer, and if you plan to leave him keeping an eye on the other kids, that he has some CPR classes and water safety classes, etc.
I do NOT leave my 12 yr old in "charge" of anyone else.