C.K.
FWIW, I have some different perspectives on this issue. If nothing was ever explicitly stated, your "general understanding" may not be the same as hers or any of the other mother's. She may not be happy with this arrangement but is afraid to be the one who upsets the apple cart. I can tell you that I would not want to be responsible for the neighborhood kids, especially if it was not planned in advance--I could be in the middle of something, or quite frankly, not in the mood. If someone broke their arm or peed in front of everyone or god knows what else--it's a big responsibility. And if someone is naughty, reprimanding other people's kids can be a sticky situation.
I am the first to admit that I am over-protective and paranoid, but I would not let a 3 & 5 year old play with other kids without being around to supervise, or knowing that another adult was providing constant supervision. A quick peak out the door every now and then wouldn't be enough for me at those ages.
It is odd that she is sort of singling out your kids. Are they the youngest? Is your daughter the only girl--anything like that? Maybe the other mothers have arranged something in advance?
In the end, I think every parent needs to be responsible for, and keeping an eye on, their own kids, unless something else is set up ahead of time. It sounds like right now the kids sort of have free reign over the 4 yards, but maybe it needs to get a little more formal--asking permission or being invited. I could also see instances where maybe a family wants one family over, but not the other two. How to respect each other's spaces and time is something you will have to work out, but communication is key. Good luck.