M.T.
If you expect kids to be allowed at YOUR house, imagine how the other parents perceive that you won't allow your kids at their houses. Set up playdates for each child with a friend their age. Let your kids know that you will not be doing daily playdates. It is okay for them to know that a six year old can have a walk over, drop off playdate and that a four year old is too young. Figure out some good rules, like outside play only, they can't go into someone's house without you knowing or without your permission, what age they are allowed to have walk-over playdates at someone's home. Remember that since your kids are two years apart, the same rules may not apply to each.
--------------------------------------------------------------------------
ETA: After seeing your explanation of the neighbor sending your kids away, she simply may prefer that people call first rather than ringing the doorbell, and when you are the house with the pool, kids turn up constantly, often expecting to use your towels and eat your snacks, and without bringing their own parent to supervise. Your kids may not have been asking for that, but you don't know how many other times the doorbell ding-donged that day.