I really would let this go and focus instead on distracting and occupying your son. The reason behind what they're doing doesn't matter in the long run; they sound pretty self-involved if they can't see and react kindly to a kid who's staring longingly through the fence. This could be as simple as their son deciding he doesn't want to play with your son -- there can be a big gap between almost-five and almost-seven, and the almost-seven can think the younger chiild is too babyish to play with, etc. Don't try to fix this; your son may be heartbroken when he sees the bouncy castle full of other kids but this is life, and not everyone in it is nice.
If you are there and hear them tell him -- if they say it in the words you describe -- "go away," then yes, I would stand up and say "I hear you and I think you are being very rude to my child. It's your property and your right to not allow him to play on it, but please use politer language to my child. He is standing right here watching other kids and being told he is excluded; if you want to exclude him, at least do it with manners." Then I'd take your son and go. But other than that -- if you are present and there is actually an order to your son to leave them alone -- I would not keep trying to talk to them. They have made it very clear with their avoidance that they do not want to talk to you.
Meanwhile, ensure your son isn't in the yard when there are bunches of kids playing in the inflatables. Take him to the park, or start inviting other kids over for play dates instead of having your son think his playmates are next door.
Also, I don't know how you would handle this with your daughter, but I would not want my daughter babysitting for this family if the family is going to behave this way toward another of my children. If your daughter wants babysitting work, I am sure she can get it -- most of us would be delighted to find an experienced babysitter looking for work.