Setting aside whether or not she forgot for the moment...
The girls are 8 and your son is 3 turning 4. Those are two MAJOR age differences. Toddler versus solid/ halfway through elementary school. While you may have made a huge effort to create activities for the older kids that your son and HIS friends couldn't play... I'm sure the vast majority of the party is toddler-centered since it's HIS party. The girls would need to be tiptoeing around younger, wobblier, more meltdown prone children that they can't really talk to/play with (you know the difference... and older kid plays at a little kid level FOR them, because the younger kids can't keep up with the big kids), any "fault" is automatically theirs as well as the responsibility (the "you should __________, know better/ be more careful/ be gentle with/ etc), playing 'baby games' and eating 'baby foods'. The adults will all be visiting with each other and chasing their little ones, the toddlers will all be having a blast doing toddler things... and the girls will just be stuck in the middle with no one to play with/ talk to.
While I'm sure they LOVE their baby cousin... a party spent with toddlers versus a party spent with kids their own age that they can actually PLAY with...
The age gap between them and your own is something that undoubtedly widened into a GULF about 2 years ago and is only going to get wider. When your son is their age... they'll be teenagers.
Just a thing to keep in mind.
As is the fact that they're still KIDS (you'll see both how 'old' -aka look back at your son being 4 when he's 8 and get blown away-, as well as how YOUNG they are). Asking them to make an 'adult' choice at age 8 (suck up being bored for a few hours with a smile on your face while all your friends at at a party) is a difficult choice even for ADULTS to make. It's like volunteering to go to a boring lecture for several hours on a subject you're completely uniterested in instead of christmas morning with your kids. On any other day, you might be able to enjoy yourself for awhile (or at least not be furious/miserable/out and out refuse)... but on a day where there is something REALLY important to you? But because they're older they're "expected" to put on a happy face and not be miserable at a choice a grownup wouldn't willingly choose could they skirt it. This is NOTHING against your son. It's just the age gap. No matter who they are and how much he loves them, at age 8 he won't want to be at a toddler party. If his bestie is having a bday, or his team is announcing their end of year gathering... and you made him skip it to go to a toddler party... he'd be crying for days. Justifyably, because he'd be missing out on something important to him to be somewhere he's just window dressing and wouldn't really be involved with.
The other thing to keep in mind (bringing in the 'forgetting') part... as soon as your little one is in school full time, look back at today and requestion whether or not they actually forgot. Weeks are crazy whirlwinds, weekends are craziness in trying to split sports, parties, playdates, family time (nuclear, since you'll mainly only have your son for homework and dinner and bedtime during the week you'll be CRAVING time with him on the weekend and trying to jam it between other obligations), homework, projects, illnesses... toddler years are exhausting, but elementary years make many to most people go crosseyed trying to 'get' everything done and not forget anything.