I went through a somewhat similar experience to what you have described. No other issues in the marriage, but for a while, every single time we went away for "vacation" it seemed to include a ton of extraneous family (almost always "his" side).
Now, don't get me wrong, I LOVE his side of the family, dysfunction and all. But if we made plans to go do something (whitewater rafting, for example) that was within driving distance of his family (within 2 hours of them), we would end up inviting "whoever wants to join us" to join us and ended up altering our plans to stay in their town instead of where we were going to stay... or to get a joint place instead of just our place. Which ordinarily is not a big deal... but once in a while, when on vacation, I'd kinda like to be able to just me and him decide what/when/where we are going to do/eat/see/drive and what schedule to be on while we're at it. Instead, every single thing became something where a group consensus needed to happen, and since my husband is the "peacemaker" type, we always were conceding to everyone else's needs/choices. Not what WE would have chosen if it had just been us.
Right down to eating at expensive places for lunch (with awful food) that our own children were too young to like anything on the menu, so basically didn't eat, or it cost us at least twice what it cost anyone else (who were only paying for themselves).
It was getting ridiculous, and causing tension between US on OUR vacation!
Husband's reasoning? They are family, and why not? Why not foster relationships between our kids and their cousins and great aunts and uncles? The will have memories that include all these people, and will bond the entire family.
Yes... all true. But some of the time, I want the memories to be mine! When we'd go as a family, someone was always hijacking our kids and we weren't seeing them do this or that, or participating with them (we were in another boat, or in another car, etc etc).
However, I was never left behind.
My problem (in your situation) would be that you agreed in advance, and your husband seems to not care about his word to you. If I ever convinced my husband that we should/shouldn't do a certain activity or whatever, he isn't one to go back on that.
As for the rest... I've been there. I get it. But at least try to go along as much as you can. Try to look at it as your kids bonding with their older cousins... not taking away from your family memories. It's hard to do. I know. But the rewards for it come later.
Just this past weekend, we went up to visit family and the entire family got together (as usual). My son's cousin (technically, my husband and his cousin both have sons, so the boys are what... first cousins once removed or something? anyway...) came with his grandparents (my husband's aunt/uncle). They saw each other about 2 months ago for 2 hours over dinner in a restaurant as we passed through town, and prior to that, it has been since at least last summer, probably over a year or more since they've seen each other. They are 14 turning 15.
When we arrived, it was like the boys had never spent a day apart. They are very close and hung out all afternoon like best friends. They just disappeared for chunks of time riding scooters or walking around the neighborhood. Who knows what they talked about... probably family gossip like all the rest of us were doing. LOL
We don't have boy cousins for him on my side of the family... and all the other cousins on husband's side are much younger. These two boys will probably always be close. And that is a direct function of how much time they spent together from the time they were probably 2 years old, at all the family stuff we did when they were younger.
It's quite a blessing NOW.
Try to look for positives... and realize that you are missing an opportunity to get to know your nieces/nephews when you stay behind-- not just for nuclear family memories. Family is bigger than that.