Family cake & ice cream only for a 2 year old, same for 3. That IS a party, for crying out loud! As you said, they don't remember. In fact, they get confused and stressed out and cranky. They want to know why they don't get presents every day. Don't get blamed or guilt-tripped by family members. If they want to throw the party, fine.
Your older child can learn that parties are for older kids. In general, if you keep all the parties under control through the years, it will be better for all.
My son never had friends over until age 5. He didn't care, he didn't know. We had a few family members and one neighbor who always gave him a gift and with whom we were very friendly.
When they started school, we followed the "age" rule - for age 5, we invited 5 kids. 6 years, 6 kids. And we stuck to it. No siblings, no expensive venues, no stress over $20 goody bags, no inviting the whole class, no receiving 25 presents, no nagging over 25 thank you notes. You had a couple of friends, you got a couple of gifts and the guests got to see you open them and hear you say thank you (without sitting for an hour while you opened all 25 gifts!), you played some games, the kids got a couple of trinkets, and everyone went home. We didn't spend the college tuition or orthodontia money on extravagant nonsense that's more for the adults to compete with other adults than to have kids really enjoy it. If you have 25 kids over, you have to go to 25 parties through the year, schedule at least one every other weekend, and spend a ton of money on gifts for kids your child barely knows and has nothing in common with. Start now to keep it reasonable.
If your family members don't like this, hang up the phone and don't invite them. They are getting over-involved, and you already know in your gut what you don't want to do!