There's a lot going on in your post, & I suspect even more going on in your emotions, regarding your family relationships & dynamics.
First, the current issue. I know a lot of people have jumped to one extreme or the other in their response to what you should do/feel. I think that you cannot ignore who your family is, nor their actions (both towards you & towards others). As human beings, we naturally communicate with others based on how we expect they will respond. And how we know that is based on their actions at the time & in the past.
So it's really not possible to say "well, remove the backstory of what they do to your mom & you, & just answer regarding this current situation" because you wouldn't be approaching the situation the same way with different people. Because you have seen how they interact with your parents (& I can only assume with you as well at times), you must take measures to protect yourself from being manipulated, both in the situation, & emotionally.
Despite the number of responses below, none of us can really tell you if you are right or wrong to set up the party the way you did, & then to compromise only on the time, but not the location. You can't express in one post the lifetime of interaction you've had with your family for any of us to give an assessment for you.
What I can tell you is that it is the heart behind the matter that truly counts. Did you set up the location just to force people to drive & make it convenient for you? Did you refuse to relocate to a different restaurant just because you want to prove that they cannot make you change your plans to accommodate them?
It doesn't sound like it from your post. I'm sure that there are circumstances that you would change the location for, but most of us do not do so for convenience alone. Unfortunately, your brother/SIL have "cried wolf" a few too many times, & you are now very sensitive to it.
Standing your ground so that you are not taken advantage of, & therefore protecting yourself from the emotional turmoil it causes you, is nothing to feel guilty of. It is unfortunate that people in our lives feel that accommodations should be made for them, but are unwilling to compromise for us when necessary (& we all have someone like that in our lives!!).
Regarding their relationship with your parents, it is best to let them handle that. Make sure to not judge them or your parents for how that is working, because unless/until someone complains to you & asks for help, it is working for them. Not always are circumstances perfect in our situations, but your parents (mother) wouldn't be doing for them if she didn't feel she was getting what she needed in return.
One thing I would like to mention that concerns me, is your quote of your mother stating "we don't mind the extra drive & neither should you". This is emotional manipulation - making you feel that you should acquiesce for someone else because they perceive your situation as not as important. Whether you address this with your mother is up to you - might be best to let this one go, but be aware for the future, & be prepared to speak up and point out (gently, firmly, respectfully) that your life and priorities are just as important to you as theirs are to them, & you have considered whether or not you are able to make certain accommodations based on yours.
Please, focus on enjoying your daughter's celebration, and I hope that you will continue to value yourself enough to set boundaries. Just always make sure to examine your heart behind your actions when agreeing or refusing requests... both are appropriate, based on situations.
Happy Birthday to your daughter!!
T.