We just had to make the same decision recently so I know how you feel. But it was pretty easy for us: Saturday is so filled with sports for my daughter's friends, and dance rehearsals for my daughter, that it had to be Sunday. We did it from 2-4. I did not check with anyone except my husband; I just knew we would lose almost all of the kids she wanted to invite, if we picked any Saturday.
You mention that there is "a big race at the kids' school that day" so it's clear you have one particular Sunday in mind; why not do it on another Sunday when there is no afternoon event? Is this Sunday you mention being chosen because it's closest to the birthday? Really, most kids won't care that much if their party is a week later, or even more, as long as something small is done to mark the day itself. I would do it on a Sunday that does not have the school event.
My daughter's party was more than two months after her birthday, due to a heap of weekend commitments she had around the time of her birthday and for weeks afterward. It turned out just fine and she looked forward to it. But she's 11, and maybe it's easier for her to defer her pleasure for a while than it would be for a younger kid. But a week shouldn't be an issue, I would hope.