That was the best thing you could have done for your son, following thru with your word. I know it's incredibly hard for a 3 yr. old to sit sooo long and be quiet, especially when there's nothing that is geared towards his level of understanding. I have tried it both ways...having them sit in church and having them in the nursery. I have always liked the nursery because it gives something to my child in the way of bible stories at their level of understanding. And I benefit as I am able to sit and concentrate on the service and what our Pastor is saying. I started having my kids sit with us when they were older in elementary school. I don't think any kid at any age likes to sit thru church. My oldest is 13 and she still doesn't like sitting thru a sermon. :o)
That being said, I have had my children sit with us in church too at that young. What I've found is even at that young of age, they are testing you and the boundaries you set. They are seeing just how much they can get away with and what you will tolerate. And believe me, they know how to push your buttons and will go right up to the brink of what is acceptable behavior and then some. Little stinkers aren't they? LOL I have found with my kids, being firm in my word has always gotten their attention. Just saying no without consequences only makes matters more difficult for you and him. Be sure you do exactly what you say you will do if he doesn't behave. Example: leaving the party or not being able to do something special, etc. This will definitely get his attention and he will start to put two and two together as it were. Unfortunately we as Moms have to deal with the fallout, the crying, the pleading, the tantrums, etc. It might take a couple of times of consequences for his behavior( Be sure to have tylenol on hand for your headache...LOL )before he'll learn.
As for ideas of keeping him quiet...I've brought puzzles along to church for my kids and books that do things...pop up or show how to tie a shoe kinds of things, coloring books. I've even brought a few little small NON Sound making toys to keep them occupied. Or when I am desperate sometimes letting them go thru my purse and discover what's in there works. Again, it's not my favorite thing as picking up the contents of my purse after church is no fun. LOL Hope that helps a little!