This all sounds like very typical behavior for that age, plus you also have the two year old competing for your attention.
But I think you ought to really look at what is 'insubordination' (is he in the Army or at day camp?!?) and what is really not that big a deal.
For example, of course you do not want to ENCOURAGE rudeness- but there is a big difference between being rude to a camp counselor and simply saying that he didn't want to play a certain game and wanted to sit out- there is nothing illegal about wanting to sit out of a game you don't like or don't enjoy, if he expressed himself politely and didn't cause trouble while he was sitting out. Maybe he just wanted to do a craft or other project instead.
I agree with the other poster that you can't realistically compare your 5 year old to your adult BIL. Sure, it is very possible that the way his parents treated him helped him become who he is today- but that is a pretty heavy guilt trip to lay on a 5 year old, and it is based on YOU and how you feel about your BIL- not even about how you feel about your own son!
Kids are always testing your boundries. My son is almost 12 now and is VERY polite and well behaved. I am a fairly strict mother, but I also know that you have to bend a little sometime, or you will go crazy- as they say "put out the big fires and step over the small ones;'
If your son is whining or being rude, make sure he knows AHEAD of time what the consequences will be for that. Sit down with him when he is in a good mood and talk to him and be VERY specific, or make a chart, so that he knows ahead of time if he throws a tantrum, X is what the consequence is, etc.
But know that whining and foot-dragging are going to happen with even the best behaved child. You will have to repeat your consequences and telling him that being rude is not acceptable THOUSANDS of times before it sinks in- and even then there will be lapses. You will occasionally grit your teeth and be frustrated- but you are the parent. Just stay calm and above all be CONSISTENT and make sure dad and other family members and baby sitters are too.
Decide now what issues are really important to you and what you can let go. I have never tolerated rudeness and talking back. But a little foot-dragging or wanting to sit out a game at camp is just a normal thing. Have some faith in your son and your own ability to parent him. It's totally asking for trouble to overdo it now just because you have a BIL who is an adult screw-up! Don't put that kind of pressure on a five year old.
I asked my son the other day if I am more strict than other moms and he thought about and said that even though he didn't always think I was right, that I was 'just the right amount' of strict! :) I was so happy- but he is almost 12 so it took YEARS to get him to that POV! :) Good luck!