Hi R. - I've had good luck with an ornery 4yr old boy on our block by doing a couple of things.
First off, kids from the block are not allowed inside of our house without a parent's direct permission. It's the same for my kids. That gives me a chance to talk to mom and kids about house rules etc. because it was more like a playdate than a dropover. It also helps me have some control over what happens inside the house, like my kids needing to finish homework or doing chores before I send them outside.
Secondly, I did the same thing as you did. I matter of factly warned him that he would need to go home if he didnt play nice. When he didnt, I asked his older brother to escort him home and I explained why the playdate was over. I think you did a good job in your situation explaining that you are the mom of your house.
Also remember to look on him not like just a snotty 4yr old but rather as a little kid with natural leadership qualities that need to be trained. On the whole redirection rather than ultimatum works better with these kinds of kids although that doesnt mean there are no natural consequences to behavior, like going home early. I can see how he wasnt really trying to control you as much as tell you all the things that he probably was hearing at home like "put your toys away" and "no, you can't watch movies all day".
Kids need to learn how to show respect to adults - they dont know it automatically. I don't know what you said but I've said before to my kids and to the neighbor kids "I dont like the way you are talking to me so you have a choice. You can go home (to your room) or you can stop being so bossy and negative." Then follow through.
He's probably going to be there a while so it's best to learn to live with him rather than create an adversarial relationship between him and your kids. He's going to be a different kid in a year or two after he starts school and matures a little.