I think it depends on what you mean by bullying. Is this kid just telling your son he doesn't want to be friends anymore? If he is just ignoring/avoiding your son, that is not bullying. This is a tough lesson to learn, but not all friendships last, and your son needs to move on and play with other friends.
However, if this kid is going out of his way to call names, tease, or create embarrassing situations for you son, that is something to worry about. Any physical contact is, of course, completely unacceptable. These situations can not be ignored. You say you hate confrontation, but don't you hate seeing your son bullied?
Start with the teacher. Let her know what your son is telling you about this kid and that other parents are saying the same thing. Ask her to keep an eye out. See if that improves things. If not, go to the Principal. Ask him/her to talk to the yard duty supervisors. These boys need to be watched and caught in the act. None of these actions should get back to you or your son as a tattle tale. Especially if you make it clear that you are speaking for a group. Other kids are being bullied too, not just your son.
Stay on it. Your son needs to know you are willing to stand up for him. Just make sure it is something you should be standing up for. He needs to learn his own lessons about friendships. Evaluate the situation and make sure it is really bullying and not just the sometimes unavoidable, painful end to a close friendship.
Be brave!