First off, {{HUGS}}!!! I went though this with my daughter's father and yes it is very hard to deal with, you need to be strong because in the end, your son will see who had his best intrest in the end when he gets older and realizes what has/had happened between you and your ex.
Now, document everything!!! Get a note book and write down the dates and times of any interaction between you, your ex and your ex and your son either good or bad. Keep is simple and keep your personal feelings out of it too. So for the phone call you'd have date and time and write down phone call:son screaming mommy in back ground, offered to pick up, so and so refused, said not till 6pm on Sunday. Told me it was my fault. Now I'm sorry, but if your ex knows anything about his own child or little children at all, they are so dearly attached to mommy at this age!! I've also got a 2yr old son and if he needs help and I'm home, he tells my hubby No, mommy do it!! My hubby isn't even allowed to push the cart at the store, only I am!! lol Also, this new living situation may need to move slower for your son in order for him to adjust. Yes, he is with daddy, but daddy isn't where he reconizes as home so that probably has him scared as well.
Now if you fear for his safety, you need to get him back into court ASAP (hence why you need to document all contact!!). If you've ever called the cops on him for anything violent or if he has ever been arrested for it then you've got great reason not to trust him alone with your son. You may also be able to refuse him visitation now if you fear his safety, check with GA law on this one so you don't get in trouble for visitation violation!!
And most importantly, don't talk bad about your ex in front of him. He may only be 2, but his brain is sponge and you'd be suprised at what he might remember!!
Good luck and get with your lawyer ASAP!!
S.