M.S.
My ex husband was very abusive also. My friends and family always used to say he kept you bare foot and pregnant. Out of the 10 years I stayed with him I had my beautiful children. My oldestbid going to be 11 in August. I left almost 5 years ago and got a pfa on him. Since then we have been to court a few times. Even with the judge ordering him 5 hours per month visitation, he hasnt seen the kids in almost 4 years. This has hit my oldest the hardest and she often asks why she doesn't have a dad. I explain things to her like this.... Your dad just isn't in the right time in his life to devote time to you and be the dad you need him to be. I tell her that I understand what she's feeling and why she feels that way. But I also tell her that she has a mommy a Grammy and a pap paw who love her very much and that we are here for her through anything and we always will be.
I would never trust my ex to be alone with my kids. If you haven't already filed something with the courts then do so. In most states if nothing is filed then it means you "share" custody and nothing can be done if he picks your son up and takes him. If you are truly fearful of this man then do what ever you can to protect your son. When he is old enough to understand, you can explain it to him and he can make his own decisions. Until then you need to protect him.
I always felt bad like I was doing the wrong thing too. Until at our last custody hearing a judge himself told me it would be more hurtful to have their father in and out of their lives then it would be to be just out. And that if he wasn't taking his regular visits then stop taking them until he can be a full time father. At that time he court ordered visitation only as I agree. It is hard. And there are times when I want to call him up and just tell him how the kids are and let him talk to them. But chances are he wouldn't care. And I don't want to hurt my babies any more than he already has. Get an attorney and do something legally!
Best of luck to you!