Yes, you should keep trying. These people are your family, your children’s family, and it will be worth the effort if you can try. Take charge of the conversation, ask them about your husband when he was a child, their own lives as children, as young people, learn about them, show interest. Tell them about your kids, what they are doing, who their friends are. As someone else mentioned, food is always a safe topic, maybe movies or television shows, whatever they like to do, whatever you like to do. Don’t sit on your phone and act bored. Don’t expect them to text or call and don’t take it personally if they don’t. Not everyone has the same toolbox of things they do, but unless they are mean to you, you should keep trying.
I'm not clear on what "drama" is here, or what being the "black sheep" looks like. If they say something that you find upsetting, it would be nice if your husband could step in and say, “mom, that sounds upsetting, I’m sorry that happened”, and then change the subject. If he can’t do it, you can. You can acknowledge their feelings without getting into it with them. If they carry on, then it’s time to make an excuse and cut the visit short, “it’s been so nice to see you all, sorry we can’t stay longer,” and leave. The only thing that would justify not having a relationship with them would be if they are actually mean to you, your husband, or your children. Then your husband should be clear that is not acceptable and you will leave if it happens, and then leave. If that doesn't change that behavior, then you should minimize contact.
You are teaching your children how to treat people, and someday, you will be glad you did.