I do think you need to examine how you feel about your niece and how much of that comes from you not liking her mother. I would never encourage my nieces or nephew in bad behavior and would feel totally comfortable correcting them- but it sounds to me like YOU have taken this a lot more personally than your daughter did.
If she is badly behaved, it is because your brother and her mom have not done a good job of TEACHING her the proper way to behave and treat people. I think you have a chance here to not only help her learn that it is not ok to be mean or bully other people- but also to give her some much needed structure and love and support.
Ask yourself- do you want to correct her out of love and concern, or just out of anger and frustration??
You sound like a good mom who wants to protect her little girl- that is a wonderful thing. Your niece doesn't sound so lucky! Be that good mom for her too, and sit down or take her out for a talk, just the two of you. Reassure her that she has your family's love and support and that if things are tough with her parents she can come and talk to you.
THEN talk with her about the behavior you saw. Ask her how she would feel if her cousins were doing that to her. Tell her that you don't have the power to ground her or punish her, etc. but that YOU know she is a better, sweeter, smarter girl than that and even though you understand she is frustrated right now, it is just NOT acceptable to take it out on her little cousin or other people- and it will just get her in trouble, not help or change anything.
Be the mom- be the bigger person here and help your niece learn to do the same. You could build a strong, loving family relationship between you and her and your daughter that you will cherish as you all get older.
Don't act out of anger- act out of love!