Hi T.,
This is a little odd, and at her age, it's really hard to know what's going on, other than that whatever it is, she's processing it at night, and if she's screaming and scared.
How does this correspond with the behavior you see while they kids are together? Does she have fun? Does she shy away from the rougher kids ? Does she hang close you?
Obviously YOU aren't comfortable with what happens after a visit to your sister's, so at this point, I think I'd keep my child home. Let her get a little older before she has sleep-overs. (Mine did stay at my sister's at a young age, but they never had nightmares aftewards)
If don't think you have to tell your sister that your daughter appears to be petrified of her cousin, but you can. Let her know that your child has nightmares after she visits, without incriminating her child in the process. Just say that she's used to a different level of activity, and she doesn't sleep after spending time in their home. If she were to spend the night there, she'd keep your sister up all night, and you don't want to do that to her anymore than you want to be up all night the next night. She'll probably outgrow it, and be fine, and when she is, then she can do overnights without mom.
Otherwise, I think I'd spend the night with my daughter at my sister's house. Let her sleep with you ane everything, so she feels safe.
Kids have a way of letting us know very very plainly when they need something, or something isn't wrong. We as adults don't always pick up the cues, but I think you caught this one load and clear. On the other hand, she isn't bruised or anything, just scared. So it doesn't sound like it's a necessarily dangerous situation, but it does mean she's most likely afraid of her cousin. If your sister wants to conquer this one, she's going to have to teach her son to be more gentle.
My sister's family was always roudier than our household. IT was okay with my older girls, cuz they were the same ages, but when my "2nd set" of 2 girls arrived, 10 years later (oopsie), they were afraid of her boys. The boys were never horrible or anything, but they were "big" and the "moved too fast" for the girls -- esp the elder of the two because she is a quiet and introverted child. That's not to say she isn't friendly. She's great with people, she just doesn't do wild in-your-face stuff. Even now, at 15, the 19 yr old plus cousins intimidate her. They are strangers, really, and she doesn't have relationships with them, so when they are around, talking loudly and not making efforts to get to know the girls, the girls simply ignore them. It's kind of too bad, cuz they are the only cousins anywhere near their ages, but that's the way life is for them. They have lots of adult cousins who are good with them, and whom they enjoy. :-)
In the end, you can't force them to like their cousins, but you can protect them when they are young from being in fearful situations unnecessarily.
Good luck !