H.H.
It is nice to have something to go away with from the party- but it seems like they will have their face painted, so that would do. Maybe a bowl of candy by the door so they could grab themselves a little treat to take home. (like lolypops, m&ms or fruit snack (my little pony fruit snack since you were looking for a "horse theme") )
And I think a photo of each child on the pony would be an excellent "favor".