We do Gymboree, which I actually would not recommend (we were signed up before we moved so I'm just finishing up my 3 month obligation here). My daughter is a bit older, so the younger program might be better, but I generally don't hear the greatest reviews.
Most of the public libraries have music/storytime programs. The one at Greenburgh library is pretty good. They separate walkers from non-walkers, and the younger program is pretty low-key. We also do Music Together, which would be good for a younger baby. It's a mixed age class, but you kind of do the class in a way that's appropriate for your little one. We started it pretty young, and I believe they're doing an abbreviated summer session, so you would probably not have a problem with the registration issue. There is also Musical Munchkins, which people seem to like a lot, but we've never done it. Not sure if they have a summer session or not.
I hear mixed reviews of Little Gym. Some love it, some hate it.