K.R.
I hate to tell you this, but your son sounds just like mine at that age. My son didn't sleep through the night until he was almost three years old. The more I tried to teach him to sleep, the more books I read on sleep training, the more routines I tried to impose on him during the day the worse things got. When I just decided that he was going to sleep with us (in a bed bumped right up to ours) we all started sleeping SO much better!
My daughter was sleeping 8 hours a night by 6 weeks old in a bassinet. She also would just fall asleep playing on the floor and take a nap on the carpet under her playmat, then wake up and resume playing. Around 6 months old she started waking up more frequently during the night (sleeping in a crib in her room), this is when we moved her into our bed to co-sleep and nurse throughout the night. She is three yrs now and wakes once during the night, calls for me and goes right back to sleep when I lay down next to her.
I honestly believe sleep habits vary between children based on so many factors, that by changing one, or even a few of those factors, doesn't guaruntee a good nights sleep in a crib in their own room. Personality plays a big part, developmental milestones (from seperation anxiety to learning to walk - emotional, cognitive, and physical changes), teething, family stresses or situations (moving, dad in the military, new baby in the house...), weather changes (ever wake up with a headache in the middle of the night cause a storm is blowing in? I have). Because of all these variables and knowing that there are just off nights for myself and I've had a whole lifetime to learn to sleep, I just figured that the best thing I can do for my kids is to be there for them in the night whenever they need me to be.
9-11 months old were the hardest months of sleep with both my kids, they were up literally every hour wanting to nurse, or crying out in their sleep, or tossing and turning. So frustrating sometimes! But hang in there and know that someday he WILL sleep, by himself, in his own room - I promise! :)
Someone pointed out once that there is a reason that "teach your baby to sleep" books are such a big business, it's because no one's baby is born knowing how to sleep! And in western cultures, we have the biggest sleep issues in the world, trying to force something so unnatural on our children when they are fighting so hard against it just doesn't help.
There are kids out there who learn to sleep after a few nights of crying, but many do not. Go with your gut!