So much great advice, and experience! I know as a first-time mom I was SO against the CIO method... but when our wonderful sleeper started not being such a wonderful sleeper (and this full-time teacher was a total mess all day and night!)-- the pediatrican told me that it would work, and it would not take long. I trusted her and within 3 days she was able to fall asleep on her own.
What our pediatrican told us was to extend the time by 5 minutes each time she cried. When we did go in, we did NOT pick her up (that just "rewards" the effort), we just spoke very gently and soothingly and patted her a little. It was really challenging at first, but little ones are much smarter than we give them credit for.
Also, I was convinced that I would "scar her for life" or give her "abandonment issues" if I didn't rush to her side, swoop in and smother her with "love and security" of rocking, holding, etc. I quickly came to realize that the love and security comes through her father and I interacting with her before bedtime, when she wakes up, and all the other wonderful time we spend together when she SHOULD be awake, not getting the sleep she so desperately needs!
I do think Erica's advice about "overtired" babies having a hard tme is worth noting. I know our daughter (who is now 2) has a much harder time settling and getting sleepy when she's TOO tired. Maybe start the settling down routine about a half hour earlier than you have been.
We are expecting our second daughter, and I can't imagine what I'd do if my firstborn was still struggling with sleep! The lessons we learn ;-) It is hard not to beat ourselves up about it--but think of it this way--he will be OKAY, he's too young to understand all of the complex emotions you are associating with his crying. He's learned to cry as his communication for getting something. When our daughter slept through the night after a 1.5 hour crying spell (with us attempting to sooth her frequently) the first two nights of CIO--she was still a happy, delightful, little one in the morning! When she got the sleep she needed (one of your major concerns), she and I were both SOOOO much happier in general.
Don't feel like a failure! You are doing everything you think is right for your son, asking your question is one more way of helping! Best wishes to you mama!!