Do you have any pictures of your son and his dad from before he passed/before the divorce? Can you remember some happy times that you all had together?
Maybe one of the things you can do to help your son is to put together a scrapbook all about his dad. Pictures, happy times, anecdotes about the two of them together. It might help him adjust to the loss, but also to get to know his dad. And he'll always have the book as something to remember his dad by.
The feelings you have about the ex/deceased may be hard to put aside - I understand, because my BIL went through a very nasty divorce with his wife, but if there is any way to put a positive spin on it and you can find it, then do it. He probably doesn't remember much about him, and he's at about the age where he identifies with the same-gender parent. My daughter is 4 1/2 and she wants to do everything I do - so that's probably where this is all coming from.