You were honest, AND only answered your son's direct questions, without elaborating into territory that he may or may not be ready to cover. This is wonderful.
Now I suggest you change the conversation into one about all the different kinds of families there are in the world, and about all the people who ARE in your son's life and who love him and care for him and play with him and help him learn things. There are some great books out there about families of all different shapes and sizes. Todd Parr has a great one (and perfect for a 3-year-old) called "the Family Book." I'm sure there are others. Maybe the next time your son is coloring, sit down with him and help him draw a picture of his family: you, his grandparents, favorite teachers, neighbors, whomever is close to him and sees him often. If he goes to day care, ask the teachers for some help reinforcing the strength of the family that he has, rather than the missing piece.
He is going to be curious about his dad. The dominant culture still constantly reinforces "one mom, one dad, more than one kid" as the perfect family - even though most families don't look like that. But you can help him see the wonderful community that loves him, and help him know that he is not the only kid "without a dad." Good luck, you're doing a great job.