You sound so sweet:)
He's your baby so maybe you aren't ready yet and maybe your son isn't either. At some point you will need to learn to let your son be apart from you for periods of time, but I guess I'm wondering if now is the right time. Because of the part of concern for your son being mistreated, I'm wondering if you are having a mama gut instinct about having your son with his grandmother? Or if you know all too well what it feels like to be mistreated?
I'd say try your best to figure out where your fears are coming from...paying attention to where your son is and protecting him is your job and he isn't too old yet so being too overprotective isn't necessarily a horrible thing (though some might disagree). Utilize your time now while he's young to address where your fears are coming from...maybe from your past or upbringing? If you can find awareness now, this will get easier in the future and you'll be in a better position to truly know whether you are being too overprotective or whether you have just cause to not have your son be somewhere specific without you there to supervise.
Just know that as a mama, I completely understand your fears and need to know your son is safe and cared for. The thought of my son experiencing what I did growing up....well, I can't go there so I don't. But it does fuel my tendency to be protective. For me, it is important I not discard my feelings because they are there for a reason. But it is also important my son not miss out on things because of my own traumatic past because that belongs to me, not him. So figuring out whether my fear is about me or is in fact a legitimate fear is an effort I take seriously.
So, how to get past this fear? Get in touch with it so it doesn't control you, so you understand it better, and so you can ensure it finds closure on a personal level. But don't dismiss it if you feel it's a legitimate fear...that would be a mistake I think.
Best of luck:)