D.W.
Diabetes is like being pregnant - you either have it or you don't. So, yes, those levels are really important because your body's ability to process increased glucose levels further in your pregnancy can change abruptly.
In addition to your diet, your liver stores a compound called glycogen. When your body isn't able to properly deliver glucose (sugars) to the cells, the body starts using the stored glycogen to provide energy. When the cells can't accommodate all the extra sugar in your bloodstream, you will have serious issues with both macrovascular and microvascular complications. They usually take years to develop, but gestational diabetes is a different beast.
It's really important - maybe not as important for your concern, but it's deeply important for your baby. One of the biggest concerns is a big baby - usually 9 lbs or greater is a concern.
Diabetes is really complicated and no one really understands it, but for the sake of your child, I'd really encourage you to go in for the 3 hour test. There's a small chance that there was a bad reading, but I'd personally not want to risk it. I spent 8 years involved in Diabetes treatments, and I've seen too many things that make me very passionate about making sure all people get treated quickly.
Good luck!