What is the thickness of the wood you are drilling? Hole saws don't clear out the wood very well. Have you tried drilling half way through, flip the board over and finish from the other side? Also drill a little, clear the kerf, then drill a little more and repeat until you're through.
I love making wheels. I usually buy the small ones and make the big ones.