Maybe, just maybe the problem is that the drill bit is flexing while drilling so that the bit is not 90 degrees in the hole, which would mean that the length of the bit isn't enough with the additional angle. So, with that said:
What material are you drilling through? I was drilling pine, near a knot, with a small bit and it would not drill straight, the drill ibt would curve away from the hard knot.
What type of drill bit are you using, i.e. HSS, carbon steel, alloy, titanium oxide coated, etc? Cobalt drill bits are the stiffest I've found.
What speed are you using, when I use small bits I crank the speed up to max.