Most people drill too fast and actually punch out the the last part ,they think they are drilling ,very small drill bits load up and need cleared several times in at drilling of a single hole ,Hard to do ,when doing dozens and dozens of holes & holes at a set ,stacking has layers ,in the center you you can feel the layers separate as drill enters new layer , as you drill ,most are seldom tight together as we think compounding the problem ,obvious on larger pc.s in center areas away from edges and pegs. Just a fact !! Also tiny drill bits tend to wander or bow even in a drill press ,JUST another fact !! Stackers ,find this out the hard way ,even single 3/4" find grain problematic on some woods .Just more facts ,drill bits like scroll saw blade need time to clear trapped sawdust and shavings to be able to cut and drill as straight as possible !
Or you can ignore and blame what ever else is You choose !