Thank you to everyone who replied and shared such great insights. I checked my drill press and found it was on the lowest speed setting (about 600 rpm!); I adjusted it to the 2nd highest (around 2100 rpm) and tried a new bit (Dewalt cobalt steel). I used the same piece of plywood for backing, and just held the baltic birch plywood to it by hand as before. The resulting hole was nice and crisp, with no delamination at all.
The bits I used before (TiN coated bits that resulted in delamination), I would say, had not seen much use. They were purchased as part of a lower priced set... I guess I got what I paid for...
It would have been better to separate variables (testing a new bit first, then testing a higher drill press speed separately), but a higher quality, new bit at a higher rpm certainly resolved the issue.
