for most of my holes I use a dremel in a router base like this I need the base to keep the holes square. I also have a keyless chuck so I can use a variety of drill sizes. I don't remember if it was part of the original dremel or if I bought it separately, but it is awesome for drilling holes.
Using the dremel with router base has really helped me from snapping drill bits. Using a cordless drill with small bits I would get a bit of wobble and snap off the small bits. Also happens when pieces are not held down well with a drill press.
I also have a 1984 harbor freight bench top drill press, but it mostly collects dust in the garage. It was my dad's, he bought it shortly after I was born, so I am a bit sentimental. I drag it out when I need bigger holes.
Almost all of the holes I drill are 1/16". For most of my work I can find a spot to put a 1/16" hole to start a cut, so I don't worry about using the extra tiny bits. I buy a few packs at a time of 10-12 1/16" bits from harbor freight for a few bucks. They work well for me, I still loose more than I dull.