I often use 320 grit carefully when finished but I don't wait until the end to do my sanding. I find that using mostly spiral blades, the fuzzies will start to drag the piece in the blade slot, so I sand them away. I find, for me anyway, that is better than waiting and facing the prospect of sanding the complete cutting. Usually by the time I am done the back requires only a few minutes and the top often needs just a little attention because the shelf liner I use does pull a little grain when it's removed.
I like your idea of scraping the surface prior to applying the pattern though!
By the way, looking at your boxes etc., you need change nothing when it comes to finishing. I love finishes that look inviting to touch. Wood can be such a tactile thing when done well. I dislike pieces that look good but scream "don't get fingerprints on me..." I used to go from sandpaper to scraping and then pumice with oil to rottenstone with oil and occasionally even cigar ashes. (although I don't smoke cigars) and then a hint of wax and a lot of elbow grease. I just realized why I don't do that stuff anymore. lol