You are correct about the fuzzies. I have learned to ignore them. Some amount of clean up of fuzzies is necessary on all cuts. I have a three step process for removing the fuzzies.
First I use a sanding mop (the one from the UK that foxfold recommended) to remove all the very loose stuff from the back of the panel. I will do this often during the cutting to prevent a build up under the panel that interferes with smooth sliding of the panel on the table.
Second, after the cut is done and the sanding mop has done its job, I actually use my palm sander with 220 grit paper on the back side. Yeah, have to be careful with delicate areas, but I really haven't had much problem as long a I keep the sander flat. This removes a lot of the fuzzies and only takes a few seconds.
Third, to get any and all fuzzies left behind, especially in those small/narrow/sharp pointy areas, I use a butane torch. Poof - as long as you are quick and carefull - they are gone.
Use the spirals more, and soon you won't want to use a regular blade much any more. I only go to a #3 or #5 Pegas mgt R blade when I cut the long straight/oval/circular lines of a frame area that is part of the pattern.