I work with both metals and wood. In my opinion the micro-fine dust that is generated from sanding and from a scroll saw is more likely to bother someone than the larger particle dust found in a saw mill. To further that statement, there are a number of exotic woods that emit toxins in the dust that you should be wearing a mask.
When I run my planer I am not so picky about the mask, same with the saw. When I use the scrollsaw or a Laser Welder I do use a light mask even though I have dust collection coming through the bottom of my cutting surface. I never shape or polish my metal parts from the laser work without a mask. The silicone particles from the polishing wheels and the compounds can cause many respiratory issues.
One of my friends wife worked in a plant that only cuts and packages paper envelopes for mailing. Due to the fine dust created she developed COPD and after a long illness died this past year.
If in doubt, wear a very light surgical mask. they cost $6.00 for a box of 50 at COSTCO and you get multiple uses per mask.