Band saw blades drift. Scroll saw blades drift. Some brands/sizes drift more than others but they all drift. It might depend on the tension, it might be affected by how you feed the wood into the blade, maybe something else. Don't worry about it, just understand and accept that it happens.
When I first started scrolling and wanted to cut a straight line, I would line up the wood and straight pattern line with the centerline of the scroll saw table and try to cut in that orientation. That did not work (because the blade drifts) and resulted in frustration.
Happiness came when I stopped trying to make it cut along the centerline of the table. Now I pay no attention to the orientation of the work vs the table. I pay attention to the line the blade is cutting and feed the patten line into the blade. The blade does cut a straight line, but you have to watch to see the line it is cutting.
Watch where/how it is cutting and look at the pattern line about a 1/2" or so in front of the blade; feed the pattern line into the blade. No more frustration.
Cheers and happy scrolling!