heck, i have days i cant follow a crooked line!
im not sure if youre aware, but scrollsaws will have a drift- trying to feed at a 90 degree to the blade doesnt work to get a straight line. i have to feed at an angle from the right side.
the best advise i can give at this time is keep constant pressure letting the blade do the work and look ahead of the cut. i have noticed better control with straight cuts when i have the saw speed up higher,too.
what type of wood are you cutting? that can factor in,too
also, cutting with the grain can be a bugger- it always easier to cut straight against the grain.