I find the smaller the blade, the more it has a tendency to wander and harder to control. But, with a bigger blade, it is harder to make tighter turns. You can try a #5 and see if that helps. I personally like a #3 for 1/4 Baltic Birch. Works well for me.
One thing to watch is if you're putting too much pressure on the side of the blade. Try this exercise next time you're cutting. Every once in awhile, loosen up on your workpiece and see if it bounces back a little. If so, you're putting sideways pressure on the blade. I do this exercise quite often when I'm cutting and usually surprised that I'm putting pressure on the side of the blade.
One thing many new scrollers worry about is trying to stay perfectly on the line. That comes with experience, sure. But it's not normally a big deal if you wander off the line a bit. A mistake that new scrollers will do is over correct. This will create wavy and irregular lines. Instead, gradually adjust so you eventually come back to the line. The gradual correction will make it less obvious. And remember, when you pull off the pattern, nobody will know if you stayed on the lines or not.