Maybe you are using too much spray adhesive. It only takes a very quick once over, let it sit for about 2 minutes before applying it to the wood. Make sure the wood is sanded well and very clean of dust. Then just a light application of mineral spirits and wait about one minute to remove the pattern. It should almost fall off on it's own. When done this way you should have no residue. If you try to remove the pattern too quickly and a little part of it does stick, you can use a paper towel barely dampened over that area and it should remove any residue completely.
I remember when I first tried doing it this way, I was very skeptical about doing it this way and used too much spray adhesive and experienced what you are talking about. Then I read some direction from someone on a different scroll saw page and explained it the way I just did above. Only this guy did not put the pattern on the wood until at least 3 minutes and up to 5 minutes. I did it the way I told you above and it worked perfectly. I was afraid the pattern would lift but I was willing to give it a try. I was pleasantly surprised and it is how I do all my scrolling now, even with a lot of very small fret holes. I never need my blue painters tape any longer and I never have any surface damage when I remove the pattern.
Dick
heppnerguy