I think you need to figure a safer way. Show a picture here of what you are doing without the table saw on and maybe someone will provide a safe suggestion. Just doing it the way you described is not good. I hope I never log on here and find out you have amputated a finger or hurt your hand with a kick back. With your description, it could happen.
I worked more than 40 years in an extremely dangerous industrial environment and safety was always the topic. If you could not do a job safely, the work was stopped until we figured it out. We were responsible for watching out for each other. In your case, if you were doing it as you described, we would have stopped the work. If you continued to do it, you would have been taken off the job.
I apologize for the rant but I have seen too many accidents which could have been prevented and I would like to prevent Kevin from having one.