How do you put your fingers/thumb into the ball? Do you put your thumb, or fingers in first. You should put your fingers in first up to the first knuckle, then your thumb.
I've been having fit problems too. However mine seems to be quite different from yours. I have a free arm swing and before I even push away the ball to begin the swing I can tell that my thumb is not coming out. I am 99% sure that the ball still would not come off if I tried to do what you do to get the ball off. My thumb swelling has a lot to do with this.
Don't give up. Keep working with your
Pro Shop until you get it figured out.