Astigmatism To Right Eye, Blind In Left, Wearing Contact Lense That Making Crossed Eyed, Need Advice?

I have an astigmatism to my right eye. My left eye is completley blind d/t a inoperable cataract.
I wear glasses with balanced lenses just to even out that thickness of the lenses. When I wear my glasses it looks as if I am not looking at you with my left eye (as if I had a lazy eye) but my vision is fine on the right. Occassionally, I will wear a contact lense in my right eye (like for a formal event).
I recently had my eyes checked and my prescription was adjusted. I notice that when I wear my contact lense that my left eye looks a bit crossed then eventually does go crossed eyed when I’m wearing it for a while. Even though I don’t wear my contact very often, when I do wear it I get somewhat of a headache after wearing it all day.
Is my eyes trying to get used tot he contact lense every time I wear it since I don’t wear it often? Or should I get rechecked for the lense prescription/shape of the lense?