Does the visual aberration/distortion appear *consistently* to you, not between different sights, but at least for each sight?
What I mean is, well, let's take take a hypothetical sight called the "X," sight, does the way the dot appears in that sight X always appear the same to you. For example, is it always a "comma" with a brighter dot at the upper left and a comet-tail off to the right? Or maybe it always appears as a "starburst" with a more distinct/sharply-resolved "beam" on the right?
If so, it's possible that you can use a visually distinct area of that aberration/distortion as a more refined aiming point.
I have not-insignificant astigmatism in both eyes. so working with RDS -particularly when combined with my nearsightedness- has always been a battle.
That said, I've learned how to stretch out to well beyond the 200 yard mark (with my goal being the 300, which is what a very well-respected SME on this topic suggested that good shooters should be able to do, even with astigmatism).
Another thing to pursue would be your corrective eyewear (if any) and eyepro. Insuring good alignment of your corrective eyewear when you are looking through the dot on the pistol or when checking a rifle. as well as making sure that your eyepro is of good optical quality and isn't itself causing any distortions, can both help significantly as well.