If these women know you are 53 and don't have an issue with it, the only complaint I hear is society's take (which shouldn't matter if you look the same age as the women you are dating, because 'society' would never know you were dating younger women anyway by looks alone).
But if you're lying about your age or withholding your age?
I say start gradually meeting people while doing the things that you already love.
Having said all of that (don't worry, my rant about dating younger women is over), you do say that younger women are attracted to you.
If that's truly the case, I don't see what the problem is.
My problem is I can easily pass for being in my late 30's.
I am fit, toned and dress younger than most men in their 50's. I am attracted to younger women because I still have the vigor and attitude of a younger man. "There are a lot of reasons why people in general are attracted to folks who are considerably (10 years) younger than themselves: to feel young again, to feel attractive to someone still in their reproductive prime, to enjoy the rush that youth provides (as opposed to the jadedness or anger some of us wear as we get older), and to feel proud that someone young and desirable is on our arm.
He may even be thinking of starting a second family with her so he can "get it right" this time around.