I heard a professor once say that Einstein's mathematics led him to E=mc^2.
Actually, forty five years before Einstein, a British mathematician and physicist, William Clifford, after reading Riemanns lectures on the new concept of manifold and curvature declared that all forces will be shown to be an aspect of curvature. His insight is broadly true, not only for Einstein's gravity but for all the other forces in the standard model.
So this was a case in which mathematics led to a breakthrough in physics. But the story doesn't end there. Riemann's novel notions of geometry began with Gauss who was his supervisor. And Gauss discovered non-Euclidean geometry after reading Kants Critique of Pure Reason fives times. What did Kant write that inspired Gauss to discover this? Well, he wrote that it was not a priori obvious that the angles in a triangle should add upto 180 degrees. That they do so is, in Kantian terms, because it is a synthetic a priori truth. For a mathematician of Gauss's calibre, this clue was enough to set Gauss on his journey towards non-Euclidean geometry.
In fact, Einstein also name-checked Hume as an inspiration for his development of special relativity which reconceptualised our physical understanding of the relationship of space and time because his natural philosophy threw a critical eye on the nature of causality. This critique was also important for Kant too. However, this critique did not begin with Hume but with al-Ghazali.
So there you have two instances of probing philosophical questions in natural philosophy bringing about breakthroughs in mathematics and physics and also from mathematics to physics.
It can happen the other way around: that is breakthroughs in the sciences can lead to revolution in the arts. It's well known that Einsteins special relativity was the inspiration for Braque and Picasso's discovery of cubism and it was also the inspiration of Dali's famous painting, The Persistence of Memory. And the ideology of progress encapsulated in Baconian science led to Italian futurism and it's offshoots like Vorticism and so on.
Does abstract maths always represent some physical reality?
Well, it's unlikely that the large cardinal axioms from set theory are going to figure directly in any physical theory soon. I would also say that prime numbers, which are a large part of number theory, are also not going to turn up in physical theory in a natural way. They haven't done so, so far; and attempts to use them have usually an air of artificiality and ade really attempts to use physical tools to answer mathematical questions.
But more deeply speaking, as Aristotle already pointed out, the reason why mathematics does so well in describing physical laws is that both are aspects of neccessity - as well as logic. This answers Wigners question about the "unreasonable effectiveness of mathematics in physics". It's not "unreasonable" at all, and a small amount of reflection, or an acquaintance of the appropriate philosophical literature will show why. But then again, asking contemporary mathematicians or physicists to be acquainted with anything outside their narrow specialisms seems to be a bit of a big ask these days - especially in philosophy - and which is one reason why good questions like Wigner's are dogmatically resurrected from time to time with no appreciation that they have already been answered. And in this case, two and a half millenia ago.