Short Answer
Your question is excellent, because it inquires after the intersection of math and physics philosophically. Be mindful this is a broad topic because one has to know math, physics, and philosophy, ideally in equal proportions with history somewhat mashed in there for perspective.
Mathematical physical theories greatly enhance and help to define what is meant by the physical universe and supplement our basic intuitions which are sometimes referred to as naive physics, but note also the psychology of naive physics is used by some to ground the origins and effectiveness of mathematical theory, a perspective known as mathematical empiricism.
Long Answer
Scope
The topic you reference is the relationship between mathematics and the physical world, which is called the mathematization of science (Google ngram) which is both a topic of interest in the philosophy of science and the philosophy of mathematics particularly relevant to the question of math foundations. Obviously, the technicals of mathematical physics are best asked about in PhysicsSE. But I'll try to give you some pointers to ideas for the philosophical grounding of their intersection.
Historical Perspective
The systematic application of mathematics to the physical world starts at least as early as the Babylonians, which goes back to around 1,800 BCE who used it for astronomy and measurement. Babylonian math was very practical and very limited and might be considered the bare minimum needed to be considered the start of the mathematization of physics. About 1,500 years later, a big step forward was taken by Euclid of Alexandria with his axiomatization of mathematics known as Euclidean Geometry which took for granted certain presumptions about the physical world, such as the physical reality of parallel lines that largely went unchallenged until the 19th century. Gauss considered the mathematics of Archimedes, who was active around the same time, to also be one of the most important contributors to mathematics making him an important contributor to math and physics.
In this abridged history, the next outstanding contribution to the mathematization of physics undoubtedly was the work Galileo Galilei who used rigorous measurement in his experiments, a relatively new practice as science began to grow out of natural philosophy. The ancient Greeks had their sciences, as discussed by GER Lloyd, but around the 16th century, science entered a new phase of mathematical rigor, the next leap forward not occurring for about 300 years, with the advent of mathematical logicism and formalism of whom both Gottlob Frege and David Hilbert are considered major contributors.
But, before we get to the end of the 1800's and the start of the 1900's where the birth of the theories of QM and relativistic physics radically transformed mathematical physics, we need to take a step back and offer two other famous contributors of mathematical physics, Isaac Newton and his mechanics of motion and gravitation and Johannes Kepler and his astronomy. The early science of Galileo was done in a time when Aristotelian physics had been defended for a very long time within the Catholic church (and was part of the reason that the Church burned Giordano Bruno and imprisoned Galileo); by the time of Kepler, mathematics irrefutably killed the geocentric model of the universe.
Next is Big Al. This notion that mathematical theories of physics face resistance by intellectuals of the day didn't end with Kepler by any means. Albert Einstein won his Nobel Prize for physics for the photoelectric effect because the scientist who wrote his recommendation thought relativity wasn't much of a theory. In Albert Einstein, we also see a physicist who did foundational work at the quantum level as well as the relativist level, and ironically didn't consider his math skills very good. Here are some quotations to show his sometimes wry humor, of which, I like:
"So far as the theories of mathematics are about reality, they are not certain; so far as they are certain, they are not about reality."
Since then mathematical physics has churned out some theories about strings, dark matter, and multiverses, which some physicists see as being very non-scientific raising questions relevant to Einstein's insight above. For an amusing challenge to the notions that some of contemporary physics today is actually science, watch this short video by Sabine where she (rightly, IMNSHO) calls the multiverse a mathematical religion.
Philosophy, Math, and Science
What's going on here with this math and science stuff? Well, that's a little involved but we can offer quickly some philosophical ideas that can serve as a starting point to explore mathematical physics.
First, mathematical physics is an example of mathematical application and mathematical models. Second, in the models of mathematical physics, one is exploring quantitative relationships between the primitives of the physical ontology, such as energy, spacetime, and matter. But there's more to mathematical physics than ontological primitives, what might be thought of as physical entities, there are epistemological aspects. How do we know what we know in physics? This is one of the primary concerns in the philosophy of physics as well as the philosophy of science more broadly.
Let's just give an example of a transformation in philosophical thinking. After Newton simultaneously invented the calculus and mathematized force, mass, acceleration, and gravitation, scientist-philosophers saw the universe as being characterized by the mechanistic interaction of particles within a Euclidean space governed by the passage of absolute space and absolute time. What does that mean? It means that physics now had a rigorous footing on mathematics extending our naive physics into a very rational and empircal physics. In fact, by the end of the 19th century, some scientists thought there wasn't much new to learn about the laws of nature.
But there had been some developments in geometry, known as non-Euclidian, that had set the mathematical communities of the world abuzz, since Euclid's geometry was thought to justify the mathematics of physics. When the non-Euclidian geometries were accepted, mathematical philosophy was transformed because it turned out that grounding of mathematical theories in physics wasn't clear cut. It is on the basis of these theories that Albert Einstein reformulated a vision of the universe where time and space were related, relative, and the shape of the universe might be considered fundamentally curved. Later, this space was shown to be something akin to Minkowski space. In fact, there are now a plurality of models of space and time, keeping alive the question of the Ancient Greeks, what exactly is space and time?
This answer is already too long, but I'll leave off with the results of the mathematization of physics which continues to renew philosophical speculation and theorization. If math goes on in the mind of people, and mathematical theories are linguistic constructions of societies (constructivism), is the universe inherently real (scientific realism) or is it only an instrument of our minds (instrumentalism)? These are the sorts of philosophical considerations that arise when people combine mathematical thinking with physics.
Summary
You ask a tough question, because of the broadness of it. The logical positivists of the 20th century spent a great deal of time theorizing about math, physics, language, and philosophy, and we can say with certainty that we are still learning from their failures. This question is a great first step in the direction of following their thinking. About the only thing that can be said for certain is that anyone who claims to have all the answers probably doesn't, and the best thing to do is keep reading and thinking. Good luck!
References
- Relativity and Common Sense, Hermann Bondi - I love this book because it examines relatively conceptually, and is a classic.
- The Philosophy of Quantum Theory, Tim Maudlin - This is an excellent introduction starting with a short list of physics results in experimentation and a subsequent series of philosophical explanation including an introduction to Bell's Theorem as well as the mathematics involved. Written by a philosopher instead of a physicist, it's very clear.
- Causality: Models, Reasoning, and Inference, 2nd Ed., Judea Pearl - This book is for you if you are interested in modern methods of mathematical causality. There are formal systems of non-quantitative and non-experimental approaches to causality, but this book has some mathematically rigorous methods, easiest to understand if pursued after a math education.
- A Companion to the Philosophy of Science, Ed. W.H. Newton-Smith - The 1st edition has over 80 brief articles on important topics including "Mathematics, Role in Science". Tough to get through if you haven't studied some philosophy, such as epistemology, ontology, metaphysics, etc.
- Logical Positivism, A.J. Ayer - Written by one of their own after the movement had its heydey, an anthology of logical positivist thought.
- The Conceptual Roots of Mathematics, J.R. Lucas - To me, this represents the clearest exposition on the topic because it's written at the level a non-mathematician undergraduate can understand.