It seems to me the problem with a strictly mathematical view is it doesn't, "think outside the box."
When you "shut up and calculate," how do you account for, "garbage in, garbage out?"
Remember epicycles really were brilliant math and laid a lot of the foundations of geometry, but for the age, understanding the earth was not the center required, literally, outside the box thinking.
Currently physics has been chasing reductionist atomism since Democritus, yet there doesn't seem to be any tiniest particle, just permutations of positive and negative charge. Fluctuations between synchronization pulling everything in and harmonization of everything equalling out. Particles and fields. Between black holes and black body radiation.
I think one of the bigger stumbling blocks is our modeling of time.
As these mobile organisms, this sentient interface between our body and its situation functions as a sequence of perceptions, in order to navigate, so our experience of time is as the point of the present, moving past to future. Codified as measures of duration.
The reality is that change turns future to past. Tomorrow becomes yesterday, because the earth turns. Duration is the present, as the events coalesce and dissolve.
There is no "dimension" of time, because the past is consumed by the present, to inform and drive it. Causality and conservation of energy. Cause becomes effect.
Time is asymmetric, because it's a measure of action and action is inertial. The earth only turns one direction.
Different clocks can run at different rates simply because they are separate actions. Think metabolism.
That culture is about getting everyone synchronized into one larger social organism, based on the same languages, rules, measures, etc. it might seem like there should be one universal, Newtonian flow of time and a great conundrum when Einstein showed otherwise, the fact is that nature is so diverse and integrated, is because everything doesn't march to the beat of the same drummer. Multicultural, not monoculture.
That events will appear in different order from different locations is no more consequential than seeing the moon as it was a moment ago, simultaneous with seeing stars as they were years ago.
It is the energy that's conserved, not the information. That the information changes is time.
Energy is conserved, because it manifests this presence, creating time, temperature, pressure, color and sound. Time is frequency, events are amplitude.
Ideal gas laws correlate volume with temperature and pressure, which are as foundational to our emotions and bodily functions, as sequence is to thought, but we don't consider them extensions of space.
So the energy goes past to future, because the patterns it generates come and go, rise and fall, future to past. Energy drives the wave, the fluctuations rise and fall.
Energy radiates out, as form coalesces in, given synchronization of waves is the path of least resistance. Thus galaxies.
The problem of "dark matter" is that while our tactile sensibilities place this relatively stable state at the center, like a geocentric cosmology puts the earth at the center, the reality is that mass is an intermediate state, between the barest bending of the light, to the vortices at the center. So mass is an effect of gravity, rather than gravity an property of mass.
Consciousness also goes past to future, while the perceptions, emotions and thoughts giving it form and structure go future to past. Though it's the gut processing the energy, while the head simply sorts the information.
Which is why the intellect tends to focus on the information, rather than the dynamic on which it rides.
Thus the almost religious obsession with math.
Now it will be interesting to see if you are willing to think through what I've argued, as a test of logic over belief. My experience is those most educated in the current paradigm/box, are unwilling/unable to look outside it.