Did all numbers exist at the beginning of the universe?
The sensible answer seems to be that they didn't but we may need to explain why.
First, inevitably, we have to say what we assume a number to be. If you want to say as some philosophers do that numbers exist in some Platonic realm, then you will have the apparently insoluble problem that there does not seem to be any good reason to claim that we know that such a Platonic realm exists at all.
Instead, what we certainly know to exist is our ideas, and in particular our ideas of numbers. I can think of the number 5 and so my idea of the number 5 exists, although I can only claim that it exists now. However, this seems good enough since to be able to say that the number 5 exists, I only need to be able to think about it. If I cannot think about it, too bad, I will be unable to say anything about it.
To say that I have an idea of number 5 just means that the number 5 exists in an imaginary realm, that is to say, a realm which exists in my imagination, and (presumably) only in my imagination.
However, this is apparently good enough to do all the operations of arithmetic, although this takes place in this imaginary realm inside my mind.
This is fine since when I have the idea of the number 5, I can talk about it, reason about it, share my ideas with other people etc. I have to assume that the same sort of thing goes on in the minds of other people, and this is fine, too, since I never worried about the fact that living in the so-called "real world" requires an enormous amount of assumptions, not least that there really is a real world to begin with. Other people also don't appear worried.
This seems good enough to explain all empirical facts about arithmetic.
This also means that numbers do not exist outside our mind, which is no ontological disaster and it seems much more economical in terms of our assumptions about the furniture of reality.
So the sensible answer is that numbers didn't exist at the beginning of the universe, and this because, presumably, there was no human being and therefore no human being to think about numbers.