It seems that reality, simulated or not, consists of some currency (eg, matter/energy) and is governed by laws. Using science, we have the ability to observe/quantify the present state of reality, and also thereby to infer the laws according to which reality changes. Given such information, I can think of two scenarios that a person might take as evidence that there is a system outside of (or encapsulating) reality: contravention of the laws of reality--especially a change that manifests in a highly complex way (direct manipulation of complex systems rather than of base systems)--or improbability of the present state of reality, given the laws of reality.
See that both of those arguments rest on probability. They are paralleled in philosophy of religion by, respectively, the argument from miracles and the teleological argument. Also, it can be seen that the simulation hypothesis overlaps (at least partially) with the god (sentient, omnipotent, omni-etc. creator) hypothesis in that both propose accounts of reality wherein a sentient being outside of, or in a realm encapsulating, our own reality is responsible for creating, and defining the outcome/behavior of, our reality. Thus the discourse relating to those aspects of philosophy of religion, and whether they can form a basis for scientifically testing the existence of a creator, also applies to the simulation hypothesis.
An obvious problem here is the fallibility and never-guaranteed completeness of empirical findings/models. How can you tell if something is improbable given, or violates, the laws of reality, if you cannot be sure of (or, as at present, are sure of your ignorance of) those laws. Was that (for example) spontaneous resurrection truly a miracle (assuming it happened at all), or was it perhaps caused by the unknown presence of complex systems within our reality? That thread of contemplation leads one to ask whether final (total and complete) reduction of physics to its most base elements will (i.e. does) lead to a mathematical understanding that, at least given faith in mathematics, is the basis for being confident that the model is complete and correct. In other words, could we derive a set of laws that fully describe reality, and in a such way that a mathematically simpler basis for reality is provably not possible (e.g. if we found out that physics results ultimately from the simplest kind of cellular automata that is capable of leading to those laws, or to such complexity). If physics does reduce to such a mathematical understanding, then there is a viable (but yet not achieved) basis for using observation of reality's present state as evidence bearing on the question of simulation.
But certainty is not the only roadblock here. For example, because of the multiple worlds hypothesis (and probably other reasons), the teleological argument fails to lead to conclusions regarding sentient intervention, even when the present state of reality (traditionally, the teleological argument considers the presently known laws of physics as part of what is improbable [along with the material composition of the universe, and sometimes the outcome], but here we can reduce that and hypothesize that those laws are actually the outcome [thus constituting part of 'the present state of reality'] of the unknown and more-base actual laws) is improbable given its laws. Of course, since we do not yet know the laws of our reality, we may propose simulation scenarios that lead to hypotheses regarding what we would expect those laws to be, and if those hypotheses are supported (or at least not refuted) by our future findings regarding the laws, then we could have scientific evidence that reality is a simulation. However, it is important here that each scenario in question leads to unique predictions; predictions that would not be entailed by any other (or at least any non-simulation) scenarios. Its worth noting here that, of course, proof against the improbability of the present state of reality does not constitute evidence against the simulation hypothesis.
The argument from miracles is seemingly more useful. If only a single miracle of the complex variety occurs, we may recognize the possibility that, in some catalog of infinite realities, some realities had to experience that sole miracle that 'defies' probability. If, however, we witness multiple miracles of the complex variety, then we can be very confident that something 'sentient' (or at least complex) exists outside our reality and is capable of influencing our reality. The question then is one of interpreting the meaning of miraculous influence: if something exists outside our reality, is its relationship one of being the substrate or genesis of our reality, or is it simply an independent entity capable, for some reason, of interaction--and does that interaction simply imply a further set of laws (laws of our reality) that govern interactions with other realities. If a reality can reach a state whereby, by virtue of the (potentially deliberate) actions of internal sentient/complex system(s), its interactions with another entity can be governed, and are governed towards the goal of influencing the outcome (state) of the other reality, how does this relate to the idea of a simulated reality?
It seems the gut human instinct is to relate worlds or universes using spatial metaphors, but space is a troubling concept even within our reality, and should probably not be recruited in reasoning about the relations between realities, worlds, universes, or whatever you want to call them. However, the relationship can be defined in other ways: does reality X influence reality Y and vice-verse? Does either influence contravene the laws of the influenced reality? Does either reality have direct access to information about the other (access that does not come from an interaction; access that does not necessarily or automatically influence the observed system during the process of observation.) Note that these refer (or at least relate) to the concepts of omnipotence and omniscience.
Ultimately, scientifically tackling the simulation hypothesis involves making testable predictions. This is complicated, for one, by the number of possible variations on the concept. If we are considering scenarios of a sentient progenitor(s) who deliberately created the simulation, then things are also complicated by issues of motive/intention and capability/knowledge. The question of motive may parallel theoretical work regarding the motives (and evolution thereof) of future AI. Substrate is potentially a factor under any scenario, though may be conceptually/metaphysically indistinguishable (at least if considering only from our side) from the laws of reality.