In his original paper on the simulation argument Bostrom stated his main claim in terms of a collection of three different propositions:
This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation.
Using De Morgan’s laws, a claim of the form (A or B or C) is logically equivalent to a claim of the form ~(~A and ~B and ~C), and I find it conceptually clearer to think of Bostrom’s claim in these terms. If we take the negations of his three statements (and paraphrase and elaborate them a little), what he is arguing is that he thinks it’s very unlikely all three of the following statements are simultaneously true (i.e. at least one is likely false):
#1 A significant fraction of species that have reached our level of technological development avoid extinction and go on to reach a post-biological stage where they command vast computational power (enough so that they could perform vast number of ancestor simulations as described in #2)
#2 A sufficiently large fraction of post-biological civilizations with such computing power devote enough of that computing power to ancestor simulations such that non-simulated biological intelligences end up being significantly outnumbered by simulations of biological intelligences inhabiting simulations of the past before the post-biological era.
#3 We are non-simulated.
Now, there are all sorts of reasons #1 and #2 might be false, since they involve questions of futurology, technological feasibility, and sociological questions about what posthuman civilizations would tend to want to do with their computing power. But certainly a lot of transhumanists are hoping that #1 is true, since if this is a common fate for intelligence in the universe (or multiverse), then there’s a decent chance that will be our future as well. And if #1 is true, #2 seems plausible as well. So, the heart of Bostrom’s argument is that, based on a particular type of anthropic selection principle which he discussed at length in his book Anthropic Bias (available on his website here), #3 is very unlikely given #1 and #2.
Before going into his anthropic argument, I want to address two related misconceptions that I’ve seen people bring up, the second of which is most relevant to your question:
A. This is not a general argument for skepticism about our ability to know anything about external reality, like Descartes’ Demon or the brain in the vat. Instead, Bostrom’s argument is directed specifically at transhumanists and others who would find the assumption I labeled #1 plausible, pointing out that it naturally leads to seeing #2 as plausible as well, and that if we grant the likelihood of both we are forced to consider #3 unlikely. See Bostrom’s comments here in the section titled “Not the Old Brain-in-a-Vat Argument” where he says “The Simulation argument, by contrast, adopts as its starting point that things are the way they seem to be and that science gives us reliable information about the world. Part of this information concerns the technological capabilities that an advanced civilization would be able to develop.” (He makes similar comments at 5:40 in the podcast here.) Also note that Bostrom himself seems to be a transhumanist or at least keenly interested in the subject, see his writings about it here and here and here for example, along with his book Superintelligence: Paths, Dangers, Strategies.
B. If we understand Bostrom as just claiming that #1 and #2 and #3 can’t all be true, then in no way does it refute his argument to point out that if we are in a simulation, it might not be an ancestor simulation and in fact the “real world” might be nothing like the apparent world we experience (different laws of nature, for example). This is certainly possible, but by postulating that we are simulated you aren’t disputing what he says about the three propositions. The point is to show that given the premise that the laws of nature are the ones we observe, we are led to some conclusions that are apt to seem strange for those that believe in the technological feasibility of this sort of transhuman future—either that we are living in a simulation, or that there is some kind of “Great Filter” (a possibility he talks about here) that makes it vanishingly unlikely that any civilization like ours will actually reach that transhuman level, or that there are some quasi-deterministic historical laws that lead the vast majority of transhuman civilizations to avoid detailed historical simulations even when they have the capability to run them.
Also note that when presented in my modified form with the statements #1-#3 that are negations of Bostrom's original 3 statements, we can see that the argument that #1 - #3 can't all be true amounts to a probabilistic version of a proof by contradiction (I think this is one of the main conceptual advantages of presenting the argument in the modified form). In a proof by contradiction one starts from some set of premises and proves they can't all be true by showing they lead to a contradiction; similarly if we start from premises #1 - #3 we can show that #1 and #2 together lead to the conclusion that #3 is extremely unlikely. #3 said that we are not in a simulation, so negating it implies we likely are in a simulation, and that may undermine whatever reasons we had for thinking #1 and #2 were plausible in the first place (since #1 and #2 are statements about the real world, and if we decide that we're in a simulation we can no longer claim to know with any confidence what the real world is like). But it's a general feature of proofs by contradiction that they undermine their own premises, so this is not in itself a reason for doubting the argument.
I think these two points address your original objection. As for Bostrom’s anthropic argument for saying #3 is unlikely given #1 and #2, it’s basically a variation on the “self-sampling principle” he details in his book Anthropic Bias linked above (I believe Bostrom invented this term, and he is more systematic in fleshing out the rules of its use, but he didn’t invent the basic idea which was already present in some of the work of Brandon Carter, who coined the term “anthropic principle”). I discussed the self-sampling assumption and its relation to the simulation argument (as well as the related doomsday argument) in a previous answer, so I won’t repeat that here. But one extra issue brought up in the comments is that Bostrom doesn’t actually invoke the self-sampling assumption in his simulation argument paper, instead he uses what he calls the “principle of bland indifference”. This is actually just a weakened version of the self-sampling assumption, as he says on p. 9 of the paper:
The Doomsday argument rests on a much stronger and more controversial premiss, namely that one should reason as if one were a random sample from the set of all people who will ever have lived (past, present, and future) even though we know that we are living in the early twenty‐first century rather than at some point in the distant past or the future. The bland indifference principle, by contrast, applies only to cases where we have no information about which group of people we belong to.
And as I noted in a comment, Bostrom says at 1:15:21 of this podcast interview that the bland indifference principle is "weaker than the self-sampling assumption" but that they are both part of his project of "trying to figure out what are the legitimate ways of reasoning about these indexical facts when observer selection effects are in play, in other words, developing a theory of anthropics."
