6

Feyerabend was critical of the scientific method and claimed in his book "Against method" that "anything goes". If I understood correctly he meant that there is no single scientific method,but he does not specify common components of every scientific method either,thus concluding that "anything goes". Are the following observations evidence that Feyerabend is wrong?

  • The replication crisis demonstrated that a huge amount of finding in the social as well as natural sciences cannot be replicated. Isn't the solution to the replication crisis more stringent statistical methodology?

  • Also Feyerabend never specified common elements of scientific methodologies such as observation, forming of hypothesis or data analysis which enabled the majority of scientific discoveries. There are common elements to all scientific methods, contrary to Feyerabend.

  • One might find suitable theories through mathematical manipulation of existing theories. However these theories will be tested against the physical world, thus although one might argue to have found truth before testing, we cannot know if these theories really hold up to reality before testing them.

  • Non-mathematical theorizing can discover "trivial" workings of reality. However this seems only to work for low hanging fruits and usually does involve at least observation of the physical world plus a lot of undocumented reasoning.

CuriousIndeed
  • 173
  • 11

3 Answers3

11

The most widespread opinion of Feyerabend is that his basic thesis is correct, that there IS no single method that applies to all of science, but that his inference to anarchism is an over extrapolation. There are instead multiple useful methods, each of which is generally useful across a range of different sciences. And this collection of methods have a familial resemblance that involves a respect for the pursuit of truth, fallibilism, and peer evaluation/critique.

However, the thesis of this question is incorrect. The replication crisis really has nothing to do with Feyerabend, and rigorous statistics will have no effect on it. One of the major features of the replication crisis is that experiment-experiment results tend to have mean solutions that fall well outside the expected range one would predict from the prior published test results. This was the surprising conclusion from multiple revisitations to historic drug effectiveness tests. More careful analysis of the prior test results would not have affected that the test-test variability is not well predicted by the experimental variability within a particular test series.

The basic cause of the replication crisis, is that science simply does not do enough replications. And repeated testing, in different labs, with slightly different protocols and experiment teams, is the best way for science to sort valid from spurious inferences. Revisiting the same question multiple times, demonstrably does not lead to the same conclusion each time. And one will not discover this, unless one actually does multiple replications of interesting results. An interesting corollary is that this principle applies to surprising UNinteresting results too. Failed experiments have not been the focus of the replication program, but the same effect of experimental variability likewise applies to them, where there are likely a large number of false negatives that were just due to bad luck in experimental variability.

There are a variety of science practices that lead to the replicability crisis.

  • Almost all academic original research is from PhD programs. And every PhD awarded has to be for original work, which means that NONE of the primary output of academic research will be replications.
  • Most journals will only publish "interesting" work -- IE only statistically significant, original, and with positive outcomes. IE, if a study has a negative or indeterminate outcome, or is a replication, it will generally be rejected for publication. This drastically skews the published literature toward innovative research rather than completing a database on a subject, an over-focus on statistical significance, when even real effects are very often not significant in every test, and a massive discouragement of ever doing replications.
  • The best funded studies, with larger sample sizes, etc, are from commercial organizations. And they will generally only report results when they are commercially useful -- IE successful applications of a commercial product. This drastically skews the published literature on these commercial products.

One of the concerns of critics of science has been about "P-hacking" where researchers multiply reanalyze their data until they finally find an effect that meets a 95% P criteria. The focus on statistical rigor, and bayesian analysis, is intended to address this supposed problem. However, this "problem" is just a sociological reaction to the publication policy of journals. Until journals publish indeterminate and replication studies too, variations of "p-hacking" will continue to skew the science database.

And setting a more stringent criteria for publishing, which is the point of the Bayesian vs frequentist approach, will not do anything prevent the lucky statistical accident effect which is so much a part of the replication crisis. What it WILL do, is drastically increase the number of false negatives, where a real effect exists, but is never explored because the publication criteria for the first time it was observed prevented publishing the interesting hints that an indeterminate but suggestive initial investigation provided.

Dcleve
  • 9,612
  • 1
  • 11
  • 44
4

There are more than a dozen different explanations for the replication crisis. Different explanations require different responses. For example, publish or perish incentive structures wouldn't be addressed by encouraging psychologists to be more statistically rigorous. I list some other explanations on slide 6 of this presentation: https://drive.google.com/file/d/11LpcCZlPKg-kzK_WhCv6rsy7PXLI5-4v/view?usp=drivesdk

Computer simulations and formal mathematical models are used in fields such as climate science, evolutionary game theory, cosmology, and computational chemistry. The relationship between these models and observation is complex and often indirect, in part because these fields often study phenomena that can't be observed directly (eg, because of time scales or abstraction). The definition of "observation" has to be stretched very thin to fit a generalization like "all sciences use observation." There are similar problems with talking about hypotheses or mathematical theories. Paleontology and archaeology often can't operate in a hypothesis testing way because they can't produce data on demand. The big claims made by these historical sciences might be qualitative explanations of particular events (what caused the KT extinction?) rather than mathematizable generalizations.

In general, it's extremely difficult to find substantive commonalities across all scientific fields.

Dan Hicks
  • 2,457
  • 7
  • 25
  • 1
    Appreciate the answer. Considering page 6 of your presentation, as I understand you list these as factors which could be improved implying some value system what is better for establishing truth..Isn't that against "anything goes"...(I absolutely agree with these recommendations however I'm not sure Feyerabend would have) – CuriousIndeed Jan 09 '22 at 21:44
  • Feyerabend's argument is that we need methodological anarchism *because* the aim of science is truth. The basic argument strategy of Against Method is to show that, for any given epistemological principle, some historical scientist made progress towards truth by violating that principle. – Dan Hicks Jan 10 '22 at 15:54
  • Couldn't findings which violated good practice also be by chance or because they were low hanging fruits?? For example, I could find an effect with a very small sample size, but it doesn't follow that sample size is not important, and obviously one should not recommend a small sample size.. – CuriousIndeed Jan 10 '22 at 16:02
  • @Rubus -- I can't speak for Dan Hicks, but will try to answer here anyway. YES, there are better practices, and larger sample sizes are among them. BUT -- many sciences cannot DO sample size easily -- Astronomy, anthropology, archaeology, sociology, economics, geology -- many sciences are primarily observational. And for, say, medicine -- case studies which are very small sample size are often very illuminating/suggestive of problems with treatments, and ways to improve them. Generalities have exceptions. Feyerabend focused on these valid exceptions to an excessive degree. – Dcleve Jan 10 '22 at 18:48
  • ok..but shouldn't it then be "anything goes but only if there aren't better alternatives"..I mean clearly there is a hierarchy of evidence... – CuriousIndeed Jan 10 '22 at 19:29
  • @Rubus -- Mostly yes. I see this as part of an overall problem with philosophy to embrace "logic" too much, such that black/white absolutes are embraced when reality is shades of grey. Logical positivism tried to do science per absolutes. Popper and Lakatos came up with more pragmatic/judgement approaches, but then each of them tried to formalize their criteria -- unsuccessfully. Dealing with shades of grey can be guided by better practices, but sometimes one has to abandon them based on good judgement. So your "only if" is too stringent -- sometimes even when available, guidelines fail. – Dcleve Jan 10 '22 at 20:21
  • "Dealing with shades of grey can be guided by better practices, but sometimes one has to abandon them based on good judgement" -> Could you make an example where abandoning the hierarchy of evidence preemptively would be justified? – CuriousIndeed Jan 10 '22 at 20:57
  • @Rubus -- the "hierarchy of evidence" tables are primarily geared toward medicine -- they often are difficult to apply in other fields. Within medicine, I noted one exception -- that the level of insight one gets from case studies, is generally far greater than for double blind studies, which are generally better for nailing down the magnitude of an effect, not initial characterizing of it. For another in CURING a patient, the rigid protocols of an experimental procedure should generally not be treated as rigid. – Dcleve Jan 10 '22 at 21:21
  • But evidence from case studies is only useful if it can be verified with stronger study design. See for example this paper (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1124898/) It clearly states several observational studies which found strong effects, which however could not be replicated in randomized controlled studies. Yes case studies might point us in the right direction (hypothesis generating studies). When is it "enough" to only do observational studies..I would argue never if RCT are possible... – CuriousIndeed Jan 10 '22 at 21:29
  • I created a chat forum to continue this discussion: https://chat.stackexchange.com/rooms/133079/chat-for-discussing-replicability-crisis – Dcleve Jan 11 '22 at 00:19
  • @Rubus -- forgot to "at" you. – Dcleve Jan 11 '22 at 01:16
1

Agreeing with @Dcleve's answer, I would like to add to that.

Feyerabend's thesis(1) of "anything goes" means, among other things, that there is no a priori reason that throwing a coin in order to derive natural laws or decide actions is on the average worse(2) that what is done using scientific methods (where applicable). In fact there is a non zero chance this random walk on possible actions can arrive at more profound and deeper outcomes.

Also there is no a priori reason that effective cures for diseases cannot be found through, for example alternative conceptions of reality, like myths and story telling, than "narratives" based on molecular biology.

Another reason is that cultural hegemony should not blind us from alternative but equally effective conceptions of physical reality and methodology.

It is a fact that a given outcome may be arrived at through different routes.

So for these and other reasons Feyerabend's thesis cannot simply be dismissed. Even if one adheres to the scientific method, it is still beneficial to keep in mind that anything goes, as it can have a positive effect on scientific narrow-mindedness which can be detrimental to the pursuit of knowledge.

References

  1. Against Method
  2. NFL theorem
Nikos M.
  • 2,113
  • 1
  • 11
  • 18
  • 1
    While there are examples that neglect of the scientific method yielded valid findings, the overwhelming majority if scientific progress has been acomplished because of adherences to scientifc standards..Also finding which have been found by unorthodox methods usually were low hanging fruits. Can you name one finding in the modern science (last 50 years) that has been found due to unorthodox methods? – CuriousIndeed Jan 13 '22 at 15:42
  • 1
    "that there is no a priori reason that throwing a coin in order to derive natural laws or decide actions is on the average worse that what is done using scientific methods (where applicable)." you can also write a novel by randomly generating words until the novel makes sense (inifite monkey theorem) however it will take like forever compared to a target oriented approach – CuriousIndeed Jan 13 '22 at 15:46
  • There are various examples of breakthroughs or intuitions that have been found by unorthodox methods even by chance. But have been reformulated in standard scientific terms. A sampling of the history will provide examples. But you fail to account for the hegemony of a certain approach in the last 400 years. So any comparison is unfair. In any case it does not diminish Feyerabend's thesis. – Nikos M. Jan 13 '22 at 15:49
  • Can you name one finding in the modern science (last 50 years) that has been found due to unorthodox methods? When sb. does personal experimentation it has to be replicated due to various methodological problems with personal experimentation..selection bias is one. non-existant sample size is another. etc etc..it's not as if results from personal experimentation are equally valid as double blinded randomized control trials..yes you can find truth through personal experimentation, however until validated it should be regarded with skepticism...also there is no "hegemony of science".. – CuriousIndeed Jan 13 '22 at 15:54
  • there is a hierarchy of evidence simply because findings confirmed by RCT are more likely to be true than findings from personal experimentation – CuriousIndeed Jan 13 '22 at 15:54
  • Feyerabend's thesis is analogous to [no free lunch theorems](https://en.m.wikipedia.org/wiki/No_free_lunch_theorem) in optimisation and decision theory. That is, under broad conditions, no method is on average better than random choice. So Feyerabend's thesis simply cannot be dismissed on principle as unreasonable. You try to dismiss it based on the last 50 years of scientific discoveries, but this is not enough, based on unfair comparisons. Hope this is clear. – Nikos M. Jan 13 '22 at 16:42
  • It's not analogous because comparing optimisation algorithms to each other is not analogous to comparing anything to the scientific method..You can choose an arbitrary timeframe and the scientific method always vastly outperforms chance findings / findings by unorthodox methods by orders of magnitude..saying that there have been chance finding / findings by unorthodox methods is not the same as saying no method is superior... – CuriousIndeed Jan 13 '22 at 17:14
  • If you read the Wikipedia entry linked you will see applications in n scientific method and "anti scientific methods" as well. – Nikos M. Jan 13 '22 at 17:15
  • *The contribution of NFL is that it tells us choosing an appropriate algorithm requires making assumptions about the kinds of target functions the algorithm is being used for. With no assumptions, no "meta-algorithm", such as the scientific method, performs better than random choice." The assumption is that we use a method based at least on observation rendering your theoretical argument invalid... – CuriousIndeed Jan 13 '22 at 19:43
  • There is no a priori reason to make that assumption and not another one based eg on beauty. At least as things stand up to now, unless one has a compelling proof otherwise, the argument holds. Many would like to get rid of NFL but the problem lies in disproving its application in the first place. As far as we know it may hold for physical reality. – Nikos M. Jan 13 '22 at 22:15
  • 1
    @NikosM. -- The scientific method has been developed to deal with this world, not with an arbitrary abstract collection of all logical possibilities. The scientific method assumes predictability, and predictability involves the assumption that phenomena behave without multiple radical discontinuities in their behavior, and that most phenomena are timewise stable, and not chaotic. These assumptions appear to be true of our universe. Assuming they will continue to be true -- is fine by pragmatism. Not for logicians. We are talking how to do science. So a pragmatic approach is justified. – Dcleve Jan 13 '22 at 22:51
  • @Dcleve true I agree, the comments got deranged a bit on various possibilities. However the original thesis of Feyerabend is that in order to do real progress one has to be open and not dogmatic. This is the true essence of "anything goes" as summarized in the last part of my answer. – Nikos M. Jan 13 '22 at 23:17
  • @NikosM. -- Feyerabend was advocating for how to do science, which is a pragmatic activity. Science cannot be rationally justified, the justification becomes circular per the Munchausen Trilemma. The NFL Theorem is a rational argument, addressing theoretical possibilities, hence is simply a restatement of the inability of science to be rationally justified. It does not support Feyerabend, who was arguing based on exceptions to any particular methodology having actual pragmatic success in our world. – Dcleve Jan 13 '22 at 23:29
  • As an aside -- I actually agree with Popper and Lakatos, that there ARE useful methods to science, and those two have articulated them. But agree with Kuhn that science itself does not always follow those methods. I also partially agree with some of Feyerabend's points -- that sometimes those methods can profitably be violated. Although I lean far closer to Dan Hicks argument, that those methods need to be customized for each science. – Dcleve Jan 13 '22 at 23:35
  • @Dcleve I agree on most, except the relevance of NFL theorem. When science is viewed as a decision process or search process in order to find or classify data, the relevance of NFL, at least on principle, is clear. – Nikos M. Jan 14 '22 at 10:22