Is there a name for the phenomenon where a system that selects for one quality eventually leads to optimizing that quality at the cost of others?

Question

There's a pattern that pops up in a wide variety of systems - biological, economic, social, etc - and can occur whenever survival and propagation within the system is determined by how well members within it embody some particular quality. Probably the most well-known example is evolution, where species survive to the extent that they are capable of reproducing. This eventually leads to species like the black-tailed antechinus, where the male literally f**ks itself to death during mating season. In short, evolution may incidentally produce beautiful species or happy species incidentally, but leave the system running long enough and it will eventually sacrifice these qualities in species if doing so gives them a reproductive edge. Non-biological examples include businesses being selected for ruthless focus on profit which end up producing worse goods and services for their customers or satisfaction for their employees, and religions which focus mainly on increasing and retaining membership at the cost of spiritually enrichment for those involved.

The only place I've managed to find it discussed is this blog, where the author calls it "the multipolar trap". But that appears to be just their name for it, as searching for this term mostly just yields references to the same blog.

It seems similar to tragedy of the commons, prisoner's dilemma, and other sad phenomena that emerge from basic game theory. But none of those quite capture this pattern where the optimizing principle of the system favors the emergence of members which cast aside other, desirable qualities in favor of the one the system actually selects on.

Answer 1

You are referring to a perverse incentive, "an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers."

Also under the heading of Goodhart's law:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

Or, expressing similar sentiment, Campbell's law:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Answer 2

Caveat

As Ted Wrigley noted, there's a difference between a system that develops to have a clash in values between the part and the whole, and one that develops parts that might lead to the failure of the whole. Let's try both.

Short Answer

Is there a name for the phenomenon where a system that selects for one quality eventually leads to optimizing that quality at the cost of others?

Yes. It's called specialization. A person who spends all of their time designing databases stands to benefit from the supply-demand curve, but might find himself homeless if the tech-bubble quickly bursts and lacks the social skills to get help from others. But you seem interested in particular cases in specialization where said specialization leads to unintended/collateral negative consequences. Your examples are cast wide, so we'll examine a few cases to suggest some terms.

Long Answer

Specialists, Generalists, and Fragility

In systems thinking from an ecological perspective, the best terms might be fragility/antifragility.

A fragile system is one that develops in such a way that certain portions of a system specialize in a way to open up the system as a whole to failure, particularly through black swan events. Instrumental convergence is nothing but puffery to characterize AI systems that avoid specialization and hedge their bets using a variety of general-purpose strategies. From WP quoting Bostrom:

Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents.

Going back to the example of a database programmer, the adage cited is "The jack of all trades is a master of none, but oftentimes better off than a master of one."

In general systems theory championed early by Bertalanffy, and added to by Nassim Nicholas Taleb, the notion that a system might evolve in such a way to open the system as a whole to failure starts with the example of how it's easy to believe all swans are white, only to find out that such a claim is a failure with the advent of the discovery of a single black swan raising the specter of the scandal of induction. The black swan example is practically an idiom in epistemology.

Black swan theory is largely epistemological in so far as it seeks to characterize systems of knowledge. From WP:

Taleb's problem is about epistemic limitations in some parts of the areas covered in decision making. These limitations are twofold: philosophical (mathematical) and empirical (human known) epistemic biases. The philosophical problem is about the decrease in knowledge when it comes to rare events as these are not visible in past samples and therefore require a strong a priori, or an extrapolating theory; accordingly predictions of events depend more and more on theories when their probability is small. In the fourth quadrant, knowledge is uncertain and consequences are large, requiring more robustness.

So, in this context, an optimization in game-theoretic terms, might make a system outperform in one domain, and be susceptible to failure in another. Cognitive biases and informal fallacies are an example of this according to behavioral economist Daniel Kahneman, the main thesis of his Thinking, Fast and Slow where he defends a perspective of the human brain and mind based on evolutionary psychology. To wit:

The book's main thesis is that of a dichotomy between two modes of thought: "System 1" is fast, instinctive and emotional; "System 2" is slower, more deliberative, and more logical. The book delineates rational and non-rational motivations or triggers associated with each type of thinking process, and how they complement each other,

The usual explanation rests in the intuition of a human on the plains of Africa whose brain has to choose between paranoid and alive, and logical and eaten when the brushes are rustling and possibly populated with hungry lions. That same paranoia today might be counterproductive since human paranoia combined with in-group-out-group thinking might lead to war and death of humans who might be better off collaborating.

Competition, Cooperation, and Coopetition

In biological systems that develop to preserve some genes over others or the organism, is related to selfish gene, biological altruism, and reciprocal altruism. Sociobiology suggests that psychological altruism is a consequence of the biological sort.

In your business example, you might conceive of a system developing so that the system collapses to benefit a part of the system. When corporate profits are put above environmental sustainability (externalizing costs and Tragedy of the Commons), a business may destroy the source of it's profit, such as the eradication of the environment, as in Dr. Seuss's The Lorax, or it may destroy it's customer base by maximizing short-term profits for shareholders and destroying the company, a phenenomeon related to the golden parachute:

A golden parachute is an agreement between a company and an employee (usually an upper executive) specifying that the employee will receive certain significant benefits if employment is terminated. These may include severance pay, cash bonuses, stock options, or other benefits. Most definitions specify the employment termination is as a result of a merger or takeover,15 also known as "change-in-control benefits",6 but more recently the term has been used to describe perceived excessive CEO (and other executive) severance packages unrelated to change in ownership

In these cases, systems clearly specialize in direct relation and at the expense of parts, wholes, collaborators, or competitors. It might be possible to categorize this as some unified class of parasitism-predation. From WP's article on parasitism:

The entomologist E. O. Wilson has characterised parasites as "predators that eat prey in units of less than one".2

Summary

Specialization has three manifestations that can lead to negative outcomes in some regard. On the one hand, a system might specialize allowing it to perform optimally under one set of conditions, and fail under another. This is fragility. The second manifestation of specialization is when in the part-whole relationship, the specialization of one part benefits or externalizes expenses in relation to other parts. This is altruism and parasitism. And if it's between whole systems, the terms altruism and predation apply. Is there a single term to describe these various exemplars? I'd say corporate employment, but I don't think that's the term you're looking for.

Answer 3

There's a category error invoked here that you ought to be aware of. There are two basic dynamics that you're referring to indiscriminately:

1. The tension between specialization and adaptability which is a common feature of biological evolution. The first is a tendency to maximize utilization of best available resources, while the second leans towards sub-optimal utilization of a wide variety of resources. The first is an optimal strategy in a stable environment; the second an optimal strategy in a variable or changing environment.
2. The tension created by value-preferences within human society. These are an assortment of cognitive dichotomies: good/evil, right/wrong, better/worse, moral/immoral, etc. As with the first point, there are different strategies: some people seek out long-term values, some short-term, some settle for sub-optimal individual values to create collective values, etc. Acculturation and socialization play large roles in mediating these strategies.

However, the fact that these two general dynamics seem parallel does not imply that they are equivalent or interrelated. Evolution is value-free. The fact that (say) female praying mantises cannibalize their mates, or that male ducks attack and force sexual activity on females, isn't intrinsically bad or good except in the human mind. Mantises and ducks are not happy or unhappy with these conditions; that is simply what they do, perfectly normal, with some (perhaps deeply) historical reason why that behavior allowed those species to thrive and persist. But value-preference dynamics are an entirely different order of activity. Value-preferences do not strive to reach some optimal strategy within an established environment; value-preference dynamics strive to coerce the environment into an optimal state for some optimal state for a (perceived) optimal strategy.

I think the best term you're going to find for this concept is path dependence, though the term doesn't normally imply the negative value judgement that you are reaching for. The basic idea is that committing to a particular path lowers the costs of continuing and raises the costs of switching paths. For instance, in the earliest days of the automobile there was a competition between internal combustion engines, steam-powered engines, and (yes) electric engines. Steam-powered engines were an already-established technology, and quite efficient, but had shortcomings for personal use (for instance, a long start-up period). Electric vehicles were hampered by the poor battery technology or the late 19th century, and only found effective use on trams or trolleys that could be wired to external power sources. internal combustion engines, though dirty and inefficient, maximized convenience and utility, and so took off. After that, a range of industries shot up to provide the necessities for IC engines, and the social and economic costs of switching back to steam or electric vehicles became progressively more steep.

Part of the reason the linked author points at capitalism as a problem is that capitalism — at least in Rightist ideology — tries to reduce value-preferences to mere questions of specialization and adaptability. Hard-line capitalism is only interested in optimizing success strategies, but as with evolution that creates a value-free dynamic in which any outcome (even the extinction of the species itself) is morally neutral.