Politics as an optimization problem
Conservatism vs. Progressivism as a Learning Rate Problem
An idealized model of the philosophical differences between conservativism and progressivism is something like: Conservatism is the view that we need to preserve our culture in order to protect the success our society has already built. Progressivism is the view that it is necessary to modify our society in order to improve it and solve outstanding problems.” These platonic idealizations of conservative and progressive positions do not necessarily neatly map to party politics in the USA or anywhere else (more about this later), but they are a useful lens through which we can understand the right-left political spectrum.
There is a marked similarity between the conservative-progressive spectrum and a common challenge that arises in machine learning tasks. Many tasks that exist in machine learning (and more broadly in life) can be framed as optimization problems; we have an equation with modifiable parameters and we want to change those parameters to achieve to some optimal outcome. The “optimal outcome” is usually defined as the minimum or maximum of some utility (AKA loss AKA cost AKA energy) function. Often the only way to solve optimization problems is by an iterative process of trial and error wherein you modify the parameters in a clever way such that you improve your utility score over time.
For example, I might be baking a cake for a friend, who will rate the quality of my cake on a scale from 1 to 10 (the utility function). I want to find the quantities of eggs, milk, flour, and sugar (the parameters) that will result in my cake getting a perfect 10 (the maximum of the utility function). So I try baking a cake with 1 cup of milk, 1/2 cup of flour, 2 eggs, and a teaspoon of sugar. My friend tries the cake and gives it a score of 3. I realize that the sugar was a bit low relative to the other ingredients, so it makes sense to increase it. But the question is: how much more sugar should I put in?
In machine learning, this is known as the learning rate problem. A large learning rate would mean that I correct the sugar content very aggressively - maybe the cake wasn’t sweet at all, so I should put in 10 more cups of sugar. But if I do that, I run the risk of overshooting and making the cake too sweet. A small learning rate, on the other hand, means that I correct cautiously - maybe I only need to add another teaspoon of sugar. But this runs the risk that if I only correct by a single teaspoon each time, I’ll have to bake a whole lot of cakes before the cake is sweet enough. In machine learning it is often the case that the learning rate problem can only be solved through trial and error, though it is sometimes possible to automate the trial and error process.
The learning rate problem is analogous to the conservatism-progressivism debate. Progressivism says 'let's aggressively change things in our society so things get better!’ while conservatism says 'hold your horses, if you change things too quickly you might end up breaking the stuff that already works!'
In machine learning, as in politics, there is usually a happy medium to be found between large and small learning rates. Conservatism and progressivism in extremis are both pathological. If a society don't maintain its culture, institutions, and values over long time horizons, whatever gains were made as a consequence of those things will dissipate as the progressive zeitgeist destroys everything in the attempt to improve everything. On the other hand, a society that is perfectly conservative will never be able to solve entrenched problems or adapt to changing circumstances.
Conservatism has an additional benefit which is inherent to the choice of small learning rates. The rapidity at which societal norms change itself has a consequential impact on people’s wellbeing. People have the ability to adapt to suboptimal circumstances that are relatively constant; they can use the reliability of their expectations to work around the sub-optimality. A rapidly changing environment of social norms, even if there is ‘progress’ on some issue can be difficult to adapt to. People used to one set of norms suddenly find themselves in a world with very different expectations, which can create friction irrespective of what the new norms are.
Reactionism vs. Whig History
In addition to conservatism and progressivism, there is also the reactionary position which can be described as 'we already broke the stuff that works, let's go back to how things were then'. The reactionary position can be associated with the algorithmic concept of a ‘pocket' (e.g. 'perceptron with pocket') where you store the best solution you've seen so far and go back to that solution if you changed things but didn't find anything better.
Reactionism has its time and place as well. Although nostalgia for past times is sometimes misguided, it is disingenuous to assume a priori that the state of a society at time t+x is always better than the state of the society at time t. This perspective is sometimes called 'Whig history', which is a quasi-religious belief that societies inevitably ‘progress' morally in time. The mainstream moral-historical narrative in the West today leans Whiggish (e.g. a progression from Judaism to Catholicism to Protestantism to Enlightenment and Democracy and secular progressive atheist humanism). Whig history is vaguely reminiscent of the Darwinist idea that life evolved from simple organisms to more complex organisms and finally to man. To the extent that cultures also evolve through selective processes, a certain Darwinian logic also pertains to memetic cultural evolution. However, there are naturalistic fallacy and is-ought problems that arise from making evolutionary arguments for moral progress.
In traditional societies, one can find a historical narrative that is the polar opposite of Whig history. For example, some circles within Orthodox Judaism maintain the doctrine of yeridas hadoros (lit. 'descent of generations') which effectively states that earlier generations are religiously superior to subsequent ones, making it effectively impossible to formally overturn legal precedent from previous epochs in Jewish history. (Halachic scholars, do, however, have many informal mechanisms of getting around precedent.) (Despite the yeridas hadoros doctrine, Orthodox Judaism tends to consider recent rulings more immediately legally relevant than older canonical documents, potentially making it more conservative than reactionary.)
The Analects of Confucius are also somewhat of a reactionary document. In the analects, Confucius bemoans the loss of the virtuous Way (Tao) among the Chinese people of his time. According to Confucius, the Zhou dynasty epitomized intuitive knowledge of proper moral behavior. Confucius therefore exhorted his students to observe the ancient rituals as they had historically been observed.
Religious reaction differs from secular versions in that secular reaction almost inevitably chooses an arbitrary point in history to return to, and this usually involves a somewhat arbitrary interpretation of why that historical period was better and an arbitrary selection of aspects of that society that should be reinstated. Religious reaction, by contrast, is rooted in revelation - the belief that God at some point revealed His will for mankind, and that will is manifest in a set of canonical documents. There may have never been a point in history when the ideal manifest in the canon was perfectly observed, but this is irrelevant to the religious person. God's will is God's will irrespective of whether people ever did a good job of observing it.
Implicit in the religious reactionary's view is that there are core aspects of policy which are invariant to a changing historical landscape. You have to honor your father and mother irrespective of whether you're a medieval serf or an industrial age factory worker or a modern email-sender. It doesn't matter whether the moral zeitgeist blows in the direction of 'everyone is equal, hierarchies are bad, therefore your parents aren't deserving of special treatment.’ The religious revelatory view is that if the moral zeitgeist contradicts God's revealed will, then the zeitgeist is wrong, period.
Even for those who are not religious, the idea of having an unchanging canon has a rational appeal, even if the canon was chosen by an arbitrary process. A fixed, unchanging set of rules is a lot simpler to deal with than the ebb and flow of public opinion over the course of centuries. A canon serves as moral schelling point among people who accept it, even if people privately hold more diverse views. A canon serves as a common point of reference for everyone involved in policymaking, and it keeps everyone honest, at least to some extent, because you can’t propose something that directly contradicts it. (In the United States, the canonical status of the U.S. constitution is pretty much the only thing holding the country together at this point.)
The revelatory reactionary view is akin to the perspective that the optimization problem of politics has a known closed-form solution, at least with respect to the issues that are addressed by revelation. Moreover, the closed-form solution is invariant to changes in the environment. Although there may be nuances that are not directly addressed by the canonical revealed texts, the assumption is that deeper study thereof will yield the results to novel problems as well. As such, politics is not an optimization problem but an analytic derivation problem from first principles.
Rationalist vs. Empiricist Utilitarian Technocracy
Those who are of a technocratic, utilitarian mindset may also be sympathetic to the derivation-based approach. Setting economic policy based on economic theory treats also treats policy as a theoretical question with a closed-form analytic solution that persists across time and cirumstances. (The solutions themselves might be parameter-sensitive, but the sensitivities are known a priori; you just need to plug the numbers in.)
The empirical approach to policy, on the other hand, is more similar to the ‘optimization' strategy of conservatism and progressivism. The empiricist tries policies, ideally motivated by theory and ideally at a small scale (at first). The empiricist carefully measures the outcome of these policy trials and determines whether the policy was beneficial or harmful, then adjusts accordingly. As opposed to the conservative and progressive, the empiricist does not have a strong a priori view that 'change is good’ or ‘change is bad’, rather he believes that change is sometimes good and sometimes bad, and you have to methodically collect and analyze results before scaling an idea up.
The empricist’s attitude seems to be a sensible golden mean between progressivism and conservatism, but there are some caveats. For one thing, the effects of a policy can be notoriously difficult to measure due to a wide range of confounding factors. One major problem is known in machine learning as the credit assignment problem. Let’s say you implement a minimum wage policy in your country, and 10 years afterwards you observe that health metrics improve. It might be that your minimum wage policy is responsible for that improvement, but you also implemented 1000 other policies during those 10 years (and beforehand) and any one of them could have contributed to the improved health metrics at varying time lags. Economists do have methods of studying the impact of specific interventions, often involving fortuitous ‘natural experiments’ which avoid many potential confounds, but these tend to be difficult to find. And even if you do have convincing evidence of the effect of a particular invention in the past, it is still not guaranteed that those results will generalize. Therefore the theoretician is within bounds to tell the empiricist, ‘I don’t care how good your experiment is, as a general rule price controls cause market distortions, which always cause more harm than good in the long run, and they should never be used.’
There isn’t really a conclusion here. The easy way out is to suggest some sort of synthesis, but without specifying precisely how these different approaches interact, that’s not very helpful. I guess because policy is ultimately created by people and not by abstract philosophies, my recommendation would be that whoever holds power should have a deep understanding of all the above meta-theories and use them to make wise decisions. Unfortunately, in democracies, these meta-theories are precisely the fault lines along which party divisions occur, making such a leader unlikely to arise.