The second book of rationality
This is part 2 of 6 in my series of summaries. See this post for an introduction.
Part II
Part II
How to Actually Change Your Mind
I
|
f truth is so handy, why do we keep making the
same mistakes? This part examines why we are so bad at acquiring accurate
beliefs and how we can do better. In particular, it looks at the problems of
motivated reasoning and confirmation bias, including self-deception and
“arguments as soldiers”.
The question “what should I
believe?” always has a right answer, even under conditions of uncertainty. As
Robin Hanson says, you are never entitled to your opinion – only your best
honest effort to find the best estimate of the way things are. The
mathematically precise answer to “who of these six people has a crush on me?”
is to multiply the 1:5 prior odds with the 10:1 likelihood ratio of Bob winking
at you (the evidence, because let’s suppose people wink at you ten times as
often when they have a crush on you) to get 10:5 or 2:1 posterior odds in favor
of Bob having a crush on you.
Unfortunately it is tricky to
revise our beliefs in this idealized way. We shield our beliefs and ideologies
from evidence through webs of cheers, illusions and biases, leading to them
getting stuck in our heads. One finding from 20th century psychology
is that human behavior is often driven by story-telling and shaky reasoning,
which evolved to defend the things we care about – our in-group, world-view or
our social standing. And since these processes are unconscious, we fail to
realize what we are doing, and introspectively we feel like we “directly
perceive” things about ourselves.
We can do better by
formalizing rational belief and rational behavior in general, using probability
theory and decision theory. Probability theory defines how to reason given some
background knowledge (priors) and a new piece of evidence, to arrive at the
best set of new beliefs (posterior). Decision theory defines which action to
take given a consistent set of beliefs and preferences. Of course, humans do
not have the time, computing power or self-control to be perfect reasoners. Yet
by comparing ourselves to the standard of Bayesian rationality, we can spot
where we messed up.
As Scott Alexander notes, it
is useful to be able to use a limited amount of evidence wisely – rationality
techniques help us get more mileage out of the evidence we have. In our
personal lives, political disagreements, and philosophical puzzles, the same
mathematical rules apply. Yet often the same cognitive biases also hold sway,
and as Luke Muehlhauser writes, we are not “corrupted” by cognitive biases – we
just are cognitive biases; they are
the substance of our reasoning. But this doesn’t mean debiasing is impossible.
With effort we can bring ourselves closer to some truth.
5
|
Overly Convenient Excuses
|
This
chapter looks at cases where the available evidence is overwhelming and we
have plenty of time to think things over… and yet, errors like confirmation
bias can still take root. Although the Bayes-optimal answer is often
infeasible to compute, some questions are probabilistically clear-cut.
|
Scientific humility means double-checking your
calculations and taking extra precaution in anticipation of your own errors.
This is proper humility, and it entails not being selectively underconfident
about uncomfortable truths (like the creationist is regarding evolution). It
does not mean social modesty, nor is
it a fully general excuse not to change your mind about something. People often use social modesty as an excuse
for not even trying to be right. Use humility to justify further action (e.g.
to plan for the case that you are wrong), not to excuse laziness and ignorance.
Humility is a complicated virtue, and we should judge it by whether applying it
makes us stronger or weaker, and by whether it is an excuse to shrug.
Avoid false
dilemmas and package-deal fallacies
by spending five minutes by the clock searching for a third, superior
alternative. Don’t assume that things traditionally grouped together must
always be so, or that there are only two options. Prefer optimal policies to defensible personally convenient policies –
otherwise you may end up justifying Noble Lies! People justify Noble Lies by
pointing out their benefits over doing nothing, but if you really need these
benefits you can construct a Third Alternative for getting them. For example,
instead of encouraging kids to believe in Santa Claus, give them science
fiction novels. Instead of belief in an afterlife, seek hope in cryonics or nanotech.
Is lottery-ticket buying a rational purchase of
fantasy? Well, you are occupying your valuable brain with a fantasy whose
probability is near zero. Lotteries are a sink of money and emotional energy.
They divert hopes and dreams away from things that people might actually
accomplish, into an infinitesimal probability. Without the lottery, people
might fantasize about things that they can actually do, which might lead to
thinking of ways to make the fantasy a reality. But to overcome bias requires
deciding that the bias is bad – something lottery advocates unfortunately fail
to do.
Selling tempting daydreams that will never
happen (lottery tickets) is not a valuable service even if people pay for them.
If that were the case, you could develop a “better lottery” which pays out
every five years on average according to a Poisson distribution of radioactive
decay, such that you could win at any moment. Thus you would buy a nearly-zero
chance to become a millionaire at any moment over the next five years, and you
could spend every moment imagining that you might become a millionaire at that
moment.
Some probabilities are so small (e.g. winning
the lottery, or the odds of humans and chimpanzees having 95% shared DNA by
coincidence) that your brain can’t keep track of how small it is, just like you
can’t spot an individual grain of sand on a beach from 100 meters away. Yet people
ignore infinitesimally small probabilities by saying “there’s still a chance”,
which they pretend is worth keeping track of. They treat extremely tiny chances
as if they were more than tiny in implication. But if you’re going to ignore
uncertain evidence, why not ignore certain proof as well? Why make the
qualitative distinction?
Nothing is perfectly black or white, but this
does not mean that everything is the same shade of gray. The fallacy of gray is treating all
uncertainties or imperfections as the same. Yet there are lighter and darker
shades of gray (some very nearly white and some very nearly black), and you can
make progress on purifying your shade. Even if you cannot eliminate bias, you
can still reduce it –which is worth
doing. Nobody is perfect, but some people are less imperfect than others. Wrong
is relative. Likewise, the scientific worldview is not on the same level as
witchdoctoring or religious faith.
Laymen don’t trust science because science
admits that it’s not perfect – and therefore not an absolute authority that
must be yielded to. This is a cultural gap (inferential distance), because
those without understanding of the Quantitative Way will often map the process
of arriving at beliefs onto the social domains of authority, and ignore a
source that admits mistakes or isn’t infinitely certain. But all knowledge is probabilistic, and you
don’t need infinite certainty to get along in life. Remember, not all grays are
the same shade. You can still choose between relatively better and worse
options.
No situation would make 2+2=4 false, but your belief
that 2+2=4 is not unconditional, because entangled evidence could convince you
that 2+2=3. For example, a combination of physical observation (such as
observing that putting two more objects down beside two objects produced three
objects), mental visualization, and social agreement that XXX – XX = XX would
convince you that 2+2=3 and that something is wrong with your past or
recollection thereof. These factors are what currently convince you that 2+2=4. A belief is only really worthwhile
if you could, in principle, be persuaded to believe otherwise.
You should not be 100% confident in
propositions like “2+2=4 is always exactly true”, because the map is not the
territory and you are unlikely to be that well-calibrated. If you say that you
are 99.9999% confident in a proposition, you’re saying that you could make one
million equally likely statements and be wrong, on average, once. Furthermore,
once you assign probability 1, you can never change your mind, because
probability 1 indicates a state of infinite certainty, and Bayes’s theorem says
that such an estimate can never be changed in response to any evidence. In
fact, it would require infinite evidence to correctly attain.
In the ordinary way of writing probabilities, 0
and 1 both seem like entirely reachable quantities. But when you transform
probabilities into log odds (or odds ratios), they extend to positive
and negative infinity. With log odds, the distance between any two degrees of
uncertainty equals the amount of evidence you’d need to move from one to the
other. This implies that infinite certainty requires infinitely strong
evidence. For example, a probability 0.9 would have 9:1 odds and log odds of 10log10(0.9/0.1)
= 9.542 decibels. But a probability 1 would have log odds of positive infinity!
So for all practical purposes (where no infinity is required), we can say that
0 and 1 are not probabilities, just like you will never count an infinity of
anything.
If people really want to buy lottery tickets,
why criticize their decision? Since I, as a human, have a proper interest in
the future of human civilization, including the human pursuit of truth, this
makes your rationality my business. We should care about the
truth because the pursuit of truth makes the future of human civilization
brighter and the gameboard fairer, so that people have to win by convincing
others using science, not by setting them on fire or telling public lies. Never
respond to irrationality with violence, only with arguments and evidence!
›
6
|
Politics and Rationality
|
This
chapter deals with the angry, unproductive discussions of mainstream national
politics. Why do we take political disagreements so personally? And why do we
not become more careful and
rigorous with the evidence when we’re dealing with issues we deem important?
|
It’s hard to learn rationality while discussing
contemporary politics, because politics is like war, and we find it hard to
resist getting in a solid dig at the other side. Arguments are soldiers, and if
you don’t support all arguments on your side or attack all arguments that favor
the enemy side, it’s like stabbing your soldiers in the back. This shouldn’t be
surprising to someone familiar with evolutionary psychology. In the ancestral
environment, being on the wrong side could have gotten you killed while being
on the right side could have gotten you access to food, sex, or let you kill
your hated rival. Thus people act funny when they talk about politics. Like the
temptation of chocolate cookies, it isn’t good for you. If you must talk about
politics for the purpose of teaching rationality, use examples from the distant
past (e.g. Louis XVI during the French Revolution).
Unlike matters of fact (like biological
evolution), the evidence for policies should rarely appear one-sided to an
objective observer, because the outcomes often have surprising, double-sided
effects. Complex actions with many consequences are not tilted so far to one
side as people think – there will be arguments both for and against, so you
must integrate the evidence and do a cost-benefit analysis. Yet people deny the
drawbacks of their favored policy and the benefits of a disfavored policy. They
want debates to be one-sided. Politics is the mind-killer and arguments are
soldiers.
Lady Justice is widely depicted as carrying a
scales, where pulling one side down pushes the other side up. In debates and
sports, a point for one side is a point against the other. Whoever wins is seen
as correct on everything and whoever loses is seen as wrong on everything. However,
the facts don’t know whose side they’re on. Two logically distinct factual
questions have two different answers, yet in complex arguments, people mix them
up – for example, judging the benefits and risks of a technology (like a
nuclear reactor) by a single overall good or bad feeling. In non-binary answer
spaces where things may have many distinct and unrelated attributes, it is
irrational to add up pro and con arguments along one simplified dimension of
“good” or “bad”.
We tend to explain people’s behavior in terms
of intrinsic enduring dispositions and traits rather than situations or
circumstance, and we do the opposite for ourselves (the correspondence bias, also known as the fundamental attribution error). This happens because our judgment
is based on a single observed episode, rather than a model of the person’s
behavior in general. Furthermore, we overestimate how likely others are to
respond the same way we do (the false
consensus effect). Despite all this, everybody sees themselves as behaving
normally.
The correspondence bias is even stronger when
someone offends us; we are especially eager to attribute bad actions to bad
character – hence patriotic Americans who see the 9/11 hijackers as innately
evil mutants. But “the enemy” is the hero of their own story, and in a
different world they might be your friend. The Enemy usually acts as you might
in their circumstances, and they think that their motives are just. That
doesn’t mean they are justified or right, but killing someone in self-defense
(even if it’s the best option available) is still a tragedy.
Due to human silliness, the fact that there are
flying saucer cults is neither evidence for
nor against real aliens (as cults can
arise around almost any idea). Stupidity and human evil do not anticorrelate
with truth. Reversing the beliefs of the foolish does not create correct
beliefs, and that foolish people disagree with you does not mean you are correct.
A car with a broken engine cannot drive backward at 200 miles per hour even if
the engine is really, really broken. Reversed stupidity is not intelligence,
and “2+2=4” isn’t false just because Stalin believed it. The world’s greatest
fool may say the sun is shining, but that doesn’t make it dark out. Likewise,
smart ideas (e.g. quantum mechanics) can have stupid followers. Since even the
strongest ideas attract weak advocates, arguing against weaker advocates proves
nothing.
A good technical argument trumps expert
authority, because argument goodness screens
off (D-separates) expert belief from the truth. In the following graph,
learning the value of the Argument node blocks the path between “Truth” and
“Expert Belief”:
If we know the arguments, we
have very little left to learn from authority. Thus we can see that
P(truth|argument, expert) = P(truth|argument). This is asymmetric: if we know
the credentials, we’re still interested in hearing the arguments. In practice,
good authorities are, ceteris paribus,
more likely to know about relevant counterevidence (and have unique intuitions
about inferences). So there are many cases in which we should take the
authority of experts into account when deciding whether or not to believe their
claims – but if two speakers both present full technical arguments with
references, a physicist has at best a minor advantage over a clown (assuming we
can process the argument).
In causal graphs, nodes closer to the target of
inquiry give more information than those farther away. The more directly your
arguments bear on the original question without intermediate inferences, the
more powerful the evidence. So stay close to the original question, screen off
as many other arguments as possible, and do the calculations – because you
can’t settle a factual dispute merely by accusing the opponent of cognitive
bias. Remember, if there are biased reasons to say the sun is shining, that
doesn’t make it dark out. Keep your eye on the ball (the closer evidence).
George Orwell noted the importance of clear
thinking and the impact of words on experience. His writings on language and
totalitarianism are important to rationalists. Orwell believed that language
should get the point across and not be used to sound authoritative, to obscure
meaning, or to convey ideas without their emotional impact. Speakers may
manipulate their phrasing to alter what aspects of a situation are noticed. The
passive voice (“the subjects were administered the drug”), clichés (“stand shoulder
to shoulder”), and static noun phrases (“unreliable elements were subjected to
an alternative justice process”) all
obscure agency and concrete imagery.
Orwell opposed the euphemisms, question-begging
and vagueness of political language, because he knew that muddled thinking and
human evil intertwine. He recommends simplifying your English to make your
stupidity obvious even to yourself. Likewise, human evil is enshrouded by
biases and stupidity, therefore overcoming bias is important! Thinking that rationality and truth-seeking is an
intellectual exercise ignores the lessons of history (e.g. the famines of
Stalin and Mao); cognitive biases and muddled thinking allow people to hide
from their own mistakes and allow evil to take root.
›
7
|
Against Rationalization
|
This
chapter speaks to the problem of rationalization; in other words,
story-telling that makes our current beliefs feel more coherent and justified
without necessarily improving their accuracy.
|
Since humans are irrational to start with, more
knowledge can hurt. Knowledge of cognitive biases can give smart people more
ammunition to argue against things they don’t like, without applying it equally
to themselves. Thus they become sophisticated
arguers who suffer from dysrationalia.
To help people obtain truth and do no harm, discussions about common biases
should start by emphasizing motivated cognition and the sophistication effect: the more someone knows, the more prone
they are to biases like seeking out supportive rather than contrary sources,
spending more time denigrating contrary arguments than supportive ones, and so
on.
Contrary to public perception, a false theory can have supporting evidence and a
correct model can take a hit of
counterevidence. Any imperfectly exact model should expect occasional opposing
evidence. Due to “conservation of expected evidence”, you cannot interpret every
possible result as confirmation. The point is to incrementally shift your
belief upwards or downwards with each incoming piece of probabilistic evidence.
Rationality is not for winning debates, but for deciding which side to join –
so acknowledge small pieces of counterevidence by shifting your belief down a
little. Supporting evidence will follow if your belief is true.
It is tempting to weigh each counterargument by
itself against all supporting arguments, so that you can easily conclude that
your theory was right (and winning this kind of battle repeatedly can make you
feel even more confident in your theory). People rehearse supporting arguments
they already know to avoid downshifting their confidence. But this
double-counts the evidence, since you already took it into account when
arriving at your current cherished belief. When facing a new contrary argument,
you have to shift your probability
down!
A clever
arguer writes down their conclusion first and then constructs arguments for
it, whereas the curious inquirer
first examines all the evidence and then estimates a conclusion. The latter is
more entangled with the chains of cause-and-effect. If you first write at the
bottom of a sheet of paper, “And therefore, the sky is green!” then it doesn’t
matter what arguments you write above it afterward because the conclusion is already
correct or already wrong. Your arguments about reality do not change the
reality about which you are arguing. So be more like the curious inquirer.
If you know someone is a clever arguer for one
side (and they only tell you the evidence that they want you to hear), remember
that spoken sentences are not the facts themselves, and have been produced
following a particular algorithm. You are not helplessly forced to update your
beliefs until you reach their position. You must condition on all your evidence! This means taking
into account what they could have
told you but didn’t. It can also
involve looking at evidence from multiple parties. (But don’t double-count
known arguments for your own side or abuse the notion of evidence-filtering as
a Fully General Counterargument.)
Here are two confusingly similar words that
describe extraordinarily different mental processes: “rationality” and
“rationalization”. Rationality flows
forward from evidence to bottom line, whereas rationalization flows backward from the conclusion to selecting
favorable evidence (thus, it is anti-rationality). Rationalization is
determining your reasoning after your conclusion. But working backwards is in
vain, because you cannot “make rational” what is not already so; that is like
calling lying “truthization”. Rationalization fixes beliefs in place, but to
improve our beliefs, they must be changed. Be curious when testing hypotheses.
You cannot specify a rational argument’s
conclusion in advance, even if the conclusion
is true! The “rational argument” is just the list of evidence that convinced
you of the rational choice; you can’t produce a rational argument for something
that isn’t already rational. Whatever actually decides your bottom line is the
only thing you can honestly write on the lines above. Flowing forward to the
bottom line requires rigor, the ability to revise your beliefs, and honesty
about the real reasons for the conclusion.
When people doubt one of their most cherished
beliefs, they instinctively ask only the questions that have easy answers. Humans
tend to consider only critiques of their position they know they can defeat. For
example, some religious people only doubt their beliefs in order to defend
them, so they target their strong points and rehearse comforting replies. In
Orthodox Judaism, you are allowed to doubt, but not successfully – you may
notice inconsistencies and contradictions, but only for the purposes of
explaining them away. But to do better, you must think about the real weak and
vulnerable points of your beliefs; think whatever thought hurts the most.
Evidence may be costly to gather, yet at some
point you have to decide. Beware motivated stopping (aka motivated credulity), which is when you want the “best” current option, because the evidence you’ve seen so
far points to a conclusion you like. Also beware motivated continuation (aka motivated skepticism), which is when you
reject the current best option and suspend judgment, or insist that more
evidence is needed because the evidence you’ve seen points to a conclusion you
dislike. The motivated skeptic asks if the evidence compels them to believe;
and the motivated credulist asks if the evidence allows them to believe. For
both, the decision to terminate a search procedure is subject to bias and
hidden motives.
Often, people’s original decision process
included no search, yet they give fake (often noble-sounding) justifications
for their bottom line. They give arguments that did not actually factor into
making their decision. For example, a Christian may revere the Bible as a
source of ethical advice, even though there are many books that are superior on
that criterion. To change your mind (or genuinely justify it), you must modify
the real algorithm behind your conclusions. Whatever process you actually use to make your decisions is
what determines your effectiveness as a rationalist.
Justifications offered for rejecting a
proposition are often not the person’s true objections, which when dispelled
would result in the proposition being accepted. A true rejection is the first reason that actually caused one to reject an idea. This is mostly
pattern-recognition, hard-to-verbalize intuitions, professional zeitgeists,
long inferential distances, and so on, rather than the justifications that
people offer. For example, some people reject Yudkowsky’s transhumanist beliefs
like artificial superintelligence, stating that “he has no PhD”, when the actual reasons are more likely related
to pattern-matching with “strange weird sci-fi idea”. The true sources of
disagreement may be hard to communicate or hard to expose.
You could learn a lot about physics from a
single pebble, because everything is inferentially entangled with something
else, and there are high-level regularities in the Great Web of Causality.
Thus, there is no perfect, risk-free lie. Humans often fail to imagine all the
facts they would need to distort to tell a truly plausible lie. Sometimes,
lying about a fact will require you to lie about an entangled fact, and then
another one, and so on. Compared to outright deception, either honesty or
silence involves less exposure to recursively propagating risks you don’t know
you’re taking.
Marcus Einfeld was an Australian judge who
received numerous awards, but in 2009 was sentenced to two years in prison over
a series of perjuries and lies that all started with a £36 fine for driving
over the speed limit. Instead of paying the ticket, Einfeld lied about lending
his car to a friend on that day… except the person whose name he gave was dead;
so he said that he meant someone else, and so on his lies continued. This is a
real-life example of how people who are bad at lying leave entangled traces
somewhere until the whole thing blows up in a black swan epic fail.
Truths
are entangled in a network, thus lies are contagious: to cover up a lie about a
fact, you must lie about lots of
specific object-level facts, or lie about general laws and even the rules of
science or reasoning. A Lie That Must Be Protected is an impediment to
rationality, for once you tell a lie, the truth is your enemy: you have to deny
that beliefs require evidence, and then deny that maps should reflect
territories, and then deny that truth is a good thing. One protected false
belief can spawn Dark Side Epistemology.
Lots of memes out there about how you learn things (e.g. “deep wisdom” like
“everyone has a right to their own opinion”) originally came from people who
were trying to convince other people to believe false statements. But remember
that on the Light Side you still have to refute the proposition for itself
rather than accuse its inventor of bad intentions.
›
8
|
Against Doublethink
|
This
chapter dives deeper into the topic of self-deception.
|
Singlethink is holding a single,
non-contradictory thought in your mind at once, noticing when you are
forgetting, recalling an uncomfortable thought, and avoiding shoving things
into the corner of your mind. Singlethink is the skill of not doublethinking – Orwell’s term for when
you forget, and then forget that you have forgotten. Young Eliezer caught the
feeling of shoving an unwanted fact into the corner of his mind and resolved to
avoid doing that, as the first step in his path as a rationalist. The drive to
improve should lead you to create new skills beyond the flawed ones in your
existing art.
George Orwell wrote about “doublethink”, which
is when a person is able to hold two contradictory thoughts in their mind
simultaneously. What if self-deception makes us happy? Maybe there are happy
stupid people, but that way is closed to rationalists. Doublethink leads only
to problems. You cannot choose to be biased, because that would involve
obtaining an accurate model, forgetting it, and then forgetting that you
forgot. If you watch the risks of doublethink enough to do it only when useful,
you cannot do it. You can’t know the consequences of being biased until you
have already debiased yourself, and then it’s too late for self-deception. And
blindly choosing to remain ignorant of the consequences of being biased is not
clever second-order rationality; it’s willful stupidity.
Belief
in self-deception is
when someone honestly believes that they have deceived themselves into
believing something, so they may receive placebo benefits, but they don’t actually believe the proposition is
true. Thus, they haven’t actually deceived themselves. For example, imagine a
person who claims to be an Orthodox Jew and expects to receive the benefits
associated with deceiving themselves into believing in God, without actually
believing that God exists (because they don’t anticipate the consequences of
God existing, only the consequences of believing in God).
Some people seem to worship the worship of God or the supposed
benefits of faith, instead of God, because they believe that they’ve deceived
themselves. This is Dark Side epistemology. To fight this, explain to people
how hard real self-deception is to get away with. Deceiving yourself is harder
than it seems. For example, saying “I believe people are nicer than they really
are” means that you believe people are bad, but you believe you believe people are nice. And just knowing the
difference can make it harder to successfully deceive yourself.
Belief-in-belief can create apparently
contradictory beliefs. Moore’s Paradox
describes statements like “it is raining but I don’t believe it is”. This may
be a failure to consciously distinguish between belief and endorsement.
The latter is driven by positive affect, and usually involves quoted (rather
than unquoted) beliefs. For example, saying “I believe in democracy” means you
endorse the concept of democracy, not that you believe that democracy exists.
You should learn to recognize the feeling of believing something and
distinguish it from having good feelings about a belief.
One way to avoid deliberate self-deception is
to believe yourself incapable of doing it. For example, keep telling yourself that
your map should correlate with the territory, and otherwise has no actual
credulity – i.e. that you don’t believe it on a gut level. If you know your belief isn’t correlated to
reality, or if you know that it’s
doublethink, how can you still believe it? Deep down you’ll know that you’re
looking at an elaborately constructed false map. Believing in your inability to
deceive yourself may become a self-fulfilling prophesy, which is why it’s a
wise precaution.
›
9
|
Seeing with Fresh Eyes
|
This
chapter is about the challenge of recognizing evidence that doesn’t fit our
expectations and assumptions.
|
The anchoring
effect is when people anchor on an irrelevant number that they’ve recently
seen and then adjust upwards or downwards when they subsequently have to
estimate a particular value. This can result in estimates that are wildly off-base,
and can affect our judgments when it comes to e.g. salary negotiations or
buying a car. Watch yourself thinking and try to notice when you are adjusting
a figure in search of an estimate. Try to mitigate it by throwing away your
initial guess or imagining a new anchor (one that is too large instead of too
small, or too small instead of too large).
Cognitive
priming is when the activation
of one concept (e.g. “water”) subconsciously spreads to linked ones (“drink”,
“splash”). When you are primed with a concept, the facts related to that
concept come to mind easier. This means that completely irrelevant observations
influence your estimates and decisions, because it shifts them in a particular
direction. Even slight exposure to a stimulus is enough to change the outcome
of a decision or estimate (even if the “information” is false or irrelevant!). Like
adjustment, priming is also a mechanism for anchoring; and these contamination effects contribute to
confirmation bias. Once an idea gets into your head, it primes information
compatible with it, and thus ensures its continued existence. This is
unfortunately how our brains work.
Contamination is exacerbated by cognitive
busyness. In other words, distraction makes it harder to identify false
statements. Some experiments on priming suggest that mere exposure to a view is
enough to get one to passively accept it, at least until it is specifically
rejected. The philosopher Descartes thought that we first comprehend and
consider a proposition before accepting or rejecting it. But perhaps his rival Spinoza
was right when he suggested (in the 17th century) that we passively
accept everything we’re told and only afterwards actively reject certain
propositions.
In principle, the human brain can parallelize
an operation but must complete it in under a hundred sequential neuron spikes. Our
brains, which are slow, rely heavily on cache
lookups: we automatically complete the pattern (based on stored answers),
rather than re-computing/thinking for ourselves whether the answer is actually
true. This may be a majority of human cognition! Answers copied from others can
end up in your head without you ever examining them closely, which can make you
say things that you’d never believe if you thought them through. So examine
your cached thoughts! Try to see memes with fresh eyes and examine them
critically (e.g. “death gives meaning to life” or “love isn’t rational”).
When asked to think creatively, there is always
a cached thought that you can fall into (e.g. thinking of something that you
heard was the latest innovation). To truly “think outside the box”, you must
actually think, and strive for
properties like truth or good design. Don’t strive for novelty or creativity
for its own sake, because then you probably want merely to be perceived as original, which leads to mimicking nonconformists.
People who aim to be optimal may in the course of time attain creativity. But
there is no conveniently labeled “Outside the Box” to which you can immediately
run off. So-called “outside the box” thinking is often a box of its own.
As Robert Pirsig notes (in Zen and the Art of Motorcycle Maintenance), one way to fight cached
patterns of thought is to focus on precise concepts. Look and see freshly for
yourself, without primary regard for what has been said before. The more you
look, the more you see. This technique also helps combat writer’s block. In
Pirsig’s story, a girl didn’t know what to write for her essay on the United
States, until she narrowed it down to the upper left-hand brick of the front of
the Opera House on the main street of Bozeman, Montana.
When telling someone from the year 1901 about
our modern world, they would find it hard to distinguish truth from fiction.
Imagine trying to explain quantum physics, the internet, or other aspects of
modern technology and culture to them –our civilization would be
unrecognizable. The internet could be seen as a “network of adding machines
which transport moving pictures of lesbian sex by pretending they are made out
of numbers.” From their perspective, the truth turned out to be surprising. So one
wonders what the world will look like 100 years from now.
Storytelling is not the same as rational
forecasting. Don’t argue or generalize real-world conclusions from fictional or
imaginary evidence! The Matrix is not
even an “example” of AI, because it never happened (nor did Terminator). This fallacy is the mirror
image of hindsight bias: updating on evidence predicted or imagined but not
observed. The data set is not at all representative of whatever real-world
phenomenon you need to understand to answer your real-world question, and leads
to inadequate models. Treating these as illustrative historical cases skews the
frame of the discussion, and makes it harder to locate the correct answer in
the space of possibilities.
As mentioned, one way to fight cached patterns
of thought is to focus on precise concepts. Moreover, it is not virtuous to
give one word as many meanings as possible, or to connect everything to
everything else. A model that connects all
things, unselectively, contains the same information as a model that
connects none. Rationalists (and poets) need narrow and precise categories that
include some things and exclude others. Not every pebble is a diamond! Good
hypotheses can only explain some possible outcomes, and not others. Some people
sneer at narrowness, like the New Age gurus with the Deep Wisdom of “everything
is connected to everything else”. But it was perfectly alright for Isaac Newton
to explain just gravity but not the
role of money in human society. Narrowness is about going out and actually looking at things; and just because
people use the same word (e.g. evolutionary biology vs. “evolution” of
technology) doesn’t mean it’s the same thing!
To seem “Deeply Wise”, just find ways to violate
the standard cache with a coherent philosophy (e.g. transhumanism) that is
within one inferential step from the listener’s current state. Concentrate on
explaining those unusual but coherent beliefs well. For example, people are
familiar with the standard Deep Wisdom of “death gives life meaning”; but
unfamiliar with the transhumanist view of “death is a pointless tragedy that
people rationalize” (so it seems deep). To actually be deep, do some original seeing, think for yourself, and aim for
the optimal rather than the merely defensible.
We all change our minds occasionally, but we
don’t constantly and honestly reevaluate every decision and course of action. Only
the actual causes of your beliefs determine your effectiveness as a
rationalist, and once you can guess what your answer will be, you have probably
already decided (for better or worse). Usually we can guess what our answer
will be within half a second of hearing the question. A number of biases (like
hindsight bias, positive bias, fake causality, anchoring/priming, and
confirmation bias) cause us to change our minds much less often than we think.
We are apt to proposing solutions to tough
problems immediately. But this implies that your answer was based on little
thought, because the effectiveness of the bottom line is determined by what
happened before writing it. Premature
conclusions are likely to be wrong, and they will weaken the data set you think
about when trying to model a phenomenon. So hold off on proposing an answer.
When working in a group, people become emotionally attached to their suggested
solutions, so be sure to discuss the problem as thoroughly as possible before
suggesting any.
The
genetic fallacy is when a belief is
judged based on its origins rather than current justificational status. The original justification for a belief does
not always equal the sum of all the evidence that we currently have available.
But on the other hand, it is very easy for people to still believe untruths
from a source that they have since rejected (e.g. the Bible). So when there is
no clear-cut evidence, be very suspicious of beliefs you acquired from an
untrustworthy source! Sometimes, like when there is no good technical argument,
you do need to pay attention to the original sources of ideas.
›
10
|
Death Spirals
|
This
chapter discusses some important hazards that can afflict groups united
around common interests and amazing shiny ideas – which will need to be
overcome if we are to get the full benefits out of rationalist communities
that encourage us to better ourselves.
|
The affect
heuristic is when we make judgments based on subjective emotional
impressions of “goodness” or “badness”, instead of rational analysis. Often we
conflate unrelated aspects of something (e.g. the risks and the benefits of
nuclear power plants) into an overall good/bad feeling. The affect heuristic
comes with a set of biases. For example, experimental subjects, when offered $1
for each red jelly bean they randomly drew from a bowl, often preferred to draw
from a bowl with a greater number of
red beans but smaller proportion
(i.e. lower probability) of red beans!
There are biases you can exploit to be seen as
generous without actually spending lots of money. For example, if you buy
someone a $45 scarf (so a relatively expensive item from an ordinarily cheap
class) you are more likely to be seen as generous than if you buy them a $55
coat. This is because making something evaluable
makes it more emotionally salient, and thus attractive. Humans tend to evaluate
options in comparison to other options in the category.
Experimental subjects
preferred the ice cream from Vendor L if they
saw a single ice cream, while they preferred Vendor H if they saw both ice
creams. Similarly, a $25 candle is more memorable and impressive than a $25
shirt; you can use this trick to display your friendship.
When asked to rate how loud an acoustic
stimulus sounds, using unbounded scales (starting at zero or “not audible at
all”, but with no upper limit), different subjects give similar ratios between
the loudness of two sounds. This is even if they make up their own modulus,
i.e. a reference point that act as a fixed scaling factor. For example, if
subject A says that sound X has a loudness of 10 and sound Y has a loudness of
15, and subject B says that sound X has a loudness of 100, then it’s a good
guess that subject B will assign sound Y a loudness around 150. But without a
scale or metric for comparison, estimates vary widely. When rating a single
item (e.g. the value of punitive damages to be awarded, or time to Artificial
General Intelligence) the answers are unpredictable, because the subjects’
choice of modulus is essentially an expression of subjective attitude or
feeling.
The affect heuristic in social situations leads
to the halo effect: we assume that
“good-looking” means “good”, thus we are more likely to vote for attractive
politicians, hire attractive people, and give them lower prison sentences. We
assume greater intelligence, honesty or kindness. All positive qualities seem
to correlate with each other, whether or not they actually do. This is
supported by research in social psychology, which has also found that people do
not realize their bias and deny that physical attractiveness influenced their
judgment.
Due to the halo effect, we see people who are
strong and invulnerable (like Superman) as more courageous and heroic than
others (like police officers), despite the latter being more virtuous. Comic
books are written about superheroes who save 200 innocent schoolchildren and
not police officers saving 3 prostitutes, even though risking your life to save
3 people reveals a more selfless nature. However, we shouldn’t aim to reveal virtue, because that would
be a lost purpose. It is still better to risk your life to save 200 people than
to save three, given the choice.
The historical Jesus deserves honor – not for
walking on water or being resurrected, but for confronting a church he believed
to be corrupt. Yet an atheistic police officer (like John Perry, a
transhumanist who died when the north tower of the World Trade Center in New
York City fell) who risks his life to save others deserves more honor, because he doesn’t anticipate his soul will survive.
Risking one’s existence to save others takes more courage than someone who
expects to be rewarded in the afterlife for their virtue.
Humans can fall into a feedback loop around
something they hold dear. The halo effect, in combination with strong positive
affect, can trigger a supercritical chain reaction, whereby a “great idea”
(e.g. a political system, leader, tonic) seems to apply everywhere; and each
observation confirms it even more. Every situation that a person considers,
they use their great idea to explain. This loop can continue, until they end up
believing that Belgium secretly controls the US banking system or that they can
use an invisible blue spirit force to locate parking spots.
A Great Thingy that feels really good can lead
to a happy death spiral (even if the
thing is Science). To avoid one, you don’t have to rehearse or rationalize
reasons why the Great Thingy/Great Idea is bad. Don’t refuse to admire anything,
and don’t forcibly shove an idea into a safe box (“restricted magisterium”).
Instead, you should split the idea into smaller independent parts, consider
each additional nice claim as a burdensome detail (remembering the conjunctive bias), be curious about the
evidence, and focus on the specifics of the causal chain. Also, don’t derive
happiness from unsettled claims that “you can’t prove are wrong”.
The mistake of religion, Communism and Nazism
was that they responded to criticism with violence. For example, the Bible in
Deuteronomy says not to listen to those who want to serve other gods, but to
kill them. The death spiral goes supercritical when it feels morally wrong to
argue against any positive claim about the Great Idea (including the idea that
faith is positive). Thinking that any argument against your favorite idea must
be wrong, or that any argument that supports your favorite idea must be right,
is one of the most dangerous mistakes a human can make and has led to massive
amounts of suffering and death in world history. Remember that in a nontrivial
answer space, the vast majority of possible supporting arguments for a true
belief are false. There is never ever
an idea so true and happy that it’s wrong to criticize any argument that
supports it.
When a cult receives a shock (e.g. when a
prophecy fails to come true, or their leader is caught in a scandal), the less
fanatic and more moderate members leave first, so the remainder becomes more radical and extreme. By analogy to
physics, a trapped collection of hot atoms will occasionally lose a high
kinetic-energy atom, decreasing the average thermal energy. Thus it’s important
for groups to have voices of moderation, skepticism and opposition; to tolerate
dissent. Without those voices, the cult will slide further in the direction of
fanaticism. On the flip side, you may want to exclude technically uninformed
trolls to make progress.
The dark mirror to the happy death spiral is
the spiral of hate. A strong negative
impression of a thing can cause us to believe related negative ideas, which we
then treat as strengthening the original impression. After the 9/11 terrorist
attacks, nobody dared to urge restraint in the US’s response, because they
would have been seen as a traitor. The ensuing spiral of hate (and concern for
image) caused the US to shoot its own foot off more effectively than any
terrorist group (e.g. the resultant wars have many times more casualties). It
is too dangerous for there to be anyone in the world that we would prefer to
say negative things about (like “the suicide hijackers were cowards”) over
saying accurate things about.
The Robbers Cave experiment (in the 1950s) was
designed to investigate the causes and remedies of problems between groups. The
researchers found that mere division into two groups caused rivalry and
conflict between the groups of “campers” (who were school boys on summer camp).
Since the first meeting, the two groups began hurling insults, formed
stereotypes of each other, raided each other’s cabins, etc. Friction was
reduced when the two groups had to cooperate to address a common problem, like
a water shortage or restarting a stalled truck (an “outside enemy”). Does this
resemble modern politics?
Simply having a good idea at the center of a
group of people is not enough to prevent that group from becoming a cult. Every
cause has a natural tendency for its supporters to become focused on defending
their group. All humans are vulnerable to the flaws in reasoning (like happy
death spirals or the ingroup-outgroup dichotomy) that cause cults. All human
groups tend to become cultish, unless the entropy is constantly resisted (even
when your cause is rationality!). The question is a quantitative one: “how much
cultishness and where?” – not a
qualitative “yes or no”. You have to actually put in the work to oppose the
slide into cultishness; to resist ordinary human nature.
The Inquisitors thought they had the truth, but
they had a mindset of guarding and preserving; whereas Richard Feynman sought
to discover truth. This is an
enormous psychological difference. If you believe that you absolutely certainly
have the truth and that it must be
protected from heretics, then torture and murder follow. But if you believe
that you are close to the truth but not there yet, someone who disagrees with
you is merely wrong, not a mortal enemy. Guardians enforce authority, while
science makes progress according to criteria of goodness (e.g. replicable experiments)
rather than criteria of comparison.
Contrary to popular conception, the Nazis were
not transhumanists, because their ideals were located in the past: they
believed in a Nordic “fall from grace” (you could call them bioconservatives). The
Nazis did not want their eugenics program to create a new breed of supermen,
but they wanted to breed back to the archetypal Nordic man which they believed previously existed. They were guardians
of the gene pool. On the other hand, the Communists had their ideals in the
future (e.g. “New Soviet Man”), although they were defective transhumanists.
Objectivism is a closed philosophy, based on the authority
of Ayn Rand. It is “closed” because you cannot be an Objectivist if you
disagree with Rand’s works. The group she created became a cult. Her followers
praise “reason” and “rationality”, but don’t actually study the sciences.
Praising rationality does not provide immunity to the human trend toward
cultishness. Science has no gods, and isn’t fair: progress can only be made by
surpassing the past milestones. An aspiring rationalist in 2007 starts with a
huge advantage over an aspiring rationalist in 1957 – that unfairness is
progress. There are things we know now which earlier generations could not have
known, so from our perspective we should expect elementary errors even in our historical
geniuses. Embracing a system explicitly tied to the beliefs of one human being,
who’s dead, is somewhere between the silly and the suicidal.
Yudkowsky presents two short stories about
individuals who are concerned that they may have joined a cult. In the first
koan, a novice rationalist learns from his master to actually apply the
techniques to real-world questions, rather than merely repeating the mentor’s
words or being anxious about their self-image as a rationalist. “You may sell a
hammer for a low or high price, but its value is clear when you use it to drive
nails.” When the novice later became a master himself, he would tell his
students to “use the techniques and don’t mention them”. In the second koan,
the master made the novice wear a silly hat to convey the point that clothing
has nothing to do with probability theory. The techniques should be judged on
their own merits. When the novice later became a grad student, he would only
discuss rationality while wearing a clown suit. But did the second novice
really understand?
Solomon Asch’s experiments revealed that people
often (about 33% to 75% of the time) give incorrect answers in order to conform
to the group. When shown the following lines, nearly three-quarters of subjects
said line C is as long as X when the experimenter’s confederates said so.
The unanimous agreement of
surrounding others can make subjects disbelieve or at least fail to report
what’s right before their eyes. This by itself isn’t necessarily irrational
(because other people’s beliefs are often legitimate evidence), but conformity
dramatically drops when another person dissents first; so people probably fear
sticking out. The evidence indicates people aren’t conforming rationally, but
because they may be nervous for social reasons. Being the first dissenter is
thus a valuable service.
Group conformity can lead to pluralistic ignorance, and thus
overconfidence. In some cases, the conformity effect can be broken. Being a
voice of dissent can benefit the group, but expressing concern is often
conflated with disagreement, which is considered socially impolite and disrupts
group harmony. Remember that raising a point that others haven’t voiced is not
a promise to disagree with the group at the end of its discussion.
Unfortunately, there is not much difference socially,
so if you choose to be a dissenter, you have to accept the costs.
The modesty
argument says that if two people disagree about a question of fact, they
should each adjust their probability estimates in the direction of the other’s
until they agree. Robert Aumann proved that genuine Bayesians
cannot agree to disagree. However, in practice it can be rational not to agree,
because your collective accuracy is different from your individual accuracy
(which should be the domain of rationality). Other people may not be epistemic
peers, and following the modesty argument can decrease individual rationality,
for example when encountering a creationist.
What if everyone agrees? Being the first to rebel is hard, but you might
need to do that to be a scientific revolutionary. To be the first person in a rebellion – to be the
only one who is saying something different – is more difficult than joining a revolution or even risking
death. It doesn’t feel like going to school in black; it feels like going to
school in a clown suit. On the other hand, not every dissenting idea is good,
and too many people fake the courage of being a lonely iconoclast (in
predictable ways). There are rebellions worth joining, but going to a rock
concert is not a rebellion. Being different for its own sake is a bias like any
other.
When encountering a group that thinks something
weird, people often nervously ask, “This isn’t a cult, is it?” Nervous people
want to be reassured that they’re not joining a cult; but cultish countercultishness can distract you from focusing on your
goals. Unusualness of belief is a risk factor, not the disease itself. Also, if
someone really were a member of a cult, they wouldn’t say so (hence the
question doesn’t make sense). Thus when considering whether or not to join a
group, consider the details of the group itself – like is their reasoning
sound? Do they do awful things to their members? Cultishness is not an essence
but an attractor, fed by failure modes like group polarization, the halo
effect, uncriticality and evaporative cooling. It can affect any group, so you must be vigilant. Yet
you must also be unafraid of some uncertainty and weirdness. Otherwise, you
will be reluctant to see any hint of any cult characteristic.
›
11
|
Letting Go
|
This
chapter concludes How to Actually
Change Your Mind. Reciting the axioms of probability theory is easier
than noticing what we are really seeing. You are not a Bayesian homunculus;
you are cognitive biases. But there is a shadow of Bayesianism present in you
as well.
|
The Enron executives never admitted to making a
large mistake – yet acknowledging a fundamental problem, saying “oops!” and
making a big change, is important to quickly fixing the whole error at once
(rather than grudgingly making minimal concessions). Big improvements require
admitting big errors. When your theory is proven wrong, just scream “oops!” and
admit your mistake fully. Don’t just admit local errors, protecting your pride
by conceding the absolute minimal patch of ground; it is far better to make big
improvements quickly. This is a lesson of Bayescraft that Traditional
Rationality fails to teach.
If you think you have a revolutionary idea, try
to find a disproof and if you do, let the idea go. The young Eliezer thought he
disproved Cantor’s Diagonal Argument until he found a counterexample to his
attempted disproof, and he realized he nearly became a math crank. If you make
a mistake, don’t excuse it or pat yourself on the back for thinking originally;
acknowledge you made a mistake and move on. Don’t reinterpret your mistakes to
make it so that you were right “deep down” or half-right. If you insist on
clinging to your idea and refusing to admit your mistake, you risk becoming a
crackpot – which may be intellectually fatal. You will stay stuck on bad ideas.
The study of rationality should teach you how not to be stupid across many
disciplines. For example, knowing when to give up and admit defeat is key to
not turning little mistakes into big mistakes. For example, Casey Serin owes
banks 2.2 million dollars after lying on mortgage applications, yet instead of
losing hope and declaring bankruptcy, he tried to buy another house. Similarly,
Merton and Scholes’s Long-Term Capital
Management lost all their profits by overleveraging. Each profession has
rules on how to be successful, but rationality helps with the greater skill of
not being stupid.
Doubt is often regarded as virtuous for the
wrong reason, i.e. it is a sign of humility and recognition of your place in
the hierarchy. But from a rationalist perspective this is not why you should
doubt. Do not doubt for the sake of appearing modest or wearing rationalist
attire; doubt because you have a reason to suspect that a particular belief is
wrong. Then investigate the doubt, without fear, and either destroy the doubt or
destroy the targeted belief. The doubt should exist to annihilate itself –
either to confirm the reason for doubting or to show the doubt to be baseless. Resolve your doubts! A doubt that fails
to either be destroyed or to destroy its belief may as well not have existed at
all.
Eugene Gendlin said in a poem: “What is true is
already so. Owning up to it doesn’t make it worse. Not being open about it
doesn’t make it go away. And because it’s true, it is what is there to be
interacted with. Anything untrue isn’t there to be lived. People can stand what
is true, for they are already enduring it.” Yudkowsky calls this the Litany of Gendlin.
Curiosity is the first virtue of rationality.
Curiosity is wanting to investigate, not
wanting to have investigated (out of a sense of duty). Investigate to shift
your beliefs, not just to get it over with; otherwise you could fall prey to
motivated stopping. If you are genuinely curious, you’ll gravitate to inquiries
that seem most promising of producing shifts in belief, or inquiries that are
least like the ones you’ve tried before. And remember that on average, you
should expect your beliefs to shift by an equal amount in either direction (due
to conservation of expected evidence).
Any process you think may confirm your beliefs, you must also think may
disconfirm them. If you can find within yourself the slightest shred of true
uncertainty, then guard it like a forester nursing a campfire; and if you can
make it blaze up into a flame of curiosity, it will make you light and eager
and give purpose to your questioning and direction to your skills.
Traditional Rationality is phrased as social
rules, which allows people to use the hypocrisy of others as defense.
Violations are interpretable as defections from cooperative norms (i.e.
cheating). For example, it wouldn’t be fair to demand evidence from you if we
can’t provide it ourselves. But under Bayesian rationality, it is a law that
you need evidence to generate accurate beliefs, otherwise your mapping-engine
won’t run. No social maneuvering can exempt you from the mathematics. Even if
I’m doing XYZ wrong, it doesn’t help or excuse you; it just means we’re both
screwed.
Sometimes, before you can fairly assess
probabilities, it helps to visualize an uncomfortable state of affairs in
detail to make it less scary. Visualize what the world would look like if the
unpleasant idea were true, and what you would do in that situation. Leave
yourself a line of retreat before you
come to the battlefield, so that you’ll have less trouble retreating. This will
allow you to reason about the idea and evaluate evidence for it without
immediately trying to reject it. Planning your retreat this way doesn’t mean
you have to accept that the premise is actually true – just admit to yourself which ideas scare you.
Some false deeply-held beliefs (e.g. religion)
require you to stage a true crisis of
faith, that could just as easily go either way, in order to defeat them. An
error can build itself a fortress. Warning signs include when a belief has long
remained in your mind, gotten mixed up in your personality generally, is
surrounded by a cloud of known arguments and refutations, and has emotional
consequences. To doubt successfully, you shouldn’t rehearse cached thoughts,
but see originally. This crisis of faith
technique requires a combination of desperate effort, original seeing,
curiosity, doubting the most painful spots, singlethink, resisting affective
death spirals, visualizing the alternative, and so on. Allocate some
uninterrupted hours and find somewhere quiet to sit down. Make a convulsive,
wrenching effort to be rational. By the time you know a belief is in error, it
is already defeated.
Yudkowsky writes a short story about a
fictional Beisutsukai (members of the
Bayesian Conspiracy) master named Jeffreyssai, to illustrate a crisis of faith.
The Ritual of Changing One’s Mind is pointless if you don’t have the ability to
exit as something of a different person. (You aren’t born knowing the truth and
right of everything.) It’s best to prepare by blocking out every thought that
previously occurred to you and holding off on proposing solutions. Then
remember: That which can be destroyed by
the truth should be. People can stand what is true, for they are already
enduring it.
›
Comments
Post a Comment