I read a lot of books, so I thought it would be fun to do some book reviews.

As I make this decision I am in the middle of reading Rationality: From AI to Zombies which is just a monstrously long and complicated book. Six parts. 1800 pages. So this is just a review of parts 1-2 which account for about the first third of the book.

This book is a collection of blog posts. To enjoy this sort of book you need to be able to enjoy a dense collection of nonlinearly organized thoughts. If you have never found a blog that intrigued you and just read all of its past posts in a sitting, this book may not be for you.

Enjoying Infinite Jest with its aggressive footnoting and endnoting is also a sign that this might be your sort of book.

If you do enjoy this sort of book, and you happen to be a manager, you might also enjoy Managing Humans.

This book is by Eliezer Yudkowsky who does several curiously nonstandard things like work on AI and write Harry Potter fanfiction.

The first two parts of this book are about rationality.

I came into this book thinking that rationality was rather overrated. I took a game theory class in college that turned me off. In particular I was disappointed by the game-theory definition of rationality which did not seem like it was always the right thing to do.

Here’s an example. We called this the “Microsoft game”, after the olden days in which dominant industry players frequently crushed newcomers just by copying their products. The way it works is, one player is Microsoft. There are thousands of other players. At each time step, one of them is chosen randomly to become a startup and challenge Microsoft.

The startup has two options:

  • The “conservative option”. The startup gets $1M, Microsoft loses $0.
  • The “aggressive option”. The startup gets $10M, Microsoft loses $1B.

The trick is, if the startup chooses the aggressive option, Microsoft then has the option to retaliate. If Microsoft retaliates, Microsoft loses an extra $1, and the startup loses all of its $10M winnings.

That’s all. The question is, what is the rational way for each side to play this game?

According to standard game theory and the theory of dominant strategies, there is no reason for Microsoft to ever retaliate, because the outcome for Microsoft is always just a dollar better when not retaliating.

The next step of standard game theory is to repeatedly eliminate dominated strategies. So if you eliminate the ability to ever retaliate, then it’s clear that all startups should choose the aggressive option. Therefore according to standard game theory the “rational” way to play this game is for startups to always be aggressive and for Microsoft to never retaliate.

In practice, this seems silly. Microsoft should obviously retaliate all the time, and in a world where Microsoft will obviously retaliate all the time then it’s clear for all startups to choose the conservative option.

I ended up arguing with my game theory professor and converging on a state where, we agreed that the “rational” thing to do according to game theory was one thing, and the intelligent thing to do in reality was something else. Which left me thinking that rationality was overrated.

The author makes a good case in this book that this is just misusing the term “rationality”. The right way to think of rationality is that if a logical argument indicates that a particular course of action is the wrong thing to do, then it should not be described as “rational”. So I now think this is just a case where game theory needs to fix a glitch.

This book contains many, many examples of ways in which you can trick yourself into believing an illogical argument, or leave a mental conclusion in place that really should be overturned on closer inspection. In particular, I think I have a tendency to underrate the likelihood of complex plans to fail for some unpredictable reason, compared to simple plans. I reflected on this tendency while reading this book and found it quite a fun experience.

I still think the author overrates rationality. Rationality is a great root philosophy when you need to determine whether a statement is true or false, or when you need to pick between a small number of clearly delineated choices. But often you need to take action when the space of possibilities is quite vast, or make a decision where your data is all so fuzzy and poorly categorized that reductionism offers little practical help. I suspect there are cases where human biases are actually really good for you, despite being irrational.

So far I would recommend this book to anyone who can stand it.