Thoughts On AI Risk

There are two ways that people think about AI risk and I am not a big fan of either of them.

One is the “paper clip” scenario. Some group of software engineers is working on AI, and all of a sudden, it works. Boom, there is now a superintelligent AI, it’s so much smarter than people it takes over the world. Now it does whatever it wants, probably some crazy thing like turning the world into paper clips. The solution is to develop better programming methods, so that when you build an all-powerful AGI (artificial general intelligence), it will do good for humanity, rather than turning us all into paper clips.

The other is the “management consultant” method of analyzing risk. You basically list off a bunch of different practical problems with AI that don’t sound like science fiction and think about it the same way that you would think about the risk of currency exchange rate fluctuation. The solution is to run your business in a slightly different way.

I do think that AI has the potential to become smarter than humans, and a world where that happened would be quite unusual, with weird things happening. I don’t think the “management consultant” method is really taking that into account. I don’t believe the scenario where AI basically works, but its impact on the world is no greater than the fax machine’s.

On the other hand, I don’t believe that AI will make a quantum leap to generic superhuman ability. Computers tend to be very good at some things, and very bad at other things. Computers can destroy me at chess, but no robot can come close to my performance at folding laundry. The sort of mastery of the real world required to turn the planet into paper clips is a lot harder than folding laundry.

It’s possible that AI will be superhuman in every way, at some point. But if we go down the path to get there, I think we will first encounter other world events that revolutionize our understanding of what AI is capable of.

The AI Apocalypse

Generic superhuman intelligence is just not necessary for AI to destroy the world. There are several ways for just a superhuman narrow AI to cause an apocalyptic disaster.

How can a superhuman narrow AI cause an apocalyptic disaster? One possibility is via hacking. Imagine an AI that worked like:

• Scan the internet for unprotected systems
• Hack into them
• Whatever you get access to, hold it hostage for money
• Spend that money on more computing power
• Repeat

If the AI gets access to nuclear weapons it can blow up the whole world.

This scenario does not require general artificial intelligence. The AI only needs to be a superhuman hacker. Once it has systems access, it can do the hostage negotiations with the equivalent of 80 IQ.

Another possibility is via real-world violence. Imagine an AI that worked like:

• Build some killer robots
• Capture some people
• Hold it hostage for money
• Spend that money building more killer robots
• Repeat

This seems harder, to me, because it seems harder to build killer robots than to build an AI hacking program. But I could be wrong.

You might wonder why anyone would build such a disastrous AI. At the beginning, it might just be a way for a criminal to make money. Build one of these AIs, hardwire it to send 10% of its take into your account, now you have a plan to make money. One day, the original creator disappears, and the killer robots just keep on going.

It does seem like a dangerous AI it likely to need access to the financial system. Cryptocurrency seems particularly risky here, because an AI can perform crypto transactions just as easily as a human can. Cryptocurrency dark markets could also enable an AI that doesn’t have to do all the hacking itself. Instead, it buys exploits with its earnings.

Don’t Worry About AGI

People worrying about AGI should be good Bayesian reasoners. If hypotheses A and B both imply an apocalypse, but A implies B, then it is more likely that B causes the apocalypse, than that A implies the apocalypse. Superhuman generic intelligence implies superhuman narrow intelligence, so fill in the blanks, A = AGI, B = narrow AI, and worry about the dangers of superhuman narrow intelligence instead.

One problem with dangerous narrow AI is that better AI methods don’t really do anything to help. The problem isn’t AI that was intended to be good and gets out of control of the humans building it. When there’s money to be made, humans will happily build AI that is intended to be evil. In general, AI research is just making this problem worse.

Instead, we should be worrying about the security of the systems that narrow AI would subvert to cause a disaster. Can we make a crypto wallet that is impossible to hack? Can we make it impossible to hack a hospital IT system? To me, research and development here seems underrated.

How To Get Promoted

Recently I read Slava’s post on How to get promoted. I have a lot of respect for Slava, but I don’t agree with this post. So I thought I would share how I think about it.

The Common Mistake

Slava is right about one important thing, that many people in Silicon Valley misunderstand what it takes to get promoted. Many people assume that promotions just come naturally to people who deserve them, and this is incorrect. The mentality here is like

1. Do good work
2. ???
3. Promotion!

To get promoted, you need to figure out Step 2. But how?

“Getting Promoted” Is Passive

I mean this in a linguistic sense! If you think about it as “getting promoted”, you are using the passive voice. You’re omitting information. The laws of nature aren’t promoting you. Some specific person at your company is the most important decisionmaker for your promotion. To get promoted, you need to understand who that decisionmaker is, and what sort of performance they want from you to get a promotion. The mentality here is like

1. Do good work
2. Convince the key decisionmaker that your work deserves a promotion
3. Promotion!

Who is the key decisionmaker? It depends on what your role is. If you’re a software engineer, the key decisionmaker is probably just your manager. Maybe it’s your manager’s manager. If your company is smaller, maybe it’s just the CEO. But if you’re trying to get promoted, you need to figure out who this person is, and what they want.

Dysfunctional Management

If you have a good manager, you can simply ask them. Ask your manager how promotion decisions are made, who the key decisionmakers are, and more specifically ask them what you need to achieve in order to get promoted.

If you have an inexperienced manager, maybe they aren’t able to explain this to you. Or, maybe you just won’t accept what your manager telling you, that you aren’t ready for a promotion because you need to do better work.

Sometimes, it sounds like your manager is asking the impossible of you, in order to get promoted. Sorry! That may or may not be your manager’s fault, or your fault. I don’t know. At least you’re having the conversation.

Sometimes it is impossible for you to get promoted on your current team. You’re a senior engineer, and you want to make staff engineer, but what you’re working on just isn’t important enough to merit a promotion. This isn’t necessarily your manager’s fault, although often a good manager can help you figure out how to navigate this.

If You Only Care About Promotions

So what’s the right strategy for someone who only cares about promotion? Slava’s thoughts:

The winning strategy is to ignore company metrics completely and move between projects every eighteen months so that nobody notices.

Wouldn’t people notice anyway? Rank and file employees will, but not the management. In a fast growing company things change very quickly.

To me, this just sounds like a case of bad management. It’s hard to manage a fast-growing startup, and bad management happens. But you won’t find bad management everywhere. And often, what seems like bad management from afar, is actually decent management grappling with a seemingly impossible problem, once you dig into the details.

Instead, I think the best strategy for someone who only cares about promotion is to do the unsexy work that management thinks is important, but is having a hard time recruiting people for. Everybody wants to work on the VR project. Nobody wants to fix the billing system. Everybody wants to add a new feature to the consumer app. Nobody wants to be responsible for fixing the database service that recently caused a big outage. Work on the important stuff that nobody wants to do.

People who do this are incredibly valuable to an organization, and usually end up rewarded. Part of being a good manager is to figure out who is willing to do the unsexy but critical work, and to reward these people when they succeed at it.

Conclusion

In summary, my career advice for opportunists is to ask your manager what it will take to get promoted, and then do it. If that isn’t working, try working on a project that nobody wants to work on, but management thinks it’s really important. The rituals and management fashions, don’t worry about it. Spend your mental energy on getting stuff done.

Knausgaard's Struggle

I just finished reading My Struggle. Definitely one of my favorite books of the past decade. Or perhaps I should say, six of my favorite books of the past decade, since it’s a six-book series. I feel compelled to write about it, but I’m not quite sure where to start.

First: what is this book? It is sort of an autobiography. Usually, an autobiography is written by someone famous, someone who has some sort of past achievement, and you are reading the autobiography to learn about this achievement. Usually autobiographies are bad, because the author is not a very good writer, they are famous for some other sort of achievement. Knausgaard is a very good writer. And when he started writing this, he wasn’t famous for anything, or at least he wasn’t anywhere near as famous as he became for writing this sort-of-autobiography. He is now famous, but pretty much because this book is so good.

Back to the point of, it isn’t one book, it is six books. Altogether they are 3600 pages long. So it is very long either for a novel or for an autobiography. Sometimes I think of it as “the book” because it kind of works like one book. It constantly goes into far, far more detail than you would intuitively expect. It reminds me of Leeuwenhoek creating one of the earliest microscopes, turning this new tool on a boring drop of pond water, and revealing a world full of crazy little monsters.

Knausgaard captures the mentality and environment from his past, dives into the details, and exposes the most embarassing, most real details. In a normal autobiographical novel, these details would be hidden away. But exposing them is precisely what Knausgaard is after. Hating and fearing his father while growing up, the disgust he feels when his grandmother gets old, the stupid things he did while chasing after girls when he was younger, having trouble masturbating, cheating on his girlfriends and wives, being cheated on himself, being depressed, his family members being alcoholics, being frustrated when his wife was depressed, being attracted to underage girls, mistreating his children, just all sorts of things that it’s hard to believe a normal person would just tell you about himself. And yet at the same time it doesn’t come off like he’s a bad person, at all. He seems very understandable and relatable. It isn’t really a high percentage of bad behavior - it’s more like every couple hundred pages you’re surprised he would admit to something.

To me these weren’t the most interesting parts, either. They are the parts that prove, in a sense, how honest he’s being, and bring more authenticity to the other parts. When he writes about struggling with writer’s block, struggling to write a great novel while spending huge chunks of time dealing with mundane family and children issues, struggling in his relationships with his extended family which are at once critical to him and the source of so much pain, these struggles in one sense are boring because nothing really happens, and in another sense fascinating because he can describe in so many ways how exactly he is feeling and how what at first seems like nothing actually breaks down into dozens of little situations.

Each of the six books covers a different time in his life, not entirely chronologically. One discusses his early childhood, one discusses the time he spent married to his first wife, one discusses the time around writing his first novel, one discusses the time spent as a schoolteacher on an isolated island. I particularly enjoyed the last book, which covers the time spent writing these books themselves. After all the struggles to get past writers’ block, the idea for these books comes almost as a coincidence, and suddenly he finds himself writing easily, writing ten pages a day when earlier in his life he had trouble writing ten pages in a summer. After decades of work, he finds overnight success. He starts to get famous, the people in his life start to get angry that he wrote about them, and finally the book ends at the very moment it is being written, with the past catching up to the present.

One part I was fascinated by is his discussion of the Nazis. After his father dies, he finds a Nazi pin in his belongings. After his grandmother dies, he finds a copy of Mein Kampf in her belongings. And that’s it. He doesn’t know anything else about it. How does he come to terms with this? Is there something to come to terms with? The last book in the series reels off into a literary-theoretical direction a few times, detached from the main autobiographical thrust, like Knausgaard has something else to come to terms with, and this is one of them. It’s like he is asking without answers, what does it mean for an average person of the 1940’s in Norway to have flirted with Nazism? Like he can’t quite bear to connect it with his own family, even after so much personal exposure, and it’s all just a surge of thoughts and questions based on a couple pieces of inconclusive evidence anyways, but he can’t just let it drop without exposing at least his own questioning.

Who should read this? I don’t know. There’s a lot of book here to read. In the end, it is a story of a man struggling to combine his intellectual career ambitions with his identity as a member of his family. By the end of the book, it blends into reality. He succeeds intellectually, in the success of these books, and his family life recedes back out of the public eye.

I feel like I know Knausgaard so well after reading these books, we have almost become friends. If you have the ability to read 3600 consecutive pages on one topic, the topic of Knausgaard’s normal-ish life, I think you will enjoy these books.

Fire Season

I moved out to California in 2004, and quickly realized that the San Francisco area had no seasons. “Winter” in January, it’s 65 degrees and occasionally rains a bit. “Summer” in July, it’s 70 degrees and never rains. That’s it. In Cincinnati you get a larger range of weather in a single day.

Now, unfortunately, we have a season. Fire season. It wasn’t always like this. A few years ago, maybe 2017, I went camping and woke up in the morning and noticed our tents were speckled with ash. Since then, every year there have been days where smoke covered the sky.

Growing up, school was canceled on snow days. Here, the kids have smoke days.

Fire season isn’t really in the traditional summer. It’s a bit later, more like September through October. Around here those months are drier and perhaps hotter than June or July. The time formerly known as “fall”. It’s appropriate to have a new name for this season, because the leaves mostly don’t fall off the trees here.

Right now, it’s a smoke day. Mid-eighties, and it’s too smoky outside to open the windows. My house, like a lot of houses around here, doesn’t even have air conditioning. The kids are all at home, because school is remote, because of coronavirus, which also means we can’t drive somewhere like an office that does have air conditioning. So it’s just everyone hanging out in this hot sweaty box, staring at the smoky hellscape outside. Waiting for the earth to fix itself.

At least the sunsets are beautiful. The smoke makes layers of glowing orange and pink and the sun itself looks like a neon ball.

A Distant Mirror: Reflections from the 14th Century

A Distant Mirror is an excellent book. It makes me want to read more history. It’s hard for me to “review” it, per se. It’s not like eating dinner, where the goal is for the dinner to taste good, and I can assess whether it tastes good, compare to the best and worst dinners of my past, and sum it up as four stars.

Instead it’s more like the title of the book itself suggests. The Middle Ages are a mirror, and we can look in that mirror to see ourselves, to see our own era from a different angle. This book provides the mirror, but if I learn something from reading the book, does that tell me more about the Middle Ages, the mirror, or the present day?

Such an alien time. The Black Death kills about a third of Europeans from 1347-1351. Enormous debates about why it happened and what should be done. Let’s convene the greatest doctors and politicans of the age. Everyone’s analysis is just completely terrible. Maybe the plague is caused by some theological error by the Pope. Maybe it’s the fault of the Jews. Maybe it’s caused by Jupiter being in a certain position. (That seems to have been the consensus of the French medical establishment.) Maybe we’re praying in the wrong way. Everyone involved, every side of the debates, was just completely wrong. Not even possessing the right mental framework to find the answer. There are zero documents from the time mentioning anything about rats or fleas in conjunction with the plague.

How could any intelligent observer of the time have been more correct? It defies my imagination to think of any plausible way. In some sense nihilism would have been more accurate than any of the prevailing belief systems.

Besides the plague, the endless wars. I used to think of the Hundreds Years’ War as like, a big war that lasted a hundred years. Learning more, it doesn’t feel like that at all. It feels more like the Middle Ages were just a mish-mash of constant little invasions and fighting here and there. Before 1430 there was no standing army in France. Lots of small armies of a few thousand people form up and disband for various reasons. All the time some city is threatened by some army, they hire a mercenary group or have a local put together an army, the threat goes away, and the newly formed army just starts wandering around extorting other cities. It might be my own inability to comprehend the politics of the time, but it just seems like chaos. Later historians sliced off one quasi-logical chunk of the chaos and called it the Hundred Years’ War.

The wars do not feel controlled. I know war always gets out of control, but the fighting units of the time seem only somewhat subordinate to the country they are fighting for. Armies get formed and operated by some individual person. Maybe for a while they fight for the King of France. But frequently there is a problem of, these mercenary-ish armies got formed up to defend against something, but now we don’t have anything to pay them for, and the King cannot simply order them disbanded. That isn’t how it’s done. The default is for them to turn into criminal gangs wandering around demanding money from the cities they pass by.

A nobleman owns territory in both England and France due to marriage, and then England and France go to war. What should he do? Ah, the perfect solution is to raise an army and go off and invade Italy. This way he has a noble excuse for neutrality. Everyone seemed happy with that solution, impressed by his chivalry.

The way wars are conducted also seems like everyone constantly overestimates their knights. Fighting sieges without bringing siege weapons, because the most important thing is the nobility of your knights. The enemy is stuck in a position where your archers can slaughter them all, but instead you keep the archers behind your knights and have your knights charge, because that is the noblest form of battle. The English destroy the French because their archers are superior, but instead of prioritizing the development of archers as the main military goal, the French don’t seem to bother. Or they bother, but do it poorly. Nothing like, hey let’s pay a good salary for good archers. No, instead let’s ban non-archery sporting events for a while, to encourage the peasants to practice their archery more.

A thousand years earlier, the Romans had half a million people in their army. Trained, made up of common people, run by military experts rather than aristocrats. All that mentality had become lost over time.

Religion, too. Was the Pope in Avignon or the Pope in Rome the true channel to God? If you wanted to be forgiven for a terrible sin like murder or blasphemy, which Pope did God want you to pay your forgiveness money to? Clearly it was one or the other and the greatest good would come if the world could just figure out which Pope was the true one, so that you didn’t have to pay off both of them just to be sure of salvation.

What does all this tell me about the modern day? It makes me wonder, how could someone looking back from the year 2650 think about our time and think, it’s funny how they had all these intense debates when both sides of the debate were just totally wrong. Not even paying attention to the most important dimensions. No ability to reach the correct answer from the state of the discourse.

In the 1300’s there are small, tiny hints at the uprising of democracy. But they are crushed and would not rise again for hundreds of years. Groups of people briefly saying, hey maybe we shouldn’t be ruled by a heriditary nobility, maybe we should have… something else? They didn’t have the time to figure things out before getting slaughtered and put back in their places.

I am grateful to be living in the Pax Americana. I hope it lasts as long as it can. One truth that holds constant throughout the ages of history is that war has a terrible cost in human lives. I don’t think we are living at the end of history, and I don’t think the Pax Americana will last forever. Totalitarianism seems to be on the retreat, but what if we have only seen the first, initial, malformed examples of what it can do, like the doomed stirrings of democracy in the Middle Ages? I don’t really want to know what comes next, what moral beliefs of our time will turn out to be quaint fictions, what hopes of mine will forever prove impossible for humanity.

So this hasn’t been a book review, really, right? I can only indirectly recommend this book, by showing a bit of how it has made me reflect, what it has made me think about. I do recommend it, though. A Distant Mirror, by Barbara Tuchman.

If nothing else, it has driven home how there are much worse things than coronavirus.