Unknown

"Some aspects of the future are predictable, with the right knowledge and effort; others are impossibly hard calls. Competent futurism is built around knowing the difference."

"Human intelligence is the source of all our power, all our technology.… So far, humanity has had no competitors for our special power. But what if machine minds get better than us at the thing that, up until now, made us unique?"

"If you were able to choose what an AI wants – the destinations toward which it steers – that might be good news for you. Or bad news, if you made a poor choice of destinations, or if some malicious person makes an AI that steers toward outcomes you dislike. But the problem facing humanity is not a problem of whether good people or bad people are in control of AI. No – we are facing an even harder problem: it’s much easier to grow artificial intelligence that steers somewhere then it is to grow AIs that steer exactly where you want."

"Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking that work tirelessly and generalize more accurately from less data, and that can turn all that intelligence to analyzing and understanding and ultimately improving themselves – these minds would exceed ours."

"When it comes to thinking, quality trumps quantity."

"If you were incredibly, incredibly optimistic, you might look at the differences and say: “Well, gradient descent is not the same as natural selection, so it won’t have all the same complications as natural selection. And I don’t know of any particular complications in the relationship between what AIs are trained for and what AIs end up wanting; so I don’t expect any complications.” But a blank map does not correspond to a blank territory: if you’re venturing across an unknown land mass and your map has a blank spot where you haven’t visited, it doesn’t mean you’ll see a vast empty space when you get there. If gradient descent is different from natural selection, that doesn’t mean that we should expect to see no complications, since we don’t know about any. Rather, we should expect to see new, interesting, unpredicted complications."

"History teaches that one kind of relatively easy call about the future involves realizing that something looks theoretically possible according to the laws of physics, and predicting that eventually someone will go do it. Heavier-than-air flight, weapons that release nuclear energy, rocket that go to the Moon with a person on board: these events were called in advance, and for the right reasons, despite pushback from skeptics who sagely observed that these things hadn’t yet happened and therefore probably never would."

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

"Once AIs get sufficiently smart, they’ll start acting like they have preferences – like they want things. We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way."

"With so many different hopes, surely there’s a chance that one of them will pan out? If you think reality works like that, go try to write a hundred different letters to someone with fifty billion dollars, giving a hundred different reasonable reasons you thought of why they ought to give you a hundred million dollars for your personal use. See if it works. The reason it all fails in the end is that the fifty-billionaire does not want to rationalize giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to. In much the same way, an artificial superintelligence will not want to find reasons to keep humanity around – not in the same way that humans desperately want to find reasons to be kept."

"You can’t grow an AI that does what you want just by training it to be nice and hoping. You don’t get what you train for."

"When it comes to AI alignment, companies are still in the alchemy phase. They are still at the level of high-minded philosophical ideals, not at the level of engineering designs. At the level of wishful grand dreams, not carefully crafted grand realities. They also do not seem to realize why this is a problem."

"It is not a matter of your own country outlawing superintelligence inside its own borders, and your country then being safe while chaos rages beyond. Superintelligence is not a regional problem because it does not have regional effects. If anyone anywhere builds superintelligence, everyone everywhere dies."

"When imagining some new, unprecedented piece of future history, there is a temptation to fall into imagining that it will all go sensibly, rather than the way things usually go in history books. People sometimes ask us: How could the AI companies possibly be doing this thing, if matters are as we say? And maybe the simplest real answer is: Because this is the sort of awful, sad, real situation that you read about in history books, and not in the sensible world that exists only in imagination."

"If anyone builds it, everyone dies. It doesn’t matter whether it’s built by benevolent corporations or selfish ones. It doesn’t matter whether it’s built by researchers in the East or researchers in the West. It doesn’t matter whether it’s built by reckless optimists or people who say they respect the problem. Nobody has the knowledge or skill to make a superintelligence that does their bidding."

"But if someone has read the history of engineering disasters, they should quickly realize this phase of the standard template for disaster. It’s the part where the most informed and most worried parties have to downplay their fears, because the rest of the system hasn’t caught up, and others would give them strange looks…. History is full of other examples of catastrophic risk being minimized and ignored…."

"It is as improbable that human thinking patterns mark the final limit of intelligent algorithms as it is that human neurons represent the limit of possible computing speeds."

"This is the normal way humanity learns to surmount challenges: We deny the problem, reality smacks us around a bit, and then we start treating the problem with more respect. The Titanic sank, and most people who were aboard died. But nowadays passenger ships have enough lifeboats, and nowadays if the captain said to board them then you’d board them. We don’t hype ships up as unsinkable anymore. We make a mistake the first time, and learn from it the second time. With ASI, there is no second time."

"Even in the face of superhuman machine intelligence, it can be tempting to imagine that the world will keep looking the way it has over the last few decades of our relatively short lives. It is true, but hard to remember, that there was a time is real as our own time, just a few short centuries ago, when civilization was radically different. Or millennia ago, when there was no civilization to speak of. Or a million years ago, when there were no humans. Or a billion years ago, when multicellular colonies had no specialized cells. Adopting a historical perspective can help us appreciate what is so hard to see from the perspective of our own short lifespans: Nature permits disruption. Nature permits calamity. Nature permits the world to never be the same again."

"And if a sci-fi writer tried to write that story, the audience would just be confused, because why did that happen? Isn’t that the opposite of what the AI was trained for? But reality is allowed to be like that. And we are, fundamentally, predicting that the world will not turn out like a sci-fi novel. We’re predicting that AI’s preferences will turn out to be complicated and weird."

"The experts in this field argue in opaque academic terms about whether everyone on earth will die quickly (our view); versus whether humanity will be digitized and kept as pets by AIs that care about us to some tiny but nonzero degree; versus whether there’s a 20 percent chance we die, and an 80 percent chance that superintelligence will be harnessed successfully by a corporation, which will then be able to wield its power as they see fit.… When these are the debates experts are having, you don’t have to be certain which experts are right to understand that the current situation is not okay."

"We believe the ASI alignment problem is possible to solve in principle, by the sort of people so inhumanly smart that they never optimistically believe some plan will work when it won’t."

"Musicians have been singing about stopping the violence for years and hopefully this will do the same. We all need to do something to break the violence and the stereotype that hostel people are violent people"

"Problems like this are why we say that if anyone builds it, everyone dies. If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them? That’s not a problem that anyone’s equipped to solve."

"We’re pretty sure, actually very very sure, that a machine superintelligence can beat humanity in a fight, even if it’s starting with fairly limited resources."

"The problem with making AIs want – and ultimately do – the exact, complicated things that humans want is a major facet of what’s known as the “AI alignment problem.” It’s what we had in mind when we were brainstorming terminology with the AI professor Stuart Russell back in 2014, and settled on the term “alignment.” Most everyone who is building AIs, however, seems to be operating as if the alignment problem doesn’t exist – as if the preferences the AI winds up with will be exactly what they train into it. This assumption lurks in the background whenever someone says, “The USA needs to build superintelligence before China, because we don’t trust China,” as if the factional allegiance of whoever ran the gradient descent determined what the resulting AI wanted."

"Revere wanted to respect historical truth in his plays, without any compromise or concession: the truth of facts and customs, psychological, ideological, and even linguistic truth, since language must be a destiny."

"Pathways are hard to predict. But we can predict the endpoint."

"We know, from years of talking to people about this subject, that some people are swayed by the abstract observation that a superintelligence could exploit options they didn’t even know were possible. For others, however, explanations such as these end up making them feel like we’re cheating in a child’s game of pretend. If we can’t even tell a story about how the bad guy is supposed to win, how is that convincing? We emphasize again: Reality has never been bound by that rule. Even if an Aztec soldier couldn’t have figured out in advance how guns work, the big boat on the horizon contained them anyway."

"We have prepared adequately for the game. I don’t see ourselves coming back with a loss. We are looking forward to winning the first leg and the second leg"

"The problem here is not that corporate executives might build AI servants and command them to do something monstrous. They’re not in control. It doesn’t matter whether they’re benevolent. Humanity is faced with an engineering challenge: How do we shape the preferences of AIs that we can’t understand? It doesn’t matter whether or not the engineers have an ethics team watching over their shoulder; the ethicists wouldn’t have any idea how to get an AI’s preferences to align with ours, either."

"We predict this with confidence: Once some AIs go to superintelligence – and nobody will delay much in pushing AIs that far, if in the middle of some great arms race – humanity does not stand a chance. Ends are sometimes easier to call than pathways. The only part of our story that is a real prediction is the ending – and then, only if the story is allowed to begin."

"It was a different experience acting again. But I missed it and I am going to go for more auditions now so that I can get more roles. This is just a start for me."

"Normality always ends. This is not to say that it’s inevitably replaced by something worse; sometimes it is and sometimes it isn’t, and sometimes it depends on how we act. But clinging to the hope that nothing too bad will be allowed to happen does not usually help."

"Many of these complications won’t show up in obvious, undeniable ways until after it’s too late for humans to do anything about them."

"The greatest and most central difficulty in aligning artificial superintelligence is navigating the gap between before and after.… Ideas and theories can only be tested before the gap. They need to work after the gap, on the first try."

"Speed is often better, but AI is different from nearly every problem we’ve faced so far. When missteps kill everyone, you can’t just run fast and accept a few early mistakes."

"Most powerful artificial intelligences, created by any method remotely resembling the current methods, would not choose to build a future full of happy, free people. We aren’t saying this because we get a kick out of being bleak. It’s just that those powerful machine intelligences will not be born with preferences much like ours."

"From each of the four curses we named, we draw these lessons: 1. An engineering challenge is much harder to solve when the underlying processes run on time scales faster than humans can react…. 2. An engineering challenge is much harder to solve when there is a narrow margin for error, especially if it’s a narrow margin between “unimpressive” and “explosive.”… 3. Self-amplifying processes, like an overheating reactor boiling off its coolant water and then overheating more, leave little room for error…. 4. Complications make engineering problems worse…."

"From these lessons in combination, we infer an additional lesson for engineers: If someone doesn’t know exactly what’s going on inside the complicated device subject to all these curses – speed, narrow margins, self-amplification, complications – then they should stop. They should shut it down immediately, at the moment the behavior looks strange; don’t wait until the behavior becomes visibly concerning."

"I am also very happy with the clean sheet—it’s something we take pride in as a team. We are now looking forward to the next game with confidence and belief."

"A human brain can learn to navigate wider-ranging paths through a larger cross-section of reality than any other animal. That is our special power."

"The preferences that wind up in a mature AI are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained."

"The responsibility of having someone who is reliant on you for everything is both totally awe-inspiring and overwhelming all at the same time."

"The rise and rise of Otsweletse Moupo 23 August 2006"

"We leverage artificial intelligence to streamline course creation and make education more accessible and adaptable."

"Mexty is a no-code, AI-powered authoring platform that helps educators, instructional designers, trainers, and parents create interactive, personalized learning content in minutes"

"I think the creation of a professional franchise women's league in South Africa needs to start happening in the next year or two. I think that is the only way we can start bridging the gap between our provincial set-up and the national team"

"We need to play more competitive franchise or club cricket throughout the year, and we can't just rely on four players that go to the WPL, the Hundred, or the Big Bash"

"Moupo weathers the storm 23 October 2006"