"Firstly, I was never into marriage. Even at home, they were shocked when the letter came. I always said I didn’t see myself getting married. I wanted kids, but never marriage. I also don’t have a really good reference of marriage."

"Code of conduct for councillors underway - Siele 14 May 2014"

"You can’t grow an AI that does what you want just by training it to be nice and hoping. You don’t get what you train for."

"Many of these complications won’t show up in obvious, undeniable ways until after it’s too late for humans to do anything about them."

"Problems like this are why we say that if anyone builds it, everyone dies. If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them? That’s not a problem that anyone’s equipped to solve."

"And if a sci-fi writer tried to write that story, the audience would just be confused, because why did that happen? Isn’t that the opposite of what the AI was trained for? But reality is allowed to be like that. And we are, fundamentally, predicting that the world will not turn out like a sci-fi novel. We’re predicting that AI’s preferences will turn out to be complicated and weird."

"The preferences that wind up in a mature AI are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained."

"But if someone has read the history of engineering disasters, they should quickly realize this phase of the standard template for disaster. It’s the part where the most informed and most worried parties have to downplay their fears, because the rest of the system hasn’t caught up, and others would give them strange looks…. History is full of other examples of catastrophic risk being minimized and ignored…."

"This is the normal way humanity learns to surmount challenges: We deny the problem, reality smacks us around a bit, and then we start treating the problem with more respect. The Titanic sank, and most people who were aboard died. But nowadays passenger ships have enough lifeboats, and nowadays if the captain said to board them then you’d board them. We don’t hype ships up as unsinkable anymore. We make a mistake the first time, and learn from it the second time. With ASI, there is no second time."

"The experts in this field argue in opaque academic terms about whether everyone on earth will die quickly (our view); versus whether humanity will be digitized and kept as pets by AIs that care about us to some tiny but nonzero degree; versus whether there’s a 20 percent chance we die, and an 80 percent chance that superintelligence will be harnessed successfully by a corporation, which will then be able to wield its power as they see fit.… When these are the debates experts are having, you don’t have to be certain which experts are right to understand that the current situation is not okay."

"With so many different hopes, surely there’s a chance that one of them will pan out? If you think reality works like that, go try to write a hundred different letters to someone with fifty billion dollars, giving a hundred different reasonable reasons you thought of why they ought to give you a hundred million dollars for your personal use. See if it works. The reason it all fails in the end is that the fifty-billionaire does not want to rationalize giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to. In much the same way, an artificial superintelligence will not want to find reasons to keep humanity around – not in the same way that humans desperately want to find reasons to be kept."

"When it comes to AI alignment, companies are still in the alchemy phase. They are still at the level of high-minded philosophical ideals, not at the level of engineering designs. At the level of wishful grand dreams, not carefully crafted grand realities. They also do not seem to realize why this is a problem."

"Speed is often better, but AI is different from nearly every problem we’ve faced so far. When missteps kill everyone, you can’t just run fast and accept a few early mistakes."

"The most fundamental fact about current AI’s is that they are grown, not crafted. It is not like how other software gets made – indeed it is closer to how a human gets made, at least in some important ways. Namely, engineers understand the process that results in an AI, but do not much understand what goes on inside the AI minds they manage to create."

"Human intelligence is the source of all our power, all our technology.… So far, humanity has had no competitors for our special power. But what if machine minds get better than us at the thing that, up until now, made us unique?"

"Humanity does not need to understand intelligence, in order to grow machines that are smarter than us."

"Once AIs get sufficiently smart, they’ll start acting like they have preferences – like they want things. We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way."

"Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking that work tirelessly and generalize more accurately from less data, and that can turn all that intelligence to analyzing and understanding and ultimately improving themselves – these minds would exceed ours."

"It is not a matter of your own country outlawing superintelligence inside its own borders, and your country then being safe while chaos rages beyond. Superintelligence is not a regional problem because it does not have regional effects. If anyone anywhere builds superintelligence, everyone everywhere dies."

Unknown

Languages

Top Categories

Timeline

Related Authors

All Quotes by This Author