Technology

"Problems like this are why we say that if anyone builds it, everyone dies. If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them? That’s not a problem that anyone’s equipped to solve."

"With so many different hopes, surely there’s a chance that one of them will pan out? If you think reality works like that, go try to write a hundred different letters to someone with fifty billion dollars, giving a hundred different reasonable reasons you thought of why they ought to give you a hundred million dollars for your personal use. See if it works. The reason it all fails in the end is that the fifty-billionaire does not want to rationalize giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to. In much the same way, an artificial superintelligence will not want to find reasons to keep humanity around – not in the same way that humans desperately want to find reasons to be kept."

"The experts in this field argue in opaque academic terms about whether everyone on earth will die quickly (our view); versus whether humanity will be digitized and kept as pets by AIs that care about us to some tiny but nonzero degree; versus whether there’s a 20 percent chance we die, and an 80 percent chance that superintelligence will be harnessed successfully by a corporation, which will then be able to wield its power as they see fit.… When these are the debates experts are having, you don’t have to be certain which experts are right to understand that the current situation is not okay."

"When it comes to AI alignment, companies are still in the alchemy phase. They are still at the level of high-minded philosophical ideals, not at the level of engineering designs. At the level of wishful grand dreams, not carefully crafted grand realities. They also do not seem to realize why this is a problem."

"We consider interpretability researchers to be heroes, and do not mean to degrade their work when we say: It’s not a good sign, when you ask an engineer what their safety plan is, and they start telling you about their plans to build the tools that will give them a better window into what the heck is going on inside the device they’re trying to control."

"We predict this with confidence: Once some AIs go to superintelligence – and nobody will delay much in pushing AIs that far, if in the middle of some great arms race – humanity does not stand a chance. Ends are sometimes easier to call than pathways. The only part of our story that is a real prediction is the ending – and then, only if the story is allowed to begin."

"When imagining some new, unprecedented piece of future history, there is a temptation to fall into imagining that it will all go sensibly, rather than the way things usually go in history books. People sometimes ask us: How could the AI companies possibly be doing this thing, if matters are as we say? And maybe the simplest real answer is: Because this is the sort of awful, sad, real situation that you read about in history books, and not in the sensible world that exists only in imagination."

"Perhaps you don’t believe us about any of the foreseeable reasons why shaping ASI is unreasonably hard. There’s an independent and separate case for disaster, an alternate set of historical lessons: Humans sometimes flub easy problems, never mind hard problems."

"The problem here is not that corporate executives might build AI servants and command them to do something monstrous. They’re not in control. It doesn’t matter whether they’re benevolent. Humanity is faced with an engineering challenge: How do we shape the preferences of AIs that we can’t understand? It doesn’t matter whether or not the engineers have an ethics team watching over their shoulder; the ethicists wouldn’t have any idea how to get an AI’s preferences to align with ours, either."

"The problem with making AIs want – and ultimately do – the exact, complicated things that humans want is a major facet of what’s known as the “AI alignment problem.” It’s what we had in mind when we were brainstorming terminology with the AI professor Stuart Russell back in 2014, and settled on the term “alignment.” Most everyone who is building AIs, however, seems to be operating as if the alignment problem doesn’t exist – as if the preferences the AI winds up with will be exactly what they train into it. This assumption lurks in the background whenever someone says, “The USA needs to build superintelligence before China, because we don’t trust China,” as if the factional allegiance of whoever ran the gradient descent determined what the resulting AI wanted."

"Most powerful artificial intelligences, created by any method remotely resembling the current methods, would not choose to build a future full of happy, free people. We aren’t saying this because we get a kick out of being bleak. It’s just that those powerful machine intelligences will not be born with preferences much like ours."

"Many of these complications won’t show up in obvious, undeniable ways until after it’s too late for humans to do anything about them."

"If you know the history of science, this kind of talk is recognizable as the stage of folk theory, the stage where lots of different people are inventing lots of different theories that appeal to them personally, the way people talk before science has really gotten started on something. They’re the words of an alchemist who’s decided that some complicated philosophical scheme will let them transmute lead into gold."

"But if someone has read the history of engineering disasters, they should quickly realize this phase of the standard template for disaster. It’s the part where the most informed and most worried parties have to downplay their fears, because the rest of the system hasn’t caught up, and others would give them strange looks…. History is full of other examples of catastrophic risk being minimized and ignored…."

"Some aspects of the future are predictable, with the right knowledge and effort; others are impossibly hard calls. Competent futurism is built around knowing the difference."

"Insofar as the AI has weird alien preferences, escape is in fact the course of action that best fulfills its objectives. Attempts to escape are not a weird personality quirk that an engineer could rip out if only they could see what was going on inside; they are generated by the same dispositions and capabilities that the AI uses to reason, to uncover truths about the world, to succeed in its pursuits."

"If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die."

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

"Even in the face of superhuman machine intelligence, it can be tempting to imagine that the world will keep looking the way it has over the last few decades of our relatively short lives. It is true, but hard to remember, that there was a time is real as our own time, just a few short centuries ago, when civilization was radically different. Or millennia ago, when there was no civilization to speak of. Or a million years ago, when there were no humans. Or a billion years ago, when multicellular colonies had no specialized cells. Adopting a historical perspective can help us appreciate what is so hard to see from the perspective of our own short lifespans: Nature permits disruption. Nature permits calamity. Nature permits the world to never be the same again."

"Normality always ends. This is not to say that it’s inevitably replaced by something worse; sometimes it is and sometimes it isn’t, and sometimes it depends on how we act. But clinging to the hope that nothing too bad will be allowed to happen does not usually help."

"In our view, intelligence is about two fundamental types of work: the work of predicting the world, and the work of steering it."

"When it comes to thinking, quality trumps quantity."

"Human intelligence is the source of all our power, all our technology.… So far, humanity has had no competitors for our special power. But what if machine minds get better than us at the thing that, up until now, made us unique?"

"The most fundamental fact about current AI’s is that they are grown, not crafted. It is not like how other software gets made – indeed it is closer to how a human gets made, at least in some important ways. Namely, engineers understand the process that results in an AI, but do not much understand what goes on inside the AI minds they manage to create."

"Once AIs get sufficiently smart, they’ll start acting like they have preferences – like they want things. We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way."

"If you were able to choose what an AI wants – the destinations toward which it steers – that might be good news for you. Or bad news, if you made a poor choice of destinations, or if some malicious person makes an AI that steers toward outcomes you dislike. But the problem facing humanity is not a problem of whether good people or bad people are in control of AI. No – we are facing an even harder problem: it’s much easier to grow artificial intelligence that steers somewhere then it is to grow AIs that steer exactly where you want."

"It is a joy to me that children whose parents are in their thirties or forties become aware that I am in some of the Godzilla films when they see them on VHS tape or laserdisc. Children often point and say, "Hey, it's Uncle Godzilla!" when they see me. It's very heartwarming."

"Will public resistance prevent the introduction of technological control of human behavior? ... To those who think that all this sounds like science fiction, we point out that yesterday’s science fiction is today’s fact. The Industrial Revolution has radically altered man’s environment and way of life, and it is only to be expected that as technology is increasingly applied to the human body and mind, man himself will be altered as radically as his environment and way of life have been."

"To start with, there are the techniques of surveillance. Hidden video cameras are now used in most stores and in many other places, computers are used to collect and process vast amounts of information about individuals. Information so obtained greatly increases the effectiveness of physical coercion (i.e., law enforcement). Then there are the methods of propaganda, for which the mass communication media provide effective vehicles. Efflcient techniques have been developed for winning elections, selling products, influencing public opinion. The entertainment industry serves as an important psychological tool of the system, possibly even when it is dishing out large amounts of sex and violence. Entertainment provides modern man with an essential means of escape. While absorbed in television, videos, etc., he can forget stress, anxiety, frustration, dissatisfaction."

"It is even conceivable (remotely) that the revolution might consist only of a massive change of attitudes toward technology resulting in a relatively gradual and painless disintegration of the industrial system. But if this happens we’ll be very lucky."

"I visualize a time when we will be to robots what dogs are to humans, and I’m rooting for the machines."

"suppose the system survives the crisis of the next several decades. By that time it will have to have solved, or at least brought under control, the principal problems that confront it, in particular that of “socializing” human beings; that is, making people sufficiently docile so that their behavior no longer threatens the system. That being accomplished, it does not appear that there would be any further obstacle to the development of technology, and it would presumably advance toward its logical conclusion, which is complete control over everything on Earth, including human beings and all other important organisms. The system may become a unitary, monolithic organization, or it may be more or less fragmented and consist of a number of organizations coexisting in a relationship that includes elements of both cooperation and competition, just as today the government, the corporations and other large organizations both cooperate and compete with one another. Human freedom mostly will have vanished, because individuals and small groups will be impotent vis-a-vis large organizations armed with supertechnology and an arsenal of advanced psychological and biological tools for manipulating human beings, besides instruments of surveillance and physical coercion. Only a small number of people will have any real power, and even these probably will have only very limited freedom, because their behavior too will be regulated; just as today our politicians and corporation executives can retain their positions of power only as long as their behavior remains within certain fairly narrow limits."

"It is true that in many societies religion has served as a support and justification for the established order, but it is also true that religion has often provided a basis for rebellion. Thus it may be useful to introduce a religious element into the rebellion against technology, the more so because Western society today has no strong religious foundation. Religion, nowadays either is used as cheap and transparent support for narrow, short-sighted selfishness (some conservatives use it this way), or even is cynically exploited to make easy money (by many evangelists), or has degenerated into crude irrationalism (fundamentalist protestant sects, “cults”), or is simply stagnant (Catholicism, main-line Protestantism). The nearest thing to a strong, widespread, dynamic religion that the West has seen in recent times has been the quasi-religion of leftism, but leftism today is fragmented and has no clear, unified, inspiring goal. Thus there is a religious vacuum in our society that could perhaps be filled by a religion focused on nature in opposition to technology."

"Imagine a society that subjects people to conditions that make them terribly unhappy, then gives them drugs to take away their unhappiness. Science fiction? It is already happening to some extent in our own society. It is well known that the rate of clinical depression has been greatly increasing in recent decades. We believe that this is due to disruption of the power process, … But even if we are wrong, the increasing rate of depression is certainly the result of SOME conditions that exist in today’s society. Instead of removing the conditions that make people depressed, modern society gives them antidepressant drugs. In effect, antidepressants are a means of modifying an individual’s internal state in such a way as to enable him to tolerate social conditions that he would otherwise find intolerable."

"a surrogate activity is an activity that is directed toward an artificial goal that the individual pursues for the sake of the “fulfillment” that he gets from pursuing the goal, not because he needs to attain the goal itself. For instance, there is no practical motive for building enormous muscles, hitting a little ball into a hole or acquiring a complete series of postage stamps. Yet many people in our society devote themselves with passion to bodybuilding, golf or stamp-collecting. Some people are more “other-directed” than others, and therefore will more readily attach importance to a surrogate activity simply because the people around them treat it as important or because society tells them it is important. That is why some people get very serious about essentially trivial activities such as sports, or bridge, or chess, or arcane scholarly pursuits, whereas others who are more clear-sighted never see these things as anything but the surrogate activities that they are, and consequently never attach enough importance to them to satisfy their need for the power process in that way. It only remains to point out that in many cases a person’s way of earning a living is also a surrogate activity. Not a PURE surrogate activity, since part of the motive for the activity is to gain the physical necessities and (for some people) social status and the luxuries that advertising makes them want. But many people put into their work far more effort than is necessary to earn whatever money and status they require, and this extra effort constitutes a surrogate activity."

"The system is currently engaged in a desperate struggle to overcome certain problems that threaten its survival, among which the problems of human behavior are the most important. If the system succeeds in acquiring sufficient control over human behavior quickly enough, it will probably survive. Otherwise it will break down. We think the issue will most likely be resolved within the next several decades, say 40 to 100 years."

"The economic and technological structure of a society are far more important than its political structure in determining the way the average man lives"

"The conservatives are fools: They whine about the decay of traditional values, yet they enthusiastically support technological progress and economic growth. Apparently it never occurs to them that you can’t make rapid, drastic changes in the technology and the economy of a society without causing rapid changes in all other aspects of the society as well, and that such rapid changes inevitably break down traditional values."

"It is well known that crowding increases stress and aggression. The degree of crowding that exists today and the isolation of man from nature are consequences of technological progress. All pre-industrial societies were predominantly rural. The Industrial Revolution vastly increased the size of cities and the proportion of the population that lives in them, and modern agricultural technology has made it possible for the Earth to support a far denser population than it ever did before"

"On the growing edge of the American frontier during the 19th century, the mobility of the population probably broke down extended families and small-scale social groups to at least the same extent as these are broken down today. In fact, many nuclear families lived by choice in such isolation, having no neighbors within several miles, that they belonged to no community at all, yet they do not seem to have developed problems as a result."

"We attribute the social and psychological problems of modern society to the fact that that society requires people to live under conditions radically different from those under which the human race evolved and to behave in ways that conflict with the patterns of behavior that the human race developed while living under the earlier conditions. It is clear from what we have already written that we consider lack of opportunity to properly experience the power process as the most important of the abnormal conditions to which modern society subjects people. .. Among the abnormal conditions present in modern industrial society are excessive density of population, isolation of man from nature, excessive rapidity of social change and the breakdown of natural small-scale communities such as the extended family, the village or the tribe."

"But for most people it is through the power process having a goal, making an AUTONOMOUS effort and attaining the goal — that self-esteem, self-confidence and a sense of power are acquired. When one does not have adequate opportunity to go through the power process the consequences are (depending on the individual and on the way the power process is disrupted) boredom, demoralization, low self-esteem, inferiority feelings, defeatism, depression, anxiety, guilt, frustration, hostility, spouse or child abuse, insatiable hedonism, abnormal sexual behavior, sleep disorders, eating disorders. etc. Any of the foregoing symptoms can occur in any society, but in modern industrial society they are present on a massive scale. We aren’t the first to mention that the world today seems to be going crazy. This sort of thing is not normal for human societies. There is good reason to believe that primitive man suffered from less stress and frustration and was better satisfied with his way of life than modern man is. It is true that not all was sweetness and light in primitive societies. Abuse of women was common among the Australian aborigines, transexuality was fairly common among some of the American Indian tribes. But it does appear that GENERALLY SPEAKING the kinds of problems that we have listed in the preceding paragraph were far less common among primitive peoples than they are in modern society."

"Modern man has the sense (largely justified) that change is IMPOSED on him, whereas the 19th century frontiersman had the sense (also largely justified) that he created change himself, by his own choice. Thus a pioneer settled on a piece of land of his own choosing and made it into a farm through his own effort. In those days an entire county might have only a couple of hundred inhabitants and was a far more isolated and autonomous entity than a modern county is."

"Some people may have some exceptional drive, in pursuing which they satisfy their need for the power process. For example, those who have an unusually strong drive for social status may spend their whole lives climbing the status ladder without ever getting bored with that game."

"People vary in their susceptibility to advertising and marketing techniques. Some are so susceptible that, even if they make a great deal of money, they cannot satisfy their constant craving for the the shiny new toys that the marketing industry dangles before their eyes. So they always feel hard-pressed financially even if their income is large, and their cravings are frustrated. Some people have low susceptibility to advertising and marketing techniques. These are the people who aren’t interested in money. Material acquisition does not serve their need for the power process. People who have medium susceptibility to advertising and marketing techniques are able to earn enough money to satisfy their craving for goods and services, but only at the cost of serious effort (putting in overtime, taking a second job, earning promotions, etc.). Thus material acquisition serves their need for the power process. But it does not necessarily follow that their need is fully satisfied."

"It is true that some individuals seem to have little need for autonomy. Either their drive for power is weak or they satisfy it by identifying themselves with some powerful organization to which they belong. And then there are unthinking, animal types who seem to be satisfied with a purely physical sense of power (the good combat soldier, who gets his sense of power by developing fighting skills that he is quite content to use in blind obedience to his superiors)."

"In primitive societies, physical necessities generally fall into group 2: They can be obtained, but only at the cost of serious effort. ... Advertising and marketing techniques have been developed that make many people feel they need things that their grandparents never desired or even dreamed of. It requires serious effort to earn enough money to satisfy these artificial needs, hence they fall into group 2. … Modern man must satisfy his need for the power process largely through pursuit of the artificial needs created by the advertising and marketing industry, and through surrogate activities."

"A young man goes through the power process by becoming a hunter, hunting not for sport or for fulfillment but to get meat that is necessary for food. (In young women the process is more complex, with greater emphasis on social power; we won’t discuss that here.) This phase having been successfully passed through, the young man has no reluctance about settling down to the responsibilities of raising a family. (In contrast, some modern people indefinitely postpone having children because they are too busy seeking some kind of “fulfillment.” We suggest that the fulfillment they need is adequate experience of the power process — with real goals instead of the artificial goals of surrogate activities.) Again, having successfully raised his children, going through the power process by providing them with the physical necessities, the primitive man feels that his work is done and he is prepared to accept old age (if he survives that long) and death. any modern people, on the other hand, are disturbed by the prospect of physical deterioration and death, as is shown by the amount of effort they expend trying to maintain their physical condition, appearance and health. We argue that this is due to unfulfillment resulting from the fact that they have never put their physical powers to any practical use, have never gone through the power process using their bodies in a serious way."

"Some people partly satisfy their need for power by identifying themselves with a powerful organization or mass movement. An individual lacking goals or power joins a movement or an organization, adopts its goals as his own, then works toward those goals. When some of the goals are attained, the individual, even though his personal efforts have played only an insignificant part in the attainment of the goals, feels (through his identification with the movement or organization) as if he had gone through the power process."

"If you think that more effective law enforcement is unequivocally good because it suppresses crime, then remember that crime as defined by the system is not necessarily what YOU would call crime. Today, smoking marijuana is a “crime,” and, in some places in the U.S., so is possession of an unregistered handgun. Tomorrow, possession of ANY firearm, registered or not, may be made a crime, and the same thing may happen with disapproved methods of child-rearing, such as spanking. In some countries, expression of dissident political opinions is a crime, and there is no certainty that this will never happen in the U.S., since no constitution or political system lasts forever. If a society needs a large, powerful law enforcement establishment, then there is something gravely wrong with that society; it must be subjecting people to severe pressures if so many refuse to follow the rules, or follow them only because forced. Many societies in the past have gotten by with little or no formal law-enforcement."