Information theory

"Claude Shannon, the founder of information theory, invented a way to measure 'the amount of information' in a message without defining the word 'information' itself, nor even addressing the question of the meaning of the message."

"In fact, an information theory that leaves out the issue of noise turns out to have no content."

"If quantum communication and quantum computation are to flourish, a new information theory will have to be developed."

"The 19th and first half of the 20th century conceived of the world as chaos. Chaos was the oft-quoted blind play of atoms, which, in mechanistic and positivistic philosophy, appeared to represent ultimate reality, with life as an accidental product of physical processes, and mind as an epi-phenomenon. It was chaos when, in the current theory of evolution, the living world appeared as a product of chance, the outcome of random mutations and survival in the mill of natural selection. In the same sense, human personality, in the theories of behaviorism as well as of psychoanalysis, was considered a chance product of nature and nurture, of a mixture of genes and an accidental sequence of events from early childhood to maturity. Now we are looking for another basic outlook on the world -- the world as organization. Such a conception -- if it can be substantiated -- would indeed change the basic categories upon which scientific thought rests, and profoundly influence practical attitudes. This trend is marked by the emergence of a bundle of new disciplines such as cybernetics, information theory, general system theory, theories of games, of decisions, of queuing and others; in practical applications, systems analysis, systems engineering, operations research, etc. They are different in basic assumptions, mathematical techniques and aims, and they are often unsatisfactory and sometimes contradictory. They agree, however, in being concerned, in one way or another, with "systems," "wholes" or "organizations"; and in their totality, they herald a new approach."

"We completely ignore the human value of the information. A selection of 100 letters is given a certain information value, and we do not investigate whether it makes sense in English, and, if so, whether the meaning of the sentence is of any practical importance. According to our definition, a set of 100 letters selected at random (according to the rules of Table 1.1), a sentence of 100 letters from a newspaper, a piece of Shakespeare or a theorem of Einstein are given exactly the same informational value."

"In fact, the science of thermodynamics began with an analysis, by the great engineer Sadi Carnot, of the problem of how to build the best and most efficient engine, and this constitutes one of the few famous cases in which engineering has contributed to fundamental physical theory. Another example that comes to mind is the more recent analysis of information theory by Claude Shannon. These two analyses, incidentally, turn out to be closely related."

"Whether computers are used for engineering design, medical data processing, composing music, or other purposes, the structure of computing is much the same. We are extremely short of talented people in this field, and so we need departments, curricula, and research and degree programs in computer science... I think of the Computer Science Department as eventually including experts in Programming, Numerical Analysis, Automata Theory, Data Processing, Business Games, Adaptive Systems, Information Theory, Information Retrieval, Recursive Function Theory, Computer Linguistics, etc., as these fields emerge in structure... Universities must respond [to the computer revolution] with far reaching changes in the educational structure."

"Incomplete knowledge of the future, and also of the past of the transmitter from which the future might be constructed, is at the very basis of the concept of information. On the other hand, complete ignorance also precludes communication; a common language is required, that is to say an agreement between the transmitter and the receiver regarding the elements used in the communication process... [The information of a message can] be defined as the 'minimum number of binary decisions which enable the receiver to construct the message, on the basis of the data already available to him.' These data comprise both the convention regarding the symbols and the language used, and the knowledge available at the moment when the message started."

"I have tried to show that psychiatric research can be empirical and experimental, controlled, and operational and not dependent on inferences, analogies, or anecdotes. Hypotheses can be derived which are testable. Theory is a different matter. At the present we rely heavily on psychoanalytic theory or on still poorly formulated and defined general systems theory, information theory, or transactional theory. To explain the depth and variety of the interrelationship of somatopsychosocial facets of the totality of human behavior in process requires a unified theory of human behavior which we have not yet even approached. Integration or synthesis of biological, psychological, and social theory is not enough."

"Cybernetics is concerned primarily with the construction of theories and models in science, without making a hard and fast distinction between the physical and the biological sciences. The theories and models occur both in symbols and in hardware, and by 'hardware* we shall mean a machine or computer built in terms of physical or chemical, or indeed any handleable parts. Most usually we shall think of hardware as meaning electronic parts such as valves and relays. Cybernetics insists, also, on a further and rather special condition that distinguishes it from ordinary scientific theorizing: it demands a certain standard of effectiveness. In this respect it has acquired some of the same motive power that has driven research on modern logic, and this is especially true in the construction and application of artificial languages and the use of operational definitions. Always the search is for precision and effectiveness, and we must now discuss the question of effectiveness in some detail. It should be noted that when we talk in these terms we are giving pride of place to the theory of automata at the expense, at least to some extent, of feedback and information theory."

"Every time we fire a phonetician/linguist, the performance of our system goes up"

"Pure mathematics, being mere tautology, and pure physics, being mere fact, could not have engendered them; for creatures to live, must sense the useful and the good; and engines to run must have energy available as work : and both, to endure, must regulate themselves. So it is to Thermodynamics and to its brother Σp log p, called Information theory, that we look for the distinctions between work and energy and between signal and noise."

"The field of 'information theory' began by using the old hardware paradigm of transportation of data from point to point."

"Without an understanding of causality there can be no theory of communication. What passes as information theory today is not communication at all, but merely transportation."

"Some authors state that the last stage in this chain of measurements involves "consciousness," or the "intellectual inner life" of the observer, by virtue of the "principle of psycho-physical parallelism." Other authors introduce a wave function for the entire universe. In this book, I shall refrain from using concepts that I do not understand."

"My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'"

"A great deal of the thinking [in Organizational Development] has been influenced by cybernetics and information theory, though this has been used as much to extend the scope of as to improve the sophistication of formulations. It was von Bertalanffy (1950) who, in terms of the general transport equation which he introduced, first fully disclosed the importance of openness or closedness to the environment as a means of distinguishing living organisms from inanimate objects."

"The cybernetics phase of cognitive science produced an amazing array of concrete results, in addition to its long-term (often underground) influence:"