"The recent development of various methods of modulation such as reM
and PPM which exchange bandwidth for signal-to-noise ratio has intensified
the interest in a general theory of communication. A basis for
such a theory is contained in the important papers of Nyquist! and Hartley"
on this subject. In the present paper we will extend the theory to include a
number of new factors, in particular the effect of noise in the channel, and
the savings possible due to the sta tistiral structure of the original message
and due to the nature of the final destination of the information.
The fundamental problem of communication is that of reproducing at
one point either exactly or approximately a message selected at another
point. Frequently the messages have meaning; that is they refer to or are
correlated according to some system with certain physical or conceptual
entities. These semantic aspects of communication are irrelevant to the
engineering problem. The significant aspect is that the actual message is
one selected from a set of possible messages. The system must be designed
to operate for each possible selection, not just the one which will actually
be chosen since this is unknown at the time of design." p379
"<p>By a communication system we will mean a system of the type indicated
schematically in Fig. 1. It consists of essentially five parts:
1. An information source which produces a message or sequence of messages
to be communicated to the receiving terminal...
2. A transmitter which operates on the message in some way to produce a
signal suitable for transmission over the channel.
3. The channel is merely the medium used to transmit the signal from
transmitter to receiver.
4. The receiver ordinarily performs the inverse operation of that done by
the transmitter, reconstructing the message from the signal.
5. The destination is the person (or thing) for whom the message is intended.
"We can think of a discrete source as generating the message, symbol by symbol. It will choose successive symbols accoring to certain probabilities depending, in general, on preceding choices as well as the particular symbols in question." p385
"A more complicated structure is obtained if successive symbols are not chosen independently but their probabilities depend on preceding letters." p386
"<p>Second-Order Word Approximation. The word transition probabilities are correct but no further structure is included.
</p><p>THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH
</p><p>WRITER THAT THE CHARACTER OF THIS POINT IS
</p><p>THEREFORE ANOTHER METHOD FOR THE LETTERS
</p><p>THAT THE TIME OF WHO EVER TOLD THE PROBLEM
</p><p>FOR AN UNEXPECTED
</p><p>The resemblance to ordinary English text increases quite noticeably at each of the above steps. Note that these samples have reasonably good
structure out to about twice the range that is taken into account in their construction. Thus in (3) the statistical process insures reasonable text
for two-letter sequence, but four-letter sequences from the sample can usually be fitted into good sentences.
"<p>We have represented a discrete information source as a Markoff process.
Can we define a quantity which will measure, in some sense, how much information
is "produced" by such a process, or better, at what rate information
Suppose we have a set of possible events whose probabilities of occurrence
are PI , P2 , ... , P3 These probabilities are known but that is all we know
concerning which event will occur. Can we find a measure of how much
"choice" is involved in the selection of the event or of how uncertain we are
of the outcome?
"Quantities of the form H = - Σp<sub>i</sub> log p<sub>i</sub> (the constant K merely amounts
to a choice of a unit of measure) play a central role in information theory as
measures of information, choice and uncertainty. The form of H will be
recognized as that of entropy as defined in certain formulations of statistical
mechanics where p<sub>i</sub> is the probability of a system being in cell <i>i</i> of its phase
space. <i>H</i> is then, for example, the H in Boltzmann's famous H theorem.
We shall call H = - ΣP<sub>i</sub> log P<sub>i</sub> the entropy of the set of probabilities" p393
"The ratio of the entropy of a source to the maximum value it could have
while still restricted to the same symbols will be called its relative ell/ropy.
This is the maximum compression possible when we encode into the same
alphabet. One minus the relative entropy is the redundancy. The redundancy of ordinary English, not considering statistical structure over greater
distances than about eight letters is roughly 50%. This means that when
we write English half of what we write is determined by the structure of the
language and half is chosen freely. The figure 50% was found by several
independent methods which all gave results in this neighborhood. One is
by calculation of the entropy of the approximations to English. A second
method is to delete a certain fraction of the letters from a sample of English
text and then let someone attempt to restore them. If they can be restored
when 50% are deleted the redundancy must be greater than 50%.
A third method depends on certain known results in cryptography.
Two extremes of redundancy in English prose are represented by Basic
English and by James Joyces' book "Finigans Wake." The Basic English
vocabulary is limited to 850 words and the redundancy is very high. This
is reflected in the expansion that occurs when a passage is translated into
Basic English. Joyce on the' other hand enlarges the vocabulary and is
alleged to achieve a compression of semantic content.
The redundancy of a language is related to the existence of crossword
puzzles. If the redundancy is zero any sequence of letters is a reasonable
text in the language and any two dimensional array of letters forms a crossword
puzzle, If the redundancy is too high the language imposes too
many constraints for large crossword puzzles to be possible. A more detailed
analysis shows that if we assume the constraints imposed by the
language are of a rather chaotic and random nature, large crossword puzzles
are just possible when the redundancy is 50%. If the redundancy is 33%,
three dimensional crossword puzzles should be possible, etc." p398-399
"The input to the
transducer is a sequence of input symbols and its output a sequence of output
symbols. The transducer may have an internal memory so that its
output depends not only on the present input symbol but also on the past
history. We assume that the internal memory is finite, i.e. there exists
a finite number 111 of possible states of the transducer and that its output is
a function of the present state and the present input symbol. The next
state will be a second function of these two quantities." p399
"If the channel is noisy it is not in general possible to reconstruct the original
message or the transmitted signal with certainty by any operation on the
received signal E. There are, however, ways of transmitting the information
which are optimal in combating noise. This is the problem which we now
"An approximation to the ideal would have the property that if the signal
is altered in a reasonable way by the noise, the original can still be recovered.
Tn other words the alteration will not in general bring it closer to another
reasonable signal than the original. This is accomplished at the cost of a
certain amount of redundancy in the coding. The redundancy must be
introduced in the proper way to combat the particular noise structure
involved. However, any redundancy in the source will usually help if it is
utilized at the receiving point. In particular, if the source already has a
certain redundancy and no attempt is made to eliminate it in matching to the
channel, this redundancy will help combat noise. For example, in a noiseless
telegraph channel one could save about 50% in time by proper encoding of
the messages. This is not done and most of the redundancy of English
remains in the channel symbols. This has the advantage, however, of
allowing considerable noise in the channel. A sizable fraction of the letters
can be received incorrectly and still reconstructed by the context." p414