Entropy, Coding and Data Compression

S Natarajan is a Professor at the Bangalore Centre of the Indian Statistical Institute.

Rapid progress in all fields of scientific and technological activity has been the hallmark of the twentieth century. Though much of these was the result of the combined efforts of several scientists, some of the path-breaking developments were due to the original work of a few gifted ones. Claude Elwood Shannon (1916-2001) was one such in the field of communication science and technology. During the nineteen forties, he was working as a communications engineer in Bell Telephone Laboratories. The major problem at that time was reliable communication over radio, telephones, telegraphy, etc. The approach to these problems was that of A N Kolmogorov and N Wiener who advocated a model of a stochastic process of signals corrupted by disturbance, called noise, due to the medium used for transmission. The problem then is to recover the original signal from the corrupted one. Shannon made a radical departure from their approach by introducing the idea of encoding the signal before passing through the medium. In a single (two part) paper [1] entitled `A Mathematical Theory of Communication' he not only provided a new set-up for the problem, but also proved a number of basic theorems and gave a few practical methods to achieve what the theorems promised. He introduced the concept of entropy of a probability distribution (which already was in use in thermodynamics as a measure of disorder) and showed how this determines the minimum possible rate of reliable communication. He also named the new subject `information theory' since entropy is a measure of the information content of a probability distribution. The entropy idea was later taken to ergodic theory in 1958 by A N Kolmogorov to solve the outstanding problem of isomorphism of Bernoulli shifts, thus giving birth to a very rich area of research known as entropy theory of measure preserving transformations.

Read full article (156  Kb)

Address for Correspondence
S Natarajan 
Indian Statistical Institute 
Bangalore 560059, India. 
E-mail: sn@isibang.ac.in