Information Theory and Coding (Elective) ...

Course description                   

This is intended to be a straightforward and accessible course on information theory. Information theory is the mathematical theory that deals with the fundamental aspects of communication systems. As such, its primary goal is not to deliver practical solutions to communications problems, but rather to answer the question whether encoding and decoding schemes exist or not for a given combination of a source model and a channel model. The two main outcomes of single-user information theory are that any source requires a minimum description rate to represent its output faithfully (source coding theorem) and that any channel is characterized by a maximum transmission rate above which the probability of error cannot be made arbitrarily small (channel coding theorem). The purpose of this course is to develop the fundamental ideas of information theory and to indicate where and how the theory can be applied.

Course learning outcomes

At the end of the course students should be able to

·         Calculate the information content of a random variable from its probability distribution;

·         Relate the joint, conditional, and marginal entropies of variables in terms of their coupled probabilities;

·         Define channel capacities and properties using Shannon’s Theorems;

·         Construct efficient codes for data on imperfect communication channels;

·         Generalize the discrete concepts to continuous signals on continuous channels;

·         Understand Fourier Transforms and the main ideas of efficient algorithms for them;

·         Describe the information resolution, compression, and efficient coding properties of wavelets.

Course content

·         Introduction and Preview

·         Entropy and Mutual Information

·         Asymptotic Equipartition Property

·         Entropy Rates of a Stochastic Process

·         Data Compression

·         Channel Capacity

·         Differential Entropy

·         The Gaussian Channel

 

Course reading Materials

·         Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, John Wiley & Sons, Inc., 1991, ISBN 0-471-06259-6.

·         Lin S, G Costillo D.J. 2004 Error Control Coding 2nd Ed., Prentice Hall, ISBN 978-0130426727.

·         Wells R.B., 1999. Applied Coding and Information Theory for Engineers, Prentice Hall, ISBN 0-13-961327-7.

·         Reed I S, Chen X 1999. Error-Control Coding for data Networks, Springer,ISBN: 978-0-7923-8528-8.