This is an introductory course on information theory.
I will be covering most of chapters 2-5, 9-10
and a few additional topics from chapters 10,15
of the book Elements of Information Theory (2nd ed.) by Cover and Thomas.
Prerequisites: EE3110 or an equivalent course on probability.
See here for a previous offering of this course with a different course number.
Textbook: Elements of Information Theory (2nd ed.) T. M. Cover and J. A. Thomas.
Course topics.
- Introduction to information theory
- Entropy, relative entropy and mutual information
- Asymptotic equipartition property
- Entropy rate of a stochastic process
- Data compression
- Channel capacity
- Differential entropy
- Gaussian channel
- Additional topics (time permitting)
Grading policy (Tentative)
5% Homework+scribing, 5-10% Project, 10% Miniquizzes, 25-30% Midsem, 50% Endsem.
Midsem 01 Oct 2019
Endsem 16 Nov 2019 (as per Institute calender)
Lectures
29 Jul Lecture 01 Introduction
30 Jul Lecture 02 Entropy, joint and conditional entropies
05 Aug Lecture 03 Relative entropy, mutual information
06 Aug Lecture 04 Jensen's inequality and consequences
13 Aug Lecture 05 Log sum inequality and consequences
19 Aug Lecture 06 Data processing inequality, Fano's inequality
22 Aug Lecture 07 AEP, Miniquiz-1
27 Aug Lecture 08 AEP, Entropy rate
29 Aug Lecture 09 Entropy rates of stochastic processes
03 Sep Lecture 10 Data compression: Source coding, Kraft's inequality
05 Sep Lecture 11 Extended Kraft's inequality, bounds
10 Sep Lecture 12 Feb Uniquely decodable codes, Introduction to Huffman codes
19 Sep Lecture 13 Huffman codes, optimality of Huffman codes
23 Sep Lecture 14 Channel capacities
26 Sep Lecture 15 Joint AEP
07 Oct Noninstructional day
10 Oct Lecture 16 Channel coding theorem
17 Oct Lecture 17 Converse of channel coding theorem, source channel separation
21 Oct Lecture 18 Differential entropy
22 Oct Lecture 19 Differential entropy, Gaussian channel
28 Oct Lecture 20 Capacity of Gaussian channel
29 Oct Lecture 21 Parallel and bandlimited Gaussian channels
31 Oct Tutorial, Miniquiz-3