{"id":2903,"date":"2024-08-16T17:31:27","date_gmt":"2024-08-16T12:01:27","guid":{"rendered":"https:\/\/study.madeeasy.in\/?p=2903"},"modified":"2025-07-16T15:05:33","modified_gmt":"2025-07-16T09:35:33","slug":"information-theory-coding","status":"publish","type":"post","link":"https:\/\/www.madeeasy.in\/study\/ec\/communication-systems\/information-theory-coding","title":{"rendered":"Information Theory and Coding"},"content":{"rendered":"<p style=\"text-align: justify;\">The basic goal of communication system is to transmit information from one point to another.<\/p>\n<p style=\"text-align: justify;\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2904 size-full\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/channel.jpg\" alt=\"Channel\" width=\"269\" height=\"32\" \/><\/p>\n<p style=\"text-align: justify;\">Information theory deal with efficient transmission of signals. For this some processing of the signals is to be done. Processing block is divided into two parts:<\/p>\n<p style=\"text-align: justify;\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2905 size-full\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/processing-block.jpg\" alt=\"Processing Block\" width=\"299\" height=\"47\" \/><\/p>\n<ul style=\"text-align: justify;\">\n<li>Source coding is done in order to achieve the efficient representation of data generated by discrete source.<\/li>\n<li>Channel coding is used to introduce controlled redundancy to improve reliability.<\/li>\n<li>Source coding provides the mathematical tool for assessing data compaction that is lossless compression of data generated by discrete memoryless source. In the context of communication, information theory deals with mathematical modelling and analysis of a communication system rather than with physical sources and physical channel.<\/li>\n<\/ul>\n<h2 style=\"text-align: justify;\">Need of Information Theory<\/h2>\n<p style=\"text-align: justify;\">Need of information theory can be explained with the help of following example: Ex. Horse Race.<br \/>\nSuppose there are 8 horses running in a race and probability of winning of horse is as specified below:<\/p>\n<p style=\"text-align: justify;\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2906 size-full\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/information-theory.jpg\" alt=\" Information Theory\" width=\"539\" height=\"102\" srcset=\"https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/information-theory.jpg 539w, https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/information-theory-300x57.jpg 300w\" sizes=\"auto, (max-width: 539px) 100vw, 539px\" \/><\/p>\n<p style=\"text-align: justify;\">Using fixed length code, 3-bits (8 symbols) will be needed for transmission of information of winning of Horses. Now, the question arises, that if less than 3-bits can be used to transmit same information without any loss. The answer to this question is yes, by using source coding theorem.<\/p>\n<p style=\"text-align: justify;\">According to source coding theorem, the number of bits transmitted for each Horse\u2019s winning information is given by log<sub>b<\/sub> 1\/P(x<sub>i<\/sub>), where P(x<sub>i<\/sub>) probability of winning of i<sup>th<\/sup> horse.<\/p>\n<p style=\"text-align: justify;\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2907 size-full\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/coding-theorem.jpg\" alt=\"Coding Theorem\" width=\"685\" height=\"325\" srcset=\"https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/coding-theorem.jpg 685w, https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/coding-theorem-300x142.jpg 300w\" sizes=\"auto, (max-width: 685px) 100vw, 685px\" \/><\/p>\n<p style=\"text-align: justify;\"><strong>Note:<\/strong><\/p>\n<ul style=\"text-align: justify;\">\n<li>Here b is unit, in which we measure information<br \/>\nb = 10 \u21d2 Information is measured in \u2018Decit\u2019 or \u2018Hartley\u2019<br \/>\nb = e \u21d2 Information is measured in \u2018nat\u2019<br \/>\nb = 2 \u21d2 Information is measured in \u2018bit\u2019<\/li>\n<li>Frequent is the event, fewer will be the bits need to transmit it, because this helps in reducing the average bit length.<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">According to the theorem, the number of bits per source symbol can be made as small as, but not smaller than the entropy of the source measured in bits.<br \/>\nInformation theory provides a quantitative measure of the information contained in message signals and allows us to determine the capacity of a communication system to transfer this information from source to destination.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79_1 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-light-blue ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.madeeasy.in\/study\/ec\/communication-systems\/information-theory-coding\/#Information\" >Information<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.madeeasy.in\/study\/ec\/communication-systems\/information-theory-coding\/#Entropy\" >Entropy<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.madeeasy.in\/study\/ec\/communication-systems\/information-theory-coding\/#Channel-Capacity\" >Channel Capacity<\/a><\/li><\/ul><\/nav><\/div>\n<h3 style=\"text-align: justify;\"><span class=\"ez-toc-section\" id=\"Information\"><\/span>Information<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p style=\"text-align: justify;\">The amount of information contained in an event is closely related to its uncertainty. Thus a mathematical measure of information should be a function of probability and should satisfy following:<\/p>\n<p>(a) Information should be proportional to uncertainty of outcome.<br \/>\n(b) Information contained in independent outcomes should add.<\/p>\n<p>Suppose that a probabilistic experiment involves observation of the output emitted by a discrete source during every signalling interval.<\/p>\n<h3 style=\"text-align: justify;\"><span class=\"ez-toc-section\" id=\"Entropy\"><\/span>Entropy<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p style=\"text-align: justify;\">In practical communication system, we usually transmit long sequences of symbols from an information source. Thus, we are more interested in the average information that the source produces rather than the information<br \/>\ncontent of a single symbol. For quantitative representation of average information, following assumptions are made:<\/p>\n<p style=\"text-align: justify;\">(a) Source is stationary so that probabilities may remain constant with time.<br \/>\n(b) Successive symbols are statistically independent and come at average rate of r symbols\/second.<\/p>\n<p>The entropy of a discrete random variable, representing the output of a source of information, is a measure of the average information content per source symbol.<img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2908 size-full\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/entropy.jpg\" alt=\"Entropy\" width=\"620\" height=\"129\" srcset=\"https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/entropy.jpg 620w, https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/entropy-300x62.jpg 300w\" sizes=\"auto, (max-width: 620px) 100vw, 620px\" \/><\/p>\n<p style=\"text-align: justify;\">H(x) is also considered as the average amount of uncertainty within source X that is resolved by the use of alphabet.<\/p>\n<h4 style=\"text-align: justify;\">Maximum Value of Entropy<\/h4>\n<p style=\"text-align: justify;\">Maximum entropy will occur when all message signals have equal probability<br \/>\nLet, total message signal = M<br \/>\nLet, each have probability = 1\/M<\/p>\n<p style=\"text-align: center;\">H(x)<sub>max<\/sub> = log<sub>2<\/sub> M<\/p>\n<p style=\"text-align: justify;\">Entropy satisfies following relation<\/p>\n<p style=\"text-align: center;\">0 \u2264 H(x) \u2264 log<sub>2<\/sub>M<\/p>\n<p style=\"text-align: justify;\">The lower bounds corresponds to no uncertainty, which occurs when one symbol has probability P(x<sub>i<\/sub>)=1 while P(x<sub>j<\/sub>) = 0 for all j \u2260 i, so emits the same symbol xi all the time. The upper bound corresponds to the maximum uncertainty which occurs when P(x<sub>i<\/sub>) = 1\/M for all i, that is, when all symbols are equally likely to be emitted by X.<\/p>\n<h4 style=\"text-align: justify;\">Information Rate<\/h4>\n<p style=\"text-align: justify;\">If the rate at which source X emits symbols is r(symbols\/s), the information rate R of the source is given by<img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2913 size-full\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/information-rate.jpg\" alt=\"Information Rate\" width=\"393\" height=\"66\" srcset=\"https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/information-rate.jpg 393w, https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/information-rate-300x50.jpg 300w, https:\/\/www.madeeasy.in\/study\/wp-content\/uploads\/2024\/08\/information-rate-390x66.jpg 390w\" sizes=\"auto, (max-width: 393px) 100vw, 393px\" \/><\/p>\n<h3 style=\"text-align: justify;\"><span class=\"ez-toc-section\" id=\"Channel-Capacity\"><\/span>Channel Capacity<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p style=\"text-align: justify;\">Channel capacity defines the intrinsic ability of a communication channel to convey information.<br \/>\nChannel capacity per symbol of a DMC is defined as<img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-2914 aligncenter\" src=\"https:\/\/study.madeeasy.in\/wp-content\/uploads\/2024\/08\/channel-capacity.jpg\" alt=\" Channel Capacity\" width=\"184\" height=\"32\" \/><\/p>\n<p style=\"text-align: justify;\">where the maximization is over all possible input probability distributions {P(x<sub>i<\/sub>)} on X. Note that channel capacity C<sub>s<\/sub> is a function of only the channel transition probabilities that define the channel. It specify maximum number of bits allowed through channel per symbol.<\/p>\n<h4 style=\"text-align: justify;\">Channel Capacity Per Second (C)<\/h4>\n<p style=\"text-align: justify;\">If r symbols are being transmitted per second, then the maximum rate of transmission of information per second is rC<sub>s<\/sub>. This is the channel capacity per second and it is denoted by C(b\/s):<\/p>\n<p style=\"text-align: center;\">C = rC<sub>s<\/sub> | bit\/sec<\/p>\n<p style=\"text-align: center;\"><a class=\"btn btn-danger\" role=\"button\" href=\"https:\/\/study.madeeasy.in\/ec\/communication-systems\/awgn-channels\/\" target=\"_blank\" rel=\"noopener\">&lt;&lt; Previous<\/a> | <a class=\"btn btn-success\" role=\"button\" href=\"https:\/\/study.madeeasy.in\/ec\/communication-systems\/additive-white-gaussian-noise-channel\/\" target=\"_blank\" rel=\"noopener\"> Next &gt;&gt;<\/a><br \/>\n<strong> Must Read: <\/strong> <a href=\"https:\/\/study.madeeasy.in\/subjects\/what-is-communication\/\" target=\"_blank\" rel=\"noopener\"><strong>What is Communication?<\/strong><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The basic goal of communication system is to transmit information from one point to another. Information theory deal with efficient<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[758,6],"tags":[804,806,807,805],"class_list":["post-2903","post","type-post","status-publish","format-standard","hentry","category-communication-systems","category-ec","tag-coding-theorem","tag-entropy","tag-information-rate","tag-source-coding"],"_links":{"self":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/posts\/2903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/comments?post=2903"}],"version-history":[{"count":0,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/posts\/2903\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/media?parent=2903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/categories?post=2903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/tags?post=2903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}