{"id":6460,"date":"2025-12-22T12:31:42","date_gmt":"2025-12-22T07:01:42","guid":{"rendered":"https:\/\/study.madeeasy.in\/?p=6460"},"modified":"2025-12-22T12:31:42","modified_gmt":"2025-12-22T07:01:42","slug":"tokens-patterns-and-lexemes","status":"publish","type":"post","link":"https:\/\/www.madeeasy.in\/study\/cs-it\/tokens-patterns-and-lexemes","title":{"rendered":"Tokens, Patterns and Lexemes"},"content":{"rendered":"<ul>\n<li style=\"text-align: justify;\">A token is a pair consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a king of lexical unit, e.g., a particular keyword or a sequence of input characters denoting an identifier.<\/li>\n<li style=\"text-align: justify;\">A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword.<\/li>\n<li style=\"text-align: justify;\">A lexeme\u00a0is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token.<\/li>\n<\/ul>\n<p><strong>Example:\u00a0<\/strong><\/p>\n<p style=\"text-align: center;\">Identify the lexeme and the respective token in the following C-statement.<br \/>\nprintf(\u201cTotal = %d\\n\u201d, score);<\/p>\n<p><strong>Solution:<\/strong><\/p>\n<p style=\"text-align: center;\">printf is lexeme corresponding to identifier score is lexeme corresponding to identifier<br \/>\n\u201cTotal = %d\\n\u201d is a lexeme corresponding to literal.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A token is a pair consisting of a token name and an optional attribute value. The token name is an<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1908,3],"tags":[1917,1915,1916],"class_list":["post-6460","post","type-post","status-publish","format-standard","hentry","category-compiler","category-cs-it","tag-lexemes","tag-patterns","tag-tokens"],"_links":{"self":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/posts\/6460","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/comments?post=6460"}],"version-history":[{"count":0,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/posts\/6460\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/media?parent=6460"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/categories?post=6460"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.madeeasy.in\/study\/wp-json\/wp\/v2\/tags?post=6460"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}