{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "87e17b68-7539-4539-aeab-9df6fc819f22",
   "metadata": {},
   "source": [
    "# Entropy of written English text\n",
    "\n",
    "Author: J. Lizier, Isabelle De Backer, 2022-; based on the original Matlab tutorials.\n",
    "\n",
    "The following block aims to import all the relevant libraries to analyse data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "55ca6967-2ca7-45e8-9858-6bb0356c2bee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "# Specifics required for the text processing here:\n",
    "import string\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc5119d3-c9d3-4139-ad43-3346098ea85d",
   "metadata": {},
   "source": [
    "# Preparing your environment\n",
    "\n",
    "As per `Module_2_notebook.ipynb` we need to use the functions we have defined in our previous work in other notebooks. So gather the new functions you wrote in this module into your `simpleinfotheory.py` script, and make sure it is referencable from here (you may need to change the folder referenced below) before you run the import line in the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8df5b524-117e-4a02-989e-9647b336dcc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 3: edit simpleinfotheory.py and past your functions into that as you write them\n",
    "import sys\n",
    "sys.path.append('../../Module1-IntroToInfoTheory/PythonCode/completed/')\n",
    "import simpleinfotheory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ecca927-eb6b-4840-8ea5-d4ea2bf79869",
   "metadata": {},
   "source": [
    "# 12. (Optional extension) Entropy of written English text\n",
    "\n",
    "Let's compute the Shannon information contents of letters in English language text ourselves, using the collected scripts from the 1990s comedy [Seinfeld](https://en.wikipedia.org/wiki/Seinfeld).\n",
    "\n",
    "1. Download the collection of text extracted from Seinfeld scripts following the links on Module 2 on Canvas.\n",
    "1. Open the data in a text file to inspect it (you should always do this!). We have each character's line on a different line of text. There is much punctuation in here as well.\n",
    "1. Load the data into Python: (_note_ you may need to alter the filename/path to match your own)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0cebd6af-93fe-4377-9bdf-adfc995bcd10",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = './Seinfeld-scripts-textOnly.txt'\n",
    "with open(filename, 'rt') as f:\n",
    "    str = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc2ec16c-3f84-49a0-ba1a-99c0b07f53d2",
   "metadata": {},
   "source": [
    "4. Now we need to pre-process it to remove punctuation characters, digits, and newlines (which we'll turn into spaces), and convert all upper case characters into lower case. We'll also convert it to a numpy array. Afterwards, let's check that we're only left with characters and spaces by examining the set of unique symbols in `processedStr` (leave the \";\" off so we see the output!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "75054903-0006-40fc-8b52-f5ec78c46e3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l',\n",
       "       'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y',\n",
       "       'z'], dtype='<U1')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile('[!\"#\\$%&\\'\\(\\)\\*\\+\\,-\\.\\/:;<=>\\?@\\[\\]\\\\\\^_`{\\|}~0-9]*');\n",
    "processedStr = p.sub('', str); # Remove punctuation characters and digits\n",
    "processedStr = ' '.join(processedStr.split('\\n')); # Replace newline characters with spaces\n",
    "processedStr = processedStr.lower(); # Convert all upper case into lower case\n",
    "processedStr = np.array(list(processedStr)); # Finally convert this into a numpy array so we can work with it\n",
    "np.unique(processedStr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e6b1195-3c3b-4570-ac49-742810f94025",
   "metadata": {},
   "source": [
    "5. Now compute the average entropy of these characters, as derived from their probabilities of occurrence in the Seinfeld script, using your `simpleinfotheory.entropyempirical()` function. Please note:\n",
    "    - You will need to have imported your `simpleinfotheory` scripts as above\n",
    "    - I would suggest that you edit your function `entropyempirical()` in `simpleinfotheory.py` to uncomment the line `[symbols, counts] = np.unique(xn, axis=0, return_counts=True)` instead of the subsequent for loop (which can be commented out), but still include the line `probabilities = counts / xnSamples`. (You can see how this is done in the `simpleinfotheory.py` solution code). This will run much faster. You will need to restart the kernel for this to take effect.\n",
    "\n",
    "    How does this compare to the stated value of the entropy of characters from [Mackay](http://www.inference.org.uk/itprnn/book.pdf) in Table 2.9 (sec 2.3; or see slide 26 of our lecture) as estimated from \"_The Frequently Asked Questions Manual for Linux_\"? Did you expect it to be the same, and why or why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "37767929-ed8a-4a8a-a8f9-1c8a2842b0cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entropy of single characters from Seinfeld scripts is 4.0846 bits\n"
     ]
    }
   ],
   "source": [
    "# Compute the entropy of the characters:\n",
    "(result, symbols, probabilities) = simpleinfotheory.entropyempirical(processedStr)\n",
    "print(\"Entropy of single characters from Seinfeld scripts is %.4f bits\" % result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ca71e8-9b68-40ed-8124-473aa4275567",
   "metadata": {},
   "source": [
    "6. Next, compute the Shannon information content of each character, and again compare these to those quoted by Mackay.<br/>\n",
    "You will have noticed that the `simpleinfotheory.entropyempirical()` function returns the probabilities of each symbol as well as the result in a tuple `(result, symbols, probabilities)` (see more details in its header). So, when you call the function, make sure that you have accepted all output variables as follows: `(result, symbols, probabilities) = simpleinfotheory.entropyempirical(processedStr)`. You can then send the probabilities as an argument to your `simpleinfotheory.infocontent()` code. On comparing to Mackay's results for each character, remember that your Shannon information contents are for the characters in a sorted order, but that order may be different to what the book displays -- yours will be displayed for each character in the order they appear in `symbols` (which is as returned by `np.unique(processedStr)` above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0c84c738-8497-481c-970f-f63ac0215b79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Info content of [' '] is 2.3116 bits\n",
      "Info content of ['a'] is 4.0322 bits\n",
      "Info content of ['b'] is 6.4422 bits\n",
      "Info content of ['c'] is 5.9460 bits\n",
      "Info content of ['d'] is 5.2915 bits\n",
      "Info content of ['e'] is 3.4355 bits\n",
      "Info content of ['f'] is 6.4115 bits\n",
      "Info content of ['g'] is 5.4955 bits\n",
      "Info content of ['h'] is 4.3057 bits\n",
      "Info content of ['i'] is 4.1372 bits\n",
      "Info content of ['j'] is 8.3484 bits\n",
      "Info content of ['k'] is 6.3208 bits\n",
      "Info content of ['l'] is 4.9140 bits\n",
      "Info content of ['m'] is 5.5827 bits\n",
      "Info content of ['n'] is 4.2814 bits\n",
      "Info content of ['o'] is 3.8170 bits\n",
      "Info content of ['p'] is 6.4067 bits\n",
      "Info content of ['q'] is 11.2288 bits\n",
      "Info content of ['r'] is 4.6424 bits\n",
      "Info content of ['s'] is 4.4777 bits\n",
      "Info content of ['t'] is 3.7499 bits\n",
      "Info content of ['u'] is 5.1079 bits\n",
      "Info content of ['v'] is 7.1339 bits\n",
      "Info content of ['w'] is 5.5464 bits\n",
      "Info content of ['x'] is 9.9738 bits\n",
      "Info content of ['y'] is 5.1689 bits\n",
      "Info content of ['z'] is 10.5129 bits\n"
     ]
    }
   ],
   "source": [
    "# Compute the Shannon information content of each character:\n",
    "characterInfoContents = simpleinfotheory.infocontent(probabilities)\n",
    "# To display more nicely:\n",
    "for ix in range(symbols.size):\n",
    "    print('Info content of %s is %.4f bits' % (symbols[ix], characterInfoContents[ix]));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fa01559-3c8e-4987-86eb-03b83a9032da",
   "metadata": {},
   "source": [
    "7. _Next level challenge_: can you move on to compute joint entropies for consecutive appearance of two characters, and then the conditional entropy of the second given the first.<br/>\n",
    "    _Hint_: to select all but the last item in a numpy array `x`, you can refer to `x[:-1]`, whilst to select all but the first item in an array `x`, you can refer to `x[1:]`<br/>\n",
    "    What does this tell us about how reading one character reduces our uncertainty about the next, and does this make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0b962f0e-208a-4017-9d2f-3bd26850bbc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entropy of characters pairs: 7.4496 bits\n",
      "Conditional entropy of character given previous: 3.3650 bits\n"
     ]
    }
   ],
   "source": [
    "# Compute the joint entropies for two characters:\n",
    "#  Need a matrix with first column being first character, and second column\n",
    "#  being the second\n",
    "characterPairSamples = np.column_stack( (processedStr[:-1],processedStr[1:]) );\n",
    "pairEntropy,__,__ = simpleinfotheory.jointentropyempirical(characterPairSamples);\n",
    "print('Entropy of characters pairs: %.4f bits' % pairEntropy);\n",
    "# Compute the conditional entropy of the second character given the first:\n",
    "conditionalEntropy = simpleinfotheory.conditionalentropyempirical(processedStr[1:], processedStr[:-1]);\n",
    "print('Conditional entropy of character given previous: %.4f bits' % conditionalEntropy);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "231703d1-a22d-4a19-a22f-fea28c7a2558",
   "metadata": {},
   "source": [
    "A more serious challenge would be to display the joint Shannon information contents, and the conditional Shannon information contents, as per Figures 2.2 and 2.3 of Mackay. This cannot be done with a simple modification to our simple Matlab scripts, as they were not set up to return the probabilities in a nicely ordered way for all possible combinations. (That was sacrificed to make your other tasks easier!). But you could attempt to pull out a list of all observed joint symbols and their probabilities, and sort them nicely yourself ready for display in such a figure. We will work further on this in the next module (and solutions are deferred to that module)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "170d20cd-1072-4a1a-b8d4-c6adab0a2d4f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
