{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "87e17b68-7539-4539-aeab-9df6fc819f22",
   "metadata": {},
   "source": [
    "# Conditional mutual information between successive letters in written English\n",
    "\n",
    "Author: J. Lizier, Isabelle De Backer, 2022-; based on the original Matlab tutorials.\n",
    "\n",
    "The following block aims to import all the relevant libraries to analyse data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "55ca6967-2ca7-45e8-9858-6bb0356c2bee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "# Specifics required for the text processing here:\n",
    "import string\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc5119d3-c9d3-4139-ad43-3346098ea85d",
   "metadata": {},
   "source": [
    "# Preparing your environment\n",
    "\n",
    "As per `Module_2_notebook.ipynb` etc. we need to use the functions we have defined in our previous work in other notebooks. So gather the new functions you wrote in this module into your `simpleinfotheory.py` script, and make sure it is referencable from here (you may need to change the folder referenced below) before you run the import line in the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8df5b524-117e-4a02-989e-9647b336dcc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 3: edit simpleinfotheory.py and past your functions into that as you write them\n",
    "import sys\n",
    "sys.path.append('../../Module1-IntroToInfoTheory/PythonCode/completed/')\n",
    "import simpleinfotheory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ecca927-eb6b-4840-8ea5-d4ea2bf79869",
   "metadata": {},
   "source": [
    "# 7. (Optional Extension) Conditional mutual information between successive letters in written English\n",
    "\n",
    "In this extension activity, we will continue our analysis of written English extracted from the [Seinfeld](https://en.wikipedia.org/wiki/Seinfeld) scripts as begun in the previous modules.\n",
    "\n",
    "1. Download the scripts from the links on Module 2 on Canvas, load into Python and preprocess as per steps 1-4 of the activity from module 2, such that we have the characters stored in the numpy array `processedStr`:<br/>\n",
    "_Note:_ you may need to alter the filename/path to match your own --"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0cebd6af-93fe-4377-9bdf-adfc995bcd10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l',\n",
       "       'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y',\n",
       "       'z'], dtype='<U1')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filename = './Seinfeld-scripts-textOnly.txt'\n",
    "with open(filename, 'rt') as f:\n",
    "    str = f.read()\n",
    "p = re.compile('[!\"#\\$%&\\'\\(\\)\\*\\+\\,-\\.\\/:;<=>\\?@\\[\\]\\\\\\^_`{\\|}~0-9]*');\n",
    "processedStr = p.sub('', str); # Remove punctuation characters and digits\n",
    "processedStr = ' '.join(processedStr.split('\\n')); # Replace newline characters with spaces\n",
    "processedStr = processedStr.lower(); # Convert all upper case into lower case\n",
    "processedStr = np.array(list(processedStr)); # Finally convert this into a numpy array so we can work with it\n",
    "np.unique(processedStr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e6b1195-3c3b-4570-ac49-742810f94025",
   "metadata": {},
   "source": [
    "2. We previously computed the mutual information between characters over several lags between these characters. In particular, we examined the mutual information between characters separated by a lag of 2, and posed the question of whether the information carried by the earlier character about the later one is also included in the character in between them. Think about how you could investigate this question using conditional mutual information?\n",
    "3. Calculate the mutual information between characters separated by another character, conditioned on the character in the middle. Use our function `simpleinfotheory.conditionalmutualinformationempirical()`.<br/>\n",
    "    _Hint_: You should create vectors of samples for each of: the earlier character, the middle character, and the later character; and pass these through to the function. Recall that to select all but the last two items in a numpy array `x`, you can refer to `x[:-2]`, and similarly to select all but the first and last items in an array you can refer to `x[1:-1]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f1225213-86e2-49f3-9ff9-dfe0f6d34876",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.739076942492698"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the mutual information between successive characters conditioned on the character in between:\n",
    "simpleinfotheory.conditionalmutualinformationempirical(processedStr[:-2],processedStr[2:],processedStr[1:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab60c5ec-592a-4e15-97c7-280f7ce91253",
   "metadata": {},
   "source": [
    "4. Compare this conditional mutual information to the mutual information between the two characters separated by another character as computed in the previous module (see the result for a lag of 2 on the sample plot in that activity). Is the conditional mutual information here larger or smaller than that? What does this tell us about the structure of the relationships in sequences of characters in English text?\n",
    "5. _Challenge_: Can you compute such conditional mutual information over lags of up to 5 characters, conditioning on all intervening characters, and then plot these? Alternatively, you could compute the joint mutual information from sets of consecutive characters (up to 5 of them) to the next character. (How are these two quantities related?).<br/>\n",
    "    Note that the calculations here will take _significantly_ longer than previous ones since we are dealing with higher and higher order multivariate spaces (not so long for a lag of 2, but ~1 minute for lag 5). What is the size of the probability space we are estimating once we are looking at a lag of 5 (i.e. 4 characters in between the previous and next)? Do you think we can properly estimate the joint probabilities here from the amount of data that we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c9a980f0-ad8a-4aea-9fbe-5d0a78282db8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cond MI over lag 1 is 0.7196\n",
      "\n",
      "cond MI over lag 2 is 0.7391\n",
      "\n",
      "cond MI over lag 3 is 0.5251\n",
      "\n",
      "cond MI over lag 4 is 0.3197\n",
      "\n",
      "cond MI over lag 5 is 0.2474\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 29920 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compute the mutual information between successive characters conditioned on up to 5 characters in between:\n",
    "maxLag = 5;\n",
    "\n",
    "condMisVsLag = np.zeros(maxLag);\n",
    "conditionalCharacters = None;\n",
    "\n",
    "nextChar = processedStr[maxLag:];\n",
    "for lag in range(1,maxLag+1): # To go from 1 up to maxLag\n",
    "    sourceChar = processedStr[maxLag-lag:-lag];\n",
    "    if (lag == 1):\n",
    "        # Nothing to condition on, just compute MI\n",
    "        condMisVsLag[lag-1] = simpleinfotheory.mutualinformationempirical(sourceChar,nextChar)[0];\n",
    "        conditionalCharacters = sourceChar; # Initialise the characters to now be conditioned on\n",
    "    else:\n",
    "        condMisVsLag[lag-1] = simpleinfotheory.conditionalmutualinformationempirical(sourceChar,nextChar,conditionalCharacters);\n",
    "        conditionalCharacters = np.column_stack( (conditionalCharacters,sourceChar) ); # Add to the characters to be conditioned on\n",
    "    print('cond MI over lag %d is %.4f\\n' % (lag, condMisVsLag[lag-1]));\n",
    "\n",
    "plt.scatter(range(1,maxLag+1), condMisVsLag, c=\"red\", marker=\"x\");\n",
    "plt.xlabel('Lag')\n",
    "plt.ylabel('Cond MI (bits)');\n",
    "plt.title('Average MI between characters separated by the\\ngiven lag conditioned on intervening chars');\n",
    "plt.savefig('CondMIversusLag.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "498d4c88-23f4-4544-922d-e812255fd914",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
