mirror of https://github.com/jlizier/jidt
262 lines
12 KiB
Plaintext
262 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "87e17b68-7539-4539-aeab-9df6fc819f22",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Mutual information of written English text\n",
|
|
"\n",
|
|
"Author: J. Lizier, Isabelle De Backer, 2022-; based on the original Matlab tutorials.\n",
|
|
"\n",
|
|
"The following block aims to import all the relevant libraries to analyse data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "55ca6967-2ca7-45e8-9858-6bb0356c2bee",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import math\n",
|
|
"\n",
|
|
"# Specifics required for the text processing here:\n",
|
|
"import string\n",
|
|
"import re"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc5119d3-c9d3-4139-ad43-3346098ea85d",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Preparing your environment\n",
|
|
"\n",
|
|
"As per `Module_2_notebook.ipynb` etc. we need to use the functions we have defined in our previous work in other notebooks. So gather the new functions you wrote in this module into your `simpleinfotheory.py` script, and make sure it is referencable from here (you may need to change the folder referenced below) before you run the import line in the next cell:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8df5b524-117e-4a02-989e-9647b336dcc5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Option 3: edit simpleinfotheory.py and past your functions into that as you write them\n",
|
|
"import sys\n",
|
|
"sys.path.append('../../Module1-IntroToInfoTheory/PythonCode/completed/')\n",
|
|
"import simpleinfotheory"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4ecca927-eb6b-4840-8ea5-d4ea2bf79869",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 6. (Optional extension) Mutual information between successive letters in written English\n",
|
|
"\n",
|
|
"In this extension activity, we will continue our analysis of written English extracted from the [Seinfeld](https://en.wikipedia.org/wiki/Seinfeld) scripts as begun in the previous module.\n",
|
|
"\n",
|
|
"1. Download the scripts from the links on Module 2 on Canvas, load into Python and preprocess as per steps 1-4 of the activity from the previous module, such that we have the characters stored in the numpy array `processedStr`:<br/>\n",
|
|
"_Note:_ you may need to alter the filename/path to match your own --"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0cebd6af-93fe-4377-9bdf-adfc995bcd10",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"filename = './Seinfeld-scripts-textOnly.txt'\n",
|
|
"with open(filename, 'rt') as f:\n",
|
|
" str = f.read()\n",
|
|
"p = re.compile('[!\"#\\$%&\\'\\(\\)\\*\\+\\,-\\.\\/:;<=>\\?@\\[\\]\\\\\\^_`{\\|}~0-9]*');\n",
|
|
"processedStr = p.sub('', str); # Remove punctuation characters and digits\n",
|
|
"processedStr = ' '.join(processedStr.split('\\n')); # Replace newline characters with spaces\n",
|
|
"processedStr = processedStr.lower(); # Convert all upper case into lower case\n",
|
|
"processedStr = np.array(list(processedStr)); # Finally convert this into a numpy array so we can work with it\n",
|
|
"np.unique(processedStr)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4e6b1195-3c3b-4570-ac49-742810f94025",
|
|
"metadata": {},
|
|
"source": [
|
|
"2. How can we now compute the mutual information between one character and the character that comes next in the text? We will need to provide samples of a previous character and the next character to our `simpleinfotheory.mutualinformationempirical()` function.<br/>\n",
|
|
" _Hint_: to select all but the last item in a numpy array `x`, you can refer to `x[:-1]`, whilst to select all but the first item in an array `x`, you can refer to `x[1:]`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f1225213-86e2-49f3-9ff9-dfe0f6d34876",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Compute the mutual information between successive characters:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ab60c5ec-592a-4e15-97c7-280f7ce91253",
|
|
"metadata": {},
|
|
"source": [
|
|
"3. Compare the mutual information that you computed above to the average entropy of each character computed as per step 5 of the activity in the previous module. Consider the following:\n",
|
|
" 1. What proportion of our uncertainty about the next character in the written text is reduced by observing the previous character?\n",
|
|
" 1. How much code could we save in communicating a character if our coding scheme took the previous character into account?\n",
|
|
" 1. The mutual information computes a measure of the relationship between the consecutive characters here. You're probably familiar with using correlation to measure a relationship between variables -- could correlation be used here? We will see more about how MI and correlation are related in the coming weeks.\n",
|
|
"4. Are there relationships between previous characters and later characters beyond those which are consecutive?<br/>\n",
|
|
" Can you modify your call to `simpleinfotheory.mutualinformationempirical()` above to compute the mutual information between characters that are not consecutive but separated by a lag of 2 (i.e. with one character in between them)? Is there still a substantial relationship? Is this information solely contained in the earlier character or is it perhaps also included in the immediately previous character?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7dc60b50-fc49-40a8-9d9c-9afc67e081fc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Compute the mutual information between characters separated by a lag of two:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "75e18172-0164-46e0-9be3-18970c294c30",
|
|
"metadata": {},
|
|
"source": [
|
|
"5. Can you see how this relationship changes over longer lags still? Plot the mutual information as a function of lag (up to say 10). At what point would you say there is no longer a relationship? We will discuss statistical approaches to answering that in the coming weeks."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aaf73d8c-a21a-4503-9281-b3bd8e7f670f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Compute and plot the MI as a function of lag:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7445b88c-d912-4976-9368-11478dbbcf74",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 7. (Optional extension) Pointwise mutual information between successive letters in written English\n",
|
|
"\n",
|
|
"_Further challenge_ -- It would be interesting to inspect the **local or pointwise mutual information** between each possible pair of consecutive letters. (See Part 3 of the lecture)\n",
|
|
"\n",
|
|
"1. To do this, first note how `simpleinfotheory.jointentropyempirical()` returns the set of symbols and their probabilities, as well as the joint entropy value.\n",
|
|
"2. Now we will alter `simpleinfotheory.mutualinformationempirical()` to similarly retrieve and return all of the relevant probabilities for each consecutive character pair:\n",
|
|
" 1. Notice how the calls for the joint entropy, $Y$ entropy and $X$ entropy already retrieve these for us.\n",
|
|
" 2. Then alter the return statement so that all of these relevant values are returned: `return result, xySymbols, xyProbs, xSymbols, xProbs, ySymbols, yProbs` (this is already done in the solution code for `simpleinfotheory.py`).\n",
|
|
" 3. You will need to restart the kernel to reload the library. You'll also need to update the function calls above, since they're now returning a list. If you append `[0]` to them, such as `simpleinfotheory.mutualinformationempirical(...)[0]`, then this will just pick out the main `result` return variable for the above as desired.\n",
|
|
"3. Next, call `simpleinfotheory.mutualinformationempirical()` again as per step 2 of the previous exercise for lag 1, but this time storing all of these return values:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "539b82af-b94e-4ca1-9362-ba70078c30cb",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Call the mutual information empirical again, this time storing all of the return values.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9ec8b8b0-92ea-444b-9b02-aeb3c01ab679",
|
|
"metadata": {},
|
|
"source": [
|
|
"4. Now, we loop over all possible joint symbols and compute the pointwise mutual information -- fill in the line of the code marked with `???` to compute the pointwise MI and then run this code block:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b3a37011-0238-47d0-85a3-b384573fa2c0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"pointwiseMIs = np.zeros((xSymbols.size, ySymbols.size)); # Create array to store the pointwise MI values for each possible character pair\n",
|
|
"for firstCharIndex in range(xSymbols.size):\n",
|
|
" firstChar = xSymbols[firstCharIndex];\n",
|
|
" probFirst = xProbs[firstCharIndex];\n",
|
|
" for secondCharIndex in range(ySymbols.size):\n",
|
|
" secondChar = ySymbols[secondCharIndex];\n",
|
|
" probSecond = yProbs[secondCharIndex];\n",
|
|
" jointSymbolIndex = np.argwhere((xySymbols[:,0] == firstChar) & (xySymbols[:,1] == secondChar));\n",
|
|
" if (jointSymbolIndex.size == 0):\n",
|
|
" pointwiseMIs[firstCharIndex, secondCharIndex] = 0; # No occurence, so set to 0\n",
|
|
" continue;\n",
|
|
" probJoint = xyProbs[jointSymbolIndex];\n",
|
|
" # Compute the pointwise MI from probJoint, probFirst and probSecond\n",
|
|
" pointwiseMIs[firstCharIndex, secondCharIndex] = np.log2( ??? );"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e7f91288-9d24-4736-ae88-b8f3562d09a4",
|
|
"metadata": {},
|
|
"source": [
|
|
"5. Can you plot these values using `plt.imshow()`? Run the command `plt.colorbar()` to insert a colour bar to show the scale. The plot will have the first letters along the y axis, and second letters along the x axis. You can label these using:\n",
|
|
"\n",
|
|
" <code>plt.xlabel('Second letter')\n",
|
|
" plt.xticks(ticks=range(0,27), labels=ySymbols.flatten()) # second letters - y - goes on x axis\n",
|
|
" plt.ylabel('First letter');\n",
|
|
" plt.yticks(ticks=range(0,27), labels=xSymbols.flatten()) # first letters - x - goes on y axis\n",
|
|
" cbar = plt.colorbar()\n",
|
|
" cbar.set_label('MI (bits)');\n",
|
|
" plt.title('MI between successive letters of text');</code>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c9a980f0-ad8a-4aea-9fbe-5d0a78282db8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Make the heatmap plot:\n",
|
|
"\n",
|
|
"# Add the labels pasting in the code from above:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37b50b62-6af9-4b66-84c3-b2c180ad9d28",
|
|
"metadata": {},
|
|
"source": [
|
|
"6. Examine the values and determine whether you can identify character pairs where the second is highly predictable from the first, and where the first character is misinformative about the second. Can you explain these results?"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.12"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|