mirror of https://github.com/jlizier/jidt
215 lines
9.4 KiB
Plaintext
215 lines
9.4 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Module 4 - What is Information? II\n",
|
|
"\n",
|
|
"Author: Julio Correa, 2020; based on the original Matlab tutorials.<br/>\n",
|
|
"Adaptations by: J. Lizier, 2023-\n",
|
|
"\n",
|
|
"The following block aims to import all the relevant libraries to analyse data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import math"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Preparing your environment\n",
|
|
"\n",
|
|
"As per the previous modules, we want to use functions we have defined in our previous work in other notebooks.\n",
|
|
"\n",
|
|
"You have several options on handling this as per previously. I suggest you update `simpleinfotheory.py` script to **add the new functions you wrote in the last module**, and import the required functions from this."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Option 1: your notebook from Module 1 is complete:\n",
|
|
"# from ipynb.fs.full.Module_1_notebook import entropy\n",
|
|
"# Option 2: you use the Module 1 note book solutions: (if so, ignore the out\n",
|
|
"# from ipynb.fs.full.Module_1_notebook_solutions import entropy\n",
|
|
"# Option 3: edit simpleinfotheory.py and past your functions into that as you write them\n",
|
|
"from simpleinfotheory import entropy, entropyempirical, jointentropy, jointentropyempirical, conditionalentropy, conditionalentropyempirical, mutualinformation, mutualinformationempirical"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 4. Coding conditional mutual information\n",
|
|
"\n",
|
|
"In this exercise we continue to alter the Python code to measure the conditional mutual information between variables $x$ and $y$, conditional on variable $z$, for a distribution $p(x,y,z)$:\n",
|
|
"\n",
|
|
"$I\\left(X;Y\\mid Z\\right)=H\\left(X\\mid Z\\right)+H\\left(Y\\mid Z\\right)-H\\left(X,Y\\mid Z\\right)$\n",
|
|
"\n",
|
|
"For the conditional mutual information, we will focus only on its _empirical_ calculation (for the most part). We will code conditional mutual information $I(X;Y|Z)$ for empirical samples `xn` and `yn` and `zn` in the cell below.\n",
|
|
"\n",
|
|
"1. Find the lines where you need to add code, and do so. _Hint_: You can call your existing code `conditionalentropyempirical` to compute $H(X,Y|Z)$, $H(X|Z)$ and $H(Y|Z)$ respectively, by passing in `np.append(xn, yn, axis=1),zn`, then `xn,zn` and `yn,zn` as arguments to these functions respectively."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"\"\"\"function conditionalmutualinformationempirical(xn,yn,zn)\n",
|
|
"Computes the mutual information over all samples xn of a random\n",
|
|
"variable X with samples yn of a random variable Y, conditioning on \n",
|
|
"samples zn of a random variable Z.\n",
|
|
"\n",
|
|
"Inputs:\n",
|
|
"- xn - numpy matrix of samples of outcomes x. May be a 1D vector of samples, or\n",
|
|
" a 2D matrix, where each row is a vector sample for a multivariate X.\n",
|
|
"- yn - numpy matrix of samples of outcomes y. May be a 1D vector of samples, or\n",
|
|
" a 2D matrix, where each row is a vector sample for a multivariate Y.\n",
|
|
" Must have the same number of rows as X.\n",
|
|
"- zn - numpy matrix of samples of outcomes z. May be a 1D vector of samples, or\n",
|
|
" a 2D matrix, where each row is a vector sample for a multivariate Z\n",
|
|
" which will be conditioned on.\n",
|
|
" Must have the same number of rows as X.\n",
|
|
"\n",
|
|
"Outputs:\n",
|
|
"- result - conditional mutual information of X with Y, given Z\n",
|
|
"\n",
|
|
"Copyright (C) 2020-, Julio Correa, Joseph T. Lizier\n",
|
|
"Distributed under GNU General Public License v3\n",
|
|
"\"\"\"\n",
|
|
"def conditionalmutualinformationempirical(xn, yn, zn):\n",
|
|
" \n",
|
|
" # First, error checking, and converting argument into standard form: \n",
|
|
" xn = np.array(xn)\n",
|
|
" # Convert to column vectors if not already:\n",
|
|
" if xn.ndim == 1:\n",
|
|
" xn = np.reshape(xn,(len(xn),1))\n",
|
|
" yn = np.array(yn)\n",
|
|
" if yn.ndim == 1:\n",
|
|
" yn = np.reshape(yn,(len(yn),1))\n",
|
|
" zn = np.array(zn)\n",
|
|
" if zn.ndim == 1:\n",
|
|
" zn = np.reshape(zn,(len(zn),1))\n",
|
|
" [rx,cx] = xn.shape\n",
|
|
" [ry,cy] = yn.shape\n",
|
|
" [rz,cz] = zn.shape\n",
|
|
"\n",
|
|
" # Should we check any potential error conditions on the input?\n",
|
|
" # Check that their number of rows are the same:\n",
|
|
" assert(rx == ry)\n",
|
|
" assert(rx == rz)\n",
|
|
"\n",
|
|
" # We need to compute H(X|Z) + H(Y|Z) - H(X,Y|Z):\n",
|
|
" # 1. conditional joint entropy:\n",
|
|
" H_XY_given_Z = conditionalentropyempirical(np.append(xn, yn, axis=1),zn); # How to compute this empirically ...?\n",
|
|
" # 2. conditional entropy of Y:\n",
|
|
" H_Y_given_Z = conditionalentropyempirical(yn,zn) # How to compute this empirically ...?\n",
|
|
" # 3. conditional entropy of X:\n",
|
|
" H_X_given_Z = conditionalentropyempirical(xn,zn) # How to compute this empirically ...?\n",
|
|
" \n",
|
|
" # Alternatively, note that we could compute I(X;Y,Z) - I(X;Z)\n",
|
|
" \n",
|
|
" result = H_X_given_Z + H_Y_given_Z - H_XY_given_Z;\n",
|
|
" return result"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"2. Test that your code works by running, e.g.:\n",
|
|
" 1. `conditionalmutualinformationempirical([0,0,1,1],[0,1,0,1],[0,1,0,1])` and validating that you get the result 0 bits.\n",
|
|
" 1. `conditionalmutualinformationempirical([0,0,1,1],[0,0,1,1],[0,1,1,0])` and validating that you get the result 1 bit.\n",
|
|
" 1. `conditionalmutualinformationempirical([0,0,1,1],[0,1,0,1],[0,1,1,0])` and validating that you get the result 1 bit.\n",
|
|
" 1. Can you explain the expected results for these boundary cases?\n",
|
|
" 1. _Challenge_: Let's make a larger empirical test of case c above. First we will generate a large sample of binary values for variable $X$, `X = np.random.randint(0, 2, (1000,1))`, and same for $Z$, `Z = np.random.randint(0, 2, (1000,1))`, then we will construct the samples of $Y$ as the exclusive OR (XOR) of these two, `Y = np.logical_xor(X, Z)`. Validate using `mutualinformationempirical` that there is (almost) no mutual information between either $X$ or $Z$ with $Y$, yet using `conditionalmutualinformationempirical` that there is (almost) one bit of conditional mutual information from $X$ to $Y$ given $Z$ (or vice versa in $X$ and $Z$). Explain the meaning of the conditioning on $Z$ increasing the apparent mutual information between $X$ and $Y$ - see part 5 of the lecture, below. (Also: why are the bit values not quite 0 and 1 in this example?)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0.0\n",
|
|
"1.0\n",
|
|
"1.0\n",
|
|
"I(X;Y) = 0.0001 bits\n",
|
|
"I(Z;Y) = 0.0013 bits\n",
|
|
"I(X;Y|Z) = 0.9987 bits\n",
|
|
"I(Z;Y|X) = 0.9998 bits\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Test the code here\n",
|
|
"print( conditionalmutualinformationempirical([0,0,1,1],[0,1,0,1],[0,1,0,1]) )\n",
|
|
"print( conditionalmutualinformationempirical([0,0,1,1],[0,0,1,1],[0,1,1,0]) )\n",
|
|
"print( conditionalmutualinformationempirical([0,0,1,1],[0,1,0,1],[0,1,1,0]) )\n",
|
|
"\n",
|
|
"X = np.random.randint(0, 2, (1000,1))\n",
|
|
"Z = np.random.randint(0, 2, (1000,1))\n",
|
|
"Y = np.logical_xor(X,Z)\n",
|
|
"print( \"I(X;Y) = %.4f bits\" % mutualinformationempirical(X,Y)[0] ) # My solution code returns multiple values, just taking the result part\n",
|
|
"print( \"I(Z;Y) = %.4f bits\" % mutualinformationempirical(Z,Y)[0] ) # # My solution code returns multiple values, just taking the result part\n",
|
|
"print( \"I(X;Y|Z) = %.4f bits\" % conditionalmutualinformationempirical(X,Y,Z) )\n",
|
|
"print( \"I(Z;Y|X) = %.4f bits\" % conditionalmutualinformationempirical(Z,Y,X) )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"3. _Challenge_: Can you alter the code in `conditionalmutualinformationempirical` to compute conditional mutual information $I(X;Y|Z)$ using the expression $I(X;Y|Z) = I(X;Y,Z) - I(X;Z)$?\n",
|
|
"\n",
|
|
"4. _Challange_: We did not code a function for `conditionalmutualinformation` in this exercise - an implementation is provided for you however in the solutions (see below). Can you read the code and understand how this is calculating the conditional mutual information for the given probability table `p`? Note that the argument `p` would be a 3D matrix, representing the probability $p(x,y,z)$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.12"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|