mirror of https://github.com/jlizier/jidt
183 lines
8.1 KiB
Plaintext
183 lines
8.1 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Module 4 - What is Information? II\n",
|
|
"\n",
|
|
"Author: Julio Correa, 2020; based on the original Matlab tutorials.<br/>\n",
|
|
"Adaptations by: J. Lizier, 2023-\n",
|
|
"\n",
|
|
"The following block aims to import all the relevant libraries to analyse data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import math"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Preparing your environment\n",
|
|
"\n",
|
|
"As per the previous modules, we want to use functions we have defined in our previous work in other notebooks.\n",
|
|
"\n",
|
|
"You have several options on handling this as per previously. I suggest you update `simpleinfotheory.py` script to **add the new functions you wrote in the last module**, and import the required functions from this."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Option 1: your notebook from Module 1 is complete:\n",
|
|
"# from ipynb.fs.full.Module_1_notebook import entropy\n",
|
|
"# Option 2: you use the Module 1 note book solutions: (if so, ignore the out\n",
|
|
"# from ipynb.fs.full.Module_1_notebook_solutions import entropy\n",
|
|
"# Option 3: edit simpleinfotheory.py and past your functions into that as you write them\n",
|
|
"from simpleinfotheory import entropy, entropyempirical, jointentropy, jointentropyempirical, conditionalentropy, conditionalentropyempirical, mutualinformation, mutualinformationempirical"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 4. Coding conditional mutual information\n",
|
|
"\n",
|
|
"In this exercise we continue to alter the Python code to measure the conditional mutual information between variables $x$ and $y$, conditional on variable $z$, for a distribution $p(x,y,z)$:\n",
|
|
"\n",
|
|
"$I\\left(X;Y\\mid Z\\right)=H\\left(X\\mid Z\\right)+H\\left(Y\\mid Z\\right)-H\\left(X,Y\\mid Z\\right)$\n",
|
|
"\n",
|
|
"For the conditional mutual information, we will focus only on its _empirical_ calculation (for the most part). We will code conditional mutual information $I(X;Y|Z)$ for empirical samples `xn` and `yn` and `zn` in the cell below.\n",
|
|
"\n",
|
|
"1. Find the lines where you need to add code, and do so. _Hint_: You can call your existing code `conditionalentropyempirical` to compute $H(X,Y|Z)$, $H(X|Z)$ and $H(Y|Z)$ respectively, by passing in `np.append(xn, yn, axis=1),zn`, then `xn,zn` and `yn,zn` as arguments to these functions respectively."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"\"\"\"function conditionalmutualinformationempirical(xn,yn)\n",
|
|
"Computes the mutual information over all samples xn of a random\n",
|
|
"variable X with samples yn of a random variable Y, conditioning on \n",
|
|
"samples zn of a random variable Z.\n",
|
|
"\n",
|
|
"Inputs:\n",
|
|
"- xn - numpy matrix of samples of outcomes x. May be a 1D vector of samples, or\n",
|
|
" a 2D matrix, where each row is a vector sample for a multivariate X.\n",
|
|
"- yn - numpy matrix of samples of outcomes y. May be a 1D vector of samples, or\n",
|
|
" a 2D matrix, where each row is a vector sample for a multivariate Y.\n",
|
|
" Must have the same number of rows as X.\n",
|
|
"- zn - numpy matrix of samples of outcomes z. May be a 1D vector of samples, or\n",
|
|
" a 2D matrix, where each row is a vector sample for a multivariate Z\n",
|
|
" which will be conditioned on.\n",
|
|
" Must have the same number of rows as X.\n",
|
|
"\n",
|
|
"Outputs:\n",
|
|
"- result - conditional mutual information of X with Y, given Z\n",
|
|
"\n",
|
|
"Copyright (C) 2020-, Julio Correa, Joseph T. Lizier\n",
|
|
"Distributed under GNU General Public License v3\n",
|
|
"\"\"\"\n",
|
|
"def conditionalmutualinformationempirical(xn, yn, zn):\n",
|
|
" \n",
|
|
" # First, error checking, and converting argument into standard form: \n",
|
|
" xn = np.array(xn)\n",
|
|
" # Convert to column vectors if not already:\n",
|
|
" if xn.ndim == 1:\n",
|
|
" xn = np.reshape(xn,(len(xn),1))\n",
|
|
" yn = np.array(yn)\n",
|
|
" if yn.ndim == 1:\n",
|
|
" yn = np.reshape(yn,(len(yn),1))\n",
|
|
" zn = np.array(zn)\n",
|
|
" if zn.ndim == 1:\n",
|
|
" zn = np.reshape(zn,(len(zn),1))\n",
|
|
" [rx,cx] = xn.shape\n",
|
|
" [ry,cy] = yn.shape\n",
|
|
" [rz,cz] = zn.shape\n",
|
|
"\n",
|
|
" # Should we check any potential error conditions on the input?\n",
|
|
" # Check that their number of rows are the same:\n",
|
|
" assert(rx == ry)\n",
|
|
" assert(rx == rz)\n",
|
|
"\n",
|
|
" # We need to compute H(X|Z) + H(Y|Z) - H(X,Y|Z):\n",
|
|
" # 1. conditional joint entropy:\n",
|
|
" H_XY_given_Z = ???; # How to compute this empirically ...?\n",
|
|
" # 2. conditional entropy of Y:\n",
|
|
" H_Y_given_Z = ???; # How to compute this empirically ...?\n",
|
|
" # 3. conditional entropy of X:\n",
|
|
" H_X_given_Z = ???; # How to compute this empirically ...?\n",
|
|
" \n",
|
|
" # Alternatively, note that we could compute I(X;Y,Z) - I(X;Z)\n",
|
|
" \n",
|
|
" result = H_X_given_Z + H_Y_given_Z - H_XY_given_Z;\n",
|
|
" return result"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"2. Test that your code works by running, e.g.:\n",
|
|
" 1. `conditionalmutualinformationempirical([0,0,1,1],[0,1,0,1],[0,1,0,1])` and validating that you get the result 0 bits.\n",
|
|
" 1. `conditionalmutualinformationempirical([0,0,1,1],[0,0,1,1],[0,1,1,0])` and validating that you get the result 1 bit.\n",
|
|
" 1. `conditionalmutualinformationempirical([0,0,1,1],[0,1,0,1],[0,1,1,0])` and validating that you get the result 1 bit.\n",
|
|
" 1. Can you explain the expected results for these boundary cases?\n",
|
|
" 1. _Challenge_: Let's make a larger empirical test of case c above. First we will generate a large sample of binary values for variable $X$, `X = np.random.randint(0, 2, (1000,1))`, and same for $Z$, `Z = np.random.randint(0, 2, (1000,1))`, then we will construct the samples of $Y$ as the exclusive OR (XOR) of these two, `Y = np.logical_xor(X, Z)`. Validate using `mutualinformationempirical` that there is (almost) no mutual information between either $X$ or $Z$ with $Y$, yet using `conditionalmutualinformationempirical` that there is (almost) one bit of conditional mutual information from $X$ to $Y$ given $Z$ (or vice versa in $X$ and $Z$). Explain the meaning of the conditioning on $Z$ increasing the apparent mutual information between $X$ and $Y$ - see the next lecture segment. (Also: why are the bit values not quite 0 and 1 in this example?)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Test the code here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"3. _Challenge_: Can you alter the code in `conditionalmutualinformationempirical` to compute conditional mutual information $I(X;Y|Z)$ using the expression $I(X;Y|Z) = I(X;Y,Z) - I(X;Z)$?\n",
|
|
"\n",
|
|
"4. _Challenge_: We did not code a function for `conditionalmutualinformation` in this exercise - an implementation is provided for you however in the solutions. Can you read the code and understand how this is calculating the conditional mutual information for the given probability table `p`? Note that the argument `p` would be a 3D matrix, representing the probability $p(x,y,z)$."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.12"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|