natas_hw
/
jidt
mirror of https://github.com/jlizier/jidt
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
jidt/course/Module01-Entropy/PythonSimpleFunctions/Module_1_notebook.ipynb

331 lines
12 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Module 1 - Uncertainty and Entropy I\n",
"\n",
"Author: Julio Correa, 2020; based on the original Matlab tutorials.<br/>\n",
"Adaptations by: J. Lizier, 2023-\n",
"\n",
"The following block aims to import all the relevant libraries to analyse data"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import math"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Coding Shannon information content\n",
"\n",
"During our Introduction to Information Theory block, we will alter several Python functions in order to compute information-theoretic quantities.\n",
"\n",
"Our first activity with these templates is to implement the Shannon information content:\n",
"\n",
"$h\\left(x\\right)=\\log_2{\\left( \\frac{1}{p(x)} \\right)}=-\\log_2 {p(x)}$\n",
"\n",
"1. Edit the Python function <code>infocontent(p)</code> below to return the Shannon information content for an outcome $x$ with probability $p(x)$. Make sure that you use the function <code>np.log2()</code> rather than <code>np.log()</code> to get your answers in bits rather than nats.\n",
" 1. the value that we want is <code>-np.log2(p)</code>\n",
" 1. and we can assign this to be returned from the function by calling this from the return line: <code>return -np.log2(p)</code>\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"function infocontent(p)\n",
"Computes the Shannon information content for an outcome x of a random variable\n",
"X with probability p.\n",
"\n",
"Inputs:\n",
"- p - probability to compute the Shannon info content for\n",
"\n",
"Outputs:\n",
"- result - Shannon info content of the probability p\n",
"\n",
"Copyright (C) 2020-, Julio Correa, Joseph T. Lizier\n",
"Distributed under GNU General Public License v3\n",
"\"\"\"\n",
"\n",
"def infocontent(p):\n",
" \n",
" # Alter the equation below to provide the correct Shannon information \n",
" # content:\n",
"\n",
" return None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2. To evaluate a function in Python, we type it's name with an appropriate argument supplied in brackets. For example, to evaluate the Shannon information content with our function for an outcome with had probability 0.2, you would call:\n",
"<code>infocontent(0.2)</code>\n",
"If you want to see the output printed to the screen, then enclose this in a <code>print</code> function:\n",
"<code>print(infocontent(0.2))</code>\n",
"\n",
" Compute the following using your function:\n",
" - h(heads) for a fair coin?\n",
" - h(1) for a 6-sided die? h(not 1) for a 6-sided die?\n",
" - h(1) for a 20-sided die? h(not 1) for a 20-sided die?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# h(heads) for a fair coin?\n",
" \n",
"# h(1) for a 6-sided die?\n",
"\n",
"# h(not 1) for a 6-sided die?\n",
"\n",
"# h(1) for a 20-sided die?\n",
"\n",
"# h(not 1) for a 20-sided die?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. Reproduce the plot below of $h(x)$ versus $p(x)$ using the matlibplot <code>plot()</code> function.\n",
"\n",
" Hints:\n",
" 1. Input <code>p</code> to <code>infocontent(p)</code> as a vector across the range <code>p = np.arange(0.01,1.001,0.01)</code>.\n",
" 2. Make an inline plot with matplotlib by calling <code>plt.plot(x, y)</code>, with <code>p</code> and <code>infocontent(p)</code>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"./ShannonInfoContentVersusP.png\" width=\"400\"/>\n",
"</div>\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Define the array for p\n",
"\n",
"# Compute the infocontent() of the array p\n",
"\n",
"# Make the plot and don't forget to label axes\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. Have a look at the characters that we could play Guess Who? with on the [Kooky characters sheet](https://web.archive.org/web/20170215034006/http://www.hasbro.com/upload/guesswho/GWc_Kooky-en_GB.pdf). Assuming that your partner selects one of these characters at random, compute the probability and then the Shannon information content of their character:\n",
" 1. being Jason?\n",
" 2. having one eye?\n",
" 3. having more than one eye?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# h(Jason)?\n",
"\n",
"# h(one eye)?\n",
"\n",
"# h(more than one eye)?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5. Based on those answers, reflect on the following:\n",
" 1. Would a good first question be \"does your character have one eye?\" ? Why / why not?\n",
" 2. Would a good first question be \"are you Jason?\" ? Why / why not?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. Coding Shannon entropy\n",
"\n",
"In this exercise we continue to generate Python code to measure the Shannon entropy for a distribution $p(x)$:\n",
"\n",
"$H(X)=-\\sum_xp\\left(x\\right)\\log_2p\\left(x\\right)$\n",
"\n",
"Your task is to edit the Python function <code>entropy(p)</code> in the next cell to return the Shannon entropy for the given distribution $p(x)$ over outcomes $x$ of $X$.\n",
"\n",
"Note the input argument to the function is a vector <code>p</code>, representing the probability mass for each outcome of $x$. That is, <code>p</code> is a vector with the $n$th entry in the vector giving the probability for the $n$th value that $x$ may take. The sum of the items in the vector <code>p</code> must be 1.\n",
"\n",
"For example, for a binary $x$ we could have <b>p = np.array([0.25, 0.75])</b> where $p(x=0) = 0.25$ and $p(x=1) = 0.75$.\n",
"\n",
" - If we knew x was a binary variable, and we only took one argument, $p = p(x=1)$, how could you write one line of code to compute H(X) from p? (_Hint_: what would $\\log_2(p(x=1))$ be as a function of $p$? What would $\\log_2(p(x=0))$ be as a function of $p$? Can you combine these to give $H(X)$ ?) \n",
"\n",
"Let's assume that we don't know how many values $x$ could take, and write the code for an arbitrary length vector $p$.\n",
"\n",
"1. Can you think of two ways to write the code to sum up the contribution for each item $p(x)$ in the vector $x$, being:\n",
" 1. for loop over the items of p, or<br>\n",
" 1. the sum of a vector multiplication or dot product in Python?\n",
"\n",
" Implement one of these in <code>entropy(p)</code>. (Usually the latter is faster)\n",
"\n",
"2. Think of possible error conditions here, and how you can handle these in your code."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"function entropy(p)\n",
"Computes the Shannon entropy for a probability distribution p.\n",
"\n",
"Inputs:\n",
"- p - (numpy array or list which much sum to 1) - a probability distribution to compute the Shannon info content for\n",
"\n",
"Outputs:\n",
"- result - Shannon entropy of the probability distribution p\n",
"\n",
"Copyright (C) 2020-, Julio Correa, Joseph T. Lizier\n",
"Distributed under GNU General Public License v3\n",
"\"\"\"\n",
"def entropy(p): \n",
" # Should we check any potential error conditions on the input?\n",
"\n",
" # First make sure the array is now a numpy array\n",
" if type(p) != np.array:\n",
" p = np.array(p)\n",
"\n",
" # We need to take the expectation value over the Shannon info content at\n",
" # p(x) for each outcome x:\n",
" # Alter the equation below to provide the correct entropy:\n",
" return None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. Write down the answer you expect, and test that your code gives answers you expect for:\n",
" 1. <code>entropy([0.5, 0.5])</code>\n",
" 2. <code>entropy([0.25, 0.25, 0.25, 0.25])</code>\n",
" 3. <code>entropy([1, 0])</code>"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# entropy([0.5, 0.5]) ?\n",
"\n",
"# entropy([0.25, 0.25, 0.25, 0.25]) ?\n",
"\n",
"# entropy([1, 0]) ?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. _Challenge_: Plot $H(X)$ as a function of $p(x=1)$ for binary $X$. (See the plot we expect on the figure below). This will involve a loop over values of $p = p(x=1)$ to call the <code>entropy(p)</code> function with a vector corresponding to each $\\{p(x=1), p(x=0)\\}$ pair."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"./ShannonEntropyVersusP.png\" width=\"400\"/>\n",
"</div>\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# Add your code to plot H(X) vs p(x=1) below, and don't forget to label axes\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5. Coming back to the characters that we could play Guess Who? with on the [Kooky characters sheet](https://web.archive.org/web/20170215034006/http://www.hasbro.com/upload/guesswho/GWc_Kooky-en_GB.pdf), validate that (using your <code>entropy</code> function):\n",
" 1. $H(who) = 4.585$ bits  (entropy of the character's identity)\n",
" 1. $H(one\\ eye?) = 0.738$ bits  (entropy of whether the character has one eye or more than one eye)\n",
" 1. $H(Jason) = 0.2499$ bits  (entropy of whether the character is Jason or not)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Add your code here to validate the entropies as above:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"6. Based on those answers and in comparison to your responses on the earlier exercise, reflect on the following:\n",
" 1. Would a good first question be _\"does your character have one eye?\"_ ? Why / why not?\n",
" 1. Would a good first question be _\"are you Jason?\"_ ? Why / why not?\n",
" 1. What is the best question to ask first that you can think of, and why? (as an optional tangent, you could watch [a video](https://youtu.be/FRlbNOno5VA) which goes into some detail about what the best questions might be -- again, think about the information-theoretic view on what is being said there)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink