{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "87e17b68-7539-4539-aeab-9df6fc819f22",
   "metadata": {},
   "source": [
    "# AIS in written English\n",
    "\n",
    "Author: J. Lizier, 2023-; based on the original Matlab tutorials.\n",
    "\n",
    "In this activity we will revisit the written English data set we first use in Module 2, considering it as a time series (where later characters have later time indices), and examine Active Information Storage (AIS) that past characters provide about the next.\n",
    "\n",
    "1. We will start by taking copies of relevant parts of your saved code from the CA activity in the AIS module. We are going to use that code to recreate a similar bias corrected AIS versus k plot for this data set.\n",
    "    1. For the first cell below, copy and paste the import/startup lines from your CA notebook.\n",
    "    2. For the second cell below, leave the code as is, aside from changing the path to where your data set is. Notice that there is a little more data plumbing this time. The vector `processedStr` holds each character in character format, but we will need to supply our symbols to JIDT as integers in the range `[0,alphabetSize-1]`, where `alphabetSize` is 27 (26 letters plus the space). So we convert the `processedStr` to a time series of integers.\n",
    "    3. For the third cell, copy and paste the code from the CA notebook which computed and plotted AIS vs k (the last code cell before we went on to examine the local AIS values).\n",
    "2. Now, make sure to delete the lines in the third cell which load in the CA data (we're no longer using this!)\n",
    "3. Now, let's check what we need to change about the calculation of AIS for each value of k in the main loop:\n",
    "    1. When we worked with the CAs, the alphabet size we were dealing with there was 2. For this data set it will be 27. Can you see where we tell the estimator this? If unsure, go back to the AutoAnalyser, change the `base` parameter to 27, and see where this is placed in the generated code.\n",
    "    2. Remove the loop over variables `v` in the code, since we've only got one time series here (unlike the CAs example where we have a time series for each cell or variable `v`). Pass your whole time series `data` to the estimator in a single call to `calc.addObservations()`\n",
    "4. Finally, change your loop over the value of `k` to stop at 4. At the moment, the multivariate state space here for `k=5` with an alphabet size of 27 is too large for JIDT to handle (I have some new algorithms coming!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7307f279-64fd-4596-b279-4b26df51f698",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paste the import and JVM startup lines here (copy from the CA activity):\n",
    "\n",
    "from jpype import *\n",
    "import numpy\n",
    "import sys\n",
    "# Our python data file readers are a bit of a hack, python users will do better on this:\n",
    "sys.path.append(\"/home/joseph/JIDT/infodynamics-dist-1.6.1/demos/python\")\n",
    "import readIntsFile\n",
    "\n",
    "if (not isJVMStarted()):\n",
    "    # Add JIDT jar library to the path\n",
    "    jarLocation = \"/home/joseph/JIDT/infodynamics-dist-1.6.1/infodynamics.jar\"\n",
    "    # Start the JVM (add the \"-Xmx\" option with say 1024M if you get crashes due to not enough memory space)\n",
    "    startJVM(getDefaultJVMPath(), \"-ea\", \"-Djava.class.path=\" + jarLocation, convertStrings=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "55ca6967-2ca7-45e8-9858-6bb0356c2bee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specifics required for the text processing here:\n",
    "import string\n",
    "import re\n",
    "\n",
    "# Load and pre-parse the scripts\n",
    "filename = '/home/joseph/TeachingPlayground/CSYS5030/WrittenText/Seinfeld-scripts-textOnly.txt'\n",
    "with open(filename, 'rt') as f:\n",
    "    str = f.read()\n",
    "p = re.compile('[!\"#\\$%&\\'\\(\\)\\*\\+\\,-\\.\\/:;<=>\\?@\\[\\]\\\\\\^_`{\\|}~0-9]*');\n",
    "processedStr = p.sub('', str); # Remove punctuation characters and digits\n",
    "processedStr = ' '.join(processedStr.split('\\n')); # Replace newline characters with spaces\n",
    "processedStr = processedStr.lower(); # Convert all upper case into lower case\n",
    "processedStr = numpy.array(list(processedStr)); # Finally convert this into a numpy array so we can work with it\n",
    "numpy.unique(processedStr)\n",
    "\n",
    "data = processedStr.view(numpy.int32) - ord('a'); # Converts the letters to an integer representing their offset from 'a'.\n",
    "# Use of int32 rather than int64 is important to ensure compatibility with Java int size.\n",
    "# But spaces have been turned into the value ord(' ') - ord('a') (i.e. the difference in character encodings of space and 'a').\n",
    "# We need to replace this with the integer value 26, such that all of our symbols are consecutive for JIDT:\n",
    "data[data == ord(' ') - ord('a')] = 26;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ab68f364-d9ed-45d3-8408-ce724d0233c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIS_Discrete(k=1, 2711979 samples) = 0.7196 bits (bias 0.0001, bias corrected 0.7195)\n",
      "AIS_Discrete(k=2, 2711978 samples) = 1.4587 bits (bias 0.0035, bias corrected 1.4552)\n",
      "AIS_Discrete(k=3, 2711977 samples) = 1.9838 bits (bias 0.0943, bias corrected 1.8895)\n",
      "AIS_Discrete(k=4, 2711976 samples) = 2.3036 bits (bias 2.5475, bias corrected -0.2439)\n",
      "Optimal k=3, giving bias corrected AIS(k=3)=1.8895 bits\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 20604 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Paste the code computing AIS vs k here (copy from the CA activity and then we'll adapt):\n",
    "\n",
    "kMax = 4;\n",
    "results = numpy.zeros((kMax))\n",
    "biases = numpy.zeros((kMax))\n",
    "biasCorrectedResults = numpy.zeros((kMax))\n",
    "\n",
    "for k in range(1,kMax+1):\n",
    "    # 1. Construct the calculator:\n",
    "    calcClass = JPackage(\"infodynamics.measures.discrete\").ActiveInformationCalculatorDiscrete\n",
    "    calc = calcClass(27, k)\n",
    "    # 2. No other properties to set for discrete calculators.\n",
    "    \n",
    "    # 3. Initialise the calculator for (re-)use:\n",
    "    calc.initialise()\n",
    "    \n",
    "    # Pass in the whole time series at once now:\n",
    "    variable = JArray(JInt, 1)(data.tolist())\n",
    "    \n",
    "    # 4. Supply the sample data:\n",
    "    calc.addObservations(variable)\n",
    "    \n",
    "    # 5. Compute the estimate:\n",
    "    result = calc.computeAverageLocalOfObservations()\n",
    "    # 6. Compute the (statistical significance via) null distribution analytically:\n",
    "    measDist = calc.computeSignificance()\n",
    "    bias = measDist.getMeanOfDistribution()\n",
    "\n",
    "    # Store the results:\n",
    "    results[k-1] = result\n",
    "    biases[k-1] = bias\n",
    "    biasCorrectedResults[k-1] = result - bias\n",
    "    \n",
    "    print(\"AIS_Discrete(k=%d, %d samples) = %.4f bits (bias %.4f, bias corrected %.4f)\" %\\\n",
    "        (k, calc.getNumObservations(), result, bias, result - bias))\n",
    "\n",
    "optimalK = numpy.argmax(biasCorrectedResults)+1 # Need to add 1 for the array offset\n",
    "print('Optimal k=%d, giving bias corrected AIS(k=%d)=%.4f bits' % (optimalK,optimalK,biasCorrectedResults[optimalK-1])) # subtract 1 for array indexing\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(range(1,kMax+1), results, marker='x', color='red', label='AIS raw')\n",
    "plt.scatter(range(1,kMax+1), biases, marker='o', color='blue', label='bias')\n",
    "plt.scatter(range(1,kMax+1), biasCorrectedResults, marker='+', color='green', label='bias corrected AIS')\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('AIS (bits)')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc5119d3-c9d3-4139-ad43-3346098ea85d",
   "metadata": {},
   "source": [
    "5. Now run the code, and observe how AIS -- both the raw and bias corrected values -- change with history length k.\n",
    "    * What value of k should we use when computing AIS for this data set?\n",
    "    * From this data set, how many previous characters are we able to confidently say have an impact on predicting the next character?\n",
    "    * What sources of error or limitations might there be in our experiment?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50cfe176-4bf8-44a1-86f6-0ef354486190",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
