{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# TensorFlow Example\n",
"\n",
"\n",
"\n",
"\n",
"_{*Photo credit: TensorFlow* "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"[![Intro to Machine Learning](https://img.youtube.com/vi/KNAWp2S3w94/0.jpg)](https://www.youtube.com/watch?v=KNAWp2S3w94 \"Intro to Machine Learning\") \n",
"*This tutorial is based on Lawrence Moroney's excellent video tutorial [Intro to Machine Learning](https://www.youtube.com/watch?v=KNAWp2S3w94).*"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"- First of all, we create some data in Python. \n",
"- In particular, we create two variables *x* and *y* with 6 values respectively. \n",
"\n",
"A Python library called **NumPy** provides lots of array type data structures to do this. \n",
"\n",
"1. Import the library ``numpy`` as ``np`` and \n",
"2. specify the values for *x* and *y* as an array in NumPy with ``np.array[]``. We use datatype *float*."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# Create some data\n",
"import numpy as np\n",
"\n",
"x = np.array([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
"y = np.array([-2.0, 1.0, 4.0, 7.0, 10.0, 13.0], dtype=float)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Now let's take a look at the numbers. \n",
"Can you see the relationship between them?\n",
"\n",
"\n",
"| Variable | Value 1 | Value 2 | Value 3 | Value 4 | Value 5 | Value 6 |\n",
"|---|---|---|---|---|---|---|\n",
"| x | -1 | 0 | 1 | 2 | 3 | 4 |\n",
"| y | -2 | 1 | 4 | 7 | 10 | 13 |"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"As you look at them, you might notice that the value of x is increasing by 1 as you read left to right and the corresponding value of y is increasing by 3. \n",
"You probably think that y equals 3 times x plus or minus something. Then, you'd probably look at the 0 on x and see that y is 1, and you'd come up with the relationship: \n",
"\n",
"$y = 3x + 1.$\n",
"\n",
"That's almost exactly how you would use code to train a model to spot the patterns in the data! How would you train a neural network to do the equivalent task? By feeding it with your set of x's and y's, it should be able to figure out the relationship between them.\n",
"\n",
"But before we train our model, let`s first use the Python library **seaborn** to visualize the realtionship between x and y."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
" (function(spec, embedOpt){\n",
" let outputDiv = document.currentScript.previousElementSibling;\n",
" if (outputDiv.id !== \"altair-viz-39ab8759178343a69a12f0f566072ee5\") {\n",
" outputDiv = document.getElementById(\"altair-viz-39ab8759178343a69a12f0f566072ee5\");\n",
" }\n",
" const paths = {\n",
" \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
" \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
" \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.17.0?noext\",\n",
" \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
" };\n",
"\n",
" function maybeLoadScript(lib, version) {\n",
" var key = `${lib.replace(\"-\", \"\")}_version`;\n",
" return (VEGA_DEBUG[key] == version) ?\n",
" Promise.resolve(paths[lib]) :\n",
" new Promise(function(resolve, reject) {\n",
" var s = document.createElement('script');\n",
" document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
" s.async = true;\n",
" s.onload = () => {\n",
" VEGA_DEBUG[key] = version;\n",
" return resolve(paths[lib]);\n",
" };\n",
" s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
" s.src = paths[lib];\n",
" });\n",
" }\n",
"\n",
" function showError(err) {\n",
" outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
" throw err;\n",
" }\n",
"\n",
" function displayChart(vegaEmbed) {\n",
" vegaEmbed(outputDiv, spec, embedOpt)\n",
" .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
" }\n",
"\n",
" if(typeof define === \"function\" && define.amd) {\n",
" requirejs.config({paths});\n",
" require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
" } else {\n",
" maybeLoadScript(\"vega\", \"5\")\n",
" .then(() => maybeLoadScript(\"vega-lite\", \"4.17.0\"))\n",
" .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
" .catch(showError)\n",
" .then(() => displayChart(vegaEmbed));\n",
" }\n",
" })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-9103e5c76b5f6cfa1b8d791d17bc6753\"}, \"mark\": \"point\", \"encoding\": {\"x\": {\"field\": \"x\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"y\", \"type\": \"quantitative\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.17.0.json\", \"datasets\": {\"data-9103e5c76b5f6cfa1b8d791d17bc6753\": [{\"x\": -1.0, \"y\": -2.0}, {\"x\": 0.0, \"y\": 1.0}, {\"x\": 1.0, \"y\": 4.0}, {\"x\": 2.0, \"y\": 7.0}, {\"x\": 3.0, \"y\": 10.0}, {\"x\": 4.0, \"y\": 13.0}]}}, {\"mode\": \"vega-lite\"});\n",
""
],
"text/plain": [
"alt.Chart(...)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Plot relationship with scatterplot\n",
"import pandas as pd\n",
"import altair as alt\n",
"\n",
"import warnings\n",
"warnings.simplefilter(action='ignore', category=FutureWarning)\n",
"\n",
"df = pd.DataFrame({\"x\":x, \"y\":y})\n",
"\n",
"alt.Chart(df).mark_point().encode(\n",
" x=\"x\",\n",
" y=\"y\"\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"\n",
"# Neural Network\n",
"\n",
"- We import **TensorFlow** and calling it ``tf`` for ease of use. \n",
"- The framework for defining a neural network as a set of sequential layers is called ``keras``, so import that, too."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-01-12 14:33:57.109574: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
"To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"TensorFlow version 2.11.0\n"
]
}
],
"source": [
"# Import TensorFlow and Keras \n",
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"\n",
"print(f\"TensorFlow version {tf.__version__}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Define and compile the neural network\n",
"\n",
"First, we create the simplest possible neural network:\n",
"\n",
"- it has one layer (``keras.layers.Dense()``), \n",
"- that layer has one neuron (``units=1``), \n",
"- and the input shape to it is only one value (``input_shape=[1]``)."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-01-12 14:34:44.630059: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
"To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
]
}
],
"source": [
"# Define neural network\n",
"\n",
"model = tf.keras.Sequential([\n",
" keras.layers.Dense(units=1, input_shape=[1])])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"- Next, we write the code to ``compile`` your neural network. \n",
"- When you do so, you need to specify two functions: \n",
"\n",
" - an ``optimizer`` and\n",
" - a ``loss`` function.\n",
"\n",
"\n",
"- We use **mean_squared_error** for the loss and **stochastic gradient descent (sgd)** for the optimizer. \n",
"- You don't need to understand the math for those yet, but you can see that they work!"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# Compile neural network\n",
"\n",
"model.compile(optimizer='sgd', \n",
" loss='mean_squared_error')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"In this example, you know that the relationship between the numbers is $y=3x+1$. \n",
"\n",
"1. When the computer is trying to learn that, it makes a guess, maybe $y=10x+10$. \n",
"2. The **loss** function measures the guessed answers against the known correct answers and measures how well or badly it did.\n",
"3. Next, the model uses the **optimizer** function to make another guess. \n",
"4. Based on the **loss** function's result, it tries to minimize the loss. At this point, maybe it will come up with something like $y=5x+5$. While that's still pretty bad, it's closer to the correct result (the loss is lower).\n",
"5. The model repeats that for the number of **epochs**, which you'll see shortly.\n",
"\n",
"Now you have all the code you need to define the neural network. The next step is to train it to see if it can infer the patterns between those numbers and use them to create a model."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"## Train the neural network\n",
"\n",
"The process of training the neural network, where it learns the relationship between the x's and y's, is in the ``model.fit`` call. That's where it will go through the loop before making a guess, measuring how good or bad it is (the loss), or using the optimizer to make another guess. It will do that for the number of **epochs** that you specify. \n",
"\n",
"When you run that code, you'll see the loss will be printed out for each epoch. \n",
"\n",
"For example, you can see that for the first few epochs, the loss value is quite large, but it's getting smaller with each step. As the training progresses, the loss soon gets very small.\n",
"By the time the training is done, the loss is extremely small, showing that our model is doing a great job of inferring the relationship between the numbers.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/50\n",
"1/1 [==============================] - 0s 258ms/step - loss: 72.6197\n",
"Epoch 2/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 57.1348\n",
"Epoch 3/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 44.9520\n",
"Epoch 4/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 35.3671\n",
"Epoch 5/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 27.8262\n",
"Epoch 6/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 21.8933\n",
"Epoch 7/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 17.2256\n",
"Epoch 8/50\n",
"1/1 [==============================] - 0s 7ms/step - loss: 13.5532\n",
"Epoch 9/50\n",
"1/1 [==============================] - 0s 5ms/step - loss: 10.6640\n",
"Epoch 10/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 8.3909\n",
"Epoch 11/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 6.6024\n",
"Epoch 12/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 5.1954\n",
"Epoch 13/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 4.0884\n",
"Epoch 14/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 3.2174\n",
"Epoch 15/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 2.5321\n",
"Epoch 16/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 1.9930\n",
"Epoch 17/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 1.5688\n",
"Epoch 18/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 1.2350\n",
"Epoch 19/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.9725\n",
"Epoch 20/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.7658\n",
"Epoch 21/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.6033\n",
"Epoch 22/50\n",
"1/1 [==============================] - 0s 5ms/step - loss: 0.4754\n",
"Epoch 23/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.3747\n",
"Epoch 24/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.2955\n",
"Epoch 25/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.2332\n",
"Epoch 26/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.1841\n",
"Epoch 27/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.1455\n",
"Epoch 28/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.1151\n",
"Epoch 29/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.0912\n",
"Epoch 30/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0724\n",
"Epoch 31/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.0576\n",
"Epoch 32/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0459\n",
"Epoch 33/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.0367\n",
"Epoch 34/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.0294\n",
"Epoch 35/50\n",
"1/1 [==============================] - 0s 3ms/step - loss: 0.0237\n",
"Epoch 36/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0192\n",
"Epoch 37/50\n",
"1/1 [==============================] - 0s 5ms/step - loss: 0.0156\n",
"Epoch 38/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0128\n",
"Epoch 39/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0106\n",
"Epoch 40/50\n",
"1/1 [==============================] - 0s 5ms/step - loss: 0.0088\n",
"Epoch 41/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0075\n",
"Epoch 42/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0063\n",
"Epoch 43/50\n",
"1/1 [==============================] - 0s 5ms/step - loss: 0.0055\n",
"Epoch 44/50\n",
"1/1 [==============================] - 0s 7ms/step - loss: 0.0048\n",
"Epoch 45/50\n",
"1/1 [==============================] - 0s 5ms/step - loss: 0.0042\n",
"Epoch 46/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0038\n",
"Epoch 47/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0034\n",
"Epoch 48/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0031\n",
"Epoch 49/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0028\n",
"Epoch 50/50\n",
"1/1 [==============================] - 0s 4ms/step - loss: 0.0027\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Train neural network\n",
"\n",
"model.fit(x, y, epochs=50)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"## Use the model\n",
"\n",
"You now have a model that has been trained to learn the relationship between x and y. You can use the ``model.predict`` method to have it figure out the y for a previously unknown x. \n",
"\n",
"For example, if x is 10, what do you think y will be? Take a guess before you run the following code:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1/1 [==============================] - 0s 70ms/step\n",
"[[30.779314]]\n"
]
}
],
"source": [
"# Make a prediction\n",
"\n",
"print(model.predict([10.0]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"You might have thought 31, but it ended up not exactly there. Why do you think that is?\n",
"\n",
"Neural networks deal with probabilities, so it calculated that there is a very high probability that the relationship between x and y is $y=3x+1$, but it can't know for sure with only six data points. Therefore, the result is very close to 31, but not necessarily 31.\n",
"\n",
"As you work with neural networks, you'll see that pattern recurring. You will almost always deal with probabilities, not certainties, and will do a little bit of coding to figure out what the result is based on the probabilities.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1/1 [==============================] - 0s 49ms/step\n",
"[[31.521452]]\n"
]
}
],
"source": [
"# Import modules\n",
"import numpy as np\n",
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"\n",
"# Create some data\n",
"x = np.array([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
"y = np.array([-2.0, 1.0, 4.0, 7.0, 10.0, 13.0], dtype=float)\n",
"\n",
"# Model definition\n",
"model = tf.keras.Sequential([\n",
" keras.layers.Dense(units=1, input_shape=[1])])\n",
"\n",
"# Compile model\n",
"model.compile(optimizer='sgd', \n",
" loss='mean_squared_error')\n",
"\n",
"# Model fitting\n",
"model.fit(x, y, epochs=50, verbose=0)\n",
"\n",
"# Model prediction\n",
"print(model.predict([10.0]))"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "tf",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
},
"rise": {
"enable_chalkboard": true
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
},
"vscode": {
"interpreter": {
"hash": "09dae4df73b471858248f7b697ec2bb8ee523eef7d0b410e464ff3946da6c31d"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}