{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# t-Test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A t-test can be used to determine if there is a systematic difference between the means of two groups. \n", "\n", "The simplest form of analysis that can be done is one with only one independent variable (outcome variable) that is manipulated in only two ways (group variable) and only one outcome is measured. The manipulation of the independent variable typically involves having an experimental condition and a control: \n", "\n", "- E.g., does the introduction of a new UX feature on a website lead to a higher revenue than the old one? We then could measure revenue for both conditions and compare them. This situation can be analysed with a t-test.\n", "\n", "Furthermore, we can use a t-test to draw conclusions from model estimates. In particular, the test can be used to decide whether there is any significant relationship between a dependent variable `y` and a feature `x` by testing the null hypothesis that the regression coefficient `b` equals 0.\n", "\n", "Among the most frequently used t-tests are:\n", "\n", "- **Dependent t-test**: Compares two means based on related data.\n", "- **Independent t-test**: Compares two means based on independent data\n", "- **Significance testing**: Testing the significance of the coefficient b in a regression." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Independent t-test example\n", "\n", "Let's conduct an independent two-sample t-test for the following hypothetical research question for an e commerce online shop: \n", "\n", "> Does the introduction of a new UX feature increases the time on site? \n", "\n", "We proceed as follows:\n", "\n", "- Imagine that we split a random sample of 1000 website members into two groups using random assignment. Random assignment ensures that, on average, everything else is held constant between the two groups.\n", "\n", "- Group “A” (500 members) receives the current product experience\n", "\n", "- Group “B” (500 members) receives some change that we think is an improvement to the experience. \n", "\n", "- We then compare metrics (time on site) between the two groups. \n", "\n", "\n", "Let's formulate a **nullhypothesis ($H_0)$:**\n", "\n", "- $H_0$ \"There is no difference in the time spent on site between the two groups.\"\n", "\n", "## Generate data\n", "\n", "We generate our own data for this example." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from scipy import stats\n", "\n", "# number of participants\n", "n_A = 500 # see old feature\n", "n_B = 500 # see new feature\n", "\n", "# generate random data with scipy\n", "np.random.seed(123)\n", "\n", "# mean =loc and standard deviation = scale\n", "ux_new = stats.norm.rvs(loc=4.0, scale=1.2, size = n_B)\n", "ux_old = stats.norm.rvs(loc=3.8, scale=1.5, size = n_A)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ux_newux_old
02.6974.928
15.1973.904
24.3403.402
32.1925.194
43.3065.691
\n", "
" ], "text/plain": [ " ux_new ux_old\n", "0 2.697 4.928\n", "1 5.197 3.904\n", "2 4.340 3.402\n", "3 2.192 5.194\n", "4 3.306 5.691" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "# create pandas DataFrame\n", "df = pd.DataFrame({'ux_new': ux_new, 'ux_old': ux_old})\n", "df.head(5)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD3CAYAAAAe5+9lAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABAfElEQVR4nO3dd3yT1f7A8U9Gk7bporQs2etaZAiUJbJUwI0iyFD0Ak5QRFERZfnTy7hecSAIIo7bUkAQ9xbUCngRkCUWkCGrjNIW2qZtmjbP749TKtAVSkaTfN+vV1+1ec6T5xtMvj09zznfo9M0TUMIIYTf03s7ACGEEJ4hCV8IIQKEJHwhhAgQkvCFECJASMIXQogAYfR2AOXZunUrZrPZ22E4zWaz+VS8rhBor1ler3/zl9drs9m48soryzxWbRO+2WwmLi7O22E4LSUlxafidYVAe83yev2bv7zelJSUco/JkI4QQgQISfhCCBEgJOELIUSAkIQvhBABQhK+EEIECEn4QggRICThCyFEgJCEL4QQAUISvhBCBAhJ+MKvFRZ651whqqNqW1pBCFcwGmH69KqdW9XzhKiupIcvhBABQhK+EEIECEn4QggRICThCyFEgHDLTdtVq1bx0UcfAaoYf0pKCklJScyYMQOdTkeLFi2YNm0aer38vhFCCE9xS8YdOHAgCQkJJCQkcMUVVzB58mTmzZvH+PHjSUpKQtM0Vq9e7Y5LCyGEKIdbp2Xu2LGDvXv3Mm3aNN544w06d+4MQM+ePVm3bh19+/Yt99yzfxn4ivz8fJ+K1xV84TXHxcWRlpZWxbNjz3t9vvB6XUler/9xa8JfuHAhY8eOBUDTNHQ6HQAWi4Xs7OwKz5UtDqs/X3nNsbGxVT733NfnK6/XVeT1+iavbHGYlZXF/v376dq1q7rQOeP1VquViIgId11aCCFEGdyW8Ddu3MhVV11V8nOrVq3YsGEDAMnJycTHx7vr0kK4j83mnrZCeIDbhnQOHDhA/fr1S36eOHEiU6ZMYc6cOTRt2pT+/fu769JCuI/ZDM52VjZtcm8sQlwktyX8++6777yfmzRpQmJiorsuJ4QQohIyEV4IIQKEJHwhhAgQkvCFECJASMIXAauoCLKyZKMTEThkAxQRcHbvhl9/hf371c96PVx2GVx9NbRoAcXrA4XwO5LwRcDIz4cvvoDff4fISOjeHaKi4PRp+OMPWLoUWraEgQPV7Esh/I0kfBEQ8vIgMRGOHYPevaFHD9WzP6tPH9Xr//57WLwYRozwWqhCuI2M4Qu/Z7erZH/8OAwdCr16nZ/sAQwG6NYN7r5b9fiXLIEzZ7wSrhBuIwlf+DVNU8M4qakweLAasqlIkyZw552QlgZDhoDD8fcxZwtradolBCyEG8mQjvBr774L27apXv3llzt3TvPmcP318OWX6vvZklBpaWnMmxdLamrF59erd2kxC+EukvCF3zp6FB5/HBo3hp49L+7c+Hh14/aTT6BpU6hTxy0hCuFRkvCFX9I0GDsWCgrglltKj9mfNeERG+ExZU/JyfzrDCk/29nyW03W/6LDYKh6XX0hqgNJ+MIvffON6p3Png25ueW3C48xk1rv7+qXhkIbFutJQvIzqeGwkwKcPhXJ7uBG1AjLo27mHvcHL4SbyE1b4XcKC+HJJ6FZMxg/3smTNI2wnOPUStuJJTeNApMFLruM0+H1Sdb3pm7hIeqe/hMeeAA0R+XPJ0Q1JD184Xfefx927oQVK8BkcuIEzUGNzAOE2E6TFxzFmYgGOAwmQupArgOCzRYanjrIh8Yh9Fu0iJigUNKjW6Dp5eMjfIv08IVfKSiA//s/6NwZ7rjDuROiM/YSYjvNmYj6ZEY1xWE4/7dE66Dd9A/5mVsKPyZt8acE2fOISd+DzlHknhchhJtIwhd+JSEBDh2CadOcqImjaTBqFMEF2WRGNsJqqV3uSU+HvYkDPf+35RYyoptjLMyjRuY+mXQvfIokfOE3Cgthxgzo2BFuuKHy9t3+9wosWUJWWD3yQmMqbFvfeJyBwZ/z9ttwxNiYM5GNCC7IJjznmIuiF8L9JOELv/HRR6oC5uTJlffu6x77jb7fPQ23305OmHOT7B8OeZeCAlhkHU5uaAy5ITUJyzlGUEGOC6IXwv0k4Qu/8frrapHULbdU3E5fZOfWT0djtcSqSmlO1kNuYjzE4MHwfu4gTjvCORPRgCKDiagzh2RoR/gEtyX8hQsXMmTIEAYOHMiKFSs4ePAgw4YNY/jw4UybNg2HQ6a2Cdf57TdYuxYeeUQVQqvIVb+8TN3jW/nyxnlQo8ZFXefZZyFHC+Nd651oegNZEQ0IKszDYj1xCdEL4RluSfgbNmxgy5YtLF26lISEBI4fP87MmTMZP348SUlJaJrG6tWr3XFpEaBefx0sFhg5suJ20Rl76f3jdP6IG0hK3MCLvk7bttDXnMzb1mFYHSHkB0eRb44kPOcYhqKCKkYvhGe4JeGvXbuWli1bMnbsWB566CF69+7Nzp076dy5MwA9e/Zk/fr17ri0CEAnT6rNS+69V21oUpHrvn+GIkMQX97wRpWv92jYO5zWoliWdysAZyIagKYRnn20ys8phCe4ZeVIZmYmqampLFiwgCNHjvDwww+jaRq64rFSi8VCdnZ2hc9hs9lISUlxR3hukZ+f71PxukJ1ec1vvhlDQUEsN964j5SU83vZcXFxpKWlAdD4xGZapXzI552e5kC+EfLTgFjs5W5qazz/WPE4fVv9Vjoat/KOdQh3m5LQ6wxkh8QQkZfG6eBYIIy8/HxCoFr8+1RVdfn/6ymB8HrdkvCjoqJo2rQpJpOJpk2bYjabOX78eMlxq9VKREREhc9hNpudrj9eHaSkpPhUvK5QHV5zQQGsXAn9+8NNNzUrs01sbCxoGoO+nE2OpRbbr5tCrCms5HiQsfyPwbnHzib/IKOR0WHLGXN6JslFvegbvJa8iHqE56dTI+8kEEZIcDDgfA396qg6/P/1JH95vRX90nLLkE7Hjh35+eef0TSNEydOkJeXR7du3diwYQMAycnJxMfHV/IsQvytvE74Z5+pbQsffbT4AZutVJvp02F6t29o8tePhM2awrMzwtRj06sez43Ba6ijP8E71qEAOPRGrJZahORnVlytTQgvcksPv0+fPmzcuJFBgwahaRpTp06lfv36TJkyhTlz5tC0aVP69+/vjksLP2U0lp2gk5IgPBw2bICNG2H69POrX9arB6lHNWLSd6E3mDj5r3dgxnt/H0/dVKV4gnRF3GtZyezsseyxN6Fl0AFyLLWxWNPQp6aqXVSEqGbcVv3p6aefLvVYYmKiuy4nAlBWFuzdC927l1/vHsBUkIPJnsvpiIagc90ftXeHruLV7Pt4J3cIsyJnoRX38sPPHIP8fJddRwhXkYVXwmdt26buo7ZvX3G7MOtxivRGckNruvT60foz3B7yNSvzbuK0IxwAa2isWsh1Qubli+pHEr7wSZoGW7dCo0YQHV1Bw9xcgm1ZxYXRXP92H2VZRp4WwrLcAQA4DEFQsyakp8M5ExWEqA4k4QufdPgwZGTAlVdW0vD4cRw6vep5u8EVQX/S1bSZ93IHU6QVf5xq11a/kd6o+lx/IdxBEr7wSVu2qM1NWrUqv42+qAAyM7GGxqLpK6m3cAlGhn7A4aLLWG27Wj0QHKxWgM2fLzN2RLUiCV/4HJtN7Wh1xRUV72hlsaoFV7lu6t2fdX3wj9TVH+cd65C/H6xVCzIz1RJgIaoJSfjC56SkgN1eyc1azUFo3imIiqLIaHZrPEZdEfdYPuTngi78aW+sHgwLgzZt1LCOVNIU1YQkfOFzduxQRS7r1y+/TUheBgZHoeppe8DwkI8xUcC7uXeqB3Q6Vbpz61aQulGimpCEL3xKTg4cOACtW1dQxl7TsFjTsBuDVU/bA2IMmdwa8i0r8m7mTJFFPXjXXRAZCfPmeSQGISojCV/4lN9/VyMkbdqU3ybInoupMBdraC2nNzdxhVGhy8nVQnnvVPEOLGfrNa9cKVM0RbUgCV/4lB07oE4diK3gPmxo3ikcOj15IRVN0He9dqYUOgRt5420O3Foxb9oxoxRNxwWLfJoLEKURRK+8Bnp6ZCaWnHvHquVkLwM8oNruHUqZnlGWZaz19aQb7O6qgdatFClPBcsUIlfCC+ShC98xo4d6nvr1hU0WrECveYgNzTGIzFd6Kbg1dQ2nmLuyXOmaD7yiPpN9fHHXolJiLMk4QufoGkq4TduDBVupbB4MXaDmYIgi6dCO49JV8iDsav4Kusq9u4tfvCGG1QNiDff9EpMQpwlCV/4hE2bVCmFioZzap7aDWvXkhca49GbtRd6MGYVBhx/T84xGOChh+CHH9QiAiG8RBK+8AlLlqi8WVEphfZb3gGDgdwQ11bFvFj1TKcYVGM177yjppECMGqUWhYsvXzhRZLwRbVXWAjLlqn7n8U7B5aiL7Jz5bb34eabVcVKL3u01nKysuC//y1+oFYtGDQI3n8frFavxiYClyR8Ue398IMqL1/RcE6LP78kzHoCRo/2XGAV6GbZTqdO8PLL52zPOGaM2rUlKcmrsYnAJQlfVHtLlqgFqy1blt+mw5bFZIfVVTdIqwGdDiZNgv37YcWK4gevugratlVVNKW+jvACSfiiWsvLg1Wr4I471L62ZQnPTqXFn1+w9cp/lt/ICwYMgLg4mDWrOL/rdKqXv3Ur/O9/3g5PBCBJ+KJa++wzyM6G4cPLb9Nu6/voNQdb2o/yXGBO0Oth4kTYvh2+/LL4wbvuUruuy81b4QVuS/i33XYbI0aMYMSIEUyaNImDBw8ybNgwhg8fzrRp03A4HO66tPAjSUlQty707l1OA02j/dZ3+KtRLzKim3syNKcMHw4NG8KMGcW9/LAwuOceWL4cTp3ydngiwLgl4dtsNgASEhJISEhg5syZzJw5k/Hjx5OUlISmaaxevdodlxZ+JCND9YyHDVNTMsvS6GAyNTP28lv76nGz9kJBQfDkk6pC8s8/Fz/48MNQUADvvOPV2ETgcUvC37VrF3l5eYwaNYp77rmHrVu3snPnTjp37gxAz549WS81wkUlVq5U5WcqGs7psGUx+eYIUlrd4bnALtLo0arY2/PPFz9wxRXQq5eqryN/6QoPcssdruDgYEaPHs3gwYP566+/uP/++9E0DV3x6keLxUJ2dnaFz2Gz2UjxoVWJ+fn5PhWvK7j7NS9c2IimTQ2EhOwH4khLSzvveLAti7g/VrLhH3eSetoKWIFY7CXzIAGMF/xcWvnHLzi3eGZNZc8HRvLy8wmBkn+fUaOimT27Nu+8c5Bu3XIJv/VW6k+YwKFFi7D27FnJ83lHoL2nA+H1uiXhN2nShEaNGqHT6WjSpAlRUVHs3Lmz5LjVaiWiwoIoYDabiYuLc0d4bpGSkuJT8bqCO1/zgQPw22/wr39Bq1bqGrEX1ESO3/QhpsI8Uq565LxjQRfM1Lnw5wtVdPzcY2cTfWXPBxBSvELs7L/P9OlqeunChY0YORJ0zZrBv/9Nw88/hwcfrPT5vCHQ3tP+8nor+qXlliGdlStXMmvWLABOnDhBTk4O3bt3Z8OGDQAkJycTHx/vjksLP7Fkifp+113lt+nw29scr92O1LodPRPUJQgOhmnT4Ndf4cMPUWUW7r8fvvgC/vrL2+GJAOGWhD9o0CCys7MZNmwYjz/+ODNmzOC5555j7ty5DBkyBLvdTv/+/d1xaeEHNA0SE6FnT1Vksiy1j2+j3rHN6matFwulXYyRI9W6q6eegvx84IEHVOxvveXt0ESAcMuQjslk4uWXXy71eGJiojsuJ/zMpk2we7ea3VKeDlsWU2gws6NtBX8CVDMGA7zyClx7LcyZA88+2wBuuQXeflt1/81mb4co/JwsvBLVTmKiyn2DBpV93FiYT9vtiaTE3e7xbQwv1TXXwMCB8OKLquwCY8ZAWlrxOI8Q7iUJX1QrdjssXao6vlFRZbeJS1lFSH4mv7W/z6Oxucprr6kKEA8/DNq110Hz5rLyVniEJHxRrXz3nerw3n13+W06/PY2GTWa8leTPp4LzIXq11crb7/9Ft55T68y/9q1qgaDEG4kCV9UKwkJEB1dftHL6Iy9NPnrB7a0H42m892375gx0KcPPPYYHOj1TzWNR3r5ws189xMj/E5Wltrne8gQNWuxLB1+exuHzqAqY/owvR7ee08N7QwdE03R4KHqt11WlrdDE35MEr6oNlatUtMVR4wo+7i+yM6VW99jT8ubyA6v59ng3KBhQ1i8WM3N/0/uGLUTVskWWUK4XvUpHi4CXmIiNGsGXbue86DNVjJdcWqHz8F6gsv/cz/Tb/ZOjBflnNjLc8cdMGECPPNyJ0Y2jKfW/PkwdqzPrC0QvkUSvqgWDh6ENWtg6tQLcp3ZDMWrsvN//5MgfRAn7p8GuullPk+91E3uD9ZZ58RekVmagV2R/+Hpw4/ynnav2tPxmms8EKAINDKkI6qFd99V30eOLKdBQQFmWxa5oTX9rvdr1BWxvMkk9sffSRoxnHr+DW+HJPyUJHzhdUVFqjR8377ll1Lg1Cl0QG5IjCdD8xiLIZ9VH+lYVeM+aiR/wq5vD1V+UvG+E0I4S4Z0hNd99x0cPgxlVONQNA3S08k3hVNk9N/yAzGXmbkx9lfIhK+vf4WiuKNcEbK//BM2VaPhK+ETpIcvvO7ttyEmBm69tZwGWVlQUEBuqH/27s/VIPwMeeG1GaElcMPuV0jJa+ztkIQfkYQvPKasfUNOnoRPPlHbvJY7oSUtDYxG8oOj3BmeyxTvk1IlDg3C6oRRk3Ru1T7hmj0L2J1f3jjX3yrdk0UIZEhHeJDRqDYCOdf69SpZ5eeXPgYwfdQhtDNn0NWpAz6yslang9RUqIf67qx6gF4HqdnhxBrMzNBNZpljKL1T3mRlzQdoYjxSqv3Zf7Oy/u2EuJBvfIKEX9I0tatVgwZqz9cyna0VH+P/wzkldDqsllpEFGbyVfgQCghicPoCDhZe5u3IhI+ThC+85sABSE+HDh3KPm4oKoBFi7CZIwOuVnxeSE0cOj1XFGxhWfQY8rRgBmcs4HBhXW+HJnyYJHzhNb/+CqGh0Lp12cfjUlbByZNYQ8vr/vsvTW8gL6QmIXkZtNH/wfKaY8lxWBicsYCjRbW9HZ7wUU4l/Pnz55/3c1m7WQlxMU6fhj17VO++vD3BO22cB02bYjNXvOG9v7KGxqJDIzQvndZBu1kaPZYzjgjuTF9AhiPS2+EJH1ThTdsVK1awcuVK9u3bR3JyMgBFRUUUFhYyYcIEjwQo/NPGjep7eZUHap/YTqNDa+Gll2DOMs8FVo0UBoVgM4UTmptGjqU27Uwp/Df6Me5MX8BDmTP5we7tCIWvqTDhDxgwgG7durFw4UIeeughAPR6PTVr1vRIcMI/2e3qZm1cHESW01GN3/gmhQYzxpEjAzbhg+rlR5/eT7DtDPnBUXQybWd25L94/MzzTJoEYWHejlD4kgqHdEwmE/Xr1+f5558nPT2d1NRUjhw5wrZt2zwVn/BD27eraZidO5d93GzLot32BH5vPRQCvHORHxxFkT4Ii/VkyWN3hn7BPaErePll+PNPLwYnfI5T8/DHjRtHeno6deuqGQI6nY5OnTpVeE56ejoDBw7knXfewWg08swzz6DT6WjRogXTpk1Dr5f7xYFI09TN2tq1VT34srTdloDJbuXXTmO50qPRVUM6HVZLLBHZqRgL8yk0BgMwLeIVtjYfzMcfq92zhHCGUwn/1KlTLFvm/J/VdrudqVOnEhys3pwzZ85k/PjxdOnShalTp7J69Wr69u1btYiFT9u/X62uvfXWcopeahqdNs3naL14Ui+ruFMRKHJDYgjPPkao9SRZkeq3ZLCugKVLoW1b+PJLdatDiMo4lfCbNGnCiRMnqF3buelgs2fPZujQobxVvGhm586ddC7++71nz56sW7eu0oRvs9lISUlx6nrVQX5+vk/F6woX+5rj4uJYs6YAi8VA3boZpKWVbtPi6Fpqpf1BQp/XSEtLA2KxFxYCxuLvlTu/XeXnlX/8gnOLayZUHsff5zkbc8Xx6LCaowjNSycjpDaa3gBAq1YQH29lwwYLq1ZBXJxr33+B9p4OhNfrVMLfvHkzffr0ITo6uuSxtWvXltl21apVREdH06NHj5KEr2kauuLunMViITs7u9Jrms1m4uLinAmvWkhJSfGpeF3hYl/zL79AaqqJfv2gTp2y59b3+2EJuSHRHOx2P7FBIQAEFc/bDCpv/uYFLmxX2XkVHT/32NlE7EwcFxtzZfHkhdUmLD2TCPsZci21Sh7v29fCX3+pzdB37YrDYrnoy5Ur0N7T/vJ6K/ql5dS78dtvv3X6Yh9++CE6nY5ffvmFlJQUJk6cSEZGRslxq9VKRERgzqsOdDNnQkgIdOxY9vGIrCNcvutjfun6OIXFyV4o9qBQCoJCseSmkRsaWzIeZjDAjTeqDWRmzIB//cvLgYpqzamEP2nSpFKPzZw5s8y2S5YsKfnvESNGMH36dF566SU2bNhAly5dSE5Oput5m5aKQLBjB3z2GfTuDZOesBEeU0aphKlvAQ66Jz5M96aejrCa0+mwhsZS48xBTAU5FJjDSw41bKiqjf7nPzB6NDSVfztRDqcS/o033giooZk//viDkydPVnLG+SZOnMiUKVOYM2cOTZs2pX///hcfqfBps2apOeOdO0N4jJnUehesuNIc1D65A7spgoyr7yx5uFrtUetleSHRRGQdwZJ78ryED+qvpxUrYPJkSEryUoCi2nMq4ffo0aPkv3v27MmoUaOcevKEhISS/05MTLzI0IS/+PNPWLYMHn9cDemUJST/NAZHIacDsG6O03R6ckNjCLOeUIXlzlGvnvr3nTEDJkwof9hMBDanJsOvXbu25Oujjz7i1KlT7o5L+JFp0yA4GJ58svw2odaTFBrMAVs3x1m5xb8QQ62lpzg9/bRapzZliqejEr7CqR7+F198UfLfJpOJGTNmuC0g4V+2bYOlS2HSJKhTp+w2RnsuZruVM+H1y5mcL84qMpqxmSMJzTsFBQWAqeRYZCQ88QQ895wqXVFe2WkRuJzq4c+cOZORI0fSo0cPhg4dSuvy6tkKcYHJkyEqCp56qvw2FmsaDnTkhgZ2GQVnWUNjMTgK4cMPSx0bO1YlfpmtI8riVMJPSEhgypQpbNmyhSlTprB48WJ3xyX8wPr18PnnaqihRo2y2+gchYTkpZMXEo2mlx03nWEzR1BoMME5ZcvPrtWKjIRHH4VVq2DnTueeT/bDDRxOfcI+//xzlixZgtFoxG63M3ToUEaPHu3u2IQP0zR49llVM2fcuPLbheamo0fDes5iIlGJ4imakWvXUrv1dk7UbnvefsE2GwQFwbBhMHBg5U8n++EGDqd6+JqmYTy7cjAoiKCgILcGJXzf11/DTz+p8eRyV39qGpbcNAqCLBQGhXo0Pl+XGxoDwcHEb3yz1LHQULXPwO+/wzlrHoVwLuF37NiRcePG8f777zNu3Djat2/v7riED7Pb1c3DFi3gwQfLb2cuyMJYZJPefRVoeiMMHUq77QmYbVmljnfrBno9lFMBRQSoShP+8uXLeeKJJxg4cCDZ2dl07tyZiRMneiI24aMWLIBdu+Dll8FkKr9dqDWNIr2RvOAoj8XmV8aMwWS30nZbQqlD4eHQrp1a4Zyb64XYRLVUYcKfO3cu69ato7CwkN69e3Pbbbfxv//9j3nz5nkqPuFjMjLUvPvrroObby6/nb7ITrDtDLkhNUEneyNUSadOHK0XT6dN80sqeZ6rSxd1Q3bzZi/EJqqlCj9pycnJvPbaa4QUL4+sX78+r7zyCmvWrPFIcKJ6Ojuro6zKgs8/D2fOwJw5FU+pD807hY7isWhRZRvjx1Ar7Q91w+QCtWqpujobN0JRkReCE9VOhbN0QkNDS8oanxUUFITFlTVYhc85OyMkLS2N2Ni/SyGkpanhnPbt1RTxC6eJl8wGcTgIzT2FzRRGUfEOTqJqdrYeQv9vJxAyfz606l3qeJcuauHbH39Amzaej09ULxX28IODgzl8+PB5jx0+fLjULwEhNE3tvGQyQZ8+lTRevRpjUUFJmQBRdfagULa0HwUffUR4dmqp4y1aQHQ0bNjgheBEtVNhD//JJ59kzJgxdOvWjQYNGpCamsratWuZPXu2p+ITPuL33+Gvv1Rt9kr/AFy0CIfOIDdrXWRT/ENc9cvLdPjtbX7qNfW8Yzqd6uV/9RUcOQL163spSFEtVNjDb9GiBUlJSbRq1Yq8vDyuuOIKli5dSqtWrTwVn/ABNht8+62q2FhZlcZQaxp8/LEqoyA3a10iI7o59O9Px80L0RfZSx2/8kowm6WXL5xYaRseHs5tt93mgVCEr/rhB8jJgaFD1dzvily57X2w28mNlJu1LjVmDBHfDOAfez4jJe785bUmk7qv8uuv0K+fmrIpApN0scQlOX5cJZKOHeGyyypprGl0+G0RXHWVbGHoajfdxOnIhnTaWPaU6fh4cDhUFU0RuCThiyo7e6M2JASuvdaJE379lZj0PeDkBjriIhgMbO74IE0PrCEmrfQm1jVrQvPmsGmTTNEMZJLwRZXt2mXm8GG1yKq8nazOk5CA3RgMgwa5PbZA9FuH+yjSBxG/aUGZxzt1UkNvu3Z5ODBRbUjCF1WSlwe//BJGgwbqpmBl9EV2WLaM3f+4VdXwFS5ntdRi5xWDuXLbewQVWEsdb95c7U2wcaPnYxPVgyR8USWrV4PNpuPGG53bpKr53q8hPZ3tbe52f3ABbGP8GIJtWbTZUXonc71e9fIPHoQTJ7wQnPA6tyT8oqIiJk2axNChQ7nrrrs4dOgQBw8eZNiwYQwfPpxp06bhcDjccWnhARs3qvosbdrklbtt4YXa7kiEmjXZ2/x69wYX4A43uIrjtduWW1+nfXu1Ulp6+YHJLQn/hx9+AGDZsmWMGzeOmTNnMnPmTMaPH09SUhKaprF69Wp3XFq4WVERPPwwhIVB587OlWE055/hH7s/haFDcRhkLwW30unYGD+Guse30uDIL6UOh4RA69awfTvk53shPuFVbkn41113HS+88AIAqampxMTEsHPnTjp37gxAz549Wb9+vTsuLdzsrbdU775fPzCZSvcgy9Iq5UOCCvPhbhnO8YQdbe8i3xxJlw2vlXm8c2e1Z8HWrZ6NS3if2zYRNRqNTJw4ke+++47XX3+dH374oaQGj8ViITs7u8LzbTYbKSmlp5dVV/n5+T4Vb1Wkpxt45plm9OljoHbtNAoLC0lLS6v0vMs3v8vJyCbU6tKFtIQ0IBb7RWykqtoanT7n/HaVn1f+8QvOLR4iqTyOv8+7mNdZeTwXsNmYPt1c8uPfWxWGQeEDtH75ZVr/8y9o3BiAMyfzePb5HIxGqFMniv/9T0fTpplAbJnv3UB4T58rEF6vW3eNnj17Nk8++SR33nknNput5HGr1UpERESF55rN5jLL71ZXKSkpPhVvVYwcqWbnzJsHy5fHlqqWWZaIM4dpeXQdP/aaRi2drqR9kNH5t97Zts6ec2G7ys6r6Pi5x84mYmfiuNiYnY3nPGYzqfXiAVXWIvWc2mn6ogJqOxxYW3chK6KBapO6idhYNX/2qqvURudZWer/R1nv3UB4T5/LX15vRb+03DKk8/HHH7Nw4UIAQkJC0Ol0tG7dmg3FxTySk5OJj493x6WFm6xdC++9BxMmwMV8JtrsSEKHxva2MpzjSQ6DibzgaEJzT6FzlF5p1aqVKnInN28Di1sSfr9+/fjjjz+46667GD16NM8++yxTp05l7ty5DBkyBLvdTv/+/d1xaeEGhYUwZgw0bAiTJ1/EiZpGu+0JHK7fjczoZm6LT5TNGlYbvab2HriQwQAdOsCePXDggBeCE17hliGd0NBQXnut9A2jxMREd1xOuNncuWpv1FWrnCh9fI46J7ZRK20nn984333BiXLZg0KxmcKwWE+UuVF8fLz6y+3NN+Hf//ZCgMLjZOGVqFBqqtqj9oYb4GKLprbdnkiRPoidV9zplthE5XIstTE67ATnZ5Y6FhGhhucWL1b3ZoT/k4QvKjRhAhQUqF7+xWx0pnMU0WZHEn+2uJG80JruC1BUyGaOxG4wE2Y9UeZCrE6d1Mbzy5Z5ITjhcZLwRblWr1aJYNIkaHaRQ/BNDqwhPOeY3Kz1Np0Oq6U2JnuuGr+5QKNGcMUV8MYbZf4+EH5GEr4ok80GY8eqRD9x4sWf3257AvnmSPa0vNn1wYmLkhdakyKdAV5+udQxnQ4eeUTVyZcdsfyfJHxRpjlzYPduNZQTHHxx5wYVWIlLWcXOVoMpNF7kycLlNJ2eXEst+OQTYtP+KHX87rvVeP4bb3ghOOFRkvBFKQcPwgsvwO23q5u1F+vyXR9jsltlOKcasVpqQWgo3dfNLnUsLAz++U/44AOpounv3LrSVvim8ePVn/qvvupc+wmP2AiP+XuJPzckQsOGjHynR6kuxd/L/4UnOfRGePBB2r76Oj/2ms7pGk3OOz5mDLz+Orz9Njz3nJeCFG4nPXxxni++gI8/hqlT1UIrZ4THqCX+qfXiOV67HdrXX5OdXkBq/c4lj6fWi4f4+JJSAMILJkzAoTfQff1LpQ794x/Qt6+ak19Q4IXYhEdIwhcl8vLg0UfV3OzHH6/ac4TkZ6AD8kKiXRqbcIHLLmNru3/Sfss7hGUfK3X4iSfg6FFIKr13ivATkvBFiVmz1DL7efPAZKrac4TkZVBgDKUwyJlNboWnrev+NHqHnW6/zCl1rH9/aNcOXnoJZH8i/yQJXwCqpsrs2TBsGPTpU7XnMNrzMNlzpXdfjWVGN+P31kPptOlNQvIyzjum08HTT8Mff6ihPeF/JOELNE3tYhUcXOZUbaeF5GWgIcM51d3aqydhslvpsuH1UsfuvFMtxppdejKP8AMyS0eQmAhr1qgbdnXrVvFJNI2Q/Axs5gjZxrC6KtkwpTWkDqB38uv0/mwCEF7SxGhU5TTGjYOTRxtfVClsUf1Jwg9w6enqZl23bvDAA1V/HlNBDsaiArLD67kuOOFa52yYElRgJTYzkzOXxRHZ8vyd6EcVBfO84XNefj2KXtd5I1DhLjKkE+CeegpOn4aFC0F/Ce+GkLwMHDo9+eYoV4Um3MhusmAzhauiahfcobUY8hlXaxmffSb73vobSfgB7Mcf4d134cknoU2bS3ii/HxC8jPJD45C0xtcFZ5ws+ywuhgchXCq9AYp42otIzISnn/eC4EJt5GEH6BsNnjoIWjSBKZMucQn++wz9FoReSFSBtmXFJjCKAiyqHoKF/Tyo4w5PPGEWoS3ZYt34hOuJwk/QM2apYqjzZ8PoaGX+GTvv0+RPgibKbzytqL60OnIDqurltZmZJQ6/NhjEBVVfjmM4j3dhQ+Rm7YBaNs2ePFFGD4crr/+0p7LknMCvv6a3JDYi9shRVQLNnMEhIZSeOQYJ201z/t/WC8SrrwSPv1U3dCvd8H9eKmL5Hukhx9g7HZVGbFmTVUs61K13bEEiopkVytfpdNB3boYiwpKLcQC6NJFrc/46ScvxCZczuU9fLvdzrPPPsvRo0cpKCjg4Ycfpnnz5jzzzDPodDpatGjBtGnT0F/KlBBRZTNmqJkXH32kkv6larftfejcmcLDRZf+ZMI7IiOxG0MIzzmmFs2d08sPDlZTdn/4AQ4fhgYNvBinuGQuz7qffvopUVFRJCUlsWjRIl544QVmzpzJ+PHjSUpKQtM0Vq9e7erLCids3fr3UM7FbkhelvqndlDnxHa4995LfzLhPcVj+cYiGyFlbHbetStYLPD997INoq9zecK//vrreeyxx0p+NhgM7Ny5k86dOwPQs2dP1q9f7+rLikoUFLh2KAegy64PKDSYYOhQ1zyh8Jr84CjsxmDCco6VyuomE/TuDYcOqZpLwne5fEjHYrEAkJOTw7hx4xg/fjyzZ89GV/xnosViITs7u9LnsdlspKSkuDo8t8nPz/dKvI0bNyckpPJSBjNmqJu1H3+skr7drhEUVPWbrOnHU4nfs5IdjfrRPjoau1NTNowl7ZxrzzltjU6fc367ys8r//gF5xYnwsrjqNrrrDyeito69+9Tdhsj9qIiTofUIjb7EEG56eQWL6BLS0sDoH59iIqqwTffQFRUZvEivVif+oxWxlufYU9yyyydY8eOMXbsWIYPH84tt9zCSy/9veGC1WolIiKi0ucwm83E+VAhj5SUFK/FW9lsiWPH1E5GbdqoOdVbtsD06boqz7KYPh26ndlMRH46KZ0foD0QZHTurXS2nbPtq3LOhe0qO6+i4+ceO5ssnYmjKq/TmXgqansxcZX1uN0Sgz33BFG5J7EX34SPjY0tadO3L6xYAampsbRvrx7zpc9oZbz5GXalin5puXxI59SpU4waNYqnnnqKQYMGAdCqVSs2bNgAQHJyMvHxsuuRpxQUwMqVagy2KvvTlqfjb4vICollb/NLnNcpqg+djpywugQV5hFsO1PqcFwcXHaZuoFrt3shPnHJXJ7wFyxYQFZWFvPnz2fEiBGMGDGC8ePHM3fuXIYMGYLdbqd///6uvqwox1dfqTU1AwdCiKv2JDlyhBZ/fsEvccOlMqafyQuJptBgVjtiXTCWr9OpXn52NshtON/k8iGdyZMnM3ny5FKPJyYmuvpSohI7d6qZOVdfDY0bu/CJFy9Gp2msa3W3LOTwNzod2WF1qHHmIHz5JXDTeYcbNYJWrWDdOjhyRI3tC98hn1c/dfo0fPaZ+hO8d2/XPa/eUQhvv82+Zv1Ij2jkuicW1UZeSE01++qFF8qch9m3ryq988wzXghOXBJJ+H7I4YBVq9Rn9Y47wODCApbN//wKjhxhU8cHXfekonrR6cgJqwMbNtBs/3elDkdFwVVXwZIl8L//eT48UXWS8P3QTz+pVZE33ww1arj2ueM3L4Q6ddjT8mbXPrGoVnJDakL9+vT66f/K7OVffbXaHe2xx2TDc18iCd/P7NsHycnQrt0l1rgvQ+SZQzTf+xWMHi03a/2dTg/PPEPDw+to/NePpQ6bTKri6q+/qp6+8A1SLdOPnD4NH34ItWrBjTdW/XkmPGIjPMZc+sDUtwEN7r+f6Y0AYku3Ef5j9Giyn/kXvZJf4K8mfUodvvtumDdPjeXffjuEhXkhRnFRJOH7icJCtSjG4YA771Q9sKoKj/l779MSmoPaJ3/Hbgon+I47SE1VC5GCjEbqpW66tOBF9RQczLruT3P9N4/T8NBaDjW8+rzDej289poqrjZrlqrTJKo3GdLxE998A6mpMGCAa6pgXigkLxODw47VUsv1Ty6qrc0dHyDHUoueyS+UebxrV9XTf+kl2LvXw8GJiyYJ3w9s2wabNqmZE25ZGa5pWKwnsBuDsZkqL4sh/Ic9KJT13Z6k+b5vuezIhjLb/PvfYDbD+PGejU1cPEn4Pm77dvj8c7Ug5tpr3XMNU0EOpsI81buXXa0CzqZOD5MbUpPeP00v83jduqq+0hdfqLUfovqShO/DTp1SQzjBwTBoELhrTxmL9SQOnUE2KQ9QBaYw1nafSIu9X9Pw4M9ltnn0UbUC97HHID/fwwEKp0nC91F2u7o5e+yYKkfvrhkShsJ8gm2nsYbGounk7RKoNnYeS3ZYXa5d82yZ8/KDgmDuXDhwQA3xiOpJPsE+asIEVbXwrbdU+QR3Ccs5AejkZm2AsweF8lPPKTQ6tJbm+74ps80116hOyMyZ8Ndfno1POEcSvg9avFj1pp54Au65x33X0RcVEJqXTm5ojCy0EmzpMJrMqCZcu/pZdJqDsvZSefllNbT4+OMVP1cV9oURLiDz8H3M+vXw8MOqgNXs2e69Vpj1BKCRY6nt3gsJn1BkMPFj7+nc/vG9xKWswmgcVOYmOt26qZ3V7r4bmjcv+7mquvmOuDTSw/chR46ouvYNG8KyZVCFzZScpi+yE5p7iryQmhQZy1h1KwLS9jZ3kRYTxzVrJpfbTe/aVa0F+eor6clXN5LwfURenlq+brXCp59CdLR7rxdmPYFOc0jvXpxH0xtYfe0MYtJ3w8KFZbYxGuH669XGO7/84uEARYUk4fsATYP77lOLq5YsUdPf3Co1FYv1pNr9KMhV22QJf7HrHwM40LgPTJ1KSF5GmW2aN4fLL4eff4YzpXdLFF4iCd8H/OtfkJSkvt96q4cuiEZ2WD0PXEz4HJ2Or69/FU6fptePz5fbrH9/1Vn59lvPhSYqJjdtq7kVK2DKFBgxAiZNcv/1ojIPwFtvkRsaI2P3gc5mY/r08t4DbUG7n65vz6Prew+RHduUl984v21UlKqb/+OPsH8/NG3q7oBFZSThV2ObNsG996oaOYsWeaaqwTU/TAajkeywuu6/mKjezGVUTT1HvVg7DgcUtO9KeH7Z4zbdu6vyH198oWaXuXOigaicDOlUU0ePquGbWrXgo49UcSp3a3B4PW13JMGTT+IwXEJ9ZREYgoLIDqtLsC1LZfQyGI1w003qBu7PZVdlEB7ktoS/bds2RowYAcDBgwcZNmwYw4cPZ9q0aThkT7TzXDh1zWpVyT47WxWjquWBRa46zcH1Xz9GVvhlsju1cJrVEovdGAxjx2IqyCmzTdOmave1tWtV/SfhPW75A2vRokV8+umnhISoGR4zZ85k/PjxdOnShalTp7J69Wr69u3rjkv7JKPx74UomqbG7XftUjVyPvxQfZXHVQtY2m5L4LLUTay6PYGBFotrnlT4P52eM5GNiDm4mz4/TOWb/nPKbNavH/z5p6rseu+9Ho5RlHBLwm/YsCFz587l6aefBmDnzp107twZgJ49e7Ju3bpKE77NZiMlJcUd4blFfn5+leONi4sjLS0NgA0bQklJsdC9ew41auRR/HAFYkvOvTh/n2fJz+Dab59kf+2OrKnTl4Go3azKZ1THNa2kXcXtLzjP6fac09bo9Dnnt6v8vPKPX3BucdGwyuOo2uusPJ6K2jr371N2m/LPdeZ9YNcHE/PQQ3RZ+BrJl13Podrty2zdtWswP/4Yztq1WUBEtft8X8pn2Fe4JeH379+fI0eOlPysaRq64juOFouF7OzsSp/DbDYT55bdPNwjJSXlkuKNjY1l+3bYvBnat4drrw1Dp3OuBGZsbNX2lj173m0fP4nFdpolt68mppZaaBVUyd21IKOxZItDZ9qfe97FtK/KORe2c+a1OHPsbPJzJo6qvE5n4qmo7cXEdamPlzo+axY5Sz5lZPITvPXAJgqNwaXa9uwJ+/bBL79EcOoU1e7zfamf4eqiol9aHrlpqz+nULvVaiUiQnZNutChQ2oFbePG6iaXp/YZab73a67c9l/Wdn+GE7Xbeuaiwv9ERvLpLW9TK20n16x+rswmOh3cfDPYbPDkkx6OTwAeSvitWrViwwa1PVpycjLx8eVP9QpEu3er2jhRUTB4MBgMnrmuOf8MN3/+IGkxl5Pcc7JnLir81t4WN/Br/Biu+t8cmhxYU2abWrXUNOP331e1doRneSThT5w4kblz5zJkyBDsdjv9+/f3xGV9wokTcMMNqqTsXXdBaKiHLqxp3PrZ/URkHeWTAe/KIivhEt/1e4lTNVty+0f3EJpb9pScXr1UeZD77oPMTA8HGODclvDr16/PBx98AECTJk1ITExk+fLlzJw5E4OnurDVnNUKt9wCx4/DsGFQo4YHL/7WW1zxxwrWXPMiR+p39eCFhT+zB4Wy8o5lhOaeYuCqu9A5ikq1MRpVD//Eicrr5gvXknVvXlJYqJL85s2qdvjmzZ67dp3jW2H2ePY268e67k977sLCf51XhqE9xM+l+QMPMM3wIkybVmb7SZPMvPiiKvntkRpRQhK+N2ia2vT5s89g3jzVy/dUwg/POsrwpJshJoaPbvuv7FMrXOPCMgyaRlRINCHTp5PxWgK24KjzmtdL3cSUKeoz8MAD0KUL1JZK3G4nn3YveO45WLAAJk6EMWM8d11TQQ7Dl96C2XYGvvgCa5h8woSb6HSciWyEPSiUGqcPYLTnlmpiMkFiImRlqd2xikqP/ggXk4TvYbNnq02eH3xQffeUIHsuQ5fdRu0T21gx6ANoK1MwhXtpOj0ZNZqh6QxEZ+5DX2Qv1aZ1a3jjDfj+e5gxwwtBBhhJ+B705puqTM3w4Woox1Nz7YMKrAxPupkmB9bwyYB32dviBs9cWAQ8h8FERnRz9I5Camb8ic5ReuXuyJGq/Pf06fDDD56PMZBIwveQN95Qwze33ALvvee5ufYW60lGJPaj0cGfWHV7Atva3eOZCwtRzB4USmaNZhgL86mZsbfUzB2dDubPh5Yt1USGQ4e8FGgAkITvAf/5j7pJO2CAKowWFOSZ69Y+sZ37F3Wm7rHfWDloOTva3uWZCwtxAZs5gsyoJgTZrdTI3KeW254jLEwVCczPhxtvhNOnvROnv5OE70aaBv/3f/DUU2oF7YoVnqlrj6bRcdNCRi/uht5RyLsjf+aPVoM8cGEhypcfUoPTkY0ILsiGIUNKJf1WrWDVKtizB+64AwoKvBSoH5OE7yYFBWol4bRpanwyKckzPfvIM4cYvvRmbvniIQ436M5b92+scNciITwpLzSG0xEN4JNPVGGdnPNr6F9zDSxeDGvWqM+PbJ3hWjIP3w0yMmDQIHUDavJkeP55VTrBnYLsuXRf92+6r5sNwJfXv87GzmPRdHomPGIjPKb0nxauqqUvxMXItdQiau6LMGoUXHcdfPklREeXHB8xAg4eVHs5BwerKczu/vwECkn4LrZ5s9q45NAh+O9/1ZvXrQoLabctiWvWPEdk1hF+v2II3103mzNRjUqahMeU3pu0Xj1ITS37KeulbnJnxELAPfdARIQa2unVS22R2LBhyeHnnoO8PDVVMz9f9fo9de/Ln0nCd5GiIjWvfupUqFNH/Unavbv7rjfhwRzCv14Bl/+L2/ftg44d4ZUkWvfoQWv3XVYI17ntNtW7HzgQOnVSA/jFHxqdDl58UfXwp05VdXeWL1cVZUXVScJ3gd9/h5EjG7Jpk7rxdPPN8N136ssZFzO0ElRgpf3Wdwn/7xw4cABCQsio0Yz8oxoMKbsSlfTYRbV17bWwYYMqptOnD8yZA2PHgk6HTqeGdS67TC1UjI+HlSvhyiu9HbTvkpGxS3DqlJpb364d7NkTzDvvqLH74q18XSos5zjXrJnME6804MavHoU6dUiv0Qzi4sgPjvLcKi4hXO3yy1XS79tXzV++7bbzdjsfNQp++glyc6FzZ5g1C+ylF+0KJ0jCr4KDB9WOPc2awVtvqaT/9df7GDnS9Xm31snfufXT+xj/aiN6/DyDA036sHjUOli/XhWkkkQv/EGNGmqH81dfha+/Vn8qL1lSso/wVVfB9u1qLcukSWob0K++KjksnCQJ30n5+aqM8eDB0LSpel/eeCNs2wZz50JUlOsqPxmKCmi9Yykj3+3JmDfb0GZHEr+1v4+5j+zmgzs/5HCDq1x2LSGqDZ0OHnsMNm5UH7K771ZDPhs3AhATAx98oD6Hubnq89ehg9otrgp7xQckGcMvh6apIfIff1Rj8Z9/rqYM16ypevePPAINGrj2mrFpf9B2eyLttywmzHqSjBpN+fa6f7O1/UhyQ2NcezEhqqu2bWHdOvXn89Spahxn4EB46il0XbsyYIDaJW7JElWMcNgweOIJNZw6eLC67yvTOMsmCR+1uGP/ftVb37IFduyA336DI0fU8dhYNdVy8GB1X6ms6WFV2u1e06iZvgdmf8yUlUsw7Nyh3qk33wxjxhDdty/99Hr6XdrLE6L6sdkqXnZuMMDDD6t9P+fMgVdegVWr0Dp3QXffaEx33MHIkdHcey98+qmaAv3WW+qv7ZgYuPpq6NkTunVTFTnDwtRfAcYAz3gB8/Lz8lSPfd8+ldzPft+/Xz2en6/a6fVqDUidOqqj0bixegPpdLB+vfoqS1paGvPmxVYchKYRkXWE+kc30HT/9zTb9w01Tv8FgKFrV85ENCAvuAaOjUdh5HPAc+U+lcy8ET7NbFbTbpx15Ai8/z66efPggQcoemgsfzXuzb6mfTnerC/t2rTlH//Qs2eP+mz/9JMa+jkrKgp69FDJv2VLNWLUrBnUrRtYfw34ZcJfswbWrj0/sV+4yCgsTP0Pj4uDm25SEwXatYMrrlB/Jl6q4PzT1Dy1m5hTu4g5tYtaaTupl7qR8JzjANhMYRxoci3rrnqam+fdAI0bY5USCEKULTxcjaOOHcvCh7bQesdSWuz9in7fPw3fQ25INMfqdiS1bkeOXt6FXQMGkJWtIzUVTp5UX/v3qxu95473BwdDkybqF0CNGrWJj1d5oVEjNWQbGelf8yI8lvAdDgfTp09n9+7dmEwmXnzxRRo1alT5iRdJ01ThpdOn1fzdZs2gXz/1/exv9aZN/+61u8N13z/D1evO+a0RFATNm8PtfdV4ZJcumNu143KTicvdE4IQ/uXsEJBOx4MLOwAdgJdUT+777wlNTqbZ5s002/AfWFdI7odf8u/tNxARoTpzoMqcOBxqFfy5ncGz33/6KZrExPMvGxamEv/Zr7p1Ve6IiVH382rWVG1CQv7+MpnUdTRN1dTS69V/FxWpL7td/dKp6Ovyy9Vzu5rHEv73339PQUEBy5cvZ+vWrcyaNYs333zT5dfR6eDwYTUEeG5PvbBQVeHbs6fi811RX2bXPwZw9YCapM9cSKExmCKDGU7r4Ps/1Bfvnde+Xj1gkwzRiMBSXmmPstQr3jO3opIgYICYNhiKCqh9+/Ww/fyjRmPpz7fFooZu27aFkyfTsFhiycxUHcasLDhzRnUcDx9W9/hOnvTMVNAhQ9TsI1fzWMLfvHkzPXr0AODKK6/k999/d9u1wsLc9tROOdKgGzzVDdsry70biBCBRqenyBhcpT/fdTr1C8Bigfr1/3783F8SRUXql0F6uloblp6upojm5f393W5Xz6XXq69vv1U/n/tYZV+vvXbJ/xJlv0ZN88zSheeee45+/frRq1cvAHr37s3333+PsZzb5lu3bsXskeLxQgjhP2w2G1eWU3/CYz38sLAwrFZryc8Oh6PcZA+UG7AQQoiq8diEpA4dOpCcnAyo3nvLli09dWkhhBB4cEjn7CydPXv2oGkaM2bMoFmzZp64tBBCCDyY8IUQQnhXAK0xE0KIwCYJXwghAoQkfCGECBCS8C+Rw+Fg6tSpDBkyhBEjRnDw4EFvh+RWdrudp556iuHDhzNo0CBWr17t7ZA8Ij09nV69erFv3z5vh+IRCxcuZMiQIQwcOJAVK1Z4Oxy3stvtTJgwgaFDhzJ8+HC//n8sCf8SnVsyYsKECcyaNcvbIbnVp59+SlRUFElJSSxatIgXXnjB2yG5nd1uZ+rUqQQHB3s7FI/YsGEDW7ZsYenSpSQkJHD8+HFvh+RWP/30E4WFhSxbtoyxY8fy6quvejskt5GEf4k8WTKiOrj++ut57LHHSn42GAxejMYzZs+ezdChQ6lVq5a3Q/GItWvX0rJlS8aOHctDDz1E7969vR2SWzVp0oSioiIcDgc5OTkVLgj1df77yjwkJyeHsHOK9xgMBgoLC/32TWOxWAD1useNG8f48eO9G5CbrVq1iujoaHr06MFbb73l7XA8IjMzk9TUVBYsWMCRI0d4+OGH+frrr9H5U53gc4SGhnL06FFuuOEGMjMzWbBggbdDchvp4V+iiy0Z4Q+OHTvGPffcw4ABA7jlllu8HY5bffjhh6xfv54RI0aQkpLCxIkTSUtL83ZYbhUVFcXVV1+NyWSiadOmmM1mMjIyvB2W27z33ntcffXVfPPNN3zyySc888wz2Gw2b4flFpLwL1GglYw4deoUo0aN4qmnnmLQoEHeDsftlixZQmJiIgkJCcTFxTF79mxiYyvZ2czHdezYkZ9//hlN0zhx4gR5eXlERUV5Oyy3iYiIIDw8HIDIyEgKCwspKiryclTu4d9dUQ/o27cv69atY+jQoSUlI/zZggULyMrKYv78+cyfPx+ARYsWBcwNzUDQp08fNm7cyKBBg9A0jalTp/r1vZp//vOfPPvsswwfPhy73c7jjz9OaGiot8NyCymtIIQQAUKGdIQQIkBIwhdCiAAhCV8IIQKEJHwhhAgQkvCFECJASMIXQogAIQlfCCECxP8DNtzgalMiAT0AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import seaborn as sns\n", "%matplotlib inline\n", "sns.set_style(\"whitegrid\")\n", "\n", "sns.histplot(x=ux_old, data=df, color=\"blue\", label=\"UX old\", kde=True)\n", "sns.histplot(x=ux_new, data=df, color=\"red\", label=\"UX new\", kde=True, alpha=0.8);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calculate t-statistic\n", "\n", "The t-statistic is used in a t-test to determine whether to support or reject the null hypothesis. The formula of the statistic is:\n", "\n", "$$t = \\frac{\\bar{x}_{1} - \\bar{x}_{2}}{{s}_p \\sqrt{\\frac{2}{n}}}$$\n", "\n", "where $s_p$ is the pooled standard deviation:\n", "\n", "$$s_p = \\sqrt{\\frac{{s}^{2}_{X_1} + {s}^{2}_{X_2}} {2} }$$\n", "\n", "Thus, the t statistic as a way of quantifying how large the difference between groups is in relation to the sampling variability of the difference between means.\n", "\n", "Note that we perform the calculation of the t-Test manually to better understand the concept. \n", "\n", "Usually we would just use the independent t-test function below and are done:\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "t = 2.49205054533012\n", "p = 0.012862137872711413\n" ] } ], "source": [ "t, p = stats.ttest_ind(ux_new, ux_old)\n", "print(\"t =\", t)\n", "print(\"p =\", p)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First of all, let`s calculate the different components of the formula. We will start with the numerator of the formula. \n", "\n", "### Difference in mean\n", "\n", "Calculate the differeence between the two means:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.21\n" ] } ], "source": [ "diff_mean = ux_new.mean() - ux_old.mean()\n", "print(round(diff_mean,2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pooled standard deviation\n", "\n", "Next, we calculate the variance: \n", "\n", "$$s^{2} = \\frac{\\sum (x_{i} - \\bar{x})^{2}}{N - 1}$$\n", "\n", "and standard deviation: \n", "\n", "$$s = \\sqrt{\\frac{\\sum (x_{i} - \\bar{x})^{2}}{N - 1}}$$\n", "\n", "\n", "We use the numpy function `var` with the optional argument `ddof` which means “Delta Degrees of Freedom” (the divisor used in the calculation of the variance is (N - ddof), where `N` represents the number of elements. By default `ddof` is zero). \n", "\n", "Note that `ddof` is equal to the number of parameters we use. We estimate the mean, therefore we only have one parameter. See this [documentation](https://docs.scipy.org/doc/numpy/reference/generated/numpy.var.html) for more information." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variance:\n", "UX new: 1.452143284234209 UX old: 2.247145174966161\n", "Standard deviation:\n", "UX new: 1.2050490795956026 UX old: 1.499048089610924\n" ] } ], "source": [ "import numpy as np\n", "\n", "# calculate variance\n", "var_ux_new = ux_new.var(ddof=1)\n", "var_ux_old = ux_old.var(ddof=1)\n", "\n", "print(\"Variance:\")\n", "print(\"UX new:\", var_ux_new, \"UX old:\", var_ux_old)\n", "\n", "# calculate standard deviation\n", "s_ux_new = np.sqrt(var_ux_new)\n", "s_ux_old = np.sqrt(var_ux_old)\n", "\n", "print(\"Standard deviation:\")\n", "print(\"UX new:\", s_ux_new, \"UX old:\", s_ux_old)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we calculate the pooled variance (also known as combined variance, composite variance, or overall variance) to estimate the combined variance of our two groups:\n", "\n", "$$s_{p}^2 = \\frac {(n_{1} - 1)s_{1}^2 + (n_{2} - 1)s_{2}^2}{n_{1} + n_{2} - 2}$$\n", "\n", "Afterwards, we take the square root to get the pooled standard deviatin (${s}_p$)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "1.849644229600185" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calculate pooled variance\n", "var_p = ( (n_B - 1) * var_ux_new + (n_A - 1) * var_ux_old ) / ( n_A + n_B - 2 )\n", "var_p" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.3600162607852102" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# pooled standard deviation \n", "s_p = np.sqrt(var_p)\n", "s_p" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Calculate t-statistic\n", "\n", "$$t = \\frac{\\bar{x}_{1} - \\bar{x}_{2}}{{s}_p \\sqrt{\\frac{2}{n}}}$$" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2.49205054533012" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calculate t-statistic\n", "t = (diff_mean) / (s_p * np.sqrt(2/500))\n", "t" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The probability density function for t is ([see documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.t.html) and [wikipedia](https://en.wikipedia.org/wiki/Student%27s_t-distribution)):" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.9935689310636443\n" ] } ], "source": [ "## Compare with the critical t-value\n", "df = n_A + n_B - 2 # Degrees of freedom\n", "crit_t = stats.t.cdf(t, df=df)\n", "print(crit_t)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.006431068936355699\n" ] } ], "source": [ "#p-value after comparison with the t \n", "p = 1 - crit_t\n", "print(p)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "t = 2.4921\n", "p= 0.0129\n" ] } ], "source": [ "print('t =', t)\n", "print('p=', 2*p) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Note that we multiply the p value by 2 because it's a two tailed t-test.*\n", "\n", "You can see that after comparing the t statistic with the critical t value (computed internally) we get a good p value way below 5% and thus we reject the null hypothesis.\n", "\n", "This means we have an indication that the mean of the two distributions are different and statistically significant." ] } ], "metadata": { "interpreter": { "hash": "463226f144cc21b006ce6927bfc93dd00694e52c8bc6857abb6e555b983749e9" }, "kernelspec": { "display_name": "Python 3.8.2 64-bit ('base': conda)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": {}, "toc_section_display": true, "toc_window_display": false }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }