34. Accuracy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "incident-spell",
   "metadata": {},
   "source": [
    "# Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "latter-trailer",
   "metadata": {},
   "source": [
    "## Confusion Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "driven-chemical",
   "metadata": {},
   "source": [
    "|Reality/Prediction |0|1|\n",
    "|----|----|----|\n",
    "|0|TN|FP|\n",
    "|1|FN|TP|\n",
    "\n",
    "矩阵的最上一行代表预测值，最左列为真实值；\n",
    "\n",
    "1. 0 - Negative、1 - Positive；\n",
    "\n",
    "2. TN、FP、FN、TP：表示预测结果的样本数量；\n",
    "\n",
    "3. TN（True Negative）：实际值为 Negative，预测值为 Negative，预测 negative 正确；\n",
    "\n",
    "4. FP（False Positive）：实际值为 Negative，预测值为 Positive，预测 Positive 错误；\n",
    "\n",
    "5. FN（False Negative）：实际值为 Positive，预测值为 Negative，预测 Negative 错误；\n",
    "\n",
    "6. TP（True Positive）：实际值为 Positive，预测值为 Positive，预测 Positive 正确；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lasting-detective",
   "metadata": {},
   "source": [
    "## Precison and Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wound-architecture",
   "metadata": {},
   "source": [
    "|Reality/Prediction |0|1|\n",
    "|----|----|----|\n",
    "|0|9978<br>TN|12<br>FP|\n",
    "|1|2<br>FN|8<br>TP|\n",
    "\n",
    "###### 1）精准率（precision）\n",
    "\n",
    "精准率：预测所关注的事件的结果中（共预测了 20 次），预测正确的概率（8 次正确，12 次错误）；\n",
    "精准率 = TP / (TP + FP) = 8 / (8 + 12) = 40%\n",
    "含义：每做 100 次患病的预测，平均会有 40 次是正确的；\n",
    "\n",
    "###### 2）召回率（recall）\n",
    "\n",
    "召回率：对所有所关注的类型（癌症患者，共 10 个），将其预测出的概率（预测出 8 个）；\n",
    "召回率 = TP / (TP + FN) = 8 / (8 + 2) = 80%\n",
    "解释：每当有 100 个癌症患者，通过该预测系统，能够成功的找出 80 个癌症患者；\n",
    "\n",
    "###### 3）精准率和召回率比准确率好的原因\n",
    "\n",
    "结合实际的业务要求：目的是要根据体检信息，更准确和高效的找出癌症病人；准确率并不能反映系统是否找出了全部的癌症病人，或者找出了多大比例的癌症病人；而精准率和召回率更能直接的反应系统能够找出癌症病人的能力；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "narrow-conjunction",
   "metadata": {},
   "source": [
    "## Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "herbal-inspector",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "adjacent-birmingham",
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = datasets.load_digits()\n",
    "X = digits.data\n",
    "y = digits.target.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "lucky-morris",
   "metadata": {},
   "outputs": [],
   "source": [
    "y[digits.target==9] = 1\n",
    "y[digits.target!=9] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dominant-constraint",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "coated-question",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yasmine/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "log_reg = LogisticRegression()\n",
    "log_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "subtle-suite",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9755555555555555"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "healthy-column",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_log_predict = log_reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "honey-lambda",
   "metadata": {},
   "outputs": [],
   "source": [
    "def TN(y_true, y_predict):\n",
    "    assert len(y_true) == len(y_predict)\n",
    "    return np.sum((y_true == 0) & (y_predict == 0))\n",
    "\n",
    "def FP(y_true, y_predict):\n",
    "    assert len(y_true) == len(y_predict)\n",
    "    return np.sum((y_true == 0) & (y_predict == 1))\n",
    "\n",
    "def FN(y_true, y_predict):\n",
    "    assert len(y_true) == len(y_predict)\n",
    "    return np.sum((y_true == 1) & (y_predict == 0))\n",
    "\n",
    "def TP(y_true, y_predict):\n",
    "    assert len(y_true) == len(y_predict)\n",
    "    return np.sum((y_true == 1) & (y_predict == 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "still-spank",
   "metadata": {},
   "outputs": [],
   "source": [
    "def confusion_matrix(y_true, y_predict):\n",
    "    return np.array([\n",
    "        [TN(y_test, y_log_predict), FP(y_test, y_log_predict)],\n",
    "        [FN(y_test, y_log_predict), TP(y_test, y_log_predict)]\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "short-sector",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[403,   2],\n",
       "       [  9,  36]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "resistant-functionality",
   "metadata": {},
   "outputs": [],
   "source": [
    "def precision_score(y_true, y_predict):\n",
    "    tp = TP(y_test, y_log_predict)\n",
    "    fp = FP(y_test, y_log_predict)\n",
    "    try:\n",
    "        return tp / (tp + fp)\n",
    "    except:\n",
    "        return 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "falling-distinction",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9473684210526315"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "funny-merchant",
   "metadata": {},
   "outputs": [],
   "source": [
    "def recall_score(y_true, y_predict):\n",
    "    tp = TP(y_test, y_log_predict)\n",
    "    fn = FN(y_test, y_log_predict)\n",
    "    # try...except：异常检测；\n",
    "        # 没有异常，执行 try 后面的语句；\n",
    "        # 出现异常，执行 except 后面的语句，\n",
    "    try:\n",
    "        return tp / (tp + fn)\n",
    "    except:\n",
    "        return 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "normal-network",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pregnant-spiritual",
   "metadata": {},
   "source": [
    "## Scikit—learn.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "certified-comparison",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[403,   2],\n",
       "       [  9,  36]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "confusion_matrix(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "accessible-cargo",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9473684210526315"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "\n",
    "precision_score(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "brazilian-amateur",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import recall_score\n",
    "\n",
    "recall_score(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "classified-parks",
   "metadata": {},
   "source": [
    "## F1 Score\n",
    "+ F1 Score：兼顾降准了和召回率，当急需要考虑精准率又需要考虑召回率，可查看模型的 F1 Score，根据 F1 Score 的大小判断模型的优劣；\n",
    "+ F1 = 2 * Precision * recall / (precision + recall)，是二者的调和平均值；\n",
    "+ $F1 = \\frac{2·precision·recall}{precision + recall}$\n",
    "\n",
    "1. F1 是 precision 和 recall 的调和平均值；\n",
    "2. 调和平均值：如果 1/a = (1/b + 1/c) / 2，则称 a 是 b 和 c 的调和平均值；\n",
    "3. 调和平均值特点：|b - c| 越大，a 越小；当 b - c = 0 时，a = b = c，a 达到最大值；\n",
    "4. 具体到精准率和召回率，只有当二者大小均衡时，F1 指标才高，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "abandoned-guest",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8674698795180723"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1_score(y_test, y_log_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "healthy-decision",
   "metadata": {},
   "source": [
    "## Threshold\n",
    "\n",
    "+ LogisticRegression() 类中的 predict() 方法中，默认阈值 threshold 为 0，再根据 decision_function() 方法计算的待预测样本的 score 值进行对比分类：score < 0 分类结果为 0，score > 0 分类结果为 1；\n",
    "+ log_reg.decision_function(X_test)：计算所有待预测样本的 score 值，以向量的数量类型返回结果；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "extraordinary-walnut",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "X = digits.data\n",
    "y = digits.target.copy()\n",
    "y[digits.target==9] = 1\n",
    "y[digits.target!=9] = 0\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "historical-parks",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yasmine/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "log_reg = LogisticRegression()\n",
    "log_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "dutch-committee",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predict_1 = log_reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "aboriginal-ecology",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[403,   2],\n",
       "       [  9,  36]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_test, y_predict_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "faced-publication",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9473684210526315"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "precision_score(y_test, y_predict_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "respiratory-vessel",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import recall_score\n",
    "recall_score(y_test, y_predict_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "controversial-moses",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[404,   1],\n",
       "       [ 21,  24]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_score = log_reg.decision_function(X_test)\n",
    "y_predict_2 = np.array(decision_score >= 5, dtype='int')\n",
    "\n",
    "confusion_matrix(y_test, y_predict_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bridal-pearl",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.96"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_test, y_predict_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "amino-encyclopedia",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5333333333333333"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_test, y_predict_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "detailed-floor",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[390,  15],\n",
       "       [  5,  40]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_score = log_reg.decision_function(X_test)\n",
    "y_predict_3 = np.array(decision_score >= -5, dtype='int')\n",
    "\n",
    "confusion_matrix(y_test, y_predict_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "about-labor",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7272727272727273"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_test, y_predict_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "collaborative-olive",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8888888888888888"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_test, y_predict_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesbian-central",
   "metadata": {},
   "source": [
    "## P-R Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "respective-campus",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "baking-librarian",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "intellectual-document",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yasmine/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "log_reg = LogisticRegression()\n",
    "log_reg.fit(X_train, y_train)\n",
    "decision_scores = log_reg.decision_function(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "stuffed-illness",
   "metadata": {},
   "outputs": [],
   "source": [
    "precisions = []\n",
    "recalls = []\n",
    "thresholds = np.arange(np.min(decision_scores), np.max(decision_scores), 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "handmade-processing",
   "metadata": {},
   "outputs": [],
   "source": [
    "for threshold in thresholds:\n",
    "    y_predict = np.array(decision_scores >= threshold, dtype='int')\n",
    "    precisions.append(precision_score(y_test, y_predict))\n",
    "    recalls.append(recall_score(y_test, y_predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "tropical-photography",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAl9klEQVR4nO3dd3hc1Z3/8fd3ZiSNuizL3ZYtNwyELoxZ2kKoaaQtISEhbdfrtE1+T5INabspu8+GkM2mbBIvCSSQTUIaOIYQCGlAYA0YsI0L7k2ukmUVq4ymnN8fd2RkWZZG0kh3yuf1PHrunTtXM9+L4cPxueeeY845REQk+wX8LkBERNJDgS4ikiMU6CIiOUKBLiKSIxToIiI5IuTXF9fU1Lg5c+b49fUiIlnp+eefb3LOTRroPd8Cfc6cOaxevdqvrxcRyUpmtvtU76nLRUQkRyjQRURyhAJdRCRHKNBFRHKEAl1EJEcMGehmdreZHTaz9ad438zsW2a2zczWmdn56S9TRESGkkoL/UfA9YO8fwOwIPmzFPje6MsSEZHhGnIcunPuCTObM8gpNwL3Om8e3lVmVmVm05xzB9JV5AkObYQND4zJR4tkvdNfD9PO9rsK3z2y/iAb97f6XcYp1c+p5vKFAz4bNCrpeLBoBrC3z+uG5LGTAt3MluK14qmtrR3ZtzVthifuGNnviuQ0B01b4KZ7/C7Ed5954CWaO3ow87uSgS27Yl7GBvpA/8gGXDXDOXcncCdAfX39yFbWOPNN3o+InOiua6HrqN9VZISunjhLL5/LZ15zut+ljKt0jHJpAGb1eT0T2J+GzxWR4QhXQnfmdjOMF+cckVicolD+DeJLxxWvBG5NjnZZArSOWf+5iJxauMprocdj3k+eiiUcCUdeBvqQXS5m9jPgb4EaM2sA/hUoAHDOLQceBl4DbAM6gfeOVbEiMoiSamjZDV+e6L1+3TegPv/+c4zEEgAUhYI+VzL+Uhnl8vYh3nfAh9JWkYiMzEXLoLTGu4P11Dfg4Et+V5R2h9u7aWyPEDBj/uQyCoInt8Ij0TgARQVqoYtItqqug8s/6e2v/Rl0t/haTrrFE46rvvY4xyJed9L0yjCXLqhhWmUx77p4NjVlRUDfFroCXURyQbgSulr8riKtjkViHIvEuKl+Jq+aUcmKF/excu1+uqMJvv2nrXzs6oW88dwZxJ03gE5dLiKSG4qr4PAm+OOXxubzp50LZ7xhbD77FDp7vJb5ebUTePviWm69eA4AT29r4nMr1vP1x7bw9ce2UFLoBbla6CKSG2bUw84n4Klvpv+zE3EonjDugd6R7GopLToxtv5mfg1//PgVPLuzmZVr9/OTZ/YAUFNeNK71ZQIFukguuuqz3s9Y+OOX4a9fB+dIx6OYd/11Jz96eudJxw3j7JmVfOLa05hTU0pHxLvZWVp4cleKmXHR3IlcNHciH716AR2ROHU1paOuLdso0EVkeMKV4BIQaYdwxag/7s8vH6arJ87lC058FP5QezcPrTvAQ+sOUFIYJFzgBXlJ4eCxNbk8DOWjLisrKdBFZHiKq7xtRyOEisACECwY8ce1dUd51YxKvv62c09678+bD/P0tiaOdkb51fMNAEzKw66UVCnQRWR4iqu97beTSx9YEG5dAXWXj+jj2rqip+weufK0yVx52mQAll4+l2g8wfzJZSP6nnygQBeR4Zl3FVz/FYh2QrTLm/20cfOIAj2ecOw60sllC4aeeXDhlDztRxkGBbqIDE9hCSz5gLcf6/ECfYRj3p/f7c0OGQpm6Dy3WSb/BmqKSPqECqGgZMRPpbZ1RQF403kz0lhU/lKgi8joFFXAqu9CLDLsX+19TL+4IP+e6hwLCnQRGZ2yyd4wxr3PDvtXu3sn0srDx/THggJdREbnxu942xGslnR8Iq08nBlxLOifooiMTrjS245gtaRIrLeFrihKB41yEZHROf6g0WHo6XjleKgYAoMHdXfUa6GH1YeeFgp0ERmdwnIIFHgzO/ad3XHeVfCuBwb91d4WeuEAC1XI8CnQRWR0AgG46V44svWVY5sehEMbhvzV53cfxQwCAY1DTwcFuoiM3qLXnPi6oymlJfC2HT5Gcj0KSQP9PUdE0q+4CmLdEO0e9LSOSIx3XFQ7PjXlAbXQRST9eke+PPoZKCh+5XioCP7mI1A8gUTC0R6JHV8LVEZPgS4i6Tf1HG9WxnU/f+WYS3gTek06necrr+ZAaxfOQWXxyKfelRMp0EUk/WZdCJ/qtwrRscPwtQUcPXKYt/z06eOHZ1SFx7m43KVAF5HxkeyG6Ww7AszhSzeeyeK6ak7TtLhpo5uiIjIuul2IqBXSfNhbeejcWVUsmlqBpWFdUvEo0EVkXKzf10p7opCz9v2c6TQxoaTQ75JyjgJdRMZFW3eUe+PXAjC3oJGJZQr0dFOgi8i42NfSzWPxegC+++Z5lBTqFl66KdBFZMz1xBJ8fsV6WvEWg66g0+eKcpP+FykiY+5Ih7ea0eJFdbATWPNTOLjOe9MCcMF7YdJC/wrMEQp0ERlzR471AHDt+Qug5wI49JL3A9486oEgXPtvPlaYGxToIjKmemIJPvuAF94Ty8PwD3868YT/XARdLeNfWA5KqQ/dzK43s81mts3Mbhvg/Uoze9DM1prZBjN7b/pLFZFs9NK+VtY2eKsZLZhcdvIJ4coRrXYkJxuyhW5mQeA7wDVAA/Ccma10zm3sc9qHgI3Oudeb2SRgs5n9xDnXMyZVi0jWOHLM6z9/6COXUjXQ2PNwlTfdbmfzye8VTwA9eJSyVLpcFgPbnHM7AMzsPuBGoG+gO6DcvEe+yoBmIJbmWkUkyxyLxHh+j7d49CnHnZdMhM2/ha/WnfzeVZ+Hyz8xhhXmllQCfQawt8/rBuCifuf8N7AS2A+UA29zziX6f5CZLQWWAtTWag5kkVz377/dxM+e3UNRKEB16SkC/ZovwtwrTj7++O3QtGVsC8wxqQT6QH/f6b/GyHXAGuAqYB7wmJk96ZxrO+GXnLsTuBOgvr5e65SI5LiGo50snFLG8ndeQFHoFAtB1yzwfvp78X91s3SYUrkp2gDM6vN6Jl5LvK/3Avc7zza8kaaL0lOiiGQj5xz7WrqorS5l7qQBboYORTdLhy2VQH8OWGBmdWZWCNyM173S1x7g1QBmNgU4DdiRzkJFJLt89L417GjsYHLFCFckKq6Cvavg6O601pXLhgx051wM+DDwKLAJ+IVzboOZLTOzZcnTvgz8jZm9BPwR+JRzrmmsihaRzLflUDsAyy6fN7IPmLnY225ckZ6C8kBKDxY55x4GHu53bHmf/f3AtektTUSyWU88wevOnkbtxJKRfcCSD8Jjnx9yoWl5hSbnEpExEYkmTn0jNBXBEARCEFOgp0qBLiJjIhKLU1QwyogJFUMskp6C8oACXUTGRCSaIDyaFjpAqEgt9GFQoIvImIjEEmlooYcV6MOgQBeRtEskHD3xBEWh0Qa6WujDoUAXkbSLxLyZP0Z1UxSgQH3ow6FAF5G064rGAQiPusulCPavgQc/BpsfGXVduU6BLiJp1dLZw/ef9B4Urykb4VOiveZcBvEeWPMTePJraagutynQRSStvv/kDr73l+0ATKkIj+7DrvkifHIrLHqtJupKgQJdRNJqb3PX8f3a6hE+JdqfJupKidYUFZG0icUTrFy7n8V11dz17nrKwwXp+eBwlQI9BWqhi0ja7DrSCcDs6pL0hTl4LfR4BKJdQ5+bxxToIpIWzjme2NIIwFsvmJneDy+u8rZqpQ9KgS4iabG2oZUvPeQtNTx7Yml6Pzxc6W11Y3RQCnQRSYvth48B8M2bz2Vq5ShHt/QXrvK2aqEPSjdFRSQt9rd4/dvXnTk1/R/eG+iP3w4V08EC3nzpk7XSZV8KdBFJiyMdPVSEQ4QLRvm4/0AmzoPJZ8LhTd5P+34oqYarv5D+78piCnQRSYu2riiVJWkc2dJXcRV88OlXXt8xX/3pA1AfuoikRVt3lPKiMQr0/jQufUAKdBFJi7auGBXF4/SX/nAldLeMz3dlEQW6iKRFS1cPFel8mGgwxVVqoQ9AgS4io+acY39LN9OrisfnC8OV6kMfgAJdREbtia1NHIvEmDlhHANdLfSTKNBFZNRe2H0UGKMx6AMJV3l96M6Nz/dlCQW6iIza3uZOpleGmZWu6XKHEq6ERAy2aBWjvhToIjJqh9sjTEn34/6DqV3ibV+4d/y+Mwso0EVkxFo6ezjzXx7hr9uamFBSOH5fXLsEZl8C3W3j951ZQIEuIiO2rqGVjh5vQeiu5Hbc6MboSRToIjJie5o7j+9vOdQ+vl8eroTOI9C6b3y/N4Mp0EVkRCKxOA+t209xQZC3L57Ft95+3vgWUDrJm6Trv86ATQ+N73dnKE3OJSIj8p+/38KqHc2cM6uK/3jz2eNfwCUfgwmz4bcfh/YD4//9GSilFrqZXW9mm81sm5nddopz/tbM1pjZBjN7PL1likimWb+vlcJggO/ecr4/BZROhLPf5u3Huv2pIcMM2UI3syDwHeAaoAF4zsxWOuc29jmnCvgucL1zbo+ZTR6jekUkQ2w9fIw3nDudGeP1uP9AQsmhklEFOqTWQl8MbHPO7XDO9QD3ATf2O+cdwP3OuT0AzrnD6S1TRDLJv/92I43tERZMLvO3kEDIW71ILXQgtUCfAezt87oheayvhcAEM/uLmT1vZrcO9EFmttTMVpvZ6sbGxpFVLCK+OhaJ8f0ndwJw+cJJ/hZj5rXSFehAaoFuAxzrP4FCCLgAeC1wHfB5M1t40i85d6dzrt45Vz9pks//IojIiPzuJe8G5NdvOofTp1X4XA3JQI/4XUVGSGWUSwMwq8/rmcD+Ac5pcs51AB1m9gRwDrAlLVWKSEZwzvHJX60DoLJ4nOY+H0ooDLEuv6vICKm00J8DFphZnZkVAjcDK/ud8xvgMjMLmVkJcBGwKb2liojfDra90rUxdTznbhlMqBCOqQsXUmihO+diZvZh4FEgCNztnNtgZsuS7y93zm0ys0eAdUAC+IFzbv1YFi4i4++pbUcA+ObN53Lm9Eqfq+llsPVRv4vICCk9WOScexh4uN+x5f1e3wHckb7SRCTT/PAp72boVYsyaGRy7cVwdKc3dLEgQ/7W4BM9+i8iKdnX0sWG/W1cffoUysdr7dBUzEg+2BTRzIsKdBFJyVPbmgB48/n9Ry37LFzlbTXzogJdRFLT2O4NDcyo7haA4ipvu/MJX8vIBAp0ERnS8se388OndlERDhEuCPpdzommnuVtG1/2t44MoEAXkUEdaO3iK797maJQgH+8Yp7f5ZysfCpUzVaXC5o+V0SGcPvvvJbvP716Pm+7sNbnak5BqxcBaqGLyBD2tXSxYHIZN9XPGvpkvxRXwaEN4PrPSpJfFOgiMqhDbRHOmF6B2UDTOmWIcBW07oX1v/a7El8p0EXklA60drGnuZPZE0v9LmVw13zJ27bs9rcOnynQReSUvvPnbQBcPHeiz5UMoboOgkV534+uQBeRAbV2RfnfVXuYN6mUi+dleKCDd2O0q8XvKnylQBeRAa3e1QzA+y+d63MlKQpXwgv3wJ5VflfiGwW6iAzo2Z3NFAQt8x71P5Uz3+Rt197nbx0+UqCLyICe3dXM2TOrMu/J0FO56rMwcX5e96Mr0EXkJLuPdPDinhYW11X7Xcrw5PkDRgp0ETnJP/74eQAumVfjcyXDFK6C5h1+V+EbBbqInCCecDQc7eKiumoumZ8Fo1v6ChZ6i120H/S7El8o0EXkuM0H2znjXx7hWCTGtWdOzeynQwdy2g3etq3/Ovb5QZNziQgAbd1RrvuGN6f4J687jTeflyWjW/qqWeBt87QfXYEuIgA8tdVbkejWi2fzoSvn+1zNCB1fvajFzyp8oy4XEQFg08F2Agaffe3pfpcycuFKb/vAB+BYo7+1+ECBLiKAN1RxelUxRaEsGXc+kLIpULMQYl2w52m/qxl3CnQRAWBXUwd1NRk+q+JQgiF45/3efh72oyvQRYQ9RzpZ29DK7Iklfpcyer2LRufhRF0KdBHhwXXeML/XnT3d50rSoLAMLAhHtsKBtdDZ7HdF40aBLiJs2N9KXU0pSzJ93vNUmHl96S/cC/9zOdx9nd8VjRsNWxQRXj7Yztxs7z/v69YVcGQbrPkpbP+z39WMG7XQRfLcgdYudjR2ZMciFqmadBosei1MOweiHRCP+l3RuFCgi+S5PUc6AVg4pdznSsbA8QeN8mPEiwJdJM8dbOsGYHpV2OdKxkDvg0YrPgjO+VvLOFCgi+S51buOEi4IMHNCDgxZ7G/WYm+79dG8GO2SUqCb2fVmttnMtpnZbYOcd6GZxc3srekrUUTGwmMbD3H+lx/jp8/u4erTp2TPykTDUV0Hb1zu7efB/C5DjnIxsyDwHeAaoAF4zsxWOuc2DnDe7cCjY1GoiKRPLJ7gCys3UBA0br14NrdcNNvvksZO74NGeRDoqbTQFwPbnHM7nHM9wH3AjQOc9xHg18DhNNYnImPgF6sb2NfSxTsWz+ZfX38m8yeX+V3S2OntR8+DG6OpBPoMYG+f1w3JY8eZ2QzgTcDy9JUmImPBOcc9T+9iRlUxH7kqS6fJHY7ekS55MBVAKoE+0JIl/W8XfwP4lHMuPugHmS01s9VmtrqxMf+mthTJBJ954CU2H2rnvZfMIRDIshWJRiKPWuipPCnaAMzq83om0H99p3rgvuRyVTXAa8ws5pxb0fck59ydwJ0A9fX1uT+GSCTD7Gg8xs+e3Ut5OMQ7l+Rwv3lfedSHnkqgPwcsMLM6YB9wM/COvic45+p6983sR8BD/cNcRPz3+Bbvb8YrP3xpbo5qGUhBCQRCaqEDOOdiZvZhvNErQeBu59wGM1uWfF/95iIZzjnHt/64je8/uYOpFeHsn/d8OMy8fnQFusc59zDwcL9jAwa5c+49oy9LRNLp8S2N/NcftjCxtJBlV8z1u5zxF67Mi5uimm1RJA/c9dedTKko4sl/vorCUB4+IF5clRct9Dz8kxXJLw+u3c+TW5t4y/kz8zPMwWuh58FN0Tz90xXJD3ubO/nIz14E4MpFk32uxkfhKjh2GPY+C239B+nlDgW6SA77/cZDAPx86RIunFPtczU+Kp8KrXvhrmvgB1f7Xc2YUaCL5KjfrNnHlx/ayIVzJrC4Lo/DHOBvPw3vvB/OuslroScSflc0JhToIjmopbOHzz2wHoB/f9NZJB/6y1/hCpj/am8FIxz0tPtd0ZhQoIvkGOccdzy6mfZIjN986JLcXIlopHJ8GgAFukiOeXTDQX7yzB4AzppR6XM1GaZ3GoBfvgcSg049lZUU6CI5pDsa51O/fun467yYfGs4ZtR7233P5+RoFwW6SA75v+1HaO2K8snrTmPVp1/tdzmZp2Ia3HSvt5+D3S56UlQkh7y45ygBg/dfWpc/k28N1/F+9BZfyxgLaqGL5JBNB9uZO6lMYT6Y3gUvdv0VXG7N4q1AF8khLx9sY9FUjWoZVNkUb/uX/4ADa3wtJd0U6CI5Ym9zJ3ubuzhnZpXfpWS2imnwhv/29tsO+FtLminQRXLA7146wGVf/TMA179qqs/VZIE5l3jbHLsxqkAXyXLP7mzmyw9tBOBzrz2dWdUlPleUBXr70XMs0DXKRSSLrXhxHx/7+RoA7nnfYq5YOMnfgrJF70iXVd+Fozvhhtv9rSdN1EIXyVItnT18/JdrKS8K8ctlFyvMhyMQhMVLvVEuzyzPmadGFegiWei5Xc18bsV64gnHj953YX5PjTtSr7kDlnzA24+0+VtLmqjLRSTLOOf4xC/XsvtIJ+9cUsv5tRP8Lil79Xa9dLVAcfb/c1Sgi2SZlWv3s/tIJ199y9ncdOEsv8vJbr2Tde1ZBdV1vpaSDupyEckimw608bkV61k0tZy3XjDT73KyX/k0b7tiGbQf9LeWNFCgi2SJP2w8xOu//VdKC0Pc9Z4LNZNiOkw/Dy77hLd/7JC/taSBAl0kS9zzf7soC4f42dIlzKgq9ruc3GAGc6/w9nNgTLoCXSQLHDkWYc2eFq4/cyp1NaV+l5Nbeh8y6mrxs4q0UKCLZLhILM47vv8MPfEE71wy2+9yck/vSJc/fhHuuwWi3f7WMwoKdJEM9+P/283mQ+18/aZzeZWWlEu/yplw9tsgWAQvPwRHtvld0Yhp2KJIBnLOsWF/GyvX7ufOJ3Zw2YIabtCkW2MjEIQ33wk7/gL33pjVfekKdJEMdMsPnuHp7UcAqKsp5T/efJZGtYy14ysZKdBFJA3iCcd7fvgsT28/wrVnTGHp5XO5YPYEzBTmY6430Lf+HhZeD4Hs65HOvopFcpRzji+s3MCTW5t46wUz+d47L6B+TrXCfLyUJic3e/6HsPNxf2sZIQW6SAZIJBy3P7KZH6/aza0Xz+arbzmboLpYxldROdzyK28/Sx8ySinQzex6M9tsZtvM7LYB3r/FzNYlf542s3PSX6pIbmo42snSH69m+ePbeesFM/nC689Uf7lfZlzgbbO0H33IPnQzCwLfAa4BGoDnzGylc25jn9N2Alc4546a2Q3AncBFY1GwSK7YsL+VFS/u44dP7QJg2RXz+NT1p6mLxU9FFd42Sx8ySuWm6GJgm3NuB4CZ3QfcCBwPdOfc033OXwVo1iCRQdz37B5uu/8lAC5bUMOnbzidM6ZX+FyVEAxBYbnXj15SDYv/we+KhiWVQJ8B7O3zuoHBW9/vB3430BtmthRYClBbW5tiiSK5IxKL86lfrWPFmv0snFLGD269kNqJWgM0o1z8QXj2+7DuFzkZ6AP9/c8NeKLZlXiBfulA7zvn7sTrjqG+vn7AzxDJVRv2t/LJX65j44E2brmoln++fhGVxQV+lyX9XfkZaNwMhzcOfW6GSSXQG4C+s+jPBPb3P8nMzgZ+ANzgnDuSnvJEspdzjmjc0XC0k8+tWM/T249QVhTijreezd/Va2GKjBauzMobo6kE+nPAAjOrA/YBNwPv6HuCmdUC9wPvcs5tSXuVIlkkEovzwAv7WP74dnYd6QS8WVr//tI6ll4+l8kVYZ8rlCEVV+VmoDvnYmb2YeBRIAjc7ZzbYGbLku8vB/4FmAh8N3mHPuacqx+7skUyR08swZNbG9nT3MnRjh7uf3EfDUe7mFxexD9dNZ8JpYVctqCG+ZPL/S5VUhWuhFi3N/NiQfb8DzilR/+dcw8DD/c7trzP/t8Df5/e0kQyWyLhaO2K8pGfvchftzUdP75wShl3vusCrlo0mVBQz+5lpd450rtbcy/QRcQTTzh2NB7jD5sOc/dTO2lsjxAMGP/2xldx5aLJTCwtJFwQ9LtMGa3jE3W1QPkUX0sZDgW6yBBaOnt4fEsjD649wNPbm+jsiQMwf3IZbz5/BteeMYULZlf7XKWkVXGVt82yfnQFusgADrZ28/Pn9vKnzYdZv6+VeMIxpaKIN503g/NqJ7BoarkWm8hlWbosnQJdpA/nHH/efJiP3reG9u4YC6eUcevFs7n2jKlcVFetOVbyRd8+9CyiQJe8Fo0nWL3rKBv2t7LpQDsv7DnKzqYO5k8u41fLzue0qRqZkpf69qFnEQW65I3uaJydTR08ta2JF/Yc5UBrNxv3txGJJQAoLghSP2cC77u0jhvPnU5FWE9x5i0FukhmSCQc2xuP8eLeFl7cc5Sth47ReCzC7uRDPgC11SVMrwrzjotqOa92ApfOr6GquEBdKuIJFUJBCRxYC5sfOfG9sskw43x/6hqCAl0yWiLhONDWze6mDiLxBLG4IxZPsL+1m5cPtNEZjRONJeiOJWho7qStO0pbV4yeuNfqLgwGOGdWJadNKefGc2cwu7qEJfMmMqOq2Ocrk4xXMQM2Pej99GUB+OR2bzbGDKNAF19EYnFau7zw3XSgjV1NHXRG4xxq7eZAazeH27tp6YzS3NmDO8U0bpPKi6gIhygIBigMBVg0rZyqkkLKwyGmVxazuK6auppSjQuXkXnfI9Cy58Rj2/8Ef/oydDQp0CX3tXZF6YjEiMUdkVic1buPsre5ky2HjnGkI8LRjh46e+K0dEaPt6J7FQSNmrIiZlQVc9rUciqLC5lUXsSk8iLmTfKCuSAQIBQ0JpYWak4UGVulNd5PX53JeQczdPSLAj3POeeIxBIcauvmaGeUaDxBV0/8eDC3d3vdF23dUXpiCeIJRyzhiMYStHZF6UyeG4nFae6I0nQsctJ3hALG3EmlTKkIM3NCCaWFQcqKQsyeWEJpUYiFU8pZMKWMopBa0pLhMvxmqQI9A/WGbHc0Tlc0zrHuGB09cToiMRrbIzR39BCJJYjE4rR1xdh9pINYwpFw3k884UgkIN67n9x2R+N0R18J5a6eGJ3R+Cm7NPoqDAUIhwIEA0YwECAUMCqLCygpClJTVkhRKMi5s0LMm1RGZXEBoWCAgqAxb1IZp00tp0Bzmkgu6B2fvv5+WHCNr6UMRIE+TImEI5pIkEhALLmNO3d8PxKL09kTJxJL0BNL0BWN09UToyfuvNc9MXY2eTfvmjt6aDoWIRJNEE0kiMYTRGOO5s4eemKJoYsBikIB6mpKKQwFCJh5gWtGIAAFgVeOBcwIFwQIFwQJBYxQMEA4FKSsKEhRQZDJ5UVMLCukMBgkXBCgoriA0qIQFeEQRaEgBUHTWpci1XO9bWdmLvmQdYG+fl8rv3q+ge5onHjCEXeORMIRd17Y9h6LxRPHW6P9W6q9+14/b+J4yzbhvNZxwuG9Tjhccj+ePJ5q0A6muCBIVUkBVSVeH3FxQYCCYO+PUVVSSFVJAeFQkHBBkPJwiNKiICWFIWrKiqgp8yaAKgwGNMxOZDwFQ1B3ufrQ02VfSxe/fr6B4kKvpRlItj69VijHW6MFwQDhAq+LoG9L1Wu9ettQ0AgXBI+3aM28zwiY9xl2fN/bYlAUDFBUEDz+WV4XxCs/hcEAJYVBigoCFAaDFBcGKC4IURgKUJgcjTG5vEhBLJKtwpXQtNXvKgaUdYF+3ZlTue6LU/0uQ0TyVbgSGl+G5p1QXed3NSfQnSoRkeGorPW2v3qvv3UMQIEuIjIcl30cpp8P7Yf8ruQkCnQRkeEIhqD2Yoi0+V3JSRToIiLDVVwFPccgHvW7khMo0EVEhqv3idH7bvG3jn4U6CIiw9X7lOjWRyEe87eWPhToIiLDVT0Xrv+Kt59BfekKdBGRkTi+kPRRX8voK+seLBIRyQi9/eibHoQJc1L/veq5MO3sMSlJgS4iMhJVs7ztH/51eL8XroTb9gx93ggo0EVERmLqWfDRddDTkfrvvHAPPLMcYhEIFaW9JAW6iMhITZg9vPMnzve23a3eYtNpppuiIiLjpfdG6hhNv6tAFxEZL703Up/+9ph8vAJdRGS8zFoM594Ccy4dk49PKdDN7Hoz22xm28zstgHeNzP7VvL9dWZ2fvpLFRHJcsVV8Mbvwtk3jcnHDxnoZhYEvgPcAJwBvN3Mzuh32g3AguTPUuB7aa5TRESGkEoLfTGwzTm3wznXA9wH3NjvnBuBe51nFVBlZtPSXKuIiAwilUCfAezt87oheWy452BmS81stZmtbmxsHG6tIiIyiFQCfaDVjN0IzsE5d6dzrt45Vz9p0qRU6hMRkRSlEugNwKw+r2cC+0dwjoiIjKFUAv05YIGZ1ZlZIXAzsLLfOSuBW5OjXZYArc65A2muVUREBjHko//OuZiZfRh4FAgCdzvnNpjZsuT7y4GHgdcA24BOIPOWwxYRyXEpzeXinHsYL7T7HlveZ98BH0pvaSIiMhzmZbEPX2zWCOz25cs9NUCTj98/XvLhOvPhGkHXmUtGc42znXMDjirxLdD9ZmarnXP1ftcx1vLhOvPhGkHXmUvG6ho1l4uISI5QoIuI5Ih8DvQ7/S5gnOTDdebDNYKuM5eMyTXmbR+6iEiuyecWuohITlGgi4jkiLwKdDM718xWmdma5KyPi/u89+nkAh2bzew6P+tMBzP7SPJaNpjZV/scz6nrBDCzT5iZM7OaPsdy5jrN7A4zezm5eMwDZlbV571cus5BF9LJVmY2y8z+bGabkv89fjR5vNrMHjOzrcnthFF/mXMub36A3wM3JPdfA/wluX8GsBYoAuqA7UDQ73pHcZ1XAn8AipKvJ+fidSavaRbetBS7gZpcvE7gWiCU3L8duD3XrhNvWpHtwFygMHldZ/hdV5qubRpwfnK/HNiS/LP7KnBb8vhtvX+uo/nJqxY63pS+Fcn9Sl6ZEfJG4D7nXMQ5txNvTprFA/x+tvgA8BXnXATAOXc4eTzXrhPgv4B/5sTpmnPqOp1zv3fOxZIvV+HNZgq5dZ2pLKSTlZxzB5xzLyT324FNeOtF3AjckzztHuCNo/2ufAv0jwF3mNle4GvAp5PHU1qgI4ssBC4zs2fM7HEzuzB5PKeu08zeAOxzzq3t91ZOXWc/7wN+l9zPpevMpWs5JTObA5wHPANMcclZaZPbyaP9/JQm58omZvYHYOoAb30WeDXw/5xzvzazm4C7gKtJcYGOTDLEdYaACcAS4ELgF2Y2l9y7zs/gdUec9GsDHMva63TO/SZ5zmeBGPCT3l8b4PyMvs5B5NK1DMjMyoBfAx9zzrWZDXTJo5Nzge6cu/pU75nZvcBHky9/CfwguZ91C3QMcZ0fAO53Xufcs2aWwJsMKGeu08zOwus3Xpv8D2Mm8ELyRnfOXGcvM3s38Drg1ck/V8jC6xxELl3LScysAC/Mf+Kcuz95+JCZTXPOHUiuwXz41J+QmnzrctkPXJHcvwrYmtxfCdxsZkVmVgcsAJ71ob50WYF3fZjZQrybTE3k0HU6515yzk12zs1xzs3BC4TznXMHyaHrBG/0B/Ap4A3Ouc4+b+XSdaaykE5WMq/FcRewyTn39T5vrQTendx/N/Cb0X5XzrXQh/APwDfNLAR0A0sBnLdgxy+AjXh/pf2Qcy7uX5mjdjdwt5mtB3qAdydbdbl2nQPKwT/P/8YbyfJY8m8jq5xzy3LpOt0pFtLxuax0uQR4F/CSma1JHvsM8BW87tD3A3uAvxvtF+nRfxGRHJFvXS4iIjlLgS4ikiMU6CIiOUKBLiKSIxToIiI5QoEuIpIjFOgiIjni/wNrk78h6GuKsgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thresholds, precisions)\n",
    "plt.plot(thresholds, recalls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "executive-account",
   "metadata": {},
   "source": [
    "###### P-R Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "royal-defensive",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAU1ElEQVR4nO3dfWxd9Z3n8fc3znMg5MkJNA8kA6FpuoUpGAZoOwXabQnMDFst7UIrWNgZIXZKt6t9EGikaXdVabWj2RlVbWmzUcuiqlrQtGUG2skUzUILHSgdHBUCKYS64SEmaew8J04cY/u7f9gUYxz7xL72uT5+v6Qr3XPOz/d885PzyS+/e875RWYiSZr8ppVdgCSpNgx0SaoIA12SKsJAl6SKMNAlqSKml3XiJUuW5OrVq8s6vSRNSlu2bNmbmY1DHSst0FevXk1zc3NZp5ekSSkiXj3ZMadcJKkiDHRJqggDXZIqwkCXpIow0CWpIkYM9Ii4JyLaIuL5kxyPiPhKRLRExNaIuLD2ZUqSRlJkhH4vcPUwxzcAa/tftwHfGHtZkqRTNeJ16Jn5eESsHqbJdcC3s+85vE9FxIKIOCszd9eqyIG2/+YIf79113h89KR16TmLufycJWWXIalktbixaDmwc8B2a/++dwR6RNxG3yieVatWjepkLW1H+eqPW0b1s1WUCT9t2cvf/qmBLk11tQj0GGLfkKtmZOYmYBNAU1PTqFbWuPb8s7j2/GtH86OVdP03nmRGg99tS6rNVS6twMoB2ysA50QmyNET3cybVdoTHCTVkVoE+kPAzf1Xu1wKHBqv+XO9U0dXN/NmNdDd01v45bKDUjWNOLSLiPuAK4AlEdEKfBGYAZCZG4HNwDVAC3AMuHW8itU7nXijlwef2cWDzxT/T9Flv7OY+267dByrklSGIle53DjC8QQ+W7OKdEr+xyfexwu7Dxdq292bfPXRXzF3ZsM4VyWpDE6+TnIfXb+Mj65fVqjtIy/soTfhUxevHLmxpEnHyyOmkL9p3smS02Zy1bqlZZciaRw4Qp8i9h49wSMvtLFmyTy+/P9eKrucypkWwScvWsmqxXPLLkVTmIE+RXSc6Oa02dN5eW8H//uxHWWXUym9mfQmLJw7k3/3wTVll6MpzECfIs5ePI9nvvCxssuopP/+g21856lX+YPzzyq7FE1xzqFLY3D0RDffbW7l2vedxdL5s8suR1OcI3RpDL6/pZWjJ7q58ZJVnOjuGfXnBMHM6Y6vNDYGujQGP3i274auf7PpqTF/1v/65AVcf9GKMX+Opi4DXRqD//Sx8/jFawfH9BnNr+znx9vbWTxvZm2K0pRloEtjcPk5S8b8LPp//Y0nWb14Lh8+r7FGVWmqctJOKtHW1oNsefUAN1+2mmnThnoStVScgS6V6N4nX2HuzAaub3LuXGNnoEsl2Xv0BD98djfXX7SC+bNnlF2OKsBAl0py389fo6unl5svW112KaoIA10qwRs9vXzn56/yobVLOHfpaWWXo4rwKhepBD96/jfsOXyCP/+D9Rzr6i61loZpwazpPiO/Cgx0qQQ/3Np3Q9Id//cXJVcCs6ZP4+//wwc5d+npZZeiMTLQpRLcceVaLly1sOwyeOjZXby05wiL5s0quxTVgIEuleB9K87gfSvOKLWGru5eNj2+g3+5fhmLvEu1EvxSVJqiHn2xjX0dXXzyIpckrApH6NIU9b0tOwF47KV2nmjZW3I1w1uxcA63fMDFQ0ZioEtT1KwZDcyb2cB3m3eWXcqwOrt7aYjgpstW0+DjEYZloEtT1N2fvrDsEgr5w6/+E3NmNBjmBTiHLqluHTzWxfO7DnH5uYvLLmVSMNAl1a2nduwjEz5w7tgeUTxVGOiS6taTv97H3JkNXLBiQdmlTAoGuqS69UTLXi5evcj1VguylyTVpd8c6uTX7R18wPnzwgx0SXXpZzv6ro0f6xJ/U4mXLUqqS0+07APgu807+d6W1gk552mzpnPTZWezbP7sCTlfrRnokurSktNmMX/2dP72F69P2Dk7unr4P0+8zOc/upZbLl8z6ebuIzNLOXFTU1M2NzeXcm5JGsorezv40g9/ySMvtnFO4zz+2x+9lw+tbSy7rLeJiC2Z2TTUsUL//ETE1RGxPSJaIuKuIY6fERE/iIhnI2JbRNw61qIlaaKtXjKPb91yMd/6t0109yY3feuf+fff2ULrgWNll1bIiIEeEQ3A3cAGYD1wY0SsH9Tss8AvM/MC4ArgryLC53FKmpQ+8p5lPPwff5//8rHz+PH2Nj7614/x1Ud+RecbPWWXNqwiI/RLgJbM3JGZXcD9wHWD2iRwekQEcBqwHyh3XS1JGoPZMxq446q1PPKfr+CqdUv5q398iT974LmyyxpWkUBfDgx8HFtr/76Bvga8B9gFPAd8PjN7B39QRNwWEc0R0dze3j7KkiVp4ixfMIevf+YiViycQ09J3zkWVSTQh3rE2eA/1ceBZ4B3Ab8LfC0i5r/jhzI3ZWZTZjY1NtbXFw2SdDK9vcmew52cdcacsksZVpFAbwUGLmmygr6R+EC3Ag9knxbgZWBdbUqUpHLt7TjBGz3JuxbU9/XpRQL9aWBtRKzp/6LzBuChQW1eAz4CEBHLgHcDO2pZqCSVZffBToC6H6GPeGNRZnZHxB3Aw0ADcE9mbouI2/uPbwS+BNwbEc/RN0VzZ2bW95pWklTQ7kPHATjrjPoeoRe6UzQzNwObB+3bOOD9LuBjtS1NkurDrv4R+rsW1PcIfXLd1ypJJdh18Dizpk9j4dwZZZcyLANdkkaw+1An71owh75bbeqXgS5JI9h16Hjdz5+DgS5JI9p9sP6vQQcDXZKG1d3TS9uRzrq/Bh0MdEka1p4jJ+jN+r8GHQx0SRrW7oOT4xp0MNAlaViv9wd6vV+DDga6JA1r5/6+xS1WLjLQJWlS27n/OEtOm8ncmfW/BLOBLknD2HngGCsXzS27jEIMdEkaxmv7j7FyoYEuSZNad08vuw91ssoRuiRNbrsPddLTm5PiC1Ew0CXppH57hcskmXKp/69tJakkr/UH+rd/9ioPPjN45c23e89Zp3PLB9ZMRFknZaBL0kmsWjyX5Qvm8MzOg8O2O3i8ix9tazDQJaleXX7OEp6466oR233uvl/wXOvB8S9oBM6hS9IY7e84waJ5M8suw0CXpLHad7SLRfNmlV2GgS5JY7W/o4tF88pfb9RAl6QxyEwOHHOELkmT3pET3bzRkyx2Dl2SJrf9R7sA/FJUkia7/ccMdEmqBEfoklQR+zsMdEmqhDenXBafZqBL0qS2v6OL2TOm1cUSdQa6JI3BvqNdLJpb/ugcDHRJGpO2I500zp9ddhlAwUCPiKsjYntEtETEXSdpc0VEPBMR2yLisdqWKUn1qe3wCZadXv5dolAg0COiAbgb2ACsB26MiPWD2iwAvg78UWa+F/hk7UuVpPrTdqSTpfMnSaADlwAtmbkjM7uA+4HrBrX5NPBAZr4GkJlttS1TkurPie4eDhx7g2WnT54pl+XAzgHbrf37BjoPWBgRP4mILRFx81AfFBG3RURzRDS3t7ePrmJJqhPtR04ATKoRegyxLwdtTwcuAq4FPg78eUSc944fytyUmU2Z2dTY2HjKxUpSPdlz+M1Ar48RepELJ1uBlQO2VwCDV0ttBfZmZgfQERGPAxcAL9WkSkmqQ+1HOgEm1ZTL08DaiFgTETOBG4CHBrV5EPhQREyPiLnA7wEv1LZUSaovb43Q62PKZcQRemZ2R8QdwMNAA3BPZm6LiNv7j2/MzBci4kfAVqAX+GZmPj+ehUtS2dqOdDJ9WtTNjUWF7lXNzM3A5kH7Ng7a/kvgL2tXmiTVtz2HT9B4+iymTRvqq8aJ552ikjRKew531s0XomCgS9KotR85wdI6uUsUDHRJGrU9hztZVidfiIKBLkmj0vlG312iZzrlIkmT2+sHjwOwfOGckit5i4EuSaPw+oH+QF8wt+RK3mKgS9IoOEKXpIp4/cBxGqZF3TwLHQx0SRqVXQePc+b82UxvqJ8YrZ9KJGkSaT14nOUL6me6BQx0SRqV1w8cr6v5czDQJemUdff08pvDnY7QJWmy232ok57eZIUjdEma3F7Z1wHA6iXzSq7k7Qx0STpFr+ztC/Q1BrokTW6v7DvGnBkNdfWkRTDQJemUvbK3g7MXzyWiPha2eJOBLkmn6JV9HaxeXF/TLWCgS9Ip6elNdu4/ztlL6uehXG8y0CXpFOw6eJyunl7WOEKXpMntzUsWd+ztYFf/ExfrxfSyC5CkyWT5gjmsWjSXTY/vYNPjO1h35ulctW4pV65byvtXLij1YV2RmaWcuKmpKZubm0s5tySNRWbS0naUR19s49EX22h+9QA9vckZc2bw4fMauWrdUj58XiML582s+bkjYktmNg15zECXpLE5dPwNfvqrdh59sY3Htrezr6OLaQG/u3LBb0fv68+aX5PLHA10SZogvb3J1tcP8eiLbfz4xTaee/0QAGfOn82V6xq54t1LuWrdUmaMcmrGQJekkrQd7uQn2/tG7//UspejJ7r5rx9/N5+98txRfd5wge6XopI0jpbOn82nLl7Jpy5eSVd3L//iiw9zpLN7XM7lZYuSNEFmTp8G4/i0AANdkirCQJekijDQJakiCgV6RFwdEdsjoiUi7hqm3cUR0RMR19euRElSESMGekQ0AHcDG4D1wI0Rsf4k7f4CeLjWRUqSRlZkhH4J0JKZOzKzC7gfuG6Idp8Dvg+01bA+SVJBRQJ9ObBzwHZr/77fiojlwCeAjbUrTZJ0KooE+lBXTQ6+vfTLwJ2Z2TPsB0XcFhHNEdHc3t5esERJUhFF7hRtBVYO2F4B7BrUpgm4v//BM0uAayKiOzP/bmCjzNwEbIK+W/9HWbMkaQhFAv1pYG1ErAFeB24APj2wQWauefN9RNwL/HBwmEuSxteIgZ6Z3RFxB31XrzQA92Tmtoi4vf+48+aSVAcKPZwrMzcDmwftGzLIM/OWsZclSTpV3ikqSRVhoEtSRRjoklQRBrokVYSBLkkVYaBLUkUY6JJUEQa6JFWEgS5JFWGgS1JFGOiSVBEGuiRVhIEuSRVhoEtSRRjoklQRBrokVYSBLkkVYaBLUkUY6JJUEQa6JFWEgS5JFWGgS1JFGOiSVBEGuiRVhIEuSRVhoEtSRRjoklQRBrokVYSBLkkVYaBLUkUY6JJUEQa6JFVEoUCPiKsjYntEtETEXUMc/0xEbO1/PRkRF9S+VEnScEYM9IhoAO4GNgDrgRsjYv2gZi8DH87M84EvAZtqXagkaXhFRuiXAC2ZuSMzu4D7gesGNsjMJzPzQP/mU8CK2pYpSRpJkUBfDuwcsN3av+9k/hj4h6EORMRtEdEcEc3t7e3Fq5QkjahIoMcQ+3LIhhFX0hfodw51PDM3ZWZTZjY1NjYWr1KSNKLpBdq0AisHbK8Adg1uFBHnA98ENmTmvtqUJ0kqqsgI/WlgbUSsiYiZwA3AQwMbRMQq4AHgpsx8qfZlSpJGMuIIPTO7I+IO4GGgAbgnM7dFxO39xzcCXwAWA1+PCIDuzGwav7IlSYMVmXIhMzcDmwft2zjg/Z8Af1Lb0iRJp8I7RSWpIgx0SaoIA12SKsJAl6SKMNAlqSIMdEmqCANdkirCQJekijDQJakiDHRJqggDXZIqwkCXpIow0CWpIgx0SaoIA12SKsJAl6SKMNAlqSIMdEmqCANdkirCQJekijDQJakiDHRJqggDXZIqwkCXpIow0CWpIgx0SaoIA12SKsJAl6SKMNAlqSIMdEmqCANdkibQ1e89k3Vnnj4un10o0CPi6ojYHhEtEXHXEMcjIr7Sf3xrRFxY+1IlafL7yo3v51+9f/m4fPaIgR4RDcDdwAZgPXBjRKwf1GwDsLb/dRvwjRrXKUkaQZER+iVAS2buyMwu4H7gukFtrgO+nX2eAhZExFk1rlWSNIwigb4c2Dlgu7V/36m2ISJui4jmiGhub28/1VolScMoEugxxL4cRRsyc1NmNmVmU2NjY5H6JEkFFQn0VmDlgO0VwK5RtJEkjaMigf40sDYi1kTETOAG4KFBbR4Cbu6/2uVS4FBm7q5xrZKkYUwfqUFmdkfEHcDDQANwT2Zui4jb+49vBDYD1wAtwDHg1vErWZI0lBEDHSAzN9MX2gP3bRzwPoHP1rY0SdKpiL4sLuHEEe3Aq6WcvHaWAHvLLqKO2B9vZ3+8xb54u7H0x9mZOeRVJaUFehVERHNmNpVdR72wP97O/niLffF249UfPstFkirCQJekijDQx2ZT2QXUGfvj7eyPt9gXbzcu/eEcuiRVhCN0SaoIA12SKsJAL6DAAh+f6V/YY2tEPBkRF5RR50QYqS8GtLs4Inoi4vqJrG+iFemPiLgiIp6JiG0R8dhE1ziRCvxdOSMifhARz/b3R2XvKo+IeyKiLSKeP8nx2i8MlJm+hnnR97iDXwO/A8wEngXWD2pzObCw//0G4Odl111WXwxo9yh9dxdfX3bdJf9uLAB+Cazq315adt0l98efAX/R/74R2A/MLLv2ceqP3wcuBJ4/yfFrgH+g72m1l9YiNxyhj2zEBT4y88nMPNC/+RR9T5usoiKLnQB8Dvg+0DaRxZWgSH98GnggM18DyMwq90mR/kjg9IgI4DT6Ar17YsucGJn5OH1/vpOp+cJABvrICi3eMcAf0/evbhWN2BcRsRz4BLCR6ivyu3EesDAifhIRWyLi5gmrbuIV6Y+vAe+h7/HazwGfz8zeiSmv7pxqtoyo0MO5prhCi3cARMSV9AX6B8e1ovIU6YsvA3dmZk/fIKzSivTHdOAi4CPAHOBnEfFUZr403sWVoEh/fBx4BrgKOAf4x4j4aWYeHufa6lHhbCnKQB9ZocU7IuJ84JvAhszcN0G1TbQifdEE3N8f5kuAayKiOzP/bkIqnFhFF3/Zm5kdQEdEPA5cAFQx0Iv0x63A/8y+SeSWiHgZWAf888SUWFdqvjCQUy4jG3GBj4hYBTwA3FTRkdebRuyLzFyTmaszczXwPeBPKxrmUGzxlweBD0XE9IiYC/we8MIE1zlRivTHa/T9b4WIWAa8G9gxoVXWj5ovDOQIfQRZbIGPLwCLga/3j0y7s4JPlivYF1NGkf7IzBci4kfAVqAX+GZmDnkZ22RX8PfjS8C9EfEcfVMOd2ZmJR+rGxH3AVcASyKiFfgiMAPGb2Egb/2XpIpwykWSKsJAl6SKMNAlqSIMdEmqCANdkirCQJekijDQJaki/j+RPh8xBMMo7AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(precisions, recalls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "freelance-water",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "acoustic-water",
   "metadata": {},
   "outputs": [],
   "source": [
    "precisions, recalls, thresholds = precision_recall_curve(y_test, decision_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "behind-tokyo",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(151,)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precisions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "opened-brazilian",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(151,)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recalls.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "injured-forestry",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150,)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thresholds.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frank-pavilion",
   "metadata": {},
   "source": [
    "1. 现象：thresholds 中的元素个数，比 precisions 和recalls 中的元素个数少 1 个;\n",
    "2. 原因：当 precision = 1、recall = 0 时，不存在 threshold；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "pediatric-funeral",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoq0lEQVR4nO3dd3hUZfrG8e+TTkJCKhBIQugdBAOoIEpRigXFgujaFVFc68+yq+sW3V3dtey6FkRFBAs2REQEFURQQAFFegkgJJQktBTSk/f3xxk0QEKGMMmZM/N8rmuuKedk5maY3BzOnPO+YoxBKaWU8wXYHUAppZRnaKErpZSP0EJXSikfoYWulFI+QgtdKaV8RJBdLxwfH29SU1PtenmllHKklStX7jPGJFS3zLZCT01NZcWKFXa9vFJKOZKI7Khpme5yUUopH6GFrpRSPkILXSmlfIQWulJK+QgtdKWU8hG1FrqITBaRbBFZW8NyEZHnRSRdRFaLSG/Px1RKKVUbd7bQpwDDT7B8BNDedRkHvHzqsZRSSp2sWo9DN8YsEpHUE6wyCphqrHF4l4lItIgkGmP2eCrkUbLWw9qP4Iw7ICKuXl5CKW+1c38hH/2YiQ577WxpqbEM7FDtuUGnxBMnFrUEMqrcz3Q9dlyhi8g4rK14UlJS6vZq+7fA4qeh6yVa6MrvvPPDTiZ+sxURu5OoUzH+nLZeW+jVfbSq3XwwxkwCJgGkpaXVbRMjNMq6Ls6r048r5WRlFZVEhASy7m8n2guq/JUnjnLJBJKr3E8CdnvgeasX5ir0kvx6ewmlvFWlMQTo5rmqgScKfRZwnetolzOA3Hrbfw6/baEXHYSK8povSvmgykpDQIAWuqperbtcRORd4FwgXkQygT8DwQDGmInAHGAkkA4UAjfWV1gAGsVY1zPHW5ea9LsdRjxZr1GUamgVxhCoha5q4M5RLmNrWW6ACR5LVJuIeBj1EuSdYK/Ohk9gyzwtdGWb3MIyggKFiFDPDmhaUYnuclE1sm343FPS65oTLxdgwRNQnAthTRokklJHfLJqFw98sJryykriG4fSvEkYkWFBNAoOIjwkkPCQQBqFBHJacjTndmhKk/Bgt5+7stIQqOd3qxo4s9Brk3iadb13DaQOsDWK8h/GGF5auJV/z9tEt5ZRnNMhgZz8ErLySigoKefA4SKKSsspLK3gwOFS3vjuFwBaxYXTsVkkybHhJMU0Ijkm/Nfbx27hVxhDoG6hqxr4aKH3tK6//Q9sXWBrFK/QvId13L6qN2UVlTz68VreW5HBqNNa8K/LexAaFHjC9Zdu3c+aXbmsycxla04Bi7bkUFxWedR6sREhJMc0Iik2nOSYcDZn5SNa6KoGvlnojZtCcj/Y9rV18WeVFRAUBp0vgoCaC0bVXV5xGRPe/pHFW/bx+8HtuO+8DrWWbnBgAAM7JBx1cokxhn0FpWQeLCTjYBEZBwrJPFhE5sFC1u3K5Yt1eymrMPRI0t2Iqnq+WegAN39hdwLvsPJN+PQuOLQTYlvbncarlFVUsuKXgyzclM236fsIChASmzSiY/NIBrSPp3FoEI1Dg4gIDSIiNLDaLe7dh4q4acpy0rML+NdlPbiyT3I1r+QeESEhMpSEyFB6pcQct7yy0pCVX+zxL1qV79BPhq9r2tm6ztmohQ5k5RWzcFM2Czfl8O2WfeSXlBMcKKS1iiUoUNiwN4+56/by3/lbjvvZ4EChY/NI+qTGkhQTTnSjYJ6au5Gi0gqm3NiXAe3j6zV7gOsfHKVqooXu6xI6WtdZa6Ht4BOvGxDkc7tlyisqWZVxiK83ZfP1xhzW77GGjEhsEsaFPVswqGMCZ7WztsaP2JyVz97cYg6XlFNQUs7hknIOl1aQV1zGyl8O8t7yDApLKwBoGd2Iabf3o2PzSFv+fEpVJXaN2paWlmZWrFhhy2v7nWe7Ql5m7etFJsKE7x1/qOe+ghIWbc7h6005LNqcQ25RGYEBwumtYhjUsSmDOiXQsVlknb9cNMZwsLCMXQeLSI0PJzLM/cMOlTpVIrLSGJNW3TLdQvcHl74MmctPvE7pYVj8DPzwKgz8v4bJ5SFrMnPJLy7j++0HWLgpm9W7cjEG4huHcl6XZgzq2JQB7eNp0sgzxSsixEaEEBsR4pHnU8pTtND9QeuB1qU2e9fA0heg320Q6oxdCOnZ+Vz0wrcAiECv5GjuG9qBQZ2a0iUxSsc9UX5FC139ZuCD8PpQWP46DLjH7jRuefv7nQA8ekFnRvdO0q1m5df0JGL1m+Q+1henS/4HpYV2p6nV4ZJyPlyRyajTWnDL2W20zJXf00JXRzvjDijcBzuX2p2kVjNX7SK/pJzrzmxldxSlvIIWujrakWET9m22N4cb3lq2k64touhdzUk4SvkjLXR1tIgEa8z5nE12J6nVtpwC+reL17FNlHLRL0XV0UQgviNkb7AOZTwZweE05OzF5ZWG4EAtc6WO0EJXx2vaCVZOgX+0OLmf63m1dcx7AzDGUFFpCArQ/2QqdYQWujre2fdDbFvgJM4iXv0B7P6x3iIdq7zSyqZb6Er9RgtdHS86BfrfdXI/k7cbfnq7fvJUY+OefABi9FBFpX6l/19VnhHVAkrzoTivQV5u0uJtNA4N4qKeJ7lbSCkfplvoyjOiWlrX8/4AYdEn//OBIXDW7yE8ttZVMw4UMmfNHm4e0JooHRhLqV9poSvPaNELIprCupl1+/nSAsDA0L/Uuurr325HgBv7p9bttZTyUVroyjPi2sIDx08K4bb3r4Plk60vZGsYGOxQYSlPfr6R6cszuOL0JJ3sQalj6D505R3OugtKcuHHacctMsYw48dMhjzzDR+szOS2gW3426huNoRUyrvpFrqyXWFpOX/8NpAHm/Si6ZIXCOo7DgKtj+a2nAIenbmWJVv30yslmrcu7U7nxCibEyvlnbTQle1+3HGImat2UxAwmNdCnuGppx5jf4cxNGkUzJtLdhAaHMATl3Tj6r4pOr65Uiegha5st31fAQC3j5vA3pkLuDNvKheuPY3txRFc1LMFf7qwM00jw2xOqZT300JXttu27zDhIYH0bhWLXDMJJvZnQde5HBjxMnGNQ+2Op5Rj6Jeiylal5ZUs3bqfNgkR1qiJCR1g4APIuo+I2/W13fGUchTdQle2evLzjWzcm8/L1/T+7cH+98DaGTD7Htg2qvYnCQiCvuMgRie6UP5NC13ZZu7aPUz+bjs3nJXKiO6Jvy0ICoFRL8KHN8DP7574SQzW4Y4RCY6ZB1Wp+qKFrmyxLaeABz5YTc/kaP44svPxKySdDvesqf2Jyorh783AVHo+pFIO49Y+dBEZLiKbRCRdRB6uZnkTEflURH4WkXUicqPnoypfkXGgkGte+57goABevLoXIUGe+CrnJIb6VcpH1bqFLiKBwIvAeUAmsFxEZhlj1ldZbQKw3hhzkYgkAJtE5G1jTGm9pFaOtftQEWNfXUZhaQXv3noGSTHhp/aER2ZIKi2EwgOn8DwB0Cj61LIoZTN3drn0BdKNMdsARGQ6MAqoWugGiBRrcsfGwAGg3MNZlcNl5RVz9avLyC0s4+1b+9GlhQfO+JRAq4wXP21dTsXQv+p+eOVo7hR6SyCjyv1MoN8x67wAzAJ2A5HAGGOO36kpIuOAcQApKSl1yascau7avTw6cw1FpRVMvbkfPZKiPfPEgUEwdjoc/OXUnmfjZ7DwSeh+OTRJ8kg0pRqaO4Ve3bnWx+6wHAasAgYDbYEvRWSxMeao2Q6MMZOASQBpaWm609MPHDxcyp9nrWPWz7vp2iKKZ67sSafmHh6LpcMwDzzHcHihD8x/HEa/curPp5QN3Pk2KhNIrnI/CWtLvKobgRnGkg5sBzp5JqJyqnnr9nLec4v4fO0e7juvAzMn9Pd8mXtKTCs48w5YPR12NdzcqEp5kjuFvhxoLyKtRSQEuApr90pVO4EhACLSDOgIbPNkUOUsz8/fwm3TVtI0MpRZdw7griHtCQ708hOTB9wH4fHwxZ/sTqJUndS6y8UYUy4idwLzgEBgsjFmnYiMdy2fCDwOTBGRNVi7aB4yxuyrx9zKi72/IoNnv9zM6N4teeqyHt5f5EeERUHfW2HhP6H0MIRE2J1IqZPi1olFxpg5wJxjHptY5fZu4HzPRlNOtGhzDn+csYaz28c7q8yPiGxuXRcd0kJXjuOw3zblzXILy7jj7R9p3yySl67p7bwyh98muC4+ZGcKperEgb9xylvtOlREQUk5dw1uR2RYsN1x6ubIyUVFh+xMoVSdaKErjykprwAgLCTQ5iSnICLBul45xdqPrpSDaKErjykus84lC/XI2Cw2adoFBtwLaz6AV86B3avsTqSU2xz8m6e8TfGRLfRgB2+hi8DQv8B1n1hb6K8NhW//A5U6mqPyflroymNKfGEL/Yg258Dt30HHEfDVn2HqxZC7y+5USp2QD/zmKW+RW2QNrhke4iPD7IfHwpVT4eIXrLNHXz4Lti+yO5VSNdJCVx6Rnl3AlCU7aBoZSqvYUxwS15uIQO9rYfxiCI2ERf+2O5FSNdJCV6fEGMO0ZTsY+uw3bNiTx3ldmhEQUN14bg4X1xY6XQgZy6Fch/lX3kkLXdVZUWkF97//M3+aufbXx0b3bmljonqW2h/Ki2C3Dt6lvJOP7OxUDS2vuIyrXlnGhr153Du0A78f3I78knKaNHLoCUXuSDnLuv5lMaScYW8WpaqhW+iqTp6YvZ6Ne/N4/fo07h7anoAA8e0yB4iIg6ZdYf0sqKywO41Sx9FCVydtwcYs3l+Rye3ntmVwp2Z2x2lYA+6Fvavhe50EQ3kfLXTlNmMM7y3fyZ3v/ETHZpHcNaS93ZEaXvfLof0wWPA4HNhudxqljqKFrtxy4HApt01byUMfraFnUjRTbupDaJCDzwitKxG48FlrcupP7wajMykq76FfiqpaZRwoZPTLS8gtLOORkZ25eUBr3zw00V1NkuD8v8Hse+G930GjmOPXkQDodxs069rw+ZTf0kJXtfpifRY5+SV8MqE/PZOj7Y7jHXrfABk/wLZvql9ekGVtzV/03waNpfybFrqq1ZrMQyQ2CdMyryogAC6dWPPyN0ZC1vqGy6MUug9duWFzVgGdE6PsjuEszbpC9nodpVE1KC10Vat9BSU0jQy1O4azNOsKpQWQu9PuJMqPaKGrE8rKK2b/4VIStNBPTrNu1vXOZfbmUH5FC12d0N9mrycwQLisd5LdUZylRS+IawdLX9RDG1WD0UJXNfpmcw6frd7DnYPakRofYXccZwkIhP53W2eVbvva7jTKT2ihq2oVl1Xw2CdraRMfwW3ntLE7jjP1GAORifDtc3YnUX5CC11V69VF29ixv5DHL+nmn2eEekJQKJxxhzXL0fbFdqdRfkALXVXrszV7OLNNHP3bxdsdxdlOvwGiW8HbV1ijNCpVj7TQFQDTlu3gzH/OZ9KirezJLWLj3nz6t4uzO5bzhUXBLV9B827w/rWw6Gn9klTVGy10RU5+CU99vpGisgr+MWcjZ/5zAQC9W1UzRok6eY2bwvWzodvl1iiNH4+H8hK7UykfpIWueH7+ForLKvjo9rN4b9xvM/F0TWxiYyofExwGl70Ggx6B1dPhzYvh8D67Uykfo4Xu53YfKmL68p1c2SeZtgmN6dcmjheu7sXYvik0CffxGYgamgic8yBc/gbsWWUNv6uUB+ngXH6qtLyS91Zk8MKCLQjChEHtfl12YY8WXNijhY3pfFy30bD6fTi4w+4kyse4tYUuIsNFZJOIpIvIwzWsc66IrBKRdSJSw5iiyhvMXr2bwc8s5E8z15IcE8674/rRMrqR3bH8S1gUlOTanUL5mFq30EUkEHgROA/IBJaLyCxjzPoq60QDLwHDjTE7RaRpPeVVpyjzYCG/f/cnOjePYsqN3TinQwIifjxZhV3CmkBxnt0plI9xZ5dLXyDdGLMNQESmA6OAqoM9Xw3MMMbsBDDGZHs6qDp1lZWGN777BWNg4u9OJyUu3O5I/is0CkryrEMY9R9U5SHu7HJpCWRUuZ/peqyqDkCMiCwUkZUicl11TyQi40RkhYisyMnJqVtiVSc/Zxzi0pe+4/Vvt3NB90Qtc7vFpIKphKx1didRPsSdLfTqNh+OPTMiCDgdGAI0ApaKyDJjzOajfsiYScAkgLS0ND27ogHs3F/Im0t/YfJ324lvHMp/xpzGqNP0C0/bdRgGCGyaY510pJQHuFPomUBylftJwO5q1tlnjDkMHBaRRUBPYDPKNkvS93HbWyvJLy4HYMH95xAZpocieoXGTSGpD2ycbR3KqJQHuLPLZTnQXkRai0gIcBVw7KAUnwBni0iQiIQD/YANno2qTsbXG7O5bvIPNI8KA2BQxwQtc2/TaSTs+RlyM+1OonxErVvoxphyEbkTmAcEApONMetEZLxr+URjzAYRmQusBiqB14wxa+szuKpZbmEZD360mvbNInnvtjMoK68kNFhHTPQ6HS+Ar/4Cmz6HvrfanUb5ALdOLDLGzAHmHPPYxGPu/xv4t+eiqbowxvDQR6s5cLiUN27oQ5RulXuvhA7WrEYbP9NCVx6hp/77mOfnpzN33V4eHt6Jbi11LBav13Ek/PItFOtJRurUaaH7iMpKw98/W89zX21mdK+W3HJ2a7sjKXd0ugAqy2DLl3YnUT5AC91HfLkhi1cXb+e6M1vx9BU99exPp0jqA+Hx1tEuSp0iLXSHK6+o5OuN2bz0dTpRYUE8dmEXAgK0zB0jIBC6u2Yzyt5odxrlcDraokNtyylg6tIdzF69m30FpUSHB/Pg8E4EBeq/0Y4z8AFY9Q588Sj87kO70ygH00J3mPziMv63IJ3J324nQIQhnZtyaa+WnNuxKSFBWuaOFBEH5zxgFXr6V9BuqN2JlENpoTvMmFeWsX5PHlemJfHAsE4kRIbaHUl5Qt9xsPx1mPcItD4XAvVXU5083aRzkAOHS1m/J4/7z+vAvy7vqWXuS4JC4by/Qc5GWPJfu9Moh9JCdwhjDE99vhEROLejDjfvkzpfBF0vhQV/hx1L7E6jHEgL3QGy84u5+c0VvLcig9vPaUv3JD1hyCeJwEXPQ0wr+PBmnURanTQtdC+361ARF/3vW75L38efL+rC/53f0e5Iqj6FRcEVU6BwP8wYB5WVdidSDqKF7sUyDxZyw+QfKCyt4OM7+nNj/9Z6jLk/SOwJw/8JW+fD0v/ZnUY5iBa6l1qdeYiLX/iOvXnFvHpdGl1aRNkdSTWktJug7RBY+iJUVtidRjmEFroXKigpZ8I7P9IoOJCZE/pzRps4uyOphiYCva+Fgixr8C6l3KCF7oWm/7CTjANFPDfmNNomNLY7jrJLh+EQ0hjWfGB3EuUQWuheZMUvBxg/bSX/mLOBXinR9G0da3ckZafgRtZojBtmQXmJ3WmUA2ihe4nv0vdx+cSlLN22n3ED2zLp2jS7Iylv0P0Ka6z0ZS/bnUQ5gJ5fbKP84jKWbN3Pos05zN+QTWhQAN89PJjGofrXolzaDYXOF8P8v0KzbtBex3lRNdPmaGCVlYZFW3J4a9lOFm7KprzSEBESyJlt47jmjFZa5upoInDpRHh9GHx4E9w6H+Lb251KeSltjwaUk1/C1a8uY0t2AfGNQ7h5QGsGdWpK75QYHSlR1SwkAsa+A5POhXfHwi1fQaNou1MpL6SF3oD+MGM1GQcL+c+Y0xjZPVFLXLkvOgWunAZTL4b3r4XLJkPjBLtTKS+jjdJACkvLWbgph+vPTOWSXi21zNXJS+1vjfWyYym82MeaFMMYu1MpL6Kt0kBm/LiL8krD+V2b2R1FOVmva2D8YojvADNvh2mXwIFtdqdSXkILvQGUVVTyxfos4iJCOL2VHluuTlHTznDjXBj5NGSuhJfOgu/+CxXldidTNtNCr2el5ZXcNGU5izbncP1ZqXbHUb4iIAD63goTvoe2g+HLx+DVQbB7ld3JlI200OvZXz9dx+It+/jn6O7cNUQPN1Me1qQlXPU2XDnVGvdl8jDIWm93KmUTLfR6tGlvPm9/v5ObB7RmbN8Uu+MoXyUCXUbBbYutsV9mjNOhAvyUFno92ZNbxIMfrSY4UJgwqJ3dcZQ/iGwGo16ArDWw4Am70ygbaKHXk7/OWs+WrHyev6oXsREhdsdR/qLjCDj9RljyP9i+2O40qoFpodeTVRmHGNa1OSO6J9odRfmbYX+H2Dbw8XgoKbA7jWpAWuj1YMnWfezNK6Z3SrTdUZQ/ComAC56BvEzY8oXdaVQDcuvUfxEZDvwXCAReM8Y8WcN6fYBlwBhjzIceS+nFDhWWsnZXHpuz8tmSnc/mrALW7MqlVVw4o3q1tDue8letB0J4HGz8DLqNtjuNaiC1FrqIBAIvAucBmcByEZlljFlfzXpPAfPqI6i3KSwtZ9rSHTz31WaKy6yZ2aPDg+nQNJKxfZKZMKgdUWHBNqdUfisg0Nqfvv5TKC+FIP0exx+4s4XeF0g3xmwDEJHpwCjg2INdfw98BPTxaEIvY4zhua+28MZ328kvLmdo52bc2D+V9s0ak9A4FBGxO6JSlo4XwE9vwY5vrZOPlM9zp9BbAhlV7mcC/aquICItgUuBwfhwoRtjmLp0B8/P38LQzk0ZN7AtfVJjtMSVd2o7CILCIH2+FrqfcKfQq2urY4d4+w/wkDGm4kTlJiLjgHEAKSnOOdHm4OFSZvy0i+k/7GRLdgH928XxwtW9CQsOtDuaUjULbgRx7WB/ut1JVANxp9AzgeQq95OA3ceskwZMd5V5PDBSRMqNMTOrrmSMmQRMAkhLS/P6cT8rKw2PzVrL+8szKa2opGdyNP++vAeX9U4iIEC3ypUDxLaGnE12p1ANxJ1CXw60F5HWwC7gKuDqqisYY1ofuS0iU4DZx5a5E23NKeCtZTvp3y6ORy/oQufEKLsjKXVyYtvA5nlQWWF9Uap8Wq2FbowpF5E7sY5eCQQmG2PWich41/KJ9ZyxwRSXVbDzQCHbcg6zfd9hPlm1C4D7z++oZa6cKbYNVJRCxg/Q6ky706h65tZx6MaYOcCcYx6rtsiNMTeceqyGdaiwlHvfW8XCzTlHTQCT2CSMJ0d3p1dytG3ZlDolbYdARFN480I4+//g7Pv1EEYf5vdziu7JLWLspGXsPlTMbQPb0jkxkjbxjUmNDydSjyNXThedbI2ZPvdh+OZJ60SjS16ExJ52J1P1wG8LvaLSkJ5dwOOz15OdX8I7t/YjLVVnE1I+KDwWRk+CLpfA7Hvg1cEw4D4Y+IBurfsYvyv0rLxiHvl4DUu27qewtAKAJy7ppmWufF+nkZByBsz9Ayz6l2tr/SVocZrdyZSH+FWh7zpUxPWTfyDzYCFj0pI5LSWaXskxpMZH2B1NqYYRHgujX4Gul8Cn98Dk4XDrfGjW1e5kygN8vtArKg0/bD/A9OU7mb16DwBPX9GDS3sl2ZxMKRt1HAEtesErA+G9a2HcQgjTI7mczmcL3RjDBysz+dfcTewrKCEiJJAbz0rlxgGtaRndyO54Stkvsjlc/ga8eRF8MsGal1SHsXA0nyz0jAOFTFu2g0mLttEnNYa/jerKuR0TCA/xyT+uUnWX2h+G/hm+fAyWvQRnTrA7kToFPtVwqzIO8ccZa1i/Jw+ApJhGvDfuTD1NX6kTOesu68SjLx+DlmmQ0q/2n1FeySdmLFqSvo9hzy3ikhe/Y2tOAb8fbE3K3Ld1rJa5UrURsY52iWoJM26B4ly7E6k6cvwW+p7cIia88yPBgQE8OLwjF3RPpFVcBFf1TSFOJ2dWyj1hTWD0q/DGcJjzoHUkjHIcxxf6Ix+vJa+4nLl3n037ZpG/Pq5ffCp1klL6WScbffMUdDgful1mdyJ1khy9y2XJ1n0s2JjN3UPaH1XmSqk6GvigtR999r1wKKP29ZVXcXShf/rzboIChFvPbmN3FKV8Q2AQXPaqNdzuO2Pg8H67E6mT4MhCP1xSztPzNvHuDxlc1TeZRiE6zrNSHhPbBsZMgwNbYerFWuoO4rhC//Tn3XT98zxe+Dqdi3u24NELutgdSSnf03YwXPWONX3d1FFQeMDuRMoNjiv0pJjfvux84tJuOq+nUvWl3RCr1Pdthjcv1lJ3AMcVeq+UGF659nTW/nUYUTpeuVL1q90QGPuuVepTL9Zj1L2c4wodYFjX5jQOdfwRl0o5Q7shMPYdyN4I06+B8hK7E6kaOLLQlVINrN1QGPUi/LIYZt4BlZV2J1LV0M1cpZR7eo6BvF0w/68Q1QLOf9zuROoYWuhKKfcNuNcq9SXPW2O/nDHe7kSqCi10pZT7RGDEvyB/L8x9CA7nwKA/QoAebeYNdB+6UurkBATCZa9Dr9/B4qetM0qLDtqdSqGFrpSqi+AwuPgFuOBZ2LYQJg2CrHV2p/J7WuhKqboRgT43ww2fQVkhvDYU1s6wO5Vf00JXSp2alH5w2yJo3h0+vBEW/dvuRH5LC10pdeoim8P1s6H7lbDgCdj4md2J/JIWulLKM4JC4OL/QYte8PF42L/V7kR+RwtdKeU5wWFw5VTrSJj3fgelh+1O5Fe00JVSnhWdYh3WmL0BPr0bjLE7kd/QQldKeV67ITDoEVjzAfzwqt1p/IYWulKqfpx9P3QYDvP+ADu/tzuNX3Cr0EVkuIhsEpF0EXm4muXXiMhq12WJiPT0fFSllKMEBMClE6FJEnxwPRRk253I59Va6CISCLwIjAC6AGNF5Nh537YD5xhjegCPA5M8HVQp5UCNYuDKadbQAB/eBBXldifyae5sofcF0o0x24wxpcB0YFTVFYwxS4wxRwZzWAYkeTamUsqxEnvAhf+xxlLXk47qlTuF3hLIqHI/0/VYTW4GPq9ugYiME5EVIrIiJyfH/ZRKKWc7bSy0Hwar3tajXuqRO4Uu1TxW7d+IiAzCKvSHqltujJlkjEkzxqQlJCS4n1Ip5Xwdh0NuBuxPtzuJz3Kn0DOB5Cr3k4Ddx64kIj2A14BRxpj9nomnlPIZbQdb1+nz7c3hw9wp9OVAexFpLSIhwFXArKoriEgKMAO41hiz2fMxlVKOF5MKsW1h6wK7k/isWmcsMsaUi8idwDwgEJhsjFknIuNdyycCjwFxwEsiAlBujEmrv9hKKUdqcy6sfs+aZDpAT4PxNLemoDPGzAHmHPPYxCq3bwFu8Ww0pZTPadYFSgsgfw80OdGxFaou9J9IpVTDiWtvXe/fYm8OH6WFrpRqOPGuQt+nhV4f3NrlopRSHhGZCCGN4ae3ICTCOvIlsrndqXyGFrpSquGIQP+7rREYZ95uPdasm1Xs7YZAypkQFGpvRgcTY9NZW2lpaWbFihW2vLZSymaVlZC1FrbOt45L37kMKssgOBxSB0DbIVbBx7Wz/hFQvxKRlTUdRahb6EqphhcQYI3xktgDBtwLJQXwy7e/FfyWL6z1mqRAu8FWwbc5B8Ka2Jvby2mhK6XsF9rYGhqg43Dr/sFfrGLfugDWfAQrp4AEQlIfa8u9x5XWiUrqKLrLRSnl3SrKIHO5q+Dnw+5VENUC7l4Ngf63TXqiXS562KJSyrsFBkOrs2DIn2DcQhgzDfJ2wea5difzOlroSiln6TACIlvAisl2J/E6WuhKKWcJDILTr7f2rx/8xe40XkULXSnlPL2utQ5nXPay3Um8iha6Usp5mrSE3tfB9xNhw6d2p/EaWuhKKWca/hS0PB0+Hg/ZG+xO4xW00JVSzhQcBmNcY8JMvxqKDtb+Mz5OC10p5VxRLeDKaXAoAz68CSor7E5kKy10pZSzpfSDC56xjnpZ/KzdaWylha6Ucr7Tr4fuV8DCf0Km/56BroWulPINI5+2dsF8dIs12Jcf0kJXSvmGRtEwehIc2gFzH7I7jS38b2QbpZTvanUWDLgPFj8NB3dATCtrCN7oZGiSbF1HtbTGh/FBWuhKKd9y7sNQWmDtS9/yJRRkHbOCWFPhVS35JskQnQJNkqzboY1tiX6qtNCVUr4lMBhGPPXb/bJia3TG3Azr8Maq17tWwPpPrNmSqmoUU6Xkk62ir1r84XFeOZOSFrpSyrcFh0FcW+tSncoKayv+17LfCbmZ1u39W2HbQmuL/6jnDHdtzSdV2cqvsmsnMtGWsdq10JVS/i0g0Do6JqoF0O/45cZYZ6HmZlhFf1TxZ8Ce1VC47+ifEddzHrVLJxnaD4OoxHr7o2ihK6XUiYhAeKx1SexZ/TplRa6y31lll45rK3/HUsj7EEwFxHeECd/X2+4aLXSllDpVwY0gvr11qU5FOfw4BT67H3YutY7GqQd6HLpSStW3wCDoORZCo2Dlm/X2MlroSinVEEIirOEJ1s+st5EhtdCVUqqhnH49lBfD6g/q5em10JVSqqEk9oRul1tfsNYDtwpdRIaLyCYRSReRh6tZLiLyvGv5ahHp7fmoSinlAy5/HbpfXi9PXWuhi0gg8CIwAugCjBWRLsesNgJo77qMA3TmVqWUamDubKH3BdKNMduMMaXAdGDUMeuMAqYayzIgWkTq7+h5pZRSx3Gn0FsCGVXuZ7oeO9l1EJFxIrJCRFbk5OScbFallFIn4E6hV3dKk6nDOhhjJhlj0owxaQkJCe7kU0op5SZ3Cj0TSK5yPwnYXYd1lFJK1SN3Cn050F5EWotICHAVMOuYdWYB17mOdjkDyDXG7PFwVqWUUidQ61guxphyEbkTmAcEApONMetEZLxr+URgDjASSAcKgRvrL7JSSqnquDU4lzFmDlZpV31sYpXbBpjg2WhKKaVOhlhdbMMLi+QAO2x5cYgH9tW6lr2ckBGckdMJGcEZOZ2QEZyRs64ZWxljqj2qxLZCt5OIrDDGpNmd40SckBGckdMJGcEZOZ2QEZyRsz4y6lguSinlI7TQlVLKR/hroU+yO4AbnJARnJHTCRnBGTmdkBGckdPjGf1yH7pSSvkif91CV0opn6OFrpRSPsKvCl1ErhCRdSJSKSJpVR5PFZEiEVnlukw80fPYkdG17A+uSUQ2icgwuzIeS0T+IiK7qrx/I+3OdERtk7N4AxH5RUTWuN67FXbnOUJEJotItoisrfJYrIh8KSJbXNcxXpjRqz6PIpIsIl+LyAbX7/bdrsc9/l76VaEDa4HRwKJqlm01xpzmuoxv4FxVVZvRNanIVUBXYDjwkmvyEW/xXJX3b07tq9c/Nydn8RaDXO+dNx07PQXrs1bVw8B8Y0x7YL7rvp2mcHxG8K7PYzlwvzGmM3AGMMH1OfT4e+lXhW6M2WCM2WR3jhM5QcZRwHRjTIkxZjvWuDl9Gzad47gzOYuqgTFmEXDgmIdHAW+6br8JXNKQmY5VQ0avYozZY4z50XU7H9iANV+Ex99Lvyr0WrQWkZ9E5BsROdvuMNVwaxIRG93pmk92st3/Da/C29+zIwzwhYisFJFxdoepRbMjI6m6rpvanKcm3vh5RERSgV7A99TDe+lzhS4iX4nI2mouJ9oy2wOkGGN6AfcB74hIlJdldGsSkfpSS+aXgbbAaVjv5TMNlasWtr5nJ6G/MaY31q6hCSIy0O5ADueVn0cRaQx8BNxjjMmrj9dwa7RFJzHGDK3Dz5QAJa7bK0VkK9ABqJcvqOqSEZsnEXE3s4i8Csyu5zjucsTEK8aY3a7rbBH5GGtXUXXf83iDLBFJNMbscc0bnG13oGMZY7KO3PaWz6OIBGOV+dvGmBmuhz3+XvrcFnpdiEjCkS8YRaQN0B7YZm+q48wCrhKRUBFpjZXxB5szASBHTwh+KdYXu97AnclZbCUiESISeeQ2cD7e8/5VZxZwvev29cAnNmaplrd9HkVEgNeBDcaYZ6ss8vx7aYzxmwvWX24m1tZ4FjDP9fhlwDrgZ+BH4CJvy+ha9giwFdgEjLD7/aySaxqwBljt+pAm2p2pSraRwGbX+/aI3XmqydfG9bn72fUZ9JqMwLtYuyzKXJ/Jm4E4rCMytriuY70wo1d9HoEBWLv6VgOrXJeR9fFe6qn/SinlI3SXi1JK+QgtdKWU8hFa6Eop5SO00JVSykdooSullI/QQldKKR+hha6UUj7i/wHFRgbg9vHXlQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thresholds, precisions[:-1])\n",
    "plt.plot(thresholds, recalls[:-1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "false-dodge",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVyklEQVR4nO3de5Bc5Xnn8e+jG+iKJDQIXQAJWwIJDGssA3btxuw6NhfjEBxXDN6YDZtEy5ZJ8k+2cFxre112eZ3yJutgg1WEYEzZBalUiIOxCI5DMLZZbEQC4iohS1gaS4xGF3QfpNE8+0c3MAyjmTNS95zuo++naqq6+7xz+sep4afT7zl9TmQmkqT2N6bsAJKkxrDQJakiLHRJqggLXZIqwkKXpIoYV9Ybz5o1KxcsWFDW20tSW3riiSe2ZWbHYMtKK/QFCxawatWqst5ektpSRPzySMuccpGkirDQJakiLHRJqggLXZIqwkKXpIoYttAj4o6I2BoRzxxheUTEzRGxLiJWR8QFjY8pSRpOkT30O4HLhlh+ObCo/rMc+Maxx5IkjdSwhZ6ZjwA7hhhyFXBX1jwGTI+IOY0KeCQ79h3ki/c/x9bdPc1+K0lqC42YQ58HbOr3vLP+2ltExPKIWBURq7q7u4/pTb/16Evc/pMN7O45dEzrkaSqaEShxyCvDXrXjMy8LTOXZeayjo5Bv7layP6Dvdz1/17i15fM5u2nTD3q9UhSlTSi0DuB0/o9nw9sbsB6j+hvV3Wyc/8hbnjfmc18G0lqK424lst9wI0RcQ9wEbArM7c0YL2D6j3cx1/9eD1nnzqVudMnsvmVA0e9rlOnnciYMYN9wJCk9jNsoUfE3cAlwKyI6AQ+B4wHyMwVwErgCmAdsB+4vllhAZ7qfIXOnbUSf++XHzqmdS3/tTP59BVLGhFLkko3bKFn5rXDLE/gkw1LNIx3zJvO1659J/sP9h71On6ybjvfe2oz58yd1sBkklSu0i6fe7QmjBvDh8+fe9S/39eX/PVPNvD2U6Zw5XlHvx5JajXH3Vf/v//0FtZ27eWP3r+Isc6fS6qQtttDP1a3/Ms6AP7m8Y387apNw4xWq1k8eyqfuXJp2TGklnTcFfoFZ8xg4oSx7D94uOwoGoHDfcnqzl1s2dXDZ8oOI7Wo467Qv3T1O8qOoKPwnZ/9ktWdu/iTDy4uO4rUso67OXS1nz09h/iLH6zlwgUzufScU8uOI7Ws424PXe3nGw//gu37DvKxd8/ggWdebui6x40J3ndWByeMG9vQ9UplsNDV8n784jYAbn34F01Z/1/89vl85IL5TVm3NJosdLW87/zBRWx5pbGXSU6SP777Sbbu6eH9S2Y3dN1SWSx0tbxpJ45n2qnjG7rOHz7XxZquPXz+N87hpImNXbdUFg+K6rhz6HAfX3rgec7smMzHLzq97DhSw1joOu7c/fONrO/ex59evoTxY/1fQNXhX7OOK7t7DvHVH77IxWfO5NeXnFJ2HKmhLHQdV275l3Xs3H+Q//mhpUR4LR9Vi4Wu48amHfv55k9f4up3zuPceSeVHUdqOM9y0XHjz3+whoO9fRzuS7608vmy4xTywaWzWbZgZtkx1CYsdB03eg71MXH8WH7wbFfZUYZ1OJODvX3s6TlkoaswC13HjRWfeFfZEQr7yx++yP/94Vp+5+Izyo6iNuIcutRidvcc4q9/sp4PLJ3NOXOd61dx7qFLLeZbP32J3T29fPj8uazv3lt2nIaaO30iJ473QmjNYqFLLeYfntoMwB/d/W8lJ2m8DyydzV9dt6zsGJVloUst5mvXvpO1XXvKjtFQr+w/xOfue5aFsyaXHaXSLHSpxSyZM40lc6aVHaOh/ubxjQBced6ckpNUmwdFJTXd/au3cPrMSbzDL3Q1lYUuqam2732VR3+xnSvPm+PlFprMQpfUVA888zKH+5Irz5tbdpTKs9AlNdX9qzdzZsdklsyZWnaUyrPQJTXN1t09/GzDDq48b67TLaPAs1wkNc3Kp7eQCWRy7792lh0HgJOnnMD7FneUHaMpLHRJTbNp5wEAbn5oXclJ3uzJz36A6ZMmlB2j4Sx0SU3z6SuWcN17WucCY195cA0/Wttd2RuDFyr0iLgM+EtgLHB7Zn55wPKTgG8Dp9fX+X8y85sNziqpzYwdE5xxcut8O/RXrxxg6ZxplZ3PH/agaESMBW4BLgeWAtdGxNIBwz4JPJeZ5wOXAH8eEdX7PCOpbR3uS17Ysoelc6v1Ldz+iuyhXwisy8z1ABFxD3AV8Fy/MQlMjdo/e1OAHUBvg7NK0lHb/MoBDhw6zLcefYnvPLZx2PG/9a75/O+PvGMUkjVOkUKfB2zq97wTuGjAmK8D9wGbganAxzKzb+CKImI5sBzg9NNPP5q8knRU5px0In96+dm8cuDQsGO/89gv2bq7ZxRSNVaRQh9ssikHPL8UeBL4T8DbgH+KiB9n5u43/VLmbcBtAMuWLRu4DklqmnFjx/Df3ve2Ycf19SXf/OkGFrThlSGLfLGoEzit3/P51PbE+7seuDdr1gEbgLMbE1GSRs/Lu3voOdTXlpf6LVLojwOLImJh/UDnNdSmV/rbCLwfICJmA2cB6xsZVJJGw4Zt+wA4sw0Lfdgpl8zsjYgbgQepnbZ4R2Y+GxE31JevAL4A3BkRT1OborkpM7c1MbckNcX6eqEv7KhgoQNk5kpg5YDXVvR7vBn4YGOjSdLo29C9j4njxzJ76ollRxkxL84lSf1s2LaXBbMmM2ZM+335yEKXpH5e2r6fhbMmlR3jqFjoklR36HAfG3fsb8szXMBCl6TXbdqxn8N9ycJZU8qOclQsdEmqe+2URffQJanNtfM56GChS9Lr1m/bx/RJ45kxuT0vFmuhS1Lduq1723bvHCx0SQIgM1nbtYezTp1adpSjZqFLEtC951Ve2X+IxbMtdElqa2u79gJwloUuSe1tTdceABZZ6JLU3l7s2sPMyROYNaU9z3ABC12SgNoe+qJTplC7NXJ7stAlHfcykxe79rJo9hT6+rLwT2Zr3Umz0PXQJanKduw7yN5Xe/n2Yxv59mMbC//e6TMn8fCfXNIyl9q10CUd92ZOnsAXf/Nctu89WPh3Hnqhi7Vde2mlGRoLXdJxLyL4nYvPGNHvrN26h909vS015+4cuiQdha5dPcyedkLZMd7EQpeko/Dy7h5OndZa9x210CVphPr6kq27X2X2SRa6JLW1HfsPcvBwn3voktTuXt7VA2ChS1K769pdL3SnXCSpvb1soUtSNXTt6mFMQMcUT1uUpLb28u4eZk05gXFjW6tCWyuNJLWBl3e/2nLTLWChS9KI1b4laqFLUlvLTDa/coC57bqHHhGXRcSaiFgXEZ86wphLIuLJiHg2In7U2JiS1Bp2H+hlz6u9nDZzUtlR3mLYqy1GxFjgFuADQCfweETcl5nP9RszHbgVuCwzN0bEKU3KK0ml2rRzPwDzZ7ReoRfZQ78QWJeZ6zPzIHAPcNWAMR8H7s3MjQCZubWxMSWpNXTuPADA/BkTS07yVkUKfR6wqd/zzvpr/S0GZkTEwxHxRERcN9iKImJ5RKyKiFXd3d1Hl1iSStRZ30M/rU330Ae7evvAG+mNA94FfAi4FPhMRCx+yy9l3paZyzJzWUdHx4jDSlLZOnceYOoJ45g2sfXuD1QkUSdwWr/n84HNg4zZlpn7gH0R8QhwPrC2ISklqUV07tzPvBkTW+pORa8psof+OLAoIhZGxATgGuC+AWP+AfgPETEuIiYBFwHPNzaqJJWvc+eBljwgCgX20DOzNyJuBB4ExgJ3ZOazEXFDffmKzHw+Iv4RWA30Abdn5jPNDC5Joy0z6dx5gIvPPLnsKIMqNAmUmSuBlQNeWzHg+VeArzQumiS1lm17D7L31V4WnNyae+h+U1SSCnpp+z4AFsyaXHKSwVnoklTQhu5aoZ85a0rJSQZnoUtSQRu272P82GDu9Na7jgtY6JJU2Evb9nHazEktdx3017RmKklqQRu27WPhya05fw4WuiQV0teXvLR9Hwtb9IAoWOiSVEjXnh56DvW17BkuYKFLUiHr62e4uIcuSW1ubdceABbNbs1TFsFCl6RC1nbtYcak8XRMOaHsKEdkoUtSAWu79rJ49tSWvMriayx0SRpGZrL25T2cderUsqMMyUKXpGFs2dXDnld7WTTbQpektramfkD0LAtdktrb2pdrhb64hc9wAQtdkoa1pmsPs6edwPRJE8qOMiQLXZKG8cKWPSxu8ekWsNAlaUiv9h7mxa17OHfeSWVHGZaFLklDeLFrL4cOJ+fObf1CL3RPUUk6Xj3zq10AzJl+Itv3vnrM6xsTwYzJzZmLt9AlaQgb6vcR/citjzZsnZ//jXP4L+9d0LD1vcZCl6Qh/O57FzB/+kSyQev7/PeeY8uungat7c0sdEkawpyTJvKJ9yxo2Pq++P3nG7augTwoKkkVYaFLUkVY6JJUERa6JFWEhS5JFWGhS1JFWOiSVBEWuiRVRKFCj4jLImJNRKyLiE8NMe7dEXE4Ij7auIiSpCKGLfSIGAvcAlwOLAWujYilRxj3Z8CDjQ4pSRpekT30C4F1mbk+Mw8C9wBXDTLuD4G/A7Y2MJ8kqaAihT4P2NTveWf9tddFxDzgamDFUCuKiOURsSoiVnV3d480qyRpCEUKPQZ5beCFx74K3JSZh4daUWbelpnLMnNZR0dHwYiSpCKKXG2xEzit3/P5wOYBY5YB90QEwCzgiojozczvNiKkJGl4RQr9cWBRRCwEfgVcA3y8/4DMXPja44i4E7jfMpek0TVsoWdmb0TcSO3slbHAHZn5bETcUF8+5Ly5JGl0FLrBRWauBFYOeG3QIs/M3z32WJKkkfKbopJUERa6JFWEhS5JFWGhS1JFWOiSVBEWuiRVhIUuSRVhoUtSRVjoklQRFrokVYSFLkkVYaFLUkVY6JJUERa6JFWEhS5JFWGhS1JFWOiSVBEWuiRVhIUuSRVhoUtSRVjoklQRFrokVYSFLkkVYaFLUkVY6JJUERa6JFWEhS5JFWGhS1JFWOiSVBEWuiRVRKFCj4jLImJNRKyLiE8Nsvw/R8Tq+s+jEXF+46NKkoYybKFHxFjgFuByYClwbUQsHTBsA/C+zDwP+AJwW6ODSpKGVmQP/UJgXWauz8yDwD3AVf0HZOajmbmz/vQxYH5jY0qShlOk0OcBm/o976y/diS/Bzww2IKIWB4RqyJiVXd3d/GUkqRhFSn0GOS1HHRgxH+kVug3DbY8M2/LzGWZuayjo6N4SknSsMYVGNMJnNbv+Xxg88BBEXEecDtweWZub0w8SVJRRfbQHwcWRcTCiJgAXAPc139ARJwO3At8IjPXNj6mJGk4w+6hZ2ZvRNwIPAiMBe7IzGcj4ob68hXAZ4GTgVsjAqA3M5c1L7YkaaAiUy5k5kpg5YDXVvR7/PvA7zc2miRpJPymqCRVhIUuSRVhoUtSRVjoklQRFrokVYSFLkkVYaFLUkVY6JJUERa6JFWEhS5JFWGhS1JFWOiSVBEWuiRVhIUuSRVhoUtSRVjoklQRFrokVYSFLkkVYaFLUkVY6JJUERa6JFWEhS5JFWGhS1JFWOiSVBEWuiRVhIUuSRVhoUtSRVjoklQRFrokVYSFLkkVYaFLUkUUKvSIuCwi1kTEuoj41CDLIyJuri9fHREXND6qJGkowxZ6RIwFbgEuB5YC10bE0gHDLgcW1X+WA99ocE5J0jCK7KFfCKzLzPWZeRC4B7hqwJirgLuy5jFgekTMaXBWSdIQihT6PGBTv+ed9ddGOoaIWB4RqyJiVXd390izSlLbu+ycUzn71KlNWfe4AmNikNfyKMaQmbcBtwEsW7bsLcslqepuvvadTVt3kT30TuC0fs/nA5uPYowkqYmKFPrjwKKIWBgRE4BrgPsGjLkPuK5+tsvFwK7M3NLgrJKkIQw75ZKZvRFxI/AgMBa4IzOfjYgb6stXACuBK4B1wH7g+uZFliQNpsgcOpm5klpp939tRb/HCXyysdEkSSPhN0UlqSIsdEmqCAtdkirCQpekioja8cwS3jiiG/jlMa5mFrCtAXFGSzvlNWtzmLV52invsWQ9IzM7BltQWqE3QkSsysxlZecoqp3ymrU5zNo87ZS3WVmdcpGkirDQJaki2r3Qbys7wAi1U16zNodZm6ed8jYla1vPoUuS3tDue+iSpDoLXZIqoi0KvcBNqq+q35z6yfodkf59GTnrWYbM2m/cuyPicER8dDTzDcgw3Ha9JCJ21bfrkxHx2TJy9ssz7LatZ34yIp6NiB+NdsZ+OYbbtv+j33Z9pv63MLNFs54UEd+LiKfq27W0q6kWyDojIv6+3gc/j4hzy8hZz3JHRGyNiGeOsDwi4ub6f8vqiLjgmN80M1v6h9ole38BnAlMAJ4Clg4YM4U3jgecB7zQqln7jXuI2hUsP9qqWYFLgPvL/hsYQd7pwHPA6fXnp7Rq1gHjPww81KpZgU8Df1Z/3AHsACa0aNavAJ+rPz4b+Ocytmv9/X8NuAB45gjLrwAeoHbHt4uBnx3re7bDHvqwN6nOzL1Z30LAZAa5/d0oKXJDbYA/BP4O2Dqa4QYomrVVFMn7ceDezNwIkJllbd+RbttrgbtHJdlbFcmawNSICGo7TzuA3tGNCRTLuhT4Z4DMfAFYEBGzRzdmTWY+Qm1bHclVwF1Z8xgwPSLmHMt7tkOhF70B9dUR8QLwfeC/jlK2gYbNGhHzgKuBFZSr0HYF3lP/qP1ARJwzOtEGVSTvYmBGRDwcEU9ExHWjlu7Nim5bImIScBm1f+DLUCTr14El1G4r+TTwx5nZNzrx3qRI1qeAjwBExIXAGdRuidmKCv+dFNUOhV70BtR/n5lnA78JfKHZoY6gSNavAjdl5uHmxxlSkaz/Su26EecDXwO+2+xQQyiSdxzwLuBDwKXAZyJicbODDaLQ32zdh4GfZuZQe3LNVCTrpcCTwFzg3wFfj4hpzY01qCJZv0ztH/UnqX0S/jfK+TRRxEj+TgopdMeiko3oBtSZ+UhEvC0iZmXmaF+op0jWZcA9tU+vzAKuiIjezPzuqCR8w7BZM3N3v8crI+LWkrYrFL9Z+bbM3Afsi4hHgPOBtaMT8U05iv7NXkN50y1QLOv1wJfr05rrImIDtfnpn49OxNcV/Zu9HmoHHYEN9Z9WNKJuK6SsAwYjOLAwDlgPLOSNAyHnDBjzdt44KHoB8KvXnrda1gHj76S8g6JFtuup/bbrhcDGMrbrCPIuoTZ/Og6YBDwDnNuKWevjTqI2xzq5jG06gu36DeB/1R/Prv//NatFs06nfsAW+ANqc9SlbNt6hgUc+aDoh3jzQdGfH+v7tfweeha7SfVvAddFxCHgAPCxrG+xFszaEgpm/Sjw3yOil9p2vaaM7Vo0b2Y+HxH/CKwG+oDbM3PQU8bKzlofejXwg6x9oihFwaxfAO6MiKeplc9NWcKntIJZlwB3RcRhamc8/d5o53xNRNxN7UyxWRHRCXwOGA+vZ11J7UyXdcB+6p8sjuk9S/r/U5LUYO1wUFSSVICFLkkVYaFLUkVY6JJUERa6JFWEhS5JFWGhS1JF/H9AYKsUnnsFWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(precisions, recalls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decreased-consortium",
   "metadata": {},
   "source": [
    "1. 外层曲线对应的模型更优；或者称与坐标轴一起包围的面积越大者越优。\n",
    "2. P - R 曲线也可以作为选择算法、模型、超参数的指标；但一般不适用此曲线，而是使用 ROC 曲线；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cloudy-tradition",
   "metadata": {},
   "source": [
    "## ROC Curve\n",
    "\n",
    "|Reality/Prediction |0|1|\n",
    "|----|----|----|\n",
    "|0|9978<br>TN|12<br>FP|\n",
    "|1|2<br>FN|8<br>TP|\n",
    "\n",
    "$$FPR = \\frac{FP}{TN+FP}$$\n",
    "\n",
    "$$TPR = \\frac{TP}{TP+FN}$$\n",
    "\n",
    "1. TPR（True Positive Rate）：真正率；被预测为正的正样本结果数 / 正样本实际数：TPR = TP /（TP + FN）；\n",
    "2. TNR（True Negative Rate）：真负率；被预测为负的负样本结果数 / 负样本实际数：TNR = TN /（TN + FP） ；\n",
    "3. FPR（False Positive Rate）：假正率；被预测为正的负样本结果数 /负样本实际数：FPR = FP /（TN + FP） ；\n",
    "4. FNR（False Negative Rate）：假负率；被预测为负的正样本结果数 / 正样本实际数：FNR = FN /（TP + FN） ；\n",
    "\n",
    "###### 关系：\n",
    "1. 随着阈值 threshold 的增大，FPR 和 TPR 都逐渐减小；\n",
    "2. FPR 和 TPR 称正相关关系，FPR 越高，TPR 相应的也越高；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "legal-jordan",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "suburban-judges",
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = datasets.load_digits()\n",
    "X = digits.data\n",
    "y = digits.target.copy()\n",
    "y[digits.target==9] = 1\n",
    "y[digits.target!=9] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "swiss-message",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "banned-talent",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yasmine/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "log_reg = LogisticRegression()\n",
    "log_reg.fit(X_train, y_train)\n",
    "decision_scores = log_reg.decision_function(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "prospective-element",
   "metadata": {},
   "outputs": [],
   "source": [
    "from playML.metrics import FPR, TPR\n",
    "\n",
    "fprs = []\n",
    "tprs = []\n",
    "thresholds = np.arange(np.min(decision_scores), np.max(decision_scores), 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "fixed-presence",
   "metadata": {},
   "outputs": [],
   "source": [
    "for threshold in thresholds:\n",
    "    # dtype='int'：将数据类型从 bool 型转为 int 型；\n",
    "    y_predict = np.array(decision_scores >= threshold, dtype='int')\n",
    "    fprs.append(FPR(y_test, y_predict))\n",
    "    tprs.append(TPR(y_test, y_predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "therapeutic-workshop",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQBUlEQVR4nO3df6zddX3H8eeL21YQkAJtQVuglbRCk8GUCzLnDxxuUkzGTFwCGtlQ15CJ8U/Ykukf/KMxS4wB7BpGnH9M/lCi6KpsiVGmDEeJWCgMVovSWga3KChVqLf3vT/uRS+3t73f0nPvuedzn4/kJvf7o/e+P7nNs9+ee873pKqQJA2+Y/o9gCSpNwy6JDXCoEtSIwy6JDXCoEtSIxb16xsvW7asVq9e3a9vL0kD6f77799bVcunO9a3oK9evZqtW7f269tL0kBK8tNDHfMhF0lqhEGXpEYYdElqhEGXpEYYdElqxIxBT3JbkqeTPHSI40nyuSQ7kmxL8qbejylJmkmXK/QvAJcd5vgGYO3Ex0bg80c/liTpSM34PPSqujvJ6sOccgXwxRq/D++9SZYmeW1VPdmrIdUbT//qBb70g10cGBvr9yjSgja8+hTevm7a1wYdlV68sGglsGvS9u6JfQcFPclGxq/iOfPMM3vwrXUkNn93J7d+73GSfk8iLWzXvuPseRv06fIw7btmVNVmYDPA8PCw76wxh8bGii0PPsm7zl3BrX91Yb/HkTQLevEsl93AGZO2VwF7evB11UM/3PUse557gfec99p+jyJplvQi6HcCV0882+Vi4DkfP599VcXogbHOH/+27UmWDB3Dpeee1u/RJc2SGR9ySfIl4BJgWZLdwCeBxQBVtQnYAlwO7AB+DVwzW8Pq9/78pu/z4M+eO6I/865zV/CaYxfP0kSS+q3Ls1yumuF4AR/t2UTqZOfI81xw1slc0vEXKwm857zXzfJUkvqpb7fP1dF74xlL+dila/s9hqR5wpf+S1IjvEKf5J4de/n+j/f2e4xO9h/wxUGSXs6gT/KZf3+UHz7xLIuOmf+vvDkmYd1pJ/Z7DEnziEGfZKzgHeuW8y8fuqjfo0jSEfMxdElqxIK+Qv/tgTHG6vd3IKjybgSSBteCDfrDe37JX9z8/YN+uXjpOSv6NJEkHZ0FG/Qnn/sN+w+M8cGLz+L0k4793f5L3tD7O6BJ0lxYsEF/yV8Or+K8VUv7PYYkHTV/KSpJjTDoktQIgy5JjTDoktQIgy5JjWj+WS6jB8amvZHVi6Pe3EpSW5oO+v7RMd7yqW+z9/kXD3nO0ADciEuSumg66C+MHmDv8y9y6TkruGjNKQcdP/HYxZxz+mv6MJkk9V7TQX/JH519Kh952+v7PYYkzSp/KSpJjWjmCv2fv/c4Tz77m5ft8119JC0kTQT9ud/8lhu/8TBLho5h8dDLf8l50nGLWes7+0haAJoI+kv3Mf+7y8/hmj9e0+dpJKk/fAxdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQP7tMWq4p/u3sn/PfeCd06UJAY46D955td86pv/w3GLh1g8FE49fgnrfAGRpAVsYIP+2FO/AuBf/+bNvPHMk/s8jST1X6fH0JNcluTRJDuS3DDN8ZOSfD3Jj5JsT3JN70d9uf+dCLov65ekcTMGPckQcDOwAVgPXJVk/ZTTPgo8XFXnA5cA/5hkSY9nfZnHnnqelUuP44RXDex/MiSpp7pcoV8E7KiqnVW1H7gduGLKOQWcmCTACcDPgdGeTjrFzr3Pc/aKE2bzW0jSQOkS9JXArknbuyf2TXYTcC6wB3gQ+HhVHfTUkyQbk2xNsnVkZOQVjjxu/+gYxy8ZOqqvIUkt6RL06d50s6Zsvxt4AHgd8IfATUkOem+3qtpcVcNVNbx8+fIjHFWSdDhdgr4bOGPS9irGr8Qnuwa4o8btAB4HzunNiJKkLroE/T5gbZI1E7/ovBK4c8o5TwCXAiQ5DXgDsLOXg0qSDm/Gp4hU1WiS64C7gCHgtqranuTaieObgBuBLyR5kPGHaK6vqr2zOLckaYpOz/mrqi3Alin7Nk36fA/wZ70dTZJ0JLw5lyQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1YiCDvm33szz21PP9HkOS5pWBDPqX798NwIWrT+nzJJI0fwxk0KvglOOX8KG3run3KJI0bwxk0CVJBzPoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjegU9CSXJXk0yY4kNxzinEuSPJBke5Lv9nZMSdJMFs10QpIh4GbgT4HdwH1J7qyqhyedsxS4Bbisqp5IsmKW5pUkHUKXK/SLgB1VtbOq9gO3A1dMOef9wB1V9QRAVT3d2zElSTPpEvSVwK5J27sn9k22Djg5yXeS3J/k6um+UJKNSbYm2ToyMvLKJpYkTatL0DPNvpqyvQi4AHgP8G7gH5KsO+gPVW2uquGqGl6+fPkRDytJOrQZH0Nn/Ir8jEnbq4A905yzt6r2AfuS3A2cDzzWkyklSTPqcoV+H7A2yZokS4ArgTunnPM14G1JFiV5NfBm4JHejipJOpwZr9CrajTJdcBdwBBwW1VtT3LtxPFNVfVIkm8B24Ax4Naqemg2B5ckvVyXh1yoqi3Alin7Nk3Z/gzwmd6NJkk6Er5SVJIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaMZBBf2bfi5x47KJ+jyFJ88pABv2xp55n7YoT+z2GJM0rAxf0/aNj/GTvPtaddkK/R5GkeWXggv6TZ/YxOlasO80rdEmabOCC/vjefQC8fvnxfZ5EkuaXgQv66IEC4NjFQ32eRJLml4ELuiRpegZdkhph0CWpEQZdkhrRKehJLkvyaJIdSW44zHkXJjmQ5H29G1GS1MWMQU8yBNwMbADWA1clWX+I8z4N3NXrISVJM+tyhX4RsKOqdlbVfuB24IppzvsY8BXg6R7OJ0nqqEvQVwK7Jm3vntj3O0lWAu8FNvVuNEnSkegS9Eyzr6Zsfxa4vqoOHPYLJRuTbE2ydWRkpOOIkqQuutyDdjdwxqTtVcCeKecMA7cnAVgGXJ5ktKq+OvmkqtoMbAYYHh6e+o+CJOkodAn6fcDaJGuAnwFXAu+ffEJVrXnp8yRfAL4xNeaSpNk1Y9CrajTJdYw/e2UIuK2qtie5duK4j5tL0jzQ6W1/qmoLsGXKvmlDXlV/ffRjSZKOlK8UlaRGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJakSnoCe5LMmjSXYkuWGa4x9Ism3i454k5/d+VEnS4cwY9CRDwM3ABmA9cFWS9VNOexx4R1WdB9wIbO71oJKkw+tyhX4RsKOqdlbVfuB24IrJJ1TVPVX1i4nNe4FVvR1TkjSTLkFfCeyatL17Yt+hfBj45nQHkmxMsjXJ1pGRke5TSpJm1CXomWZfTXti8k7Gg379dMeranNVDVfV8PLly7tPKUma0aIO5+wGzpi0vQrYM/WkJOcBtwIbquqZ3ownSeqqyxX6fcDaJGuSLAGuBO6cfEKSM4E7gA9W1WO9H1OSNJMZr9CrajTJdcBdwBBwW1VtT3LtxPFNwCeAU4FbkgCMVtXw7I0tSZqqy0MuVNUWYMuUfZsmff4R4CO9HU2SdCR8pagkNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1IjBi7op590LJf/wemc8KpF/R5FkuaVTkFPclmSR5PsSHLDNMeT5HMTx7cleVPvRx13wVknc8sHLuB1S4+brW8hSQNpxqAnGQJuBjYA64GrkqyfctoGYO3Ex0bg8z2eU5I0gy5X6BcBO6pqZ1XtB24HrphyzhXAF2vcvcDSJK/t8aySpMPoEvSVwK5J27sn9h3pOSTZmGRrkq0jIyNHOqsk6TC6BD3T7KtXcA5VtbmqhqtqePny5V3mkyR11CXou4EzJm2vAva8gnMkSbOoS9DvA9YmWZNkCXAlcOeUc+4Erp54tsvFwHNV9WSPZ5UkHcaMT+auqtEk1wF3AUPAbVW1Pcm1E8c3AVuAy4EdwK+Ba2ZvZEnSdDq9OqeqtjAe7cn7Nk36vICP9nY0SdKRyHiL+/CNkxHgp6/wjy8D9vZwnEHgmhcG17wwHM2az6qqaZ9V0regH40kW6tquN9zzCXXvDC45oVhttY8cPdykSRNz6BLUiMGNeib+z1AH7jmhcE1LwyzsuaBfAxdknSwQb1ClyRNYdAlqRHzOujz6Y015kqHNX9gYq3bktyT5Px+zNlLM6150nkXJjmQ5H1zOd9s6LLmJJckeSDJ9iTfnesZe63D3+2Tknw9yY8m1jzQrzhPcluSp5M8dIjjve9XVc3LD8ZvM/Bj4PXAEuBHwPop51wOfJPxuz1eDPyg33PPwZrfApw88fmGhbDmSed9m/FXLL+v33PPwc95KfAwcObE9op+zz0Ha/574NMTny8Hfg4s6ffsR7HmtwNvAh46xPGe92s+X6EvxDfWmHHNVXVPVf1iYvNexu9sOci6/JwBPgZ8BXh6LoebJV3W/H7gjqp6AqCqBn3dXdZcwIlJApzAeNBH53bM3qmquxlfw6H0vF/zOeg9e2ONAXKk6/kw4//CD7IZ15xkJfBeYBNt6PJzXgecnOQ7Se5PcvWcTTc7uqz5JuBcxm+9/SDw8aoam5vx+qLn/ep0c64+6dkbawyQzutJ8k7Gg/7WWZ1o9nVZ82eB66vqwPjF28DrsuZFwAXApcBxwH8lubeqHpvt4WZJlzW/G3gA+BPgbOA/kvxnVf1ylmfrl573az4HfSG+sUan9SQ5D7gV2FBVz8zRbLOly5qHgdsnYr4MuDzJaFV9dU4m7L2uf7f3VtU+YF+Su4HzgUENepc1XwN8qsYfYN6R5HHgHOC/52bEOdfzfs3nh1wW4htrzLjmJGcCdwAfHOCrtclmXHNVramq1VW1Gvgy8LcDHHPo9nf7a8DbkixK8mrgzcAjczxnL3VZ8xOM/4+EJKcBbwB2zumUc6vn/Zq3V+i1AN9Yo+OaPwGcCtwyccU6WgN8p7qOa25KlzVX1SNJvgVsA8aAW6tq2qe/DYKOP+cbgS8keZDxhyOur6qBva1uki8BlwDLkuwGPgkshtnrly/9l6RGzOeHXCRJR8CgS1IjDLokNcKgS1IjDLokNcKgS1IjDLokNeL/ATiKZ/j3+oV1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(fprs, tprs)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attractive-survival",
   "metadata": {},
   "source": [
    "###### 分析：\n",
    "\n",
    "1. ROC 曲线下方与图形边界围成的面积，作为衡量模型优劣的标准，面积越大，模型越优；\n",
    "2. 可以是同样算法不同超参数所得的不同模型，也可以是不同算法所得的不同模型；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "affecting-blowing",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "fprs, tprs, thresholds = roc_curve(y_test, decision_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "loose-tuition",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9823319615912208"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "roc_auc_score(y_test, decision_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "complimentary-abuse",
   "metadata": {},
   "source": [
    "## Confusion Matrix in Muti-class Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "indonesian-relations",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "X = digits.data\n",
    "y = digits.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "bored-florist",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "ranging-snapshot",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yasmine/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "log_reg = LogisticRegression()\n",
    "log_reg.fit(X_train, y_train)\n",
    "log_reg.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "social-dance",
   "metadata": {},
   "source": [
    "+ 多分类问题，精准率、召回率、阈值的计算，需要修改 average 参数：average = 'micro'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "clinical-logic",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predict = log_reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "selective-naples",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9485396383866481"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "\n",
    "precision_score(y_test, y_predict, average='micro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "former-correlation",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfm = confusion_matrix(y_test, y_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "middle-suicide",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[149,   0,   1,   0,   0,   1,   0,   0,   0,   0],\n",
       "       [  0, 137,   0,   0,   0,   0,   2,   0,   2,   0],\n",
       "       [  0,   1, 139,   2,   0,   0,   0,   0,   0,   0],\n",
       "       [  0,   0,   0, 138,   0,   6,   0,   0,   2,   1],\n",
       "       [  0,   1,   0,   0, 124,   0,   7,   1,   1,   0],\n",
       "       [  0,   1,   1,   0,   0, 136,   0,   1,   0,   2],\n",
       "       [  1,   1,   0,   0,   1,   0, 141,   0,   0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0, 143,   2,   1],\n",
       "       [  0,   9,   4,   0,   0,   4,   0,   1, 113,   2],\n",
       "       [  1,   0,   0,   4,   0,   3,   0,   2,   5, 144]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cfm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "spiritual-morocco",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAKkElEQVR4nO3dz4td9RnH8c+ndxJ0kiZKEgIm0olQbI1QIkOJDggmLtoquunCgkLdzKbVKIJoN/4DIroowhDrxqCLmEWRohb8Ad2EziSCTsaCRJtEo04lNTIJTJx5uri3kGTS3DPN+c65J8/7BULm5uTrw828OefenPsdR4QAXN1+0PQAAMojdCABQgcSIHQgAUIHEiB0IIHGQrf9C9v/sP2J7aeamqMq2zfaftf2jO1p23uanqkK2x3bh22/0fQsVdi+zvZ+2x/3nuvbm56pH9uP974nPrL9qu1rmp7pYo2Ebrsj6Y+SfinpFkm/sX1LE7Msw/eSnoiIn0raKel3LZhZkvZImml6iGV4QdKbEfETST/TgM9ue4ukRyWNRsStkjqSHmh2qqWaOqP/XNInEXE0IuYlvSbp/oZmqSQiTkbEod6vv1P3G3BLs1Ndnu2tku6RtLfpWaqwvU7SnZJekqSImI+Ifzc6VDVDkq61PSRpWNIXDc+zRFOhb5F0/LyvT2jAozmf7RFJOyQdbHiUfp6X9KSkxYbnqOomSbOSXu693Nhre03TQ11ORHwu6VlJxySdlPRtRLzd7FRLNRW6L/FYK+7Ftb1W0uuSHouI003P87/YvlfS1xEx1fQsyzAk6TZJL0bEDklzkgb6/Rvb16t7NbpN0g2S1th+sNmplmoq9BOSbjzv660awMudi9lepW7k+yLiQNPz9DEm6T7bn6n70miX7VeaHamvE5JORMR/r5T2qxv+ILtb0qcRMRsR5yQdkHRHwzMt0VTof5f0Y9vbbK9W982LPzc0SyW2re5rx5mIeK7pefqJiKcjYmtEjKj7/L4TEQN3pjlfRHwp6bjtm3sP7ZZ0pMGRqjgmaaft4d73yG4N4BuIQ038TyPie9u/l/SWuu9S/ikippuYZRnGJD0k6UPbH/Qe+0NE/KW5ka5Kj0ja1zsBHJX0cMPzXFZEHLS9X9Ihdf9l5rCkiWanWsp8TBW4+nFnHJAAoQMJEDqQAKEDCRA6kEDjodseb3qG5WjbvBIzr4RBn7fx0CUN9BN0CW2bV2LmlTDQ8w5C6AAKK3LDzMaNG2NkZKTSsbOzs9q0aVOlY6em2vT5DKAZEbHkQ2NFboEdGRnR5ORk7et2byUuo9TabbzzkOfi6sOlO5AAoQMJEDqQAKEDCRA6kECl0Nu2BzuAC/UNvaV7sAM4T5Uzeuv2YAdwoSqht3oPdgDVQq+0B7vtcduTtidnZ2evfDIAtakSeqU92CNiIiJGI2K06r3rAFZGldBbtwc7gAv1/VBLS/dgB3CeSp9e6/2QAn5QAdBS3BkHJEDoQAKEDiRA6EAChA4kUGRzSNtFNgcreccdN/mU1+l0iq29sLBQZN02znypzSE5owMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kECrtnu2l+xiW5tvvvmmyLolt5EutV0w2o3tnoGkCB1IgNCBBAgdSIDQgQQIHUiA0IEE+oZu+0bb79qesT1te89KDAagPkMVjvle0hMRccj2DyVN2f5rRBwpPBuAmvQ9o0fEyYg41Pv1d5JmJG0pPRiA+izrNbrtEUk7JB0sMg2AIqpcukuSbK+V9LqkxyLi9CV+f1zSeI2zAahJpQ+12F4l6Q1Jb0XEcxWO50MtPXyoBSvt//pQi7t1vSRppkrkAAZPldfoY5IekrTL9ge9/35VeC4ANer7Gj0i/iap3DUzgOK4Mw5IgNCBBAgdSIDQgQQIHUigVbvAtlGpG3EkacOGDUXWHR4eLrLumTNniqxbUqfTKbb24uJi7WtGBLvAAlkROpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQQKu2ey7589FLPA+lTU9PF1l3+/btRdZdu3ZtkXUlaW5urtjapZT6nmO7ZyApQgcSIHQgAUIHEiB0IAFCBxIgdCCByqHb7tg+bPuNkgMBqN9yzuh7JM2UGgRAOZVCt71V0j2S9pYdB0AJVc/oz0t6UtJiuVEAlNI3dNv3Svo6Iqb6HDdue9L2ZG3TAahFlTP6mKT7bH8m6TVJu2y/cvFBETEREaMRMVrzjACuUN/QI+LpiNgaESOSHpD0TkQ8WHwyALXh39GBBIaWc3BEvCfpvSKTACiGMzqQAKEDCRA6kAChAwkQOpAAu8AW1sbdZb/66qsi627evLnIulK5742Sf3+dTqf2NRcWFtgFFsiK0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IoNgusG3bsbWNu7W2befT06dPF1lXktatW1ds7bZhF1ggKUIHEiB0IAFCBxIgdCABQgcSIHQggUqh277O9n7bH9uesX176cEA1Geo4nEvSHozIn5te7Wk4YIzAahZ39Btr5N0p6TfSlJEzEuaLzsWgDpVuXS/SdKspJdtH7a91/aawnMBqFGV0Ick3SbpxYjYIWlO0lMXH2R73Pak7cmaZwRwhaqEfkLSiYg42Pt6v7rhXyAiJiJiNCJG6xwQwJXrG3pEfCnpuO2bew/tlnSk6FQAalX1XfdHJO3rveN+VNLD5UYCULdKoUfEB5K4JAdaijvjgAQIHUiA0IEECB1IgNCBBAgdSKDYds+1LwpcxtzcXJF1S24jvbi4WPuaEcF2z0BWhA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAq3aBXb9+vUllpUknT17tsi68/PzRdYtafXq1UXWLflc2Es2Pq3F+++/X2RdSbrrrrtqX3NhYYFdYIGsCB1IgNCBBAgdSIDQgQQIHUiA0IEEKoVu+3Hb07Y/sv2q7WtKDwagPn1Dt71F0qOSRiPiVkkdSQ+UHgxAfapeug9Jutb2kKRhSV+UGwlA3fqGHhGfS3pW0jFJJyV9GxFvlx4MQH2qXLpfL+l+Sdsk3SBpje0HL3HcuO1J25P1jwngSlS5dL9b0qcRMRsR5yQdkHTHxQdFxEREjEbEaN1DArgyVUI/Jmmn7WF3PyK0W9JM2bEA1KnKa/SDkvZLOiTpw96fmSg8F4AaDVU5KCKekfRM4VkAFMKdcUAChA4kQOhAAoQOJEDoQAKEDiRQbLvnEtvvlpi1tFJbJ0vltk9etWpVkXXPnTtXZF1J6nQ6RdYt9VxI0qlTp2pfc2xsTFNTU2z3DGRE6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kUGoX2FlJ/6x4+EZJ/6p9iHLaNq/EzCthUOb9UURsuvjBIqEvh+3JiBhtdIhlaNu8EjOvhEGfl0t3IAFCBxIYhNAnmh5gmdo2r8TMK2Gg5238NTqA8gbhjA6gMEIHEiB0IAFCBxIgdCCB/wDq4qoAPFae9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(cfm, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "selective-billion",
   "metadata": {},
   "outputs": [],
   "source": [
    "row_sums = np.sum(cfm, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "creative-airline",
   "metadata": {},
   "outputs": [],
   "source": [
    "err_matrix = cfm / row_sums"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "adult-measurement",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.fill_diagonal(err_matrix, 0)\n",
    "# np.fill_digonal(矩阵， m)：将矩阵对角线的数据全部改为 m；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "polish-convert",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAKrUlEQVR4nO3d34tc9RnH8c8nuxs0saLQILgbGoViK0LJssQfWbwwXrRV9KaIAYV6kwvbGIMgWi/8B0QjUoQl1huDIjEXRYq1oCJ7s3ZNBI2bgom6mRgxe9EoudnIPr2YEZLs1jljznfPTJ73CwLZyfHrw7DvnDOTM991RAjApW1N0wMAKI/QgQQIHUiA0IEECB1IgNCBBBoL3fZvbf/H9me2n2hqjqpsb7T9ru0524dt72p6pipsD9k+ZPvNpmepwvZVtvfbPtJ5rm9teqZubO/ufE98YvtV25c1PdOFGgnd9pCkv0r6naQbJW23fWMTs/Tge0mPRcSvJd0i6U8DMLMk7ZI01/QQPXhe0lsR8StJv1Gfz257VNIjkiYi4iZJQ5Lub3aq5Zo6o2+R9FlEHIuIRUmvSbq3oVkqiYiTEXGw8/vv1P4GHG12qh9ne0zSXZL2Nj1LFbavlHS7pJckKSIWI+K/jQ5VzbCky20PS1on6auG51mmqdBHJR0/5+uW+jyac9neJGmzpJmGR+lmj6THJS01PEdV10s6JenlzsuNvbbXNz3Uj4mIE5KekTQv6aSk0xHxdrNTLddU6F7hsYG4F9f2FZLekPRoRHzb9Dz/j+27JX0TER82PUsPhiWNS3oxIjZLOiOpr9+/sX212lej10m6VtJ62w80O9VyTYXekrTxnK/H1IeXOxeyPaJ25Psi4kDT83SxVdI9tr9Q+6XRHbZfaXakrlqSWhHxw5XSfrXD72d3Svo8Ik5FxFlJByTd1vBMyzQV+r8l/dL2dbbXqv3mxd8bmqUS21b7teNcRDzb9DzdRMSTETEWEZvUfn7fiYi+O9OcKyK+lnTc9g2dh7ZJ+rTBkaqYl3SL7XWd75Ft6sM3EIeb+J9GxPe2/yzpn2q/S/m3iDjcxCw92CrpQUkf2/6o89hfIuIfzY10SdopaV/nBHBM0kMNz/OjImLG9n5JB9X+l5lDkqaanWo58zFV4NLHnXFAAoQOJEDoQAKEDiRA6EACjYdue0fTM/Ri0OaVmHk19Pu8jYcuqa+foBUM2rwSM6+Gvp63H0IHUFiRG2ZsD9xdOCMjI5WOW1pa0po11f9+PHv27E8dqTFVn4teDeJzMYgiYtmHxhq5BbYfXXPNNUXWbbVaRdYtiefi0sOlO5AAoQMJEDqQAKEDCRA6kECl0AdtD3YA5+sa+oDuwQ7gHFXO6AO3BzuA81UJfaD3YAdQ7c64Snuwdz6909c39gNZVQm90h7sETGlzu6Xg3ivO3Apq3LpPnB7sAM4X9cz+oDuwQ7gHJU+vdb5IQX8oAJgQHFnHJAAoQMJEDqQAKEDCRA6kACbQ2LV3HzzzcXWnpmZKbLu5ORkkXUlaXp6usi6K20OyRkdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEEBmq757GxsRLLSpJarVaRdbds2VJkXUn64IMPiq2NwcV2z0BShA4kQOhAAoQOJEDoQAKEDiRA6EACXUO3vdH2u7bnbB+2vWs1BgNQn+EKx3wv6bGIOGj7Z5I+tP2viPi08GwAatL1jB4RJyPiYOf330makzRaejAA9enpNbrtTZI2S5opMg2AIqpcukuSbF8h6Q1Jj0bEtyv8+Q5JO2qcDUBNKoVue0TtyPdFxIGVjomIKUlTneOLfKgFwE9T5V13S3pJ0lxEPFt+JAB1q/IafaukByXdYfujzq/fF54LQI26XrpHxLSkZZ9vBTA4uDMOSIDQgQQIHUiA0IEECB1IYKB2gcXqeP3114use9999xVZt6TJycliax89erT2NRcWFrS4uMgusEBGhA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJDBQ2z2PjY2VWFaS1Gq1iq2Ntvfff7/Y2tu3by+y7tDQUJF1JWl+fr7IuhHBds9ARoQOJEDoQAKEDiRA6EAChA4kQOhAApVDtz1k+5DtN0sOBKB+vZzRd0maKzUIgHIqhW57TNJdkvaWHQdACVXP6HskPS5pqdwoAErpGrrtuyV9ExEfdjluh+1Z27O1TQegFlXO6Fsl3WP7C0mvSbrD9isXHhQRUxExERETNc8I4CJ1DT0inoyIsYjYJOl+Se9ExAPFJwNQG/4dHUhguJeDI+I9Se8VmQRAMZzRgQQIHUiA0IEECB1IgNCBBNgFtjB2l10do6OjRdY9ceJEkXUlaXx8vPY1jxw5ojNnzrALLJARoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQQJFdYNeuXRsbNmyofd01a8r9vTSIu7Vu3LixyLrHjx8vsi5WR0SwCyyQEaEDCRA6kAChAwkQOpAAoQMJEDqQQKXQbV9le7/tI7bnbN9aejAA9RmueNzzkt6KiD/YXitpXcGZANSsa+i2r5R0u6Q/SlJELEpaLDsWgDpVuXS/XtIpSS/bPmR7r+31hecCUKMqoQ9LGpf0YkRslnRG0hMXHmR7h+1Z27NLS0s1jwngYlQJvSWpFREzna/3qx3+eSJiKiImImKi5IdPAPSua5ER8bWk47Zv6Dy0TdKnRacCUKuq77rvlLSv8477MUkPlRsJQN0qhR4RH0maKDsKgFJ4MQ0kQOhAAoQOJEDoQAKEDiRA6EACRbZ7tl3/okADJicni6199OjR2tdcWFjQ4uIi2z0DGRE6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwlU/WmqfaHEjrU/2L17d5F19+zZU2Tdkko9F88991yRdSVpdHS0yLrT09NF1pWk8fHx2tc8ffr0io9zRgcSIHQgAUIHEiB0IAFCBxIgdCABQgcSqBS67d22D9v+xParti8rPRiA+nQN3faopEckTUTETZKGJN1fejAA9al66T4s6XLbw5LWSfqq3EgA6tY19Ig4IekZSfOSTko6HRFvlx4MQH2qXLpfLeleSddJulbSetsPrHDcDtuztmfrHxPAxahy6X6npM8j4lREnJV0QNJtFx4UEVMRMRERE3UPCeDiVAl9XtItttfZtqRtkubKjgWgTlVeo89I2i/poKSPO//NVOG5ANSo0ufRI+JpSU8XngVAIdwZByRA6EAChA4kQOhAAoQOJEDoQAJFtnseGRnRhg0bal+3fb/OYNm5c2extV944YUi67ZarSLrllRqu+ennnqqyLqS9PDDDxdb+0Kc0YEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBBwR9S9qn5L0ZcXDfy5pofYhyhm0eSVmXg39Mu8vImLZFsxFQu+F7dmImGh0iB4M2rwSM6+Gfp+XS3cgAUIHEuiH0KeaHqBHgzavxMyroa/nbfw1OoDy+uGMDqAwQgcSIHQgAUIHEiB0IIH/AdaVeW7L5oeVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(err_matrix, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}