Update Text Representation (Embeddings).ipynb

Author: bansalkanav

Module 9 - GenAI (LLMs and Prompt Engineering)/1. Text Embeddings/Text Representation (Embeddings).ipynb

@@ -422,13 +422,16 @@
    "id": "1cabe4ef-4f68-4194-99b3-784d56e5cc09",
    "metadata": {},
    "source": [
-    "At this point, we can notice that the classifier is doing poorly with identifying relevant articles, while it is doing well with non-relevant ones. Our large feature vector could be creating a lot of noise in the form of very rarely occurring features that are not useful for learning. Let us change the count vectorizer to take a certain number of features as maximum.\n",
+    "At this point, we can notice that the classifier is doing poorly with identifying relevant articles, while it is doing well with non-relevant ones. \n",
     "\n",
     "**Potential Reasons for poor classifier performance**\n",
     "1. Perhaps we need to balance the data - Clearly there is class imbalance\n",
     "2. Perhaps we need a better learning algorithm - Implement Logistic Regression, SVM, RF, etc...\n",
     "3. Perhaps we should look for tuning the classifier's parameter with the help of Hyperparameter Tuning\n",
-    "4. Perhaps we need a better pre-processing and feature extraction mechanism - Right now we have a sparse and large feature vector"
+    "4. Perhaps we need a better pre-processing and feature extraction mechanism - Right now we have a sparse and large feature vector\n",
+    "\n",
+    "**Let's work with potential reason number 4.**  \n",
+    "Our large feature vector could be creating a lot of noise in the form of very rarely occurring features that are not useful for learning. Let us change the count vectorizer to take a certain number of features as maximum."
    ]
   },
   {