Chapter2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 2: Your first practical NLP application, spam filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in spam and ham file lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import codecs\n",
    "\n",
    "def read_in(folder):\n",
    "    files = os.listdir(folder)\n",
    "    a_list = []\n",
    "    for a_file in files:\n",
    "        if not a_file.startswith(\".\"):\n",
    "            f = codecs.open(folder + a_file, \"r\", encoding = \"ISO-8859-1\", errors=\"ignore\")\n",
    "            a_list.append(f.read())\n",
    "            f.close()\n",
    "    return a_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise lists and print out length – this should return 1500 for `enron1/spam` and 3672 for `enron1/ham`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1500\n",
      "Subject: what up,, your cam babe\r\n",
      "What are you looking for?\r\n",
      "If your looking for a companion for friendship, love, a date, or just good ole'\r\n",
      "Fashioned * * * * * *, then try our brand new site; it was developed and created\r\n",
      "To help anyone find what they' re looking for. A quick bio form and you' re\r\n",
      "On the road to satisfaction in every sense of the word.... No matter what\r\n",
      "That may be!\r\n",
      "Try it out and youll be amazed.\r\n",
      "Have a terrific time this evening\r\n",
      "Copy and pa ste the add. Ress you see on the line below into your browser to come to the site.\r\n",
      "Http:// www. Meganbang. Biz/bld/acc /\r\n",
      "No more plz\r\n",
      "Http:// www. Naturalgolden. Com/retract /\r\n",
      "Counterattack aitken step preemptive shoehorn scaup. Electrocardiograph movie honeycomb. Monster war brandywine pietism byrne catatonia. Encomia lookup intervenor skeleton turn catfish.\r\n",
      "\n",
      "3672\n",
      "Subject: ena sales on hpl\r\n",
      "Just to update you on this project' s status:\r\n",
      "Based on a new report that scott mills ran for me from sitara, I have come up\r\n",
      "With the following counterparties as the ones to which ena is selling gas off\r\n",
      "Of hpl' s pipe.\r\n",
      "Altrade transaction, l. L. C. Gulf gas utilities company\r\n",
      "Brazoria, city of panther pipeline, inc.\r\n",
      "Central illinois light company praxair, inc.\r\n",
      "Central power and light company reliant energy - entex\r\n",
      "Ces - equistar chemicals, lp reliant energy - hl & p\r\n",
      "Corpus christI gas marketing, lp southern union company\r\n",
      "D & h gas company, inc. Texas utilities fuel company\r\n",
      "Duke energy field services, inc. Txu gas distribution\r\n",
      "Entex gas marketing company union carbide corporation\r\n",
      "Equistar chemicals, lp unit gas transmission company inc.\r\n",
      "Since i' m not sure exactly what gets entered into sitara, pat clynes\r\n",
      "Suggested that I check with daren farmer to make sure that i' m not missing\r\n",
      "Something (which I did below). While I am waiting for a response from him\r\n",
      "And/or mary smith, I will begin gathering the contractual volumes under the\r\n",
      "Above contracts.\r\n",
      "- - - - - - - - - - - - - - - - - - - - - - forwarded by cheryl dudley/hou/ect on 05/10/2000 07: 56\r\n",
      "Am - - - - - - - - - - - - - - - - - - - - - - - - - - -\r\n",
      "Cheryl d king\r\n",
      "05/08/2000 04: 11 pm\r\n",
      "Sent by: cheryl dudley\r\n",
      "To: daren j farmer/hou/ect@ ect, mary m smith/hou/ect@ ect\r\n",
      "Cc:\r\n",
      "Subject: ena sales on hpl\r\n",
      "I am working on a project for brenda herod & was wondering if one of you\r\n",
      "Could tell me if i' m on the right track & if this will get everything for\r\n",
      "Which she is looking.\r\n",
      "She is trying to draft a long - term transport/storage agreement between ena &\r\n",
      "Hplc which will allow ena to move the gas to their markets. In order to\r\n",
      "Accomplish this, she needs to know all of the sales to customers that ena is\r\n",
      "Doing off of hpl' s pipe.\r\n",
      "I had scott mills run a report from sitara showing all ena buy/sell activity\r\n",
      "On hpl since 7/99. If I eliminate the buys & the desk - to - desk deals, will\r\n",
      "This give me everything that I need?\r\n",
      "Are there buy/sell deals done with ena on hpl' s pipe that wouldn' t show up in\r\n",
      "Sitara? Someone mentioned something about deals where hpl transports the gas\r\n",
      "On it' s own behalf then ena sells it to a customer at that same spot - -\r\n",
      "????? Do deals like that happen? Would they show up in sitara?\r\n",
      "Is there anything else that i' m missing? I' m not real familiar with how some\r\n",
      "Of these deals happen nowadays so am very receptive to any\r\n",
      "Ideas/suggestions/help that you can offer!!!\r\n",
      "Thanks in advance.\n"
     ]
    }
   ],
   "source": [
    "spam_list = read_in(\"enron1/spam/\")\n",
    "print(len(spam_list))\n",
    "print(spam_list[0])\n",
    "ham_list = read_in(\"enron1/ham/\")\n",
    "print(len(ham_list))\n",
    "print(ham_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine all emails together, keeping the label, and shuffle them: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset size = 5172 emails\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "all_emails = [(email_content, \"spam\") for email_content in spam_list]\n",
    "all_emails += [(email_content, \"ham\") for email_content in ham_list]\n",
    "random.seed(42)\n",
    "random.shuffle(all_emails)\n",
    "print (f\"Dataset size = {str(len(all_emails))} emails\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocess the texts by tokenising them and removing the stopwords:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'participate': True, 'in': True, 'our': True, 'new': True, 'lottery': True, 'now': True, '!': True}\n",
      "5172\n",
      "38\n",
      "38\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk import word_tokenize\n",
    "\n",
    "def get_features(text): \n",
    "    features = {}\n",
    "    word_list = [word for word in word_tokenize(text.lower())]\n",
    "    for word in word_list:\n",
    "        features[word] = True\n",
    "    return features\n",
    "\n",
    "all_features = [(get_features(email), label) for (email, label) in all_emails]\n",
    "\n",
    "print(get_features(\"Participate In Our New Lottery NOW!\"))\n",
    "print(len(all_features))\n",
    "print(len(all_features[0][0]))\n",
    "print(len(all_features[99][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply Naive Bayes classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set size = 4137 emails\n",
      "Test set size = 1035 emails\n"
     ]
    }
   ],
   "source": [
    "from nltk import NaiveBayesClassifier, classify\n",
    "\n",
    "def train(features, proportion):\n",
    "    train_size = int(len(features) * proportion)\n",
    "    # initialise the training and test sets\n",
    "    train_set, test_set = features[:train_size], features[train_size:]\n",
    "    print (f\"Training set size = {str(len(train_set))} emails\")\n",
    "    print (f\"Test set size = {str(len(test_set))} emails\")\n",
    "    # train the classifier\n",
    "    classifier = NaiveBayesClassifier.train(train_set)\n",
    "    return train_set, test_set, classifier\n",
    "\n",
    "train_set, test_set, classifier = train(all_features, 0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the training set = 0.9615663524292966\n",
      "Accuracy on the test set = 0.936231884057971\n",
      "Most Informative Features\n",
      "               forwarded = True              ham : spam   =    200.5 : 1.0\n",
      "                    2004 = True             spam : ham    =    148.6 : 1.0\n",
      "                     nom = True              ham : spam   =    125.8 : 1.0\n",
      "                    pain = True             spam : ham    =    103.6 : 1.0\n",
      "                    spam = True             spam : ham    =     92.4 : 1.0\n",
      "                  health = True             spam : ham    =     81.1 : 1.0\n",
      "                     sex = True             spam : ham    =     79.5 : 1.0\n",
      "                     ect = True              ham : spam   =     75.7 : 1.0\n",
      "              nomination = True              ham : spam   =     74.8 : 1.0\n",
      "                   super = True             spam : ham    =     74.7 : 1.0\n",
      "                featured = True             spam : ham    =     73.1 : 1.0\n",
      "                creative = True             spam : ham    =     71.5 : 1.0\n",
      "                  differ = True             spam : ham    =     71.5 : 1.0\n",
      "                      cc = True              ham : spam   =     64.9 : 1.0\n",
      "                     ibm = True             spam : ham    =     63.4 : 1.0\n",
      "                   adobe = True             spam : ham    =     61.8 : 1.0\n",
      "                  shares = True             spam : ham    =     61.8 : 1.0\n",
      "            solicitation = True             spam : ham    =     61.8 : 1.0\n",
      "               clearance = True             spam : ham    =     60.2 : 1.0\n",
      "             medications = True             spam : ham    =     60.2 : 1.0\n",
      "                congress = True             spam : ham    =     58.6 : 1.0\n",
      "                     pro = True             spam : ham    =     57.0 : 1.0\n",
      "                 dealers = True             spam : ham    =     55.4 : 1.0\n",
      "                    draw = True             spam : ham    =     53.8 : 1.0\n",
      "                    2005 = True             spam : ham    =     53.5 : 1.0\n",
      "                    2001 = True              ham : spam   =     53.4 : 1.0\n",
      "                   cheap = True             spam : ham    =     51.6 : 1.0\n",
      "                       u = True             spam : ham    =     49.6 : 1.0\n",
      "                 doctors = True             spam : ham    =     45.8 : 1.0\n",
      "               publisher = True             spam : ham    =     45.8 : 1.0\n",
      "                  sexual = True             spam : ham    =     45.8 : 1.0\n",
      "               thousands = True             spam : ham    =     44.2 : 1.0\n",
      "              affordable = True             spam : ham    =     42.6 : 1.0\n",
      "                licensed = True             spam : ham    =     42.6 : 1.0\n",
      "                mailings = True             spam : ham    =     42.6 : 1.0\n",
      "                reliable = True             spam : ham    =     42.6 : 1.0\n",
      "                   julie = True              ham : spam   =     42.2 : 1.0\n",
      "                 advises = True             spam : ham    =     41.0 : 1.0\n",
      "                      ex = True             spam : ham    =     41.0 : 1.0\n",
      "                 popular = True             spam : ham    =     41.0 : 1.0\n",
      "                    lisa = True              ham : spam   =     40.0 : 1.0\n",
      "                pipeline = True              ham : spam   =     39.4 : 1.0\n",
      "                 foresee = True             spam : ham    =     39.4 : 1.0\n",
      "                powerful = True             spam : ham    =     39.4 : 1.0\n",
      "                      wi = True             spam : ham    =     39.4 : 1.0\n",
      "                   risks = True             spam : ham    =     39.0 : 1.0\n",
      "                   steve = True              ham : spam   =     38.3 : 1.0\n",
      "                   susan = True              ham : spam   =     38.3 : 1.0\n",
      "                     ali = True             spam : ham    =     38.1 : 1.0\n",
      "                    anti = True             spam : ham    =     37.7 : 1.0\n"
     ]
    }
   ],
   "source": [
    "def evaluate(train_set, test_set, classifier):\n",
    "    # check how the classifier performs on the training and test sets\n",
    "    print (f\"Accuracy on the training set = {str(classify.accuracy(classifier, train_set))}\")\n",
    "    print (f\"Accuracy on the test set = {str(classify.accuracy(classifier, test_set))}\")    \n",
    "    # check which words are most informative for the classifier\n",
    "    classifier.show_most_informative_features(50)\n",
    "\n",
    "evaluate(train_set, test_set, classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore the contexts of use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STOCKS in HAM:\n",
      "Displaying 1 of 1 matches:\n",
      "ur member directory . * follow your stocks and news headlines , exchange files\n",
      "Displaying 1 of 1 matches:\n",
      "ur member directory . * follow your stocks and news headlines , exchange files\n",
      "Displaying 1 of 1 matches:\n",
      "ur member directory . * follow your stocks and news headlines , exchange files\n",
      "Displaying 1 of 1 matches:\n",
      "ad my portfolio is diversified into stocks that have lost even more money than\n",
      "\n",
      "\n",
      "STOCKS in SPAM:\n",
      "Displaying 3 of 3 matches:\n",
      "report reveals this smallcap rocket stocks newsletter first we would like to s\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 3 of 3 matches:\n",
      "might occur . as with many microcap stocks , today ' s company has additional \n",
      "is emai | pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this emai | . none \n",
      "Displaying 6 of 6 matches:\n",
      "hem : ( big money was made in these stocks by savvy investors who timed them r\n",
      "g filthy , stinking ri ' ch in tiny stocks no one has ever heard of until now \n",
      "ynamic things . some of these small stocks have absolutely exploded in price r\n",
      "'' occur . as with many micro - cap stocks , today ' s company has additional \n",
      " ema - il pertaining to investing , stocks or securities must be understood as\n",
      "ntative before deciding to trade in stocks featured within this ema - il . non\n",
      "Displaying 2 of 2 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this emai | . none \n",
      "Displaying 1 of 1 matches:\n",
      "in apple investments , inc profiled stocks . in order to be in full compliance\n",
      "Displaying 3 of 3 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      " lose money from investing in penny stocks . if you wish to stop future mailin\n",
      "Displaying 3 of 3 matches:\n",
      " plays . widespread gains in energy stocks are inflating the portfolios of agg\n",
      "st levels of the year , with energy stocks outperforming all other market sect\n",
      "utions that sma | | and micro - cap stocks are high - risk investments and tha\n",
      "Displaying 4 of 4 matches:\n",
      "watch this one trade . these little stocks can surprise in a big way sometimes\n",
      "might occur . as with many microcap stocks , today ' s company has additional \n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 1 of 1 matches:\n",
      "_ _ _ _ _ _ _ _ _ _ _ _ _ _ penny - stocks are considered highly speculative a\n",
      "Displaying 3 of 3 matches:\n",
      "might occur . as with many microcap stocks , today ' s company has additiona |\n",
      "is emai | pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 1 of 1 matches:\n",
      "ne trade thursday ! go fcdh . penny stocks are considered highiy specuiative a\n",
      "Displaying 2 of 2 matches:\n",
      "ims and do your own due diligence . stocks to play ( s 2 p ) profiles are not \n",
      "s obtained . investing in micro cap stocks is extremely risky and , investors \n",
      "Displaying 2 of 2 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 2 of 2 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 1 of 1 matches:\n",
      "scovering value in natural resource stocks elgin resources ( elr - tsx ) extra\n",
      "Displaying 2 of 2 matches:\n",
      "ck monday some of these little voip stocks have been really moving lately . an\n",
      " one trade monday ! go ypil . penny stocks are considered highiy specuiative a\n",
      "Displaying 1 of 1 matches:\n",
      " one trade monday ! go ndin . penny stocks are considered highly speculative a\n",
      "Displaying 1 of 1 matches:\n",
      " one trade monday ! go wysk . penny stocks are considered highiy specuiative a\n",
      "Displaying 1 of 1 matches:\n",
      "fessionally not multi - level - not stocks - not real estate no cost tele - se\n",
      "Displaying 2 of 2 matches:\n",
      " % on regular price we have massive stocks of drugs for same day dispatch fast\n",
      "e do have the lowest price and huge stocks ready for same - day dispatch . two\n",
      "Displaying 5 of 5 matches:\n",
      "ck monday some of these littie voip stocks have been really moving lately . an\n",
      "t can happen with these sma | | cap stocks when they take off . and it happens\n",
      " statements . as with many microcap stocks , today ' s company has additiona |\n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 3 of 3 matches:\n",
      "might occur . as with many microcap stocks , today ' s company has additiona |\n",
      "is emai | pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this emai | . none \n",
      "Displaying 1 of 1 matches:\n",
      "subject : fwd : screw doctors . stocks available . vlagr @ . x _ a _ nax .\n",
      "Displaying 1 of 1 matches:\n",
      " one trade monday ! go wysk . penny stocks are considered highiy speculative a\n",
      "Displaying 2 of 2 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 4 of 4 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "eep in mind that when trading small stocks like the company above there is a c\n",
      "t professional before investing any stocks or mutual funds .\n",
      "Displaying 1 of 1 matches:\n",
      " one trade monday ! go wysk . penny stocks are considered highiy specuiative a\n",
      "Displaying 1 of 1 matches:\n",
      "dge - ksige are you tired of buying stocks and not having them perform ? our s\n",
      "Displaying 4 of 4 matches:\n",
      "ck monday some of these little voip stocks have been rea | | y moving lately .\n",
      " statements . as with many microcap stocks , today ' s company has additiona |\n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 4 of 4 matches:\n",
      "tion is key to stock success rocket stocks newsletter u r g e n t i n v e s t \n",
      "ht occur . as with many micro - cap stocks , today ' s company has additional \n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 4 of 4 matches:\n",
      " the last 12 months , many of these stocks made triple and even quadruple retu\n",
      " statements . as with many microcap stocks , today ' s company has additiona |\n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 3 of 3 matches:\n",
      "ancements but may be one of the few stocks left in this industry group that is\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 3 of 3 matches:\n",
      "might occur . as with many microcap stocks , today ' s company has additiona |\n",
      "is emai | pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 2 of 2 matches:\n",
      " % on regular price we have massive stocks of drugs for same day dispatch fast\n",
      "e do have the lowest price and huge stocks ready for same - day dispatch . two\n",
      "Displaying 1 of 1 matches:\n",
      "or information puposes only . penny stocks are considered highly speculative a\n",
      "Displaying 4 of 4 matches:\n",
      "hree days . play of the week tracks stocks on downward trends , foresees botto\n",
      "mark is our uncanny ability to spot stocks that have bottomed - out and antici\n",
      "ound and upward trend . most of the stocks we track rebound and peak within ju\n",
      "om third party . investing in penny stocks is high risk and you should seek pr\n",
      "Displaying 4 of 4 matches:\n",
      "nt opportunity drummond , small cap stocks alert newsletter must read - alert \n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      " lose money from investing in penny stocks . - - - - - - - - - - - - - - - - -\n",
      "Displaying 4 of 4 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "eep in mind that when trading small stocks like the company above there is a c\n",
      "t professional before investing any stocks or mutual funds .\n",
      "Displaying 1 of 1 matches:\n",
      "the | ast 12 months , many of these stocks made triple and even quadruple retu\n",
      "Displaying 1 of 1 matches:\n",
      "cautions that small and micro - cap stocks are high - risk investments and tha\n",
      "Displaying 4 of 4 matches:\n",
      "k tuesday some of these littie voip stocks have been reaily moving lateiy . an\n",
      " statements . as with many microcap stocks , today ' s company has additional \n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 2 of 2 matches:\n",
      "ck monday some of these little voip stocks have been realiy moving lately . an\n",
      " one trade monday ! go ypil . penny stocks are considered highiy specuiative a\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 2 of 2 matches:\n",
      " % on regular price we have massive stocks of drugs for same day dispatch fast\n",
      "e do have the lowest price and huge stocks ready for same - day dispatch . two\n",
      "Displaying 2 of 2 matches:\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 4 of 4 matches:\n",
      "n this stock . some of these smal | stocks are absoiuteiy fiying , as many of \n",
      " statements . as with many microcap stocks , todays company has additional ris\n",
      "biication pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this publication . \n",
      "Displaying 1 of 1 matches:\n",
      "s obtained . investing in micro cap stocks is extremely risky and , investors \n",
      "Displaying 2 of 2 matches:\n",
      "is emai | pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 3 of 3 matches:\n",
      " statements . as with many microcap stocks , todays company has additional ris\n",
      "blication pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this publication . \n",
      "Displaying 1 of 1 matches:\n",
      " the last 12 months , many of these stocks made tripie and even quadruple retu\n",
      "Displaying 3 of 3 matches:\n",
      "torage inc. play of the week tracks stocks on downward trends , foresees botto\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 1 of 1 matches:\n",
      "ecializing in undervalued small cap stocks for immediate breakout erhc and exx\n",
      "Displaying 3 of 3 matches:\n",
      "5 how many times have you seen good stocks but you couldn ' t get your hands o\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 5 of 5 matches:\n",
      "5 where were you when the following stocks exploded : scos : exploded from . 3\n",
      "d . 80 on friday . face it . little stocks can mean big gains for you . this r\n",
      "might occur . as with many microcap stocks , today ' s company has additional \n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 2 of 2 matches:\n",
      "subject : penny stocks are about timing nomad internationa\n",
      " one trade friday ! go ndin . penny stocks are considered highiy speculative a\n",
      "Displaying 2 of 2 matches:\n",
      "ng their gains . select gold mining stocks are the hot flyers of the otc . his\n",
      "is letter cautions that micro - cap stocks are high - risk investments and tha\n",
      "Displaying 1 of 1 matches:\n",
      "cautions that small and micro - cap stocks are high - risk investments and tha\n",
      "Displaying 5 of 5 matches:\n",
      "hursday ! some of these littie voip stocks have been realiy moving lateiy . an\n",
      "t can happen with these sma | | cap stocks when they take off . and it happens\n",
      " statements . as with many microcap stocks , today ' s company has additiona |\n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 2 of 2 matches:\n",
      "rt identifying defense and security stocks ready to explode look at the moves \n",
      " actual exchanges where small - cap stocks are traded . silica stopband doorkn\n",
      "Displaying 6 of 6 matches:\n",
      " if you knew about these low priced stocks : otcbb : zapz : closed march 31 st\n",
      " following points : * many of these stocks are undiscovered and uncovered ! wh\n",
      " ! ! * * many of these undiscovered stocks are like coiled springs , wound tig\n",
      "might occur . as with many microcap stocks , today ' s company has additional \n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 2 of 2 matches:\n",
      " the last 12 months , many of these stocks made tripie and even quadruple retu\n",
      "one trade tuesday ! go mogi . penny stocks are considered highly speculative a\n",
      "Displaying 3 of 3 matches:\n",
      " statements . as with many microcap stocks , todays company has additional ris\n",
      "blication pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this publication . \n",
      "Displaying 3 of 3 matches:\n",
      " statements . as with many microcap stocks , today ' s company has additiona |\n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 4 of 4 matches:\n",
      "y agree , some , not all , of these stocks move in price because they are prom\n",
      "tands or that as with many microcap stocks , today ' s company has additional \n",
      "is report pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this report . none \n",
      "Displaying 3 of 3 matches:\n",
      "might occur . as with many microcap stocks , today ' s company has additiona |\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 3 of 3 matches:\n",
      "n how many times have you seen good stocks but you couldn ' t get your hands o\n",
      "his email pertaining to investing , stocks , securities must be understood as \n",
      "ntative before deciding to trade in stocks featured within this email . none o\n",
      "Displaying 1 of 1 matches:\n",
      " receive first notice on run - away stocks traders ' monthly alert january pic\n"
     ]
    }
   ],
   "source": [
    "from nltk.text import Text\n",
    "\n",
    "def concordance(data_list, search_word):\n",
    "    for email in data_list:\n",
    "        word_list = [word for word in word_tokenize(email.lower())]\n",
    "        text_list = Text(word_list)\n",
    "        if search_word in word_list:\n",
    "            text_list.concordance(search_word)\n",
    "\n",
    "\n",
    "print (\"STOCKS in HAM:\")\n",
    "concordance(ham_list, \"stocks\")\n",
    "print (\"\\n\\nSTOCKS in SPAM:\")\n",
    "concordance(spam_list, \"stocks\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input some of your own messages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the training set = 0.9615663524292966\n",
      "Accuracy on the test set = 1.0\n",
      "Most Informative Features\n",
      "               forwarded = True              ham : spam   =    200.5 : 1.0\n",
      "                    2004 = True             spam : ham    =    148.6 : 1.0\n",
      "                     nom = True              ham : spam   =    125.8 : 1.0\n",
      "                    pain = True             spam : ham    =    103.6 : 1.0\n",
      "                    spam = True             spam : ham    =     92.4 : 1.0\n",
      "                  health = True             spam : ham    =     81.1 : 1.0\n",
      "                     sex = True             spam : ham    =     79.5 : 1.0\n",
      "                     ect = True              ham : spam   =     75.7 : 1.0\n",
      "              nomination = True              ham : spam   =     74.8 : 1.0\n",
      "                   super = True             spam : ham    =     74.7 : 1.0\n",
      "                featured = True             spam : ham    =     73.1 : 1.0\n",
      "                creative = True             spam : ham    =     71.5 : 1.0\n",
      "                  differ = True             spam : ham    =     71.5 : 1.0\n",
      "                      cc = True              ham : spam   =     64.9 : 1.0\n",
      "                     ibm = True             spam : ham    =     63.4 : 1.0\n",
      "                   adobe = True             spam : ham    =     61.8 : 1.0\n",
      "                  shares = True             spam : ham    =     61.8 : 1.0\n",
      "            solicitation = True             spam : ham    =     61.8 : 1.0\n",
      "               clearance = True             spam : ham    =     60.2 : 1.0\n",
      "             medications = True             spam : ham    =     60.2 : 1.0\n",
      "                congress = True             spam : ham    =     58.6 : 1.0\n",
      "                     pro = True             spam : ham    =     57.0 : 1.0\n",
      "                 dealers = True             spam : ham    =     55.4 : 1.0\n",
      "                    draw = True             spam : ham    =     53.8 : 1.0\n",
      "                    2005 = True             spam : ham    =     53.5 : 1.0\n",
      "                    2001 = True              ham : spam   =     53.4 : 1.0\n",
      "                   cheap = True             spam : ham    =     51.6 : 1.0\n",
      "                       u = True             spam : ham    =     49.6 : 1.0\n",
      "                 doctors = True             spam : ham    =     45.8 : 1.0\n",
      "               publisher = True             spam : ham    =     45.8 : 1.0\n",
      "                  sexual = True             spam : ham    =     45.8 : 1.0\n",
      "               thousands = True             spam : ham    =     44.2 : 1.0\n",
      "              affordable = True             spam : ham    =     42.6 : 1.0\n",
      "                licensed = True             spam : ham    =     42.6 : 1.0\n",
      "                mailings = True             spam : ham    =     42.6 : 1.0\n",
      "                reliable = True             spam : ham    =     42.6 : 1.0\n",
      "                   julie = True              ham : spam   =     42.2 : 1.0\n",
      "                 advises = True             spam : ham    =     41.0 : 1.0\n",
      "                      ex = True             spam : ham    =     41.0 : 1.0\n",
      "                 popular = True             spam : ham    =     41.0 : 1.0\n",
      "                    lisa = True              ham : spam   =     40.0 : 1.0\n",
      "                pipeline = True              ham : spam   =     39.4 : 1.0\n",
      "                 foresee = True             spam : ham    =     39.4 : 1.0\n",
      "                powerful = True             spam : ham    =     39.4 : 1.0\n",
      "                      wi = True             spam : ham    =     39.4 : 1.0\n",
      "                   risks = True             spam : ham    =     39.0 : 1.0\n",
      "                   steve = True              ham : spam   =     38.3 : 1.0\n",
      "                   susan = True              ham : spam   =     38.3 : 1.0\n",
      "                     ali = True             spam : ham    =     38.1 : 1.0\n",
      "                    anti = True             spam : ham    =     37.7 : 1.0\n"
     ]
    }
   ],
   "source": [
    "test_spam_list = [\"Participate in our new lottery!\", \"Try out this new medicine\"]\n",
    "test_ham_list = [\"See the minutes from the last meeting attached\", \n",
    "                 \"Investors are coming to our office on Monday\"]\n",
    "\n",
    "test_emails = [(email_content, \"spam\") for email_content in test_spam_list]\n",
    "test_emails += [(email_content, \"ham\") for email_content in test_ham_list]\n",
    "\n",
    "new_test_set = [(get_features(email), label) for (email, label) in test_emails]\n",
    "\n",
    "evaluate(train_set, new_test_set, classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how they get classified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Participate in our new lottery!\n",
      "spam\n",
      "Try out this new medicine\n",
      "spam\n",
      "See the minutes from the last meeting attached\n",
      "ham\n",
      "Investors are coming to our office on Monday\n",
      "ham\n"
     ]
    }
   ],
   "source": [
    "for email in test_spam_list:\n",
    "    print (email)\n",
    "    print (classifier.classify(get_features(email)))\n",
    "for email in test_ham_list:\n",
    "    print (email)\n",
    "    print (classifier.classify(get_features(email)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run in an interactive manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type in your email here (or press 'Enter'): Buy new meds\n",
      "This email is likely spam\n",
      "\n",
      "Type in your email here (or press 'Enter'): Buy new meds here!\n",
      "This email is likely spam\n",
      "\n",
      "Type in your email here (or press 'Enter'): Get your stock options fast\n",
      "This email is likely spam\n",
      "\n",
      "Type in your email here (or press 'Enter'): Let's schedule a meeting for tomorrow\n",
      "This email is likely ham\n",
      "\n",
      "Type in your email here (or press 'Enter'): \n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    email = input(\"Type in your email here (or press 'Enter'): \")\n",
    "    if len(email)==0:\n",
    "        break\n",
    "    else: \n",
    "        prediction = classifier.classify(get_features(email))\n",
    "        print (f\"This email is likely {prediction}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run on a different dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment:\n",
    "\n",
    "Apply the classifier to a different test set, e.g. the emails from `enron2/`. As before, you need to read in the data, extract textual content, extract the features and evaluate the classifier. What do the results tell you?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1496\n",
      "Subject: big range of all types of downloadable software.\n",
      "Need software? Click here.\n",
      "Our american professors like their literature clear, cold, pure and very dead.\n",
      "Being another character is more interesting than being yourself.\n",
      "4361\n",
      "Subject: re: telephone interview with enron corp. Research dept.\n",
      "Dear shirley:\n",
      "Confirming that I will be waiting for the telephone interview at 1 pm\n",
      "Tomorrow.? I would like to give you my cell phone number, 713/907 - 6717, as a\n",
      "Back - up measure.? Please note that my first preference is to receive the call\n",
      "At my home number, 713/669 - 0923.\n",
      "Sincerely,\n",
      "RabI de\n",
      "?\n",
      "? Shirley. Crenshaw@ enron. Com wrote:\n",
      "Dear rabi:\n",
      "I have scheduled the telephone interview for 1: 00 pm on friday, july 7 th.\n",
      "We will call you at 713/669 - 0923. If there are any changes, please let\n",
      "Me know.\n",
      "Sincerely,\n",
      "Shirley crenshaw\n",
      "713 - 853 - 5290\n",
      "RabI deon 06/26/2000 10: 37: 24 pm\n",
      "To: shirley crenshaw\n",
      "Cc:\n",
      "Subject: re: telephone interview with enron corp. Research dept.\n",
      "Dear ms. Crenshaw:\n",
      "Thanks for your prompt response.? July 6 or 7 th will work best for me..? I\n",
      "Would prefer to be called at my home number.? Please let me know the\n",
      "Schedule and other details, if any.\n",
      "Sincerely,\n",
      "RabI de\n",
      "? Shirley crenshawwrote:\n",
      "Good afternoon mr. De:\n",
      "Your resume has been forwarded to the enron corp. Re! Search dept. And\n",
      "They would like to conduct a telephone interview with you at your\n",
      "Convenience.\n",
      "The interviewers would be:\n",
      "Vince kaminskI managing director\n",
      "Stinson gibner vice president\n",
      "Grant masson vice president\n",
      "P. V. Krishnarao director\n",
      "Paulo issler manager\n",
      "Please give me some dates and times this week or july 5, 6, and 7 th when\n",
      "You might be available and I will coordinate with the other calendars.\n",
      "I look forward to hearing from you.\n",
      "Sincerely,\n",
      "Shirley crenshaw\n",
      "Administrative coordinator\n",
      "Enron corp. Research\n",
      "713/853 - 5290\n",
      "Email: shirley. Crenshaw@ enron. Com\n",
      "Do you yahoo!?\n",
      "Get yahoo! Mail - free email you can access from anywhere!\n",
      "Do you yahoo!?\n",
      "Send instant messages & get email alerts with yahoo! Messenger.\n",
      "Accuracy on the training set = 0.9615663524292966\n",
      "Accuracy on the test set = 0.7430425132320301\n",
      "Most Informative Features\n",
      "               forwarded = True              ham : spam   =    200.5 : 1.0\n",
      "                    2004 = True             spam : ham    =    148.6 : 1.0\n",
      "                     nom = True              ham : spam   =    125.8 : 1.0\n",
      "                    pain = True             spam : ham    =    103.6 : 1.0\n",
      "                    spam = True             spam : ham    =     92.4 : 1.0\n",
      "                  health = True             spam : ham    =     81.1 : 1.0\n",
      "                     sex = True             spam : ham    =     79.5 : 1.0\n",
      "                     ect = True              ham : spam   =     75.7 : 1.0\n",
      "              nomination = True              ham : spam   =     74.8 : 1.0\n",
      "                   super = True             spam : ham    =     74.7 : 1.0\n",
      "                featured = True             spam : ham    =     73.1 : 1.0\n",
      "                creative = True             spam : ham    =     71.5 : 1.0\n",
      "                  differ = True             spam : ham    =     71.5 : 1.0\n",
      "                      cc = True              ham : spam   =     64.9 : 1.0\n",
      "                     ibm = True             spam : ham    =     63.4 : 1.0\n",
      "                   adobe = True             spam : ham    =     61.8 : 1.0\n",
      "                  shares = True             spam : ham    =     61.8 : 1.0\n",
      "            solicitation = True             spam : ham    =     61.8 : 1.0\n",
      "               clearance = True             spam : ham    =     60.2 : 1.0\n",
      "             medications = True             spam : ham    =     60.2 : 1.0\n",
      "                congress = True             spam : ham    =     58.6 : 1.0\n",
      "                     pro = True             spam : ham    =     57.0 : 1.0\n",
      "                 dealers = True             spam : ham    =     55.4 : 1.0\n",
      "                    draw = True             spam : ham    =     53.8 : 1.0\n",
      "                    2005 = True             spam : ham    =     53.5 : 1.0\n",
      "                    2001 = True              ham : spam   =     53.4 : 1.0\n",
      "                   cheap = True             spam : ham    =     51.6 : 1.0\n",
      "                       u = True             spam : ham    =     49.6 : 1.0\n",
      "                 doctors = True             spam : ham    =     45.8 : 1.0\n",
      "               publisher = True             spam : ham    =     45.8 : 1.0\n",
      "                  sexual = True             spam : ham    =     45.8 : 1.0\n",
      "               thousands = True             spam : ham    =     44.2 : 1.0\n",
      "              affordable = True             spam : ham    =     42.6 : 1.0\n",
      "                licensed = True             spam : ham    =     42.6 : 1.0\n",
      "                mailings = True             spam : ham    =     42.6 : 1.0\n",
      "                reliable = True             spam : ham    =     42.6 : 1.0\n",
      "                   julie = True              ham : spam   =     42.2 : 1.0\n",
      "                 advises = True             spam : ham    =     41.0 : 1.0\n",
      "                      ex = True             spam : ham    =     41.0 : 1.0\n",
      "                 popular = True             spam : ham    =     41.0 : 1.0\n",
      "                    lisa = True              ham : spam   =     40.0 : 1.0\n",
      "                pipeline = True              ham : spam   =     39.4 : 1.0\n",
      "                 foresee = True             spam : ham    =     39.4 : 1.0\n",
      "                powerful = True             spam : ham    =     39.4 : 1.0\n",
      "                      wi = True             spam : ham    =     39.4 : 1.0\n",
      "                   risks = True             spam : ham    =     39.0 : 1.0\n",
      "                   steve = True              ham : spam   =     38.3 : 1.0\n",
      "                   susan = True              ham : spam   =     38.3 : 1.0\n",
      "                     ali = True             spam : ham    =     38.1 : 1.0\n",
      "                    anti = True             spam : ham    =     37.7 : 1.0\n"
     ]
    }
   ],
   "source": [
    "test_spam_list = read_in(\"enron2/spam/\")\n",
    "print(len(test_spam_list))\n",
    "print(test_spam_list[0])\n",
    "test_ham_list = read_in(\"enron2/ham/\")\n",
    "print(len(test_ham_list))\n",
    "print(test_ham_list[0])\n",
    "\n",
    "test_emails = [(email_content, \"spam\") for email_content in test_spam_list]\n",
    "test_emails += [(email_content, \"ham\") for email_content in test_ham_list]\n",
    "random.shuffle(test_emails)\n",
    "\n",
    "new_test_set = [(get_features(email), label) for (email, label) in test_emails]\n",
    "\n",
    "evaluate(train_set, new_test_set, classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine the two datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2996\n",
      "8033\n",
      "11029\n",
      "Training set size = 8823 emails\n",
      "Test set size = 2206 emails\n",
      "Accuracy on the training set = 0.9819789187351241\n",
      "Accuracy on the test set = 0.9810483182516647\n",
      "Most Informative Features\n",
      "                   meter = True              ham : spam   =    263.8 : 1.0\n",
      "                   vince = True              ham : spam   =    200.3 : 1.0\n",
      "                     sex = True             spam : ham    =    195.1 : 1.0\n",
      "                     nom = True              ham : spam   =    194.9 : 1.0\n",
      "                     php = True             spam : ham    =    182.1 : 1.0\n",
      "            prescription = True             spam : ham    =    169.2 : 1.0\n",
      "                     ect = True              ham : spam   =    167.7 : 1.0\n",
      "                    spam = True             spam : ham    =    145.8 : 1.0\n",
      "               forwarded = True              ham : spam   =    136.4 : 1.0\n",
      "                     fyi = True              ham : spam   =    134.6 : 1.0\n",
      "                    2005 = True             spam : ham    =    128.1 : 1.0\n",
      "                   logos = True             spam : ham    =    121.2 : 1.0\n",
      "              nomination = True              ham : spam   =    112.6 : 1.0\n",
      "              macromedia = True             spam : ham    =    109.5 : 1.0\n",
      "                  studio = True             spam : ham    =    108.3 : 1.0\n",
      "           uncertainties = True             spam : ham    =    108.3 : 1.0\n",
      "                   corel = True             spam : ham    =    104.4 : 1.0\n",
      "                  dealer = True             spam : ham    =    104.4 : 1.0\n",
      "                 readers = True             spam : ham    =     96.6 : 1.0\n",
      "                     pat = True              ham : spam   =     89.6 : 1.0\n",
      "                      cc = True              ham : spam   =     89.2 : 1.0\n",
      "                    2004 = True             spam : ham    =     81.9 : 1.0\n",
      "                     713 = True              ham : spam   =     81.0 : 1.0\n",
      "              materially = True             spam : ham    =     79.7 : 1.0\n",
      "             medications = True             spam : ham    =     78.4 : 1.0\n",
      "                     853 = True              ham : spam   =     78.3 : 1.0\n",
      "                identity = True             spam : ham    =     75.8 : 1.0\n",
      "                    8859 = True             spam : ham    =     72.0 : 1.0\n",
      "                 shirley = True              ham : spam   =     71.3 : 1.0\n",
      "                   penis = True             spam : ham    =     69.4 : 1.0\n",
      "                 removal = True             spam : ham    =     69.4 : 1.0\n",
      "                      wi = True             spam : ham    =     69.4 : 1.0\n",
      "                 artwork = True             spam : ham    =     68.1 : 1.0\n",
      "                 beliefs = True             spam : ham    =     68.1 : 1.0\n",
      "                   canon = True             spam : ham    =     66.8 : 1.0\n",
      "                 foresee = True             spam : ham    =     66.8 : 1.0\n",
      "                    pain = True             spam : ham    =     65.7 : 1.0\n",
      "             speculative = True             spam : ham    =     65.5 : 1.0\n",
      "                   epson = True             spam : ham    =     61.6 : 1.0\n",
      "                 factual = True             spam : ham    =     61.6 : 1.0\n",
      "                featured = True             spam : ham    =     60.8 : 1.0\n",
      "           advertisement = True             spam : ham    =     59.2 : 1.0\n",
      "                    drug = True             spam : ham    =     59.0 : 1.0\n",
      "                 hewlett = True             spam : ham    =     59.0 : 1.0\n",
      "                  weight = True             spam : ham    =     58.7 : 1.0\n",
      "                 packard = True             spam : ham    =     57.7 : 1.0\n",
      "                  sexual = True             spam : ham    =     57.2 : 1.0\n",
      "               enquiries = True             spam : ham    =     56.4 : 1.0\n",
      "              scheduling = True              ham : spam   =     55.4 : 1.0\n",
      "                   adult = True             spam : ham    =     55.1 : 1.0\n"
     ]
    }
   ],
   "source": [
    "spam_list = read_in(\"enron1/spam/\") + read_in(\"enron2/spam/\")\n",
    "print(len(spam_list))\n",
    "ham_list = read_in(\"enron1/ham/\") + read_in(\"enron2/ham/\")\n",
    "print(len(ham_list))\n",
    "\n",
    "all_emails = [(email_content, \"spam\") for email_content in spam_list]\n",
    "all_emails += [(email_content, \"ham\") for email_content in ham_list]\n",
    "random.shuffle(test_emails)\n",
    "\n",
    "all_features = [(get_features(email), label) for (email, label) in all_emails]\n",
    "print(len(all_features))\n",
    "\n",
    "train_set, test_set, classifier = train(all_features, 0.8)\n",
    "evaluate(train_set, new_test_set, classifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}