Building a Spam Filter with Naive Bayes¶

In this project, we're going to build a spam filter for SMS messages using the multinomial Naive Bayes algorithm. Our goal is to write a program that classifies new messages with an accuracy greater than 80% — so we expect that more than 80% of the new messages will be classified correctly as spam or ham (non-spam).

To train the algorithm, we'll use a dataset of 5,572 SMS messages that are already classified by humans. The dataset was put together by Tiago A. Almeida and José María Gómez Hidalgo, and it can be downloaded from the The UCI Machine Learning Repository. The data collection process is described in more details on this page, where you can also find some of the papers authored by Tiago A. Almeida and José María Gómez Hidalgo.

Exploring the Dataset¶

We'll now start by reading in the dataset.

In [1]:
import pandas as pd

sms_spam = pd.read_csv('SMSSpamCollection', sep='\t', header=None, names=['Label', 'SMS'])

print(sms_spam.shape)
sms_spam.head()

(5572, 2)
Out[1]:
Label SMS
0 ham Go until jurong point, crazy.. Available only ...
1 ham Ok lar... Joking wif u oni...
2 spam Free entry in 2 a wkly comp to win FA Cup fina...
3 ham U dun say so early hor... U c already then say...
4 ham Nah I don't think he goes to usf, he lives aro...
In [2]:
sms_spam['Label'].value_counts(normalize=True)
Out[2]:
ham     0.865937
spam    0.134063
Name: Label, dtype: float64

We see that about 87% of the messages are ham, and the remaining 13% are spam. This sample looks representative, since in practice most messages that people receive are ham.

Training and Test Set¶

We're now going to split our dataset into a training and a test set, where the training set accounts for 80% of the data, and the test set for the remaining 20%.

In [3]:
# Randomize the dataset
data_randomized = sms_spam.sample(frac=1, random_state=1)

# Calculate index for split
training_test_index = round(len(data_randomized) * 0.8)

# Training/Test split
training_set = data_randomized[:training_test_index].reset_index(drop=True)
test_set = data_randomized[training_test_index:].reset_index(drop=True)

print(training_set.shape)
print(test_set.shape)

(4458, 2)
(1114, 2)

We'll now analyze the percentage of spam and ham messages in the training and test sets. We expect the percentages to be close to what we have in the full dataset, where about 87% of the messages are ham, and the remaining 13% are spam.

In [4]:
training_set['Label'].value_counts(normalize=True)
Out[4]:
ham     0.86541
spam    0.13459
Name: Label, dtype: float64
In [5]:
# test_set['Label'].value_counts(normalize=True)
(test_set['Label']== 'ham').mean()
Out[5]:
0.8680430879712747

The results look good! We'll now move on to cleaning the dataset.

Data Cleaning¶

To calculate all the probabilities required by the algorithm, we'll first need to perform a bit of data cleaning to bring the data in a format that will allow us to extract easily all the information we need.

Essentially, we want to bring data to this format:

newformat

Letter Case and Punctuation¶

We'll begin with removing all the punctuation and bringing every letter to lower case.

In [6]:
# Before cleaning
training_set.head()
Out[6]:
Label SMS
0 ham Yep, by the pretty sculpture
1 ham Yes, princess. Are you going to make me moan?
2 ham Welp apparently he retired
3 ham Havent.
4 ham I forgot 2 ask ü all smth.. There's a card on ...
In [7]:
# After cleaning
training_set['SMS']=training_set['SMS'].str.replace('\W', ' ', regex=True) # str is the pd accessor
training_set['SMS']=training_set['SMS'].str.lower()
training_set.head()
Out[7]:
Label SMS
0 ham yep by the pretty sculpture
1 ham yes princess are you going to make me moan
2 ham welp apparently he retired
3 ham havent
4 ham i forgot 2 ask ü all smth there s a card on ...

Creating the Vocabulary¶

Let's now move to creating the vocabulary, which in this context means a list with all the unique words in our training set.

In [8]:
training_set['SMS'] = training_set['SMS'].str.split()

vocabulary = []
for sms in training_set['SMS']:
    for word in sms:
        vocabulary.append(word)
        
vocabulary = list(set(vocabulary))
print(len(vocabulary))

7783

The Final Training Set¶

We're now going to use the vocabulary we just created to make the data transformation we want.

In [9]:
# dictionary comprehension
word_counts_per_sms = {unique_word: [0] * len(training_set['SMS']) for unique_word in vocabulary}

for index, sms in enumerate(training_set['SMS']):
    for word in sms:
        word_counts_per_sms[word][index] += 1
In [10]:
word_counts = pd.DataFrame(word_counts_per_sms)
word_counts.head()
Out[10]:
220cm2 paths benefits progress 07xxxxxxxxx play galileo accent academic gayle ... 09050000928 offered idea eating cuddling booking 84122 6669 buzzzz burnt
0 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0

5 rows × 7783 columns

In [11]:
training_set_clean = pd.concat([training_set, word_counts], axis=1)
training_set_clean.head()
Out[11]:
Label SMS 220cm2 paths benefits progress 07xxxxxxxxx play galileo accent ... 09050000928 offered idea eating cuddling booking 84122 6669 buzzzz burnt
0 ham [yep, by, the, pretty, sculpture] 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
1 ham [yes, princess, are, you, going, to, make, me,... 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
2 ham [welp, apparently, he, retired] 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
3 ham [havent] 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
4 ham [i, forgot, 2, ask, ü, all, smth, there, s, a,... 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0

5 rows × 7785 columns

Calculating Constants First¶

We're now done with cleaning the training set, and we can begin creating the spam filter. The Naive Bayes algorithm will need to answer these two probability questions to be able to classify new messages:

\begin{equation} P(Spam | w_1,w_2, ..., w_n) \propto P(Spam) \cdot \prod_{i=1}^{n}P(w_i|Spam) \end{equation}\begin{equation} P(Ham | w_1,w_2, ..., w_n) \propto P(Ham) \cdot \prod_{i=1}^{n}P(w_i|Ham) \end{equation}

Also, to calculate $P(w_i|Spam)$ and $P(w_i|Ham)$ inside the formulas above, we'll need to use these equations:

\begin{equation} P(w_i|Spam) = \frac{N_{w_i|Spam} + \alpha}{N_{Spam} + \alpha \cdot N_{Vocabulary}} \end{equation}\begin{equation} P(w_i|Ham) = \frac{N_{w_i|Ham} + \alpha}{N_{Ham} + \alpha \cdot N_{Vocabulary}} \end{equation}

Some of the terms in the four equations above will have the same value for every new message. We can calculate the value of these terms once and avoid doing the computations again when a new messages comes in. Below, we'll use our training set to calculate:

  • $P(Spam)$ and $P(Ham)$
  • $N_{Spam}$, $N_{Ham}$, $N_{Vocabulary}$

We'll also use Laplace smoothing and set $\alpha = 1$.

In [12]:
# Isolating spam and ham messages first
spam_messages = training_set_clean[training_set_clean['Label'] == 'spam']
ham_messages = training_set_clean[training_set_clean['Label'] == 'ham']

# P(Spam) and P(Ham)
p_spam = len(spam_messages) / len(training_set_clean)
p_ham = len(ham_messages) / len(training_set_clean)

# N_Spam
n_words_per_spam_message = spam_messages['SMS'].apply(len)
n_spam = n_words_per_spam_message.sum()

# N_Ham
n_words_per_ham_message = ham_messages['SMS'].apply(len)
n_ham = n_words_per_ham_message.sum()

# N_Vocabulary
n_vocabulary = len(vocabulary)

# Laplace smoothing
alpha = 1

Calculating Parameters¶

Now that we have the constant terms calculated above, we can move on with calculating the parameters $P(w_i|Spam)$ and $P(w_i|Ham)$. Each parameter will thus be a conditional probability value associated with each word in the vocabulary.

The parameters are calculated using the formulas:

\begin{equation} P(w_i|Spam) = \frac{N_{w_i|Spam} + \alpha}{N_{Spam} + \alpha \cdot N_{Vocabulary}} \end{equation}\begin{equation} P(w_i|Ham) = \frac{N_{w_i|Ham} + \alpha}{N_{Ham} + \alpha \cdot N_{Vocabulary}} \end{equation}
In [13]:
# Initiate parameters
parameters_spam = {unique_word:0 for unique_word in vocabulary}
parameters_ham = {unique_word:0 for unique_word in vocabulary}

# Calculate parameters
for word in vocabulary:
    n_word_given_spam = spam_messages[word].sum()   # spam_messages already defined in a cell above
    p_word_given_spam = (n_word_given_spam + alpha) / (n_spam + alpha*n_vocabulary)
    parameters_spam[word] = p_word_given_spam
    
    n_word_given_ham = ham_messages[word].sum()   # ham_messages already defined in a cell above
    p_word_given_ham = (n_word_given_ham + alpha) / (n_ham + alpha*n_vocabulary)
    parameters_ham[word] = p_word_given_ham

Classifying A New Message¶

Now that we have all our parameters calculated, we can start creating the spam filter. The spam filter can be understood as a function that:

  • Takes in as input a new message ($w_1$, $w_2$, ..., $w_n$).
  • Calculates $P(Spam|w_1, w_2, ..., w_n)$ and $P(Ham|w_1, w_2, ..., w_n)$.
  • Compares the values of $P(Spam|w_1, w_2, ..., w_n)$ and $P(Ham|w_1, w_2, ..., w_n)$, and:
    • If $P(Ham|w_1, w_2, ..., w_n)$ > $P(Spam|w_1, w_2, ..., w_n)$, then the message is classified as ham.
    • If $P(Ham|w_1, w_2, ..., w_n)$ < $P(Spam|w_1, w_2, ..., w_n)$, then the message is classified as spam.
    • If $P(Ham|w_1, w_2, ..., w_n)$ = $P(Spam|w_1, w_2, ..., w_n)$, then the algorithm may request human help.
In [14]:
import re

def classify(message: str) -> None:
    '''
    The spam filter to identify the message is a spam or ham

    Parameters
    ----------
    message: string
        a SMS message

    Return
    ------
    None
    '''
    
    message = re.sub('\W', ' ', message)
    message = message.lower().split()
    
    p_spam_given_message = p_spam
    p_ham_given_message = p_ham

    for word in message:
        if word in parameters_spam:
            p_spam_given_message *= parameters_spam[word]
            
        if word in parameters_ham:
            p_ham_given_message *= parameters_ham[word]
            
    print('P(Spam|message):', p_spam_given_message)
    print('P(Ham|message):', p_ham_given_message)
    
    if p_ham_given_message > p_spam_given_message:
        print('Label: Ham')
    elif p_ham_given_message < p_spam_given_message:
        print('Label: Spam')
    else:
        print('Equal proabilities, have a human classify this!')
In [15]:
classify('WINNER!! This is the secret code to unlock the money: C3421.')

P(Spam|message): 1.3481290211300841e-25
P(Ham|message): 1.9368049028589875e-27
Label: Spam

Measuring the Spam Filter's Accuracy¶

The two results above look promising, but let's see how well the filter does on our test set, which has 1,114 messages.

We'll start by writing a function that returns classification labels instead of printing them.

In [16]:
def classify_test_set(message: str) -> None:    
    '''
    The spam filter to identify the message is a spam or ham
    for the test_set

    Parameters
    ----------
    message: string
        a SMS message

    Return
    ------
    a string of the classification
    '''
    
    message = re.sub('\W', ' ', message)
    message = message.lower().split()
    
    p_spam_given_message = p_spam
    p_ham_given_message = p_ham

    for word in message:
        if word in parameters_spam:
            p_spam_given_message *= parameters_spam[word]
            
        if word in parameters_ham:
            p_ham_given_message *= parameters_ham[word]
    
    if p_ham_given_message > p_spam_given_message:
        return 'ham'
    elif p_spam_given_message > p_ham_given_message:
        return 'spam'
    else:
        return 'needs human classification'

Now that we have a function that returns labels instead of printing them, we can use it to create a new column in our test set.

In [17]:
test_set['predicted'] = test_set['SMS'].apply(classify_test_set)
test_set.head()
Out[17]:
Label SMS predicted
0 ham Later i guess. I needa do mcat study too. ham
1 ham But i haf enuff space got like 4 mb... ham
2 spam Had your mobile 10 mths? Update to latest Oran... spam
3 ham All sounds good. Fingers . Makes it difficult ... ham
4 ham All done, all handed in. Don't know if mega sh... ham

Now, we'll write a script to measure the accuracy of our spam filter to find out how well our spam filter does.

In [18]:
correct = 0
total = test_set.shape[0]
    
for row in test_set.iterrows():
    row = row[1]
    if row['Label'] == row['predicted']:
        correct += 1
        
print('Correct:', correct)
print('Incorrect:', total - correct)
print('Accuracy:', correct/total)

Correct: 1100
Incorrect: 14
Accuracy: 0.9874326750448833

The accuracy is close to 98.74%, which is really good. Our spam filter looked at 1,114 messages that it hasn't seen in training, and classified 1,100 correctly.

Next Steps¶

In this project, we managed to build a spam filter for SMS messages using the multinomial Naive Bayes algorithm. The filter had an accuracy of 98.74% on the test set we used, which is a pretty good result. Our initial goal was an accuracy of over 80%, and we managed to do way better than that.

Next steps include:

  • Analyze the 14 messages that were classified incorrectly and try to figure out why the algorithm classified them incorrectly
  • Make the filtering process more complex by making the algorithm sensitive to letter case