Spooky Author Identification
Introduction¶
This notebook tutorial will demonstrate how to use natural language processing to identify authors from text excerpts. The authors include Edgar Allen Poe, HP Lovecraft and Mary Shelley. It will also serve as my initial entry into Kaggle's natural language contest of the same name. The purpose is not to get to the top of the Kaggle leaderboard, but to help beginners develop their understanding of text learning. We'll start by exploring the structure of the data, then clean up any inconsistencies and create a few visualizations before feeding the data into our algorithm.
Load Modules¶
There are a lot of libraries below, so it may be prudent to explain the purpose of a few of the major players. Pandas and Numpy will be used to manipulate our CSV files as necessary, while matplotlib and seaborn will be used to create visualizations so that we can better understand our data. The natural language processing modules will be used to reformat and manipulate our text data so that we can feed it into our machine learning models.
# Data Manipulation
import pandas as pd
import numpy as np
pd.set_option('max_colwidth', 200)
from IPython.display import display
# Visualizations
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [12, 8]
import seaborn as sns
sns.set_style('white')
colors = sns.color_palette()
import missingno as msno
# Misc
from time import time
# Natural Language Processing
import re
from google.cloud import language
from textblob import TextBlob as tb
import nltk
from nltk.corpus import stopwords
from nltk.stem.snowball import SnowballStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.corpus import wordnet
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, TfidfTransformer
# Machine Learning Models
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
Load Data¶
Our data is given to us in the form of two CSV files containing excerpts from each of the above 3 authors. We have a labeled dataset ("train.csv") which we will use to train our classifier and an unlabeled testing dataset aptly named "test.csv".
train = pd.read_csv('data/train.csv')
test = pd.read_csv('data/test.csv')
sample_submission = pd.read_csv('data/sample_submission.csv')
train_df = pd.DataFrame(train)
test_df = pd.DataFrame(test)
Data Structure¶
Our training data is composed of 19,579 records and 3 variables: id, text and author. According to the contest page, the id column is a unique identifier for each sentence, which doesn't seem very useful right now. Assuming all exerpts were randomly selected, there shouldn't be any clear patterns from each exerpt's ID.
display(train_df.shape)
display(train_df.info())
display(train_df.describe())
Feature Engineering¶
The unicode characters aren't playing nicely with textblob or nltk. Let's convert them to ascii strings for now. We risk losing information from special characters, but if that becomes an issue we'll come back to this later.
train_df.text = train_df.text.str.decode('utf-8', 'ignore')
Stem Words¶
Our next step will be to convert our words into stem words. Stemming is the process of reducing words into their word stem. It may be helpful to go into detail here. For example, let's say we are trying identify whether an excerpt came from an MLA style guide or a baseball article and our classifier encountered the following words:
- formater
- formatted
- formats
- formatting
The above words would all be converted to their stem: format. "So what" you might say. "Why is this even useful?" The main advantage of stemming is that it condenses our featureset, which will help improve the accuracy of our classifier. As we will see further below, text classification involves sparse data, which actually hurts the accuracy of our models (more data is not always more helpful). By stemming our words beforehand, we can reduce the number of features two-fold or three-fold for English words and even more than that in other languages.
def get_stems(words):
stemmer = SnowballStemmer("english")
stems = []
if len(words)>0:
for word in words:
stems.append(stemmer.stem(word))
return " ".join(stems)
stemmer = SnowballStemmer("english")
stemmer.stem('reduced')
Lemmas¶
We will also apply lemmatization to our features. Lemmatization is very similar to stemming in that it involves reducing words, but it does so in a more nuanced way. Whereas stemming applies a coarse algorithm to chop off the ends of words, lemmatization reduces words properly, taking into account part of speech and inflection.
Stemming often results in a non-word whereas lemmatization gets to the morphological base of a word. For example, consider the word 'reduced'. Stemming would convert that word to 'reduc', which is not a word. Lemmatizing the verb 'reduced' would result in its base word 'reduce'.
For our purposes, we will use lemmatization to reduce the number of features in our dataset, but there are other use cases for this technique. Both lemmatization and stemming can be used to improve the accuracy of search algorithms. If a user enters the search terms "code" and "challenges", it would be helpful to include websites that contain the phrase "coding challenge".
def get_lemmas(blob):
lemmatizer = WordNetLemmatizer()
lemmas = []
for word, pos in blob.tags:
wordnet_pos = get_wordnet_pos(pos)
if wordnet_pos:
lemmas.append(lemmatizer.lemmatize(word, wordnet_pos))
else:
lemmas.append(lemmatizer.lemmatize(word))
return lemmas
train_df['text_blobs'] = train_df.text.apply(tb)
train_df['text_blob_words'] = train_df.text_blobs.apply(lambda x: (x.words))
train_df['lemmas'] = train_df.text_blobs.apply(get_lemmas)
train_df['lemma_stems'] = train_df.lemmas.apply(get_stems)
train_df['lemmas'] = train_df.lemmas.apply(lambda x: " ".join(x))
train_df['stems'] = train_df.text_blob_words.apply(get_stems)
train_df['words'] = train_df.text_blob_words.apply(lambda x: " ".join(list(x)))
train_df['word_count'] = train_df.text_blob_words.apply(len)
train_df.head()
Visualizations¶
Breaking down the excerpts by author, we can see that the data is slightly unbalanced with Edgar Allen Poe contributing about 25% more texts than Mary Shelley or HP Lovecraft.
author_counts = train_df.author.value_counts()
sns.barplot(x=author_counts.index, y=author_counts.values)
plt.show()
plt.hist(train_df[train_df.word_count<120].word_count, bins=50)
plt.show()
sns.distplot(
train_df[(train_df.author=='EAP') & (train_df.word_count<120)].word_count,
color=colors[0]
)
sns.distplot(
train_df[(train_df.author=='MWS') & (train_df.word_count<120)].word_count,
color=colors[1]
)
sns.distplot(
train_df[(train_df.author=='HPL') & (train_df.word_count<120)].word_count,
color=colors[2]
)
plt.show()
Edgar Allen Poe appears to have the lowest average word count, followed by Mary Shelley, but it's difficult to tell whether it's significant. In the next iteration, that may be a useful feature, but we'll omit it for now.
Modeling¶
Now that we have a better understanding of our data structure, we can build our machine learning models. We will vectorize our features before feeding them into our algorithms. Vectorization converts a corpus into a sparse matrix of numbers. Every word in the corpus has a unique location within the vector.
Count vectorization transforms each word into a column and transforms each document into a row. Each cell contains a count of the number of times that column's word occurs. While this method can be useful, we may be more interested in words that are unique to certain documents, as opposed to the most popular words. For this reason, we will be using term frequency document inverse frequency (TF-IDF).
In TF-IDF vectorization, columns still represent words and rows still represent documents, but it goes a step further by measuring how unique a word is across all documents. Each cell in the TF-IDF matrix is a product of how frequently the word occurs in a document by how unique it is in the entire corpus of documents. In this exploration, we will be using TF-IDF for its enhanced discriminating power.
There are a number of hyperparameters that can be tuned in tfidf, which can be found here. We will focus on bigrams and unigrams. When tokenizing our documents, we can break them down to individual words like "Frankenstein", "raven" or "octopus", or into phrases like "you know", "down by", or "any day now". There are advantages and disadvantages to each approach, but we will not know which will lead to better accuracy until we implement them. It is tempting to just use the widest variety of n-grams our processor can handle, but again, more data is not always more helpful. Generating millions of features can actually hurt our accuracy.
Unigram Vectorization¶
We'll start with unigrams and try several of the features we engineered earlier on in the analysis. It's not clear whether words, stems, lemmas or stemmed lemmas will yield the highest accuracy, so we'll try all four in the next coding blocks.
With Stemming¶
def vectorize(name, data):
vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(data)
print "{0} Vector Shape: {1}".format(name, vectors.shape)
return vectors
def get_scores(vectors, df):
#x_train, x_test, y_train, y_test = train_test_split(vectors, df.author, test_size=.3, random_state=42)
nb = MultinomialNB(alpha=.1)
svm = LinearSVC(random_state=42)
logr = LogisticRegression(random_state=42)
sgd = SGDClassifier(random_state=42, max_iter=1000, tol=1e-3)
rf = RandomForestClassifier(random_state=42)
classifiers = [
("Naive Bayes", nb),
("Support Vector Machines", svm),
("Logistic Regression", logr),
("Stochastic Gradient Dissent", sgd),
("Random Forests", rf)]
scores = []
for classifier in classifiers:
name = classifier[0]
clf = classifier[1]
accuracies = cross_val_score(clf, vectors, df.author, cv=3)
scores.append((name, np.mean(accuracies)))
#clf.fit(x_train, y_train)
#pred = clf.predict(x_test)
#scores.append((name, accuracy_score(pred, y_test)))
return scores
def plot_scores(scores):
scores_df = pd.DataFrame(scores, columns=["name", "score"])
scores_df.score *= 100
plt.bar(range(len(scores_df.name)), scores_df.score, alpha=.7, color=colors[0])
plt.xlabel('Model')
plt.ylabel('Accuracy')
plt.xticks(range(len(scores_df.name)), scores_df.name)
for x,y in zip(range(len(scores_df.name)), scores_df.score):
plt.text(x, y, "{0:0.1f}".format(y), ha='center', va='bottom')
plt.show()
vectors = vectorize("Stem", train_df.stems)
scores = get_scores(vectors, train_df)
plot_scores(scores)
Lemmatization¶
vectors = vectorize("Word", train_df.lemmas)
scores = get_scores(vectors, train_df)
plot_scores(scores)
Stemming and Lemmatization¶
vectors = vectorize("Word", train_df.lemma_stems)
scores = get_scores(vectors, train_df)
plot_scores(scores)
Without Stemming¶
vectors = vectorize("Word", train_df.words)
scores = get_scores(vectors, train_df)
plot_scores(scores)
Raw words led to the highest accuracy by a narrow margin (84% vs 83.7% for stems and 83.6% for lemmas). Because the results were so close, it may be prudent to test all 3 when using bigrams.
Bigram Vectorization¶
With Stemming¶
vectorizer = TfidfVectorizer(ngram_range=(1,2))
vectors = vectorizer.fit_transform(train_df.stems)
vectors.shape
scores = get_scores(vectors, train_df)
plot_scores(scores)
With Lemmatization¶
vectorizer = TfidfVectorizer(ngram_range=(1,2))
vectors = vectorizer.fit_transform(train_df.lemmas)
vectors.shape
scores = get_scores(vectors, train_df)
plot_scores(scores)
Without Stemming¶
vectorizer = TfidfVectorizer(ngram_range=(1,2))
vectors = vectorizer.fit_transform(train_df.words)
vectors.shape
scores = get_scores(vectors, train_df)
plot_scores(scores)
Again, our stemless words came out on top with 85.1% accuracy this time. Not bad for the humbly named Naive Bayes approach.
Conclusions¶
Overall, this was a great first step. The 85.1% accuracy is not as high as I had hoped but it is a great baseline from which to improve. In the next iteration, I will use GridSearchCV to optimize our parameters and spend more on feature engineering. Thanks for taking the time to read this.