Agenda¶
- What is a recommender system?
- Why do we need recommender systems?
- How does it work?
- Collaborative Filtering
- Content-based Filtering
- Tutorial using MovieLens dataset
What is a Recommender System?¶
- an application of machine learning
- predicts a user's preference towards a given item
- aims to drive user engagement
Examples of Recommenders
Examples of Recommenders
Examples of Recommenders
Why Do We Need Recommender Systems?
Before e-Commerce...
Things were sold exclusively in brick-and-mortar stores
limited inventory
mainstream products
Why Do We Need Recommender Systems?
e-Commerce...
Introduction of the online marketplace
unlimited inventory
niche products
Why Do We Need Recommender Systems?
"With the evolution of online retail, however, has come the revelation that being able to recategorize and rearrange products on the fly unlocks their real value." - Chris Anderson
Why Do We Need Recommender Systems?
The Tasting Booth Experiment (Iyengar and Lepper, 2000)
"30% of the consumers in the limited-choice condition subsequently purchased a jar of jam; in contrast, only 3% of the consumers in the extensive-choice condition did so"
Recommender Systems = Machine Learning
Recommender Systems = Machine Learning
Recommender Systems = Machine Learning
Recommender Systems = Machine Learning
Collaborative Filtering¶
- Based on the assumption that similar users like similar things
- "Customers who bought this item also bought..."
- "Because you watched Movie X..."
User-item ("utility") matrix
Content-based Filtering¶
- Generates recommendations using user and item features
- Handles the cold-start problem for new users and items
- User features:
age,gender,spoken language - Item features:
movie genre,year of release,cast
Step 1: Import Dependencies¶
We will be representing our data as a Pandas DataFrame.
- a two-dimensional data structure
columnsrepresent features,rowsrepresent items- analogous to an Excel spreadsheet or SQL table
- documentation can be found here
import numpy as np
import pandas as pd
import sklearn
import matplotlib.pyplot as plt
import seaborn as sns
Ratings¶
Dataframe contains 4 columns:
userIdmovieIdratingtimestamp
We need this data to perform collaborative filtering.
ratings = pd.read_csv("https://s3-us-west-2.amazonaws.com/recommender-tutorial/ratings.csv")
ratings.head()
Movies¶
Dataframe contains 3 columns:
movieIdtitlegenres
We need this data to interpret the results of our collaborative filtering recommender.
movies = pd.read_csv("https://s3-us-west-2.amazonaws.com/recommender-tutorial/movies.csv")
movies.head(3)
Step 3: Exploratory Data Analysis¶
Ratings contains users' ratings for a given movie. Let's see how many ratings, unique movies, and unique users are in our dataset.
n_ratings = len(ratings)
n_movies = ratings['movieId'].nunique()
n_users = ratings['userId'].nunique()
print(f"Number of ratings: {n_ratings}")
print(f"Number of unique movieId's: {n_movies}")
print(f"Number of unique users: {n_users}")
print(f"Average number of ratings per user: {round(n_ratings/n_users, 2)}")
print(f"Average number of ratings per movie: {round(n_ratings/n_movies, 2)}")
How many ratings did each user make? Let's use pandas' groupby() and count() to:
- group the data by userId's
- count the number of ratings for each userId.
user_freq = ratings[['userId', 'movieId']].groupby('userId').count().reset_index()
user_freq.columns = ['userId', 'n_ratings']
user_freq.head()
mean_n_ratings = user_freq['n_ratings'].mean()
print(f"Mean number of ratings for a given user: {mean_n_ratings:.2f}.")
Let's visualize the distribution of movie ratings in this dataset, and distribution of user rating frequency. We can do this using a Python package called seaborn.
sns.set_style("whitegrid")
plt.figure(figsize=(14,3))
# PLOT 1
plt.subplot(1,2,1)
ax = sns.countplot(x="rating", data=ratings, palette="viridis")
ax.set(title="Distribution of movie ratings")
# PLOT 2
plt.subplot(1,2,2)
ax = sns.kdeplot(user_freq['n_ratings'], shade=True, legend=False)
ax.set(title="Number of movies rated per user", xlabel="# ratings per user", ylabel="density")
plt.axvline(user_freq['n_ratings'].mean(), color="k", linestyle="--")
plt.show()
What are the highest and lowest rated movies?¶
To find the "best" and "worst" movies, we need to calculate the mean rating for each movie in our dataset. We can do this by grouping by movieId and calculating the mean of the rating column.
mean_rating = ratings.groupby('movieId')[['rating']].mean()
mean_rating.head()
What are the highest and lowest rated movies?¶
lowest_rated = mean_rating['rating'].idxmin()
movies.loc[movies['movieId'] == lowest_rated]
highest_rated = mean_rating['rating'].idxmax()
movies.loc[movies['movieId'] == highest_rated]
The highest rated movie is Lamerica?!¶
I'm sure that this movie is great but it doesn't sound like a familiar blockbuster. Let's dig deeper and see who rated this movie.
ratings.loc[ratings['movieId']==highest_rated]
While Lamerica may have the "highest" average rating, it only received 2 ratings. This isn't a good measure of a movie's popularity. The quality of a movie's rating depends not only on the average rating but also on the number of ratings.
Bayesian Average¶
- A weighted average that accounts for how many ratings there are
- Useful when there isn't much data available
- Used extensively in baseball statistics (i.e., batting average)
- Calculate with the following equation:
$$r_{i} = \frac{C \times m + \Sigma{\text{ratings}}}{C+N} $$
where:
- $C$ = our confidence (average number of ratings for a given movie)
- $m$ = our prior (global average rating)
- $N$ = total number of ratings for movie $i$
We first need to get the count and mean for each movie in our dataset. We can do this with a sophisticated groupby that applies both mean and count using the agg method.
movie_stats = ratings.groupby('movieId')[['rating']].agg(['count', 'mean'])
movie_stats.columns = movie_stats.columns.droplevel()
movie_stats.head()
Let's calculate the Bayesian average. We first write a function that computes the Bayesian average for a given movie.
C = movie_stats['count'].mean()
m = movie_stats['mean'].mean()
def bayesian_avg(ratings_i):
bayesian_avg = (C*m+ratings_i.sum())/(C+ratings_i.count())
return bayesian_avg
We can apply bayesian_avg to our entire ratings dataset using the agg method.
bayesian_avg_ratings = ratings.groupby('movieId')['rating'].agg(bayesian_avg).reset_index()
bayesian_avg_ratings.columns = ['movieId', 'bayesian_avg']
bayesian_avg_ratings.head()
Which movies have the highest Bayesian average rating?
movie_stats = movie_stats.merge(bayesian_avg_ratings, on='movieId')
movie_stats = movie_stats.merge(movies[['movieId', 'title']])
movie_stats.sort_values('bayesian_avg', ascending=False).head()
Using Bayesian averages, we can see that Lamerica is no longer the top movie.
Which movies have the lowest Bayesian average rating?
movie_stats.sort_values('bayesian_avg', ascending=True).head()
Step 4: Transforming the Data¶
- Need to transform data into user-item matrix for collaborative filtering
- scipy.sparse_matrix: columns are movies and rows are users
- Each cell is populated with a user's rating towards a movie
- Empty cell = no rating available
from scipy.sparse import csr_matrix
def create_X(df):
"""
Generates a sparse matrix from the ratings dataframe.
Args:
df: ratings dataframe
Returns:
X: sparse matrix
user_mapper: dict that maps user id's to user indices
user_inv_mapper: dict that maps user indices to user id's
movie_mapper: dict that maps movie id's to movie indices
movie_inv_mapper: dict that maps movie indices to movie id's
"""
N = df['userId'].nunique()
M = df['movieId'].nunique()
user_mapper = dict(zip(np.unique(df["userId"]), list(range(N))))
movie_mapper = dict(zip(np.unique(df["movieId"]), list(range(M))))
user_inv_mapper = dict(zip(list(range(N)), np.unique(df["userId"])))
movie_inv_mapper = dict(zip(list(range(M)), np.unique(df["movieId"])))
user_index = [user_mapper[i] for i in df['userId']]
movie_index = [movie_mapper[i] for i in df['movieId']]
X = csr_matrix((df["rating"], (movie_index, user_index)), shape=(M, N))
return X, user_mapper, movie_mapper, user_inv_mapper, movie_inv_mapper
X, user_mapper, movie_mapper, user_inv_mapper, movie_inv_mapper =create_X(ratings)
Calculating Sparsity of the Matrix¶
Let's see how sparse our matrix, $X$, is. We can calculate matrix density, $d$, with the following equation:
$d=\frac{\text{# non-zero elements}}{\text{total # elements}}$
density = X.count_nonzero()/(X.shape[0]*X.shape[1])
print(f"Matrix density: {round(sparsity*100,2)}%")
Wow, our matrix is quite sparse. But don't be discouraged! User-item matrices are typically very sparse. A general rule of thumb is that your matrix density should be no lower than 0.5% to generate decent results.
from scipy.sparse import save_npz, load_npz
save_npz('user_item_matrix.npz', X)
We can load it again using load_npz.
X = load_npz('user_item_matrix.npz')
X
Step 5: Finding similar movies using k-Nearest Neighbours¶
k-Nearest Neighbours (kNN) is a classification algorithm that predicts the label of a given sample based on majority vote of its nearest $k$ neighbours.
Distance between two samples can be measured using cosine similarity, Euclidean distance, Manhattan distance, etc.
Cosine Similarity vs. Euclidean Distance¶
- Cosine Similarity: measures similarity of two points in orientation (i.e., the cosine angle between $A$ and $B$)
- the closer the cosine similarity is to 1, the more similar the items are
$$\text{similairity}=\frac{A \cdot B}{\mid{A}\mid\mid{B}\mid}$$
- Euclidean Distance: measures distance between two items in a n-dimensional space (i.e., measures straight line from $A$ to $B$)
- unlike cosine similarity, Euclidean distance takes magnitude into account
$$d(p,q) = \sqrt{(p_1-q_1)^2 + (p_2-q_2)^2 + ... + (p_n-q_n)^2}$$
Cosine Similarity vs. Euclidean Distance¶
- A-B and D-E have the same cosine similarity
- A-B has a smaller Euclidean distance than D-E
Let's create a function that finds $k$ similair movies for a given movie.
from sklearn.neighbors import NearestNeighbors
def find_similar_movies(movie_id, X, k, metric='cosine', show_distance=False):
"""
Finds k-nearest neighbours for a given movie id.
Args:
movie_id: id of the movie of interest
X: user-item utility matrix
k: number of similar movies to retrieve
metric: distance metric for kNN calculations
Returns:
list of k similar movie ID's
"""
neighbour_ids = []
movie_ind = movie_mapper[movie_id]
movie_vec = X[movie_ind]
k+=1
kNN = NearestNeighbors(n_neighbors=k, algorithm="brute", metric=metric)
kNN.fit(X)
if isinstance(movie_vec, (np.ndarray)):
movie_vec = movie_vec.reshape(1,-1)
neighbour = kNN.kneighbors(movie_vec, return_distance=show_distance)
for i in range(0,k):
n = neighbour.item(i)
neighbour_ids.append(movie_inv_mapper[n])
neighbour_ids.pop(0)
return neighbour_ids
We can test out our function by passing in a movieId. We'll create a dictionary that maps movieId to movie title so that we can better interpret our results.
In this case, movie_id = 1 is Toy Story. We'll set k = 10 and use our default metric, cosine similarity. This means that we're looking for the 10 most similar movies to Toy Story.
movie_titles = dict(zip(movies['movieId'], movies['title']))
movie_id = 1
similar_ids = find_similar_movies(movie_id, X, k=10)
movie_title = movie_titles[movie_id]
print(f"Because you watched {movie_title}...")
for i in similar_ids:
print(movie_titles[i])
Let's repeat the process but this time using Euclidean distance.
movie_titles = dict(zip(movies['movieId'], movies['title']))
movie_id = 1
similar_ids = find_similar_movies(movie_id, X, k=10, metric="euclidean")
movie_title = movie_titles[movie_id]
print(f"Because you watched {movie_title}...")
for i in similar_ids:
print(movie_titles[i])
Matrix Factorization¶
- A linear algebra technique that can help discover latent features between users and movies
- Latent features give a more compact representation of user tastes and item descriptions
- Can enhance the quality of recommendations when $X$ is very sparse
- Factorizes the user-item matrix into two "factor matrices":
- user-factor matrix
(n_users, k) - item-factor matrix
(k, n_items)
- user-factor matrix
Singular Value Decomposition¶
- Singular Value Decomposition (SVD) is a type of matrix factorization that is used for data reduction and de-noising
- scikit-learn has a class called
TruncatedSVDthat we can use to reduce our matrix from(n_users, n_movies)to(n_users, n_components)wheren_componentsrepresents number of latent features.
from sklearn.decomposition import TruncatedSVD
svd = TruncatedSVD(n_components=25, n_iter=10)
Z = svd.fit_transform(X.T)
print(f"The shape of our compressed Z matrix is: {Z.shape}.")
Let's apply our find_similar_movies function to our new Z matrix.
movie_id = 1
similar_movies = find_similar_movies(movie_id, X=Z.T, metric='cosine', k=10)
movie_title = movie_titles[movie_id]
print(f"Because you watched {movie_title}:")
for i in similar_movies:
print(movie_titles[i])
When we reduce the dimensions of our original user-item matrix to (n_users, 30), we get the recommendations shown above.
Top N Recommender¶
- Matrix factorization allows us to predict missing ratings in our original $X$ matrix
- Reconstruct matrix by getting inner product of the user-factor matrix and movie-factor matrix
new_X = svd.inverse_transform(Z).T
print(f"Dimensions of original user-item matrix: {X.shape}, {type(X)}")
print(f"Dimensions of SVD-reconstructed X matrix: {new_X.shape}, {type(new_X)}")
Let's try generaing top N recommendations for a user in our dataset. We'll look at userId 5. Which movies did this user rate highly?
userId = 5
user_preferences = ratings[(ratings['userId']==userId)&(ratings['rating']>=4)]
user_preferences = user_preferences.merge(movies[['movieId', 'title']])
user_preferences.sort_values('rating', ascending=False).head(5)
Now, let's predict which movies userId 5 will also like based on their previous ratings. We need to do the following:
- get row from $X$ which represents userId=5
- sort their predicted ratings in descending order
We can use np.argsort() to grab top $N$ indices for userId=5.
top_N = 10
movie_titles = dict(zip(movies['movieId'], movies['title']))
top_N_indices = new_X[user_mapper[userId]].argsort()[-top_N:][::-1]
print(f"Top {top_N} Recommendations for UserId {userId}:")
for i in top_N_indices:
movie_id = movie_inv_mapper[i]
print(movie_titles[movie_id])