{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Recommender Systems in Keras\n", "> A programming introduction to recommender systems using Keras!\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Nipun Batra\n", "- categories: [ML]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I have written a [few](../nnmf-tensorflow.html) [posts](../nmf-autograd.html) [earlier](../nmf-cvx.html) [about](../nmf-out-matrix.html) [matrix](../mf-autograd-adagrad.html) [factorisation](../contrained-nmf-cvx.html) [using](../nmf-nnls.html) various Python libraries. The main application I had in mind for matrix factorisation was [recommender systems](https://en.wikipedia.org/wiki/Recommender_system). In this post, I'll write about using [Keras](https://keras.io) for creating recommender systems. [Various](https://github.com/maciejkula/triplet_recommendations_keras) [people](http://blog.richardweiss.org/2016/09/25/movie-embeddings.html) [have](https://github.com/bradleypallen/keras-movielens-cf) [written](https://github.com/hexiangnan/neural_collaborative_filtering) [excellent](https://github.com/sonyisme/keras-recommendation) [similar](http://course.fast.ai/lessons/lesson4.html) [posts](https://github.com/maciejkula/spotlight) and code that I draw a lot of inspiration from, and give them their credit! I'm assuming that a reader has some experience with Keras, as this post is not intended to be an introduction to Keras.\n", "\n", "Specifically, in this post, I'll talk about:\n", "\n", "1. Matrix Factorisation in Keras\n", "2. Adding non-negativitiy constraints to solve non-negative matrix factorisation (NNMF)\n", "3. Using neural networks for recommendations\n", "\n", "I'll be using the Movielens-100k dataset for illustration. There are 943 users and 1682 movies. In total there are a 100k ratings in the dataset. It should be noted that the max. total number of rating for the would be 943*1682, which means that we have about 7% of the total ratings! All rating are on a scale of 1-5. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### **Task**\n", "\n", "Given this set of ratings, can we recommend the next set of movies to a user? This would translate to: for every user, estimating the ratings for all the movies that (s)he hasn't watched and maybe recommend the top-k movies by the esimtated ratings! " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Peak into the dataset" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dataset = pd.read_csv(\"/Users/nipun/Downloads/ml-100k/u.data\",sep='\\t',names=\"user_id,item_id,rating,timestamp\".split(\",\"))" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_iditem_idratingtimestamp
01962423881250949
11863023891717742
2223771878887116
3244512880606923
41663461886397596
\n", "
" ], "text/plain": [ " user_id item_id rating timestamp\n", "0 196 242 3 881250949\n", "1 186 302 3 891717742\n", "2 22 377 1 878887116\n", "3 244 51 2 880606923\n", "4 166 346 1 886397596" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, each record (row) shows the rating for a user, item (movie) pair. It should be noted that I use item and movie interchangeably in this post." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(943, 1682)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(dataset.user_id.unique()), len(dataset.item_id.unique())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We assign a unique number between (0, #users) to each user and do the same for movies." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dataset.user_id = dataset.user_id.astype('category').cat.codes.values\n", "dataset.item_id = dataset.item_id.astype('category').cat.codes.values" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_iditem_idratingtimestamp
01952413881250949
11853013891717742
2213761878887116
3243502880606923
41653451886397596
\n", "
" ], "text/plain": [ " user_id item_id rating timestamp\n", "0 195 241 3 881250949\n", "1 185 301 3 891717742\n", "2 21 376 1 878887116\n", "3 243 50 2 880606923\n", "4 165 345 1 886397596" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train test split\n", "\n", "We'll now split our dataset of 100k ratings into train (containing 80k ratings) and test (containing 20k ratings). Given the train set, we'd like to accurately estimate the ratings in the test set." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "train, test = train_test_split(dataset, test_size=0.2)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_iditem_idratingtimestamp
90092832122875036139
50879941323888954341
67994436124880141129
497697103444884485683
110321217364879270874
\n", "
" ], "text/plain": [ " user_id item_id rating timestamp\n", "90092 832 12 2 875036139\n", "50879 94 132 3 888954341\n", "67994 436 12 4 880141129\n", "49769 710 344 4 884485683\n", "11032 121 736 4 879270874" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train.head()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_iditem_idratingtimestamp
892849074933879723046
60499550254892785056
110903732225880394520
360961991404884129346
21633713175880037702
\n", "
" ], "text/plain": [ " user_id item_id rating timestamp\n", "89284 907 493 3 879723046\n", "60499 550 25 4 892785056\n", "11090 373 222 5 880394520\n", "36096 199 140 4 884129346\n", "21633 71 317 5 880037702" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Matrix factorisation\n", "\n", "One popular recommender systems approach is called Matrix Factorisation. It works on the principle that we can learn a low-dimensional representation (embedding) of user and movie. For example, for each movie, we can have how much action it has, how long it is, and so on. For each user, we can encode how much they like action, or how much they like long movies, etc. Thus, we can combine the user and the movie embeddings to estimate the ratings on unseen movies. This approach can also be viewed as: given a matrix (A [M X N]) containing users and movies, we want to estimate low dimensional matrices (W [M X k] and H [M X k]), such that: $A \\approx W.H^T$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Matrix factorisation in Keras\n", "\n", "We'll now write some code to solve the recommendation problem by matrix factorisation in Keras. We're trying to learn two low-dimensional embeddings of users and items.\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import keras\n", "from IPython.display import SVG\n", "from keras.optimizers import Adam\n", "from keras.utils.vis_utils import model_to_dot\n", "n_users, n_movies = len(dataset.user_id.unique()), len(dataset.item_id.unique())\n", "n_latent_factors = 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The key thing is to learn an embedding for movies and users, and then combine them using the dot product! For estimating the rating, for each user, movie pair of interest, we'd take the dot product of the respective user and item embedding. As an example, if we have 2 dimensions in our user and item embedding, which say correspond to [how much user likes action, how much user likes long movies], and the item embedding is [how much action is in the movie, how long is the movie]. Then, we can predict for a user `u`, and movie `m` as how much `u` likes action $\\times$ how much action is there in `m` $+$ how much `u` likes long movies $\\times$ how long is `m`.\n", "\n", "Our model would optimise the emebedding such that we minimise the mean squared error on the ratings from the train set." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "movie_input = keras.layers.Input(shape=[1],name='Item')\n", "movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors, name='Movie-Embedding')(movie_input)\n", "movie_vec = keras.layers.Flatten(name='FlattenMovies')(movie_embedding)\n", "\n", "user_input = keras.layers.Input(shape=[1],name='User')\n", "user_vec = keras.layers.Flatten(name='FlattenUsers')(keras.layers.Embedding(n_users + 1, n_latent_factors,name='User-Embedding')(user_input))\n", "\n", "prod = keras.layers.merge([movie_vec, user_vec], mode='dot',name='DotProduct')\n", "model = keras.Model([user_input, movie_input], prod)\n", "model.compile('adam', 'mean_squared_error')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here's a visualisation of our model for a better understanding." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "G\n", "\n", "\n", "4651743104\n", "\n", "Item: InputLayer\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1)\n", "\n", "\n", "4651743216\n", "\n", "Movie-Embedding: Embedding\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1, 3)\n", "\n", "\n", "4651743104->4651743216\n", "\n", "\n", "\n", "\n", "4651744392\n", "\n", "User: InputLayer\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1)\n", "\n", "\n", "4651743888\n", "\n", "User-Embedding: Embedding\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1, 3)\n", "\n", "\n", "4651744392->4651743888\n", "\n", "\n", "\n", "\n", "4651744000\n", "\n", "FlattenMovies: Flatten\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1, 3)\n", "\n", "(None, 3)\n", "\n", "\n", "4651743216->4651744000\n", "\n", "\n", "\n", "\n", "4468062472\n", "\n", "FlattenUsers: Flatten\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1, 3)\n", "\n", "(None, 3)\n", "\n", "\n", "4651743888->4468062472\n", "\n", "\n", "\n", "\n", "4651881696\n", "\n", "DotProduct: Merge\n", "\n", "input:\n", "\n", "output:\n", "\n", "[(None, 3), (None, 3)]\n", "\n", "(None, 1)\n", "\n", "\n", "4651744000->4651881696\n", "\n", "\n", "\n", "\n", "4468062472->4651881696\n", "\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "SVG(model_to_dot(model, show_shapes=True, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that in the `Merge` layer, we take the dot product of the user and the item embeddings to obtain the rating.\n", "\n", "We can also summarise our model as follows:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "__________________________________________________________________________________________________\n", "Layer (type) Output Shape Param # Connected to \n", "==================================================================================================\n", "Item (InputLayer) (None, 1) 0 \n", "__________________________________________________________________________________________________\n", "User (InputLayer) (None, 1) 0 \n", "__________________________________________________________________________________________________\n", "Movie-Embedding (Embedding) (None, 1, 3) 5049 Item[0][0] \n", "__________________________________________________________________________________________________\n", "User-Embedding (Embedding) (None, 1, 3) 2832 User[0][0] \n", "__________________________________________________________________________________________________\n", "FlattenMovies (Flatten) (None, 3) 0 Movie-Embedding[0][0] \n", "__________________________________________________________________________________________________\n", "FlattenUsers (Flatten) (None, 3) 0 User-Embedding[0][0] \n", "__________________________________________________________________________________________________\n", "DotProduct (Merge) (None, 1) 0 FlattenMovies[0][0] \n", " FlattenUsers[0][0] \n", "==================================================================================================\n", "Total params: 7,881\n", "Trainable params: 7,881\n", "Non-trainable params: 0\n", "__________________________________________________________________________________________________\n" ] } ], "source": [ "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, we have 7881 parameters to learn! Let's train our model now!" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "history = model.fit([train.user_id, train.item_id], train.rating, epochs=100, verbose=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Train error v/s epoch number" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we test how well our model does in the test setting, we can visualise the train loss with epoch number." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGplJREFUeJzt3X+QJOV93/HPt+f37uyvuz0O2OM40F0knbFlyFkisqJy\nTOwgxYiUnIpEyYmjEBO74oTEThTkOKkkZVe5XCmVrQTLQRKRquJIUWFcAlsGq7AUWzGWOBASYEwB\nZ3HcccBxe7d7+2t+fvNH9+zOHXO7s7PdO9sz71fV1Mx09858twv2c08//TyPubsAALhY0O8CAAA7\nEwEBAOiIgAAAdERAAAA6IiAAAB0REACAjggIAEBHBAQAoCMCAgDQUbbfBWzF9PS0HzhwoN9lAECq\nPP7442+4+56Njkt1QBw4cEBHjx7tdxkAkCpm9lI3x3GJCQDQEQEBAOiIgAAAdERAAAA6IiAAAB0R\nEACAjggIAEBHqQ6I2cVqv0sAgIFFQAAAOkp1QDTd+10CAAysVAdEo0lAAEBSUh0Q5AMAJCflAeFy\nLjMBQCJSHRCStFRt9LsEABhIqQ+IxUq93yUAwEBKfUAsEBAAkIjUBwSXmAAgGakPCFoQAJCMHRMQ\nZnatmX3WzO7bzM/RBwEAyUg0IMzsXjN73cyevmj7zWb2nJm9YGZ3SZK7H3P32zf7HbQgACAZSbcg\nPifp5vYNZpaRdLek90k6LOk2Mzvc6xcsVuiDAIAkJBoQ7v4nkmYv2vxOSS9ELYaqpC9KurXX7+AS\nEwAkox99EDOSXm57f0LSjJntNrPflnS9mX38Uj9sZneY2VEzOypxiQkAkpLtdwEt7n5G0s92cdw9\nku6RpNKVf81pQQBAMvrRgjgp6aq29/uibZsWmLTIOAgASEQ/AuIxSYfM7Bozy0v6sKQHevmgwIw+\nCABISNK3uX5B0qOS3mpmJ8zsdnevS/p5SQ9LelbSl9z9mV4+PwgICABISqJ9EO5+2yW2f0XSV7b6\n+RkzOqkBICE7ZiR1L8I+CAICAJKQyoAws1vM7J5Go85AOQBISCoDwt0fdPc7ivk8l5gAICGpDIgW\nOqkBIDnpDggL14NoNlmXGgDilu6ACEyStFSjHwIA4pbqgMhYGBBcZgKA+KU6IIIoIOioBoD4pTsg\nouppQQBA/FIdEBlaEACQmFQGRGug3OLioiRWlQOAJKQyIFoD5SbGxyRxiQkAkpDKgGihkxoAkpPq\ngMhE1S8xYR8AxC7VAbHWgqAPAgDiluqAkKTRfIY+CABIQPoDopAlIAAgAakPiHIhSyc1ACQg9QFB\nCwIAkpHKgGgNlJubm9NIPsNAOQBIQCoDYnWg3MQEl5gAICGpDIh2o4Us4yAAIAEDERCMgwCA+KU+\nIMoFxkEAQBJSHxCjhayWaw01WJcaAGKV+oAoF7KSpEX6IQAgVqkPiNFWQHCZCQBiRUAAADpKf0Dk\nM5KY0RUA4pb+gKAFAQCJSGVAtE+1USYgACARqQyI9qk2RrmLCQASkcqAaDdaoA8CAJKQ+oDgEhMA\nJCP1AVHKZRQYAQEAcUt9QJiZRvNM+Q0AcUt9QEisKgcASRiQgGBVOQCI24AERJbbXAEgZoMREHku\nMQFA3AYjIFhVDgBiNxABwapyABC/gQgI7mICgPilMiDaJ+uTwtHUjIMAgHilMiDaJ+uTwhZEpd5U\nvdHsc2UAMDhSGRAXW1sTgo5qAIjLQAREOZrRlbEQABCfgQiIVguCfggAiM9ABESZgACA2A1EQIwV\nw4A4v0JAAEBcBiIgyoWcJGmBgACA2AxEQLRaEAuVWp8rAYDBMRABUeYSEwDEbt2AMLPAzN61XcX0\najRPJzUAxG3dgHD3pqT/sU219CwTmEbzGVoQABCjbi4xfc3Mbk28ki0aK+bopAaAGGW7OOYfS7rT\nzCqSliWZJHf3XUkWtlnlIhP2AUCcugmI6cSriEG5kNV5AgIAYrNhQLh7w8zeL+m90aavu/tDyZa1\nPjO7RdItBw8eXN02VsxqYYXbXAEgLhv2QZjZr0r6mKRj0eNjZvYrSRe2noun+5aiFgR9EAAQm24u\nMd0i6Xp3b0iSmd0r6QlJv5xkYZs1Rh8EAMSq24Fy422vx5IoZKvKBe5iAoA4ddOC+HVJT5jZIwrv\nYPoRSf8hyaJ6US5mtVCtq9l0BYH1uxwASL11A8LMTNIjkr4mqTWi+j+6+8mkC9ussUJW7tJSrbE6\n/TcAoHfr/iV1dzezr7r7dZLu36aaerI2H1ONgACAGHTTB/GkmV2feCVbtDqjK/0QABCLbv6pfb2k\nx8zsRUmLWhtJfUOilW1Sq9XAYDkAiEc3AfGBxKuIAS0IAIjXRp3UGUkPuPv3bVM9PVtdVY4WBADE\nYqPpvhuSjpnZzDbV07P2TmoAwNZ1c4mpLOlZM3tUYR+EJMndP5hYVT0YY1U5AIhVNwHR13mXusWq\ncgAQr0sGhJkdcvfn3f0RM8u6e71t3w9tT3nda60qRyc1AMRjvT6I/9P2+lsX7duRy5CyaBAAxGe9\ngLBLvO70fkdgym8AiM96AeGXeN3p/Y4wVswxUA4AYrJeJ/U+M/uEwtZC67Wi9zvytldWlQOA+KwX\nEB+/xGtJ+qUEatmyciGr1+ZX+l0GAAyESwaEu392OwvZjE5rUkthQHAXEwDEo9sV5XaUTmtSS+Fd\nTHRSA0A8UhkQlzJWzK2uKgcA2JrBCoi2VeUAAFuz4VQbZjYt6Z9IOtB+vLvfkVxZvSm3TfnNqnIA\nsDXd/BX9sqQ/l/QNSTv6n+atUFio1CQV+1sMAKRcNwEx6u6/mHglMWjN6DpPRzUAbFk3fRB/aGY/\nnnglMWBVOQCITzcB8bOSHjKzBTObNbOzZjabdGG9YFU5AIhPN5eYphOvIiZlWhAAEJsN14OQdKn1\nqL+bTEm9a3VSzzMfEwBs2XotiLsk3S7p7g77XNJ7E6loC9buYqIFAQBbtd5cTLdHz39z+8rZGlaV\nA4D4dDWazMzeJumw2gYXuPv/TqqorWBVOQCIRzcjqX9Z0o9LepukhyX9HYWD5nZmQBSyLBoEADHo\n5jbXD0n6W5JOufs/lPQOSaOJVrUF5WKOGV0BIAbdBMSyuzck1c1sTNKrkq5OtqzejbOqHADEops+\niG+b2aSkeyUdlTQv6VuJVrUFrCoHAPFYNyDMzCT9J3c/J+luM3tY0ri7P7Et1fWAVeUAIB7rBoS7\nu5l9VdJ10fsXtqWqLSgX6aQGgDh00wfxpJldn3glMRkrhLe5sqocAGzNelNtZN29Lul6SY+Z2YuS\nFiWZwsbFDdtU46aMFXOrq8qxaBAA9G69v6DfknSDpA9sUy2xYFU5AIjHen9BTZLc/cVtqiUWrCoH\nAPFYLyD2mNkvXGqnu38igXq2rNWCYLAcAGzNegGRkVRW1JJIi7ECAQEAcVgvIE65+3/ZtkpiMlZk\nVTkAiMN6t7mmquXQwqpyABCP9QLipm2rIkatTmoGywHA1lwyINx9djsL2Qwzu8XM7pmbm3vTvtW7\nmGhBAMCWdDOSesdx9wfd/Y6JiYk37csEprFCVmeXqn2oDAAGRyoDYiNXTpZ08txyv8sAgFQbyIDY\nN1XSibMEBABsxUAGxMxUSSfPLvW7DABItYEMiH1TJc2v1DXPynIA0LMBDYgRSdJJLjMBQM8GMiBm\nJkuSRD8EAGzBQAbEvqlWQNAPAQC9GsiA2DWaVzEXcIkJALZgIAPCzLRvaoRLTACwBQMZEFLYD8Fg\nOQDo3cAGRDhYjj4IAOjVwAbEzFRJZ5dqWmRWVwDoycAGxOpYCC4zAUBPBjgguNUVALZicAMiGizH\nra4A0JuBDYjpckH5bMCtrgDQo4ENiCAwzUwy7TcA9GpgA0KKbnWlkxoAejLQATEzyboQANCrgQ6I\nfVMlvbFQ1Uqt0e9SACB1BjogZqaY9hsAejXQAdEaLMdYCADYvAEPiGgsBB3VALBpAx0Ql40VlQ2M\nS0wA0IOBDohMYLpyssRoagDowUAHhKRosBx9EACwWQMfEPumSjo+uyx373cpAJAqAx8Q379vQm8s\nVOiHAIBNGviAuPHa3ZKkR4+d6XMlAJAuAx8Qhy4ra9doXn9OQADApgx8QJiZbrx2l755bJZ+CADY\nhIEPCCm8zHTy3DL9EACwCUMTEBL9EACwGUMREPRDAMDmDUVA0A8BAJs3FAEhSe+6hn4IANiMoQmI\nVj8El5kAoDtDExBr/RCz/S4FAFJhaAIiCEzvumYXLQgA6NLQBIS0Nh7i5VlmdwWAjQxVQLzn0LQk\n6f4nTva5EgDY+YYqIN6yp6wfO7xXn/3GMc2v1PpdDgDsaDsmIMxs1Mw+b2afNrOPJPU9d950SPMr\ndX3u/30vqa8AgIGQaECY2b1m9rqZPX3R9pvN7Dkze8HM7oo2f1DSfe7+M5I+kFRN181M6McO79Vn\n/pRWBACsJ+kWxOck3dy+wcwyku6W9D5JhyXdZmaHJe2T9HJ0WCPJomhFAMDGEg0Id/8TSRcPPHin\npBfc/Zi7VyV9UdKtkk4oDInE66IVAQAb60cfxIzWWgpSGAwzku6X9JNm9ilJD17qh83sDjM7amZH\nT58+3XMRrVbEb3/9xZ4/AwAGWbbfBbS4+6Kkj3Zx3D2S7pGkI0eO9Dzz3nUzE/rgDTP6ra+/qJmp\nkj7yrqt7/SgAGEj9CIiTkq5qe78v2rbtfu2DP6BzSzX9+997WoVsRn//r+/b+IcAYEj04xLTY5IO\nmdk1ZpaX9GFJD/ShDuWzgX7rIzfoPQen9bH7vqMvP8kAOgBoSfo21y9IelTSW83shJnd7u51ST8v\n6WFJz0r6krs/k2Qd6ynmMvr0PzqiIwd26c4vPqlf+NKTOjXHlOAAYGleQOfIkSN+9OjRWD5rqVrX\nJx95Qfd+468UBNI/e+9b9NEfPqDJkXwsnw8AO4WZPe7uRzY8joC40MuzS/q1P/xL/cFTp5QNTO8+\nOK33X3e5bnr7Xu0ZK8T6XQDQDwMdEGZ2i6RbDh48+DPPP/98It/xzCtzevA7p/SVp07peDT76/5d\nI7ph/6TecdWkrpke1dW7RzUzWVI+u2NmLAGADQ10QLQk0YK4mLvrmVfm9WcvvqEnXjqnJ46f1evn\nK6v7A5P2jhd1xURRV0yUtHe8qN3lvHaNrj2mRsLn8WJW2QxhAqC/ug2IHTMOYqcyM103M6HrZiYk\nhYFx+nxFL80u6aUzSzp+ZlEnz63o1Nyynj01r68/97oWq5eeKaSYC1Qu5FQuZDSSz2okn9FIIauR\nXEalfEbFXEYj+YxKbe+LuUDFbNvr6LmQbX/OqBAdl8uYzGy7ThGAAUVAbJKZ6bLxoi4bL+qHDuzq\neMxKraGzS1WdWajq7FJVs4vhY365roVKTQuVhhYqdS1X61qqNjS3XNNrcytarjXCR7WhpWpdzR4b\nd4GFd2eVcmuhshpGUSCV81mNFrJhUBWyGs2HgTVeymm8mNXESE6TpbwmR3Iq5jJbOGMA0oqASEAx\nl9EVEyVdMVHq+TPcXbWGa7na0Eq9oZVaQyu1ZvTc0Eo9fF1pPUf7K/W24+oNLVfD18u1hhYrdb2x\nUNXi7JIWK3UtVhparNa10VXGUi6jXaN57S7ntXs0r93lgna3Lp+N5rWnXNBl4wXtHS9q10heQUDr\nBRgEBMQOZWbKZ035bKAJ5RL7nmbTtVJvaKkaBsj8cl3zKzXNLdd0bqmms0tVnVuqanaxpjOLFZ1e\nqOjZU+c1u1hVtdF80+dlA1vtk7l8oqiZqZL27xrR/l0jumpqRFfSqQ+kBgEx5ILAostPWU2Xu7+N\n1921WG1odqGq0wsVnT6/otfmK3p1fkWvza3olbllPXVyTg8/86pqjbUmipl0+XhR+6ZK2r9rVNdM\nj+jq3aO6encYIhOlHP0nwA5BQKAnZqZyIatyIav9u0cueVyj6XptfkXHZ5f08uySTpxd1omzy3p5\ndknfeOG0fveJygXHjxWz2r9rRAemR3Xt9KgO7B7VgekRXbVrRHvKBcID2EYEBBKVCUxXTpZ05WRJ\nN167+037l6p1vXQmvCPsxNkwRF6aXdIzJ+f00NOvqtHWUz+Sz+iqqRFdtaukfVMjq5eurt4dBgid\n6UC8UhkQbQPl+l0Ktmgkn9XbrxjX268Yf9O+ar2pl88u6fiZJb10ZlHHZ5d1fHZRJ84u69EXz7zp\nduLpckGXTxS0d6yovRNFzUyWdOVkUVdOlHT5RFGXjRVVyhMiQLcYKIdUcnedXapFwRG2QF45t6zX\n5lf06nxFr84t6+zSm1cLHC9mddl4cfXOqz3lgqbHwruyWs9TI+HdWaP5DJe0MJAYKIeBZmarI9Wv\n3z/V8ZjlakOvzC1HwVHRa/Mrem1+Ra/Ph3djffv4OZ0+X9FyrfPAxnwmWL2Vd/doXhOlnMZLOU2U\nchorhv0vo1E/zORITpMjOU2N5DVWzKqUI1yQfgQEBlYpn9Fb9pT1lj3ldY8Lx4dU9MZCRbOLF97a\nO7tY0exiVWcWq3plblnz0e2/9Q1GMQam1U789oGIo4WMRqNgGc1nVgNmJJ9VKR+NmG+NpI9G01/w\nnMswzgTbhoDA0Gv9wb5692hXx7u7KvWmFip1LVbqOr9S1/xyTWejcSMLlboWVurhc6Wu5Wo4IHGx\nUtcr52qrrxcqda3U3jyWZCOFbLA6Ur6QC1TIBspnwylXCtkgemSUzwbKZVr72h65zNpzJlAua8oG\ngXIZUy4TrD7yWVM+k1Eua8pH24LAlI0erc/JBkztMqgICGCTzCyawiSzqbEjndQbTS1GU6u0RsAv\nt0bL18KR8OH0K3Ut18IBjcu1hlaq4etqo6lKrRk+1xuq1Jo6v1LXSq2hWqOpWiMMs1prf7254cj5\nzTKTAjNZ9DoTtAeNrYZUPhMoE9jq8YHZ6v5sJgyawMLwyQSmIDBlTGuhFB3TCrNsxpQJAmXMlAnC\n40zh55vW6shmTLkgDLcgqi8TWPRz4Xe2fo/wsbYvE313EB1n7b9v2++RCaRMEKwemzFTEEjZIFBg\nkiksqlVbSzYILjxuhwUtAQH0UTYTaKIUaKKU3Gj5du6uaqOpar25Ok1LveGrYRI+N1ePaW2r1puq\nN12NZlONpla3tUKn6S53qelS08Ofqbf9bOvzmu5qelhHvemrxyxWG2o2ve07wuMaTV991JuuerOp\nRiP8HerR9kHUCp9MFEQWhUzQFk6tAJNaYRUF0UWfkc2EnxNl1KZCiIAAhoiZRZeiMhrrdzExcF8L\nklZIucLnMIDCIKk1wpZTo+lquLeF0drPSGsB19rfjI6vN13e+vzoGI++v9EM3zcuCDip4a5GoymP\nPvfiO0bdo2OioGxG+3319wp/t2bT27atfX/rdwl/zNW86Gpl0y/8XaPD5HI90uX5JSAApFZ4OUjR\nv6TRrU/9VHfHpXLWNDO7xczumZub63cpADCwUhkQ7v6gu98xMTHR71IAYGClMiAAAMkjIAAAHREQ\nAICOCAgAQEcEBACgIwICANBRqteDMLPzkp7rdx07yLSkN/pdxA7BubgQ5+NCw34+rnb3PRsdlPaR\n1M91s+jFsDCzo5yPEOfiQpyPC3E+usMlJgBARwQEAKCjtAfEPf0uYIfhfKzhXFyI83EhzkcXUt1J\nDQBITtpbEACAhKQyIMzsZjN7zsxeMLO7+l3PdjOzq8zsa2b2F2b2jJndGW3fZWZfNbPno+epfte6\nXcwsY2bfNrPfj94P87mYNLP7zOwvzexZM/sbQ34+/nX0/8nTZvYFMysO8/nYjNQFhJllJN0t6X2S\nDku6zcwO97eqbVeX9IvufljSjZL+eXQO7pL0iLsfkvRI9H5Y3Cnp2bb3w3wuflPSQ+7+NknvUHhe\nhvJ8mNmMpH8p6Yi7XycpI+nDGtLzsVmpCwhJ75T0grsfc/eqpC9KurXPNW0rdz/l7k9Er88r/AMw\no/A8fD467POS/l5/KtxeZrZP0t+V9Jm2zcN6LiYkvVfSZyXJ3avufk5Dej4iWUklM8tKGpH0iob7\nfHQtjQExI+nltvcnom1DycwOSLpe0jcl7XX3U9GuVyXt7VNZ2+03JH1MUvuqvMN6Lq6RdFrS/4wu\nuX3GzEY1pOfD3U9K+q+Sjks6JWnO3f9IQ3o+NiuNAYGImZUl/a6kf+Xu8+37PLw9beBvUTOzn5D0\nurs/fqljhuVcRLKSbpD0KXe/XtKiLrp8MkznI+pbuFVhcF4padTMLliReZjOx2alMSBOSrqq7f2+\naNtQMbOcwnD4HXe/P9r8mpldEe2/QtLr/apvG/2wpA+Y2fcUXm78UTP7XxrOcyGFLeoT7v7N6P19\nCgNjWM/H35b0V+5+2t1rku6X9G4N7/nYlDQGxGOSDpnZNWaWV9jh9ECfa9pWZmYKrzE/6+6faNv1\ngKSfjl7/tKQvb3dt283dP+7u+9z9gML/Fv7Y3X9KQ3guJMndX5X0spm9Ndp0k6S/0JCeD4WXlm40\ns5Ho/5ubFPbZDev52JRUDpQzs/crvO6ckXSvu/9qn0vaVmb2Hkl/KukprV13/yWF/RBfkrRf0kuS\n/oG7z/alyD4wsx+R9G/c/SfMbLeG9FyY2Q8q7LDPSzom6aMK/zE4rOfjP0v6kMK7/74t6Z9KKmtI\nz8dmpDIgAADJS+MlJgDANiAgAAAdERAAgI4ICABARwQEAKAjAgJYh5k1zOzJtkdsk7qZ2QEzezqu\nzwPilu13AcAOt+zuP9jvIoB+oAUB9MDMvmdmv25mT5nZt8zsYLT9gJn9sZl918weMbP90fa9ZvZ7\nZvad6PHu6KMyZvbpaL2CPzKzUt9+KeAiBASwvtJFl5g+1LZvzt2/X9J/VziyX5L+m6TPu/sPSPod\nSZ+Mtn9S0v9193conBvpmWj7IUl3u/v3STon6ScT/n2ArjGSGliHmS24e7nD9u9J+lF3PxZNnPiq\nu+82szckXeHutWj7KXefNrPTkva5e6XtMw5I+mq0aI3M7N9Jyrn7ryT/mwEbowUB9M4v8XozKm2v\nG6JfEDsIAQH07kNtz49Gr/9M4ayykvQRhZMqSuGylj8nra6fPbFdRQK94l8rwPpKZvZk2/uH3L11\nq+uUmX1XYSvgtmjbv1C4mtu/Vbiy20ej7XdKusfMblfYUvg5hSucATsWfRBAD6I+iCPu/ka/awGS\nwiUmAEBHtCAAAB3RggAAdERAAAA6IiAAAB0REACAjggIAEBHBAQAoKP/Dwje+zONsUruAAAAAElF\nTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "pd.Series(history.history['loss']).plot(logy=True)\n", "plt.xlabel(\"Epoch\")\n", "plt.ylabel(\"Train Error\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Prediction error" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now see how our model does! I'll do a small post-processing step to round off our prediction to the nearest integer. This is usually not done, and thus just a whimsical step, since the training ratings are all integers! There are better ways to encode this intger requirement (one-hot encoding!), but we won't discuss them in this post." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [], "source": [ "y_hat = np.round(model.predict([test.user_id, test.item_id]),0)\n", "y_true = test.rating" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.6915" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.metrics import mean_absolute_error\n", "mean_absolute_error(y_true, y_hat)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not bad! We're able to get a $MAE$ of 0.69! I'm sure with a bit of parameter/hyper-parameter optimisation, we may be able to improve the results. However, I won't talk about these optimisations in this post. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Extracting the learnt embeddings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can extract the learnt movie and item embeddings as follows:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
count1683.0000001683.0000001683.000000
mean-0.9354200.8578620.954169
std0.5174580.4474390.458095
min-2.524487-0.459752-0.989537
25%-1.3234310.5463640.642444
50%-0.9491880.8512430.993619
75%-0.5508621.1595881.283555
max0.5006182.1406072.683658
\n", "
" ], "text/plain": [ " 0 1 2\n", "count 1683.000000 1683.000000 1683.000000\n", "mean -0.935420 0.857862 0.954169\n", "std 0.517458 0.447439 0.458095\n", "min -2.524487 -0.459752 -0.989537\n", "25% -1.323431 0.546364 0.642444\n", "50% -0.949188 0.851243 0.993619\n", "75% -0.550862 1.159588 1.283555\n", "max 0.500618 2.140607 2.683658" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movie_embedding_learnt = model.get_layer(name='Movie-Embedding').get_weights()[0]\n", "pd.DataFrame(movie_embedding_learnt).describe()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
count944.000000944.000000944.000000
mean-1.1262311.1716091.109131
std0.5174780.4090160.548384
min-2.883226-0.500010-0.415373
25%-1.4581970.9035740.735729
50%-1.1594801.1995171.084089
75%-0.8367461.4566101.468611
max0.8994362.6053302.826109
\n", "
" ], "text/plain": [ " 0 1 2\n", "count 944.000000 944.000000 944.000000\n", "mean -1.126231 1.171609 1.109131\n", "std 0.517478 0.409016 0.548384\n", "min -2.883226 -0.500010 -0.415373\n", "25% -1.458197 0.903574 0.735729\n", "50% -1.159480 1.199517 1.084089\n", "75% -0.836746 1.456610 1.468611\n", "max 0.899436 2.605330 2.826109" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "user_embedding_learnt = model.get_layer(name='User-Embedding').get_weights()[0]\n", "pd.DataFrame(user_embedding_learnt).describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that both the user and the item embeddings have negative elements. There are some applications which require that the learnt embeddings be non-negative. This approach is also called non-negative matrix factorisation, which we'll workout now." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Non-negative Matrix factorisation (NNMF) in Keras" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code for NNMF remains exactly the same as the code for matrix factorisation. The only change is that we add `non-negativity` constraints on the learnt embeddings. This is done as follows:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from keras.constraints import non_neg\n", "movie_input = keras.layers.Input(shape=[1],name='Item')\n", "movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors, name='NonNegMovie-Embedding', embeddings_constraint=non_neg())(movie_input)\n", "movie_vec = keras.layers.Flatten(name='FlattenMovies')(movie_embedding)\n", "\n", "user_input = keras.layers.Input(shape=[1],name='User')\n", "user_vec = keras.layers.Flatten(name='FlattenUsers')(keras.layers.Embedding(n_users + 1, n_latent_factors,name='NonNegUser-Embedding',embeddings_constraint=non_neg())(user_input))\n", "\n", "prod = keras.layers.merge([movie_vec, user_vec], mode='dot',name='DotProduct')\n", "model = keras.Model([user_input, movie_input], prod)\n", "model.compile('adam', 'mean_squared_error')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now verify if we are indeed able to learn non-negative embeddings. I'll not compare the performance of NNMF on the test set, in the interest of space." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": true }, "outputs": [], "source": [ "history_nonneg = model.fit([train.user_id, train.item_id], train.rating, epochs=10, verbose=0)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
count1683.0000001683.0000001683.000000
mean0.8384500.8403300.838066
std0.3016180.3015290.301040
min-0.000000-0.000000-0.000000
25%0.6577490.6639510.656453
50%0.9014950.9041920.895934
75%1.0727061.0735911.072926
max1.3657191.3790061.373672
\n", "
" ], "text/plain": [ " 0 1 2\n", "count 1683.000000 1683.000000 1683.000000\n", "mean 0.838450 0.840330 0.838066\n", "std 0.301618 0.301529 0.301040\n", "min -0.000000 -0.000000 -0.000000\n", "25% 0.657749 0.663951 0.656453\n", "50% 0.901495 0.904192 0.895934\n", "75% 1.072706 1.073591 1.072926\n", "max 1.365719 1.379006 1.373672" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movie_embedding_learnt = model.get_layer(name='NonNegMovie-Embedding').get_weights()[0]\n", "pd.DataFrame(movie_embedding_learnt).describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks good!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Neural networks for recommendation\n", "\n", "We'll now create a simple neural network for recommendation, or for estimating rating! This model is very similar to the earlier matrix factorisation models, but differs in the following ways:\n", "\n", "1. Instead of taking a dot product of the user and the item embedding, we concatenate them and use them as features for our neural network. Thus, we are not constrained to the dot product way of combining the embeddings, and can learn complex non-linear relationships.\n", "2. Due to #1, we can now have a different dimension of user and item embeddings. This can be useful if one dimension is larger than the other." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": true }, "outputs": [], "source": [ "n_latent_factors_user = 5\n", "n_latent_factors_movie = 8\n", "\n", "movie_input = keras.layers.Input(shape=[1],name='Item')\n", "movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors_movie, name='Movie-Embedding')(movie_input)\n", "movie_vec = keras.layers.Flatten(name='FlattenMovies')(movie_embedding)\n", "movie_vec = keras.layers.Dropout(0.2)(movie_vec)\n", "\n", "\n", "user_input = keras.layers.Input(shape=[1],name='User')\n", "user_vec = keras.layers.Flatten(name='FlattenUsers')(keras.layers.Embedding(n_users + 1, n_latent_factors_user,name='User-Embedding')(user_input))\n", "user_vec = keras.layers.Dropout(0.2)(user_vec)\n", "\n", "\n", "concat = keras.layers.merge([movie_vec, user_vec], mode='concat',name='Concat')\n", "concat_dropout = keras.layers.Dropout(0.2)(concat)\n", "dense = keras.layers.Dense(200,name='FullyConnected')(concat)\n", "dropout_1 = keras.layers.Dropout(0.2,name='Dropout')(dense)\n", "dense_2 = keras.layers.Dense(100,name='FullyConnected-1')(concat)\n", "dropout_2 = keras.layers.Dropout(0.2,name='Dropout')(dense_2)\n", "dense_3 = keras.layers.Dense(50,name='FullyConnected-2')(dense_2)\n", "dropout_3 = keras.layers.Dropout(0.2,name='Dropout')(dense_3)\n", "dense_4 = keras.layers.Dense(20,name='FullyConnected-3', activation='relu')(dense_3)\n", "\n", "\n", "result = keras.layers.Dense(1, activation='relu',name='Activation')(dense_4)\n", "adam = Adam(lr=0.005)\n", "model = keras.Model([user_input, movie_input], result)\n", "model.compile(optimizer=adam,loss= 'mean_absolute_error')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now see how our model looks like:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "G\n", "\n", "\n", "112307868840\n", "\n", "Item: InputLayer\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1)\n", "\n", "\n", "112308383136\n", "\n", "Movie-Embedding: Embedding\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1, 8)\n", "\n", "\n", "112307868840->112308383136\n", "\n", "\n", "\n", "\n", "4659651864\n", "\n", "User: InputLayer\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1)\n", "\n", "\n", "112310319536\n", "\n", "User-Embedding: Embedding\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1)\n", "\n", "(None, 1, 5)\n", "\n", "\n", "4659651864->112310319536\n", "\n", "\n", "\n", "\n", "112308383416\n", "\n", "FlattenMovies: Flatten\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1, 8)\n", "\n", "(None, 8)\n", "\n", "\n", "112308383136->112308383416\n", "\n", "\n", "\n", "\n", "112307982232\n", "\n", "FlattenUsers: Flatten\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 1, 5)\n", "\n", "(None, 5)\n", "\n", "\n", "112310319536->112307982232\n", "\n", "\n", "\n", "\n", "112308313840\n", "\n", "dropout_1: Dropout\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 8)\n", "\n", "(None, 8)\n", "\n", "\n", "112308383416->112308313840\n", "\n", "\n", "\n", "\n", "112310320768\n", "\n", "dropout_2: Dropout\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 5)\n", "\n", "(None, 5)\n", "\n", "\n", "112307982232->112310320768\n", "\n", "\n", "\n", "\n", "4659651360\n", "\n", "Concat: Merge\n", "\n", "input:\n", "\n", "output:\n", "\n", "[(None, 8), (None, 5)]\n", "\n", "(None, 13)\n", "\n", "\n", "112308313840->4659651360\n", "\n", "\n", "\n", "\n", "112310320768->4659651360\n", "\n", "\n", "\n", "\n", "112308749368\n", "\n", "FullyConnected-1: Dense\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 13)\n", "\n", "(None, 100)\n", "\n", "\n", "4659651360->112308749368\n", "\n", "\n", "\n", "\n", "112310118104\n", "\n", "FullyConnected-2: Dense\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 100)\n", "\n", "(None, 50)\n", "\n", "\n", "112308749368->112310118104\n", "\n", "\n", "\n", "\n", "4653345424\n", "\n", "FullyConnected-3: Dense\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 50)\n", "\n", "(None, 20)\n", "\n", "\n", "112310118104->4653345424\n", "\n", "\n", "\n", "\n", "4653179904\n", "\n", "Activation: Dense\n", "\n", "input:\n", "\n", "output:\n", "\n", "(None, 20)\n", "\n", "(None, 1)\n", "\n", "\n", "4653345424->4653179904\n", "\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "SVG(model_to_dot(model, show_shapes=True, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It should be noted that we use a different number of embeddings for user (3) and items (5)! These combine to form a vector of length (5+3 = 8), which is then fed into the neural network. We also add a dropout layer to prevent overfitting!" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "__________________________________________________________________________________________________\n", "Layer (type) Output Shape Param # Connected to \n", "==================================================================================================\n", "Item (InputLayer) (None, 1) 0 \n", "__________________________________________________________________________________________________\n", "User (InputLayer) (None, 1) 0 \n", "__________________________________________________________________________________________________\n", "Movie-Embedding (Embedding) (None, 1, 8) 13464 Item[0][0] \n", "__________________________________________________________________________________________________\n", "User-Embedding (Embedding) (None, 1, 5) 4720 User[0][0] \n", "__________________________________________________________________________________________________\n", "FlattenMovies (Flatten) (None, 8) 0 Movie-Embedding[0][0] \n", "__________________________________________________________________________________________________\n", "FlattenUsers (Flatten) (None, 5) 0 User-Embedding[0][0] \n", "__________________________________________________________________________________________________\n", "dropout_1 (Dropout) (None, 8) 0 FlattenMovies[0][0] \n", "__________________________________________________________________________________________________\n", "dropout_2 (Dropout) (None, 5) 0 FlattenUsers[0][0] \n", "__________________________________________________________________________________________________\n", "Concat (Merge) (None, 13) 0 dropout_1[0][0] \n", " dropout_2[0][0] \n", "__________________________________________________________________________________________________\n", "FullyConnected-1 (Dense) (None, 100) 1400 Concat[0][0] \n", "__________________________________________________________________________________________________\n", "FullyConnected-2 (Dense) (None, 50) 5050 FullyConnected-1[0][0] \n", "__________________________________________________________________________________________________\n", "FullyConnected-3 (Dense) (None, 20) 1020 FullyConnected-2[0][0] \n", "__________________________________________________________________________________________________\n", "Activation (Dense) (None, 1) 21 FullyConnected-3[0][0] \n", "==================================================================================================\n", "Total params: 25,675\n", "Trainable params: 25,675\n", "Non-trainable params: 0\n", "__________________________________________________________________________________________________\n" ] } ], "source": [ "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the number of parameters is more than what we had in the Matrix Factorisation case. Let's see how this model works. I'll run it for more epochs given that we have more parameters." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "history = model.fit([train.user_id, train.item_id], train.rating, epochs=250, verbose=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Prediction performance of Neural Network based recommender system" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.6957\n", "0.708807692927\n" ] } ], "source": [ "y_hat_2 = np.round(model.predict([test.user_id, test.item_id]),0)\n", "print(mean_absolute_error(y_true, y_hat_2))\n", "\n", "print(mean_absolute_error(y_true, model.predict([test.user_id, test.item_id])))\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pretty similar to the result we got using matrix factorisation. Maybe, we need to tweak around a lot more with the neural network to get better results?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thanks for reading. This post has been a good learning experience for me. Hope you enjoyed too! " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 1 }