notebooks/create_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run only once\n",
    "import os\n",
    "os.chdir('../')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a SeqDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main class used to create a  is `SeqData` from `dataset.preprocesing`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataset.preprocessing import SeqDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When instantiating the dataset we have to set the size (in MBs) of chunks that dask will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = SeqDataset(chunksize=120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactions\n",
    "The fundamental and mandatory part of `SeqDataset` is the interactions. The interaction file(s) contain interactions between a user and an item and can be loaded using the `load_interactions` function.\n",
    "\n",
    "Currently we support only interaction file(s) in the form of jsonl."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-06 11:40:31,605 - preprocessing.py:104 - INFO - Total size of interactions: 311 MB. Repartitioning interactions to 2 partitions\n",
      "2024-11-06 11:40:36,844 - preprocessing.py:124 - INFO - Dropping duplicates in interactions\n",
      "2024-11-06 11:40:38,138 - preprocessing.py:134 - INFO - Interactions loaded\n"
     ]
    }
   ],
   "source": [
    "# File path of interactions\n",
    "path = './data/amazon_reviews_2023/reviews/All_Beauty.jsonl'\n",
    "# Column name indicating item id\n",
    "c_iid= 'asin'\n",
    "# Column name indicating user id\n",
    "c_uid= 'user_id'\n",
    "# Column name indicating timestamp of interaction in unix format\n",
    "c_timestamp= 'timestamp'\n",
    "# Column name indicating score assigned to item by user (Optional)\n",
    "c_score= 'average_rating'\n",
    "\n",
    "\n",
    "data.load_interactions(\n",
    "    path=path,\n",
    "    c_uid=c_uid,\n",
    "    c_iid=c_iid,\n",
    "    c_timestamp=c_timestamp,\n",
    "    c_score=c_score,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B00YQ6X8EO</td>\n",
       "      <td>AGKHLEW2SOWHNMFQIJGBECAF7INQ</td>\n",
       "      <td>1588687728923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B081TJ8YS3</td>\n",
       "      <td>AGKHLEW2SOWHNMFQIJGBECAF7INQ</td>\n",
       "      <td>1588615855070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B07PNNCSP9</td>\n",
       "      <td>AE74DYR3QUGVPZJ3P7RFWBGIX7XQ</td>\n",
       "      <td>1589665266052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B09JS339BZ</td>\n",
       "      <td>AFQLNQNQYFWQZPJQZS6V3NZU4QBQ</td>\n",
       "      <td>1643393630220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B08BZ63GMJ</td>\n",
       "      <td>AFQLNQNQYFWQZPJQZS6V3NZU4QBQ</td>\n",
       "      <td>1609322563534</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      item_id                       user_id      timestamp\n",
       "0  B00YQ6X8EO  AGKHLEW2SOWHNMFQIJGBECAF7INQ  1588687728923\n",
       "1  B081TJ8YS3  AGKHLEW2SOWHNMFQIJGBECAF7INQ  1588615855070\n",
       "2  B07PNNCSP9  AE74DYR3QUGVPZJ3P7RFWBGIX7XQ  1589665266052\n",
       "3  B09JS339BZ  AFQLNQNQYFWQZPJQZS6V3NZU4QBQ  1643393630220\n",
       "4  B08BZ63GMJ  AFQLNQNQYFWQZPJQZS6V3NZU4QBQ  1609322563534"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.interactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metadata\n",
    "We can also load and add metadata to items if there are available using the `load_metadata` fucntion. The metadata file(s) contain information regarding an item.\n",
    "\n",
    "The user can select which columns to consider by passing a list with the name of the columns (metadata parameter). All the information will concatenated as strings fro example:\n",
    "\n",
    "```\n",
    "in: metadata_cols = ['title', 'brand', 'category']\n",
    "out: 'title: Foobar brand: Foo category: Bar'\n",
    "```\n",
    "\n",
    "Currently we support only metadata file(s) in the form of jsonl.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-06 11:40:42,497 - preprocessing.py:164 - INFO - Total size of metadata: 203MB. Repartitioning metadata to 1 partitions\n",
      "2024-11-06 11:40:43,343 - preprocessing.py:172 - INFO - Dropping duplicates in metadata\n",
      "2024-11-06 11:40:45,161 - preprocessing.py:227 - INFO - Metadata loaded and mapped.\n"
     ]
    }
   ],
   "source": [
    "# File path of metadata\n",
    "path = './data/amazon_reviews_2023/meta/meta_All_Beauty.jsonl'\n",
    "# Column name indicating item id\n",
    "c_iid= 'parent_asin'\n",
    "# List of column names to be loaded from metadata\n",
    "metadata_cols = ['title', 'store', 'main_category']\n",
    "# flag on whether to drop rows with missing values in ALL of the specified columns\n",
    "dropna = True\n",
    "\n",
    "data.load_metadata(\n",
    "    path,\n",
    "    c_iid=c_iid,\n",
    "    metadata_cols=metadata_cols,\n",
    "    dropna=dropna,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B00YQ6X8EO</td>\n",
       "      <td>AGKHLEW2SOWHNMFQIJGBECAF7INQ</td>\n",
       "      <td>1588687728923</td>\n",
       "      <td>title: Herbivore - Natural Sea Mist Texturizin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B081TJ8YS3</td>\n",
       "      <td>AGKHLEW2SOWHNMFQIJGBECAF7INQ</td>\n",
       "      <td>1588615855070</td>\n",
       "      <td>title: All Natural Vegan Dry Shampoo Powder - ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B09JS339BZ</td>\n",
       "      <td>AFQLNQNQYFWQZPJQZS6V3NZU4QBQ</td>\n",
       "      <td>1643393630220</td>\n",
       "      <td>title: muaowig Ombre Body Wave Bundles 1B Grey...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B08BZ63GMJ</td>\n",
       "      <td>AFQLNQNQYFWQZPJQZS6V3NZU4QBQ</td>\n",
       "      <td>1609322563534</td>\n",
       "      <td>title: Yinhua Electric Nail Drill Kit Portable...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B00R8DXL44</td>\n",
       "      <td>AGMJ3EMDVL6OWBJF7CA5RGJLXN5A</td>\n",
       "      <td>1598567408138</td>\n",
       "      <td>title: China Glaze Nail Polish, Wanderlust 138...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      item_id                       user_id      timestamp  \\\n",
       "0  B00YQ6X8EO  AGKHLEW2SOWHNMFQIJGBECAF7INQ  1588687728923   \n",
       "1  B081TJ8YS3  AGKHLEW2SOWHNMFQIJGBECAF7INQ  1588615855070   \n",
       "2  B09JS339BZ  AFQLNQNQYFWQZPJQZS6V3NZU4QBQ  1643393630220   \n",
       "3  B08BZ63GMJ  AFQLNQNQYFWQZPJQZS6V3NZU4QBQ  1609322563534   \n",
       "4  B00R8DXL44  AGMJ3EMDVL6OWBJF7CA5RGJLXN5A  1598567408138   \n",
       "\n",
       "                                            features  \n",
       "0  title: Herbivore - Natural Sea Mist Texturizin...  \n",
       "1  title: All Natural Vegan Dry Shampoo Powder - ...  \n",
       "2  title: muaowig Ombre Body Wave Bundles 1B Grey...  \n",
       "3  title: Yinhua Electric Nail Drill Kit Portable...  \n",
       "4  title: China Glaze Nail Polish, Wanderlust 138...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.interactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading multiple files\n",
    "We can select multiple files to load for both `load_interactions` and `load_metadata` functions. However, the files should share a common schema (i.e. same column names)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_multiple_files = SeqDataset(chunksize=120)\n",
    "data_multiple_files.load_interactions(\n",
    "    path= ['./data/amazon_reviews_2018/reviews/All_Beauty.json', './data/amazon_reviews_2018/reviews/Video_Games.json'],\n",
    "    c_uid=c_uid,\n",
    "    c_iid=c_iid,\n",
    "    c_timestamp=c_timestamp,\n",
    "    c_score=c_score\n",
    ")\n",
    "\n",
    "data_multiple_files.load_metadata(\n",
    "    path= ['./data/amazon_reviews_2018/meta/meta_All_Beauty.json', './data/amazon_reviews_2018/meta/meta_Video_Games.json'],\n",
    "    c_iid=c_iid,\n",
    "    metadata_cols=metadata_cols,\n",
    "    dropna=dropna,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying k-core filtering.\n",
    "\n",
    "`k-core` filtering is a common preprpocessing method that aims to reduce the sparsity in the dataset by filtering users and items that appear less than `k` times.\n",
    "\n",
    "This can be done by using the `kcore_filtering` function. The function returns `True` if the filtering was applied succesfully otherwise it returns `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of interactions: 633,693\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of interactions: {data.interactions.shape[0].compute():,}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-06 11:41:19,827 - preprocessing.py:298 - INFO - Applying k-core filtering.\n",
      "2024-11-06 11:41:42,314 - preprocessing.py:310 - INFO - Counted users, items and user-item interactions.\n",
      "2024-11-06 11:41:42,497 - preprocessing.py:329 - INFO - Filtered out users and items with less than 5 appearances. Creating sparse matrix...\n",
      "2024-11-06 11:41:43,595 - preprocessing.py:609 - INFO - Iteration 5 - Users: 284 Items: 424\n",
      "2024-11-06 11:41:43,598 - preprocessing.py:609 - INFO - Iteration 10 - Users: 229 Items: 318\n",
      "2024-11-06 11:41:43,600 - preprocessing.py:609 - INFO - Iteration 15 - Users: 218 Items: 313\n",
      "2024-11-06 11:41:43,601 - preprocessing.py:355 - INFO - Filtered matrix computed. Applying filtering to interactions...\n",
      "2024-11-06 11:42:05,016 - preprocessing.py:375 - INFO - K-core filtering applied\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "k-core filtering applied: True. Number of interactions after filtering: 2,149\n"
     ]
    }
   ],
   "source": [
    "# set k threshold\n",
    "kcore = 5\n",
    "kcore_applied = data.kcore_filtering(kcore=kcore)\n",
    "\n",
    "print(f\"k-core filtering applied: {kcore_applied}. Number of interactions after filtering: {data.interactions.shape[0].compute():,}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B08P2DZB4X</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1627391044559</td>\n",
       "      <td>title: NIRA Skincare Laser &amp; Serum Bundle - In...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B07RBSLNFR</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1621184430697</td>\n",
       "      <td>title: OGANA CELL Peptide Concentrating Amazin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B07SLFWZKN</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1619737501209</td>\n",
       "      <td>title: Keratin Secrets Do It Yourself Home Ker...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B08JTNQFZY</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1617904219785</td>\n",
       "      <td>title: GAINWELL ~ store: GAINWELL ~ main_categ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B07KG1TWP5</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1596473351088</td>\n",
       "      <td>title: Organic Bamboo Cotton Ear Swabs by Bali...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      item_id                       user_id      timestamp  \\\n",
       "0  B08P2DZB4X  AFSKPY37N3C43SOI5IEXEK5JSIYA  1627391044559   \n",
       "1  B07RBSLNFR  AFSKPY37N3C43SOI5IEXEK5JSIYA  1621184430697   \n",
       "2  B07SLFWZKN  AFSKPY37N3C43SOI5IEXEK5JSIYA  1619737501209   \n",
       "3  B08JTNQFZY  AFSKPY37N3C43SOI5IEXEK5JSIYA  1617904219785   \n",
       "4  B07KG1TWP5  AFSKPY37N3C43SOI5IEXEK5JSIYA  1596473351088   \n",
       "\n",
       "                                            features  \n",
       "0  title: NIRA Skincare Laser & Serum Bundle - In...  \n",
       "1  title: OGANA CELL Peptide Concentrating Amazin...  \n",
       "2  title: Keratin Secrets Do It Yourself Home Ker...  \n",
       "3  title: GAINWELL ~ store: GAINWELL ~ main_categ...  \n",
       "4  title: Organic Bamboo Cotton Ear Swabs by Bali...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.interactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding users and items\n",
    "\n",
    "If needed we can encode items and users as integers by using the `encode_entries` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-06 11:43:51,194 - preprocessing.py:252 - INFO - Number of users: 218\n",
      "2024-11-06 11:43:51,199 - preprocessing.py:260 - INFO - Users encoded\n",
      "2024-11-06 11:43:51,200 - preprocessing.py:271 - INFO - Number of items: 312\n",
      "2024-11-06 11:43:51,204 - preprocessing.py:279 - INFO - Items encoded\n"
     ]
    }
   ],
   "source": [
    "data.encode_entries()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>features</th>\n",
       "      <th>user_id_encoded</th>\n",
       "      <th>item_id_encoded</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B08P2DZB4X</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1627391044559</td>\n",
       "      <td>title: NIRA Skincare Laser &amp; Serum Bundle - In...</td>\n",
       "      <td>104</td>\n",
       "      <td>247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B07RBSLNFR</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1621184430697</td>\n",
       "      <td>title: OGANA CELL Peptide Concentrating Amazin...</td>\n",
       "      <td>104</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B07SLFWZKN</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1619737501209</td>\n",
       "      <td>title: Keratin Secrets Do It Yourself Home Ker...</td>\n",
       "      <td>104</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B08JTNQFZY</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1617904219785</td>\n",
       "      <td>title: GAINWELL ~ store: GAINWELL ~ main_categ...</td>\n",
       "      <td>104</td>\n",
       "      <td>216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B07KG1TWP5</td>\n",
       "      <td>AFSKPY37N3C43SOI5IEXEK5JSIYA</td>\n",
       "      <td>1596473351088</td>\n",
       "      <td>title: Organic Bamboo Cotton Ear Swabs by Bali...</td>\n",
       "      <td>104</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      item_id                       user_id      timestamp  \\\n",
       "0  B08P2DZB4X  AFSKPY37N3C43SOI5IEXEK5JSIYA  1627391044559   \n",
       "1  B07RBSLNFR  AFSKPY37N3C43SOI5IEXEK5JSIYA  1621184430697   \n",
       "2  B07SLFWZKN  AFSKPY37N3C43SOI5IEXEK5JSIYA  1619737501209   \n",
       "3  B08JTNQFZY  AFSKPY37N3C43SOI5IEXEK5JSIYA  1617904219785   \n",
       "4  B07KG1TWP5  AFSKPY37N3C43SOI5IEXEK5JSIYA  1596473351088   \n",
       "\n",
       "                                            features  user_id_encoded  \\\n",
       "0  title: NIRA Skincare Laser & Serum Bundle - In...              104   \n",
       "1  title: OGANA CELL Peptide Concentrating Amazin...              104   \n",
       "2  title: Keratin Secrets Do It Yourself Home Ker...              104   \n",
       "3  title: GAINWELL ~ store: GAINWELL ~ main_categ...              104   \n",
       "4  title: Organic Bamboo Cotton Ear Swabs by Bali...              104   \n",
       "\n",
       "   item_id_encoded  \n",
       "0              247  \n",
       "1               35  \n",
       "2               37  \n",
       "3              216  \n",
       "4               20  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.interactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating sequences\n",
    "\n",
    "We can group all the interactions for each user/session^ by using the `create_sequences` function.\n",
    "\n",
    "`create_sequences` will join all the relevant columns (features, item_id, score, item_id_encoded) for each user/session in chronological order (older to most recent).\n",
    "\n",
    "^*when k-core filtering is applied each user will have mutliple sessions*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-06 11:43:55,876 - preprocessing.py:387 - INFO - Setting index to user_id\n",
      "2024-11-06 11:43:55,878 - preprocessing.py:393 - INFO - Index set to user_id\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'Column not found: score'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[13], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mdata\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcreate_sequences\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Desktop/panko/dataset/preprocessing.py:408\u001b[0m, in \u001b[0;36mSeqDataset.create_sequences\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    405\u001b[0m \u001b[38;5;66;03m# merge target columns for each user\u001b[39;00m\n\u001b[1;32m    406\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m col \u001b[38;5;129;01min\u001b[39;00m cols_to_merge:\n\u001b[1;32m    407\u001b[0m     result \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m--> 408\u001b[0m         \u001b[43mddf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgroupby\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43muser_id\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m[\u001b[49m\u001b[43mcol\u001b[49m\u001b[43m]\u001b[49m\n\u001b[1;32m    409\u001b[0m         \u001b[38;5;241m.\u001b[39mapply(\u001b[38;5;28mlist\u001b[39m, meta\u001b[38;5;241m=\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mcol\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m_list\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mobject\u001b[39m\u001b[38;5;124m\"\u001b[39m))\n\u001b[1;32m    410\u001b[0m         \u001b[38;5;241m.\u001b[39mto_frame()\n\u001b[1;32m    411\u001b[0m     )\n\u001b[1;32m    413\u001b[0m     ddf \u001b[38;5;241m=\u001b[39m ddf\u001b[38;5;241m.\u001b[39mmerge(result, how\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mleft\u001b[39m\u001b[38;5;124m\"\u001b[39m, left_index\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, right_index\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m    414\u001b[0m     ddf \u001b[38;5;241m=\u001b[39m ddf\u001b[38;5;241m.\u001b[39mpersist()\n",
      "File \u001b[0;32m~/anaconda3/envs/panko/lib/python3.12/site-packages/dask_expr/_groupby.py:1618\u001b[0m, in \u001b[0;36mGroupBy.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   1616\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__getitem__\u001b[39m(\u001b[38;5;28mself\u001b[39m, key):\n\u001b[1;32m   1617\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m is_scalar(key):\n\u001b[0;32m-> 1618\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mSeriesGroupBy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1619\u001b[0m \u001b[43m            \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1620\u001b[0m \u001b[43m            \u001b[49m\u001b[43mby\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mby\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1621\u001b[0m \u001b[43m            \u001b[49m\u001b[38;5;28;43mslice\u001b[39;49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mkey\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1622\u001b[0m \u001b[43m            \u001b[49m\u001b[43msort\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msort\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1623\u001b[0m \u001b[43m            \u001b[49m\u001b[43mdropna\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdropna\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1624\u001b[0m \u001b[43m            \u001b[49m\u001b[43mobserved\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mobserved\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1625\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1626\u001b[0m     g \u001b[38;5;241m=\u001b[39m GroupBy(\n\u001b[1;32m   1627\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mobj,\n\u001b[1;32m   1628\u001b[0m         by\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mby,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   1633\u001b[0m         group_keys\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgroup_keys,\n\u001b[1;32m   1634\u001b[0m     )\n\u001b[1;32m   1635\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m g\n",
      "File \u001b[0;32m~/anaconda3/envs/panko/lib/python3.12/site-packages/dask_expr/_groupby.py:2217\u001b[0m, in \u001b[0;36mSeriesGroupBy.__init__\u001b[0;34m(self, obj, by, sort, observed, dropna, slice)\u001b[0m\n\u001b[1;32m   2214\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   2215\u001b[0m         obj\u001b[38;5;241m.\u001b[39m_meta\u001b[38;5;241m.\u001b[39mgroupby(by, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39m_as_dict(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mobserved\u001b[39m\u001b[38;5;124m\"\u001b[39m, observed))\n\u001b[0;32m-> 2217\u001b[0m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\n\u001b[1;32m   2218\u001b[0m \u001b[43m    \u001b[49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mby\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mby\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mslice\u001b[39;49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mslice\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mobserved\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mobserved\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdropna\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdropna\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msort\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msort\u001b[49m\n\u001b[1;32m   2219\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/envs/panko/lib/python3.12/site-packages/dask_expr/_groupby.py:1561\u001b[0m, in \u001b[0;36mGroupBy.__init__\u001b[0;34m(self, obj, by, group_keys, sort, observed, dropna, slice)\u001b[0m\n\u001b[1;32m   1559\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\u001b[38;5;28mslice\u001b[39m, \u001b[38;5;28mtuple\u001b[39m):\n\u001b[1;32m   1560\u001b[0m     \u001b[38;5;28mslice\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(\u001b[38;5;28mslice\u001b[39m)\n\u001b[0;32m-> 1561\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_meta \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_meta\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;28;43mslice\u001b[39;49m\u001b[43m]\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/envs/panko/lib/python3.12/site-packages/pandas/core/groupby/generic.py:1951\u001b[0m, in \u001b[0;36mDataFrameGroupBy.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   1944\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(key, \u001b[38;5;28mtuple\u001b[39m) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(key) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m   1945\u001b[0m     \u001b[38;5;66;03m# if len == 1, then it becomes a SeriesGroupBy and this is actually\u001b[39;00m\n\u001b[1;32m   1946\u001b[0m     \u001b[38;5;66;03m# valid syntax, so don't raise\u001b[39;00m\n\u001b[1;32m   1947\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1948\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot subset columns with a tuple with more than one element. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1949\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mUse a list instead.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1950\u001b[0m     )\n\u001b[0;32m-> 1951\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__getitem__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/envs/panko/lib/python3.12/site-packages/pandas/core/base.py:244\u001b[0m, in \u001b[0;36mSelectionMixin.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    242\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    243\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m key \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mobj:\n\u001b[0;32m--> 244\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mColumn not found: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mkey\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    245\u001b[0m     ndim \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mobj[key]\u001b[38;5;241m.\u001b[39mndim\n\u001b[1;32m    246\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_gotitem(key, ndim\u001b[38;5;241m=\u001b[39mndim)\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Column not found: score'"
     ]
    }
   ],
   "source": [
    "data.create_sequences()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.interactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Splitting data into train/val/test\n",
    "\n",
    "The dataset can be slitted in train, val, test sets by using the `split_data` function. \n",
    "\n",
    "Currently we support only LOO (leave one out) method (train: [:-2], val: [-2], test: [-1]). In this method, depending on the length of sequences, the train/val/test sets may not be of the same size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.split_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the dataset\n",
    "\n",
    "Use the `save` method to save the dataset.\n",
    "\n",
    "The function will save:\n",
    "\n",
    "- interactions\n",
    "- metadata\n",
    "- train, val, and test splits\n",
    "- encoder_items.json (mapping used for encoding items)\n",
    "- encoder_users.json (mapping used for encoding users)\n",
    "- stats.json (dataset statistics: num_items, num_users, num_interactions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# directory to save the dataset\n",
    "path = './sequential_dataset'\n",
    "\n",
    "data.save(\n",
    "    './sequential_dataset',\n",
    "    save_metadata=True,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ./sequential_dataset/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a dataset with a yaml file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user can also utilise the `preprocessing/create_dataset.py` script to create a SeqDataset by creating a yaml file with all the configurations. The file should be saved under \"configs_hydra/dataset\"\n",
    "\n",
    "Example of yaml file:\n",
    "```\n",
    "c_iid: asin\n",
    "c_uid: reviewerID\n",
    "c_score: overall\n",
    "c_timestamp: unixReviewTime\n",
    "dropna: false\n",
    "kcore: 5\n",
    "metadata_cols:\n",
    "  - title\n",
    "  - brand\n",
    "  - category\n",
    "chunksize: 500\n",
    "path_reviews: ./data/amazon_reviews_2018/reviews/Video_Games.json\n",
    "path_metadata: ./data/amazon_reviews_2018/meta/meta_Video_Games.json\n",
    "path_output: ./data_experiments/amazon_reviews_2018/Video_Games\n",
    "```\n",
    "\n",
    "Assuming the file is stored under 'configs_hydra/dataset/custom_data.yaml' run:\n",
    "\n",
    "`python preprocessing/create_dataset.py dataset:new_data`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user can pass multiple files too e.g.\n",
    "```\n",
    "path_reviews:\n",
    "  - ./data/amazon_reviews_2018/reviews/Automotive.json\n",
    "  - ./data/amazon_reviews_2018/reviews/Cell_Phones_and_Accessories.json\n",
    "\n",
    "path_metadata:\n",
    "  - ./data/amazon_reviews_2018/meta/meta_Automotive.json\n",
    "  - ./data/amazon_reviews_2018/meta/meta_Cell_Phones_and_Accessories.json\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"preprocessing/create_dataset.py\" will instantiate a local dask cluster automatically. The default values can be found in \"configs_hydra/dask/local.yaml\"\n",
    "\n",
    "```\n",
    "n_workers: 10\n",
    "threads_per_worker: 2\n",
    "memory_limit: '20GB'\n",
    "local_directory: /tmp/dask\n",
    "dashboard_address: 8999\n",
    "```\n",
    "\n",
    "Custom settings can be passed by adding a new yaml file under \"configs_hydra/dask/\". For example for a config file \"my_settings.yaml\" run:\n",
    "\n",
    "`python preprocessing/create_dataset.py dask:my_settings.yaml`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "panko",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}