Playlist Recommendation System

This project studies real Spotify playlists to understand what music people listen to together. It lays the groundwork for recommending songs based on those patterns.

It works in two stages:

1. ETL (Extract, Transform, Load). Reads raw playlist data and organizes it into clean, analyzable files
2. Dashboard & API. Displays insights like most popular songs and artists

---

Project Structure

etl/            Spark ETL: extract, transform, load, build stats + interaction matrix
  run_etl.py    ETL entry point
api/
  main.py       FastAPI app and routes
  templates/    Jinja2 templates (server-rendered dashboard)
  static/       CSS served by the FastAPI app
utils/          Shared config, logging, Spark session, schema, S3 upload
tests/          pytest test suite
config.yaml     Dataset paths, output paths, Spark tuning
dataset/
  mpd.example.json  Small bundled sample slice (tracked in git, used by default)
  data/             Full MPD JSON slices (not tracked in git; download separately)
output/           ETL outputs (not tracked in git)
hadoop/           Windows winutils for local Spark (not tracked in git)
pyproject.toml    Project metadata, dependencies, ruff config
README.md

The project is a single Python service. Everything, ETL and the dashboard, runs from the project root, with no separate frontend build or Node toolchain required.

---

How the ETL Process Works

Step-by-step

1. Extract. Read JSON playlist files from the dataset folder
2. Transform. Flatten nested playlists into rows, clean up song IDs, assign unique playlist numbers
3. Build lookups. Create ID tables for songs and playlists
4. Compute stats. Count how often songs and artists appear
5. Build interaction matrix. Map which songs appear in which playlists
6. Load. Save everything to the output/ folder

Run the ETL

python -m etl.run_etl

Run this once before starting the dashboard. It populates the output/ folder.

ETL evolution (naive vs optimal)

The pipeline went through three stages as the dataset grew:

Naive. Read entire JSON files into memory, process sequentially, output CSV. Works for small samples but does not scale.

Optimized. Switch to generator-based extraction (one row at a time), Parquet output (columnar, compressed), and ProcessPoolExecutor to process multiple slices in parallel. This cut end-to-end ETL from ~277 seconds to ~50 seconds.

Current. Apache Spark handles the heavy JSON parsing and flattening across partitions. Post-processing (ID mappings, stats, interaction matrix) still runs in pandas/scipy on the written Parquet files. See etl/track_etl.py.

---

Input: What the Raw Data Looks Like

The project uses the Spotify Million Playlist Dataset. About 1 million real playlists that Spotify users created.

How it's organized

The dataset comes as ~1,000 JSON files (called "slices"). Each file contains 1,000 playlists, and each playlist contains a list of songs.

dataset/
  mpd.example.json          ← small sample (3 playlists) for testing
  data/
    mpd.slice.0-999.json    ← 1,000 playlists
    mpd.slice.1000-1999.json ← another 1,000 playlists
    ... (about 1,000 files total)

What one playlist looks like inside a JSON file

Each file is structured like a folder of playlists, where each playlist has a list of songs:

{
  "playlists": [
    {
      "name": "Throwback Party",
      "pid": 0,
      "tracks": [
        {
          "track_uri": "spotify:track:1301WleyT98MSxVHPZCA6M",
          "track_name": "HUMBLE.",
          "artist_name": "Kendrick Lamar",
          "album_name": "DAMN."
        },
        {
          "track_uri": "spotify:track:7BKLCZ1jbUBVqRi2FVlTVw",
          "track_name": "Closer",
          "artist_name": "The Chainsmokers",
          "album_name": "Collage EP"
        }
      ]
    }
  ]
}

Important detail: pid is only unique within one file. Playlist #0 in file A and playlist #0 in file B are different playlists. The ETL handles this by assigning each playlist a globally unique playlist_id.

Configuration

In config.yaml:

dataset_path: "dataset/mpd.example.json"   # small sample (default)
# dataset_path: "dataset/data"            # full dataset (after download)

---

Output: What Gets Produced

After ETL runs, the output/ folder contains organized data files:

---

Running the App

uvicorn api.main:app --reload

Open http://localhost:8000 for the dashboard (stat cards, top-tracks list, top-artists list), rendered server-side with Jinja2 from the ETL output. If the ETL hasn't been run yet, the page shows a notice instead of erroring.

Interactive API docs: http://localhost:8000/docs

---

Setup

Dependencies are declared in pyproject.toml. The base install covers the API and dashboard; ETL (PySpark) and S3 backup are optional extras.

API only (no ETL, no PySpark). Serve an already-populated output/ folder:

pip install -e .
uvicorn api.main:app --reload

Full local dev (ETL + API + S3 backup + tests/lint):

pip install -e ".[all]"
python -m etl.run_etl   # uses dataset/mpd.example.json by default
uvicorn api.main:app --reload

Requires Java 8+ and Hadoop winutils on Windows for PySpark. See hadoop/ directory.

---

Tests

pytest -q