Pharmaceutical Supply Chain Optimisation

A machine learning pipeline for predicting pharmaceutical shipment delivery risk. The system processes supply chain data, engineers features, and trains models to identify shipments at risk of late delivery — enabling proactive supply chain management.

---

Table of Contents

• Overview
• Architecture
• Project Structure
• Prerequisites
• Installation
• Configuration
• Usage
  • Running the Airflow Pipeline
  • Running the Notebook
• Data Pipeline
• Dataset
• Technologies
• Development

---

Overview

This project builds an end-to-end ML pipeline to optimise pharmaceutical supply chains by predicting the probability that a shipment will arrive late. The pipeline covers:

• Data ingestion – loading raw supply chain records from CSV
• Feature engineering – parsing dates, encoding categoricals, handling missing values
• Data splitting – stratified train/test split with optional class balancing via upsampling
• Model training – experiment tracking through MLflow (Databricks integration)
• Pipeline orchestration – scheduled Airflow DAG running each step in sequence

---

Architecture

Raw CSV Data
     │
     ▼
┌─────────────┐     ┌──────────────┐     ┌─────────────┐     ┌───────────────┐
│ import_data │ ──▶ │ process_data │ ──▶ │  split_data │ ──▶ │ Model Training│
│  (ingest)   │     │  (engineer)  │     │  (prepare)  │     │   (MLflow)    │
└─────────────┘     └──────────────┘     └─────────────┘     └───────────────┘
         ▲
         │  Orchestrated by Apache Airflow (Docker)

---

Project Structure

.
├── config/
│   ├── config.yaml          # Model and dataset configuration
│   └── airflow.cfg          # Airflow settings
├── dags/
│   └── example_dag.py       # Airflow DAG defining the pipeline
├── data/
│   └── supply_chain.csv     # Raw pharmaceutical supply chain dataset
├── notebook/
│   └── Supply_Chain_Optimisation.ipynb  # Exploratory analysis and modelling
├── src/
│   ├── config.py            # YAML configuration loader
│   ├── import_data.py       # CSV data ingestion
│   ├── main.py              # CLI entry point
│   ├── process_data.py      # Data cleaning and feature engineering
│   └── split_data.py        # Train/test splitting with class balancing
├── docker-compose.yaml      # Docker Compose for Airflow services
├── requirements.txt         # Python dependencies
└── README.md

---

Prerequisites

• Python 3.9+
• Docker and Docker Compose (for Airflow orchestration)
• An MLflow tracking server (or Databricks workspace) for experiment logging

---

Installation

Python environment

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Airflow (Docker)

# 1. Initialise the Airflow metadata database
docker compose up airflow-init

# 2. Start all Airflow services in the background
docker compose up -d

The Airflow web UI will be available at http://localhost:8080.

---

Configuration

All pipeline parameters are stored in config/config.yaml:

system:
  raw_dataset_path: "./data/supply_chain.csv"
  processed_dataset_path: "./data/supply_chain_processed.csv"
  model_save_path: "./models/model.pkl"

model:
  test_size: 0.2       # Fraction of data held out for testing
  random_state: 42

model_params:
  n_estimators: 100
  max_depth: 10
  random_state: 42

---

Usage

Running the Airflow Pipeline

Once the Docker services are running, trigger the hello_world_dag DAG from the Airflow UI or via the CLI:

docker compose exec airflow-scheduler airflow dags trigger hello_world_dag

The DAG executes the following tasks in order:

print_hello → fetch_data → preview_data → process_data → split_data → done

Running the Notebook

Open the Jupyter notebook for interactive exploration and model training:

jupyter notebook notebook/Supply_Chain_Optimisation.ipynb

The notebook connects to MLflow for experiment tracking. Set the MLFLOW_TRACKING_URI environment variable before launching if you are using a remote tracking server.

Running the Entry Point Directly

python src/main.py

---

Data Pipeline

Intermediate data files (X_train, X_test, y_train, y_test) are written to /tmp/ml_splits/ during pipeline execution.

---

Dataset

File: data/supply_chain.csv
Rows: 10,325 pharmaceutical shipment records
Encoding: Latin-1

Key column groups:

---

Technologies

---

Development

Linting

ruff check .
ruff check . --fix   # auto-fix safe issues

Tests

pytest

The CI/CD pipeline (.github/workflows/run_test.yaml) runs both Ruff and Pytest on every push.