Building production-grade data pipelines that turn raw data into reliable, analytics-ready systems.
I'm a Data Engineer specializing in building end-to-end data pipelines, ELT systems, and analytics infrastructure.
Final year AI & Data Science student at SIT — graduating June 2026.
What I build:
- Ingestion → API integrations, schema validation, quarantine pipelines
- Transformation → dbt staging, intermediate, and mart layers
- Orchestration → Airflow DAGs with retry logic and failure alerting
- Warehousing → BigQuery data models optimized for analytics and ML
✔ Production ELT pipelines — not just tutorials
✔ dbt data modeling across staging, intermediate, and mart layers
✔ Airflow orchestration with monitoring and alerting
✔ ML-ready feature engineering on top of warehouse data
✔ Real-time anomaly detection systems
• Apache Airflow • dbt Core • BigQuery • Docker • Python • SQL
• GCP (Cloud Storage, Cloud Run, Secret Manager) • Git & GitHub
• Pandas • NumPy • scikit-learn
• Exploratory Data Analysis
• Anomaly Detection (IsolationForest)
• ML-ready feature engineering (14-feature vectors)
• Docker Compose • Shell Scripting • Linux
• Schema validation & data quality contracts
• Backfill pipelines with checkpoint restartability
Python • Airflow • dbt • BigQuery • Docker
End-to-end ELT platform answering a real analytical question: does rainfall materially improve city AQI, and for how long?
- Ingests AQI from 150+ government monitoring stations (~4,000 records/day) via CPCB API + OpenWeatherMap
- Schema validation before every BigQuery load — quarantines 3–8% of records daily into
raw_aqi.invalid_records - dbt transformation across 3 layers: staging → intermediate → marts (daily summaries, city rankings, 14-feature ML vectors)
- Airflow orchestration with
max_active_runs=1to prevent nondeterministic mart snapshots - Backfill scripts with checkpoint files for restartable historical loads
- Fully containerised — runs end-to-end with a single
docker compose up
Python • scikit-learn • Streamlit
Production-style anomaly detection on server log streams.
- IsolationForest with rule-based alert filtering — hybrid approach catches known patterns and novel behavioral drift
- Severity classification layer (LOW / MEDIUM / CRITICAL) on top of raw anomaly flags
- Joblib model persistence for reuse across detection runs without retraining
React • Node.js • DICOMweb • Orthanc • PostgreSQL • Docker
Production-style medical imaging workflow platform enabling browser-based DICOM upload, metadata indexing, and diagnostic viewing — without any local installation.
- Designed a modular healthcare imaging architecture integrating:
- OHIF Viewer (zero-footprint rendering)
- Orthanc (DICOM server)
- PostgreSQL (metadata-only storage)
- Implemented full DICOMweb pipeline:
- STOW-RS → upload
- QIDO-RS → metadata query
- WADO-RS → image retrieval
- Built backend orchestration layer handling:
- streaming DICOM uploads (no local storage)
- metadata normalization and persistence
- read-only DICOMweb proxy for viewer integration
- Developed React dashboard for:
- multi-file DICOM upload
- study search and filtering
- seamless OHIF launch using StudyInstanceUID
- Ensured strict separation of concerns:
- no DICOM stored in DB
- frontend does not render images
- viewer fully decoupled
- Containerized full system with Docker Compose (Orthanc + backend + frontend + viewer + DB)
System Flow:
Upload → Orthanc → Metadata Sync → PostgreSQL → OHIF Viewer
- Streaming pipeline → Kafka + PySpark + BigQuery (real-time data processing)
- Natural language query layer → plain English → SQL on BigQuery using LLM APIs
- GCP cloud deployment → moving local pipelines to production cloud infrastructure
Turning raw data into systems that matter.