Clinical Data Ingestion Pipeline

A production-grade, HIPAA Safe Harbor-compliant data engineering platform that ingests raw clinical imaging metadata and DICOM manifests from two independent pediatric hospital networks operating on heterogeneous EHR systems (CHOC on Cerner, Rady Children's on Epic), unifies them through a PySpark Medallion architecture inside Databricks, and delivers a fully anonymized, ML-ready Parquet dataset to a downstream computer vision team training brain lesion segmentation models. The system is built around a hybrid decoupled ingest pattern: a stateless Streamlit portal handles clinical coordinator uploads and writes directly to AWS S3 (independent of cluster compute), while a scheduled Databricks Workflow Job executes the Bronze → Silver → Gold transformation pipeline on demand. Raw patient identifiers are cryptographically masked with deterministic SHA-256 surrogate keys, and exact admission dates are generalized to enrollment years prior to ML export — satisfying HIPAA Safe Harbor de-identification standards (45 CFR §164.514(b)).