01. Problem Identification
Amid rapid digitalization in DKI Jakarta, the younger generation faces escalating anxiety and depression due to excessive screen exposure, late-night social media scrolling, and high academic pressure. Traditionally, public health responses have been reactive—the government only becomes aware of a crisis when patients are already in severe condition. The lack of integration between self-reported telemetry logs, clinic clinical assessments, and counseling history leads to delayed early interventions.
Multi-Silo Data Fragmentation: 300,000 records of device telemetry, self-reported psychometric questionnaires, and counseling logs are scattered across different formats without a single source of truth.
Sensor Anomalies & Medical Privacy Standards: High sensor noise (screen time > 20 hours/day) and the obligation to protect medical privacy records (HIPAA/GDPR) to prevent youth identity leaks.
Lack of Spatial Predictive Systems: Absence of high-risk regional mapping and forecasting systems to anticipate seasonal surges (e.g., during school exam periods).
02. The Solution
Built a unified Early Warning System using the Databricks Lakehouse ecosystem. The Medallion Architecture cleans and integrates 300,000 data records in a distributed manner using PySpark, secures privacy via SHA-256 cryptography, automates data flows with Lakeflow Jobs, and provides a Databricks Genie AI agent for natural language exploration along with interactive BI dashboards featuring 30-day predictive projections.
Controlled ingestion in Bronze with StructType, automated isolation of anomalous records into a Quarantine table, and smart logic cleansing in Silver.
Transformation of user_id into consistent one-way SHA-256 hashes to maintain inter-table relationships without exposing patient personal identities.
Execution of multi-stage parameterized SQL batches using dbutils.widgets with automated integrity validation tests (PASS/FAIL).
Provisioning a semantic Business View, medical terminology dictionary, domain instructions, and 8 trusted benchmark queries for instant hallucination-free analytics chat.
Interactive visualization with 5 KPI cards, regional morbidity maps, screen time vs sleep loss correlations, and projected healthcare facility needs via AI_FORECAST.
03. Crafting & Engineering
System Architecture Flow
Medallion Lakehouse & Automation Pipeline
Raw Data Onboarding & Schema Enforcement
Unity Catalog Volumes / Delta FormatStorage of raw CSVs in Unity Catalog Volume (raw_dataset), strict StructType schema enforcement, recording provenance metadata (_metadata.file_path, ingested_at), and compilation into ACID-guaranteed Delta Lake.
Data Quality Engine, Quarantine & Masking
PySpark 3.4 (DataFrame API)Data profiling, separating screen time anomalies (>18 hours) to Quarantine Delta Table, capping late-night social media, smart imputation of null values for sleep index, and PII identity masking with SHA-256 Hashing.
Curated Business Data Marts & Aggregations
Databricks SQL / Delta LakeMulti-table relational joins based on user_id_masked with ROW_NUMBER() window functions, creation of regional & stressor analytical data marts, and data reconciliation assertions to guarantee data integrity.
Lakeflow Automation & Validation Workflows
Databricks Lakeflow JobsPipeline automation using Parameterized Notebooks (dbutils.widgets), 4-stage SQL Batch processing, dependency DAGs, validation assertions (PASS/FAIL), and automated Repair Run mechanisms.
Databricks Genie AI Agent & EWS Dashboard
Databricks Genie + Databricks SQLProvision of Semantic Business View for GenAI Agent, clinical synonym dictionary, Trusted Queries, benchmark testing suite, and EWS visualization complete with 30-day AI_FORECAST.
Multimodal Datasets (300,000 Total Rows)AI-Generated Dummy Data
Daily activity telemetry logs: screen time, late-night social media time (22:00–04:00), sleep quality index, exercise duration, and device OS type.
Standardized medical questionnaire results: GAD-7 anxiety scores (0–21), PHQ-9 depression scores (0–27), major stress triggers, and age range categories.
Peer tele-counseling intervention logs: satisfaction scores, message sentiment analysis scores, session modalities, and clinic/hospital referral recommendation status.
Technology Selection
Key Engineering Challenges Solved
- Handling timestamp format heterogeneity and sensor logic anomalies (late-night social media use > total screen time) without breaking daily time-series data integrity.
- Implementing a data quarantine architecture to isolate anomalous data (screen time > 18 hours) to prevent bias in calculating regional average morbidity.
- Maintaining patient medical record privacy according to HIPAA/GDPR standards via one-way masking (Cryptographic Hashing SHA-256) on user_id while preserving inter-table relationships.
- Eliminating AI hallucination risks in the Databricks Genie Agent by designing a semantic Business View, clinical synonym dictionary, and a collection of Trusted Queries.
- Performing Data Reconciliation using automated cross-layer assertion checks to guarantee zero data loss after multi-table joins.
04. Interactive Lakehouse Dashboard
Live embedded Databricks SQL Early Warning System with interactive filters and predictive analytics.
Initializing Databricks AI/BI Session...
Menghubungkan ke Databricks Workspace & rendering visualisasi
Memantau 23.562 remaja aktif, 5.853 kasus rujukan (24.84%), rata-rata durasi layar 8.01 jam, dan rata-rata skor depresi PHQ-9 (13.8) di DKI Jakarta.
Jakarta Selatan dan Timur mencatatkan tingkat rujukan tertinggi. Pemicu stres utama didominasi beban akademik (31.8%) dan konflik keluarga (18.6%).
Fitur prediktif built-in Databricks memproyeksikan volume kedaruratan krisis untuk perencanaan kapasitas intervensi faskes & konselor sebaya secara antisipatif.
05. Databricks Genie AI Agent
Interaksi kecerdasan buatan berbasis Natural Language Text-to-SQL dengan 2 mode eksplorasi.
Databricks Genie AI
ConnectedSemantic Text-to-SQL Lakehouse Intelligence