Data Engineer

Value Presence Verification (RPV) + Last-Seen Tracking Framework

by

🎯 Purpose This framework ensures that values in complex metadata structures (e.g. project earned value, invoice accounting, or employee attributes) are verified for presence even when using incremental loads. It protects against stale data persistence or silent value loss, using metadata-driven auditing. 🧩 Problem Context In many CDM views, values are upserted using procedures like … Read more

CDM Value Presence Reconciliation Framework

by

🎯 Objective To proactively expire or flag metadata rows that are no longer present in the source data even if their values haven’t changed, avoiding persistence of stale or misleading data. 1. 🏗 Recommended Table Extension Add these optional columns to any table using patch_upsert or put_update: 2. 📥 Staging Input Table – stage.project_values_current Your … Read more

Row Presence Verification (RPV)

by

🎯 Core Issue: 🔍 Risks of the Current Approach Scenario Problem Value disappears entirely You keep the last-seen value forever unless a new version overwrites Value moves to another key The old one is retained, and the new one appears as duplicate context Periodic reporting gaps Reporting shows a “stale” value that’s no longer present … Read more

DE306: Data Governance, Security & Compliance

by

SFIA Reference: SCAD Level 4/5 Data Engineer (Grade 3) Competency Description Implements access controls, data classification, and auditing to ensure data security and regulatory compliance. Learning Outcomes Evidence Requirements Level 4: Level 5: Suggested Learning Activities Recommended Resources