Modern data platforms are not built on single scripts or ad‑hoc notebooks. They rely on reusable, well‑designed utility functions that handle extraction, transformation, auditing, and historical tracking in a consistent way. This article walks through a real‑world Python utility module used in Microsoft Fabric, explaining every major function, what problem it solves, and how it … Read more
Overview The cdm_Archive_to_STG notebook is a critical component in the cdm_today and cdm_Archive pipeline. Its primary role is to create a staging table that represents a point-in-time snapshot of source data, which is then used by the cdm__Archive_upsert notebook to accurately update the main dimension table. This staging layer acts as a buffer between raw … Read more
Overview The cdm__Archive_upsert notebook is designed to help keep accurate historical records of data. It implements a technique called Slowly Changing Dimension Type 2 (SCD2), which means instead of overwriting old information, it creates a timeline of changes so you can see what was true at any point in time. This method ensures that previous … Read more