Our data architecture follows a structured, layered approach based on the Medallion model. It aligns with Microsoft Fabric principles and best practices, supporting efficient data ingestion, transformation, governance, and reporting. Landing Layer This layer receives raw data files (typically CSV format) from external systems into secure landing zones. Bronze Layer Bronze provides the initial structured … Read more
Scope: Updates to Common Data Model (CDM) and project-specific viewsTotal File Changes: 34 🔄 Summary of Key Changes ✅ Stabilised Views for 202409 Reporting These views are now stable and finalised for the 202409 reporting cycle, helping to prevent reuse issues in future CDM deployments. 🛠️ Updated PRUs for Project Views CAD Region Updates: USD … Read more
Summary of Changes by View 1. project_details_plus_USD & project_details_plus_CAD Status Logic Updated Project Manager Email Added 2. project_details_plus_CAD & project_details_USD Column Name Adjustment 3. project_details_plus_APC, customer_details_APC, project_details_APC PRU Code Mapping Enhancement 4. global_company & global_company_202409 Expanded Source 5. global_organisation & global_organisation_202409 Same as Above 6. employee_item_codes_absence, employee_item_dates_absence, employee_item_values_absence Minor Logic Refinements 7. dates_DIM_DATE_202409 Performance Optimisation … Read more
🆕 What’s New New Project View (Item 207) Added project_details_plus_APC to complete the set for regional project reporting. Improved Date Views (Items 20–21, 27–30, 204–205) Programme ID Enhancements (Items 206, 208) Integrated clearer programme identification fields from MIS80 into global organisation views, allowing for cross regional identification. 🔂Development/Improvements for cdm Notes The following files were … Read more
Commits: 24Files Changed: 33 🔄 Alterations 1. global_organisation_ifs.sql3. global_organisation_202409.sql Update: Retention of previous IFS logic for PRU codes and Programme assignment.Note: 202409 now draws from the MIS80 source and includes PRU codes and Programmes across all regions. 4–5, 6–12, 30–33:Includes: Update: Continued development of CDM views for the new generation.Main change: Standardisation on the field … Read more
Additions Common Data Model New Common Data Model (CDM) views have been introduced, conforming to the latest standard: Employee Domain Customer & Supplier Domains Project Domain These views establish a consistent and scalable approach to data modelling across core business domains, streamlining downstream reporting and integration. CDM Functions Key functions have been revised to support … Read more
Additions 202409 ⚠️ Created:global_company_202409.sql ⚠️ Created:global_organisation_202409.sql⚠️ Created:customer_details_202409.sql Datamart views have been created to ensure a seamless transition for clients migrating from 202404 to 202409, providing a structured and usable view for ongoing data integration. These views have been explicitly versioned with the 202409 signature to: Common Data Standard ⚠️ Created:project_core.sql ⚠️ Created:project_codes.sql ⚠️ Created:employee_item_codes_expense.sql ⚠️ … Read more
Additions Customer Details & Project Details APC Alterations General Ledger Transaction Customer Details (CAD & USD) Business Opportunity Dates (Busopp Dates) Notes The following SQL scripts have been modified and differ from previous versions: There were additional alterations to cdm to support transaction error logging 1. ⚠️ DIFFERENT: create_table_comment.sql2. ⚠️ DIFFERENT: create_table_log.sqlSource ID altered to … Read more
When importing rich text data into a Data Warehouse (DWH), it’s often necessary to remove unwanted HTML tags while preserving essential formatting. This guide outlines a method using Python’s BeautifulSoup library to clean and structure the data efficiently. Why Strip HTML Tags? Rich text from sources like web applications, CMS platforms, and APIs often includes … Read more
Additions Customer Details USD ✅ New Addition customer_details_USD – Part of the CAD/USD series of changes.⚠️ Contains Created (Hash) Customer IDs – This will be corrected in the next release. Considerations & Next Steps 🔹 Customer ID Fix Pending – Plan for a smooth transition when correcting the Customer ID field.🔹 UNION-based “mega” list for … Read more