What are Deployment Pipelines? Pipeline Structure Stage Comparison in UI Item Pairing & Status Selective Deployment Deployment Rules Lakehouse & Deployment Pipelines in Fabric Deployment pipelines move Lakehouse metadata, including shortcuts, but do not copy data or table schemas. After deployment, shortcuts still point to the original source, and the Lakehouse will be empty unless … Read more
When working with Git repositories in Azure DevOps, you might need to move a file from one folder to another without losing its history. The best way to do this is by using the git mv command. This ensures Git recognizes the move as a rename and keeps the file’s commit history intact. Why Not … Read more
Issued by: Data & Analytics TeamApplies to: All data integrations, pipelines, and views entering or leaving the Data WarehouseStatus: Approved DraftDate: [Insert Date]Version: 1.0 1. Purpose This standard defines the structure and principles for integrating data from source systems—including ERP (IFS), Dynamics 365, Power Apps, and SharePoint—into the enterprise Data Warehouse. It ensures that data … Read more
A calm, consistent, and clever approach to data engineering. 🔹 1. Look in the Box First Before inventing a workaround, check what already exists.If Microsoft or IFS built it, use it — it’s likely more robust, secure, and supported. 🔹 2. Easy Landings Data should arrive safely and predictably.Keep import containers simple, standardised, and transformation-free.Bronze … Read more
This process explains how Data Engineering (DE) and Analytics Reporting (A) work together to turn a business need into a trusted, production-ready report. It’s designed for non-technical managers: no jargon, just what happens, who’s responsible, and what the outcomes are. Phase 0: Definition – Ground Zero PurposeLay the foundation before vision-shaping begins. Ensure clarity on … Read more
The Data Engineering Team plays a crucial role in ensuring the smooth operation and continuous improvement of our data services. However, many people may not fully understand how we structure our development cycles and why we operate in a particular way. This article aims to provide insight into a typical week in the life of … Read more
Role: Project ManagerContext: Large-scale technical consultancy engagement requiring coordination across engineering, finance, and delivery teams 🔪 Challenge The project manager needed to: Problems included: “I spent hours each week trying to make sense of data from five different places—and still couldn’t guarantee it was accurate.” 🚀 Solution The Data Engineering team: 🌟 Outcome “Now I … Read more
At the heart of every report, dashboard, and project delivery, there’s data—clean, consistent, and flowing on time. That’s what Data Engineering delivers. Whether you’re chasing a deadline, answering a tough question, or leading a major project—we make sure you have the data you need, when you need it. ⚖️ Clarity You get clean, reliable data … Read more
Proposal for a white paper on Common Data Model (Datamarts) Proposed Structure for the White Paper 1. Executive Summary 2. Introduction 3. Objectives 4. Proposed Standardised Format 5. Naming Conventions 6. Standardised Structure for Views 7. Metadata Format 8. Implementation Roadmap 9. Benefits 10. Conclusion Naming Suggestions for Views Here are some options for naming … Read more
This white paper proposes a comprehensive Data Protection Impact Assessment (DPIA) framework for the Data Warehouse, detailing the potential commercial and personal data risks across individual data marts—including Projects, Employee, and Customer—and introducing a “Defence by Design” methodology to mitigate these risks effectively. Objectives Data Marts Overview and Risk Assessment 1. Projects Data Mart 2. … Read more