A calm, consistent, and clever approach to data engineering. 🔹 1. Look in the Box First Before inventing a workaround, check what already exists.If Microsoft or IFS built it, use it — it’s likely more robust, secure, and supported. 🔹 2. Easy Landings Data should arrive safely and predictably.Keep import containers simple, standardised, and transformation-free.Bronze … Read more
Should We Reuse Standard Dynamics 365 / Dataverse Tables for Our In‑House Apps? 1) Why We’re Talking About This We’re starting an important conversation: when we build apps internally, should we lean on the standard tables already in Dynamics 365 / Dataverse (like Account, Contact, Opportunity, Project), or should we keep spinning up new SharePoint … Read more
Competency Area Element Indicative Score (1–5) Target (6 mo) Evidence / Example Activities Priority Data Management Devise & implement MDM processes (classification, security, quality, ethics, retrieval & retention) 4 5 Introduced soft/hard delete strategy; implemented RPV & last-seen frameworks; designed employee / project / customer domain MDM structures; strong data security awareness (container-level). M Derive … Read more
Month Theme / Focus Area Key Actions Tangible Evidence / Deliverables Success Indicators Month 1 Baseline review & planning • Confirm current Level 5 competency scores and priorities• Align with manager/mentor on fellowship expectations• Map your influence across Data Engineering + Analytics disciplines Updated self-assessment & evidence tracker Clear alignment between Data Engineering objectives and … Read more
This proposal outlines a phased approach to broaden the scope of our Data Engineering capability, incorporating Microsoft Fabric as a unified analytics platform and evolving our approach to include foundational elements of AI Engineering. The goal is to create an agile, scalable, and intelligent data platform that supports operational analytics, predictive insights, and innovation. 1. … Read more
A novel about IT, DevOps and Helping your Business Win
Role Purpose The Head of Data Engineering & Analytics provides strategic and operational leadership for the organisation’s data platforms, data engineering capability, and analytical services. The role exists to ensure that: The role balances: in order to maximise the value of data across the organisation. Scope and Impact The role: The Head of Data Engineering … Read more
🎯 Purpose This framework ensures that values in complex metadata structures (e.g. project earned value, invoice accounting, or employee attributes) are verified for presence even when using incremental loads. It protects against stale data persistence or silent value loss, using metadata-driven auditing. 🧩 Problem Context In many CDM views, values are upserted using procedures like … Read more
🎯 Objective To proactively expire or flag metadata rows that are no longer present in the source data even if their values haven’t changed, avoiding persistence of stale or misleading data. 1. 🏗 Recommended Table Extension Add these optional columns to any table using patch_upsert or put_update: 2. 📥 Staging Input Table – stage.project_values_current Your … Read more
🎯 Core Issue: 🔍 Risks of the Current Approach Scenario Problem Value disappears entirely You keep the last-seen value forever unless a new version overwrites Value moves to another key The old one is retained, and the new one appears as duplicate context Periodic reporting gaps Reporting shows a “stale” value that’s no longer present … Read more