From Silos to Streaming: Engineering a Real-Time Service-Performance Dashboard
Context
A large organisation with several semi-autonomous business units wanted to replace a weekly "roll-up" spreadsheet with a real-time dashboard that exposed key service metrics—turn-around times, queue backlogs, and online-transaction success rates. Each division already owned its own BI tooling, so forcing wholesale migration onto a single platform was out of scope. The target was a live public site for customers plus a richer internal workspace for analysts, built in six months and capable of near-real-time updates.
Target Platform and Design Principles
We approached the problem as a federated data-mesh. Every division would publish its curated dataset to a shared lakehouse but retain ownership of ingestion pipelines and domain logic. Azure was selected for hosting because enterprise agreements and identity primitives were already in place. The architecture had three layers: a per-division data-product zone in Synapse, a central SQL Pool for cross-domain joins and KPI definitions, and a front-end built with React - served from Azure Static Web Apps and embedding Power BI visuals where those existed.
Data Ingestion and Governance
Divisions exposed data in three patterns. Some streamed CDC events into Event Hubs, others delivered nightly parquet drops to ADLS Gen 2, and smaller teams allowed direct queries against their existing datamarts. Each dataset carried a versioned JSON schema stored in Azure Purview plus a YAML descriptor that captured refresh SLAs and data-quality expectations. Great Expectations tests ran inside Synapse pipelines; any failure blocked promotion to the canonical store and alerted the owning steward. KPI logic was materialised with dbt; a cross-division metrics council met fortnightly to review pull requests.
Front-End, Caching, and Interactivity
The public portal cached responses in Azure Front Door for two minutes and invalidated the cache whenever a pipeline completed. Charts used Apache ECharts for accessibility and performance. Where a metric already existed as a polished Power BI report, we embedded it through Power BI Embedded and restyled via CSS to maintain visual coherence. The internal workspace leveraged Arrow Flight over Synapse Serverless, streaming parquet into a browser-side DuckDB running in WebAssembly, which gave sub-second drill-downs without hitting the SQL pool for every interaction.
Security, Cost, and Observability
Authentication used Azure AD B2C for the public site and the corporate AD tenant for staff. Row-level security definitions in Synapse propagated automatically to Power BI via DirectQuery, ensuring consistent entitlements. The dedicated SQL Pool scaled down during off-hours through an Automation runbook, and storage costs were kept predictable by tiering aged OpenSearch indices to UltraWarm. All pipelines emitted OpenTelemetry traces that landed in Grafana Cloud, so engineers could follow a record from landing zone to public widget in a single span chain.
Production Outcomes
The operations team eliminated 15 person-hours of manual reporting each week. During an unplanned spike of twenty-two-thousand concurrent external users, cache-hit ratios stayed above ninety-six percent and origin load remained flat. Transparency drove data quality improvements: once metrics were publicly visible, divisions prioritised upstream fixes instead of patching dashboards downstream.
Technical Takeaways
Contract-first schemas let multiple teams publish data without centralising ETL code, provided that automated tests enforce adherence. WebAssembly databases such as DuckDB enable genuine interactive analytics in the browser, but they need client-side memory guards for low-spec machines. Finally, static web hosting plus CDN invalidation outperforms heavy report servers for citizen-scale dashboards; embedded BI visuals are best reserved for specialist drill-downs.
About Us
Arxium is a software consultancy focused on helping government agencies, banks, and enterprises build systems that matter. We specialise in modernising legacy platforms, designing digital services, and delivering scalable, cloud-native architectures.
Leaders across Australia trust us to solve complex technical problems with clarity, pragmatism, and care. Whether it's migrating infrastructure, integrating systems, or launching a public-facing portal—we make software work in the real world.
Contact us↗ to start a conversation about your next project.