Building Modern Data Platforms Without Legacy Bottlenecks ft Andrew Jones

In this episode of The Data Engineering Show, host Benjamin Wagner sits down with Andrew Jones, Staff Data Engineer at LocalStack, to explore how data teams can transition from centralized, bottlenecked operations to scalable, self-serve platforms - while managing real-time reliability demands that now define modern data infrastructure.

- How to free up data team capacity by automating support workflows—Andrew's hackathon project automated 15% of access requests using Slack bot agents, reclaiming engineering time for platform-building rather than firefighting.

- Why data reliability has shifted from a "nice-to-have" to a customer-facing requirement—As companies embed data into product features and AI agents, pipeline failures cascade directly to customers, demanding disciplined software engineering practices like observability, SLOs, and on-call protocols.

- The framework for segmenting data personas and enabling autonomy without chaos—Technical teams (engineers, product managers) get self-serve tooling to own their data lifecycle; business teams receive curated, pre-built datasets through accessible tools like BI platforms and AI agents.

- How to shift left on data governance by rethinking upstream data contracts—Rather than relying on manual documentation and checklists, enforce reliability guarantees at the source, ensuring downstream pipelines can't fail due to upstream data quality degradation.

- Why semantic models and context layers are becoming critical for AI agent grounding—As agents proliferate across organizations, they need structured knowledge of business definitions and data relationships to make contextually accurate decisions and queries.

- The importance of questioning assumptions during rapid industry transformation—Admit what you don't know yet, stay open-minded about rethinking data ownership and governance, and avoid blindly replicating legacy processes in a fundamentally changed landscape.

Andrew Jones is a Staff Data Engineer at LocalStack, bringing a decade of platform-building expertise across companies including GoCardless and Arm. Specializing in data platform architecture and the shift toward data-driven product features, Andrew has pioneered approaches to data reliability and contracts that enable organizations to leverage data as a competitive advantage. In this episode, Andrew shares invaluable insights on transitioning centralized data teams into self-serve platforms, managing the tension between supporting legacy systems while building modern infrastructure, and rethinking data governance in the age of AI agents. His pragmatic approach to solving real-world data challenges makes this conversation essential for data leaders navigating the evolving landscape of analytics and data engineering.

"I've been building platforms really since I started, and for the last ten years, I was focused on data platforms. That's really my passion." - Andrew Jones

"The idea there is you can avoid all the cloud costs that you have when you're trying to develop against cloud and work much quicker because it's all local, it's all instant, it's all emulated." - Andrew Jones

"Data warehouses can be quite expensive if they're up and running, and that's one of the things we're seeing from our customers—they want to make it quicker and also reduce the cost associated with development and CI checking against data warehouses." - Andrew Jones

"We want to enable people in the business to be more self-serve and make use of the data and add more telemetry to our product without having to involve a person each time." - Andrew Jones

"One of the biggest challenges is you're trying to build this new world while you're still trying to support the old world—you can't just drop everything and say we're going away for six months." - Andrew Jones

"With agents now, we can move a lot quicker and build general tooling much quicker than we could before, which means you can automate more than you could before." - Andrew Jones

"People want to use data either directly or via AI and agents to create product features that are differentiated by the unique data a company has, and that requires a high level of reliability." - Andrew Jones

"If you want the output to be reliable, then your bits gotta be reliable, but also the bit upstream—which means you have to talk to those people and explain to them how important it is." - Andrew Jones

"Data contracts are getting more and more traction because more and more companies are trying to use data for customer-facing features and realizing that if you want that to be reliable, you need to go upstream and make sure it's fixed there." - Andrew Jones

"Things are changing so fast that we're still working a lot of these things out, and I think one of the best things we can do is be open-minded and question some of the assumptions we've had before." - Andrew Jones

If you enjoyed this episode, make sure to subscribe, rate, and review it on Apple Podcasts, Spotify, and YouTube Podcasts. Instructions on how to do this are here: https://www.fame.so/follow-rate-review 

The Data Engineering Show is brought to you by firebolt.io and handcrafted by our friends over at: fame.so

Previous guests include: Joseph Machado of Linkedin, Metthew Weingarten of Disney, Joe Reis and Matt Housely, authors of The Fundamentals of Data Engineering, Zach Wilson of Eczachly Inc, Megan Lieu of Deepnote, Erik Heintare of Bolt, Lior Solomon of Vimeo, Krishna Naidu of Canva, Mike Cohen of Substack, Jens Larsson of Ark, Gunnar Tangring of Klarna, Yoav Shmaria of Similarweb and Xiaoxu Gao of Adyen.

Check out our three most downloaded episodes: