Start / Data Engineering Podcast / Reflecting on the past 6 years of data engineering

Reflecting On The Past 6 Years Of Data Engineering

32 min • 6 februari 2023

Summary

This podcast started almost exactly six years ago, and the technology landscape was much different than it is now. In that time there have been a number of generational shifts in how data engineering is done. In this episode I reflect on some of the major themes and take a brief look forward at some of the upcoming changes.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Your host is Tobias Macey and today I'm reflecting on the major trends in data engineering over the past 6 years

Interview

Introduction
6 years of running the Data Engineering Podcast
Around the first time that data engineering was discussed as a role
- Followed on from hype about "data science"
Hadoop era
Streaming
Lambda and Kappa architectures
- Not really referenced anymore
"Big Data" era of capture everything has shifted to focusing on data that presents value
- Regulatory environment increases risk, better tools introduce more capability to understand what data is useful
Data catalogs
- Amundsen and Alation
Orchestration engine
- Oozie, etc. -> Airflow and Luigi -> Dagster, Prefect, Lyft, etc.
- Orchestration is now a part of most vertical tools
Cloud data warehouses
Data lakes
DataOps and MLOps
Data quality to data observability
Metadata for everything
- Data catalog -> data discovery -> active metadata
Business intelligence
- Read only reports to metric/semantic layers
- Embedded analytics and data APIs
Rise of ELT
- dbt
- Corresponding introduction of reverse ETL
What are the most interesting, unexpected, or challenging lessons that you have learned while working on running the podcast?
What do you have planned for the future of the podcast?

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Senaste avsnitt

Revolutionizing Python Notebooks with Marimo

28 juli | 52 min

Warehouse Native Incremental Data Processing With Dynamic Tables And Delayed View Semantics

21 juli | 55 min

Reflecting On The Past 6 Years Of Data Engineering

Summary

Announcements

Interview

Parting Question

Closing Announcements

Senaste avsnitt

Revolutionizing Python Notebooks with Marimo

Warehouse Native Incremental Data Processing With Dynamic Tables And Delayed View Semantics

Streamlining Data Pipelines with MCP Servers and Vector Engines

Foundational Data Engineering At Two Sigma

Enabling Agents In The Enterprise With A Platform Approach

Dagster's New Era: Modularizing Data Transformation in the Age of AI

AI and the Lakehouse: How Starburst is Pioneering New Workflows

Amazon S3: The Backbone of Modern Data Systems

Scaling Data Operations With Platform Engineering

From Data Discovery to AI: The Evolution of Semantic Layers

Balancing Off-the-Shelf and Custom Solutions in Data Engineering

StarRocks: Bridging Lakehouse and OLAP for High-Performance Analytics

Exploring NATS: A Multi-Paradigm Connectivity Layer for Distributed Applications

Advanced Lakehouse Management With The LakeKeeper Iceberg REST Catalog

Simplifying Data Pipelines with Durable Execution

Overcoming Redis Limitations: The Dragonfly DB Approach

Bringing AI Into The Inner Loop of Data Engineering With Ascend

Astronomer's Role in the Airflow Ecosystem: A Deep Dive with Pete DeJoy

Accelerated Computing in Modern Data Centers With Datapelago

The Future of Data Engineering: AI, LLMs, and Automation

Evolving Responsibilities in AI Data Management

CSVs Will Never Die And OneSchema Is Counting On It

Breaking Down Data Silos: AI and ML in Master Data Management

Building a Data Vision Board: A Guide to Strategic Planning

How Orchestration Impacts Data Platform Architecture

An Exploration Of The Impediments To Reusable Data Pipelines

The Art of Database Selection and Evolution

Bridging Code and UI in Data Orchestration with Kestra

Streaming Data Into The Lakehouse With Iceberg And Trino At Going

An Opinionated Look At End-to-end Code Only Analytical Workflows With Bruin

Feldera: Bridging Batch and Streaming with Incremental Computation

Accelerate Migration Of Your Data Warehouse with Datafold's AI Powered Migration Agent

Bring Vector Search And Storage To The Data Lake With Lance

The Role of Python in Shaping the Future of Data Platforms with DLT

Build Your Data Transformations Faster And Safer With SDF

Scaling Airbyte: Challenges and Milestones on the Road to 1.0

Enhancing Data Accessibility and Governance with Gravitino

The Evolution of DataOps: Insights from DataKitchen's CEO

Achieving Data Reliability: The Role of Data Contracts in Modern Data Management

How Generative AI Is Impacting Data Engineering Teams

The Role of Product Managers in Data-Centric Organizations

Neon: A Serverless And Developer Friendly Postgres

Improve Data Quality Through Engineering Rigor And Business Engagement With Synq

Stitching Together Enterprise Analytics With Microsoft Fabric

Being Data Driven At Stripe With Trino And Iceberg

X-Ray Vision For Your Flink Stream Processing With Datorios

Practical First Steps In Data Governance For Long Term Success

Data Migration Strategies For Large Scale Systems

Zenlytic Is Building You A Better Coworker With AI Agents

Release Management For Data Platform Services And Logic

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Build Your Second Brain One Piece At A Time

Making Email Better With AI At Shortwave

Designing A Non-Relational Database Engine

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Reconciling The Data In Your Databases With Datafold

Version Your Data Lakehouse Like Your Software With Nessie

When And How To Conduct An AI Program

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Sharing Across Business And Platform Boundaries

Tackling Real Time Streaming Data With SQL Using RisingWave

Build A Data Lake For Your Security Logs With Scanner

Modern Customer Data Platform Principles

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Designing Data Platforms For Fintech Companies

Troubleshooting Kafka In Production

Adding An Easy Mode For The Modern Data Stack With 5X

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Designing Data Transfer Systems That Scale

Addressing The Challenges Of Component Integration In Data Platform Architectures