Decoding the Open Lakehouse - Sahil Walia

In this talk, Sahil Walia, Senior Technical Architect at Snowflake, shares his extensive data engineering expertise from writing simple SQL queries early in his career to architecting enterprise-scale Apache Iceberg lakehouses today. We explore the evolution of the modern data platform, the mechanics of open lakehouses, and how decoupling storage from compute is transforming data architecture.

Links:

- https://medium.com/@sahil-walia/

- Iceberg Spec- https://iceberg.apache.org/spec/

- https://ossie.apache.org

- https://github.com/Snowflake-Labs/schemachange

- https://opensource.guide/how-to-contribute/

You’ll learn about:

- What defines a modern data platform and the importance of interoperability.

- Key differences between traditional storage, Hive metastores, and Apache Iceberg.

- How Iceberg metadata simplifies GDPR compliance, data deletions, and file compactions.

- When a lakehouse architecture actually makes sense (and when to avoid it).

- Standardizing business logic across compute engines using Apache OSI.

- Actionable tips for making impactful open-source data engineering contributions.

TIMECODES:

00:00 Defining the Modern Data Platform

11:34 What Exactly is a Lakehouse?

17:20 Comparing Hive Metastore and Apache Iceberg

22:34 Handling GDPR and Data Deletions in Iceberg

27:33 Apache Iceberg vs. Delta Lake

34:10 Do You Need a Lakehouse for Small Data?

40:11 Understanding Apache OSI (Open Semantic Interchange)

49:00 Managing Schema Changes in Data Warehouses

54:20 How to Contribute to Open Source Projects

01:00:40 The Role of AI in Open Source Contributions

This talk is designed for data engineers, data architects, and analytics professionals looking to modernize their data infrastructure. It provides essential architectural insights for technical teams evaluating Apache Iceberg, dealing with large-scale data transformation, or aiming to scale their open-source contributions.

Connect with Sahil:

- Linkedin - https://www.linkedin.com/in/sahilwaliasyracuse

- Github - https://github.com/sahil-walia

- Website - https://medium.com/@sahil-walia

Connect with DataTalks.Club:

- Join the community - https://datatalks.club/slack.html

- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

- Check other upcoming events - https://lu.ma/dtc-events

- GitHub: https://github.com/DataTalksClub

- LinkedIn - https://www.linkedin.com/company/datatalks-club/

- Twitter - https://twitter.com/DataTalksClub

- Website - https://datatalks.club/