The 90/9/1 rule of Databricks performance work - how to triage Spark optimization in 60 seconds

Your team is three weeks into a Databricks performance push. Broadcast hints in PRs. AQE flags toggled like christmas lights. Partition counts re-tuned for the third time. The manager is asking, gently, when the gains are showing up in the bill.

The staff DE on the next team finished theirs in two afternoons. Same workloads, bigger drop. They were running a triage you have never been taught.

In this episode:

- Why most of what your team calls Spark optimization is cosmetic and will never move the bill, no matter how clean the PR

- The two named tests senior Databricks engineers run on every workload before they touch a config

- Why the same change (caching, salted joins, skew handling) can be cosmetic on one workload and structural on the one next to it

- Where the real leverage in a Spark workload actually lives, and why it is almost always visible from outside the code

For Databricks data engineers stuck in a performance push that is not converting effort into runtime or bill drops. Whether you are mid-level drowning in config tweaks, or senior watching the bill refuse to move, you will walk away with a one-minute triage you can run on any Spark workload tomorrow morning.

---

Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday.

LinkedIn: linkedin.com/in/jakublasak

Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake