41: Efficiency at Any Scale, with Marlon Dutra

Wolf and Jim welcome their first-ever guest, longtime friend Marlon Dutra of Meta, for a practical tour of software efficiency. Marlon argues that efficiency isn't just a concern for huge systems. Your code may run on someone's aging phone or on a car's hardware for decades, and it may be running up a cloud bill. Efficiency also isn't the same as performance: if something gets 20% slower but 10x cheaper, and nobody notices, that's a win.

What we cover

• Measure first. You can't optimize what you don't measure, and even a text file beats no data.
• CPU. Understand algorithmic complexity, keep hot paths small, and use profilers to find where time really goes. Learn the difference between CPU time and wall time, and when threads, multiprocessing, or GPUs fit the problem. We also look at how accidental O(n²) code happens, including a documentation build that dropped from 20 minutes to 24 seconds after profiling.
• Memory. Memory is expensive, scarce, and slower than most people think. Marlon covers leaks, reference cycles, pre-allocation, stack vs. heap, cache lines, spatial and temporal locality, and memoization.
• I/O and storage. Favor sequential access and larger reads and writes, and use async I/O instead of a thread per operation. Avoid N+1 queries, and store numbers as numbers rather than strings. We also get into file formats like Parquet, compression trade-offs, data retention, SSD wear, and logging discipline.
• Networking. Latency often matters more than bandwidth. Reduce round trips with batching and pipelining, watch payload sizes, use CDNs, expect unreliable networks, and add exponential backoff so retries don't take down your own servers.

Links

• Marlon's code examples: https://github.com/Runtime-Arguments/efficiency

Hosts:
Jim McQuillan can be reached at [email protected]
Wolf can be reached at [email protected]

Follow us on Mastodon: @[email protected]

If you have feedback for us, please send it to [email protected]

Check out our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky