Podd: Oxide and Friends

AI Discourse with Steve Klabnik

5 juni 2025 | 94 min

AI, Materials, and Fraud with Ben Shindel

24 maj 2025 | 71 min

RIP USENIX ATC

15 maj 2025 | 85 min

Solutions Software Engineering with Matthew Sanabria

7 maj 2025 | 91 min

Shootout at the CNCF Corral

1 maj 2025 | 83 min

Bringing up Cosmo

24 april 2025 | 116 min

Oxide is bringing up its next generation server. To discuss the (amazingly smooth) bringup process, Bryan and Adam were joined by members of the oxide team. Tales of adversity, re-work, un-re-work, and triumph!

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Nathanael Huffman, Ian Sobering, Matt Keeter, and Aaron Hartwig.

We mentioned quite a few terms! Here's a helpful guide:

Cosmo - Oxide’s next-generation sled (currently in development) with an AMD Turin CPU
Gimlet - Oxide’s current-generation sled with an AMD Milan CPU
Turin - AMD Epyc 9005 Series
Milan - AMD Epyc 7003 Series
Genoa - AMD Epyc 9004 Series (Oxide chose to skip this generation)
Sequencing - the precise control of when power rails are energized throughout a PCB
Sled - One of the (max 32) computers in an Oxide rack; a custom form-factor optimized for power and cooling efficiency
IBC - Intermediate Bus Converter (Our 54VDC -> 12VDC converter)
RoT - Root of Trust
SP - Service Processor, the small computer (running Hubris) that allows for low-level control
Ignition - An even lower-level control network for power management (including power of the SP)
Ruby - The AMD reference platform (Oxide has used this to prepare Cosmo software in advance of bringup)
DC-SCM - https://www.opencompute.org/documents/ocp-dc-scm-spec-rev-1-0-pdf and OpenCompute standard form factor.
Grapefruit - OCP DC-SCM form-factor board with our SP, RoT, and FPGA on it, used to replace the OCP DC-SCM baseboard management controller in the Ruby platform.
Cadence - Software Oxide previously used for PCB design
Altium - Software Oxide now uses for PCB design
Hubris - Oxide’s embedded operating system, run on the SP and RoT
Humility - The Hubris debugger
PLM - Product Lifecycle Management – a class of software used for managing hardware BOMs
BOM - Bill of Materials – the components required to build a hardware product
RFK - Our colleague, Robert Keith (to distinguish him from our other colleague, Robert, and our former colleague, Keith)
FPGA - Field Programmable Gate Array – Also referred to as “soft logic” – effectively programmable hardware
ILA - Integrated Logic Analyzer
JTAG - A debugging interface for various processors
UART - A serial port or connection

For previous tales from the bringup lab:

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Character Limit with Kate Conger and Ryan Mac

17 april 2025 | 87 min

Hell is other networks

4 april 2025 | 87 min

Raiding the Minibar

1 april 2025 | 112 min

Lip-Bu Tan's Intel

21 mars 2025 | 83 min

A Happy Day For Rust

15 mars 2025 | 80 min

A Crate is Born

6 mars 2025 | 103 min

Transparency in Hardware/Software Interfaces

28 februari 2025 | 109 min

A Half-Century of Silicon Valley with Randy Shoup

24 februari 2025 | 117 min

Textual UIs with Orhun Parmaksız

13 februari 2025 | 93 min

AI Disruption: DeepSeek and Cerebras

6 februari 2025 | 91 min

Holistic Engineering with Robert Mustacchi

23 januari 2025 | 111 min

Crates We Love

16 januari 2025 | 93 min

Predictions 2025

10 januari 2025 | 122 min

OxF 2024 Wrap-Up

30 december 2024 | 94 min

Scaling Bluesky with Paul Frazee

19 december 2024 | 101 min

Conferences in Tech

14 december 2024 | 90 min

Bryan and Adam were joined by Theo Schlossnagle, KellyAnn Fitzpatrick, and Steve O'Grady to talk about conferences in tech. A lot has changed in the past couple of decades about the impetus for conferences and what makes it worthwhile to attend.

In addition to Bryan Cantrill and Adam Leventhal, speakers included Theo Schlossnagle, KellyAnn Fitzpatrick, and Steve O'Grady.

The lightly edited live chat from the show:

ellie.idb: 2005, huh? y’all met when i was 2
goodjanet: yea i was younger than 10 lol
jgrillo_: I was just thinking I feel very young because I was a junior in high school but not anymore lol
aka_pugs: my first conference - 1975
ellie.idb: oxide appeals to the youth
jbk1234: my first one was LISA in 05 or 06... mostly because it took a near act of god because my director didn't believe in sending his people to conferences
jgrillo_: "before software ate the world" is what I usually call "when the internet was still fun"
ellie.idb: my earliest memory was, uhhh, Google I/O 2008 when they gave every attendee that android phone
ellie.idb: i don’t recall which one it was, but i do remember playing with it when i was 5 hahahaha
taitomagatsu: I've only been to one tech conference in person, and it was a very tame SIGGRAPH that happened in Santiago, CL (I live in Chile). It was a lot about animation. I wanted it to have talks on image processing like the ones over on the US x3 but oh well, beggars can't be choosers
goodjanet: I've never been to a tech conference
devdsp2175: The Germans know how to run a conference. The chaos communications congress is wild.
ellie.idb: same!! never actually attended one as an adult hahaha
taitomagatsu: Have you attended one remotely?
goodjanet: nope, closest is just watching recorded talks after the fact
taitomagatsu: I attended the rustconf of 2 years ago remotely. It was amazing and I was soooo tired by the end of it. Brain got depleted of juice for the day
network2501: looking forward to in person dtrace conference with a dedicated zball room
ahl0003: more of a trade show, but I went to the MacWorld conference in the late '90s
ahl0003: I still have some BeOS install CDs from then
goodjanet: im so thankful for recorded talks
ahl0003: this is kind of wild: I went with my brother who was 12 or so and we met a guy at Be... my brother would go on to work with him 30 years later!
ellie.idb: https://en.m.wikipedia.org/wiki/Motorola_Droid the OG droid with the flip up keyboard and everything
tocococa: ISCA this year was just around the corner from Santiago in Buenos Aires and it was pretty cool, and CARLA took place this year in Santiago too
blacksmithforlife: Since I can never get a conference approved from work, I live off recorded conference videos on YouTube
network2501: best mom
devdsp2175: The shade! Sending hugs to Bryan's inner child.
taitomagatsu: daaaaaamn, I didn't know about either! I might keep an eye on ISCA, maybe I can go next year ❤️
devdsp2175: You can't record the hallway track...
jh179: Bryan's talk for Papers We Love on the History of Containers is how I found out about him, Oxide and all the rest. Had an incredible tangent about jails...
zeanic: Conference idea: all hallway tracks
devdsp2175: YouTube keeps recommending Bryan's talks on running containers on the metal at Joyant.
devdsp2175: And I keep watching them!
ellie.idb: wow, ISCA had some really fucking cool talks this year
ellie.idb: damn. i’m adding this to my watch list too!!! i’ll try and see if i can get funding for next year hahaha
tocococa: yeah, 100%, but my brain was melted after every day
nahumshalman: Bryan has the luxury of working on OSS. I think the point that Theo was making is that Surge (I only attended the very last one) was a space where you could be open about proprietary stuff. Talking about failure in a safe space, etc.
nahumshalman: Ah, Theo is now making that point.
taitomagatsu: Does ISCA have any sort of official YT channel?
taitomagatsu: Because I might... have a handful of talks to watch
goodjanet: 18 years ago isnt that long ago?
network2501: 18 years ago is almost 3 generations of lives/eras ago
ellie.idb: what HPC conferences are going on? i need to hear about the deets going on with CXL
jgrillo_: although 18yr is ~half my life it doesn't feel very long ago..
tocococa: I am not sure, I know that all keynotes were recorded, but I don´t know where they might be
ellie.idb: 21 years ago i was not alive 😅
network2501: What if the second time you do the talk it's even better than the last? Like book revisions?
ahl0003: https://www.youtube.com/watch?v=5i1OK4y9x0w
taitomagatsu: I've found a channel that has older ISCA videos https://www.youtube.com/@acmsigarch2299, imma keep looking for one that might have the 2024 one
blacksmithforlife: Working in government, watching "old" conference videos is great because they're "cutting edge" for where my organization is at currently. Case in point, we are just now going to the cloud and doing micro services
taitomagatsu: https://xkcd.com/979/
ahl0003: https://craft-conf.com/2025
srockets: That’s why I liked !!con so much. No one tries to sell you anything.
jgrillo_: I've never owned a car newer than 20yo, that's kind what it's like when you look at the car ads from its era
devdsp2175: Are you also doing an "Agile Transformation" which is neither transformative nor optimising for agility?
ahl0003: https://monktoberfest.com/
srockets: (Also, Ghent had better bike racing than Budapest)
srockets: But worse weather
bcantrill: https://youtu.be/stMEuZJJDck?list=PLvsKqlNNP3R8JKE97pwewsDmZdcO5MEWV
drkellyannfitz: Here are the talks from this year: https://redmonk.com/?series=monktoberfest-2024
blacksmithforlife: What does "hallway track" mean?
zeanic: Cr...

Intel after Gelsinger

5 december 2024 | 111 min

Technical Blogging

21 november 2024 | 100 min

Books in the Box IV

1 november 2024 | 95 min

Unshrouding Turin (or Benvenuto a Torino)

16 oktober 2024 | 114 min

Querying Metrics with OxQL

2 oktober 2024 | 95 min

RTO or GTFO

26 september 2024 | 100 min

Reflecting on Founder Mode

20 september 2024 | 82 min

RFDs: The Backbone of Oxide

30 augusti 2024 | 102 min

Whither CockroachDB?

21 augusti 2024 | 94 min

The Saga of Sagas

14 augusti 2024 | 118 min

Pragmatic LLM usage with Nicholas Carlini

9 augusti 2024 | 93 min

CrowdStrike BSOD Fiasco with Katie Moussouris

25 juli 2024 | 102 min

Heterogeneous Computing with Raja Koduri

18 juli 2024 | 114 min

Innovation Tokens with Charity Majors

11 juli 2024 | 86 min

Is NVIDIA like Sun from the Dot Com Bubble?

27 juni 2024 | 89 min

Musing with Changelog's Adam Stacoviak

17 juni 2024 | 107 min

Rebooting a datacenter: A decade later

30 maj 2024 | 101 min

Bookclub: How Life Works by Philip Ball

22 maj 2024 | 111 min

The long-awaited Oxide and Friends bookclub! Bryan and Adam were joined by special guest--and real life biologist--Greg Cost to discuss Philip Ball's terrific book, How Life Works: A User’s Guide to the New Biology. Spoiler: Alan Turing makes a very expected appearance!

In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Greg Cost.

Some of the topics we hit on, in the order that we hit them:

Bryan's reading notes

Central themes

Power and limitations of metaphor – especially mechanical ones
The fundamental, diametrical opposition between life and machines. (Nature does not use simulations!)
Rejecting the neo-Darwinian paradigm

Passages of note:

p. 91: “of the common SNPs seen in human populations, fully 62 percent are associated with height” … “the most common genomic associations for complex traits like this are in the noncoding regions” What is cognition? p. 137: “Life is, as biologist Michael Levin Jeremy Gunawardenaand philosopher Daniel Dennet have argued, ‘cognition all the way down’” AlphaFold2 p. 148 “AlphaFold does not so much solve the infamously difficult protein-folding problem as sidestep it. The algorithm makes no predictions about how a polypeptide chain folds, but simply predicts the end result based on the sequence.”
p. 156: allostery refers to how a
🤯 p. 160: “The popular view that science is the process of studying what the world is like needs to be given an important qualification: science tends to be the study of what we can study.”
p. 166: “The misfolding pathology of PrPs (prion proteins) is the price paid for the benefits of disorder. … Disordered proteins can increase the complexity and versatility of our regulatory networks, but at the cost of increased risk of toxic aggregates formed from misfolded proteins.”
p. 181: “The [training] analogy is far from perfect, not least because proteins don’t need to be ‘trained’ to acquire their roles.” Ball himself loves to use computing a metaphor, even when it is inapt or imperfect!
p. 189: “What you’re really looking at here is a diagram not of a molecular event but of a failed paradigm.”
p. 201: Clifford Brangwynne: “Many of the textbooks and even our language conveys this kind of factory-floor image of what goes on inside the cell. But the reality is that the computational logic underlying life is much more soft, wet and stochastic than anyone appreciates.” To which I would add: the information machine is MUCH more deterministic than anyone appreciates!
p. 205: “Because the binding of BMPs to BMP receptors can be altered by other molecules, the BMP pathway can interact with other pathways to create crosstalk between cells during development.” Mike Olson’s observation of everything working through side-effect. 🤯 p. 212: “It seems likely that metazoans have evolved this evolvability. One of the odd features of transcription factors that bind to DNA is that, in eukaryotes, the base sequences that they recognize are often surprisingly short – perhaps six or so base pairs long. … But there’s no reason the selectivity has to be this approximate; in prokaryotes the binding sites are longer and therefore more specific. It seems that eukaryotes have, so to speak, chosen this sloppiness – probably because it allows new regulatory pathways to develop.”
p 217: “While causal emergence seems to be a general design principle for life, it is rarely evident in our own technologies.” Disagree with: “...maybe the better computers of the future will be more causally emergent.” We can’t even get asynchronous systems working!
🤯 p. 222: “Is there, after all, really such an obvious advantage to being multicellular? If so, we don’t know what it is.” … “If [evolutionary biologist Michael] Lynch is right, the implication is humbling: we are here not because the multicellular lifestyle of metazoans like us is superior or even advantageous, but because chance mutations created possibilities for new regulatory and multicellular behaviors that natural selection merely found no reason to eliminate.”
p. 226: “If we want to understand the mechanisms behind some key evolutionary shifts – for example, the emergence of complex body shapes and lifestyles in the Cambrian explosion, the emergence of nervous systems and of new modes of cognition, and the divergence of mammals and other vertebrates – genomes are the wrong place to look.”
p. 245: “The switching of cell states often happens gradually rather than by abrupt switching at a sharply defined fork in the landscape.”
p. 248: “One of the most useful pieces of advice I heard from Nature’s biology editor many years ago was that the answer in biology is always ‘yes’”
p. 258: “Such leveraging of noise, the researchers suggested, might represent ‘a central and unifying principle underlying the properties of stem and progenitor cells that are central to the evolution of metazoan life.’ Noisiness helps to keep all the cell-fate options open.”
p. 263: “In short, says biologist Dennis Bray, the cells circuitry (if that is even a good metaphor at all) ‘is a long way from a silicon chip or any circuit a human would design.’ The more we learn about living systems, Bray writes, ‘the more we realize how idiosyncratic and discontinuous they are’ relative to computers.”
p. 267: “Planarians challenge our notions of what life can be.”
p. 276: “Lewis Wolpert is said to have once claimed, ‘It is not birth, marriage, or death, but gastrulation which is truly the most important time in your life.’”
p. 291: “The heart drives and shapes its own formation, bootstrapping itself into existence by virtue of its very function.”
p. 293: “If there’s a central feature of how life works, it is surely in this ability to create outcomes that are neither arbitrary nor wholly prescribed.”
p. 295: “The plasticity of form shown by living organisms might be not only a good way but the only way of making entities as complex as us”, but goes onto to liken us to universal computation
p. 296: “But what really is ‘normal’? … There are many types of benign skin growths; I’ve had a lipoma on my upper arm throughout my adult life.”
🤔 p. 297: “Conjoined twins like the Hensels are the result of an incomplete separation.” I think this is wrong? Certainly, it is odds with Mutants by by Armand Marie Leroi.
😡 p. 309: “Once a relatively obscure figure, Turing is now widely hailed as a visionary genius, thanks in part to the 2014 biopic The Imitation Game and the decision to feature him on the British fifty-pound note.” WTAF
p. 326: “The positioning of our organs on the correct side is controlled by stirring!” Mutants goes into this as well
p. 331: “Hsp90 acts as a kind of ‘capacitor for morphological evolution,’ storing up variation of form that might be released in times of stress”
p. 333: “It seems that the exploration of shape in the early Cambrian was excessively profligate: some of the body plans found in the fossil record of that time soon vanished. How could they have been selected for, only then to be so rapidly selected a...

All we have to fear is FUD itself

25 april 2024 | 81 min

A Baseball Startup with Paul Freedman and Bryan Carmel

17 april 2024 | 73 min

Discovering the XZ Backdoor with Andres Freund

10 april 2024 | 97 min

Cultural Idiosyncrasies

3 april 2024 | 87 min

Adversarial Machine Learning

27 mars 2024 | 84 min

Data Visualization

15 mars 2024 | 86 min

Crucible: The Oxide Storage Service

14 februari 2024 | 99 min

Innovation Stagnation?

7 februari 2024 | 61 min

Helios

1 februari 2024 | 108 min

What's taking so long?!

24 januari 2024 | 95 min

Open Source LLMs with Simon Willison

17 januari 2024 | 93 min

Predictions 2024!

10 januari 2024 | 117 min

AMD's MI300 and the Future of Accelerated Compute

12 december 2023 | 74 min

Framework Computer with Nirav Patel

5 december 2023 | 67 min

OpenAI's Boardroom Brawl

28 november 2023 | 70 min

Hiring Processes with Gergely Orosz

7 november 2023 | 105 min

Launching the Cloud Computer

31 oktober 2023 | 94 min

Open Source and Capitalism with Ashley Williams and Adam Jacob

24 oktober 2023 | 100 min

Settling Beef

17 oktober 2023 | 99 min

Mr. Nagle's Wild Ride

3 oktober 2023 | 91 min

DTrace at 20

12 september 2023 | 124 min

Open Source Anti-Patterns with Kelsey Hightower

29 augusti 2023 | 99 min

Kelsey Hightower joined Bryan and Adam to revisit a topic Bryan had spoken about a decade ago: corporate open source anti-patterns. Kelsey brought his typical sagacity to a complex and fraught topic.

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from August 28th, 2023.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Kelsey Hightower.

Here is the (lightly edited) live chat from the show:

xxxxbubbler: https://www.youtube.com/watch?v=Pm8P4oCIY3g here is Bryan's talk from 1 decade ago, for reference
rolipo.li: web3 is going great
rolipo.li: https://web3isgoinggreat.com/
ahl0003: Last time Kelsey joined us for predictions
blainehansen: "Governance orgies" happen when the governance mechanisms aren't well-designed ha. If they are well-designed then governance is good!
jbk: opsware maybe? or tivoli?
uptill3: hp openview was one as well
sevanj: "they've got us working for trinkets"
sevanj: this was mentioned on the bugzilla anouncement regarding funded staff being pulled from working on project in the last 3 years.
blainehansen: All open source problems are secretly public goods problems haha
carpetbomberz.com: Hashicorp DID do a "thing"
blacksmithforlife: Just like taxes fund roads, we should have a internet usage tax that then funds these open source projects that everyone finds value in. The person taxed should get to decide which open source project gets the money
kaliszad: The problem is, you can help other people, but first you have to sustain yourself. 🙂
aarondgoldman: Too boring to be evil
rolipo.li: too busy to be evil?
aarondgoldman: Angular never got budget even when Inbox used it and had millions of users
blainehansen: Most open source projects are probably not best led/governed by a for-profit company ha
aarondgoldman: HP had a huge repair service business when their hardware got much more reliable it almost killed the company
geekgonecrazy: Never actually considered using CNCF membership as a qualification for using a tool
ahl0003: it's the nintendo seal of quality!
geekgonecrazy: It’s an interesting thought now that I’ve heard it 🙈 especially for any sort of core utility like this
saone: On the topic of patterns that seem to be working, Docker Desktop's license requiring subscriptions for larger organizations for use of their product and focusing on providing a really good developer experience seems to be a really good spot for them to be
goodjanet: The term freeloading comes up only when there's a "problem" (usually fiscal in a company/group), the rest of the time the exact same actions are fine or often encouraged
mrdanack: I disagree, there are freeloaders. Multi-billion companies like IBM and Oracle have benefited from the PHP project for multiple decades and really haven't contributed even a modest amount back.
geekgonecrazy: Anytime hitting CLA I always use that as clue to take hesitation and think about contributing. 🙈
quasarken: I love that bit about community Adam
blacksmithforlife: https://www.linux.com/news/us-government-opens-access-federal-source-code-codegov/
blainehansen: Sometimes a community of passionate contributors is more a burden than a gift. Every project is different, not every project can be supported by many well-paid engineers at vc-funded incentive-aligned companies. I don't think the BUSL is smart or good, but there's a funding/support problem here that legitimately needs to be solved, and the existing open source social contract hasn't solved it. https://www.jeffgeerling.com/blog/2022/burden-open-source-maintainer
blacksmithforlife: Disclaimer: I'm a federal employee who tried to get more software open source while I was working at various agencies. For the most part it was soundly ignored and the agencies just claimed it was too hard and they didn't have enough funding to do it, which in my opinion is just false
blacksmithforlife: But, if you want it, just do a FOIA, then they have to give it to you
saone: There's a great deal of fear at my company that software being open sourced must be carefully vetted to avoid potential embarassment so the hurdles to open source anything are very high
girgias: The French government has released code which was pure garbage, and I don't think one can do worse than the APB code
geekgonecrazy: That sucks. 😬I can totally see individual developers being afraid. I’ve faced that with my team. Weird to think org would be especially if trusting engineers
northrup: Adam to your point though - I don't see how that's any different than other open source projects that aren't corporate backed. No open source projected is obligated to honor your issue to drive a project in a direction, or accept your PR to add a feature or function...
ahl0003: Great point!
blainehansen: The open source cooperative idea is the best I can come up with to solve the problem
blacksmithforlife: What is dev rail?
bcantrill: Developer relations
ahl0003: developer relations
jbk: dev rel(ations)?
bcantrill: JYNX
blacksmithforlife: Never heard that term before
geekgonecrazy: Curious at what scale you think devrel is needed vs the engineers in company directly involved
geekgonecrazy: I’ve often wondered if doesn’t create unnecessary barrier between engineers and community. Especially at certain size
quasarken: Dev Rel seems a lot like community solutions engineering
geekgonecrazy: I’ve personally seen some companies use devrel as sole tie to open source and “community” in place of more of company getting involved
rolipo.li: devrel as a service. now it's a consulting firm?
northrup: When I worked at GitLab in the early days, some of my most favorite experiences were going to conferences and hanging out in the GitLab booth to answer questions and talk with / help users. SOO much great feedback, clear "oh wow!" edge cases brought forward, and amazing feedback of "yeah, you made this feature, but that wasn't what we needed"
ahl0003: I remember liking this book on devrel:

Fork in the road for Terraform?

22 augusti 2023 | 80 min

No Silver Bullets

15 augusti 2023 | 78 min

Books in the Box III

25 juli 2023 | 90 min

The Frontend of the Computer

18 juli 2023 | 64 min

Tales from Manufacturing: Shipping Rack 1

11 juli 2023 | 84 min

Shipping the first Oxide rack: Your questions answered!

4 juli 2023 | 123 min

On this week's show, Adam Leventhal posed questions from Hacker News (mostly) to Oxide founders Bryan Cantrill and Steve Tuck. Stick around until the end to hear about the hardest parts of building Oxide--great, surprising answers from both Bryan and Steve.

They were also joined by Steve Klabnik.

Questions for Steve and Bryan:

[@6:38] Q:

Congrats to the team, but after hearing about Oxide for literal years since the beginning of the company and repeatedly reading different iterations of their landing page, I still don't know what their product actually is. It's a hypervisor host? Maybe? So I can host VMs on it? And a network switch? So I can....switch stuff? (*)

A:

Steve: A rack-scale computer; "A product that allows the rest of the market that runs on-premises IT access to cloud computing."Bryan: agrees

[@8:46] Q:

It's like an on prem AWS for devs. I don't understand the use case but the hardware is cool. (*)I didn’t understand the business opportunity of Oxide at all. Didn’t make sense to me.However if they’re aiming at the companies parachuting out of the cloud back to data centers and on prem then it makes a lot of sense.It’s possible that the price comparison is not with comparable computing devices, but simply with the 9 cents per gigabyte egress fee from major clouds. (*)

A:

Bryan: "Elastic infrastructure is great and shouldn't be cloistered to the public cloud"; Good reasons to run on-prem: compliance, security, risk management, latency, economics; "Once you get to a certain size, it really makes sense to own"Steve: As more things move onto the internet, need for on-prem is going to grow; you should have the freedom to own

[@13:31] Q:

Somebody help me understand the business value. All the tech is cool but I don't get the business model, it seems deeply impractical.

You buy your own servers instead of renting, which is what most people are doing now. They argue there's a case for this, but it seems like a shrinking market. Everything has gone cloud.
Even if there are lots of people who want to leave the cloud, all their data is there. That's how they get you -- it costs nothing to bring data in and a lot to transfer it out. So high cost to switch.
AWS and others provide tons of other services in their clouds, which if you depend on you'll have to build out on top of Oxide. So even higher cost to switch.
Even though you bought your own servers, you still have to run everything inside VMs, which introduce the sort of issues you would hope to avoid by buying your own servers! Why is this? Because they're building everything on Illumos (Solaris) which is for all practical purposes is dead outside Oxide and delivering questionable value here.
Based on blogs/twitter/mastodon they have put a lot of effort into perfecting these weird EE side quests, but they're not making real new hardware (no new CPU, no new fabric, etc). I am skeptical any customers will notice or care and would have not noticed had they used off the shelf hardware/power setups.

So you have to be this ultra-bizarre customer, somebody who wants their own servers, but doesn't mind VMs, doesn't need to migrate out of the cloud but wants this instead of whatever hardware they manage themselves now, who will buy a rack at a time, who doesn't need any custom hardware, and is willing to put up with whatever off-the-beaten path difficulties are going to occur because of the custom stuff they've done that's AFAICT is very low value for the customer. Who is this? Even the poster child for needing on prem, the CIA is on AWS now.I don't get it, it just seems like a bunch of geeks playing with VC money?(*)

A:

Bryan: "EE side quests" rant; you can't build robust, elastic infrastructure on commodity hardware at scale; "The minimum viable product is really, really big"; Example: monitoring fan power draw, tweaking reference desgins doesn't cut it Example: eliminating redundant AC power suppliesSteve: "Feels like I’m dealing with my divorced parents" post

[@32:24] Q (Chat):

It would be nice to see what this thing is like before having to write a big checkSteve: We are striving to have lab infrastructure available for test drives

[@32:56] Q (Chat):

I want to know about shipping insurance, logistics, who does the install, ...Bryan: "Next week we'll be joined by the operations team" we want to have an indepth conversation about those topics

[@34:40] Q:

Seems like Oxide is aiming to be the Apple of the enterprise hardware (which isn't too surprising given the background of the people involved - Sun used to be something like that as were other fully-integrated providers, though granted that Sun didn't write Unix from scratch). Almost like coming to a full circle from the days where the hardware and the software was all done in an integrated fashion before Linux turned-up and started to run on your toaster. (*)

A:

Bryan: We find things to emulate in both Apple and Sun, e.g., integrated hard- and software; AS/400Steve: "It's not hardware and software together for integration sake", it's required to deliver what the customer wants; "You can't control that experience when you only do half the equation"

[@42:38] Q:

I truly and honestly hope you succeed. I know for certain that the market for on-prem will remain large for certain sectors for the forseeable future. However. The kind of customer who spends this type of money can be conservative. They already have to go with on an unknown vendor, and rely on unknown hardware. Then they end up with a hypervisor virtually no one else in the same market segment uses.Would you say that KVM or ESXi would be an easier or harder sell here?Innovation budget can be a useful concept. And I'm afraid it's being stretched a lot. (*)

A:

Bryan: We can deliver more value with our own hypervisor; we've had a lot of experience in that domain from Joyent. There are a lot of reasons that VMware et al. are not popular with their own customers; Intel vs. AMDSteve: "We think it's super important that we're very transparent with what we're building"

[@56:05] Q:

what is the interface I get when I turn this $$$ computer on? What is the zero to first value when I buy this hardware? (*)

A:

Steve: "You roll the rack in, you have to give it power, and you have give it networking [...] and you are then off on starting the software experience"; Large pool of infrastructure reosources for customers/devs/SREs/... in a day or less; Similar experience to public cloud providers

[@01:02:06] Q:

One of my concerns when buying a complete so...

Okay, Doomer: A Rebuttal to AI Doom-mongering

27 juni 2023 | 71 min

Software Verificationpalooza

20 juni 2023 | 82 min

Virtualizing Time

13 juni 2023 | 66 min

Jordan Hendricks joined Bryan and Adam to talk about her work virtualizing time--particularly challenging when migrating virtual machines from one physical machine to another!

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from June 12th, 2023.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague Jordan Hendricks.

The (lightly edited) live chat from the show:

DanCrossNYC: The TSC ticks at a fixed rate now days, regardless of voltage scaling on the CPU.
jbk: just x86 doesn't provide a consistent want to determine what the rate is
jbk: (I guess some chips will tell you via CPUID, but I've yet to actually encounter such chips)
jbk: some hypervisors will tell you via an MSR
zorg24: Looks the Linux kernel docs have some documentation on the x86 TSC and PIT https://www.kernel.org/doc/html/next/virt/kvm/x86/timekeeping.html
DanCrossNYC: CPUID or an MSR, but yeah, most systems sample over a fixed interval (determined by another time source) to figure it out.
jbk: no, versus some other present component that allows you to measure the frequency
DanCrossNYC: No, the PIT or HPET or something.
jbk: https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/tscc_pit.c?r=236cb9a8
jbk: is how it uses the PIT
jbk: (the HPET code needs to improve it's accuracy, so it's only used when the PIT isn't there at the moment)
jbk: some Intel NUCs have no PIT
jbk: so HPET is the only option
bcantrill: https://github.com/illumos/illumos-gate/commit/717646f7112314de3f464bc0b75f034f009c861e
DanCrossNYC: Two big ones: system maintenance without disturbing guest workloads, and also load balancing across a rack.
"Sevan: ah, thanks.
https://github.com/illumos/illumos-gate/blob/717646f7112314de3f464bc0b75f034f009c861e/usr/src/test/bhyve-tests/tests/common/common.c#L166"
bcantrill: https://github.com/oxidecomputer/tsc-simulator/tree/master
DanCrossNYC: The guest may well be running NTP itself.
iangrunert: I assume you could also check that NTP is alive / has synced recently before doing a migration right?
aka_pugs: Do people use IEEE 1588/PTP in datacenters? Maybe finance wackos?
zorg24: also it might be tricky to check if NTP synced recently if it is happening in usermode
iangrunert: Might've missed this - is it just the hypervisor that has to run NTP recently or the VM as well?
saone: I believe it was just the hypervisor
DanCrossNYC: The host.
DanCrossNYC: A guest may or may not; that's up to the guest.
jbk: but IIUC, if the guest IS running NTP, then the host definitely needs it to avoid any time warps
DanCrossNYC: Yup.
DanCrossNYC: Fortunately, there's a bit of an out for the blackout window during migration: SMM mode can effectively pause a machine for an indefinite period of time.
DanCrossNYC: We don't USE SMM anywhere, but robust systems software kinda needs to handle the case where the machine goes out to lunch for a minute.
zorg24: 🙌 hooray for hardware with no SMM use
DanCrossNYC: We have done everything we can to turn it off.
ahl: https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md
ahl: https://github.com/oxidecomputer/propolis
earltea: it worked so well I almost thought the VM didn't migrate 😅
saone: It's easy to forget that there's a world outside the cloud, but edge deployments that have physical peripherals hooked up need to maintain those connections to peripherals; migrating those peripherals to cloud environments and managing that integration has been a big challenge for my group.
iangrunert: https://signalsandthreads.com/clock-synchronization/ Good listen about clock synchronization and PTP in the ""finance weirdos"" world. MiFID 2 time sync requirements require timestamping key trading event records to within 100 microseconds of UTC.
jhendricks: a bit belated, but the propolis side of these changes: https://github.com/oxidecomputer/propolis/commit/7ed480843d3b5cfd9fd07dce41772f8eac4e9171
saethlin: The calvalry??
saethlin: Are we just going to let that slide
saethlin: Is this a pronunciation situation again
zorg24: not the first time I've heard it pronounced that way 🤷
saethlin: Well maybe it's me learning this time
DanCrossNYC: Calvary
DanCrossNYC: That's the religious thing.
ahl: https://github.com/illumos/illumos-gate/blob/0c5967db436935325af441af2b27d337f4e64af5/usr/src/uts/common/os/cyclic.c#L44
zooooooooo: thought this was rust typescript at first 😳
DanCrossNYC: Dunno... I missed it. 🙂
ahl:

* Starting in about 1994, chip architectures began specifying high resolution * timestamp registers. As of this writing (1999), all major chip families * (UltraSPARC, PentiumPro, MIPS, PowerPC, Alpha) have high resolution * timestamp registers, and two (UltraSPARC and MIPS) have added the capacity * to interrupt based on timestamp values. These timestamp-compare registers * present a time-based interrupt source which can be reprogrammed arbitrarily * often without introducing error. Given the low cost of implementing such a * timestamp-compare register (and the tangible benefit of eliminating * discrete timer parts), it is reasonable to expect that future chip * architectures will adopt this feature.

aka_pugs: Bryan's TSC is overflowing.
DanCrossNYC: That's Tom.
DanCrossNYC: Riding in with the cavalry.
aka_pugs: Good session.
ahl: Thanks...

Open Source Governance

30 maj 2023 | 86 min

Building Together: Oxide and Samtec

16 maj 2023 | 80 min

The Network Behind the Network

9 maj 2023 | 99 min

Bryan and Adam are joined by Oxide colleagues Arjen, Matt, John, and Nathaneal to talk about the management network--the brainstem of the Oxide Rack. Just as it ties together so many components, this episode ties together many many (many!) topics we've discussed in other episodes.

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from May 8th 2023.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Arjen Roodselaar, Matt Keeter, John Gallagher, and Nathanael Huffman.

This built on work described in many previous episodes:

Cabling the Backplane Prior to going all-in on a cabled backplane with blind-mated server sleds (i.e. no plugging, unplugging, mis-plugging network cables). We (Bryan) espoused an "NC-SI or bust" mantra... at least in part to avoid doubling the cable count. With the cabled backplane, the reasons for NC-SI disappeared (which let the many reasons against truly shine).
The Pragmatism of Hubris in which we talk about our embedded operating system, Hubris (and it's companion debugger, Humility). Hubris runs on the service processors that are the main endpoints on the management network. Matt's work controlling the management network switch (the VSC7448) is in the context of Hubris, as is John's work communicating with the sleds over the management network.
The Power of Proto Boards showed and told about the many small boards we've used in development. Several of those were purpose built for controlling and simulating parts of the management network.
The Oxide Supply Chain Kate Hicks joined us to talk about the challenges of navigating the supply chain. Mentioned here in the context of "supply-chain-driven design": we designed around the parts we could procure! Tip: stay away from "automotive-quality" parts when the auto industry is soaking them all up.
Holistic Boot in which we talked about how (uniquely!) Oxide boots from nothing to its operating system and services. Over the management network, we can drive server recovery by piping in a RAMdisk over the network and then (slowly) through the UART to the CPU.
Get You a State Machine for Great Good Andrew joined us to talk about his work on a state-machine driven text-UI and its companion replay debugger. We mentioned this in the context of John replaying the long upload process in seconds rather than hours to fix a UI bug.

Major components of the management network

Matt's VSC7448 dev kit

Matt's remote tuning setup via webcam

Management network debugging
Management network debugging

Blue Skies Over Mastodon (with Erin Kissane and Tim Bray)

2 maj 2023 | 101 min

Rust Trademark: Argle-bargle or Foofaraw?

18 april 2023 | 82 min

Cabling the Backplane

4 april 2023 | 61 min

Get You a State Machine for Great Good

28 mars 2023 | 68 min

Andrew Stone of Oxide Engineering joined Bryan, Adam, and the Oxide Friends to talk about his purpose-built, replay debugger for the Oxide setup textual UI. Andrew borrowed a technique from his extensive work with distributed systems to built a UI that was well-structured... and highly amenable to debuggability. He built a custom debugger "in a weekend"!

Some of the topics we hit on, in the order that we hit them:

The (lightly) edited live chat from the show:

MattCampbell: I'm gathering that this is more like the fancy pseudo-GUI style of TUI, which is possibly bad for accessibility
ahl: we are also building with accessibility in mind, stripping away some of the non-textual elements optionally
MattCampbell: oh, cool
ahl: Episode about the "Sidecar" switch: https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.md
MattCampbell: ooh! That kind of recording is definitely better for accessibility than a video.
uwaces: Were you inspired by Elm? (The programming language for web browsers?)
bcantrill: Here's Andrew's PR for this, FWIW: oxidecomputer/omicron#2682
uwaces: Elm has a very similar model. They have even had a debugger that let you run events in reverse: https://elm-lang.org/news/time-travel-made-easy
bch: I’m joining late - 1) does this state-machine replay model have a name 2) expand on (describe ) the I/o logic separation distinction?
ahl: http://dtrace.org/blogs/ahl/2015/06/22/first-rust-program-pain/
zk: RE: logic separation in consensus protocols: the benefit of seperating out the state machine into a side-effect free function allows you to write a formally verified implementation in a pure FP lang or theorem prover, and then extract a reference program from the proof.
we're going to the zoo: lol i’m a web dev && we do UI tests via StorybookJS + snapshots of each story + snapshots of the end state of an interaction
ig: At that point you could turn the recording into an “expect test”. https://blog.janestreet.com/the-joy-of-expect-tests/
we're going to the zoo: TOFU but for tests 🥰
uwaces: Are you at all worried that you are replicating the horror that is the IBM 3270 terminal? — I have personal history programming on z/OS where the only interface is a graphical EBCDIC 3027 interface — the horror is that people write programs to interact with graphical window (assuming a certain size).
ahl: https://docs.rs/serde/latest/serde/#data-formats
ahl: SHOW NOTES Bryan as "semi-elderly" engineer
MattCampbell: didn't Bryan write a blog post on this?
MattCampbell: http://dtrace.org/blogs/bmc/2008/11/16/on-modalities-and-misadventures/
uwaces: https://www.replay.io
ahl: https://devtools.fm/episode/9
ahl: e.g. https://altsysrq.github.io/proptest-book/intro.html
we're going to the zoo: https://github.com/AFLplusplus/LibAFL
ig: Are you using proptest, quickcheck, or something else?
nickik: This really started with Haskell https://hackage.haskell.org/package/QuickCheck Its also cool that it does 'narrowing' meaning it will try to find an error, and then try to generate a simpler error case.
endigma: how different is something like this from what go calls "fuzzing"
Riking: Fuzzing does also have a minimization step
we're going to the zoo: https://github.com/dubzzz/fast-check
Riking: Property-based testing tends to be structured differently in philosophy, while fuzzers are more aligned to "give you a bag of bytes"
nickik: http://www.quviq.com/products/erlang-quickcheck/
endigma: yeah I can tell its a different structure, but the overall goal seems similar
we're going to the zoo: they are nonexclusive approaches to testing
papertigers: I think Kelly was doing a bunch of tests at Joyent based on quick check and prop test. First time I encountered it
we're going to the zoo: libafl provides a #[derive(Arbitrary)] macro that will provide the correct values for a struct
uwaces: Lots of stuff in Rust existed first in Haskell (build.rs, quote!, Derive macros, Traits, ect….)…
nixinator: https://tenor.com/view/%C3%B3culos-escuro-exterminador-terminator-arnold-schwarzenegger-gif-14440790
we're going to the zoo: “what do these means” depends on who you ask lol
we're going to the zoo: fast-check is 🔥 for TypeScript
endigma: if the tested function is deterministic and the test is testing arbitrary input and testing against the result to be derivative in some way of the input function by some f(x), don't you end up re-implementing the tested function to provide the expected result? how does the author choose what properties of a system to test without falling into a "testing the test" pit?
we're going to the zoo: Rust: “Here comes the Haskell plane!”
nixinator: Isn’t rust == oxidation
endigma: yes
endigma: in a scientific sense
nixinator: Iron oxide 🙂 lol
nixinator: Very good!
GeneralShaw: Is prop test a way of formal verification? Is it same/different?
ahl: https://dl.acm.org/conference/aadebug
ig: I mean, Haskell is an academic rese...

Does a GPT future need software engineers?

21 mars 2023 | 99 min

Bryan and Adam and the Oxide Friends take on GPT and its implications for software engineering. Many aspiring programmers are concerned that the future of the profession is in jeopardy. Spoiler: the Oxide Friends see a bright future for human/GPT collaboration in software engineering.

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from March 20th, 2023.

In addition to Bryan Cantrill and Adam Leventhal, speakers on MM DD included Josh Clulow, Keith Adams, Ashley Williams, and others. (Did we miss your name and/or get it wrong? Drop a PR!)

Live chat from the show (lightly edited):

ahl: John Carmack's tweet
ahl: ...and the discussion
Wizord: https://twitter.com/balajis/status/1636797265317867520 (the $1M bet on BTC, I take)
dataphract: "prompt engineering" as in "social engineering" rather than "civil engineering"
Grevian: I was surprised at how challenging getting good prompts could be, even if I wouldn't quite label it engineering
TronDD: https://www.aiweirdness.com/search-or-fabrication/
MattCampbell: I tested ChatGPT in an area where I have domain expertise, and it got it very wrong.
TronDD: Also interesting https://www.youtube.com/watch?v=jPhJbKBuNnA
Wizord: the question is, when will it be in competition with people?
Wizord: copilot also can review code and find bugs if you ask it in a right way
ag_dubs: i suspect that a new job will be building tools that help make training sets better and i strongly suspect that will be a programming job. ai will need tools and data and content and there's just a whole bunch of jobs to build tools for AI instead of people
Wizord: re "reading manual and writing DTrace scripts" I think it's possible, if done with a large enough token window.
Wizord: (there are already examples of GPT debugging code, although trivial ones)
flaviusb: The chat here is really interesting to me, as it seems to miss the point of the thing. ChatGPT does not and can not ever 'actually work' - and whether it works is kind of irrelevant. Like, the Jaquard Looms and Numerical Control for machining did not 'work', but that didn't stop the roll out.
Columbus: Maybe it has read the dtrace manual 😉
JustinAzoff: I work with a "long tail" language, and chatgpt sure is good at generating code that LOOKS like it might work, but is usually completely wrong
clairegiordano: Some definite fans of DTrace on this show
ag_dubs: a thing i want to chat about is how GPT can affect the "pace" of software development
sudomateo: I also think it's a lot less than 100% of engineers that engage in code review.
Wizord: yes, I've had some good experience with using copilot for code review
ag_dubs: chatgpt is good at things that are already established... its not good at new things, or things that were just published
Wizord: very few people I know use it for the purpose of comments/docs. just pure codegen/boilerplayes
chadbrewbaker: "How would you write a process tree with dtrace?" (ChatGPT4)

#!/usr/sbin/dtrace -s BEGIN { printf(""%5s %5s %5s %s\n"", ""PID"", ""PPID"", ""UID"", ""COMMAND""); } proc:::exec-success { printf(""%5d %5d %5d %s\n"", pid, ppid, uid, execname); }

TronDD: That's interesting as expensive, specialized code analysis tools have been varying level of terrible for a long time
JustinAzoff: I did an experiment before where I asked it to write me some php to insert a record into a database. so of course it generated code with sql injection
chiefnoah: It's ability seems to scale with how many times someone has done the exact thing you're trying to do before
JustinAzoff: but then I asked if sql injection was bad, which it explained that it was. then I asked if the code it wrote me was vulnerable to sql injection. it then explained it was
Columbus: It misses empirical verification; forming a hypothesis, testing it, and learning from the result. There have been some attempts to implement this by feeding back e.g. command output into the prompt
JustinAzoff: so then the crazy part, I asked if sql injection was bad, why did it give me code that was vulnerable to sql injection. It the went on to say that the first thing it gave me was just for example purposes
JustinAzoff: so no wonder people get into "prompt engineering" since it's clear that you need to do things like ask for code that does something, and is secure, high performance, does not have buffer overflows or sql injection vulns
MattCampbell: In my test case ("Write a Win32 UI Automation provider in C++"), all it did was produce plausible-sounding crap
ag_dubs: pattern matching over very very large data sets
clairegiordano: Bryan just said this and I wanted to write it down, re GPT-3: "the degree that it changes its answers when you tell GPT-3 to think like someone else"
JustinAzoff: or even just, "do that, but better"
ag_dubs: i think a lot of the awe of gpt is recognizing how simple our own strategies are instead of how complex the AI's strategy is
chadbrewbaker: "How would Bryan Cantrill re-write this script?" (ChatGPT4)

#!/usr/sbin/dtrace -s #pragma D option quiet dtrace:::BEGIN { printf(""%5s %5s %5s %s\n"", ""PID"", ""PPID"", ""UID"", ""COMMAND""); self->indent = 0; } sched:::off-cpu { self->indent = 0; } proc:::exec-success { this->curpid = pid; this->curppid = ppid; this->curuid = uid; this->indent = self->indent; } proc:::exec-success /execname != ""dtrace""/ { printf(""%*s"", this->indent * 2, """"); printf(""%5d %5d %5d %s\n"", this->curpid, this->curppid, this->curuid, execname); } proc:::exec-success, sched:::on-cpu /this->curpid == pid/ { self->indent++; }

antranigv: that's pretty hit on!
chiefnoah: Yup. Most people are not doing things that haven't been done before. A significant portion of software is just building blocks of libraries
Wizord: intelligence is compression, in some sense.
dataphract: "critique the epoll API as though you are Bryan Cantrill"
ag_dubs: a brain would be much stranger!!
Wizord: the ability to reduce a large dataset to a coherent set of rules
antranigv: "Explain the issues of epoll, write as if it's a Bryan ...

On Silicon Valley Bank with Eric Vishria

17 mars 2023 | 59 min

Eric Vishria of Benchmark and Oxide CEO, Steve Tuck, join Bryan and Adam to talk about Silicon Valley Bank, its role in the startup ecosystem, and the short- and long-term effects of its collapse.

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from March 17th, 2023.

In addition to Bryan Cantrill and Adam Leventhal, we were joined special guests Eric Vishria and Steve Tuck.

(Did we miss your name and/or get it wrong? Drop a PR!)

Curated chat log from the show:

davidf: Sharing this here because I loved every bit of it: My Startup Banking Story by Mitchell Hashimoto
ewen: 'The teller looks at the paper, then looks at me, then looks back at the paper, then asks ""Are you the HashiCorp guy?"" ' 😮 (Definitely agree that post looks relevant, and is worth reading; thanks for sharing. There's quite the impedance mismatch between ""traditional banking"" and ""startup"" approaches to things. Which I suspect in part explains how SVB was so widely used by startups.)"
antranigv: Question: Are there any reasons why the US is behind in these banking things? all countries in the EU and developing countries have solved these problems decade(s) ago.
statuscalamitous: my personal, barely informed take: we built this infra earlier, so we have more legacy
a172: It sounds like what SVB was providing that was so rare was a kind of business as a service.
statuscalamitous: my favorite "scare a developer" story: the way ACH payments work. that's right, SFTP!
antranigv: I think you mean FTPS? did they move to SFTP? 😄
drkamoz: I think the opposite is also true, without the infra, Africa’s been very early to adopt mobile banking https://www.bbc.com/worklife/article/20131217-east-africa-a-mobile-banking-hub
drkamoz: Can you explain sweep funds?
Eric Likness - carpetbomberz.com: 6 months of runway some place else. Not what Peter Thiel was telling people.
antranigv: What was his response?
arjenroodselaar: Eject! Eject!
ahl: this was a fun summary: https://svbhallofshame.wordpress.com/
ahl: https://www.fdic.gov/news/press-releases/2023/pr23016.html
antranigv: This Venture Debt is intriguing, specially for startups who have a good background but are having a hard time... kinda? I guess?
ahl: Acquired: Benchmark Part I
ahl: Acquired: Benchmark Part II: The Dinner

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Rack-scale Networking

28 februari 2023 | 94 min

Bryan and Adam are joined by a number of members of the Oxide networking team to talk about the networking software that drives the Oxide rack. It turns out that rack-scale networking is hard... and has enormous benefits!

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from February 27th, 2023.

In addition to Bryan Cantrill and Adam Leventhal, speakers included Ryan Goodfellow, Levon Tarver, Ben Naecker, and Arjen Roodselaar.

Links

Here's (much of) the live chat from the show:

ahl https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.md
ahl That's the Sidecar switch episode
bcantrill https://p4.org/
admchl What does "at line rate" mean?
Riking Line rate = As fast as the packets could possibly come. 1Gbit, 10Gbit, 100Gbit, etc
admchl Do you need ASICs to hit that speed? I assume x86_64 is not going to be fast enough for these specialised operations?
levon Yes, the Tofino 2 is the ASIC
bcantrill You need ASICs
bnaecker Yes, you really can't do these kinds of operations on a general purpose CPU.
rng_drizzt Yeah, you need specialized silicon here.
JustinAzoff Right, also often across all ports at the same time in both direction. a 48 port 10gbps switch will have a line rate of 960gbps (10 ** 48 ** 2)
duckman So the advantage is being able to offload compute to the switch?
bnaecker Yes, and specifically that you can separate the data plane (operations on the packets) from the control plane (decisions about what operations to allow or make).
tahnok What's TCAM?
levon Ternary Content Addressable Memory
bnaecker https://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMs
ryaeng Sure beats logging into a number of Cisco switches and making changes at the console.
admchl This is my favourite episode in a long time, this is all really fascinating.
rng_drizzt the first Sidecar episode was nearly 1.5 years ago ü§Ø , right after we cut the first rev
levon That episode blew my mind
duckman This sounds like a big deal on the scale of ebpf
duckman Or bigger
bnaecker It is extremely useful for understanding the processing pipelines. As long as you only run single-packet integration tests üôÇ
od0 just want to go out and find things to write P4 code for
JustinAzoff <@354365572554948608> yeah one way to think about that sort of thing is that xdp can be used to run little programs on a nic, where p4 is kind of like that, but running on effectively a nic with 48+ ports
bcantrill https://github.com/oxidecomputer/p4
SyntheticGate sidecar is the "codename" of our switch box
SyntheticGate "gimlet" is our server sled
bcantrill https://github.com/oxidecomputer/propolis
wmf So you have P4 and OPTE in the hypervisor at the same time?
bnaecker OPTE is in the host kernel.
arjenroodselaar The P4 runtime Ry described only exists in the test bed, where it high level simulates the switches. OPTE is part of the production environment.
arjenroodselaar The rough difference between P4 and OPTE is that P4 works on individual packets without much concept of a session (so it can't reason about TCP streams, packet order etc, so no firewall like functionality), while OPTE aims to operate on streams of packets.
JustinAzoff So you can run 100 VMs on a test system and wire them up to your virtual switch compiled by x4c?
arjenroodselaar Correct.
bcantrill OPTE == Oxide Packet Transformation Engine
admchl Gimlet?
rng_drizzt Compute server
rng_drizzt The Sidecar switch is actually just a PCIe peripheral to a Gimlet.
bnaecker The Gimlet managing the Sidecar is often called a "Scrimlet" for "Sidecar attached Gimlet"
Riking and "how do i reconfigure this giant network without hosing my ability to reconfigure this giant network"
ShaunO can identify with that - we seriously struggle to keep our own products inter-operating, let alone anyone else's
levon It can feel like a Sisyphean task.
a172 Setup a much smaller/simpler network in parallel that is accessible from "not your network" that gets you to the management interface.
levon It's a whole new world when you can look at the actual table definitions in P4
rng_drizzt Owning all the layers here is immensely beneficial
levon Those DTrace probes have been very helpful
bnaecker Those probes turned out to be everywhere. They are are in: SQL queries, HTTP queries, log messages, Propolis hypervisor state, virtual storage system, networking protocol messages, the P4 emulator, and probably more that I'm forgetting about.
levon For those unfamiliar with the DTrace tool, or the rationale behind leveraging DTrace over other tracing / debugging tools: https://www.cs.princeton.edu/courses/archive/fall05/cos518/papers/dtrace.pdf
bcantrill https://github.com/oxidecomputer/progenitor
ahl some notes on rust codegen: https://github.com/ahl/codegen-template
arjenroodselaar DDM! Bring us home!
a172 it astonishes me how many "cloud" type architectures are built on v4 only or v4 first.
a172 IPv6 is older than Wi-Fi
a172 It solves real problems. PLEASE use it.
nyanotech yessss fina...

Memory Safety with Yael Grauer

14 februari 2023 | 78 min

Yael Grauer joined Bryan, Adam, Steve Klabnik, and the Oxide Friends to talk about her recent Consumer Reports article on memory safety and memory safe languages. How do we inform the general public? How do we persuade practitioners and companies? Thanks for joining us, Yael!

In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Yael Grauer, and Steve Klabnik.

Some of the topics we hit on, in the order that we hit them (experiment in turning the show live-chat into notes):

Nahum: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ if anyone wants to read up on the 3-2-1 Backup strategy. 👅
Cyborus: can we get a link to the talk?
Nahum: https://www.youtube.com/watch?v=Q9s2NxILBK8
Nahum: https://digital-lab-wp.consumerreports.org/wp-content/uploads/2023/01/Memory-Safety-Convening-Report-.pdf via https://digital-lab-wp.consumerreports.org/2023/01/23/new-report-future-of-memory-safety/
Nahum: https://en.wikipedia.org/wiki/Pegasus_(spyware)
Cyborus: "can we talk" => "hey. you. have a panic attack. anyways i got a cool sandwich"
AaronW: "of course we should have seatbelts" 😄
MattCampbell: but then you've got the C die-hards who say that Rust itself is too complex
AaronW: https://twitter.com/markrussinovich/status/1571995117233504257?s=46
DanCrossNYC: People used to say the same thing about PL/I and recently the COBOL people have been saying the same thing. Nothing new under the sun.
statuscalamitous: https://blog.yossarian.net/2023/02/11/The-unsafe-language-doom-principle
DanCrossNYC: People who still want to treat C as a high-level assembler are saying the same stuff the PL/I people were saying when I was young.
Eric Likness - carpetbomberz.com: In support of Yael, Ralph Nader wasn't/isn't an automotive engineer and he could still argue for lowering safety risks to car buyers. It's advocacy.
cdaringe: As an ocaml user, i was hoping revery would take off https://github.com/revery-ui/revery
statuscalamitous: https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metrics
Saethlin: Wake up babe, new 0xide reading assignment dropped
AaronW: Labelled like a can of pringles -- "20% more malloc() free()!"
Nahum: Relevant to rules based accounting: https://www.schneier.com/blog/archives/2023/02/hacking-the-tax-code.html
drew: Rigorous definitions of “unsafe code” just wont cut it ig
ig: 40% less direct pointer arithmetic than the leading brand of operating systems
a172: How does principle based accounting even work? Like, how do you define if something violates the principle or not, without just turning it back into rules based?
Eden: Checkboxes are meaningful for operational checklists. Aviation and medicine use them pretty heavily. Not so meaningful for systemic work like developing a new aircraft or a new surgery.
Eden: So I guess a rules-based approach works for lines of code, but breaks down for project-level decisions such as which language to use.
Saethlin: The S in IoT is for security
benstoltz: ifixit repairability score for HW should have an analog for SW/FW.
DanCrossNYC: That's precisely what the pl/i folks acted like 25 years ago.
sam801: c++ will live on thru carbon, cppfront, and val.
DanCrossNYC: Prediction: carbon is doa.
Saethlin: I'll believe it once anyone uses those
ig: I think the other part is there's some really important pieces of software that everyone uses daily which use memory unsafe languages. Our web browsers, and our operating systems.
AaronW: I live in a condo and I still unplug expensive electronics during a thunderstorm. Maybe it's because I had many electronics fried when I was young, and my first language was C++.
Eric Likness - carpetbomberz.com: Same with answering a landline during a thunderstorm.
DanCrossNYC: Had to stop training during thunderstorms in the Marines.
Eden: My day job is security. 😉 I rail against compliance checklists on a regular basis because a lot of auditors insist on the checkbox rather than proper security consideration. For example, PCI-DSS requires password rotation, which everyone has known for decades leads to users picking worse passwords.
alilleybrinker: https://www.usenix.org/system/files/sec22summer_alexopoulos.pdf
statuscalamitous: https://security.googleblog.com/2022/12/memory-safe-languages-in-android-13.html
a172: Google and Mozilla are making pretty good strides in migrating their browser to Rust. Still a ton of work to go, but entire systems have been moved to Rust.
JamesBrock: "Lindy" https://en.wikipedia.org/wiki/Lindy_effect
statuscalamitous: https://security.googleblog.com/2021/04/rust-in-android-platform.html
DanCrossNYC: Another issue with C/C++ in particular is that UB causes latent bugs to surface years later.
alilleybrinker: In the paper linked above, the average lifetime looks to have been about 3.5 years.
Saethlin: I learned Rust faster than C++
alilleybrinker: Related, you might be interested in EPSS: https://www.first.org/epss/
DanCrossNYC: Rust requires a bit of humility. For veteran C programmers, that can be a gut punch.
srockets: “Compiler says no” is something that Haskell was proud of, but Rust is the first language I’ve seen that managed to get popular despite of it
alilleybrinker: Humility also requires a lot of Rust https://github.com/oxidecomputer/humility
Eden: I do like the checklist item that every change must be...

Oxide and the Chamber of Mysteries

7 februari 2023 | 101 min

Revisiting Unikernels

24 januari 2023 | 83 min

The Power of Proto Boards!

17 januari 2023 | 102 min

Predictions 2023!

10 januari 2023 | 97 min

Breaking it down with Ian Brown

27 december 2022 | 85 min

A Debugging Odyssey

19 december 2022 | 95 min

Podcasts for Podcast-Lovers

13 december 2022 | 89 min

Leaving Twitter with Tim Bray

29 november 2022 | 73 min

Mastodon with Kris Nova

14 november 2022 | 91 min

Tech Layoffs

8 november 2022 | 100 min

Open Source Firmware

31 oktober 2022 | 85 min

Let That Sink In! (Whither Twitter?)

28 oktober 2022 | 103 min

Open Source Inside Baseball (with Stephen O'Grady)

25 oktober 2022 | 71 min

Holistic Boot

11 oktober 2022 | 91 min

Engineering Incentives... and Misincentives

4 oktober 2022 | 66 min

Losing the Signal with Sean Silcoff

27 september 2022 | 68 min

Bryan and Adam interview Sean Silcoff, co-author of "Losing the Signal: The Untold Story Behind the Extraordinary Rise and Spectacular Fall of BlackBerry"... Soon to be a major motion picture!

Losing the Signal with Sean Silcoff (The Rise and Fall of BlackBerry)

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 26th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, our esteemed guest was one of the authors of Losing the Signal, Sean Silcoff.

Not many links, mostly anecdotes from Losing the Signal: The Untold Story Behind the Extraordinary Rise and Spectacular Fall of BlackBerry, the book Sean co-wrote with Jacquie McNish

Some of the topics we hit on, in the order that we hit them:

@05:01 The aha moment for BBM
@09:42 Ruffled feathers from partner interactions
- Bell South thought they had an exlusive deal and they didn't
- Years later, people didn't seem to hold it against them
@12:40 Brian's anecdote about a meeting with Balsillie and Lazaridis
@15:30 Lost opportunities to course-correct
- The touch screen button
@20:00 The Blackberry Storm
- Potentially rushed to market when it was not up to standards
- RIM's own testing lab found serious problems but shipped it anyway
- RIM was a great innovator and a terrible follower, some have said that of Apple, though
@25:40 Lazaridis as the product guy
- This I get (keypad), this I don't get (touchscreen)
- Story on the way up is as important as on the way down
@30:20 Parallels with DEC
- Amazing rise
- Failure to pass control
- co-CEO model at RIM - worked really well until it didn't
@34:19 NTP Patent case
- Case brief
- Jury selection was weighed incredibly far towards lay folks with very little technical understanding
- Technical demonstration goes sideways
@45:10 Trusting the other co-CEO and the option backdating scandal
- Lazaridis didn't really understand the options stuff and felt Balsillie had put the company at risk
- Kept their fights private, but people could tell "mom and dad weren't getting along"
- Is it right or wrong if everyone was doing it?
- They left an extensive digital paper trail making it easy to make a case
- Thanks to Tom for the clarification - options backdating was okay, failing to report it was not
@52:50 Larry Conlee
- Fire and brimstone, had a pocket veto, spoke with the voice of the CEOs
- Co COOs!
- Carriers were afraid of becoming dumb pipes, so were anti-app store
- Blackberry didn't care about doing an app store, then AT&T bent to Apple and allowed them to have an app store
- RIM did not believe that Silicon Valley would be let in the front door at the carriers
- RIM would talk about Apple as "the toy company" while being actively devoured

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Threads, async/await, Promises, Futures

13 september 2022 | 75 min

Potpourri: Product, Platform, Paravirtualization

6 september 2022 | 109 min

The Oxide Supply Chain

30 augusti 2022 | 69 min

Bringup Lab Chronicles: A Measurement Two Years in the Making

23 augusti 2022 | 95 min

Surviving Conventional Wisdom

16 augusti 2022 | 82 min

RIP Optane

9 augusti 2022 | 103 min

Deep Tech Investing

2 augusti 2022 | 75 min

Across the Chasm with Rust

19 juli 2022 | 104 min

Integrating Hardware and Software Teams

12 juli 2022 | 103 min

Books in the box redux

28 juni 2022 | 104 min

Paths into Systems Programming

21 juni 2022 | 107 min

The Rise and Fall of DEC

14 juni 2022 | 112 min

Surviving the Dot-Com Bust

24 maj 2022 | 92 min

Oxide and Friends Twitter Space: May 23rd, 2022

Surviving the Dot-Com Bust

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for May 23rd, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on May 23rd included Dan McDonald, Dan Cross, Joshua Clulow, Steve Tuck, Matt Campbell and Theo Schlossnagle. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

@1:37 Pick and shovels story circulating at Sun
- Oakes Ames "King of Spades", pbs article, wiki
- boo.com wiki
- pets.com wiki
@11:00 IPOs and public exposure
- theGlobe.com wiki
@18:20 "The Correction"
- Feasting like 19th century robber barons
- Nov 2000, free fall
- Trilogy, Inc wiki
@28:49 Students looking for placement
- Clarity of the bust
@36:35 Billboards on the 101
- garden.com, cnn blurb
@39:13 Theo's story, roulette at Trilogy
Expansion and contraction of CS student enrollment
@46:20 Matt's memories
- AllAdvantage wiki
- Excite@Home wiki
John Talbott "The Fall of Silicon Graphics" article
Fucked Company wiki
Camaraderie over watching your companies imploding
@53:39 Looking towards the looming housing bubble
- During Oxide raise, race against time for VC funding
Pandemic
- Hot venture environment, over-valued companies
- Stimulus, spending on non-essentials, exacerbating income inequality
@58:56 Differences from the dot-com era, more defined revenue models?
- Food delivery services, harbingers of bust?
- Steve anecdote: Dellionaires, margin call day, layoffs
@1:08:12 Dan's second startup experience
- Windows on the World wiki
@1:10:15 Matt's question: can Oxide weather a tech bust?
- Adam: downturn can motivate seeking value, looking away from (stable, pricey) incumbents to (riskier, cheaper) new offerings
- Bryan: dot com bust pushed us toward open-source, for economic reasons.
@1:15:51 Are we headed for a bust? How deep?
- How does a company survive the lean times?
- Negative human consequences
- mammon (money, material wealth) wiki
- Advice for practitioners?
Dan McDonald: controlling inflation, starting companies
Theo: downturn will hit industries differently, concerns over global supply chain
Adam: don't forget about helping others, looking out for other people, for the future of our world.
Dan Cross: if it looks to good to be true, it probably is.

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Debugging Methodologies

17 maj 2022 | 90 min

Oxide and Friends Twitter Space: May 16th, 2022

Debugging Methodologies

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for May 16th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, our special guests on May 16th were Jordan Hendricks and Luqman Aden. Other speakers included jasonbking, Rick Altherr and Ben Kimock. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Green Room wiki
NVMe wiki (Non-Volatile Memory. PCI Express)
@3:38 Jordan's story
- Jordan's thorough bug write-up, (reported by Josh Clulow as "nvme_quiesce() can hang preventing reboot")
- Non-maskable interrupt wiki
- @8:04 Adam interrupts a box with a kitchen knife
- kmdb man page and page in the mdb book
- @14:11 Josh recites a poem about timeouts
- Avoiding getting stuck, experimenting
- @20:10 A previous encounter with NVMe/PCIe issues (see also: Jordan's NVMe Hotplug discussion video ~26mins)
- mdb format character "j" (for Jordan!) (and jazzed-up) feature
- @26:50 Normal and abrupt shutdown notification, breakthrough, writing up a narrative
@32:27 Luqman's story
- The blog post "Achievement Unlocked: rustc segfault"
- dtrace usdt
- cscope, rust analyzer
@43:50 Inspecting LLVM IR, RustC MIR
- async blocks, inline assembly
- boiling down reproducible cases
- making quality write-ups, telling a story, teaching debugging
- popular on Hacker News
- dead reproducible?
@1:03:02 Bugs: psychotic, non reproducible
- Debugging mindset
- Different tools and methodologies for different problems
- anonymous tracing book page, speculative tracing page
@1:10:03 Jason: number literal formats with underscores, now in mdb
@1:12:35 Ben prompts a debugging story, checking conditions in debug, program abort on error
- ud2 instruction
- Rick describes the Oxide boot loader
- XMODEM wiki
- Triple fault wiki
- Rust "heapless" crate

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Fail Whaling

26 april 2022 | 102 min

Oxide and Friends Twitter Space: April 25th, 2022

Fail Whaling

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for April 25th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Jason Hoffman. Other speakers included Joshua Clulow, Matt Campbell and Ian. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Debugging Rails
Jason walks chain of events leading to "twttr"
@10:46 The first mock up, SMS
@13:42 Twitter goes live, early days
- Tcl, Mongrel, LiteSpeed
@19:30 First problems
- Bryan's debugging story, exceptions and backtraces, index out of bounds
- Discovery of the problem was not met with gratitude
@29:53 Jason tells another problem story, production directories full of junk test files
@38:30 Story of the first Hadoop cluster
@42:22 Matt's comment on directory limits
@46:35 Companies growing up, on-prem and cloud infrastructure
@49:26 The Fail Whale
- Ruby runtime, Ryan Dahl
- Moved to Java, Scala eventually
- DTrace and dynamic languages
- Raku, Parrot VM, MoarVM
@59:53 Changing language and hardware landscapes, video presentation sharing, short social media handles, ahl, getting into hockey
@1:12:30 Billionaire's playground?
- Quick diversion, history trivia bet
- @1:18:43 Moderation
- Microsoft Tay bot (shutdown 16 hours after launch)
- Can anything kill Twitter?
- @1;29:26 Matt: what replaces Spaces?
- How could an alternative be built? What would it look like?
- Bryan predicts: change of headquarters, "burning the flag"
- Adam predicts: resale or IPO within 3 years
See also: Jason Hoffman and Bryan Cantrill CTO vs VP Engineering video ~45mins (audio is rough, content is good)

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

More Tales from the Bringup Lab

19 april 2022 | 127 min

Oxide and Friends Twitter Space: April 18th, 2022

More Tales from the Bringup Lab

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for April 18th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by members of the Oxide team: Arjen Roodselaar, Nathaneal Huffman, Robert Mustacchi, Aaron Hartwig, Steve Tuck, Matt Keeter, Eric Aasen, Rick Altherr, and RFK.

Some of the topics we hit on, in the order that we hit them:

@2:25 Overview of upcoming themes related to the bringup lab
@4:28 Defining the different terms and code-names of the hardware in development at oxide
@4:40 Gimlet, the compute node
@5:10 Sidecar, a board based on a switching ASIC from Intel
@7:24 Arjen's twitter thread with details related to the bringup and Eric's description of the challenges in designing the PDN (Power Delivery Network) ATT
@15:34 The load-slammer, an electronic load to simulate the power draw of an ASIC / BGA-part LS
@19:06 Bouncing supply cables on load steps
@22:27 FPGA that controls everything on the Sidecar board
@24:05 TOML's unstable table order made the team pop a couple ICs off the board searching for bugs
@31:41 Brown-out in the hotel during first bringup session from a blown bus duct
@33:45 Debugging ground bounce issues while testing the PDN with the load-slammer (phantom over/undershoot)
@40:15 Hardware team pranks the management during a meeting with a potential investor
@43:20 Chonky heat sink that weighs 8 pounds / moment arm crisis
@48:19 First time powering up, checking temperature with thermal camera, learning about "puppy dog warm"
@52:12 Matt talks about the second, "lesser" network switch on the Sidecar board
@57:28 Secret 8051 cores, slew-rate woes: impedance missmatch on SPI traces that manifested in unreliable communication in full bandwidth mode of the SPI/GPIO driver
@1:03:19 PLL config issues and Matt's verbose config tool to fix them
@1:04:26 Load-bearing dongles
@1:06:37 Debugging PCIe link, Arjen's Frankenstein PCIe analyzer/exerciser
@1:22:36 Gimlet, stumbling blocks found in January
@1:30:08 Arjen's big breakthrough on the Sidecar, shouting at the T6
@1:32:08 Cursed pull-downs, Rick's remote hardware debugging support by incrementally breaking his T6 boards to find issue with the DUT
@1:36:24 T6 finally comes out of reset, "we're gonna live! we're gonna live!"
@1:41:06 Rick reworks gnarly footprint error, on multiple ICs, to verify design for Rev. B - dead bug style.
@1:53:12 Sidecar progress continuation, cable oupsi, off-by-one error
@1:59:42 Dedicated support by IC vendor with very understanding wives
@2:01:20 Summary and parting thoughts

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Another LPC55 ROM Vulnerability

5 april 2022 | 81 min

Oxide and Friends Twitter Space: April 4th, 2022

Another LPC55 ROM Vulnerability

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for April 4th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Laura Abbott.

Other speakers on April 4th included Ian, jasonbking, Todd Gamblin?, Ben ?, MattSci, jasonbking and Evan?. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Jonathan Goldstein's Heavyweight podcast
Oxide and Friends podcast
- transistor.fm launch point, has links to Spotify, Google, Amazon etc players
Laura did talk about the first LPC55 vulnerability in the May 3, 2021 space, but the recording for that day missed it.
- Laura Abbott (30 April, 2021) Exploiting Undocumented Hardware Blocks in the LPC55S69 write-up
  - And DEF CON talk with Rick Altherr
@4:01 Today's topic: Laura Abbott (23 March 2022) Another vulnerability in the LPC55S69 ROM write up
- How do you brick a chip?
@7:20 The spreadsheet, ROM patch after boot
- Company dismisses or downplays vulnerabilities
- Sees CVEs as optional??
@15:19 CVEs as more software focused. What does a CVE for hardware even mean?
- NXP doesn't want to open their software
"Even though we are not believers in security by obscurity, the product specific ROM code is not open to external parties except for approved test labs for vulnerability reviews"
@19:43 The story of the current vulnerability
- Ghidra
@27:26 Picking apart the code
- Bounds checks, writing outside the bounds of the buffer
- DICE by Trusted Computing Group
- Request for Discussion
- Evaluating potential chips when building a product
@41:09 Secure hardware, work around potential pitfalls
- Open source would help
@45:37 Disclosed to NXP, more receptive this time
- Discussion on HN
- @54:21 Security review industry
@57:11 Ian: building up your own (open) documentation on LPC55?
@1:01:31 Jason: questionable definitions of "open" source
- Access to source as building confidence in the product
@1:05:20 Todd: securing supply chain for code in large scale projects with lots of contributors
- Vulnerabilities can occur so easily
@1:08:54 Ben: custom setups abound. Hard to trust a whole stack of assembled pieces
@1:12:16 Matt: what is the ROM doing? Assembly or C? Could the provider's hands be tied as far as releasing proprietary code?
@1:17:19 Jason: X.509 parsing as a good place to look for vulnerabilities?
@1:18:25 Evan: encouragement around fuzzing X.509
Next time: more tales from the bringup lab!

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!

Time, Timezones, Metric Time, Losing and Saving

29 mars 2022 | 66 min

Oxide and Friends Twitter Space: March 28th, 2022

Time, Timezones, Metric Time, Losing and Saving

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 28th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on March 28th included Tom Lyon, jasonbking, Matt Campbell, Akshay Kumar, Aaron Goldman and Simeon Miteff. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@8:07](https://youtu.be/BHtfqleSHAs?t=487) Y2K, leap years
- The Staff of Ra
- “at” command
[@15:28](https://youtu.be/BHtfqleSHAs?t=928) Matt’s stories
- elm email
[@23:29](https://youtu.be/BHtfqleSHAs?t=1409) Jason: daylight saving time in Indiana
- “Time in Indiana” wiki
[@26:31](https://youtu.be/BHtfqleSHAs?t=1591) Time zone database
- John Bemelmans Marciano (2014) Whatever Happened to the Metric System? How America Kept Its Feet book
- Geopolitical aspects of time
- Eastman plan calendar
[@32:23](https://youtu.be/BHtfqleSHAs?t=1943) Aaron’s stories, setting clocks back, Leap Day
[@35:54](https://youtu.be/BHtfqleSHAs?t=2154) Akshay: Ken Thompson’s six day work week?
Leap seconds
- Time of day hardware bug
[@48:54](https://youtu.be/BHtfqleSHAs?t=2934) 2038 - the end of time
- Y2K problems
- GPS week number rollover wiki
[@57:58](https://youtu.be/BHtfqleSHAs?t=3478) Matt: Cory Doctorow’s “Epoch” short story podcast commissioned by Mark Shuttleworth
[@1:00:28](https://youtu.be/BHtfqleSHAs?t=3628) Ultimate, penultimate, antepenultimate
Oxide and Friends podcast!!
- transistor.fm launch point, has links to Spotify, Google, Amazon etc players
Laura Abbott (23 March 2022) Another vulnerability in the LPC55S69 ROM write up

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Trolltron, Assemble!

22 mars 2022 | 71 min

Oxide and Friends Twitter Space: March 21st, 2022

Trolltron, Assemble!

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 21st, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on March 21st included Antranig Vartanian, Dan Cross, Ian, jasonbking, Jason Ozolins, Ken and Drew Vogel. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@9:23](https://youtu.be/WrEef_bsWas?t=563) I was learning from people who were further down the track than I was
- Startups can have problems when founders fail to learn from the experiences of others
[@12:43](https://youtu.be/WrEef_bsWas?t=763) Dan: hubris of youth is an age old problem, see middle ages nobility
For some “child wonders”, their childhood is effectively sacrificed because their adulthood arrives too early
[@16:22](https://youtu.be/WrEef_bsWas?t=982) When I went to school, there was a math prodigy..
- Challenging operating system course
[@25:44](https://youtu.be/WrEef_bsWas?t=1544) Ian: for early accelerated learners, the work is easy until it isn’t. They didn’t need to spend long hours studying, so they didn’t practice it. > You have to take that youthful ego, and gently massacre it. Then build them up
The Dropout series, premiered March 2022
[@31:26](https://youtu.be/WrEef_bsWas?t=1886) Jason O: praising ability vs effort, negative effects
[@34:55](https://youtu.be/WrEef_bsWas?t=2095) 30 under 30, and such things
Empathy, learning to compromise, learning from being a parent
[@41:04](https://youtu.be/WrEef_bsWas?t=2464) How venture views human capital
- Student loans, (some predatory lenders)
- Does making a young person comfy lead to their best work?
- Taking a share of future earnings, kinda demotivating. Misaligned incentives
- Lambda school, coding bootcamps. Fixed costs and incoming sharing
[@50:17](https://youtu.be/WrEef_bsWas?t=3017) Sourcing these kids?
- “You’re a baseball card for someone” story, why are these kids at this party??
[@57:40](https://youtu.be/WrEef_bsWas?t=3460) Sometimes kids who are extremely comfortable aren’t terribly motivated to put in the hours
[@59:25](https://youtu.be/WrEef_bsWas?t=3565) Drew: Income sharing and other schemes to pay for education
- Ken: doesn’t feel like kids would be set up for success
[@1:04:07](https://youtu.be/WrEef_bsWas?t=3847) Background beef leading to this hairy scheme
- Some entrepreneurs have trouble seeing the role of luck in their success
- Thiel Fellowship wiki

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Ukraine

15 mars 2022 | 85 min

Oxide and Friends Twitter Space: March 14th, 2022

Ukraine

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 14th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Andrey Akselrod.

Some of the topics we hit on, in the order that we hit them:

If you’re interested in donating to support Ukrainians, Andrey recommends Nova Ukraine
[@1:52](https://youtu.be/EdJU8mSWzQk?t=112) Andrey introduces himself, background in computing
[@11:20](https://youtu.be/EdJU8mSWzQk?t=68) Andrey talks about where he lived in Ukraine, Dnipro
- Confluence of cultures
- Moves to New York
[@22:53](https://youtu.be/EdJU8mSWzQk?t=1373) Events of 2014, family and coworkers in Ukraine
- Crimea
[@29:12](https://youtu.be/EdJU8mSWzQk?t=1752) Earlier disputed regions (Crimea, Donbas) and relations to current events
- Ukrainian national identity
[@38:21](https://youtu.be/EdJU8mSWzQk?t=2301) Armed forces, self governance
- Business as usual, life goes on
[@44:45](https://youtu.be/EdJU8mSWzQk?t=2685) Characterizing Ukraine as European democracy, and economic functions/trade
- Nuclear reactors
[@49:12](https://youtu.be/EdJU8mSWzQk?t=2952) Invasion
- Leadership disconnect with reality
[@1:02:28](https://youtu.be/EdJU8mSWzQk?t=3748) Family still in Dnipro
- Electronic communications
- Kids understanding of what’s happening
[@1:07:59](https://youtu.be/EdJU8mSWzQk?t=4079) How to help?
[@1:16:50](https://youtu.be/EdJU8mSWzQk?t=4610) Andrey’s coworkers and team members remaining in Ukraine > Yes it’s war, but, the economy needs to continue to be healthy.
[@1:21:24](https://youtu.be/EdJU8mSWzQk?t=4884) Where is this going?

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

The Future Of Work

8 mars 2022 | 118 min

Oxide and Friends Twitter Space: March 7th, 2022

The Future Of Work

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for March 7th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on March 7th included Lucas Ives, Dan McDonald, Steve Tuck, Ian, Matt Campbell, MattSci, Jim Rybarski, Austin, Aaron Goldman, Jake Demarest-Mays, Jason Ozolins, Tom Lyon, Timon, Matthew Amdur, jasonbking, and Horace. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@8:15](https://youtu.be/GTluipbKeII?t=495) Lucas’ story
Remote before pandemic, comparisons
[@16:29](https://youtu.be/GTluipbKeII?t=989) Sidebar chat, backchannel
[@22:49](https://youtu.be/GTluipbKeII?t=1369) Pre-recorded talks, speaker commenting in chat engaging with questions
- Multitasking during meetings, different from in-person single-threaded meetings
- Recording meetings for later review
- Holding onto a thought may detract from fully listening to another’s point
[@34:40](https://youtu.be/GTluipbKeII?t=2080) Oxide’s full team meetup, what did they focus on?
[@38:01](https://youtu.be/GTluipbKeII?t=2281) Austin’s remote experience
[@44:30](https://youtu.be/GTluipbKeII?t=2670) Dan’s question: remote employees “pilgrimage” back to home often, how often?
[@50:23](https://youtu.be/GTluipbKeII?t=3023) Disadvantages to full remote?
[@56:15](https://youtu.be/GTluipbKeII?t=3375) Jake’s experience, asynchronous work style
- Meetings as unprepared group think sessions, not valuable as decision making
- Requests for discussion, as decision making tools
[@1:02:29](https://youtu.be/GTluipbKeII?t=3749) Jason: service delivery vs product delivery
- Class devision between “the desked” and “the un-desked”
[@1:07:17](https://youtu.be/GTluipbKeII?t=4037) Is “back to office” about command and control?
- Other factors: big tech companies receive substantial local subsidies
[@1:14:00](https://youtu.be/GTluipbKeII?t=4440) Timon on working in different timezones
- Recorded meetings/discussions as valuable content
- Pandemic boosted remote work tool quality
[@1:23:32](https://youtu.be/GTluipbKeII?t=5012) Difficulties with remote?
- Building rapport, judging emotions and nuanced communication
- Organic, unplanned communications with in-person office spaces (watercooler)
[@1:33:24](https://youtu.be/GTluipbKeII?t=5604) Matt: remote work as cost savings?
Value of “down time” communication, unstructured
[@1:43:50](https://youtu.be/GTluipbKeII?t=6230) Starting career, making connections, in all-remote world
[@1:47:58](https://youtu.be/GTluipbKeII?t=6478) Future of remote work since pandemic
[@1:51:30](https://youtu.be/GTluipbKeII?t=6690) Horace’s experience with remote work

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Engineering Culture

22 februari 2022 | 104 min

Oxide and Friends Twitter Space: February 21st, 2022

Engineering Culture

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for February 21st, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on February 21st included Tom Lyon, Tom Killalea, Ian, Antranig Vartanian, Matt Campbell, Simeon Miteff, Matt Ranney and Aaron Hartwig. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Alex Heath’s tweet on FB meeting about updated values: “meta, metamates, me”
[@4:44](https://youtu.be/w9MQJbC26h4?t=284) Can an established company “change its values” in any sense?
[@8:43](https://youtu.be/w9MQJbC26h4?t=523) Draw the owl > Twilio CEO: Yes, it was a meme, but it’s a great representation of our job. > There is no instruction book and no one is going to tell us how to do our work. > It’s now woven into our culture and used as a cheeky, but encouraging reply to > those who email colleagues at Twilio asking how to do something.
[@12:42](https://youtu.be/w9MQJbC26h4?t=762) How do you establish engineering culture?
- Copy-paste values?
[@20:44](https://youtu.be/w9MQJbC26h4?t=1244) When are values set down in a company’s history?
- Amazon’s brand image, expanding beyond books
- Assessing values when hiring
[@27:51](https://youtu.be/w9MQJbC26h4?t=1671) Principles vs values
- Principles are absolutes, cannot be taken too far
- Values are about relative importance, in balance with other values
- ACM Code of Ethics
- Relative importance of values. Can some values be learned, while others cannot?
[@45:11](https://youtu.be/w9MQJbC26h4?t=2711) “Turn-around CEOs”, trying to change an established company culture
[@47:39](https://youtu.be/w9MQJbC26h4?t=2859) Sun culture, early days
[@54:32](https://youtu.be/w9MQJbC26h4?t=3272) Connection between values and business model
- Urgency in context, requires nuance
[@1:03:37](https://youtu.be/w9MQJbC26h4?t=3817) Values on the wall. When are values simply ignored?
- Jack Handey wiki, Deep Thoughts recurring SNL short sketches, eg Thanksgiving ~30secs
- “Sharpen fast”
[@1:13:49](https://youtu.be/w9MQJbC26h4?t=4429) What are the important things to get set early?
- Bryan and Adam on Joyent and Delphix
[@1:22:05](https://youtu.be/w9MQJbC26h4?t=4925) Matt Ranney on his time at Uber
- Trying to shape an established culture
- Leadership’s values vs engineers
- Business ethics
[@1:35:47](https://youtu.be/w9MQJbC26h4?t=5747) GE
Thomas Gryta and Ted Mann (2020) Lights Out: Pride, Delusion, and the Fall of General Electric book
[@1:37:03](https://youtu.be/w9MQJbC26h4?t=5823) Conclusions
- Adam: Get it right first, but it’s not a lost cause if you don’t.
- Bryan: Look for value alignment in organizations you might want to join, it’s tough to change course after the fact.
- Matt: generous compensation has an effect on how closely one cares to scrutinize their organization’s values ¯_(ツ)_/¯

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Breakthroughs Delayed

15 februari 2022 | 69 min

Oxide and Friends Twitter Space: February 14th, 2022

Breakthroughs Delayed

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for February 14th, 2022

In addition to Bryan Cantrill and Adam Leventhal, speakers on February 14th included Chris DiBona, Tom Lyon, Ian, MattSci, Jeff Nickoloff, Ahmed, Tim Burnham and vint serp. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Adam’s tweet
Steven Johnson (2021) Extra Life: A Short History of Living Longer book
[@6:00](https://youtu.be/MyGgkBxz-mg?t=360) Pasteurization
- 1850’s swill milk scandal wiki
[@10:25](https://youtu.be/MyGgkBxz-mg?t=625) Automotive safety
- Three-point seat belt wiki
- Windshield safety glass wiki
- Ralph Nader (1965) Unsafe at Any Speed book
[@16:25](https://youtu.be/MyGgkBxz-mg?t=985) Bryan proposes a rubric, are multiple teams racing?
- Walter Isaacson (2021) The Code Breaker book
- Edward Jenner, 1796 smallpox vaccine
[@24:32](https://youtu.be/MyGgkBxz-mg?t=1472) DTrace
- Compact C Type Format CTF
[@27:25](https://youtu.be/MyGgkBxz-mg?t=1645) Docker
- OverlayFS
- Bryan’s Papers We Love talk on Jails and Zones video ~100mins
- 1963 Honeywell H200 wiki
- Bryan on harware virtualization history video ~10mins, also here
[@37:22](https://youtu.be/MyGgkBxz-mg?t=2242) The Greate Stirrup Controversy wiki
Steve Kemper (2005) Reinventing the Wheel: A Story of Genius, Innovation, and Grand Ambition book
Jevons paradox wiki
[@47:51](https://youtu.be/MyGgkBxz-mg?t=2871) Wikipedia
- Bryan gets worked up at a dinner party
- Cliff Clavin (Cheers character) wiki
- [@52:54](https://youtu.be/MyGgkBxz-mg?t=3174) Hello Chris!
- [@57:23](https://youtu.be/MyGgkBxz-mg?t=3443) Wordle trolling
  - [@57:40](https://www.youtube.com/watch?v=MyGgkBxz-mg&t=3460s) Audio editing
[@1:01:03](https://youtu.be/MyGgkBxz-mg?t=3663) JSON
[@1:02:22](https://youtu.be/MyGgkBxz-mg?t=3742) Chris on HBO Silicon Valley
[@1:07:05](https://youtu.be/MyGgkBxz-mg?t=4025) Antikythera mechanism wiki

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

I Know This! (Purpose-built systems with general-purpose guts)

8 februari 2022 | 81 min

Oxide and Friends Twitter Space: February 7th, 2022

I Know This! (Purpose-built systems with general-purpose guts)

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for February 7th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on February 7th included MattSci, Ian, Matt Ranney and Ken. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Calendly tweet context
[@11:47](https://youtu.be/WsvJT6i_atw?t=707) Hacker News post
[@18:15](https://youtu.be/WsvJT6i_atw?t=1095) James Garfield shooting
[@21:29](https://youtu.be/WsvJT6i_atw?t=1289) Adam’s story about customers taking on heroic interventions themselves, learning the value of logging all commands, and digging through email chains for paydirt
- Developed “three strikes” rule, focus on fixing the proximate issues (and defer general health boosters for another time) so as not to lose the faith of the customer
[@27:35](https://youtu.be/WsvJT6i_atw?t=1655) E-cache parity error
[@33:38](https://youtu.be/WsvJT6i_atw?t=2018) Support personnel remaining calm in the face of unknown damage
[@41:22](https://youtu.be/WsvJT6i_atw?t=2482) Outages, postmortem, software as a service and public cloud providers
- Vendor transparency or lack thereof
[@48:28](https://youtu.be/WsvJT6i_atw?t=2908) Ken: transparency as part of legal compliance?
MITRE CVE List of publicly disclosed cybersecurity vulnerabilities
[@52:45](https://youtu.be/WsvJT6i_atw?t=3165) Adventures in shady pay to play industry events
- Fixed raffles
[@1:01:30](https://youtu.be/WsvJT6i_atw?t=3690) “We never lost anyone’s data but it took some long vacations”
- Incident where someone corrupted kernel data structures
- Adam pulls a fast one
- Paul Newman and Robert Redford in (1973) The Sting movie
- Different ways to structure support contracts
- mdb -kw, the w is load bearing

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Taxonomy of Hype

25 januari 2022 | 83 min

Oxide and Friends Twitter Space: January 24th, 2022

Taxonomy of Hype

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for January 24th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on January 24th included MattSci, Todd Gamblin, Aaron Goldman and Tom Lyon. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

The tweet about the topic: Johannes Klingebiel’s (2022) The five Levels of Hype taxonomy
[@8:24](https://youtu.be/qrWgmkBfn9s?t=504) Roko’s Basilisk (slate.com)
[@10:21](https://youtu.be/qrWgmkBfn9s?t=621) Cloud Computing
[@12:09](https://youtu.be/qrWgmkBfn9s?t=729) Mobile, Wi-Fi (introduced in 1997)
- Adam broke his hand, but can still type dtrace with one hand
[@15:14](https://youtu.be/qrWgmkBfn9s?t=914) Java
- Write once run anywhere
- Cross platform graphical interfaces
Windows NT
[@17:47](https://youtu.be/qrWgmkBfn9s?t=1067) Storage technology
- Dedup
- ZFS copies setting and redundant_metadata
- InfiniBand, iSCSI Extensions for RDMA (iSER), SCSI RDMA Protocol (SRP)
[@26:15](https://youtu.be/qrWgmkBfn9s?t=1575) 3D XPoint (Intel Optane) wiki
- HP Memristor FAQ
- HP “The Machine”
  - HP research’s pure hype marketing pitch
  - The (absolutely incredible) Star Trek crossover ad > I’m gonna provide you the emotion of a revolution, but not the technical detail to > support it, not yet, but it’s coming.
[@31:02](https://youtu.be/qrWgmkBfn9s?t=1862) Segway (wiki)
- Dean Kamen wiki
- Decoder Ring podcast (June 2021) Who Killed the Segway? ~40mins slate.com, Apple podcasts
- 2001 Good Morning America Segway unveiling, Diane Sawyer is underwhelmed > I’m tempted to say “that’s it??” (nervous laughter) > But that can’t be it!?
[@34:29](https://youtu.be/qrWgmkBfn9s?t=2069) Maglev, Cold fusion
- Walter Isaacson (2021) The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race book
- Human Genome Project wiki
- Hype booms and busts
- Todd’s story on working on fusion at a national lab, and the nature of gaining funding for large projects
[@45:30](https://youtu.be/qrWgmkBfn9s?t=2730) Rust
[@48:43](https://youtu.be/qrWgmkBfn9s?t=2923) DTrace
[@52:14](https://youtu.be/qrWgmkBfn9s?t=3134) Nanotechnology
- K. Eric Drexler wiki
- Expert Systems, AR/VR
[@56:23](https://youtu.be/qrWgmkBfn9s?t=3383) Chatbots
- Dan Olson (Jan 2022) Line Goes Up - The Problem with NFTs ~2hr video (worth every minute)
[@59:11](https://youtu.be/qrWgmkBfn9s?t=3551) Serverless
- Itanium IA-64, Very long instruction word VLIW
- Fibre Channel over Ethernet FCoE, ATA over Ethernet AoE > A solution in search of a problem
[@1:06:50](https://youtu.be/qrWgmkBfn9s?t=4010) Taligent wiki
- Tom Hormby (2014) Pink: Apple’s First Stab at a Modern Operating System post
- Be Inc wiki
  - Bryan’s Be whiteboard story
[@1:13:47](https://youtu.be/qrWgmkBfn9s?t=4427) Docker
- Monetizing open source
[@1:20:28](https://youtu.be/qrWgmkBfn9s?t=4828) 5G

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Flying Blind with Peter Robison

11 januari 2022 | 90 min

Oxide and Friends Twitter Space: January 10th, 2022

Flying Blind with Peter Robison

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for January 10th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Peter Robison.

Other speakers on January 10th included MattSci and Simeon Miteff. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@5:02](https://youtu.be/q6i9NPslfE4?t=302) Peter on Japan Air Lines Flight 123
Boeing 777 > Bryan: The things I am the most proud of are the things I’ve worked with other people on, > when a team does something that feels beyond an individual’s grasp.
[@12:25](https://youtu.be/q6i9NPslfE4?t=745) Peter’s history covering aerospace
- McDonnell Douglas
[@15:53](https://youtu.be/q6i9NPslfE4?t=953) Jack Welch, corporate culture
- Investors over customers
- John Godson 1975 The Rise and Fall of the DC-10 book
[@24:12](https://youtu.be/q6i9NPslfE4?t=1452) Questionable morals from execs
John Newhouse 1982 The Sporty Game book
[@27:41](https://youtu.be/q6i9NPslfE4?t=1661) When did it become clear that the 737 MAX was problematic?
- Lion Air Flight 610
- Maneuvering Characteristics Augmentation System MCAS
- Ethiopian Airlines Flight 302
- Shifting blame, public messaging vs behind closed doors opinion forming
[@36:31](https://youtu.be/q6i9NPslfE4?t=2191) Why pilots had no training (or knowledge of) the MCAS system
[@39:23](https://youtu.be/q6i9NPslfE4?t=2363) Angle of Attack indicator
[@48:48](https://youtu.be/q6i9NPslfE4?t=2928) MCAS software, writing safety critical computer code
- [@53:19](https://youtu.be/q6i9NPslfE4?t=3199) “Blood on the seats”
[@58:48](https://youtu.be/q6i9NPslfE4?t=3528) Matt asks about “fly-by-wire” and MCAS. “Optional” safety features
[@1:08:04](https://youtu.be/q6i9NPslfE4?t=4084) Testing safety, lack of technical scrutiny
[@1:12:31](https://youtu.be/q6i9NPslfE4?t=4351) Simeon asks about the FAA’s relationship with Boeing
[@1:15:05](https://youtu.be/q6i9NPslfE4?t=4505) Bryan: what are the lessons for other disciplines?
- Peter: Valuing employee views. Tolerating bad news.
- Adam: The engineering culture at Boeing was so arduous to build, and so quick to corrode
[@1:18:39](https://youtu.be/q6i9NPslfE4?t=4719) Matt: relationship to F-35? Military vs commercial
[@1:23:23](https://youtu.be/q6i9NPslfE4?t=5003) Gene Kim: CEO congressional testimony
[@1:26:22](https://youtu.be/q6i9NPslfE4?t=5182) Passing certifications, alternatives to MCAS

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Predictions 2022

4 januari 2022 | 126 min

Oxide and Friends Twitter Space: January 3rd, 2022

Predictions 2022

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for January 3rd, 2022.

In addition to Bryan Cantrill and Adam Leventhal, our special guest on January 3rd included was tech prediction expert and noted Red Sox fan Steven O’Grady.

Below is a table of the oracles and their predictions: (If you made predictions, please submit a PR to add or clarify yours)

Futurist 1 year 3 year 6 year
| @openlabbott
47:15 | Discord are going to annoy their userbase. | We’ll finally get a RISC V server in a datacenter, in some shape or form. | Email goes the way of the landline.
| @MattSci2
1:10:05 | The framework laptop company will be unsuccessful. Existing laptops are not substantially different; with some retooling. | One major FPGA vendor will have a completely open toolchain for high end FPGAs. | At least 1 RISC-V supercomputer in the Top 500.
| @tomk_
1:16:45 | At least one of the hyperscalers will become startlingly good at partnering. | Stablecoins will become regulated. | The biggest datacenter server provider (outside the hyperscalers) will be a company that hasn’t yet shipped its first server.
| @tinco
1:18:57 | Multiple companies will have demonstrated a AGI (one shot machine learning system). It’s not gonna be useful for anything, but I think the problem is less hard than many critics think it is and several companies/organizations are actually going to be showing the first versions of these systems. | Drones autonomously flying around private properties will be a common thing. Factory managers, powerlines inspectors, large building sites etc. will have commonly available and affordable options to inspect or patrol their properties. | Web3 will actually happen, but not in the way it’s currently being talked about. In 6 years time bots will have improved to the point that they can not be warded off the major platforms (or any platforms) and will make the web absolutely unusable due to them disrupting all established crowd funded moderation systems. A new paradigm will have to emerge that fundamentally changes how we use the web (thus web3), so that we can still derive value from it.
| Ben Stoltz
1:24:40 | Smart glasses become a viable alternative for computer monitors youtube. People who used to look away from their phones to have their own thoughts, and are now using smart glasses in real life situations, are subjected to an ads vs. attention “Tragedy of the commons”. As costs per unit decrease leading to ubiquity, this forces a modern-day “Highway Beautification Act” to legislate Ad Blocking. | A significant percentage of commercial office space will be converted to housing. | The best AIs have emotional problems. We don’t really know how they work. AI specialists are more therapists than programmers.
| @kelseyhightower
1:29:30 | This year will be more of the same, competition to define the new normal as the pandemic winds down. | Pandemic-era solutions will backfire; crypto-currencies will give governments an excuse to track all actual spending. “We will give you the transparency, but not the kind you wanted.” | Technology will be recognized as sovereignty like money and land used to be. Governments will be wary of using technology from weak allies or competitors. Local hardware manufacturing, growth of local university training, etc. Possibly manifesting as national protectionism, or a reprise of the space-race. Open source will be the default model.
| @orangecms
1:53:45 | a major OS from China emerges | high performance computing from Europe | ARM no longer as relevant
| @ahl
1:58:00 | web3 is done; we’re not talking about it, it’s not a thing, we don’t use the term and we only vaguely recall what it was supposed to mean. | Productivity per watt becomes a highly important metric in computing. Tools tell us about our power use. We spin workloads up and down depending on power cost and availability. | AWS offers RISC-V instance types.
| @AaronDGoldman
1:07:14 | Single-node computing: people will realize that that distributed computing has a lot of overhead and that one server can do a lot of work. This will lead people to people doing business analytics jobs by pulling all their data to a single a computer and doing the calculation, getting the result 100x faster than splitting data over many computers. | Microservices inlining: taking a lot of microservices and statically linking them together. This will enable calling functions without network overhead, making things run 100x faster. | We will start do scaling properly. Instead of thinking “how can I make this big data and scale up to infinity”, we will try to get the most out of single node. Only once a single node has been pushed to its limit will we scale up to first a rack, then a datacenter, and then the world.
| @dancrossnyc
2:01:10 | Major workplace changes due to the pandemic will amplify and accentuate the wealth gap and disparity. Only some industries are privileged enough to be able to work from home. This will create social problems. | Regulation of social media in the aftermath of widespread political unrest, particularly after the US 2024 political season. | The effects of climate change will be sufficiently apparent that people will get serious about retooling around compute and power efficiency.
| @iangrunert
56:06 | No one year prediction. | CCPA copycat laws in other states, perhaps US federal legislation, plus changing global regulatory environment lead to GDPR-like protections to no longer be geo-fenced by bigger players. This’ll also have impacts on SaaS adoption - spreading data around makes right to amendment and right to deletion harder. | RISC-V chip in mainstream phone (likely Samsung). Previously moving target, but longer upgrade times and slower pace of improvements will cause Samsung to chase RISC-V for high volume phones due to better unit economics. Will have prior experience in RISC-V fab for other applications. <...

The Pragmatism of Hubris

14 december 2021 | 124 min

Oxide and Friends Twitter Space: December 13th, 2021

The Pragmatism of Hubris

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for December 13th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on December 13th included special guests Cliff Biffle and Steve Klabnik as well as Laura Abbott, Rick Altherr, James Tucker, Simeon Miteff and MattSci. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Hubris and Humility context tweet
Cliff’s written version of his Hubris talk
Hubris Fervently Anticipated Questions FAQ
[@8:07](https://youtu.be/cypmufnPfLw?t=487) Prehistory of Hubris, Cliff’s story
Project Loon wiki
[@14:23](https://youtu.be/cypmufnPfLw?t=863) Did Cliff know what he wanted to build at Oxide?
Tock embedded OS
QNX Unix-like real-time OS
[@17:55](https://youtu.be/cypmufnPfLw?t=1075) Laura on evaluating existing OS options
[@22:03](https://youtu.be/cypmufnPfLw?t=1323) Alignment of values and goals with other projects
- Bryan’s 2017 Platform as a Reflection of Values video ~30mins
[@25:00](https://youtu.be/cypmufnPfLw?t=1500) Steve: convincing low-level people that they are allowed to have nice things
RISC-V ROPI/RWPI Specification (Embedded PIC)
- Position-independent code wiki
[@28:59](https://youtu.be/cypmufnPfLw?t=1739) Secure FPGAs?
Laura Abbott’s Exploiting Undocumented Hardware Blocks in the LPC55S69 write-up
- And DEF CON talk with Rick Altherr
[@32:20](https://youtu.be/cypmufnPfLw?t=1940) Early implementation, journal club
Jonathan Shapiro 2003 Vulnerabilities in synchronous IPC designs paper
Heiser and Elphinstone’s L4 Microkernels: The Lessons from 20 Years of Research and Deployment paper
[@37:20](https://youtu.be/cypmufnPfLw?t=2240) Microkernels. Mach
- L4 microkernel family wiki
- Jochen Liedtke
- Bryan decides not to go to graduate school
- Fuchsia OS
[@51:09](https://youtu.be/cypmufnPfLw?t=3069) Origin of Humility. Debugging
- Tockilator
- Semihosting
[@1:03:15](https://youtu.be/cypmufnPfLw?t=3795) Archive files, self-descriptive binaries, debugging
[@1:10:33](https://youtu.be/cypmufnPfLw?t=4233) CORRECTION Windows does have a package manager: Windows Package Manager was released May 13, 2020
[@1:14:15](https://youtu.be/cypmufnPfLw?t=4455) Build tools and build systems
- cargo xtask
[@1:18:59](https://youtu.be/cypmufnPfLw?t=4739) DWARF
- Ada language
[@1:25:01](https://youtu.be/cypmufnPfLw?t=5101) Tock: Rust kernel, C userspace
- IDL
- Ozymandias
[@1:32:28](https://youtu.be/cypmufnPfLw?t=5548) build.rs build scripts
- Simeon’s story, code generation
- Software-hardware codesign
- [@1:52:14](https://youtu.be/cypmufnPfLw?t=6734) Conway’s law
[@1:54:30](https://youtu.be/cypmufnPfLw?t=6870) Diagnosing problems, failing tasks, formatting error messages
Joe Rozner and Rick Altherr getting Hubris and Humility running on a STM32, tweet from Dec 1, and video ~2hrs

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Tales from the Bringup Lab

7 december 2021 | 102 min

Oxide and Friends Twitter Space: December 6th, 2021

Tales from the Bringup Lab

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for December 6th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on December 6th included special guests Nathanael Huffman, Eric Aasen, as well as Rick Altherr, MattSci, Dan Cross and Steve Tuck. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@5:57](https://youtu.be/lhji-kP3Lhk?t=357) Lay of the land
[@6:58](https://youtu.be/lhji-kP3Lhk?t=418) Power
[@11:14](https://youtu.be/lhji-kP3Lhk?t=674) Matt: what goes in the middle of the board?
[@14:32](https://youtu.be/lhji-kP3Lhk?t=872) iCE40 FPGA
[@21:20](https://youtu.be/lhji-kP3Lhk?t=1280) Taking meticulous notes
[@25:41](https://youtu.be/lhji-kP3Lhk?t=1541) Power-on sequencing
- Using service processor flash to store FPGA bitstream
- Solder rework
- include_bytes
[@32:37](https://youtu.be/lhji-kP3Lhk?t=1957) “Zombie board”
- Flying probe video ~2mins
- Thermal cameras
[@46:41](https://youtu.be/lhji-kP3Lhk?t=2801) Main chip power-on
- Level shifters, I2C
- Googly Eye of Sauron
- [@55:24](https://youtu.be/lhji-kP3Lhk?t=3324) SPI wiggles (Serial Peripheral Interface)
  - Precious cargo in a rented minivan
- [@1:02:00](https://youtu.be/lhji-kP3Lhk?t=3720) Value of record keeping
- Power management
[@1:09:49](https://youtu.be/lhji-kP3Lhk?t=3720) “Valley of despair”, infinite reset loop
- SP3 socket
- Magnet wire connecting to a pin, see picture with dime for scale > Book on ENIAC quote: when things wouldn’t work, frustrated workers > referred to the machine as the MANIAC.
[@1:24:10](https://youtu.be/lhji-kP3Lhk?t=5050) Eric’s big breakthrough > Boom! SPI wiggles
[@1:30:59](https://youtu.be/lhji-kP3Lhk?t=5459) “The next day we had a demo!”
- Yet another hurdle..
- DuPont wire
[@1:39:39](https://youtu.be/lhji-kP3Lhk?t=5979) “These are the stories that don’t get told..”

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

The Sidecar Switch

30 november 2021 | 75 min

Oxide and Friends Twitter Space: November 29th, 2021

The Sidecar Switch

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 29th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Arjen Roodselaar; other speakers on November 29th included Rick Altherr, Simeon Miteff, MattSci, Jason Ozolins, Thomas and Edwin Peer. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@3:04](https://youtu.be/yl24yHlLRy0?t=184) Arjen’s announcement about the rack switch
Cadence Allegro PCB editor
[@11:35](https://youtu.be/yl24yHlLRy0?t=695) Should we do our own switch?
- “We’re just going to tweak existing designs…”
Intel Tofino 2 page
- Barefoot Networks wiki
- P4 language wiki
[@24:07](https://youtu.be/yl24yHlLRy0?t=1447) What makes this chip a beast?
[@33:24](https://youtu.be/yl24yHlLRy0?t=2004) Cable backplane, sleds
[@37:11](https://youtu.be/yl24yHlLRy0?t=2231) Sidecar
[@38:52](https://youtu.be/yl24yHlLRy0?t=2332) Management network (out of band)
- NC-SI network controller sideband interface wiki > Rick: A lot of the BMC style management functionality just > kinda got tacked on to PC systems.
[@48:36](https://youtu.be/yl24yHlLRy0?t=2916) SDN software-defined networking wiki
- NCI National Computational Infrastructure (Australia) wiki
- Network function virtualization wiki
[@55:12](https://youtu.be/yl24yHlLRy0?t=3312) The tofino simulator
[@59:51](https://youtu.be/yl24yHlLRy0?t=3591) Trust model, root of trust, service processor
[@1:02:31](https://youtu.be/yl24yHlLRy0?t=3751) Can the switch run independent of the PCIe host?
[@1:08:35](https://youtu.be/yl24yHlLRy0?t=4115) The journey. The time scale of these signaling components. Heat sinks and practice boards
Happy Hanukkah!

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Talking Turkeys

23 november 2021 | 79 min

Oxide and Friends Twitter Space: November 22nd, 2021

Talking Turkeys

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 22nd, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on November 22nd included Rick Altherr, Ian, Simeon Miteff, MattSci, Nahum Shalman, Jason Ozolins, pgray, Bill Blum, Matt Ranney, Matt Campbell, FesterCluck, Rahul Saxena and Bartz the Man. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@4:26](https://youtu.be/U10SuAHV8kQ?t=266) Thanksgiving
[@6:13](https://youtu.be/U10SuAHV8kQ?t=373) David Tolnay
- twitter and github
- Projects
London hip hop musician Loyle Carner
[@8:16](https://youtu.be/U10SuAHV8kQ?t=496) Adam is thankful for:
- ANTLR parser generator
- pest
  - usdt DTrace probes
[@11:35](https://youtu.be/U10SuAHV8kQ?t=695) Bryan is thankful for:
- build.rs Rust build scripts
- Saleae logic analyzers
[@16:33](https://youtu.be/U10SuAHV8kQ?t=993) Ian: YubiKey
[@19:09](https://youtu.be/U10SuAHV8kQ?t=1149) Matt Campbell: open source, Python accessibility Windows libraries from Chapel Hill
[@23:52](https://youtu.be/U10SuAHV8kQ?t=1432) FesterCluck: Nodejs
[@26:03](https://youtu.be/U10SuAHV8kQ?t=1563) Patrick: RabbitMQ
[@28:19](https://youtu.be/U10SuAHV8kQ?t=1699) Nahum: WireGuard and Tailscale
[@32:04](https://youtu.be/U10SuAHV8kQ?t=1924) Jason: truss by Roger Faulkner
[@37:37](https://youtu.be/U10SuAHV8kQ?t=2257) Rahul: tldp.org Linux documentation
[@42:11](https://youtu.be/U10SuAHV8kQ?t=2531) Simeon: sigrok, PulseView, Anyhow, thiserror
[@44:35](https://github.com/dtolnay/thiserror) Adam: QMK, Magic Lantern by Trammell Hudson (twitter)
[@47:36](https://youtu.be/U10SuAHV8kQ?t=2856) Matt: eBPF, (wiki)
[@54:59](https://youtu.be/U10SuAHV8kQ?t=3299) MattSci: CUDA, Ethernet
- GPS
- John Bloom (2016) Eccentric Orbits book
- Differential GPS
- BeiDou Chinese satellites, GLONASS Russian satellites, and Galileo European Union satellites
[@1:09:20](https://youtu.be/U10SuAHV8kQ?t=4160) Bartz: grep
[@1:10:30](https://youtu.be/U10SuAHV8kQ?t=4230) Rick: Ghidra reverse engineering tool
- Interactive Disassembler IDA
[@1:12:28](https://youtu.be/U10SuAHV8kQ?t=4348) Bill: Fastest Fourier Transform in the West FFTW, and gnuplot > I’m thankful that everywhere I look there’s always something that hits my > sense of wonder. That’s the thing I love about working in this industry.
Adam appreciates spreadsheets as tools for analysis

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

The Wrath of Kahn

16 november 2021 | 59 min

Oxide and Friends Twitter Space: November 15th, 2021

The Wrath of Kahn

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 15th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on November 15th included Dan Cross, Tom Lyon, Antranig Vartanian, Mat Trudel, Gabe Rudy, Simeon Miteff and bch. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Severo Ornstein (2002) Computing in the Middle Ages: A View from the Trenches 1955-1983 book
- TX-2 computer in 1958
- LINC Laboratory INstrument Computer in 1962
- Wesley Clark
- IMP
[@6:21](https://youtu.be/oft5i5RzIC8?t=381) Quote on paternity of ARPANET and the Internet
[@7:51](https://youtu.be/oft5i5RzIC8?t=471) Bryan meets Knuth… briefly
- SOAP
[@20:00](https://youtu.be/oft5i5RzIC8?t=1200) Quote from oral history of Bob Taylor (2008)
[@21:37](https://youtu.be/oft5i5RzIC8?t=1297) Dan meets Knuth?
[@25:23](https://youtu.be/oft5i5RzIC8?t=1523) The lone inventor
[@26:40](https://youtu.be/oft5i5RzIC8?t=1600) The patent race with Alexander Graham Bell and Elisha Gray (wiki)
- “Inventor” of email
[@30:49](https://youtu.be/oft5i5RzIC8?t=1849) Fathering and parenting (pioneers and settlers)
Any lone inventors?
Credit where credit is due. Teams as more than the sum of the parts.
- Turing Awards
[@35:49](https://youtu.be/oft5i5RzIC8?t=2149) Science papers, teams
[@37:14](https://youtu.be/oft5i5RzIC8?t=2234) Andy van Dam (wiki)
- “Hypertext ’87 Keynote” address
- “Reflections on a Half Century of Hypertext” (2019) ~100mins presentation
Ron Minnich (On the Metal podcast)
[@39:11](https://youtu.be/oft5i5RzIC8?t=2351) Dennis Klatt and DECtalk
- DECtalk DTC01 used a 68000 and a TI 32010 DSP; DECtalk DTC03 used a 80186 and the same TI 32010. mame
Doug Engelbart (wiki)
[@44:37](https://youtu.be/oft5i5RzIC8?t=2677) Who’s going to lead the charge?
- Michael Stonebraker (wiki)
- Seeing things through
[@49:23](https://youtu.be/oft5i5RzIC8?t=2963) bch: communications and crediting
[@50:53](https://youtu.be/oft5i5RzIC8?t=3053) DTrace, ZFS
[@53:15](https://youtu.be/oft5i5RzIC8?t=3195) Mat: The Dream Machine
- M. Mitchell Waldrop (2001) “The Dream Machine: JCR Licklider and the Revolution that Made Computing Personal” book
- DARPA, private public research funding
[@56:57](https://youtu.be/oft5i5RzIC8?t=3417) The hero narrative sells well

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Supercomputers, Cray, and How Sun Picked SGI's Pocket

9 november 2021 | 92 min

Oxide and Friends Twitter Space: November 8th, 2021

Supercomputers, Cray, and How Sun Picked SGI’s Pocket

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 8th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on November 8th included Tom Lyon, Shahin Khan, Darryl Ramm, Dan Cross, Courtney Malone, MattSci, Aaron Goldman, Simeon Miteff, and Jason Ozolins. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Bryan’s tweet about George Brown’s recommending “The Supermen”
Charles Murray (1997) “The Supermen: The story of Seymour Cray and the Technical Wizards Behind the Supercomputer” book
[@1:28](https://youtu.be/y07PyBrrzMw?t=88) Tom’s story meeting Boris
- Tom’s tweet on meeting Boris Babayan
- Elbrus computers
[@9:27](https://youtu.be/y07PyBrrzMw?t=567) Supercomputers and power
[@15:16](https://youtu.be/y07PyBrrzMw?t=916) Cray designs
- Engineering Research Associates wiki
- Control Data Corporation wiki, CDC 1604
[@20:36](https://youtu.be/y07PyBrrzMw?t=1236) ETA Systems wiki
[@23:57](https://youtu.be/y07PyBrrzMw?t=1437) On to the next big thing
- Steve Chen
- Cray X-MP
[@29:37](https://youtu.be/y07PyBrrzMw?t=1777) Super computers as one-offs
- National Computational Infrastructure in Australia, NCI
- Gallium arsenide
- GPGPU
[@33:47](https://youtu.be/y07PyBrrzMw?t=2027) Shahin on interconnects
- Jason on failure caused by a storm
- Cray C90
[@41:06](https://youtu.be/y07PyBrrzMw?t=2466) Courtney on bespoke toolchains and systems
[@42:42](https://youtu.be/y07PyBrrzMw?t=2562) Influence of Cray on Sun
- 1996 Sun to purchase Cray Business Systems Division, hpcwire
- Floating Point Systems Inc wiki > Shahin: SGI really had no use for this system. They should have just killed it.
[@50:10](https://youtu.be/y07PyBrrzMw?t=3010) Origin story of DTrace (2006 article)
- E10k
[@56:14](https://youtu.be/y07PyBrrzMw?t=3374) Thinking Machines Corp, wiki
[@57:36](https://youtu.be/y07PyBrrzMw?t=3456) Seymour Cray
- Les Davis “The ultimate team player” write up
- 2010 Oral history of Les Davis pdf
[@1:00:08](https://youtu.be/y07PyBrrzMw?t=3608) Business Systems Division history, long road to Starfire
[@1:04:20](https://youtu.be/y07PyBrrzMw?t=3860) SGI and Sun early history
- Non-uniform memory access NUMA
[@1:10:40](https://youtu.be/y07PyBrrzMw?t=4240) Cray T3E
- Massively parallel MPP
[@1:12:33](https://youtu.be/y07PyBrrzMw?t=4353) E10k stories
- boo.com wiki
[@1:18:37](https://youtu.be/y07PyBrrzMw?t=4717) Cray, spooks, pop count
[@1:20:45](https://youtu.be/y07PyBrrzMw?t=4845) Chen
- Cray X-MP and Y-MP
- Sequent
[@1:24:04](https://youtu.be/y07PyBrrzMw?t=5044) An engineer sees his defunct machine being scrapped
[@1:26:27](https://youtu.be/y07PyBrrzMw?t=5187) Jason’s story of capacitors popping off the board
- The Capacitor plague

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

On Code Review

2 november 2021 | 91 min

Oxide and Friends Twitter Space: November 1st, 2021

On Code Review

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for November 1st, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on November 1st included Kendall Morgan, Edwin Peer, Ryan Zezeski, Ian, Joshua Hoeflich, ZK Miyavi, Jason Ozolins, Nick Sherron and Austin Wise. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Context tweet
Kendall Morgan (2021) “Thoughts on Code Review” essay
[@3:57](https://youtu.be/JZdXDyeSvtc?t=237) Adam’s story, first code review at Sun
[@6:32](https://youtu.be/JZdXDyeSvtc?t=392) Choosing a reviewer
[@9:43](https://youtu.be/JZdXDyeSvtc?t=583) Unblocking others. Empathy in feedback. Asking questions, learning.
[@15:43](https://youtu.be/JZdXDyeSvtc?t=943) Bryan reviewing Jeff Bonwick’s code at Sun
- Odd working hours
- Screaming Red Chairs
[@19:47](https://youtu.be/JZdXDyeSvtc?t=1187) In-person code review vs digitized. Tools
[@24:29](https://youtu.be/JZdXDyeSvtc?t=1469) Not just finding bugs. Darin’s Law
[@25:59](https://youtu.be/JZdXDyeSvtc?t=1559) Adam’s story around a bug in a big diff, tracepoints in the kernel
[@32:28](https://youtu.be/JZdXDyeSvtc?t=1948) Adam’s favorite useless code review comment
- Marginally useful changes, what to do with multiple good alternatives
- Matters of style and taste > Joe Kowalski: Is there a problem with this code, or is it not > implemented the way you would implement it?
[@38:41](https://youtu.be/JZdXDyeSvtc?t=2321) Ian on tools. Different languages, mediums. loom for short video messages
[@44:37](https://youtu.be/JZdXDyeSvtc?t=2677) Tools designed for specific tasks.
- Gerrit
- Code review policies
[@49:31](https://youtu.be/JZdXDyeSvtc?t=2971) Jason’s story about HPE project with SCSI bug. Patch submitted to kernel group
[@54:59](https://youtu.be/JZdXDyeSvtc?t=3299) Bryan’s story about an n^3 algorithm in SCSI target code
[@56:55](https://youtu.be/JZdXDyeSvtc?t=3415) Rust compiler, resource awareness, error paths
- Often more modular than C code
- rust-analyzer, seeing inferred types
[@1:01:15](https://youtu.be/JZdXDyeSvtc?t=3675) Joshua’s experience with in-person reviews, whiteboarding
- Working arm-in-arm with people
- Sourcegraph Dev Tool Time videos
[@1:05:21](https://youtu.be/JZdXDyeSvtc?t=3921) How do you scale quality code review in bigger teams?
- Culture of code review at a company
[@1:07:15](https://youtu.be/JZdXDyeSvtc?t=4035) How to convince your team of the value of code review?
- Review can catch bugs
- Cross team knowledge, bus factor
- Speed in the short term vs speed in the long term
[@1:14:39](https://youtu.be/JZdXDyeSvtc?t=4479) Ian on cultivating organizational review practices
[@1:16:32](https://youtu.be/JZdXDyeSvtc?t=4592) Austin’s story on assuaging management fears around new practices
- Joshua: communication, writing, and accountability
- What code don’t we review?
Code review as quality check
[@1:23:55](https://youtu.be/JZdXDyeSvtc?t=5035) Engineering product quality, not always obviously of benefit to the business
- Skipping code reviews to show quality consequences
Adopting code review practices, incrementally

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Coder's Block

26 oktober 2021 | 81 min

Oxide and Friends Twitter Space: October 25th, 2021

Coder’s Block

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 25th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on October 25th included Brigid Gaffikin, Tom Lyon, MattSci, Simeon Miteff, Edwin Peer, Ian, Nima Johari, Matt Campbell, Joshua Hoeflich, Bill, Ariel Machado, and Kendall Morgan. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

BattleTris stories
[@10:15](https://youtu.be/QGs5hlH6cLk?t=615) Writer’s block, flow (instigating tweet)
National Novel Writing Month NaNoWriMo
Flow wiki
[@16:54](https://youtu.be/QGs5hlH6cLk?t=1014) “If you’re just problem solving, you can’t have writers block”
- Many degrees of freedom
- Shiny new object
[@20:39](https://youtu.be/QGs5hlH6cLk?t=1239) Remedies for writer’s block?
- Decide if you’re looking for tactics or strategy; is it small technical issues or not?
- Tactics: Hone in on ‘the craft’ – work on the language
- Strategy: Is this going to reach an audience/get an agent?
- Write a scene from a different character’s PoV; write a vignette
- This sounds like prototyping in software
- If you’re stuck on debugging, write some debug infrastructure
[@24:16](https://youtu.be/QGs5hlH6cLk?t=1456) Doing something else entirely
- Brigid: ceramics, sound walks
[@27:43](https://youtu.be/QGs5hlH6cLk?t=1663) Not everything is burnout
[@34:13](https://youtu.be/QGs5hlH6cLk?t=2053) Software analogies to writer’s techniques
[@36:04](https://youtu.be/QGs5hlH6cLk?t=2164) Personal productivity obsession
- Writer Emergency Pack by John August, site
- “You’ve got to get back to the coal face. You’ve got to finish it.”
[@41:00](https://youtu.be/QGs5hlH6cLk?t=2460) Does Rust make this indecision worse?
- Pressure to find the “right” way
[@43:56](https://youtu.be/QGs5hlH6cLk?t=2636) Arthur Whitney (wiki) > The best analog for software is poetry
Pandemic life, collaboration and conferences
[@51:51](https://youtu.be/QGs5hlH6cLk?t=3111) Hallway track. Software is collaborative but ultimately programming is a solitary act
- Nimo’s experience, it’s all collaborative. Code review, art
[@59:36](https://youtu.be/QGs5hlH6cLk?t=3576) Cliff code reviews, how to do good reviews
- Lack of code reviewers for Rust at Google
[@1:04:16](https://youtu.be/QGs5hlH6cLk?t=3856) Writer’s groups, different focuses
[@1:08:04](https://youtu.be/QGs5hlH6cLk?t=4084) Grad school during pandemic, gather.town - video chat platform for virtual interactions
[@1:11:54](https://youtu.be/QGs5hlH6cLk?t=4314) Goals, take the wins that you can, boundaries between work life and home life
Kendall Morgan “Thoughts on Code Reviews” blog post
[@1:17:38](https://youtu.be/QGs5hlH6cLk?t=4658) Bill’s experience switching things up, and enjoying computing again
Wrap up tweet

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Dijkstra's Tweetstorm

19 oktober 2021 | 87 min

Oxide and Friends Twitter Space: October 18th, 2021

Dijkstra’s Tweetstorm

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 18th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on October 18th included Edwin Peer, Dan Cross, Ryan Zezeski, Tom Lyon, Aaron Goldman, Simeon Miteff, MattSci, Nate, raycar5, night, and Drew Vogel. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Dijkstra’s 1975 “How do we tell truths that might hurt?” EWD 498 tweet > PL/1 > belongs more to the problem set than to the solution set

The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offenceAPL is a mistake, carried through to perfection. It is the language of the future for the programming techniques of the past: it creates a new generation of coding bums - [@3:08](https://youtu.be/D-Uzo7M-ioQ?t=188) Languages affect the way you think It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration. - [@4:33](https://youtu.be/D-Uzo7M-ioQ?t=273) Adam’s Perl story - The Camel Book, not to be confused with OCaml - “You needed books to learn how to do things” - CGI - [@9:04](https://youtu.be/D-Uzo7M-ioQ?t=544) Adam meets Larry Wall - [@11:59](https://youtu.be/D-Uzo7M-ioQ?t=719) Meeting Dennis Ritchie - “We were very excited; too excited some would say…” - [@15:04](https://youtu.be/D-Uzo7M-ioQ?t=904) Effects of learning languages, goals of a language, impediments to learning - Roger Hui of APL and J fame, RIP. - Accessible as a language value - Microsoft Pascal, Turbo Pascal - Scratch - LabVIEW - [@25:31](https://youtu.be/D-Uzo7M-ioQ?t=1531) Nate’s experience - Languages have different audiences - [@27:18](https://youtu.be/D-Uzo7M-ioQ?t=1638) Human languages - The Esperanto con-lang - Tonal langages - Learning new and different programming languages - [@37:06](https://youtu.be/D-Uzo7M-ioQ?t=2226) Adam’s early JavaScript (tweet) - <SCRIPT LANGUARE="JavaScript"> circa 1996 - [@44:10](https://youtu.be/D-Uzo7M-ioQ?t=2650) Learning from books, sitting down and learning by typing out examples - How do you learn to program in a language? - Zed Shaw on learning programming through spaced repetition blog - Rigid advice on how to learn - ALGOL 68, planned successor to ALGOL 60 - ALGOL 60, was, according to Tony Hoare, “An improvment on nearly all of its successors” - [@50:41](https://youtu.be/D-Uzo7M-ioQ?t=3041) Where does Rust belong in the progression of languages someone learns? Rust is what happens when you’ve got 25 years of experience with C++, and you remove most of the rough edges and make it safer? - “Everyone needs to learn enough C, to appreciate what it is and what it isn’t” - [@52:45](https://youtu.be/D-Uzo7M-ioQ?t=3165) “I wish I had learned Rust instead of C++” - [@53:35](https://youtu.be/D-Uzo7M-ioQ?t=3215) Adam: Brown revisits intro curriculum, teaching Scheme, ML, then Java - Adam learning Rust back in 2015 (tweet) “First Rust Program Pain (So you can avoid it…)” Tom: There’s a tension in learning between the people who hate magic and want to know how everything works in great detail, versus the people who just want to see something useful done. It’s hard to satisfy both. - [@1:00:02](https://youtu.be/D-Uzo7M-ioQ?t=3602) Bryan coming to Rust - “Learn Rust with entirely too many linked lists” guide - Rob Pike interview Its concurrency is rooted in CSP, but evolved through a series of languages done at Bell Labs in the 1980s and 1990s, such as Newsqueak, Alef, and Limbo. - [@1:03:01](https://youtu.be/D-Uzo7M-ioQ?t=3781) Debugging Erlang processes. Ryan on runtime v. language - Tuning runtimes. Go and Rust - [@1:06:42](https://youtu.be/D-Uzo7M-ioQ?t=4002) Rust is its own build system - Bryan’s 2018 “Falling in love with Rust” post - Lisp macros, Clean, Logo, Scratch - [@1:11:27](https://youtu.be/D-Uzo7M-ioQ?t=4287) The use of anthropomorphic terminology when dealing with computing systems is a symptom of professional immaturity. - [@1:12:09](https://youtu.be/D-Uzo7M-ioQ?t=4329) Oxide bringup updates - I2C Inter-Integrated Circuit - SPI Serial Peripheral Interface - iCE40

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Economics and Open Source

5 oktober 2021 | 99 min

Oxide and Friends Twitter Space: October 4th, 2021

Economics and Open Source

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 4th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on October 4th included Edwin Peer, James Todd, Peter Corless, Matt Campbell, jasonbking, Simeon Miteff, Josh Clulow, Ian, Joe Thompson, Dan Cross, Tom Lyon, Tim Burnham, and vint serp. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Mark Jones Lorenzo (2017) Endless Loop: The History of the BASIC Programming Language book
- John Kemeny wiki
[@3:11](https://youtu.be/JDd8xGSP9DA?t=191) Tim’s excellent tweet
- William Gibson wiki
- John Browne (1996) The Bug Count Also Rises short story
[@5:38](https://youtu.be/JDd8xGSP9DA?t=338) Growing up with BASIC
[@8:03](https://youtu.be/JDd8xGSP9DA?t=483) Braille ’n Speak PDA (intro video), BASIC programming
TI-BASIC language
[@10:39](https://youtu.be/JDd8xGSP9DA?t=639) Speaking program reading off system calls in real time
- snoop could output to /dev/audio
[@13:39](https://youtu.be/JDd8xGSP9DA?t=819) Joel Spolsky (2002) Strategy Letter V blog
- Bryan’s (2004) The Economics of Software blog
- Software “maintenance”
[@20:02](https://youtu.be/JDd8xGSP9DA?t=1202) Cathedral and the Bazaar, wiki
- “Forkophilic” development model and the Alan Cox -ac Linux tree
[@26:07](https://youtu.be/JDd8xGSP9DA?t=1567) Open source as something in the commercial best interest of a business
- SCO v IBM wiki
- Halloween documents wiki
- Steve Ballmer’s “Linux is a cancer” quote in the Chicago Sun-Times
- OpenOffice.org wiki (open sourced from StarOffice)
[@30:29](https://youtu.be/JDd8xGSP9DA?t=1829) Document editing as a service. Services and open source
Richard Stallman on SaaS
[@33:34](https://youtu.be/JDd8xGSP9DA?t=2014) The Joel Test link
- Joel’s (2007) Strategy Letter VI blog
- “Everybody wants to be a platform”
[@38:58](https://youtu.be/JDd8xGSP9DA?t=2338) Joel’s take on Sun
- Making the pie larger. Porting NFS to rival platforms
- The Sun Network Filesystem: Design, Implementation and Experience has a section on porting experiences.
- Monetizing software - “Sun could never monetize software, only hardware”
[@44:44](https://youtu.be/JDd8xGSP9DA?t=2684) Window toolkits, “cross platform”, write once run anywhere
“Write once, debug everywhere”
What’s the directory separator on MVS? or Stratos VOS?
[@51:40](https://youtu.be/JDd8xGSP9DA?t=3100) James’ experience working on Tomcat
- Joel’s (2002) Lord Palmerston on Programming blog
- Graphics toolkits, Electron/Web vs Native
[@1:05:21](https://youtu.be/JDd8xGSP9DA?t=3921) “OpenSolaris downloads are potential buyers for the ZFS appliance”
[@1:06:17](https://youtu.be/JDd8xGSP9DA?t=3977) Jason Hoffman “The Sun does not shine on me”
- Strategy cannot make up for poor execution
- Sun CEO Jonathan Schwartz didn’t travel to meet customers
- Demoing to a hostile audience
- “Asteroid named Linux on a collision course” tweet
[@1:13:20](https://youtu.be/JDd8xGSP9DA?t=4400) Open-core, AWS services, monetizing open source
- “People will pay for a service”
- Could Apple open source?
[@1:18:43](https://youtu.be/JDd8xGSP9DA?t=4723) Packaged solutions; giving mom a linux box. Free software: free for whom?
- Support relationships. People want support
[@1:22:05](https://youtu.be/JDd8xGSP9DA?t=4925) Why didn’t Sun embrace Linux?
- ZFS on Linux, Ubuntu
- The Sourceware Operating System Proposal – Larry McVoy’s open source SunOS 4 proposal.
- Sun bought Cobalt wiki
[@1:25:33](https://youtu.be/JDd8xGSP9DA?t=5133) “The writing was on the wall for Sun..”
- x86 price-performance
- “Couldn’t you buy like 100 x86 computers for that price?”
- RISC machine in-fighting, while Intel undercuts the market
[@1:31:01](https://youtu.be/JDd8xGSP9DA?t=5461) Josh’s work on frustrating hardware configuration
[@1:33:25](https://youtu.be/JDd8xGSP9DA?t=5605) Peter’s experience as a Sun customer
- Vertical scaling, but not so much horizontal scaling
- Clusters of cheap commodity hardware outperforming big multiway boxes
- Importance of open source for big internet companies
- Traders used Sun workstations, for fast trading
[@1:38:39](ht...

The Books in the Box

28 september 2021 | 77 min

Oxide and Friends Twitter Space: September 27th, 2021

The Books in the Box

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 27th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on September 27th included Tom Lyon, Dan Cross, Antranig Vartanian Simeon Miteff Matt Campbell, Jeremy Tanner, Joshua Clulow, Ian, Tim Burnham, and Nathaniel Reindl. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Not recommended :-(
- Dave Hitz and Pat Walsh (2008) How to Castrate a Bull book
- Peter Thiel (2014) Zero to One book
[@2:45](https://youtu.be/zrZAHO89XGk?t=165) David Jacques Gerber (2015) The Inventor’s Dilemma: The Remarkable Life of H. Joseph Gerber book
[@7:21](https://youtu.be/zrZAHO89XGk?t=441) Sidney Dekker (2011) Drift into Failure: From Hunting Broken Components to Understanding Complex Systems book
[@13:08](https://youtu.be/zrZAHO89XGk?t=788) Robert Buderi (1996) The Invention that Changed the World: The Story of Radar from War to Peace book
- MIT Rad Lab Series info
- Nuclear Magnetic Resonance wiki
- Richard Rhodes (1995) Dark Sun: The Making of the Hydrogen Bomb book
- Michael Riordan and Lillian Hoddeson (1997) Crystal Fire: The Birth of the Information Age book
- Craig Canine (1995) Dream Reaper: The Story of an Old-Fashioned Inventor in the High-Tech, High-Stakes World of Modern Agriculture book
- David Fisher and Marshall Fisher (1996) Tube: The Invention of Television book
- Michael Hiltzik (2015) Big Science: Ernest Lawrence and the Invention that Launched the Military-Industrial Complex book
[@18:05](https://youtu.be/zrZAHO89XGk?t=1085) Ben Rich and Leo Janos (1994) Skunk Works: A Personal Memoir of My Years at Lockheed book
Network Software Environment
Lockheed SR-71 on display at the Sea, Air and Space Museum in NYC.
[@26:52](https://youtu.be/zrZAHO89XGk?t=1612) Brian Dear (2017) The Friendly Orange Glow: The Untold Story of the Rise of Cyberculture book
[@30:15](https://youtu.be/zrZAHO89XGk?t=1815) Randall Stross (1993) Steve Jobs and the NeXT Big Thing book
[@32:21](https://youtu.be/zrZAHO89XGk?t=1941) Christophe Lécuyer and David C. Brock (2010) Makers of the Microchip: A Documentary History of Fairchild Semiconductor book
[@33:06](https://youtu.be/zrZAHO89XGk?t=1986) Lamont Wood (2012) Datapoint: The Lost Story of the Texans Who Invented the Personal Computer Revolution book
Charles Kenney (1992) Riding the Runaway Horse: The Rise and Decline of Wang Laboratories book
- Tom’s tweet
[@34:06](https://youtu.be/zrZAHO89XGk?t=2046) Bryan’s Lost Box of Books!
Edgar H. Schein et al (2003) DEC is Dead, Long Live DEC: The Lasting Legacy of Digital Equipment Corporation book
[@36:56](https://youtu.be/zrZAHO89XGk?t=2216) Alan Payne (2021) Built to Fail: The Inside Story of Blockbuster’s Inevitable Bust book
- Videotape format war wiki
Hackers (1995) movie. Watch the trailer ~2mins
Steven Levy (1984) Hackers: Heroes of the Computer Revolution book
[@42:32](https://youtu.be/zrZAHO89XGk?t=2552) Paul Halmos (1985) I Want to be a Mathematician: An Automathography book
Paul Hoffman (1998) The Man Who Loved Only Numbers about Paul Erdős book
1981 text adventure game for the Apple II by Sierra On-Line, “Softporn Adventure” (wiki)
[@49:16](https://youtu.be/zrZAHO89XGk?t=2956) Douglas Engelbart The Mother of All Demos wiki
- John Markoff (2005) What the Dormouse Said: How the Sixties Counterculture Shaped the Personal Computer Industry book
Katie Hafner and Matthew Lyon (1998) Where Wizards Stay Up Late book
1972 Computer Networks: The Heralds of Resource Sharing documentary ~26mins (wiki) included big names like Corbató, Licklider and Bob Kahn.
Gordon Moore (1965) Cramming more components onto integrated circuits paper and Moore’s Law wiki
[@52:3...

Theranos, Silicon Valley, and the March Madness of Tech Fraud

21 september 2021 | 73 min

Oxide and Friends Twitter Space: September 20th, 2021

Theranos, Silicon Valley, and the March Madness of Tech Fraud

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 20th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on September 20th included Land Belenky, Toasterson, Cole Frederick, and Simeon Miteff. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

John Carreyrou on Theranos
- “Bad Blood: Secrets and Lies in a Silicon Valley Startup” 2018 book
- “Bad Blood the Final Chapter” podcast as the trial proceeds (announcement), on apple, spotify
Cole’s tweet linking to a ~5min video of a would-be Theranos competitor commenting on its collapse > The lone inventor is a dangerous impression to give people.
Related: Brian Fitzpatrick and Ben Collins-Sussman “The Myth of the Genius Programmer” 2009 talk ~55mins
[@9:47](https://youtu.be/YWdk9CKML2g?t=587) Companies that drive scientific people nuts
- uBeam “claims to be developing a wireless charging system to work via ultrasound. Scientists have strongly criticised the plausibility under physics of this proposal.”
- uBiome > To innovate, you have to balance the world as it is with the world as it isn’t.
[@13:44](https://youtu.be/YWdk9CKML2g?t=824) Theranos’ fantastical vision. European attitudes around business and innovation.
- PCR Polymerase chain reaction invented 1983 by Kary Mullis.
[@18:39](https://youtu.be/YWdk9CKML2g?t=1119) Fake it till you make it?
- Optative voice > The secrecy of Theranos should have been a red flag
[@23:57](https://youtu.be/YWdk9CKML2g?t=1437) Whistleblower Avie Tevanian. Smoke and mirrors, giving the board the run around.
[@29:05](https://youtu.be/YWdk9CKML2g?t=1745) “Everyone was relying on someone else to do their due diligence”
- Tech risk, venture capital
- Cerebras Systems wafer scale processors
- Ellen Pao NYT editorial “The Elizabeth Holmes Trial is a Wake-up Call for Sexism in Tech”
[@35:20](https://youtu.be/YWdk9CKML2g?t=2120) Software cure-all
- 737 MAX failures
[@40:14](https://youtu.be/YWdk9CKML2g?t=2414) Founding myths
- Jean-Louis Gassée
  - 2015 “Theranos Trouble: A First Person Account” blog
  - 2018 “Theranos Could Have Been Stopped” blog
[@44:06](https://youtu.be/YWdk9CKML2g?t=2646) Tesla “Autopilot”, Uber self driving
- Anthony Levandowski > Judge Alsup: This is the biggest trade secret crime I have ever seen. > This was not small. This was massive in scale.
[@48:21](https://youtu.be/YWdk9CKML2g?t=2901) March Madness of Silicon Valley Fraudsters
- Solyndra bankrupt 2011
- Tether
[@59:02](https://youtu.be/YWdk9CKML2g?t=3542) Levandowski jeopardizes employee
- Better Place
- The Economist Obituary
- Juicero
- Flip Video bought by Cisco 2009
[@1:04:35](https://youtu.be/YWdk9CKML2g?t=3875) Warning signs of fraudulent companies
- Transparency, celebrity boards
- Optane
- Inconsistency between board and leadership on what the coming milestones are

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Docker, Inc., an Early Epitaph

14 september 2021 | 72 min

Oxide and Friends Twitter Space: September 13th, 2021

Docker, Inc., an Early Epitaph

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 13th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on September 13th included Steve Tuck, Tom Lyon, Dan Cross, Josh Clulow, Ian, Nick Gerace, Aaron Goldman, Drew Vogel, and vint serp. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Topic: Scott Carey’s article How Docker broke in half
- More by Carey on Docker:
  - Docker Desktop is no longer free for enterprise users
  - What is Docker? The spark for the container revolution
Andrej Karpathy’s tweet showing InfoWorld.com spamming ads
Carey talked to:
- Solomon Hykes (Docker cofounder with Sebastien Pahl)
- Ben Golub (Docker CEO 2013-2017)
- Craig McLuckie (Kubernetes cofounder)
- Nick Stinemates (early employee and former VP of Business Development)
[@5:21](https://youtu.be/l9LTJdT0sZ8?t=321) Akira Kurosawa’s 1950 Rashomon ~90mins. Watch a 2min trailer
Box office bomb “The Hottie and the Nottie” movie. Other stinkers: Gigli, Gotti
[@9:31](https://youtu.be/l9LTJdT0sZ8?t=571) Jerry Kaplan’s 1996 book Startup: A Silicon Valley Adventure
Steve’s take on commercialization > Bryan: There’s no question that they hit on something very big. > We saw a container as an operational vessel, but we failed to see > a container as a development vessel.
[@14:36](https://youtu.be/l9LTJdT0sZ8?t=876) dotCloud (PaaS) struggles to find a buyer; ultimately open sources as last resort > All of a sudden a company that nobody had heard of, > was a company that everybody had heard of.
They took too much money.
[@17:40](https://youtu.be/l9LTJdT0sZ8?t=1060) Pitfalls in raising money and scaling sales by imitating big companies
- HBO’s Silicon Valley
  - Clip ~1min with Jan the Man, Keith, and Doug (I’m shadowing Keith) > Everybody should be spending time arm in arm with customers understanding > how is this technology going to solve a problem > which they’ll want to pay to have a solution.
Tom: Was there actually a business anyways? Or was it just technology?
What if developers are attracted to those things they know cannot be monetized?
There was this belief that if a technology is this ubiquitous, it will be readily monetizable.
[@27:26](https://youtu.be/l9LTJdT0sZ8?t=1646) Docker Swarm and Kubernetes > Hykes: We didn’t work at Google, we didn’t go to Stanford, > we didn’t have a PhD in computer science.
Stinemates: (The Kubernetes team) had strong opinions about the need for a service level API and Docker technically had its own opinion about a single API from a simplicity standpoint. We couldn’t agree.
- DockerCon 2015: No mentioning Kubernetes!
  - Brendan Burns’ talk “The distributed system toolkit: Container patterns for modular distributed system design” was unfortunately made private by Docker sometime in the last two years. The internet archive only has this. Burns wrote a blog post about the topics from his talk.
- rkt (“Rocket”), CoreOS
[@36:11](https://youtu.be/l9LTJdT0sZ8?t=2171) Docker coming to market
- Enterprise teams wanted support
- Initial support offerings were expensive and limited (no after hours, no weekends) > Bryan: I floated to Solomon in 2014: run container management as a service.
- Rancher Labs, K3s (lightweight kubernetes)
- People care about GitHub stars (for better or worse)
[@48:02](https://youtu.be/l9LTJdT0sZ8?t=2882) Monetizing open source technologies
- Triton implementing the Docker API
- The support relationships are the foothold to figure out the product.
[@54:36](https://youtu.be/l9LTJdT0sZ8?t=3276) Venture capital going into Docker
- Docker acquires Tutum
Product market fit
- Acquisitions
[@1:04:42](https://youtu.be/l9LTJdT0sZ8?t=3882) Could the outcome have been materially different?
- Who made money on Docker? Cloud companies? Developers?
- VMware acquires Heptio
- Who invented containers?
  - BSD Jails, Plan9 namespaces?
- Tyler Tringas’ post about how small teams can create value with little outside investment, as a result of the Peace Dividend of the SaaS Wars.

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Put the OS back in OSDI

7 september 2021 | 73 min

Oxide and Friends Twitter Space: September 6th, 2021

Put the OS back in OSDI

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 6th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on September 6th included Dan Cross, Josh Clulow, Tom Lyon, Simeon Miteff, Daniel Maslowski, Matt Campbell and Moritz. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Adam’s tweets on recording Twitter Spaces.
Tweet on recovering a recording!
[@4:57](https://youtu.be/PVJfqjJJCkg?t=297) Timothy Roscoe’s Keynote
- Screenshots teasing his slides
- Conf video
Complicated relationship with academia and industry
- [@8:09](https://youtu.be/PVJfqjJJCkg?t=489) Adam’s MS graphics experience
- Bryan’s USENIX 2016 keynote ~1hr: A Wardrobe for the Emperor – Stitching Practical Bias into Systems Software Research
  - Conferences as the publishing vector for CS research
[@13:47](https://youtu.be/PVJfqjJJCkg?t=827) What a modern OS does > … accreted and not designed.
> They were not designed, they congealed.
[@17:10](https://youtu.be/PVJfqjJJCkg?t=1030) Rob Pike’s 2000 “Systems Software Research is Irrelevant” paper
- The value of incremental improvements
[@21:47](https://youtu.be/PVJfqjJJCkg?t=1307) Building on extant working components and interfaces
- Opaque, proprietary hardware
- AMD Platform Security Processor > Artifacts of the OS implementation tend to have outsized impact > on overall system performance
[@26:27](https://youtu.be/PVJfqjJJCkg?t=1587) Performance is not the only axis of a system
- Security, malleability, convenience, reliability
[@31:12](https://youtu.be/PVJfqjJJCkg?t=1872) Specialization
- HarmonyOS, Fuchsia
- Different chips performing different tasks
- Firmware everywhere
- Intel Optane
- Intel 8051
[@37:02](https://youtu.be/PVJfqjJJCkg?t=2222) Open hardware and firmware
- ARM Cortex-M0 > That’s why we land at incrementalism, we ossify at some boundary. > And it’s very hard to change things on either side without moving in lockstep.
Tom: The PC architecture was a great thing, but now the OS vendors have abdicated any knowledge of the hardware. Give us UEFI and we don’t care what happens beneath that.
- Should ARM have UEFI? (or something like it)
[@45:29](https://youtu.be/PVJfqjJJCkg?t=2729) Developing hardware is still challenging, but has never been easier than today (especially low-speed)
- Tom’s tweet about parallels with homebrew computing in the 70’s
- Precursor and Xous
[@50:58](https://youtu.be/PVJfqjJJCkg?t=3058) Where will new systems development fit in with our existing (working) systems?
- Low-speed is an opportunity area
- RISC-V for peripherals
[@56:37](https://youtu.be/PVJfqjJJCkg?t=3397) Backwards compatibility seems to be more important than marginal gains:
- Shingled magnetic recording offered <25% density gain at the cost of compatibility
- Optane: gains didn’t justify the cost
- Smart NICs only made sense in hyperscale server fleets > Josh: If you’re going to change the programming model, you have to blow the doors off on at least one axis
[@1:00:45] Moving management plane to a NIC.
- AWS Nitro implements this with a series of PCIe offload cards.
[@1:01:22](https://youtu.be/PVJfqjJJCkg?t=3682) Abstraction boundaries not designed for the current circumstances
- Coordination problems between vendors
- Vestigial components
- AMI, AST2500
- Arcane boot processes and shortcuts available for cloud compute
  - xhyve
[@1:08:57](https://youtu.be/PVJfqjJJCkg?t=4137) Removing things is so hard
- Things change given enough time
- Graham Lee’s essay on legacy and software dependencies …and in the end will be the command line
- If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

A brief history of talking computers

31 augusti 2021 | 93 min

Oxide and Friends Twitter Space: August 30th, 2021

A brief history of talking computers

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for August 30, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on August 30th included special guest Matt Campbell, as well as MattSci, TVRaman, Jessamyn West and Dan Cross. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Brian Dear’s The Friendly Orange Glow
- Brodie Lockard created amazing software on PLATO
- Control Data Corp Homework
[@2:47](https://youtu.be/b9GVJg0LRX4?t=167) Matt’s intro
- Deane Blazie created TotalTalk, a speaking terminal. See his 2004 interview.
- Apple IIe computer and the Echo II speech synthesizer card.
[@4:15](https://youtu.be/b9GVJg0LRX4?t=255) The Echo ][ sound sample
- Wargames computer: GREETINGS PROFESSOR FALKEN. Listen > SHALL WE PLAY A GAME?
  > Love to. How about Global Thermonuclear War?
  > …
  > Is this a game or is it real?
  > WHAT'S THE DIFFERENCE?
  > …
  > What’s it doing?
  > It’s learning…
  > …
  > A STRANGE GAME.
  > THE ONLY WINNING MOVE IS
  > NOT TO PLAY.
[@7:46](https://youtu.be/b9GVJg0LRX4?t=466)
- Prose 2000 sample
- DECtalk audio sample
[@12:14](https://youtu.be/b9GVJg0LRX4?t=734) Apple to PC
- Keynote Gold, Master Touch, Zoom Text
[@14:53](https://youtu.be/b9GVJg0LRX4?t=893) Keynote Gold sample
- Talking Moose. Watch a sample.
[@17:17](https://youtu.be/b9GVJg0LRX4?t=1037) GUI screen readers
- outSPOKEN used QuickDraw
- Window Bridge 1992
[@21:58](https://youtu.be/b9GVJg0LRX4?t=1318) Meeting another sight impaired person on a MUD
- pwWebSpeak
- Emacspeak
[@26:44](https://youtu.be/b9GVJg0LRX4?t=1604) Early programming experiences
- Apple IIGS
[@28:47](https://youtu.be/b9GVJg0LRX4?t=1727) Emacspeak user base
[@31:34](https://youtu.be/b9GVJg0LRX4?t=1894) Things were getting better on the Windows side..
- JAWS, patch parody sample
- Microsoft Narrator
[@36:12](https://youtu.be/b9GVJg0LRX4?t=2172) Linux
- Speakup
- Mixing multiple sound streams, hardware limitations
- Slackware
- ZipSpeak by Matthew Campbell
[@44:53](https://youtu.be/b9GVJg0LRX4?t=2693) Editors for the visually impaired?
- ed text editor
- Edbrowse
[@49:36](https://youtu.be/b9GVJg0LRX4?t=2976) Working on accessibility (a11y) for pay
- FreedomBox
- GNOME EsounD
- KDE aRts
- Gnopernicus
- Orca
[@57:46](https://youtu.be/b9GVJg0LRX4?t=3466)
- Microsoft Active Accessibility
- AT-SPI
- CORBA, D-Bus
[@1:03:11](https://youtu.be/b9GVJg0LRX4?t=3791) Handheld devices
- Apple VoiceOver
- Google TalkBack
- iPhone Screen Recognition article
[@1:08:09](https://youtu.be/b9GVJg0LRX4?t=4089) What should software engineers know about accessibility?
- Use a mature UI framework!
- Microsoft UI Automation is the successor to MSAA.
AccessKit by today’s speaker Matt Campbell!
[@1:12:34](https://youtu.be/b9GVJg0LRX4?t=4354) DECtalk samples!
[@1:15:25](https://youtu.be/b9GVJg0LRX4?t=4525) One of the most important settings a blind person will want to change in their speech synthesizer is how fast it talks.
- JAWS parody clip
Alt text image captions

Topical recent conference presentation: - Emily Shea (2019) Voice Driven Development video

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

The episode formerly known as ℔

24 augusti 2021 | 66 min

Oxide and Friends Twitter Space: August 23rd, 2021

The episode formerly known as ℔

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for August 23rd, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on August 23rd included Neal Gompa, Tom Lyon, Laura Abbott, Jeremy Tanner, Matt Campbell, Simeon Miteff and others. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Last week’s recording on “Showstopper” with author G. Pascal Zachary, and Jessamyn West.
Ashton-Tate history (there never was any Ashton, and dBASE II was the first version)
- dBASE IV was “slow, buggy” and didn’t get fixed in a timely manner
- Last week, Pascal mentioned that CEO Ed Esber “in a fit of insanity admitted to me (a journalist) he didn’t know how to use his company’s own product!”
- Friday! personal information manager, and Sidekick from Borland (like Google calendar for DOS)
[@3:01](https://youtu.be/-ZRv6EHaQYM?t=181) Phrasing: operating program (vs operating system)
- Steve Jobs 1992 MIT Sloan talk ~72mins on consultants, hiring people and leaving Apple (see mit.edu summary) > Jobs: NeXTSTEP is not an operating system, it’s an operating environment
- July 5th recording discussing NeXT. Randall Stross book: Steve Jobs and the NeXT Big Thing (1993) > Mac OSX focused on user capabilities of the desktop environment, but they considered it one and the same with the operating system
[@7:42](https://youtu.be/-ZRv6EHaQYM?t=462) Windows NT had “multiple personalities” > Adam: I was instantly transported to the 90’s. > Bryan: I could hear Smashing Pumpkins playing on the radio.
- Sun’s Spring OS was the ne plus ultra of this approach
- Mach microkernel, GNU Hurd, Apple M1,
- Windows Subsystem for Linux WSL > Adam: Docker takes static linking to the extreme and just ships everything
[@12:40](https://youtu.be/-ZRv6EHaQYM?t=760) Microkernels > Simeon: (Oxide) is working on a microkernel for Hubis, tell us about that
- Minix, and the Tanenbaum-Torvalds 1992 microkernel vs monolithic debate
- QNX Unix-like real-time OS
  - See ACM ByteCast interview with Rashmi Mohan, Bryan tells the story ~3mins of coming to QNX after reading about it in the “Operating Systems Roundup” of Byte Magazine 1993 (see also Bryan’s blog post and remembering Dan Hildebrand)
  - L4 microkernel
  - The QNX 1.44M demo disk
    - The GUI was called Photon. > Bryan: why would we not run this (QNX) absolutely everywhere?
- Oberon OS. Photon microGUI
[@15:49](https://youtu.be/-ZRv6EHaQYM?t=949) Laura on writing a microcontroller operating system
- Cliff Biffle’s website
- Microkernels, root of trust, embedded systems
- There is very little (or no) dynamic memory allocation in Hubris.
- Tock multitasking embedded OS, and Bryan’s “Tockilator: Deducing Tock execution flows from Ibex Verilator traces” video ~12mins
- In Tock, dynamic program loading is central. Hubris functions as a security-minded service processor. The programs it will use are all known in advance; so dynamic loading (and the accompanying security concerns) can be left out.
- Fit-to-purpose OSs
[@24:19](https://youtu.be/-ZRv6EHaQYM?t=1459) ROPI/RWPI (aka “Ropy Rippy”) and the growing pains of RISC-V
- GitHub issue ROPI/RWPI Specification (Embedded PIC)
- OpenTitan, ARM Cortex-M > When we set out to write Hubris, we spent a lot of time reading > and learning what’s out there.
- QNX vs monolithic systems. QNX was robust against module failure, so bugs in modules were tolerable. At Sun, faults in a module were system faults, so bugs were unacceptable.
- Memory protection. Stack growing into (and corrupting) data segment, hard to debug.
- Stack corruption, a hit and run.
[@32:39](https://youtu.be/-ZRv6EHaQYM?t=1959) Humor: Oxide rustfmt bot is named Ozymandias
- Percy Bysshe Shelley’s “Ozymandias” poem > LOOK UPON MY REFORMATTING YE MIGHTY AND DESPAIR!
- stale bot, open source maintainers, communicating bugs and issues
[@39:54](https://youtu.be/-ZRv6EHaQYM?t=2394) Fun QNX bug story
- QNX wrote their own POSIX utilities, they wrote their own AWK
- QNX developers, incl. Peter van der Veen
[@43:00](https://youtu.be/-ZRv6EHaQYM?t=2580) How do you say…
- vi, ed > Tom: Off with their eds!
- sed, ps, kubectl, /etc/passwd, QNX (...

The Showstopper Show

17 augusti 2021 | 86 min

Oxide and Friends Twitter Space: August 16th, 2021

The Showstopper Show

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for August 16th, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on August 16th included special guests G. Pascal Zachary (see gpascalzachary.com), and Jessamyn West (see jessamyn.medium.com), as well as Dan Cross, Tom Lyon, Josh Clulow, and others. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

G. Pascal Zachary’s “Showstopper! The Breakneck Race to Create Windows NT and the Next Generation at Microsoft” book
Tracy Kidder’s “The Soul of a New Machine” book
[@0:46](https://youtu.be/hlQuF75L4TE?t=46) “The endless debate of NT vs Unix.”
- Bryan: My whole career was kind of defined by going where Windows wasn’t. I don’t know what I was expecting, but what I found was a real time capsule from software development in the 90’s.
[@2:46](https://youtu.be/hlQuF75L4TE?t=166) Jessamyn: There was some familial impact (from developing DG Eclipse) that wasn’t mentioned in the book.
- “O, Engineers!” retrospective from wired
[@6:30](https://youtu.be/hlQuF75L4TE?t=390) What was Kidder’s process? “He lived in my house!”
[@8:32](https://youtu.be/hlQuF75L4TE?t=512) Zachary interviewed family members extensively. > People couldn’t leave, they were staying at the office all the time.
[@14:23](https://youtu.be/hlQuF75L4TE?t=863) I do feel this is a time capsule. A time before two mega-trends hit: the Internet and open source.
[@17:33](https://youtu.be/hlQuF75L4TE?t=1053) Microsoft was kind of a joke software company in the early 90’s. > Dave Cutler was a force of nature.
[@19:59](https://youtu.be/hlQuF75L4TE?t=1199) No one understood why someone was good at coding. It was a mystery to everyone, why there was such a wide stratification of coders. > There were projects that never saw the light of day.
- Ashton-Tate, dBase > There was a sense from Cutler and Perazzoli, that leadership of the team, > that these guys at Microsoft really didn’t get how serious the process > of building this battleship was.
- I think the level of anguish did surprise me.
[@23:59](https://youtu.be/hlQuF75L4TE?t=1439)
In “Soul of the New Machine,” the machine was the star, and people served it. East Coast vs West Coast attitudes. > On the West Coast, the personal computer were supposed to help you > actualize your counter-cultural values.
- Ken Olsen of DEC > Computing is equivalent with IBM. There was no software industry > so long as IBM gave all the software away for free.
[@26:09](https://youtu.be/hlQuF75L4TE?t=1569) Crashes. > Wozniak dreamed of owning > his own PDP > computer, which cost as much as a house. So he was aware of the robustness > of the minicomputer, and by contrast, the puny power of a personal computer.
- Thirtysomething > Dave Cutler was not cuddly. He was menacing, he could lose his temper. > And I tried not to get to close to him physically for that reason. > There were two looming father figures in Cutler and Gates. > And I think it created a lot of anxiety.
[@29:52](https://youtu.be/hlQuF75L4TE?t=1792) The stakes for NT at Microsoft were high.
- Fred Brooks’ “The Mythical Man-Month” book > It was a watershed moment in the history of computing. > It was more like the last battleship, rather than the next frontier.
- Bryan: I didn’t realize this, that Gates was arguing against memory protection with Cutler. From our perspective, shipping an operating system without memory protection, in an era when microprocessors supported it, is malpractice.
[@33:14](https://youtu.be/hlQuF75L4TE?t=1994) Cutler’s vendetta against Unix. > Conflict was at the heart of innovation at Microsoft at that time.
- Mitch Kapor of Lotus. > These early personal computer innovators were dismissed and sometimes > humiliated by mainstream big iron people of the 60’s and 70’s.
- Bill Gates’ “The Road Ahead” book doesn’t mention the internet.
- Zachary’s “Endless Frontier: Vannevar Bush, Engineer of the American Century” book > Computers on the West Coast were seen as extensions of your creativity, > and a tool for liberation. And for a long time that dominated the horizons.
In 2005 Gates and Ballmer don’t want to do cloud computing. “Who’s gonna want to put their stuff in the cloud?” We’ve found that computing is a collective experience.
[@38:28](https://youtu.be/hlQuF75L4TE?t=2308) Email and personal messaging
- Sun Ray thin client computer
- Dennis Ritchie and Ken Thompson’s “The UNIX time-sharing system” paper > Unix was an experiment in collaboration.
- RSX-11 for the PDP-11. And VMS for the VAX. > The attitude of looking down on Unix (as undesigned, academic) is > carried forward by Microsofties today.
- Tom: You can forgive Cutler’s misgivings, because Unix pretty much stole the thunder out of VMS.
[@42:24](https://youtu.be/hlQuF75L4TE?t=2544) Interviews for the book. Family members perspective on workplace behavior.
- Betty Shanahan, Society of Women Engineers. Brief Q&A
- EAGLE (Eclipse Appreciation and Gratitude for Lonely Evenings) award > Betty’s husband got an award for having to do his own laundry…
- Jessamyn’s “Women in Early Tech” blog entry about Shanahan

Agile + 20

27 juli 2021 | 72 min

Oxide and Friends Twitter Space: July 26, 2021

Agile + 20

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for July 26, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on July 26 included Tom Lyon, Tom Killalea, Dan Cross, Aaron Goldman, and others. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Al Tenhundfeld’s Agile at 20: The Failed Rebellion
The Agile Manifesto
[@0:55](https://youtu.be/3tp5EtPdPwY?t=55) Adam’s experiences
From the Agile Manifesto history > The only concern with the term agile came from Martin Fowler > (a Brit for those who don’t know him) who allowed that > most Americans didn’t know how to pronounce the word ‘agile’.
[@6:25](https://youtu.be/3tp5EtPdPwY?t=385) > The problem with agile is when it became so prescriptive that it > lost a lot of its agility.
[@8:06](https://youtu.be/3tp5EtPdPwY?t=486) > There’s so much that is unstructured in the way we develop software, > that we are constantly seeking people to tell us how to do it. > The answer is it’s complicated.
Steve Yegge’s Good Agile, Bad Agile > So the consultants, now having lost their primary customer, were at > a bar one day, and one of them (named L. Ron Hubbard) said: > “This nickel-a-line-of-code gig is lame. You know where > the real money is at? You start your own religion.” > And that’s how both Extreme Programming and Scientology were born.
[@9:15](https://youtu.be/3tp5EtPdPwY?t=555) Edward Yourdon
- “Decline and Fall of the American Programmer” book
[@10:26](https://youtu.be/3tp5EtPdPwY?t=626) “The principles are not all wrong. Some today even feel obvious.” > There’s also a lack of specificity, which gives one lots of opportunity > for faith healers to come in.
[@14:43](https://youtu.be/3tp5EtPdPwY?t=883) “Something I found surprising about Agile was how rigid it became.”
- Dan’s perils of personal tracking methodology
- Sun’s engineers connecting directly with customers
The Agile Ceremonies. (an ultimate guide)
- Sprint Planning, Daily Stand-Up, Sprint Review, Sprint Retrospective
[@20:48](https://youtu.be/3tp5EtPdPwY?t=1248) “I think we overly enshrine schedule estimation. If there are any unknowns it becomes really hard.” > I think there’s a Heisenberg principle at work with software: > you can tell what’s in a release or when it ships, but not both.
[@23:25](https://youtu.be/3tp5EtPdPwY?t=1405) Tom Killalea talks to success stories he’s seen with Agile
- Building S3 at AWS
[@28:31](https://youtu.be/3tp5EtPdPwY?t=1711) Sprint planning and backlogs
- Big work chunks, responding to changing priorities
[@33:39](https://youtu.be/3tp5EtPdPwY?t=2019) Success or failure of an Agile team?
- “Do demos and retrospectives”
- Unknowns in software development make estimation hard
[@39:11](https://youtu.be/3tp5EtPdPwY?t=2351) Dan’s experiences
- Personal Software Process, Team software process, Software Engineering Institute > Some people really benefit from the level of rigidity that is set out > by these processes. Prior to that, they just weren’t having > these conversations with their sales team, product owners, etc.
- Construction analogies, repeatability.
- Self-anchored suspension bridge
[@46:40](https://youtu.be/3tp5EtPdPwY?t=2800) Software as both information and machine.
- Consultancies, repeatability, incremental results.
- “For each success story, there are many failures.”
- Manifesto as a compromise between different methodologies
- Silver Bullet solutions, cure-alls. See Fred Brooks’ (1987) “No Silver Bullet” paper
[@51:18](https://youtu.be/3tp5EtPdPwY?t=3078) Demos: “Working software is the primary measure of progress.”
- Experimentation and iteration
- No true Scotsman fallacy
- What does Agile even mean anymore?
- “Letting people pretend to agree while actually disagreeing, but then going off and building working software anyway.”
[@59:45](https://youtu.be/3tp5EtPdPwY?t=3585) Ed Yourdon and the Y2K problem
Maybe there are too many Agile books already.
Tom Killalea conversation with Werner Vogels
- AWS development
Agile is more like a guideline than a target to hit.
Consistent team composition over time
“Soul of a New Machine”: trust is risk
The answer can’t be “you’re doing it wrong.”
How do you know if it’s working for your team?

(Did we miss anything? PRs always welcome!)

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

NeXT, Objective-C, and contrasting histories

6 juli 2021 | 71 min

Oxide and Friends Twitter Space: July 5, 2021

NeXT, Objective-C, and contrasting histories

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for July 5, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on July 5th included Tom Lyon, Ian, bch, Theo Schlossnagle, Rick Altherr, and Nate. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

First Twitter Space, May 3rd
- the lost recording (~31mins)
- (possible?) genesis of the idea to record spaces
- Adam’s process for recording spaces
- Someone (Sid?) mentioned NeXT’s transparent compensation model
- Oxide: Compensation as a Reflection of Values
[@2:28](https://youtu.be/2H9XQBdLB0Y?t=148) Randall Stross book: Steve Jobs and the NeXT Big Thing (1993)
[@4:42](https://youtu.be/2H9XQBdLB0Y?t=282) The SPARCstation 1 and the Sun-4c (campus) architecture > The hardware was not competitive, but dammit they sure looked good!
NeXTcube
[@9:15](https://youtu.be/2H9XQBdLB0Y?t=555) It’s nuts how much time and energy they spent on the look of it. > They were building a huge factory, just about the time people were > starting to outsource everything.
Sun was doing incremental things, and Steve was going for the 100 yard pass.
- Apple Lisa computer > NeXT refused to interoperate with anything. > They had this idea that a NeXT customer is going to buy all NeXT machines.
[@13:20](https://youtu.be/2H9XQBdLB0Y?t=800) NeXT was a really proprietary company, contrasted with Sun, a really open company. > Bill Gates volunteers that he would gladly urinate on a NeXT machine.
They are attempting to reinvent absolutely everything, so they need all software to be written from scratch, effectively.
Jobs does this over and over again at NeXT. He does things to make NeXT look bigger than it is.
[@16:23](https://youtu.be/2H9XQBdLB0Y?t=983) Jobs blows off important meeting with IBM
[@18:56](https://youtu.be/2H9XQBdLB0Y?t=1136) Mathematica went whole hog on NeXT
[@20:55](https://youtu.be/2H9XQBdLB0Y?t=1255) “Steve Jobs yells at your dad a lot?”
- Quark Software Inc, QuarkXPress
[@22:22](https://youtu.be/2H9XQBdLB0Y?t=1342) Story of Jobs trying to sell NeXT machines to Brown’s CS dept > “Your product looks great, I’m just not sure your company is > going to be around for as long as we need it to be.”
> Then Steve Jobs calls him an a**hole and storms out.
[@23:35](https://youtu.be/2H9XQBdLB0Y?t=1415) NeXT spent very freely. Lavish offices, catering, etc > He did not take VC money. He had weird money from beginning to end.
> Ross Perot thought Jobs was a total genius. Then realized that whether > he was a genius or not, he wasn’t selling any computers.
The 80’s were all about fear of Japan.
Ultimately they had to pivot away from hardware.
[@26:38](https://youtu.be/2H9XQBdLB0Y?t=1598) In contrast to Sun
- Metaphor Computer Systems
- Bryan’s tweet from July 3 > Measured by most any yardstick one could choose, Sun was one of > the most successful stories of the 1980’s for all of industrial America.
- John Gage
[@32:43](https://youtu.be/2H9XQBdLB0Y?t=1963)
- the NeXTSTEP operating system, based on the Mach microkernel
- Objective-C HOPL paper
- Walter Isaacson biography on Steve Jobs
- Be Inc, computer company. Jean-Louis Gassée
- Stepstone (originally PPI) > Not that I’ve read a ton of HOPL papers, but I don’t think HOPL papers > spill the tea quite this much..
[@39:53](https://youtu.be/2H9XQBdLB0Y?t=2393) Named parameters in programming languages
- The software crisis, Object Orientation, “Software ICs”
[@44:40](https://youtu.be/2H9XQBdLB0Y?t=2680) NeXT was building real things with Objective-C, PPI wasn’t.
[@45:54](https://youtu.be/2H9XQBdLB0Y?t=2754) Rick’s experience with Objective-C at Apple
- Objective-C, Objective-C++, and Swift
[@54:08](https://youtu.be/2H9XQBdLB0Y?t=3248) Objective-C and Swift are mandated. If it were an open ecosystem, would they be significant? > There was a feeling that the hardware didn’t matter. > You shouldn’t trouble yourself with any details.
[@57:46](https://youtu.be/2H9XQBdLB0Y?t=3466) Secrecy at NeXT and Apple
- NDAs signed per project > Secrecy is a lot of work.
It was all about being able to walk on stage, and dramatically drop something that was going to be life changing.
It seems like the secrecy was being used to manipulate people.
[@1:03:13](https://youtu.be/2H9XQBdLB0Y?t=3793) x86 port at Apple
[@1:05:34](https://youtu.be/2H9XQBdLB0Y?t=3934) Jobs tells them to make it great, because it’s currently sh*t.
[@1:08:04](https://youtu.be/2H9XQBdLB0Y?t=4084) Is Objective-C being used anywhere today outside the Apple ecosystem?
- GNUstep, Agent-based modeling

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

What's a bug? What's a debugger?

22 juni 2021 | 66 min

Oxide and Friends Twitter Space: June 21, 2021

What’s a bug? What’s a debugger?

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for June 21, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on June 21st included Dan Cross, Sean Klein, Aram Hăvărneanu, and the mononymous Nate. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

Adam’s toddler (being chased by a rooster) > Don’t get me wrong, some of my best friends are three-year-olds.
[@3:12](https://youtu.be/UOucW3F7nCg?t=192) Sy Brand’s tutorial Writing a Debugger
- Lobsters – when HN isn’t enough!
[@4:34](https://youtu.be/UOucW3F7nCg?t=274) Bryan’s debuggers
- MDB Modular Debugger > Adam: I think people are using cargo-cult debugging, rather than getting to the root cause > of these things, or being satisfied until they get to the root cause.
  > Bryan: I think with software systems, it’s really hard to know what they’re actually doing.
- Procedure Linkage Table aka “the plits”
- “Runtime Performance Analysis of the M-to-N Scheduling Model” (pdf) 1996 undergrad thesis (Brown CS dept website)
- [@6:29](https://youtu.be/UOucW3F7nCg?t=389) Threadmon website and 1997 paper (a retooling of the ’96 paper) > When I built that tooling, it revealed this thing > is not doing at all what anyone thought it was doing.
- TNF Trace Normal Form > Part of the problem with debuggers… debuggers are historically written by compiler folks, > and not system folks. As a result, debuggers are designed to debug the problem that > compiler folks have the most familiarity with, and that’s a compiler.
  > Debuggers are designed for reproducible problems, way too frequently.
I view in situ breakpoint debugging as one sliver of debugging that’s useful for one particular and somewhat unusual class of bugs. That’s actually not the kind of debugger I want to use most of the time.
Software breakpoints
[@11:59](https://youtu.be/UOucW3F7nCg?t=719) > libdis was my intern project in 2000. The idea was to take the program text, > and interpret it in some structural form, and try to infer different things about the program.
- Ghidra: software reverse engineering tool
- Laura Abbott’s Exploiting Undocumented Hardware Blocks in the LPC55S69
- Volatility: the memory forensics framework Adam couldn’t quite remember.
[@14:59](https://youtu.be/UOucW3F7nCg?t=899) I meant this question earnestly, what is a debugger?
- The first bug > The term is somewhat regrettable… It implies a problem, when there may not be a problem. > It may just be I want to understand how the system is operating, independent of whether > it’s doing it badly.
Wikipedia on Observability (control theory)
Oxide’s embedded OS and companion debugger: Hubris and Humility
[@19:01](https://youtu.be/UOucW3F7nCg?t=1141) Using DTrace to help customers understand their systems. > If you strings the DTrace binary, > you’re not gonna find any mention of raincoats.
- Cliff Moon on Boundary
[@22:13](https://youtu.be/UOucW3F7nCg?t=1333) Cardinal rule of debuggers: Don’t kill the patient! (see also: Do No Harm) > Not killing the patient is really important, > this was always an Ur principle for us.
The notion that the debugger has now become load bearing in the execution of the program, is a pretty grave responsibility.
[@26:54](https://youtu.be/UOucW3F7nCg?t=1614) Post-mortem debugging > It is a tragedy of our domain that we do not debug post-mortem, routinely.
Heisenbug (when the act of observing the problem, hides the problem)
[@31:11](https://youtu.be/UOucW3F7nCg?t=1871) > What’s going on in the system? It’s not crashing, there’s no core dump. > But the system is behaving in a way I didn’t expect it to, and I want to know why.
[@33:51](https://youtu.be/UOucW3F7nCg?t=2031) Pre-production reliability techniques > All of our pre-production work has gotten way better than it was, and I think that’s > compensation for the fact we can’t understand these systems when we deploy them.
[@37:58](https://youtu.be/UOucW3F7nCg?t=2278) > The move to testing has in fact obviated some of the need for > what we consider traditional debuggers.
> (Bryan audibly cringes)
[@39:08](https://youtu.be/UOucW3F7nCg?t=2348) Automated and Algorithmic Debugging conference AADEBUG 2003
- HOPL History of Programming Languages > There was a test suite of excellence when it comes to automated program debugging. > And it was some pile of C programs with known bugs, and you would throw your new > paper at it, and it would find 84% of the bugs, and there would be a lot of > slapping each other on the back on that. Really focused on the simplest of simple bugs.
- [@43:15](https://youtu.be/UOucW3F7nCg?t=2595) Bryan’s Postmortem Object Type Identification paper > Who is my neighbor in memory? Because my neighbor just burned down my house basically.
- mdb’s ::kgrep > I need to pause you there because it’s so crazy, and I want to emphasize that > he means what he’s saying. We look for the 64 bit value, and see where we find it. > This is a game of bingo across the entire address space.
We can follow the pointers and propagate types.
[@48:49](https://youtu.be/UOucW3F7nCg?t=2929) printf/println debugging – everyone’s doing it > I think it’s a mistake for people to denigrate printf debugging. > If you’ve got a situation that you ca...

Barracuda 7200.11: broken firmware is broken software!

8 juni 2021 | 57 min

Oxide and Friends Twitter Space: June 7, 2021

Barracuda 7200.11: broken firmware is broken software!

We’ve been holding a Twitter Space weekly on Mondays at 5p (PT) for about an hour. In addition to Bryan Cantrill and Adam Leventhal, speakers included Laura Abbott, Joshua Clulow, Dan Cross, Bill Blum, Rick Altherr, Tom Lyon, and others. The recording is here.

(Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

The Seagate ST3000DM001
- class action
[@2:01](https://youtu.be/qisoAIx8EE8?t=121) Bryan and Adam’s experience
- Fishworks
- HGST
- Bryan is unable to forget SU0D > This thing damn near ruined our lives
- Broad Institute
- The Seagate Barracuda product line. 7200.10, 7200.11
[@8:10](https://youtu.be/qisoAIx8EE8?t=490) Tough customers
[@10:17](https://youtu.be/qisoAIx8EE8?t=617) Cargo cultism and bad interview questions
- What is a Good New Englander? We’re not a hugging people.
[@12:35](https://youtu.be/qisoAIx8EE8?t=755) Adam and Bryan after Sean Manaea’s 2018 no-no
The Gift of the Magi, LBA
[@15:11](https://youtu.be/qisoAIx8EE8?t=911) Adam torments the interns
[@16:41](https://youtu.be/qisoAIx8EE8?t=1001) Bill and the HP Z620s
- The Wisdom of James Mickens
[@19:21](https://youtu.be/qisoAIx8EE8?t=1161) Rick’s story
- Fast and loose firmware source control
- Western Digital’s Sparta drive, flying too low
[@25:34](https://youtu.be/qisoAIx8EE8?t=1534) Need for open source firmware (see also: Bryan explains why proprietary firmware is a problem ~3mins)
- Vendor gaslighting
[@27:48](https://youtu.be/qisoAIx8EE8?t=1668) Tom on custom firmware
- Rent seeking
- S.M.A.R.T.
- ADM-3A “dumb terminal”
[@32:08](https://youtu.be/qisoAIx8EE8?t=1928) Adam’s firmware horror story flashbacks
- HBA
- When turning it off and on again isn’t enough: unplug and replug
- Sun’s ILOM bug
- Sun’s embarrassing ticker symbol change
[@38:10](https://youtu.be/qisoAIx8EE8?t=2290) After Sun > Stay the hell away from hardware
[@39:55](https://youtu.be/qisoAIx8EE8?t=2395) Hard drive API wish list?
- Adam’s series on APFS > There is no bit rot here..
- Networking vs Storage. Intermittent, transient failure
[@44:40](https://youtu.be/qisoAIx8EE8?t=2680) Firmware as differentiator
- Heat-assisted magnetic recording (HAMR)
- Microwave-assisted magentic recording (MAMR)
- see also: Jessie’s Life of a Data Byte surveys storage media tech through history
- Amazing physics, mediocre firmware. Firmware is software
[@48:23](https://youtu.be/qisoAIx8EE8?t=2903) The only firmware that didn’t give us problems..
- Adam on flash: A File System All Its Own, Flash Storage Today, History of SSDs blog entry mentioning sTec and Gnutek
- sTec aquires Gnutek Ltd
- The SEC’s complaint against Manouchehr Moshayedi of sTec
- Channel stuffing
- See also: Bryan mentions sTec misconduct on the Data Center Podcast
[@54:04](https://youtu.be/qisoAIx8EE8?t=3244) Sans firmware?
- FPGA to ASIC transition article 2011. (aside: treat yourself to this amazing vintage mouse-themed site announcing the same) > It’s when microprocessors show up that all the trouble starts.

(Did we miss anything? PRs always welcome!)

Our next Twitter Space will be on Monday at 5p Pacific Time. Join us; we always love to hear from new speakers!

Silicon Cowboys

1 juni 2021 | 66 min

Oxide and Friends Twitter Space: May 31, 2021

Silicon Cowboys

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. In addition to Bryan Cantrill and Adam Leventhal, speakers included Steve Tuck, Tom Lyon, Dan Cross, and others. The recording is here.

(Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

[@1:25](https://youtu.be/faY7kWHQuNE?t=85) Bryan: Have you listened to the Reply All episode “Is the Facebook Microphone On?”

The truth is actually scarier, Facebook doesn’t need the mic to be on … to read your mind.

Silicon Cowboys

[@2:46](https://youtu.be/faY7kWHQuNE?t=166) The 2016 documentary “Silicon Cowboys” follows the rise of the Compaq computer company. (IMDb) (Watch the trailer ~3mins)

I was trying to watch “Halt and Catch Fire” with my kid … and there’s a lot of spontaneous sex breaking out…Fastest to one billion in revenue… fastest to Fortune 500… a meteoric rise

Open by Canion

[@7:05](https://youtu.be/faY7kWHQuNE?t=425) The 2013 book “Open” by Rod Canion (cofounder and CEO of Compaq): “How Compaq Ended IBM’s PC Domination and Helped Invent Modern Computing.”

[@10:02](https://youtu.be/faY7kWHQuNE?t=602) Steve: Ben Rosen was the venture capitalist who wrote the first check to Compaq, really got them off the ground. On the board for 20 years.

Their timing was right. The way they did the company was right. And they executed really really well.To go from zero to 50 thousand units, of almost anything, in the time span they did, is incredible.

[@14:40](https://youtu.be/faY7kWHQuNE?t=880) Tom: The thing that really put them on the map was having the portable when nobody else did. And being 100% compatible.

Those portables were barely luggable, they were huge!Back in a time when there was no network. Being able to pick up your computer and take it to a place, was your network.

[@16:47](https://youtu.be/faY7kWHQuNE?t=1007) Steve: A big catalyst for their success was the channel. People were able to pick it up and go, they didn’t need special training.

[@19:25](https://youtu.be/faY7kWHQuNE?t=1165) Dad used to bring home the luggable so I could play Space Invaders, and he would work on spreadsheets.

Portable before Compaq

[@20:49](https://youtu.be/faY7kWHQuNE?t=1249) There were portable solutions before Compaq, but for timesharing.

You had the T.I. Silent 700, in the 70’s, you could tote that home and plug it into the modem.

Osborne Effect

[@22:41](https://youtu.be/faY7kWHQuNE?t=1361) Tom: They killed their company with the famous Osborne Effect

Bryan and Steve (clearly excited): What was the Osborne Effect!?
Tom: Pre-announcing the next machine.Telling customers: man, if you love the Osborne 1, just wait till the Osborne 2… So they did!

[@24:40](https://youtu.be/faY7kWHQuNE?t=1480) Bryan: Something I found surprising about the history of Compaq was the different organizational approach that they had.

Early on, before even thinking about what to go do, they were talking about the kind of company they wanted to build.

PBS Silicon Valley

[@26:14](https://youtu.be/faY7kWHQuNE?t=1574) The 2013 PBS documentary “Silicon Valley” tells the story of Fairchild Semiconductor. (Watch chapter one ~17mins)

[@28:14](https://youtu.be/faY7kWHQuNE?t=1694) We ask people, when they apply to Oxide, when they’ve been most unhappy in their careers. And it all boils down to people not feeling listened to, not having agency.

IBM’s role

[@29:41](https://youtu.be/faY7kWHQuNE?t=1781) How much of Compaq’s success is just pure mis-execution from IBM? IBM inadvertently creates this pseudo open architecture, and makes exactly the wrong move in trying to reproprietarize it with the PS/2 and Micro Channel architecture; which is an absolute disaster.

In many ways the story of Compaq is as much the story of the failed PS/2.It was such a mis-execution to do this analysis on the market and say: we need to grab our existing customers and lock them in, before they slip through our fingers, and in doing so, just hasten their departure.
And Compaq was in the right spot to pick up the pieces.

MCA (Micro Channel architecture), ISA, EISA

[@33:22](https://youtu.be/faY7kWHQuNE?t=2002) We were ripping out a bunch of ISA and EISA drivers..

I am a sacrificial sheep, I can’t possibly go. You are a sacrificial lamb.The machines themselves are anemic, if you want any functionality you go to a third party..
There were magazines filled with advice on which sound-generating card you should buy.

IBM PC XT – Hercules graphics card

[@37:00](https://youtu.be/faY7kWHQuNE?t=2220) Driver for Token Ring.

PCI – SBus – VME – VLB – AGP

[@40:20](https://youtu.be/faY7kWHQuNE?t=2420) Speaking of Intel, a big part of the Compaq story is what happens with the 386.

IBM clearly thought Intel would never give some clone manufacturer the first rights to the 386.They went from fast follower to innovator.

OS/2 supported both 16 bit (for the

from /proc to proc_macro

25 maj 2021 | 66 min

Oxide and Friends Twitter Space: May 24, 2021

from /proc to proc_macro

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. In addition to Bryan Cantrill and Adam Leventhal, speakers included Brian Cantrell (not making that one up!), Nima Johari, Joshua Clulow, Laura Abbott, and Tom Lyon. The recording is here.

(Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

The other Adam Leventhal [1] and the other AHL [2]
[@3:16](https://youtu.be/85eApYSj3ic?t=196) Hockey
- Calder Cup
- Charlotte Checkers
- Grand Rapids Griffins
[@4:02](https://youtu.be/85eApYSj3ic?t=242) Roger Faulkner invented the /proc filesystem
Gerald Ford Presidential Library and Museum
- Gerald Ford inaugural address (including its most famous line, “our long national nightmare is over”) > I went in a Gerald Ford cynic, and came out a Gerald Ford super-fan
Roger’s “The Process File System and Process Model in UNIX System V” paper
[@7:43](https://youtu.be/85eApYSj3ic?t=463) “I am on a mission from God to make programs debuggable”
- AVL trees and linked lists > Performance is the root of all evil.
- Trace Normal Form
- Watchpoints, libwatchmalloc > Watchpoints are magical, when they work. It feels like a superpower.
[@11:37](https://youtu.be/85eApYSj3ic?t=697) > Roger made this incredible contribution about debugging infrastructure > being an attribute of a production system.
- strace, truss
- BONUS: 1986 USENIX: A System Call Tracer in UNIX
- The ptrace(2) system call
- ptrace’s overloading of the wait(2) system call
- The German word that we’re seeking: Misappropriation-of-mechanism-in-a-seemingly-clever way-but-is-ultimately-a-disaster > ptrace is the x86 of system calls
[@16:45](https://youtu.be/85eApYSj3ic?t=1005) A long-coming apology..
- Linux branded zones (LX)
- “Method and system for child-parent mechanism emulation via a general interface” patent > You have to be bug-for-bug compatible.
- LX vfork/signal bug that broke golang > vfork: unsafe at any speed, toxic in any quantity
[@20:16](https://youtu.be/85eApYSj3ic?t=1216) Upstart’s problematic use of ptrace(2)
- Celebrating Joshua getting ptrace correct for LX branded zones
- Stack shenanigans breaking LX
- Red zone, segmented stacks
[@24:39](https://youtu.be/85eApYSj3ic?t=1479) The application was fishing in its own stack..
- Clozure Common Lisp, mcontext > These kinds of lies just don’t nest. Magic does not layer well.
[@28:56](https://youtu.be/85eApYSj3ic?t=1736) Windows Subsystem for Linux WSL
illumos on an M1?
- QEMU, ARM Cortex-M > It’s hard to get the machine really properly emulated
- AWS Mac minis
[@33:55](https://youtu.be/85eApYSj3ic?t=2035) It’s kind of amazing that Apple has never had much interest in the server space.
- Apple Xserve
- CHRP
The story of the stolen laptop. Little endian PowerPC
- OpenPOWER
[@37:35](https://youtu.be/85eApYSj3ic?t=2255) Language H!
- NCR
- Language H: An informal overview ( part 1, part 2)
The (other) D language
[@39:12](https://youtu.be/85eApYSj3ic?t=2352)
- AADEBUG’03
- Postmortem Object Type Identification
[@41:31](https://youtu.be/85eApYSj3ic?t=2491) It all comes back to awk
- Bourne shell source code / Algol68 #defines
- Thompson shell
Bryan’s 2007 Dtrace review, Google TechTalk ~80mins
[@48:07](https://youtu.be/85eApYSj3ic?t=2887) Dtrace language inspiration
- Dtrace clones > It was all based on us exploring some phenomenon, > something being kind of a pain in the ass or impossible, > and inventing something that was easy to use.
- Architectural review board: “This reminds us a lot of awk..” > What’s the most powerful one-liner you can crank out with awk?
- CUDA, Bluespec
[@52:35](https://...

golang asserts and the PLATO terminal

18 maj 2021 | 29 min

Oxide and Friends Twitter Space: May 17, 2021

golang asserts and the PLATO terminal

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. In addition to Bryan Cantrill and Adam Leventhal, speakers included Adam Jacob, Matt Ranney, Nima Johari, Antranig Vartanian, Joshua Clulow, Tom Lyon, and Bob Mader (and thanks to Jeremy Morris for catching Bob’s profile!).

(Did we miss your name and/or get it wrong? Drop a PR!)

We recorded the space, but we had some challenges, and we lost the recording when the first Twitter Space died at around 5:30p. We recorded the second half though; the recording is here.

Some of the topics we hit on, in the order that we hit them:

Khan Academy blog entry on Go
Adam’s blog entry, I Love Go, I Hate Go > I found novelty in the strictures, but objected to some of the specifics
[@2:40](https://youtu.be/8tJEwCvZWsg?t=160) Go’s assertion assertion
The Elm Language
[@4:40](https://youtu.be/8tJEwCvZWsg?t=280) Lionizing Unix > 7th edition is amazing, incredible, a break through.. > and it’s also kind of a shitty engineering artifact that needed a lot of work.
[@6:32](https://youtu.be/8tJEwCvZWsg?t=392) Core dumps
[@7:03](https://youtu.be/8tJEwCvZWsg?t=423) Impromptu PSA:
- Happy 81st Birthday Alan Kay!
- Alan Kay tribute video to Ted Nelson, including the story of how Alan Kay and his wife – Bonnie MacBird – were brought together by Ted Nelson, and how PARC inspired her to write TRON (!)
- Bedknobs and Broomsticks (WAT)
[@13:18](https://youtu.be/8tJEwCvZWsg?t=798) Brian Dear’s The Friendly Orange Glow
- The PLATO Terminal
- Control Data Corp (CDC)
- Dr. David Gräper’s Grapenotes
- Empire game
[@20:05](https://youtu.be/8tJEwCvZWsg?t=1205) Write your own lessons in TUTOR
- Dartmouth BASIC
- SNOBOL
[@23:12](https://youtu.be/8tJEwCvZWsg?t=1392) Dr. David Gräper’s Grapenotes started in 1977
Xerox Alto computer

(Did we miss anything? PRs always welcome!)

Our next Twitter Space will be on May 24th, 2021 at 5p Pacific! We’ll be kicking off the discussion with Silicon Cowboys (aka the real and sexless Halt and Catch Fire) on the rise of Compaq – and their aspiration to be a different kind of company. Join us; we always love to hear from new speakers!

A Requiem for SPARC with Tom Lyon

11 maj 2021 | 93 min

Oxide and Friends Twitter Space: May 10, 2021

A Requiem for SPARC with Tom Lyon

We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. In addition to [@bcantrill](https://twitter.com/bcantrill) and [@ahl](https://twitter.com/ahl), speakers included special guest Tom Lyon plus Joshua Clulow, Dan McDonald, Dan Cross, Tom Killalea, Theo Schlossnagle, Antranig Vartanian, and [@perlhack](https://twitter.com/perlhack).

We recorded the space; the recording is here.

Some of the topics we hit on, in the order that we hit them:

[@2:06](https://youtu.be/79NNXn5Kr90?t=126) SPARC 30th anniversary dinner > SPARC was an amazing achievement for its time, > but there were some nasty trade-offs made.
[@2:56](https://youtu.be/79NNXn5Kr90?t=176) illumos announcement on the end of SPARC support
- SPARCstation 2
[@4:37](https://youtu.be/79NNXn5Kr90?t=277) “There is no photography allowed in the bring-up lab” story
- SPARCstation 1 (code-named Campus) > They bricked their first CPU..
[@6:23](https://youtu.be/79NNXn5Kr90?t=383) UltraSPARC-II E-cache parity error
[@8:51](https://youtu.be/79NNXn5Kr90?t=531) Register windows > Most people don’t know, about that first SPARC, > there was no integer multiply or divide..
> It would trap on the instructions.
I feel so decadent, I’ve just been sprinkling multiplications around my code for years.
[@9:55](https://youtu.be/79NNXn5Kr90?t=595) popc instruction (also called Hamming Weight)
- IBM Stretch 1961, and the one-of-a-kind IBM Harvest made for the NSA
- Henry Warren’s 2002 Hacker’s Delight Ch. 5 shows a ~20 instruction algorithm (no branches, only adds/shifts/masks by constants) > Warren: According to computer folklore, the population count function is important to the > National Security Agency. No one (outside of NSA) seems to know just what they use it for, > but it may be in cryptography work or in searching huge amounts of material.
- According to Agner Fog, Ice Lake performs popcnt with a 3 cycle latency, and Zen 3 with just 1 cycle latency.
- Phil Bagwell’s 2001 Ideal Hash Trees depend on pop count > Bagwell: Note that the performance of the algorithm is seriously impacted > by the poor execution speed of the POPCT emulation in Java, a problem > the Java designers may wish to address.
  - Persistent versions of Bagwell’s trees are used for the built-in hash maps of Clojure, and in libraries for Scala etc.
[@11:39](https://youtu.be/79NNXn5Kr90?t=699) This was the debate between Roger Faulkner and Jeff Bonwick: register windows
- Roger Faulkner (RIP) thought they were horrific
[@12:35](https://youtu.be/79NNXn5Kr90?t=755) Register fishing: Bryan’s version and Adam’s version > When you want to know the state of some other process, you have to flush > those register windows to memory to be able to recover the stack trace.
- [@14:30](https://youtu.be/79NNXn5Kr90?t=870) Delay slot > We sat around the lunch table talking about how crazy it would > be to have a branch that executed right after a branch.
- DCTI couple (delayed control transfer instruction)
- [@15:31](https://youtu.be/79NNXn5Kr90?t=931) “Well, the instruction set doesn’t allow that..” story > Bedlam. As far as Solaris kernel discussions go, bedlam.
- Leibniz vs. Newton
[@20:14](https://youtu.be/79NNXn5Kr90?t=1214) Annulled branches
[@22:17](https://youtu.be/79NNXn5Kr90?t=1337) Praise for SPARC
- SPARC address space identifiers > When we were porting Solaris to x86, and deciding what fraction of the > address space would belong to the kernel vs the user, it felt disgusting to me.
[@25:26](https://youtu.be/79NNXn5Kr90?t=1526) Software-filled TLB > They just didn’t have the room to cram a hardware page table walk into the chip.
- MIPS would give you a trap on a VAC conflict (virtual address cache)
[@27:34](https://youtu.be/79NNXn5Kr90?t=1654) It was slow, it was late, and it had a lot of problems, it was wrong.
- UltraSPARC-III, code-named “Cheetah” > It’s weird, I compile this thing over and over, and every 80th time when > I compile and run it, it’s 40x slower..
- UltraSPARC-IV+, code-named “Panther”
[@32:17](https://youtu.be/79NNXn5Kr90?t=1937) Does the Viking I-cache bug ring a bell?
- SuperSPARC, code-named “Viking” > You’d have to DC balance the I-cache. If you had too many zeros, > they’d start flipping to ones.
- E-cache parity error > It was due to everything but high energy particle strikes.
- Radioactive boron in our SRAM manufacturing process
[@38:52](https://youtu.be/79NNXn5Kr90?t=2332) “Move it further from the tube” story > When you’re going to have a customer do something, you have to remember there’s > a human being on the other end of that. You cannot have them chasing your theories. > You need to be transparent and honest with them.
[@42:25](https://youtu.be/79NNXn5Kr90?t=2545) Micron DRAM story
[@44:38](https://youtu.be/79NNXn5Kr90?t=2678) High priced consultants and cosmic rays > They literally lined the roof with lead.. and it didn’t change the error rat...

Mr. Leventhal, Come here I want to see you

4 maj 2021 | 31 min

Oxide and Friends Twitter Space: May 3, 2021

Mr. Leventhal, Come here I want to see you

We’ve been holding a Twitter Space on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for May 3, 2021.

In addition to Bryan Cantrill and Adam Leventhal, speakers on May 3rd included Laura Abbott, Nate, Antranig Vartanian, François Baldassari, Tom Killalea, Land Belenky, and Sid?. (Did we miss your name and/or get it wrong? Drop a PR!)

Before the recording started, we discussed:

2011 Solaris Family Reunion video ~20mins
Katie Moussouris’s blog entry on the Clubhouse vulnerabilities
Laura’s blog entry on the LPC55 vulnerability
Land pointing us to the Atmega 328p MCU in a BK Medical endorectal probe
François on the STM32F103 found in Pebble
Intel Management Engine

Some of the topics we hit on, in the order that we hit them:

ASPEED BMC chip
[@1:24](https://youtu.be/h-WSU3kiXVg?t=84) So formal correctness is something that I think we are all very sympathetic with. > It’s very laudable, it’s also very hard.
- From L3 to seL4 What Have We Learnt in 20 Years of L4 Microkernels? (paper)
- Who guards the guards? Formal validation of the Arm v8-m architecture specification (paper) > Hardware architecture is an area where formal verification is more tenable, > a level you can readily reason about.
Our challenge is how can we satisfy our need for formalism without getting too pedantic about it. You don’t want to lose the forest for the trees.
A system we never deliver doesn’t actually improve anyone’s lives, that’s the challenge.
[@5:20](https://youtu.be/h-WSU3kiXVg?t=320) Journal club experiences
- Bootstrapping Trust in Modern Computers (book) > [@9:45](https://youtu.be/h-WSU3kiXVg?t=585) > We’ve tried to build a culture of looking to other work that’s been done. > Not because everything’s been done before, but because you don’t want to have to > relearn something that someone has already learned and talked about.
  > If you can leverage someone’s wisdom, that’s energy well spent.
[@11:46](https://youtu.be/h-WSU3kiXVg?t=706) When systems repeat mistakes, engineers feel deprived of agency: “I suffered for nothing.” > Engineering is this complicated balance between seeing the world as it could be, > and accepting the world as it is. > As you get older as an engineer, it’s too easy to no longer see what could be, > and you get mired in the ways the world is broken. You can become pessimistic.
Caitie McCaffrey on Distributed Sagas: A Protocol for Coordinating Microservices (video ~45min)
[@14:17](https://youtu.be/h-WSU3kiXVg?t=857) It’s dangerous to live only in the future, detached from present reality. Optative voice
[@16:45](https://youtu.be/h-WSU3kiXVg?t=1005) At Oxide, we ask applicants “when have you been happiest and why? Unhappiest?” Interesting to see that unhappy is all the same story: we were trying to do the right thing and management prevented it. > When I was younger and maybe more idealistic and willing to charge at the windmills, > I stayed too long with a company. > All the developers that interviewed me were gone by the time I got there. > I should have walked out the door, but I was too young and didn’t know better.
[@18:43](https://youtu.be/h-WSU3kiXVg?t=1123) “How do you and your cofounder resolve conflicts?” > I don’t want to hear about how you don’t have conflicts, tell me about how you resolve them.
Folks aren’t able to walk away, they’ve got this commitment both to the work and to their colleagues.
I’ve been a dead-ender a couple of times, I’ll go down with the ship.
[@20:28](https://youtu.be/h-WSU3kiXVg?t=1228) In “Soul of a New Machine” (wiki) Tom West says he wants to trust his engineers, but that trust is risk. > I just love that line: that trust is risk. > That’s part of the reason some of these companies > have a hard time trusting their technologists, > they just don’t want to take the risk.
People are so not versed in how to deal with conflict, and there’s nothing scarier than salary negotiation.
They need you, that’s why you’re here, you’ve made it all the way through the interview to this point, you’ve got leverage, now’s the time to use it.
[@23:04](https://youtu.be/h-WSU3kiXVg?t=1384) Oxide: Compensation as a Reflection of Values > It takes the need for negotiation out, > because it replaces it with total transparency.
Sometimes it’s not about what you’re getting paid, it’s about what the other person is getting paid. Not wanting to get taken advantage of.
It’s a social experiment for sure.
[@28:07](https://youtu.be/h-WSU3kiXVg?t=1687) Steve Jobs famously tried this at NeXT: pay was transparent but not equal.
- History of compensation at NeXT (wiki) (quora post) > I think that’s the worst of both worlds, a recipe for disaster.

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!

Oxide and Friends

Avsnitt

AI Discourse with Steve Klabnik

AI, Materials, and Fraud with Ben Shindel

RIP USENIX ATC

Solutions Software Engineering with Matthew Sanabria

Shootout at the CNCF Corral

Bringing up Cosmo

Character Limit with Kate Conger and Ryan Mac

Hell is other networks

Raiding the Minibar

Lip-Bu Tan's Intel

A Happy Day For Rust

A Crate is Born

Transparency in Hardware/Software Interfaces

A Half-Century of Silicon Valley with Randy Shoup

Textual UIs with Orhun Parmaksız

AI Disruption: DeepSeek and Cerebras

Holistic Engineering with Robert Mustacchi

Crates We Love

Predictions 2025

OxF 2024 Wrap-Up

Scaling Bluesky with Paul Frazee

Conferences in Tech

Intel after Gelsinger

Technical Blogging

Books in the Box IV

Unshrouding Turin (or Benvenuto a Torino)

Querying Metrics with OxQL

RTO or GTFO

Reflecting on Founder Mode

RFDs: The Backbone of Oxide

Whither CockroachDB?

The Saga of Sagas

Pragmatic LLM usage with Nicholas Carlini

CrowdStrike BSOD Fiasco with Katie Moussouris

Heterogeneous Computing with Raja Koduri

Innovation Tokens with Charity Majors

Is NVIDIA like Sun from the Dot Com Bubble?

Musing with Changelog's Adam Stacoviak

Rebooting a datacenter: A decade later

Bookclub: How Life Works by Philip Ball

All we have to fear is FUD itself

A Baseball Startup with Paul Freedman and Bryan Carmel

Discovering the XZ Backdoor with Andres Freund

Cultural Idiosyncrasies

Adversarial Machine Learning

Data Visualization

Crucible: The Oxide Storage Service

Innovation Stagnation?

Helios

What's taking so long?!

Open Source LLMs with Simon Willison

Predictions 2024!

AMD's MI300 and the Future of Accelerated Compute

Framework Computer with Nirav Patel

OpenAI's Boardroom Brawl

Hiring Processes with Gergely Orosz

Launching the Cloud Computer

Open Source and Capitalism with Ashley Williams and Adam Jacob

Settling Beef

Mr. Nagle's Wild Ride

DTrace at 20

Open Source Anti-Patterns with Kelsey Hightower

Fork in the road for Terraform?

No Silver Bullets

Books in the Box III

The Frontend of the Computer

Tales from Manufacturing: Shipping Rack 1

Shipping the first Oxide rack: Your questions answered!

Okay, Doomer: A Rebuttal to AI Doom-mongering

Software Verificationpalooza

Virtualizing Time

Open Source Governance

Building Together: Oxide and Samtec

The Network Behind the Network

Blue Skies Over Mastodon (with Erin Kissane and Tim Bray)

Rust Trademark: Argle-bargle or Foofaraw?

Cabling the Backplane

Get You a State Machine for Great Good