Was I wrong about Etched?

Or did they just fake it until they made it?

I’ve been a public critic of Etched for a while. I’ve put out a couple articles rebuking the idea that a chip focused purely on transformers would significantly outperform a modern GPU. That idea was ostensibly Etched’s core premise. Put simply, modern datacenter GPUs are already ruthlessly optimized for running transformers, so “focusing on transformers” entails going head-to-head with Nvidia. There have also been a number of negative rumors about Etched going around the chip world, from senior talent leaving to their first tapeout failing due to horrible thermal issues. So recently, when they announced their first chips were going into production, I was surprised. Their inference servers lean on a couple technologies: highly under-volted logic, and pooled rack-level memory modules. So, was I wrong about Etched the whole time?

Well, maybe not. All the doubters were skeptical of Etched’s original plan, which was just a dedicated transformer ASIC. If their under-volted logic and rack-level memory solutions were invented after those initial announcements, that doesn’t mean the skeptics were wrong. It just means that Etched managed to fundraise enough on their initial bad ideas to hire the silicon engineers who developed new, better ideas -- a common route for startups. Separately, though, it remains to be seen whether their new technology proposals actually result in better chips.

So today, we’ll be going through both. I’ll briefly discuss the history of Etched and all the rumors surrounding it, and then move on to the substance of their current technology, to actually answer the question: was Etched good before, and is Etched actually good now?

The Rumor Mill

Etched was founded in 2022, ostensibly on the idea that a chip purely focused on LLMs would significantly outperform GPUs. They drew an analogy to dedicating Bitcoin ASICs, which genuinely offer massive performance improvements over GPUs by focusing purely on running SHA-256. There are two big problems with this comparison, though. Firstly, SHA-256 is a highly specialized algorithm, with a bunch of bit-shifts and XORs and modular additions that conventional GPUs aren’t optimized for. LLM inference, on the other hand, is mostly just big matrix multiplies, which GPUs and TPUs are already pretty great at. Also, more importantly, Nvidia is actively focusing on optimizing their datacenter GPUs for running LLMs! So “focusing on LLMs” means competing directly with Nvidia, without any differentiating technology or market counterpositioning. In my book, that’s a bad idea.

After Etched was founded, they hired a team to execute on that first transformer ASIC. From the rumors I’ve heard, this did not go super well -- though the following is all unconfirmed, so take it with a grain of salt. I’ve heard about senior talent leaving due to conflict with company leadership, but there were technical issues too. Rumor has it that they were originally planning on leveraging a combination of AlphaChip-like AI techniques and external contractors for physical design of that first Sohu transformer chip. That didn’t go well, and their Sohu transformer chip apparently had major thermal issues, which forced them to reevaluate their entire system design. Etched’s leadership even publicly admits the issues they’ve had with Indian physical design contractors. Clive Chan called it a “rocky start”, which is an understatement.

So what do you do when your chips are melting, your senior talent is leaving, but you have a ton of capital raised from high-frequency-trading firms and venture funds¹ who don’t know enough about chips to question the too-good-to-be true “focus on transformers” story? Well, you pivot. And when your chief architect has experience working on Bitcoin miners, which are often run at aggressively low voltages to reduce power consumption, it makes sense to try to do the same for AI chips.

Thus, Etched’s Low Voltage Inference technology was born. It means that their next chip won’t melt like the first one did. But is it actually good?

Is low voltage inference actually good?

The formula for the power consumption of a standard digital switching circuit is:

P ∝ CV²f

where C is the load capacitance, V is the voltage, and f is the frequency you run the circuit at. At the same time, the maximum frequency of a circuit is inversely proportional to the supply voltage -- lower voltage circuits are slower.

That means if they lower the supply voltage by a factor of N, the circuits run slower by a factor of N, but they reduce power by a factor of N². So LVI improves the datacenter economics (tokens per watt) but not the datacenter throughput (tokens per second). This isn’t an issue for bitcoin miners, where they can trivially add parallel SHA-2 hash cores to make up for the lost throughput due to under-volting. But it could be an issue for AI agents, where adding latency increases a system’s time-to-first-token. This is basically the opposite of the SRAM machines like Groq and Cerebras, which are extremely expensive to operate but very, very fast.

Datacenter economics and total cost of ownership is about more than just power consumption, though. There’s also some fixed cost to build out all of the hardware in a datacenter, which is dominated by the cost of the chips themselves. If you want to get the maximum token throughput from a fixed upfront hardware cost, LVI isn’t the right solution. But if power efficiency is the main goal, it could be.

The other challenge with ultra-low-voltage computing is actually designing circuits that work well at low voltages. It’s been all-but-confirmed that Etched is using at least some custom standard cells to operate their design below the supply voltage that TSMC’s standard cells are designed to operate at. This is a lot of work, but if they can actually operate at a meaningfully lower supply voltage, it could be a big win for power consumption.

Etched also discussed another technology in their announcement: their cluster-scale memory technology. This one has less of a rumor-filled backstory, and they shared a lot less about it, so I have less to say, but it’s still worth discussing.

What about cluster-scale memory?

There are fewer details available on Etched’s memory solution, which is supposed to allow shared memory pools across the scale-up domain. They’re right that HBM-based systems are slower than all-SRAM-based systems, but it’s unclear how their shared memory subsystem solves the latency challenges with HBM that they’re claiming to solve.

In practice, I would guess that their system allows chips to write directly to, and potentially read directly from, the HBM in other chips packages. On GPUs, these sorts of peer writes are normally routed through L2 cache, which would increase overall latency. This is likely a meaningful improvement, though it would put more pressure on the compilers and the programmers to properly route data without the coherency guarantees of caches. Though also, I’m just speculating, because Etched hasn’t made any technical details on their architecture public…

She said it, not me!

The old Etched website made claims about absurd tokens-per-second numbers that never materialized. The new Etched website touts a technology that improves power efficiency at the cost of speed. Ultimately, I think it’s fair to say that those of us who were skeptical about the initial Etched PR had every right to be, even if they end up releasing a chip that has a genuine advantage for customers optimizing for tokens-per-watt over tokens-per-second.

Expect another article once they actually release technical specs!

1. To be diplomatic about it: Radical Ventures partners have treated me and founders I know poorly, and I cannot recommend you work with them.