Sid Sheth, Founder and CEO of d-Matrix, joins Nataraj Sindam to explain why AI inference needs a different compute architecture from training. Sid discusses digital in-memory compute, the shift from chips to rack-scale systems, d-Matrix's NVIDIA partnership, and what it takes to build a semiconductor company from first silicon to production infrastructure.
5 Things You'll Learn from This Episode
- Why Sid and his co-founder bet on datacenter inference in 2019, three years before ChatGPT made the opportunity obvious.
- How digital in-memory compute places memory and computation together to reduce data movement, latency, and energy use.
- What it takes for a 10-person semiconductor startup to design and tape out its first chip within 12 months.
- Why AI infrastructure companies must now deliver complete rack-scale systems instead of standalone chips or accelerator cards.
- How d-Matrix complements GPUs on interactive, latency-sensitive workloads and why it partnered with NVIDIA on its next-generation Raptor XPU.
- How stacked custom DRAM can provide 20 to 30 times more memory capacity per rack for larger models, longer contexts, and more users.
About the Episode
Sid Sheth, Founder and CEO of d-Matrix, joins Nataraj to explain why the next phase of AI infrastructure is about inference, interactivity, and latency rather than training alone. They cover d-Matrix's memory-centric architecture, the journey from first silicon to rack-scale systems, its NVIDIA collaboration, acquisitions, customer model, and Sid's vision for deploying gigawatts of inference compute.
Timestamps
- 0:00 — Introduction: rethinking AI inference infrastructure
- 0:48 — Why Sid started a semiconductor company in 2019
- 5:46 — The original thesis for dedicated inference compute
- 7:33 — Betting early on transformers and large language models
- 10:17 — Building d-Matrix's first in-memory compute chip
- 12:44 — Finding product-market fit with hyperscalers
- 16:17 — Taping out first silicon with a 10-person team
- 19:05 — From analog concepts to digital in-memory compute
- 22:58 — Why AI hardware moved from chips to rack-scale systems
- 26:25 — How d-Matrix differs from GPUs and other accelerators
- 30:18 — d-Matrix's hardware business model and target customers
- 32:39 — Partnering with NVIDIA on the Raptor XPU
- 36:35 — How stacked DRAM improves inference
- 38:26 — Acquisitions that accelerated rack-scale deployment
- 41:23 — The investment cycle for AI hardware startups
- 44:52 — d-Matrix's three-to-five-year vision
Key Insights
Q: What is digital in-memory compute?
Digital in-memory compute places memory and computation together so data does not need to move constantly between separate components. d-Matrix uses SRAM-based digital in-memory computing to improve speed and energy efficiency without relying on exotic memory or sacrificing the predictability customers expect from fully digital systems. The architecture is designed specifically for inference.
Q: Why are GPUs not ideal for every AI inference workload?
GPUs paired with HBM are effective throughput engines, especially when workloads can be heavily batched. Interactive applications such as coding agents and reasoning models prioritize latency, so users need each response quickly rather than maximizing total work per batch. d-Matrix targets that low- and ultra-low-latency side of the inference performance curve and can operate alongside GPUs.
Q: Why is AI hardware moving from chips to rack-scale systems?
Frontier labs and hyperscalers deploy compute in clusters measured at enormous scale and do not want to integrate individual chips or cards themselves. Hardware companies therefore need to package silicon into servers, connect servers through networking, provide software, and deliver complete racks that can be rolled into a datacenter. d-Matrix spent 2024 through 2026 evolving from a chip-scale company into a rack-scale infrastructure provider.
Q: How does stacked DRAM help AI inference?
Larger models, longer contexts, prefill, KV caches, and multiple users all require more memory. d-Matrix's Raptor architecture stacks custom DRAM directly on its XPU to provide extremely high bandwidth and roughly 20 to 30 times more capacity per rack than its SRAM-only approach. Sid says this makes it possible to support frontier models within a single rack.
About Sid Sheth
Sid Sheth is the Founder and CEO of d-Matrix, a leader in memory-centric compute for AI inference. Before founding d-Matrix in 2019, he was SVP and GM of Inphi's Networking Business Unit, where he grew the business to more than $1 billion before Inphi was acquired by Marvell. Earlier, he held leadership roles at NetLogic Microsystems and was a founding member of Aeluros.
Founder & CEO at d-Matrix
About the Host
Nataraj Sindam is the creator of The Startup Project, a podcast featuring founders, investors, and operators building the future.
#SidSheth #dMatrix #AIInference #Semiconductors #InMemoryCompute #AIInfrastructure #RackScaleSystems #NVIDIA #StartupProject #Podcast #Entrepreneurship #Tech