Transcript: Inside d-Matrix: Memory-Centric Compute for Low-Latency AI Inference | Sid Sheth | Startup Project #130

Nataraj: Hello everyone, welcome to Startup Project. Today I'm joined by Sid. He's the co-founder and CEO of d-Matrix, one of the most interesting companies which is rethinking infrastructure behind AI inference. I think we have seen two phases of this last compute era where the first part was all focused on training and getting as many GPUs for your training, but I think last two years the focus has been AI inference and low latency AI inference. So in this conversation we'll talk about future of AI inference, why GPUs may not be the answer for every workload, what does it take to build a semiconductor startup, partnering and competing with NVIDIA, and maybe where AI hardware goes from here. With that, Sid, welcome to the show.

Sid: Thank you, Nataraj. Thanks for having me.

Nataraj: So I guess the first question is: starting a hardware startup I think in the last three years has been pretty popular, I would say. But I think you started this before ChatGPT, I believe around 2019ish. So tell me about like what was thesis back then? Because it wasn't obvious to start a semiconductor startup. I remember at that point Most venture was into investing into SaaS, maybe no-code tools. I know there was some deep tech always happening in the industry, but it's not common for someone to go ahead and start a semiconductor company. The only company I can remember is the other one that recently went IPO was Cerebras, which was I think a couple of years around that time itself. I think maybe 2016 or 17. So give me an idea of like the origin of d-Matrix.

Sid: Yeah. So, just before I answer that question, right? So thanks for having me, obviously. I and my background has been in the semiconductor industry for thirty plus years, right? So I grew up in India, I came to the United States for my graduate studies and then joined Intel and this was back in the nineties, mid to late nineties.

Nataraj: That's

Sid: And I've been I've been in the semiconductor and system space for thirty, thirty odd years. So this comes naturally to me, right? Building chips and building companies that build chips is something that I've been doing for many, many years. Now in the case of d-Matrix, before I started d-Matrix, I was running a very thriving business that was building connectivity chips for datacenters. This was at Inphi, the company I was at before I started d-Matrix. And I was I was in my mid-40s, right? At the time. And I'd spent about seven years building that business and it was very successful. We were shipping product to all the big hyperscalers, not just in the US, but also in China. That time we didn't have these export control restrictions that were so strict. So it was a very thriving business and I could have continued running that business. There was really no need for me to step out and start a company, right? I could have continued on that path. That business eventually got acquired by Marvell in a $10 billion acquisition. So it was very successful outcome for that team and that business. But coming from a family of entrepreneurs in India, I'm a third generation entrepreneur in my family. I think I wanted something more. I was looking to, build a business from the from the foundation, from scratch. And I never had the opportunity to be a CEO at any of those businesses, right? So this was my opportunity. And so that's there was there was this there was this inner calling that I wanted to go off and Do something on my own. But what happened also right around 2018, 2019 is that the AI opportunity was becoming quite clear to at least me and my co-founder. So we started studying that for at least a year and ideated the company for almost a year before we founded it. We spent a lot of time understanding what the landscape looked like, what people were building, which portion and segment of the market did we want to focus on. And after spending a year, On that, we concluded that inference would be the big opportunity that we should go after, right? We felt training, was the focus at the time, but all the narrative in 2019 was focused around training bigger and bigger models, right? But we felt that really inference would be the much, much larger opportunity, and it was not very obvious back then to anyone. And the gap we saw in the market was there was a lot of inferencing companies focused on edge applications. There were companies that were focusing on building chips for cameras and surveillance cameras and drones and smart industry, et cetera. We didn't see any company focused on building dedicated inferencing solutions for the datacenter and the cloud. So that is what d-Matrix became. We became the first company, amongst the first companies, to build a focused inference solution for the cloud and the datacenter. And then of course, we started working backwards from there to figure out what we should build, what the platform should look like. And over the years that has turned out to be the right bet.

Nataraj: Yeah. What was the sort of initial insight that said, okay, we should build an inference solution? Like what were you looking at back then that gave you that idea?

Sid: Yeah, I mean, it was pretty intuitive to us, to be honest. I like to give this analogy about human beings. We go to school and college for twenty years, educate ourselves, acquire knowledge. That's that's like training ourselves, right? Training the human mind. And then the next forty years of our life we apply all of that knowledge and we still continuously learn through that process, but really most of it is around application of knowledge that you've already acquired or the fundamental skills, the foundational skills.

Nataraj: Mm.

Sid: You have acquired. And that is inference. So, obviously the monetization happens in the 40 years that we as humans work. The training happens in the first 20 years, right? And the inferencing or the application part is a much bigger opportunity for human beings and should be no different for AI, right? So we just intuitively felt inferencing when it happens, when it really goes mainstream, would be the much, much larger. Opportunity and building a computing solution for inference dedicated and foundationally purpose built for inference would feed into a much, much larger opportunity. So, the leap of faith was obviously we did not know when that would happen. It could take three years, it could take five years, it could take ten years, right? And there was we had no clue that ChatGPT was just around the corner, maybe three years away. When we started in 2019, ChatGPT happened 2022. We had no we had no idea, obviously, that. Was going to be the case, but that was the leap of faith we took that when it happens, it will be the opportunity to go after.

Nataraj: Did you aim at inference? Because inference I think today what we say inference and maybe how we look at inference around 2018 was slightly different, right? Because large language models were not the predominant models in production workloads. Right. So were you targeting like traditional ML models at that point of time? Or you thought of like, okay, this model there'll be a new set of models that will come and inference as a use case Its TAM will expand.

Sid: Yeah, yeah. So keep in mind, Transformers had already arrived in 2019, right? I mean, we started the company in 2019. The first year I don't think we spent we were basically focused on the foundational compute engine, what kind of compute engine we should build and the compute substrate, understanding the core bottlenecks of inference. And then we started looking for product-market fit. But that really started only in 2020. And That's when we kind of when we studied the landscape, we had been we'd been looking at recommender systems, we had been looking at computer vision models. There was obviously RNNs that were popular at the time. As you said, there were ML, traditional ML models, right? And is there anything we could do there? But deep learning was taking off in a big way, obviously, at least inside all the hyperscalers. And the one data point we picked up was around how popular transformers were becoming at places like Google and Microsoft, right? So Google Was telling us that transformer workloads like BERT models, which were still small at the time, were being used for English language search and foreign language search already, and we're seeing a very quick uptake. Right. Microsoft was telling us that they are actively using transformers internally for pretty much all their all their internal first party properties, right? And so we just felt that there was something happening here. GPT-3 had just launched. There was a lot of buzz in the tech circles about the capabilities of GPT-3. And so you kind of we put all this together and said, there might be something we can do around transformer workloads because clearly there's not too much focus on that right now. Not many companies focused on building accelerated solutions for transformers. So why don't we go do that? And so it was it was the timing was good. We just happened to be out there looking for a problem to solve in 2020, and enough data had emerged. Around LLMs, transformers, et cetera, where we felt that, making a bet on that segment of the market would be a good one. And yeah, it turned out to be a great one.

Nataraj: Once you saw this, what was the initial product or system that you guys built? Like what was that the first product or the first demo that you got to towards building?

Sid: Yeah. So we actually, built our first chip in 2020. We started the company in 2019 and within twelve months we had already taped out our first silicon. And early twenty one we had that silicon back and our focus was really just showcasing that time it was like an MNIST model. It's a tiny character recognition model, right? And all it is really doing is can you recognize characters, and predict it's not even trying to predict anything, it's just trying to kind of classify, right? And what we wanted to show was accuracy because we were taking a very different approach with in-memory compute. So our thesis was memory will be the bottleneck. As models get bigger, people want to use more data, they want to put the data in memory and the compute and memory need to be co-located. So we had this idea about a memory-centric computing architecture way back in 2019 and 2020. And we just wanted to prove thesis that it could work because the kind of the consensus or the feeling in the industry was when you use techniques like in memory compute, that you tend to you have to use analog approaches, you need to use exotic memory types. When you put the two together, you typically sacrifice accuracy, you sacrifice predictability. And all the hyperscaler customers we were talking to wanted us to kind of first go address that issue, right? So we took an approach where we use SRAM memory, which is a very readily available memory, no exotic memory. It is the most readily available memory today. And we created a compute substrate with SRAM. So, how do you build an in-memory computing substrate with SRAM? And what we showed was that you can actually achieve levels of accuracy and predictability that are very close to a true digital engine, right? Like a GPU. And that was really the first demo we did, right? And that happened in early twenty one and we were able to show that to customers and investors to kind of address some of their concerns.

Nataraj: So yeah, y you said you were talking to hyperscalers. So what was your approach there? Like you were talking to different companies and trying to find like what is the solution they are looking for. Like tell me that process of why were you talking to those hyperscalers? And because I think different types of companies approach, like how do you build the first product differently? I was talking to a founder which sort of enabled one of the biggest well known startups. I don't want to name the startup. But this API platform sort of now is very highly valued. But their first API that they created was basically to support this now 10 billion Unicorn startup. And it's because of that AP, that startup wanted that API. They built it and that sort of like created, okay. And that led them to this insight that okay, agents are the ones that are going to use the web. Right. So I think that's their way of like. Finding the product market fit. And it feels like you were talking to hyperscalers to figure out where exactly the product market fit is there. Like what was this and how was the process going to be? Like someone who's like a new startup founder and listening to this and he will think, I should just go talk to Microsoft and find what is the gap they're trying to address.

Sid: Yeah, yeah. So there's different ways you can find your product market fit, right? It is it is more of an art right than a science. I mean, you really have to talk to lots of people. It's about gathering lots of signals, right? And the more signals you gather and the more you are adept at reading those signals and sifting through those signals to find some kind of a pattern, a demand pattern. That is really the job of the entrepreneur, right? So. In our case, we were exactly doing that, right? We wanted inputs from customers, we wanted inputs from industry experts, we wanted inputs from anyone who was in the supply chain, we wanted inputs from investors who talked to a lot of people. So there was there was we were gathering these signals from anyone and everyone who we could find and talk to or who would talk to us, right? I mean, keep in mind, we are we were first time entrepreneurs. Starting our own company and we were it's not like it's you really have to be persistent to get to the right people to find their time and get them to spend time with you and give you the feedback you're looking for, right? So that process typically takes a lot of time and then we went through that and we and there's the good thing was there was a lot of research. That was being published at the time. Now people have become a little careful about the kind of research they publish, especially the Frontier Labs and the Hyperscalers. But back in 2019 and 2020, there was plenty of research that was being published. So we were able to tap into all of these and kind of sift through at that and create a demand signal and a demand pattern that we felt would allow us to serve a very large opportunity, right? And yeah, I mean that eventually got validated when Microsoft decided to invest in our Series A round. They co-led that with Playground Global. But we had been talking to Microsoft for almost a year before that, right? Understanding their demand signals and the need for specific types of workloads. And so when that investment happened, that was a that was a validation of the fact that we were moving in the right direction.

Nataraj: You mentioned like within a year, you had your first your first silicon out. I mean you mentioned it casually but like it's one of the most hardest thing to do. Like in the process of like getting a first chip out is actually not clear. When you try to like find out, hey, how do you create a new chip and who are all the players involved? Can you just break down, what does it look like from design to like getting that first chip in your hand process look like and like How many companies are involved, how many vendors are involved and doing that.

Sid: Yeah. So, I it just building chips comes naturally to us, but it is not a natural process to most people, right? And especially when you are a startup of 10 people, you're trying to with three million dollars in the bank, you're trying to build your first chip. It is it is not an easy process, right? At all, at all. So you're doing many things in parallel. In our case, we had just incubated the company we were hiring, our first. Team of ten people. You are raising money while you are incubating or you're just incubated, so you're looking to raise money, so you're fundraising. You're doing all the work I just talked about, talking to customers, investors, partners, because you want to make sure that you're building the right technology for the right application and the right market. So you're spending a lot of time doing market research, right? Obviously. You're working withe partners to, enable the Building of that chip, right? We had to work with EDA partners, sign contracts with EDA partners, IP partners, with TSMC, with our supply chain, right? I mean, essentially who's going to build that chip and help us test it. And so you're doing all of these, and then while all of this is happening, we enter COVID, right? So COVID hits us in early 2020. And we are kind of all forced to go home and work for the first time in our lives. So all this is happening in parallel, right? And through all of that, we still maintained. The ability to get our first chip into TSMC's hands within 12 months of starting the company, right? So that is quite a feat. And that only comes from experience. We have done it 20 plus times before. So, when you're doing it for the 21st time, it is you've built the muscle over many, many years. So you exactly knowhat to look for, where the pitfalls are, where can you trip up, right? And that paid off. That really did pay off when it came to And then hiring the right people, hiring the right types of people who can get this done. So what types of people you need to go hire and help you, and which partners can help you. And of course the relationships you had already built over the many, many years. It all came to it all came together and helped us kind of get it all done in the first twelve months.

Nataraj: Once you had this first chip out, what was your next pivotal point or did you start selling chips? Did you start now thinking, okay, should we create a whole system around a new architecture? Like what was the next big sort of like iteration after getting that first chip done?

Sid: Yeah, it's a journey. Yeah, it's a journey. It's a journey. So I think, you start, we were chip guys, right? So we started withe silicon foundation. And getting the silicon foundation right was super critical to us, right? And we realized that at the end of the day, you when you're build when you're you're putting a building together, you gotta get the foundation right. Right. You can't rush through the foundational process and then expect that you'll build a strong structure on top of that. But you gotta get your foundation right. So we spent a lot of years, getting that foundation right, the in memory compute. We talked about the first chip. We did a second chip after that, a year and a half later, which was mainly around taking any analog functionality in the in the product and making it digital. Right. So now we have something called digital in memory computing engine. That we that we are is the foundation of everything that we do here, right? And that was a pivot. I mean, there was a pivot where we took our SRAM-based architecture that was still had some analog functionality and fully digitized it. And that was strictly based on customer feedback. Customers said, yeah, this is a great step in the right direction, but we still want to not sacrifice any level of accuracy when you compare this approach that you guys have come up with to a fully digital computing approach, right? So there is there is, maybe a few percentage points of accuracy that we still have to trade off. You don't want to do that. So we essentially found a way to fully digitize the computing engine. So we spent a solid four years, first three to four years on just foundationally getting the compute substrate right. And then creating a solution around it, like how do we build it with chiplets, what kind of ML techniques, numerical Innovations do we introduce into the product? What kind of software should we build, right? How should we program this chip, right? I mean, because tomorrow developers will want to program this chip. How should they program this chip? So all these things were very well thought out, right? And that led to our first product in 2024. And then what happened is the industry quickly started making the transition from chip scale to rack scale. If you remember, in 24 when NVIDIA announced The Blackwell chip, it was all rack scale GPUs. There was no cards, there was no trays. Prior to that, the Hopper GPU was really cards and trays. But Blackwell immediately made the transition to liquid cooled racks, right? Hopper was all air-cooled infrastructure. So in 2024, when we had our first chip, it was all air-cooled. We and it was all card scale. We had built card scale solutions. What we quickly realized is the industry is going to rapidly migrate to rack scale. So we had to essentially go on a journey to become a rack scale company in two years. So 24 to 26, we spent all of our time converting or essentially growing d-Matrix from a chip scale, car scale company to a rack scale company. And that, we'll talk about this in a bit, but you that led to a couple of acquisitions which I did in 2026. We'll talk about some, the NVIDIA partnership that we did recently. All these were done to quickly transition the company to a rack scale company. So that we can take our product to market, much faster in that form factor.

Nataraj: So wha what does it mean for the layman listening to convert a convert to rack scale architecture?

Sid: Yeah. So let's take an example, right? It's like everybody's everyone's familiar with a chip, right? But the chip is really not the product anymore, right? What happens is you build a chip, customers, let's take an example of customers like the Frontier Labs, whether it's OpenAI or Anthropic or somebody like that, they are deploying gigawatts of compute annually, right? Typically, this year alone. I think OpenAI was looking to deploy at least four to five gigawatts of compute, right? And so companies of that scale, I mean especially the inferencing problem, became so big post-ChatGPT, post-Claude Code and codecs, et cetera. When agentic coding took off, people are now deploying, companies are deploying gigawatts of compute, right? And when they deploy compute, they are deploying entire solutions. And The solutions are clusters or racks, right? So you are essentially taking the silicon, you're packaging it up into servers, servers are put into racks, racks are connected to each other, right? And the entire solution has to be deployed very quickly and rapidly because who, these customers don't want to worry about having to take a chip or a card and have to convert that into a server or a rack for deployment purposes, right? So that is really the job of the company building the chip or the card. Right. So in our case, because we build the chip or the card, we cannot go to these customers who are deploying compute at gigawatt scale and expect them to take what we have and convert it into a solution that they can deploy. We have to essentially convert it into a solution that they can deploy. And that is where the transition from just building chips to building racks that are solutions that these customers can deploy. Not only just racks, you have to be the ability to connect these racks. Using networking technology. And so the entire networking solution, the computing solution, the rack solution, a solution that we can roll in to a customer's datacenter is something that fell upon us, right, as a company. And we made that transition and are still in the process of completing it. But we started making that transition very rapidly in 2020 four to 2020 six now.

Nataraj: Well also like when datacenters were being built, in general the industry was like someone produces the chips, someone else produces the commodity racks, someone else is installing it and someone else is maintaining it. There was there was wh this whole ecosystem of like who is actually building the rack versus, G you put you take GPUs from somewhere, you take memory chips from somewhere, you all assemble somewhere. There was a lot of like whole stack of players involved, right? Which is sort of like different from what Now NVIDIA suddenly changed into like, we own the whole system. Instead of just like giving you the GPU and you figure out where to put it and how do you cool it and everything else. I think that's sort of like the shift that also happened. In terms of other approaches that are happening out there, like I think I've seen Etched we've seen SambaNova, we've seen Cerebras, I think each have the like sort of different approaches to how they're solving. Either inference problem or something else problem. When you compare like architecturally speaking, how should I think about like what d-Matrix is doing versus or like let's say even NVIDIA, right? Before there were CPUs and then GPUs came up and which are really good at matrix multiplications. So which coincidentally is what all the AI applications need. Now in the architectural sense, where should I how should I think about d-Matrix position when we compared the entire ecosystem.

Sid: Yeah, yeah. So I mean, it's so we are focused on inference only, not training, right? So that's very clear. We don't our the company started as an inference compute company and that's what we are. We took a memory centric computing approach. Why? Because we wanted to break through the memory wall, right? We wanted to essentially build a solution that puts memory and compute together. And where is that kind of solution very useful? It is very useful when the biggest advantages you get by putting memory and compute together. Is speed because you're not moving data around that much anymore, right? So you get a big, big step up in speed and you get a big step up in energy efficiency, right? Because you are again, you don't spend a lot of energy and time moving data around. So our focus is really on the low latency segment of the market, right? If you look at that chart of throughput versus interactivity, this is something that now Jensen loves to show in every keynote he does, right? So it's it's really the inference Pareto chart where you on the y axis you have throughput per kilowatt and on the x axis you have interactivity or speed. And the two of them are typically kind of work against each other, right? If you n if you are doing more work, more throughput is more work, right? So if you're doing more work, you typically need more time. And if you want to do something in less time, that means you probably are gonna do less work, right? So these two kind of pull in opposite directions, right? So you can't have a solution that can do both equally well in the same chip, right? So you have to basically pick which task you want to do really well. And I think GPUs with HBM memory have really very good at throughput. They're workhorses. They tend to do extremely well with lots of batching, lots of work, right? But that means it comes at the cost of interactivity, right? At d-Matrix, we are focused on interactivity. So we are not, we are not a throughput play. We don't use HBM memory at d-Matrix. We don't use so by not using HBM and using SRAM memory, in-memory compute, we'll talk about our next generation product stacks, DRAM, custom DRAM directly on top of compute. So again, we are creating this memory-centric compute paradigm around DRAM. These approaches are very good for low latency and ultra low latency applications, right? And so if you were to kind of really simplify it, a lot of the companies whose names you mentioned are HBM based solutions. And the moment you become an HBM based solution, you are going to focus on some kind of throughput play. You are going to compete withe GPU. Cerebras, Dmatrix, Groq, etc, we don't use HBM In our solutions. We use SRAM, and in the case of d-Matrix, we are quickly migrating to 3D stacked DRAM. But we don't use HBM, we don't use CoWoS, right? So that's where our solution is playing at a very different point on that performance curve. It is playing more on the latency, ultra-low latency part of the curve, not so much on the throughput part of the curve.

Nataraj: It talk to me a little bit about the business model. Right now, like how does d-Matrix sell its product slash solution? Like how does how should I think about like what kind of business segment or like how do you think about your own business and how do you sell your business?

Sid: Yeah. So we are a classic infra player, right? That is how we have built the company. That's how we are building the business. That is how we engage with customers. It is a traditional infra business where we sell hardware, right? So we are a traditional CapEx play for our customers, right? And what we go is we tell our customers like, look, we have unique hardware solutions that solve a very specific problem that they have around. A, there's just a need of raw compute. Everybody needs compute and we have compute available today because we are in production. But number two, we have compute that can really augment the GPU. Right? So GPUs are very good on throughput, as I was telling you earlier. We can augment those GPUs on interactivity. Right. So if you are a customer that is serving different types of models to different customers, we could come in and say, look, you already have GPUs in your fleet that are serving your customers on throughput. But guess what? Now withe new agentic coding tools, many of your customers will care about latency a lot. Right. So and given that a lot of the coding models are all reasoning models, they use test time compute and latency matters because you want these models to think very quickly if you want them to interact very quickly, then our solution can augment that type of application alongside a GPU. Right? So I think that is That is what we do. We go talk to, we identify customers who have these specific problems around latency. And then we are a classic infra play to those customers who want to deploy hardware and serve tokens or serve models to their end customers. So inference clouds would be customers, hyperscalers who are deploying models for the various frontier labs. The frontier labs themselves, sovereign clouds. Even the high frequency traders who really care about latency, right, all of these would be target customers for somebody like us.

Nataraj: Talk to me a little bit about like I think on a higher level, initially when you start creating a chip business, especially for inference, it might look like you're competing with NVIDIA. But I think you also partnered with NVIDIA. So what does that what does that indicate about like the general industry and how do you see that partnership evolving?

Sid: Yeah. So the NVIDIA partnership, it's a deep technology partnership and a co-development engineering partnership, right, on our next generation product, which is called a product called Raptor, which is a 3D stacked DRAM product, right? So it's gonna be the world's first stacked DRAM product where we are stacking custom DRAM directly on top of compute. So we have our SRAM based in-memory compute already, right? That is our first generation product. We are adding a layer of custom DRAM on top of that to add more capacity and to make that capacity available to the compute at extremely high memory bandwidth, right? By stacking it directly on top of the compute. Right. So that is a that is a real innovation. In fact, d-Matrix will be amongst the first, if not the first, company to take that innovation and commercialize it in the market. And we could we could Take that and essentially build our own solutions around it. And keep in mind the world is now quickly transitioning to liquid cooled solutions, liquid cooled racks. I mean, it is not easy. It is, it is, it would take a pretty substantial lift for a startup like us to build all of that stuff. And we can certainly do it because there is enough knowledge in the ecosystem now. But it is feels like it's like reinventing the wheel. Right, you're reinventing something that has already been solved for. And our focus should be on getting our technology to market as soon as we can. Right. So we have this very innovative XPU architecture, right? That can get to market a lot faster if we partner with somebody like NVIDIA, who already has their liquid-cooled racks deployed pretty much in every datacenter in the world. Right. It is available in every cloud, every country, right? And when you have such a broad-based deployment of rack-scale infrastructure with our AI factory ecosystem, we can get our technology in the hands of customers a lot faster if we just plug into their infrastructure, right? We don't have to go reinvent some of that stuff. And so that is why we chose to partner. This will allow us to take what is the world's first stacked DRAM XPU. And we don't want to lose any time, getting it to our customers, right? By doing things that have already been invented. So we just partnered and then focus on what we do really well, which is building the world's best XPU, building the software to program that XPU, and building, the subsystem essentially for that XPU. And then the subsystem plugs into the NVIDIA infrastructure, right? So we focus on what we do really well and then we kind of leverage what they have already done extremely well. And we the it's a win, right? Because they are able to partner with us on a solution that augments the GPU. We work withem to get to market a lot sooner, and our customers win because they get access to this technology a lot faster and in a scalable, robust way, right? Because they as a startup, obviously if you try to do all this yourself, it's pretty risky. So customers who want to deploy compute at scale now feel that there is a very large partner like NVIDIA, who is got the scale of deployment, a robust supply chain, right, that can support their needs.

Nataraj: This might sound a little bit more technical, but like when one once you stack CP so you're saying you're stacking more compute or more in more memory. How does that help inference? Like are you able to store more KV caches on memory or are you storing, a pre-fill phase of the inference like How does the that sort of like inference advantage like where are we getting that inference advantage from?

Sid: It's both. It's a all of the above, right? I think what's happening is context lengths are increasing dramatically, models are getting bigger. Obviously you want to support you wanna support multi multiple users. So, batch sizes, are in our case the batch sizes are not very, very large because we are not a throughput player. For inferencing, typically the kind of applications we serve, you typically have low to medium batch, but you still have some batched users. You have big context lengths, you have big models. If you want to do pre-fill, then you also need access to memory for pre-fill, right? Because that's where you're contextualizing all the queries. And you need memory for all of this. And SRAM, which was our first chosen memory technology for the first generation product, is not scaling as well with process technology, right? So you're not getting the kind of capacity you need with SRAM. So we came up with this very innovative way of essentially stacking. Custom DRAM directly on the XPU. So you get more capacity. We get almost twenty X more capacity, twenty to thirty X more capacity in a rack, right, with this approach. So now suddenly you can support frontier models in a single rack with this approach.

Nataraj: And because you have more memory, you don't have to make additional calls to pull the context from different places and that is reducing the inference cost. Got it. Got it. Talk to me about the you mentioned you made a couple of acquisitions. Like talk to me about those acquisitions and how those fit into what you're trying to do.

Sid: That's right. That's right. That's right. Yeah. Yeah, so the acquisitions were mainly done because of what I stated earlier, as a company we had to quickly transition from chip scale, card scale company to rack scale, solution scale company, right? Because customers were like, We need to deploy solutions. D-Matrix, you need to come to us with solutions not your technology, which is great. You have great technology. You've got great chips, you've got great, cards that can house those chips, but we don't knowhat to do with it, right? I mean So please come to us with solutions. And so the first acquisition we did was a company called GigaIO that happened in March, April of this year. And that was a rack-scale company. They were building rack-scale solutions for disaggregated compute. So they would buy GPUs, they would buy XPUs. They didn't build their own GPUs or XPUs, they just bought GPUs, XPUs, and they would partner with somebody like Supermicro. Was building the servers and they had a rack scale solution that they had built that they were deploying in the market. So I think in our case, that was perfect because we have a partnership with Supermicro on our first generation product where they are building the racks for us. Right. And bringing this team in into our company meant that we could quickly scale our first generation product into a rack scale solution with Supermicro, right, for deployment. Right. So it allowed us to go on that rack scale journey a lot faster. And then the second acquisition I did was a software company, which was building API endpoints. So this was a company that was orchestrating is essentially once you have compute available and a solution available, then you need to put the software on top of that solution to make an API endpoint available so that users like you and me can go use the AI model, right? So that software orchestration for an API is what they specialized in. And so we acquired that company because we wanted the ability to take our solution and quickly create an endpoint that customers could trial, right? Or investors could trial or partners could trial so that in developers mainly did it for developers, developers could trial that, right? So this would allow developers to come in and play with different models on our hardware and essentially create a playground where they can go and trial all this out. And then based on their experience, they would kind of map it to the right applications that they're trying to build. And we would be able to then sell them the hardware or the compute that they can deploy for those applications.

Nataraj: Well, I have a question about in general, like what's happening with AI hardware. There's obviously a lot of funding going into AI hardware post-ChatGPT, and especially I think last one and a half year we've seen plenty of startups that demonstrate almost like, hey, with this particular model like 10X or inference or some crazy number like that. Is that all going to play out or like how do you do you disagree with any of that going on like because you've been in 30 years, right? Like what it means to like show a demo versus like what it means to actually getting there or I'm just curious like what your general thoughts are on the investment cycle that's going on into the hardware chip stars.

Sid: So we are certainly we are certainly in an environment where the tide is up, right? Right now you if you are you if you're trying to surf a wave, it's much easier, right? Because the tide is up. And the need for inference is infinite in our mind, right? Because people have, tried to size up the inferencing market and they can't seem to put a number on it, right? Because And that's very it's kind of logical to me because just think about it, right? You're creating a new type of intelligence, you're growing a new type of intelligence, right? So you had human intelligence. We are like what 8 billion people, seven, seven, eight billion people on the planet. So you can you can you can size up the amount of human intelligence on the planet, right? But and that is a function of population growth, right? But when you look at AI or this alternate. Intelligence that we are creating and growing, there is really no limit on that. And how much of that do you need to answer some of the toughest problems that we as humans need to answer, right? I mean, we want to cure all these diseases. We haven't really explored deep space. Right? There's all these physical phenomena that we are trying to get down to the bottom of mathematical problems that we are now trying to kind of solve that are becoming easier to solve with all this intelligence available. So there are just a lot of questions that we want to answer and The need and the demand for intelligence seems infinite, right? So now in a world like that, one, two, three, four, five companies are not going to be able to serve all the needs. You need lots and lots of compute. And how do you how do you kind of serve a market of this size? So I can see why you're seeing such a huge interest from investors in investing in startups, in investing in new approaches. The need for compute is so massive that there could be many, many winners here. And that is why you're seeing a huge flood of interest from investors in this in this upcycle, right? And anyone who has a slightly different approach or a novel approach that shows that they can sh offer some benefit, some advantage, is going to get attention right now, right? Because of this tremendous disconnect between supply and demand. There's such a massive disconnect between supply and demand that anyone who has A reasonably good approach which is reasonably well thought out will be able to attract investment dollars.

Nataraj: What do you think next three to five years would look like for d-Matrix? Assuming things go in the way you are thinking about like do we think d-Matrix becomes both sort of like a neo cloud provider plus a systems company or you're primarily focused on, accelerating, deployment of gigawatts of compute and GPU clusters? Out there and d-Matrix is part of those architectures. How like what does what will success look like for d-Matrix in say like after five years?

Sid: Yeah. So in our vision has always been that we want to make d-Matrix one of the most consequential companies of this AI era, right? So with that we had a vision that someday in the future, at least fifty percent of the AI decisions made would run on or would touch d-Matrix, some d-Matrix infra or hardware, right? D-Matrix compute. And so we carry that vision. We have not, fulfilled that vision yet. And the way we make this company consequential is by attaining that vision is proliferating our compute into the into the world and diffusing it where anytime somebody is trying to make a decision using AI, they touch d-Matrix, right? And so that will require us deploying gigawatts and gigawatts of compute over the next three, five, ten years, right? And that is that is what we are working towards at d-Matrix. I mean scaling this company from where we are today into that opportunity is going to be a big mountain to climb. And we're kind of looking forward to climbing that mountain. And but once we get, once we get to the top of that, I think it will be it will make d-Matrix one of the most consequential companies of this generation, right? And that is that is what we're working towards. So that is really what success looks like for us. That is what we are working towards as a team.

Nataraj: Well, I think that's a good note to end the conversation. Said, thanks for coming on the show. I'm looking forward what d-Matrix will do. It is a very exciting time to be like a semiconductor company. So yeah, all the best. And thanks for coming in then.

Sid: Nataraj, thanks for having me. Wonderful chatting.