Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity. Which makes it economically unreasonable unless you want privacy or something like that.
In each similar thread there are posts disputing the ROI - most often that's a minor positive side-effect of self-hosting. Independence, privacy and unchanging, uncensored models, that makes it already worth it.
Exactly, in fields like health, police, and military you really can’t accept the risks of your data leaking to a provider even if they claim to offer zero data retention.
Since when do software engineers complain about maintaining privacy, security or compliance?
I have very much the opposite experience, they trash the place like toddlers then complain relentlessly when our auditors or anyone else points it out.
Presumably they don't and op does. I think the engineers are complaining they want LLMs and op is stuck handing data over to a 3rd party or denying them access. This would be the compromise (self hosted LLM)
Presumably some organizations are able to contain the rapacious software engineer. In those organizations, the software engineer is better able to offer this. As it happens, though, almost all of those would rather use ZDR Claude over this kind of thing just because of the BCDR issues etc.
It seems like hosting your frontier models don't make any kind of economic sense at this point given the exorbitant premiums on hardware in this space right now.
What will prices look like once production has caught up with demand though?
If you’re doing any work near and dear to your heart, that you think is gonna have some type of significance, some type of value, if you don’t care about privacy and are not doing anything about it upfront, don’t cry on the back end.
For anyone actually considering this $200k+ setup for home or even small business use the power usage and noise is extreme. 1.6kw each with the fans to match. 600tokens/s at concurrency 16 is not what you need for home use or even sub 100 employee business use either. Really reconsider please (do you seriously need people to tell you this?). I get tech wages are high and you can probably afford this but seriously.
Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.
What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.
In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory is the blocker.
In 2027 there’s expected to be many multi terabyte HBF options coming. NAND/NOR flash is massively cheaper than RAM per byte (multiple states per logical unit so it’s denser) and it’s not that hard to wire it in a more parallel way to increase its bandwidth. Nvidia has the Kioxia solutions and Sandisk, Samsung, Tenstorrent, Mythic have also previewed HBF with GPUs to do local inference on the open source frontier models at reasonably high tok/s. Even if HBF is sold for 10x the price per terabyte of todays flash that’s still an extremely small fraction of the price of any other option right now.
Make do with the subscriptions for now. Don’t go crazy trying to run the frontier at home today. We should have something much better solutions are coming.
Curious about the >2 case. You get a Quantum X800 switch? Or just a fast Ethernet switch? These are quite cool because of the amount of VRAM on them (though I find the “unified RAM” marketing unnecessarily confusing) but I avoided them because I wasn’t sure I could network them. A B300 node is $500k but I think I’d do MI355X now at that size because new Claudes will be able to port easily.
The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.
There is something for macOS that uses Thunderbolt for high speed interconnect. Obviously it doesn't scale beyond a few nodes but I think that's a decent and fairly approachable middleground
This doesn't seem all that impressive for $200k of hardware? You could serve 4bit off 8xB6000 for ~half the price. No idea what throughput you'd get tho so maybe that's the flex.
Ultimately, that’s where most people want to be local and on their desktop. It keeps coming up over and over again. Most working in this area want privacy and want to work on their desktop or in a small cluster within a small team office environment. It keeps coming up over and over again. Rental is fine temporarily but it isn’t the end point no matter what Sam Altman wants, personal computers won out for a reason over the mainframe solutions of the past. Those reasons have not changed.
I mean, barely. If you want a video out you have to add another GPU which reduces the power the main GPU gets. And most offices don't have 800Gb fibre to the desk. The Supermicro version is rackmountable, which makes more sense.
> Is there a reason they chose glm and not deepseek
GLM-5.3's architecture (same as GLM-5.2) is simpler and it came out earlier.
Hybrid attention and Manifold-Constrained Hyper-Connections only arrived with GLM-5.3-Flash.
Also, NVIDIA acquired HuggingFace, who used GLM-5.2 for their forensic analysis when they got hacked, so I presume that they have more familiarity with that model.
And DeepSeek-V4.1-Flash is more memory-efficient, so it'd sell fewer GPUs.
DeepSeek’s claim to fame is massively shrinking the per-user memory footprint, enabling many more concurrent users in your fixed hardware footprint. Having more concurrent users leads to much higher overall throughput on that same hardware. That’s why DS is so cheap to host.
See table 2 in this article to see the tradeoffs of overall throughput vs per user latency as an example.
Deepseek would let you host higher concurrency than GLM.
Presumably that compression doesn’t come for free on the quality side. (This is totally speculative on my part. I haven’t personally used DS, and I don’t particularly feel benchmarks correlate well to my own experiences)
If you’ve got a small enough user count, perhaps there are tradeoffs in DeepSeek that are not buying you anything.
Having used both, they really are comparable for most things, which leaves the cheaper Deepseek a better choice for most folks. The one area that I've found GLM to be a clear winner is in offensive Cyber.
If you run a red team GLM is hard to beat, and most of the providers who want to serve an abliterated model cost significantly more so it makes more sense to spend money on hardware.
I'll get right on that after an elderly relative passes away and leaves me a valuable piece of real estate in a major city that will sell for enough money to buy two DGX stations in the current market. Or I win the lottery.
Eg I can put 2x of these workstations in an office handling banking/medical/secret/sensitive data and have my engineers stop complaining.
I have very much the opposite experience, they trash the place like toddlers then complain relentlessly when our auditors or anyone else points it out.
What will prices look like once production has caught up with demand though?
I've heard numbers between 10k-38k tokens per second per GB300 GPU.
Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.
What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.
In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory is the blocker.
In 2027 there’s expected to be many multi terabyte HBF options coming. NAND/NOR flash is massively cheaper than RAM per byte (multiple states per logical unit so it’s denser) and it’s not that hard to wire it in a more parallel way to increase its bandwidth. Nvidia has the Kioxia solutions and Sandisk, Samsung, Tenstorrent, Mythic have also previewed HBF with GPUs to do local inference on the open source frontier models at reasonably high tok/s. Even if HBF is sold for 10x the price per terabyte of todays flash that’s still an extremely small fraction of the price of any other option right now.
Make do with the subscriptions for now. Don’t go crazy trying to run the frontier at home today. We should have something much better solutions are coming.
The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.
For an enterprise, that price seems doable. But if you want 50-100 tps like Claude does, and you only have 10 users, maybe you're not an enterprise.
To pull a recent analogy, I guess we're still at the early Model S stage of LLMs.
Hybrid attention and Manifold-Constrained Hyper-Connections only arrived with GLM-5.3-Flash.
Also, NVIDIA acquired HuggingFace, who used GLM-5.2 for their forensic analysis when they got hacked, so I presume that they have more familiarity with that model.
And DeepSeek-V4.1-Flash is more memory-efficient, so it'd sell fewer GPUs.
See table 2 in this article to see the tradeoffs of overall throughput vs per user latency as an example.
Deepseek would let you host higher concurrency than GLM.
Presumably that compression doesn’t come for free on the quality side. (This is totally speculative on my part. I haven’t personally used DS, and I don’t particularly feel benchmarks correlate well to my own experiences)
If you’ve got a small enough user count, perhaps there are tradeoffs in DeepSeek that are not buying you anything.
If you run a red team GLM is hard to beat, and most of the providers who want to serve an abliterated model cost significantly more so it makes more sense to spend money on hardware.