Open-weight models like Llama 3.1, Mistral, and Falcon are technically free to download. But running them at anything approaching production scale is a different story altogether.
The reason for this is that just a handful of hyperscalers and well-funded inference platforms basically control the GPU layer. So, if you need AI infra to serve your 70B parameter model (NVM3 storage, high-bandwidth interconnects, A100s or H100s in quantity), then your AI startup or LLM research project will run headlong into the hypercaler or inference platform gate. AWS, for instance, offers on-demand H100 SXM5 and A100 PCIe GPUs at $12.30/hr and $3.97/hr respectively. VOLT, by comparison, lists H100 SXM5 nodes at $2.79/hr and A100 PCIe at $1.49/hr.
As you can see, a gaping maw exists between a model being technically “free” and it being actually affordable. Let’s take a deep dive into the mechanics of the Inference Cartel, those handful of hyperscalers and inference platforms controlling access to and pricing for open-weight models.
Who actually controls the inference stack
The open-weight model ecosystem is now fragmented into three layers. Of these three, centralization is most visible at the bottom two layers.
Model layer
Meta, Mistral AI, Technology Innovation Institute, and others release weights under permissive or open licenses. A genuinely decentralized layer, the weights are on Hugging Face and anyone with a hard drive can download them.
Serving layer
Companies like Together AI, Fireworks AI, Replicate, and Anyscale have built managed inference APIs atop of those weights. While they abstract away infrastructure, the reality is that under the hood they're still running on AWS, GCP, or Azure. As you might expect, the pricing reflects it.
Infrastructure layer
AWS, Azure, and GCP sit at the bottom, and every managed inference provider ultimately rents from them. This is where the cartel logic actually lives and thrives. You can publish the weights, but if you don’t own the GPUs the weights run on, you’re not the real, foundational infra layer in centralized cloud computing.
The hyperscaler numbers normalize to roughly $12–13/hr per H100.
VOLT's per-GPU price is 4–5x lower because the supply comes from a distributed network of independent operators rather than a single datacenter owner.
Managed inference: simple deployment but with a markup
Managed inference APIs simplify deployment at the cost of margin transparency. On Together AI, a Llama 3.1 70B inference call costs approximately $0.0009 per 1,000 output tokens. The marketing numbers look cheap, that is, until you need to run a production workload.
Imagine a workload of 100M tokens/day inference endpoint:
- Daily token volume – 100,000,000 output tokens
- Together AI rate – $0.0009 / 1K tokens = $90/day = $2,700/month
- VOLT A100 80GB PCIe at $1.49/hr, running Llama 3.1 70B at ~4,000 tokens/sec on a single node –
- Tokens/day capacity per GPU – 4,000 × 86,400 = ~345M tokens/day
- GPUs needed for 100M tokens/day – 1 (with headroom)
- Monthly cost – $1.49 × 720 hours = $1,073/month
Choosing VOLT over Together AI amounts to 2.5x cost difference for the same workload. And that is before any volume discounts on VOLT's side. This also illustrates an important point: when you control the GPU, as you do when using VOLT’s GPU clusters, the per-token cost collapses.
Decentralized supply fixes compute’s cartel crisis
The inference cartel is a supply concentration problem. If GPU supply were as distributed as the model weights themselves, managed inference margins would compress to near zero. The alternative to the inference cartel is decentralized GPU marketplaces like VOLT.
VOLT's network aggregates GPUs from independent operators — data centers, mining farms, and institutional holders — into a unified cluster marketplace. Beyond its pricing advantages, distributed compute introduces true supply diversity through real competition, neither of which a hyperscaler can match; which, of course, they wouldn’t want to do anyway, as it would undermine their entire business model.
VOLT currently lists nodes across H100 SXM5, H100 PCIe, A100 SXM4, A100 PCIe, RTX 4090, and A6000 tiers. An engineer running a 7B model for a latency-tolerant batch job can rent an RTX 4090 for a fraction of A100 pricing. An engineer who needs top-end throughput for a 70B model can access H100 SXM5 nodes without signing a cloud commitment.
VOLT also creates a notable distinction from managed inference APIs: customers get raw GPU, not a markup-bearing API wrapper. You bring your own serving stack (vLLM, TGI, SGLang) and you control the runtime. While there might be a bit more setup work, the lower costs are profound, and you’ll never be forced to deal with some third-party sitting between your models and its users.
Why VOLT
VOLT was designed to exist as a real GPU supply solution outside of the hyperscaler stack. Remember: AWS, Azure, and GCP operate the infrastructure layer that every managed inference provider rents. VOLT operates an entirely parallel supply chain sourced from distributed GPU operators, so that its pricing is never set by hyperscaler wholesale rates.
For open-weight model inference specifically, this matters in two ways. First, the cost differences between VOLT and a hyperscaler like AWS have to do with two very different supply models. VOLT’s GPU cluster supply is distributed, global, and already online—meaning, you can get it right now, almost anywhere in the world, without waiting for a new AI data center to be built to enhance supply.
Second, there's no model dependency, so you're not locked into a specific model version or API contract. So, when Meta releases Llama 3.2, you can deploy it the same day on the same cluster on VOLT without waiting for Together AI or Replicate to update their API endpoints.
To outmaneuver the inference cartel, you must break free from the GPU layer that they control. This means accessing the distributed GPU supply available right now on VOLT.
Put your business first, not big tech's profits. Save up to 70%.
FAQ
What open-weight models can I run on VOLT GPU nodes?
Any open-weight model that fits in the GPU's VRAM can run on VOLT nodes. Common choices are Llama 3.1 8B (fits on a single RTX 4090 with 24GB VRAM), Llama 3.1 70B (requires multiple A100 or H100 nodes for full-precision inference), and Mistral 7B (single A100 PCIe at $1.49/hr is sufficient). VOLT's cluster builder lets you specify node type and count before committing. If you need quantized inference — GPTQ or AWQ — the 70B models compress to fit on two A100 PCIe nodes.
How does VOLT pricing compare to managed open-weight inference APIs?
Managed APIs charge per token and absorb the infrastructure margin. Together AI's Llama 3.1 70B rate is $0.0009/1K output tokens, which is workable for low volume but expensive at scale. At 100M output tokens per day, that's $2,700/month. Running the same workload on a single A100 PCIe node at VOLT's $1.49/hr runs approximately $1,073/month. The crossover point where direct GPU rental beats managed APIs is around 30–40M tokens per day for most 70B workloads.
Can I run open-weight model inference on VOLT without a long-term contract?
Yes. VOLT nodes are available on an hourly basis with no commitment. There are no reserved-instance requirements, no 1-year or 3-year lock-in, and no egress fees. You spin up the cluster, run the job, and release the nodes. This matters for inference workloads with uneven demand profiles like batch jobs, research runs, and production endpoints that scale with user traffic rather than a flat commitment.
What is the inference cartel and why does it affect open-weight models?
Open-weight models have freely available weights, but running them requires GPUs. The major GPU supply is concentrated among AWS, Azure, and Google Cloud, who set the floor on infrastructure pricing. Managed inference providers build on top of that supply and add their own margin layer. The result is that access to open-weight model inference — despite the model being free — is effectively priced and gatekept by the same three companies that control cloud GPU supply. Distributed GPU networks like VOLT represent the structural alternative.
Which GPU tier should I use for open-weight model inference on VOLT?
It depends on model size and latency requirements. Llama 3.1 8B in FP16 fits on a single RTX 4090 (24GB VRAM, most cost-effective option). Llama 3.1 70B in FP16 needs 140GB VRAM — two H100 SXM5 nodes at $2.79/hr each, or four A100 PCIe at $1.49/hr each. For latency-sensitive production endpoints, H100 SXM5 nodes deliver higher memory bandwidth (3.35 TB/s vs 2.0 TB/s on A100 SXM4) and generate tokens faster. For batch inference where throughput matters more than latency, A100 PCIe is the more cost-efficient choice.