Hacker Noon · সাধারণ

Open Inference Networks: Democratic Access to Intelligence

২৩/১০/২০২৬

Weights you cannot run are a file

An open model on your disk is only a large file until something runs it. If the only place that can run it is a rented data center, you are still a tenant.

I come at this from edge work. Models on a drone have to run on the device, with no network at all. That makes me sympathetic to the local-first argument, and also impatient with claims that ignore physics. I wrote about that shift in my piece on edge AI.

There are three ways to run a model outside a hyperscaler. I will take them in order of how well they work.

Tier one: run it yourself

This is the part of the story that has clearly arrived.

Tools such as llama.cpp and Ollama made local inference a one-line install. Quantization, which stores each weight in about 4 bits, cut memory needs to roughly a quarter of the original with modest quality loss.

A compilation of published benchmarks from October 2026 gives a sense of current speeds. Each figure is one measurement, so treat them as indications.

Hardware	Model	Tokens per second
RTX 5090 (32 GB)	Llama 3.1 8B, 4-bit	150
RTX 5090 (32 GB)	Gemma 3 27B, 4-bit	47
Mac Studio M5 Max	27B-class model	65 to 69
DGX Spark (128 GB)	Llama 3.1 70B	4.4

The same source treats 30 tokens per second as the threshold for comfortable chat. By that measure, a model in the 8B to 30B range is fully interactive on one high-end consumer machine.

Memory is the hard limit. A 70B model at 4 bits needs about 35 GB for the weights alone, which is more than any single consumer graphics card holds. Laptops with 16 GB handle the 8B class. Phones run models in the 1B to 4B range.

Switching is easy, because Ollama serves the same API shape that cloud providers use:

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")

reply = client.chat.completions.create(
    model="gemma3:12b",
    messages=[{"role": "user", "content": "Summarize this log file."}],
)
print(reply.choices[0].message.content)

Change one URL and your application talks to your own machine. No subscription, no rate limit, and no data leaving the room.

The limit is quality. A 12B or 27B model is very capable at summarizing, drafting, extraction, and routine coding. It is not equal to the largest hosted models on long, multi-step reasoning. Anyone who says otherwise is selling something.

[YOUR EXPERIENCE: a short paragraph on a model you run locally or on-device, what it handles well, and where you still reach for a cloud API.]

Tier two: pool the devices

Now the harder claim. If one laptop cannot hold a frontier-size model, can a thousand laptops hold it together?

Do the arithmetic first. A 405-billion-parameter model at 4 bits is about 200 GB of weights. If each volunteer device offers 8 GB, you need at least 26 devices in a chain, each holding a block of layers. A prompt enters the first device, and the intermediate result is passed along until the last device produces one token. Then the whole chain runs again for the next token.

That is the design of Petals, the best-known project of this kind, which describes itself as running models "BitTorrent-style." It works. The project reports up to 6 tokens per second for a 70B model in the best case. That is usable for batch jobs and well below chat speed.

Three problems explain the gap.

Every hop costs time. Each token crosses every device in the chain, over home internet. A July 2026 review of Petals names bandwidth between nodes as the main documented obstacle, and describes the project's recent maintenance as limited.

Pooling adds memory, not speed. This is true even under ideal conditions. In benchmarks of the exo framework, four top-end Mac Studios linked by Thunderbolt 5 ran a very large model at 32.5 tokens per second, against 21.1 for one machine. Four times the hardware bought 1.5 times the speed, and that was over a direct cable link built for low latency. Ordinary Wi-Fi and broadband are far worse.

Peers can see your data. In a layer-split network, the device holding the first block receives your prompt in a form that can be read. Devices further along see intermediate values. This is weaker privacy than a cloud provider bound by a contract, not stronger. And as part three showed, you also cannot easily confirm that a peer ran the layers it claimed.

So the honest answer to "can consumer devices collectively host a frontier model" is: they can hold it, and they cannot serve it at a speed most people would accept. Pooling on a local network with fast links is a real option for a lab or a small office. Pooling across the public internet is, for now, a research result.

Tier three: pay people to run it

Petals relies on volunteers, and volunteer networks have a known weakness. Many people use them and few contribute.

Token-incentive networks try to fix this with payment. Bittensor is the largest example. In outline:

The network is divided into subnets, each focused on one task such as text generation.
Miners run models and answer requests.
Validators score the answers.
The network issues new tokens every day and splits them according to those scores.

This is a real mechanism for getting hardware online without a company buying it. It is the supply-side idea from part one, applied to inference.

It is also where I think the "subscriptions to tokens" story needs care, because it mixes two different things.

Metered payment means paying per request to whoever served it. That is a sound model. It is how every API already bills, and an open network can do the same with any currency.

Token emissions mean paying suppliers with newly created tokens, whether or not customers showed up. That is a subsidy. It gets a network started, and it lasts only while the token holds value.

The criticisms in the source I cited are mostly about the second:

A small group of large validators controls how rewards are distributed.
Subnet owners can run validators and miners inside their own subnet.
Competitive mining is estimated to cost $1,500 to $6,000 a month in GPU rental, which favors funded teams over individuals.
Judging answer quality is far harder to verify than validating a block.

For a buyer there is a simpler issue. A $20 subscription costs $20 next month. A token-priced service costs whatever the token is worth that day. Businesses budget in currency, and volatility is a cost.

None of this means incentive networks fail. It means the comparison is between a predictable service with a landlord and an open market with a subsidy and an exchange rate.

The case against

Data centers are more efficient. A cloud GPU serves many users at once by batching their requests together. A laptop serves one person and sits idle most of the day. Cost and energy per token favor the data center by a wide margin.

The best models are not open. The strongest systems are closed, and the largest open ones need hardware almost no individual owns. Open inference gives broad access to good models, not to the best ones.

Local is not free. A machine that runs a 27B model well costs thousands of dollars. That is several years of a subscription, paid upfront, for hardware that ages.

Peer-to-peer weakens privacy. Sending your prompt through strangers' machines is a step down from a provider with a legal agreement, unless the network adds protections that do not yet exist at scale.

Incentives can be captured. Any system that pays for scores invites people to game the scores. Concentrated validators are the predictable result.

Convenience usually wins. Most people will choose the option that needs no setup. That has been true of email, hosting, and storage.

Better small models cut both ways. They make local inference stronger, and they remove the main reason to split a huge model across a swarm in the first place.

Where this leaves us

I expected to write about swarms of phones running frontier models. The evidence pointed somewhere less dramatic and more useful.

Local inference is ready for private data, offline use, low latency, and predictable cost. This is where most of the real gain in access is.
Pooling on a local network makes sense when you need to fit a model that no single machine can hold, and you accept the speed.
Public peer-to-peer networks suit batch work and resilience. They are not yet a chat service.
Cloud APIs remain the answer when you need the strongest model.

The practical pattern is a hybrid. Run a small model locally for most requests and send the hard ones to a larger model elsewhere. That is how I would build a product today.

"Democratic access to intelligence" is arriving, mainly because a model that needed a data center two years ago now fits on a desk. That is progress in research and hardware. The networks and tokens are a second-order story so far.

Sources

মূল প্রতিবেদনে যান ↗