tensorturnA Distributed Training Platform for Your Own Machines
TensorTurn is a distributed training platform that combines several of your own machines into a single run — not a marketplace of strangers' GPUs. You connect each machine with a one-line outbound-only agent, then choose how they cooperate: ensemble mode, where every machine trains a diverse model on the full dataset and they vote, or fused mode, where one job is sharded across machines with periodic DiLoCo-style weight-averaging into a single final model. A work-stealing scheduler keeps every machine busy and requeues anything that drops.
Two ways to distribute a run
TensorTurn deliberately supports the two distribution patterns that work well over ordinary internet links between your machines, and is honest about the one it does not attempt. Both modes run on the pool of machines you have already connected.
| Ensemble mode | Fused mode | |
|---|---|---|
| What each machine does | Trains a full, diverse model on the whole dataset | Trains on its shard of the data |
| Coordination | None during training — fully independent | Periodic weight-averaging (DiLoCo-style) |
| Final output | Multiple models that vote together | One merged model |
| Network need | Very low — results sent at the end | Low — weights synced every N steps, not every step |
| Best for | Accuracy from model diversity | One deployable model, more data throughput |
| Failure handling | Missing model just drops from the vote | Shard requeued to another machine |
How a distributed run executes
- Connect two or more of your machines; each appears as a worker with its live telemetry.
- Pick ensemble or fused for the run.
- The scheduler assigns work — full-data copies for ensemble, shards for fused — and steals idle capacity so faster machines do more.
- In fused mode, machines periodically average their weights instead of exchanging gradients every step, keeping WAN traffic low.
- TensorTurn merges the result: a voting ensemble or one fused model, ready to deploy as an authenticated API.
Why weight-averaging instead of all-reduce
Classic multi-GPU training exchanges gradients every step with all-reduce, which needs the fast interconnect you find inside a single server or datacenter rack. Over the public internet that chatter would dominate and stall training. Fused mode instead uses periodic weight-averaging — each machine trains locally for many steps, then the weights are averaged — the same idea behind DiLoCo. It tolerates slow, high-latency links between homes and offices, at the cost of not being mathematically identical to single-node training.
Honest limits
This is embarrassingly-parallel ensembling or periodic weight-averaging over sharded data. It is not tensor parallelism, not pipeline parallelism, and not gradient all-reduce across the WAN.
- You cannot split one layer or one model across machines — each machine must hold the whole model in its own VRAM.
- A model too big for a single machine is not made to fit by adding more machines.
- This is your single-tenant pool of machines, not a marketplace and not web3/blockchain compute.
- Fused results approximate, rather than exactly reproduce, what one big GPU would produce; ensemble accuracy depends on genuine model diversity.
When to use which mode
Reach for ensemble mode when you want the accuracy bump that comes from averaging several different models and each machine can hold the whole dataset — it needs almost no network. Reach for fused mode when your dataset is large, you want one deployable model, and you would rather shard the data across machines. If you only have one machine, a normal single-GPU run is simpler and just as correct.
Frequently asked questions
Can I combine different GPUs, like a 3090 and a laptop 4060?
Yes. The work-stealing scheduler gives faster machines more work, so mixed hardware still contributes. Each machine only needs enough VRAM to hold the model on its own.
Does fused mode need a fast network between my machines?
No. Weights are averaged periodically rather than syncing gradients every step, so it tolerates ordinary high-latency home and office internet links.
Is this like training on 8 GPUs in one server?
Not exactly. In-server training uses all-reduce over a fast interconnect. TensorTurn uses ensembling or periodic weight-averaging, which is designed for slower links between separate machines and is not bit-for-bit identical.
What if one machine goes offline mid-run?
In fused mode its shard is requeued to another machine; in ensemble mode its model simply drops out of the vote. The run does not fail.
Can I rent other people's GPUs through this?
No. A distributed run only uses machines you connect yourself. It is single-tenant, not a GPU marketplace.