Hivemind — learning-at-home

When it fits

Consider Hivemind when developing a PyTorch training system that must coordinate across independent machines, tolerate slow or unresponsive peers, or experiment with decentralized parameter averaging and mixture-of-experts models.

Who it is for

• Researchers and developers building collaborative or volunteer-computing training systems.
• PyTorch teams exploring training across machines with slow or unreliable network peers.
• Developers adapting existing training pipelines through the documented PyTorch Lightning integration.

Documented uses

• Coordinate a model-training workload across computers without relying on a master node.
• Aggregate parameter updates across workers without synchronizing across the entire network.
• Distribute parts of neural-network layers among participants using Decentralized Mixture-of-Experts.
• Explore a multi-peer training example or collaborative Transformer training through the project tutorials.