ruCCL User Guide

Compute libraries · Tensors and frameworks · 中文 | 日本語 | Deutsch | Русский

1. Layers and entry points

The Cargo package is ruCCL and the Rust crate is ruccl.

ruCCL includes tensor Backend collectives, a rank core, and in-process implementations. ruda-communication provides communication infrastructure. The orchestrator feature enables orchestration entry points.

See ranks, devices and distributed training for explicit rank/device mapping, TCP rendezvous, initialization order, token-weighted replicas and rank-local recovery.

2. Tensor collective API

These registered interfaces use B: ruda_tensor::Backend and B::FloatTensorPrimitive. Register the inner backend for low-level calls; use ruda_autodiff::collective for explicit-rank tensor graph operations.

3. Registration and call contracts

Create configuration with CollectiveConfig::default(). Use with_num_devices for the number of local participating devices. Configure strategies and multinode addresses through their configuration methods.

Participants must agree on device counts, use unique peer IDs, and call matching collectives in the same order. Shape, reduction operation, root, and other parameters must agree. Each broadcast must have exactly one sender.

For multinode execution, configure node counts, global and local addresses, and data-service ports together.

4. Errors and lifecycle

CollectiveError covers duplicate or missing registrations, shape mismatches, inconsistent reduction operations or roots, and invalid broadcast sender counts.

Use finish_collective for normal completion. reset_collective forgets in-progress state; it does not complete an operation, checkpoint a device task, or provide lossless recovery.

5. CUDA example

The cuda feature enables the CUDA tensor backend. Run cargo run --locked -p ruCCL --features cuda --example all_reduce to execute Ring AllReduce with four logical ranks on GPU 0. It checks Sum/Mean over 257 FP32 elements, input preservation, and session exit.

Device adapters are in tensor_device. For the optimizer interface, see explicit-rank gradient reduction. Transfers include a host-staged path, not zero-copy P2P.

Source: collective API, configuration, rank, and in-process implementation.

6. Collective training

Enable collective in ruda-optim. With an explicitly owned rank communicator, convert backward gradients into GradientsParams, call grads.all_reduce_with::<InnerBackend>(&communicator, ReduceOperation::Mean)?, then pass the returned gradients to optimizer.step. Parameter IDs, gradient shapes, dtypes, and call order must match across ranks. For autodiff training, InnerBackend is the backend without the Autodiff wrapper.

Run the two-rank training example from the source tree:

cargo run --locked -p ruda-optim --features collective,cuda --example collective_training -- run ./collective-training-state
cargo run --locked -p ruda-optim --features collective,cuda --example collective_training -- resume ./collective-training-state

run requires a directory that does not yet exist. It saves each rank's model and optimizer after the first update, then executes the second update. resume restores that directory and executes the second update. With CUDA enabled, both logical ranks in this example use the same default device.

See the collective training example for the complete call sequence. To also save scheduler state and pending accumulated gradients, use TrainingRecord from Training and saving state.

7. Explicit-rank tensor collectives

RankCommunicator<TensorDevice<B>> provides floating and I32/I64 broadcast, all-reduce, all-gather and reduce-scatter. Gather concatenates equal axis-zero shards in rank order; scatter requires axis zero to divide evenly by world size. Integer tensors retain their width without floating-point conversion. This transport remains host-staged.

Use ruda_autodiff::collective for tracked all_gather, reduce_scatter_sum, reduce_scatter_mean, all_reduce_sum, all_reduce_mean and broadcast. The gather/scatter _dim variants accept arbitrary positive or negative axes. Forward and backward must have matching rank order, shapes, dtypes, gradient tracking and root. The training guide describes each backward rule and token-weighted DataParallel::reduce_fp32.

DataParallel<B, C> accepts a DataParallelCommunicator<B::InnerBackend>; rust-ascend supplies native HCCL tensor transport while retaining TCP only for training metadata. Changing transport does not change parameter IDs, frozen/tied semantics or token/sample weighting.