Model Loading and Inference

Documentation · Training · 中文 | 日本語 | Deutsch | Русский

ruLLM provides model loading, tokenization, cached autoregressive generation, and request scheduling. Its Cargo package is ruLLM; its Rust import name is rullm.

Prepare a local model

Keep the model files in one directory:

Loaders accept local directories and do not download models. Select the loader for your architecture:

B is a tensor Backend; R is a device Runtime. The Qwen3.5 text loader uses a qwen3_5_text configuration with per-layer layer_types and rejects quantization configurations. Do not pass AWQ files to a floating-point loader.

Run the text generation examples

After setting up the environment, run these commands from the source root. The relative paths point to two local model directories:

$env:RUDA_CUDA_COMPILER = 'nvrtc'
cargo run --locked -p ruda-llm --features nvidia --example qwen2_generate -- ./models/qwen2 "Hello" 32 1

The Qwen3.5 example defaults to Ruda IR → PTX when RUDA_CUDA_COMPILER is unset. Use --release for optimized execution. Unsupported direct PTX operations return errors rather than falling back to NVRTC. The PTX version must match the target GPU and driver; see the PTX backend reference.

Remove-Item Env:RUDA_CUDA_COMPILER -ErrorAction SilentlyContinue
$env:RUDA_PTX_VERSION = '8.0'
cargo run --release --locked -p ruda-llm --features nvidia-ptx --example qwen35_generate -- ./models/qwen35 "Hello" 8 1

The CUDA C++ / NVRTC path remains available with an explicit selection:

$env:RUDA_CUDA_COMPILER = 'nvrtc'
cargo run --release --locked -p ruda-llm --features nvidia-ptx --example qwen35_generate -- ./models/qwen35 "Hello" 8 1

Arguments are the model directory, raw text prompt, maximum new tokens, and run count. The last two default to 32 and 1 for Qwen2, or 8 and 1 for Qwen3.5. Run count must be positive. Examples print JSON lines for loading and generation. Generation lines include text, generated_token_ids, and stopped_on_eos.

These examples encode the prompt directly without adding a chat template. For conversational models, prepare the prompt with render_chat as below.

Generate a chat reply in an application

For an application directory alongside the RUDA source directory, use these dependencies.

[dependencies]
rullm = { package = "ruLLM", path = "../RUDA/ruLLM", features = ["nvidia"] }
ruda-tensor-device = { path = "../RUDA/ruda-tensor-device", default-features = false, features = ["std", "cuda"] }
ruda-driver-cuda = { path = "../RUDA/ruda-driver-cuda", default-features = false, features = ["std"] }
half = "=2.7.1"

This complete src/main.rs accepts a model directory and user message as command-line arguments. Later function examples reuse its B alias, imports, and dependencies:

use half::bf16;
use ruda_tensor_device::cuda::{Cuda, CudaDevice};
use rullm::{
    GreedyGenerationConfig, Qwen2ChatMessage, Qwen2ChatRole,
    load_huggingface_qwen2_pipeline,
};
use std::{error::Error, io};

type B = Cuda<bf16, i32>;

fn main() -> Result<(), Box<dyn Error>> {
    let mut args = std::env::args().skip(1);
    let directory = args.next().ok_or_else(|| io::Error::other("model directory required"))?;
    let prompt = args.next().ok_or_else(|| io::Error::other("prompt required"))?;
    let device = CudaDevice::default();
    let pipeline = load_huggingface_qwen2_pipeline::<B>(&directory, &device)?;
    let messages = [Qwen2ChatMessage::new(Qwen2ChatRole::User, prompt)];
    let prompt = pipeline.render_chat(&messages, true)?;
    let output = pipeline.generate_text(
        &prompt,
        GreedyGenerationConfig {
            max_new_tokens: 32,
            eos_token_ids: pipeline.loaded.default_eos_token_ids.clone(),
        },
        false,
        true,
        &device,
    )?;
    println!("{}", output.generated_text);
    Ok(())
}

render_chat handles plain-text Qwen2/Qwen2.5 ChatML messages. Passing true adds the assistant generation prefix. Roles also include System and Assistant; supply messages in conversation order. This API does not accept tool-call messages.

The two boolean arguments to generate_text are add_special_tokens and skip_special_tokens. Do not add special tokens again after rendering ChatML; decoding can skip them. generated_text contains only the new reply, while text includes both prompt and reply.

Temperature, top-k, and top-p sampling

Pass the rendered prompt to generate_text_sampled:

use rullm::{
    GeneratedText, HuggingFaceLoadError, HuggingFaceQwen2Pipeline,
    SamplingConfig, SamplingGenerationConfig,
};

fn sample_text(
    pipeline: &HuggingFaceQwen2Pipeline<B>,
    prompt: &str,
    device: &CudaDevice,
) -> Result<GeneratedText, HuggingFaceLoadError> {
    pipeline.generate_text_sampled(
        prompt,
        SamplingGenerationConfig {
            max_new_tokens: 32,
            eos_token_ids: pipeline.loaded.default_eos_token_ids.clone(),
            sampling: SamplingConfig {
                temperature: 0.8,
                top_k: 40,
                top_p: 0.9,
                seed: Some(42),
            },
        },
        false,
        true,
        device,
    )
}

Use generate_text for maximum-logit decoding rather than setting temperature to zero. A fixed seed fixes the sampling random sequence, not numerical consistency across models or backends.

Load AWQ weights

The AWQ loader takes CudaRuntime, not Cuda<bf16, i32>, as its generic parameter:

use ruda_driver_cuda::CudaRuntime;
use rullm::{HuggingFaceAwqQwen2Pipeline, load_huggingface_awq_qwen2_pipeline};

fn load_awq(
    directory: &str,
    device: &CudaDevice,
) -> Result<HuggingFaceAwqQwen2Pipeline<CudaRuntime>, HuggingFaceLoadError> {
    load_huggingface_awq_qwen2_pipeline::<CudaRuntime>(directory, device)
}

The returned pipeline also provides render_chat, encode, decode, generate_text, and generate_text_sampled. AWQ configuration requires quant_method = "awq", bits = 4, version = "gemm", and zero_point = true. group_size must be positive or -1. Linear projections use F16 activations. See ruBLAS INT4 for packed weight layouts.

Qwen3.5 image input

generate_image_files_greedy takes token IDs for one prompt, JPEG/PNG paths in prompt order, an image processor, and generation options:

use rullm::{
    Qwen35ImageProcessor, TokenGenerationOutput,
    load_huggingface_qwen35_multimodal,
};

fn generate_from_images(
    directory: &str,
    prompt_ids: &[i32],
    paths: &[&str],
    device: &CudaDevice,
) -> Result<TokenGenerationOutput, Box<dyn Error>> {
    let model = load_huggingface_qwen35_multimodal::<B>(directory, device)?;
    let processor = Qwen35ImageProcessor::from_huggingface(directory)?;
    let output = model.generate_image_files_greedy(
        prompt_ids,
        paths,
        &processor,
        &GreedyGenerationConfig {
            max_new_tokens: 32,
            eos_token_ids: vec![model.text.config.eos_token_id],
        },
    )?;
    Ok(output)
}

Encode prompt_ids with the model's tokenizer.json. Follow the model's prompt template, keeping one placeholder token per image using image_token_id from config.json. Image paths must match placeholder count and order. This entry point expands placeholders internally; do not expand them beforehand. It handles still images and rejects video tokens.

For existing RGB8 data, use Qwen35RgbImage { pixels, height, width } with generate_rgb_greedy. Pixels are interleaved RGB bytes without row padding. For separate preprocessing, preprocess_files and preprocess_rgb return FP32 patches, a two-dimensional shape, and per-image grids.

When serving multiple requests, load the model and processor outside the request loop. Decode the returned generated_token_ids with the same tokenizer.

Caches and continuous batching

Single-request generation functions create and maintain a cache for that generation. To drive Qwen3.5 manually, call model.new_cache(), pass the prompt tensor to forward_cached_last(tokens, &mut cache), then pass only new tokens. Do not resubmit the entire cached prefix.

To integrate your own batch executor, use ContinuousBatchScheduler:

1. Create it with ContinuousBatchScheduler::new(config, kv_config). ContinuousBatchConfig sets max_active_sequences and max_batch_tokens. PagedKvCacheConfig sets block_size, num_pages, and max_sequence_length. All must be positive.
2. Submit requests with submit(prompt_token_ids, generation) and retain each returned RequestId.
3. Call schedule() for a ScheduledBatch. Its kind distinguishes Prefill/Decode; sequences supplies each row's tokens, start position, context length, and block table.
4. Run the model in row order, write the corresponding KV pages, and select each row's next token. On success, call complete_batch(batch.id, &generated_token_ids) with one result token per row. On failure, call fail_batch(batch.id) to cancel that batch's page reservations.
5. Retrieve completed requests and generated tokens with pop_finished(), then continue scheduling.

The scheduler manages requests and page-allocation metadata. The caller's executor owns device KV storage and model execution. Use token_matrix(), context_lengths(), and flattened_block_table() to construct batch inputs. Complete or cancel an outstanding batch before calling schedule() again.

API reference: ruLLM exports, Generation options, Scheduler.

Qwen3.5 fused paged attention

Qwen35BatchCache selects its attention path when new_batch_cache creates it. Set RUDA_PAGED_ATTENTION=fused before cache creation to use ruDNN's fused paged attention in Qwen3.5 continuous batching. Unset or legacy retains the existing path; other values are rejected. Changing the variable does not switch an existing cache, and it is not a global switch for every generation entry point.

The fused path keeps a physical KV arena per full-attention layer and uses the scheduler's page tables for prefill/decode without gathering historical KV into padded tensors. Arena dimensions use num_pages, block_size, KV heads and head dimension; reserve GPU memory for that capacity. Inputs follow ruDNN's contiguous F32/F16/BF16 contract. Shared-prefix writes require scheduler-level copy-on-write. After a failed device append, rebuild the cache rather than continuing from its previous position.

Reusable attention and MoE components

rullm::gpu_inference exposes TensorOps, gqa_scores, gqa_value_product, mla_absorb_query, mla_scores and mla_value_product. These compose caller-supplied device tensor operations without repeating cached GQA heads or expanding historical MLA latent values. The caller supplies masking, softmax, positional encoding and model-specific scaling.

rullm::device_inference re-exports ruDNN's HostPlan, DevicePlan, PagedAttentionError, SwiGluExperts, RoutingOptions, GroupRoutingOptions, route_sigmoid_grouped and GroupedStrategy. These are building blocks, not a complete MLA/MoE model loader; adapters still provide projections, routing configuration and cache ownership.