Llama Crab Emoji

Topics:

Candle: a Minimalist ML Framework via @MasonStallmo

Slides: candle-2025-07-09.pdf

Candle is a lightweight, fast ML framework written in Rust that aims to provide a familiar PyTorch-like experience while addressing the performance and deployment limitations of Python-based frameworks. Created by Hugging Face, Candle is designed to be particularly well-suited for serverless ML deployments and browser-based inference.

Key Features and Performance

Candle significantly outperforms traditional Python frameworks in both memory usage and speed. According to benchmark data, Candle uses only 3.2GB peak RAM compared to torch-rs's 4.7GB, with substantially lower memory growth (18 MB/min vs 42 MB/min). Performance benchmarks show Candle executing BERT Base in 8.3ms compared to torch-rs's 15.7ms, and LLaMA 2 7B inference at 45.2ms/token versus 72.8ms/token.

// Example: Basic tensor operations in Candle
use candle_core::{Device, Tensor};

let device = Device::Cpu;
let a = Tensor::new(&[[1f32, 2.], [3., 4.]], &device)?;
let b = Tensor::new(&[[5f32, 6.], [7., 8.]], &device)?;
let result = a.matmul(&b)?;

Unique Advantages

The framework's most distinctive feature is its ability to compile models to WebAssembly (WASM), enabling ML inference directly in browsers without server dependencies—something impossible with PyTorch. This makes Candle particularly valuable for privacy-conscious applications and edge computing scenarios. The framework provides all essential ML components including model structure, weight serialization, training capabilities with optimizers and data loaders, backpropagation, and inference engines.

// Example: MNIST training loop structure
for epoch in 1..=epochs {
    let mut sum_loss = 0f32;
    for (bimages, blabels) in train_iter {
        let logits = model.forward(&bimages)?;
        let loss = loss::cross_entropy(&logits, &blabels)?;
        optimizer.backward_step(&loss)?;
        sum_loss += loss.to_vec0::<f32>()?;
    }
}

While Candle trades some of PyTorch's extensive feature set for simplicity and performance, it maintains familiar APIs that make it approachable for PyTorch users. The framework supports popular models and can be explored through various Hugging Face demos showcasing YOLO, Whisper, LLaMA 2, and other models running entirely in the browser.

🦙🦀 Tauri-Served Local LLMs with Mistral.rs via @DanielPBank

Repo: https://github.com/danielbank/tauri-mistral-chat

Daniel built a simple desktop chatbot demo with Tauri (a cross-platform framework), React (frontend JS framework), and mistral.rs (a cross-platform, highly-multimodal inference engine written in Rust). The demo integrates Mistral AI models for local inference.

Hidden Gems in the Mistral.rs Documentation

The mistral.rs Docs can be a little hard to navigate. Here are a few things Daniel found REALLY HELPFUL:

Rust Examples!

The Rust examples are all here and are a good starting point for simple programs that demonstrate the models

❗ Chat Templates (IMPORTANT)

You will need to specify a chat template (e.g. mistral.json) with your model builder:

   builder = builder.with_chat_template(template_path);

These templates are readily available in the mistral.rs repo, but you have to look for them: https://github.com/EricLBuehler/mistral.rs/tree/master/chat_templates

As an aside, you can use remote tokenizer as backup: .with_tok_model_id("mistralai/Mistral-7B-Instruct-v0.1")

Model Management

Downloading Models

The demo features two examples in ./src-tauri which are meant to demonstrate Mistral.rs inference with a Local LLM without the added complexity of Tauri. The first example downloads relevant models and the second example runs the inference:

cd src-tauri
cargo run --example download-models list
cargo run --example download_models download llama-vision --force --yes
cargo run --example hello_world

Instantiating a Model

Each model type has a different builder. For example, the TextModelBuilder is used for text-based models:

async fn load_remote_smollm3_model() -> Result<mistralrs::Model, String> {
    println!("Loading remote SmolLM3 3B model...");

    // Build the remote SmolLM3 model using TextModelBuilder
    let model = TextModelBuilder::new("HuggingFaceTB/SmolLM3-3B")
        .with_isq(IsqType::Q8_0)
        .with_logging()
        .build()
        .await
        .map_err(|e: anyhow::Error| format!("Failed to build remote SmolLM3 model: {}", e))?;

    println!("Remote SmolLM3 model loaded successfully!");
    Ok(model)
}

Different formats of the same model type also have different builders. For example, to load a text-model using the UQFF format, you would use the UqffTextModelBuilder:

async fn load_remote_llama_uqff_model() -> Result<mistralrs::Model, String> {
    println!("Loading remote Llama 3B UQFF model...");

let model = UqffTextModelBuilder::new(&model_path, uqff_files)
    .into_inner()
    .with_isq(IsqType::Q5_0)
    .with_logging()
    .build()
    .await
    .map_err(|e: anyhow::Error| format!("Failed to build Llama UQFF text model: {}", e))?;

println!("Llama UQFF text model loaded successfully!");
return Ok(model);

Inference

Once you have a model, you can run inference with the Model struct. For example, to run inference with a text-model, you would use the Model::chat method:

async fn run_inference(model: mistralrs::Model, message: String) -> Result<String, String> {
    let messages = TextMessages::new()
        .add_message(
            TextMessageRole::User,
            &format!("You are a helpful AI assistant. Keep your responses concise and friendly.\n\n{}", message)
        );

    model
        .send_chat_request(messages)
        .await
        .map_err(|e| format!("Failed to send text chat request: {}", e))?
}

Integration with the Frontend

The Tauri application exposes commands to the frontend JavaScript: discover_models and ai_chat. The frontend is a React app that uses the Tauri API to call the Rust functions:

// Call Tauri backend
console.log("Calling Tauri backend with:");
console.log("- message:", message.content);
console.log("- modelId:", modelId);
console.log("- hasImage:", !!imageData);
console.log("- imageDataLength:", imageData?.length || 0);

const response =
  (await invoke) <
  string >
  ("ai_chat",
  {
    message: message.content,
    modelId: modelId,
    imageData: imageData,
  });

console.log("Received response from Tauri backend:", response);

Universal Quantized File Format (UQFF) and GGML Universal File (GGUF) Format

What is UQFF?

Think of UQFF as a new way to package AI models so they run faster and use less computer memory. It's like having a ZIP file specifically designed for AI models. Specifically, it uses a technique called "quantization" to compress AI models to make them smaller and faster - kind of like how you might compress a video file to make it smaller.

What Makes UQFF Special

  • One File, Multiple Options - Instead of having separate files for different compression levels, UQFF lets you pack multiple compression types into one file. It's like having a ZIP file that contains both the HD version and the compressed version of a movie.

  • No More Waiting - Previously, if you wanted to use a compressed AI model, you had to wait for your computer to compress it first (which could take a while). With UQFF, someone already did the compression work for you - you just download and use it.

  • Works with Many Types - It supports different compression methods (they have nerdy names like Q4_0, Q8_1, etc.) but basically just think of them as different quality/speed settings.

What is GGUF?

GGUF stands for "GGML Universal File" (or sometimes "Generic GPT Unified Format") - it's a way to store AI models that makes them run faster and use less memory on regular computers like yours. It's essentially a special compression method that squishes models down so they can run on your laptop or desktop computer instead of needing a supercomputer.

What GGUF Does

  • Compresses big AI models so they can run on CPUs or low-power devices
  • Enables running complex models on everyday hardware like CPUs
  • Optimized for quick loading and saving of models, making it highly efficient for inference purposes

Advantages

  • One file format, one compression method
  • Very popular and widely supported
  • Works great, but limited to just GGUF-style compression

Crates you should know

mistralrs = { git = "https://github.com/EricLBuehler/mistral.rs.git" }