Models and local chat

Chat is the default Studio mode: pick a model, talk to it, keep sessions. The same packaged file can run locally, on a Compute host, or become the base for a fine-tune.

Model formats

Training exports a merged .smm by default. An adapter alone cannot load without its base; the merged model is the simplest thing to chat with.

Add a model

1. Training → Models → Find GGUF models. Search HuggingFace, inspect license, quantisation, size and gated status, then download to your Models folder with progress and resume.
2. Convert if needed. The converter is directly under the browser. It suggests the destination, verifies the result and refuses to overwrite without confirmation.
3. Pick it in Chat. The model chooser shows a "runs well / runs slowly / won't fit" badge based on this machine.
4. For a host, upload it. Settings → Compute → host card → Browse → Upload and use. Hosts keep models under /data/SharpMind/Models.

Local models live under %USERPROFILE%\SharpMind\Models. Adapters and checkpoints live under Training.

Local chat

• Saved sessions with titles, a system prompt and a tools drawer.
• Streaming replies; stop generation from the message controls.
• Per-session inference settings, with the effective value and the layer that supplied it.
• Model badges from the machine's memory and the model size.
• Long contexts truncate from the oldest turns; the session shows what was sent.

Chat works with a local .smm, a local GGUF where supported, OpenRouter, or a SharpMind Compute host. The provider picker is on the welcome and chat screens.

Inference tuning

Settings → Models and Settings → Model families expose the layered settings: System → Family → Model → Session. Each value shows its source, so you can tell whether a setting came from the family default or the session.

• Sampling: temperature, top-p, top-k, repetition penalty, max tokens.
• Context and KV-cache behaviour, where supported by the model shape.
• Tool-call recovery: Strict, Lenient or Loose. Chat and Work can use different defaults.
• Reasoning tags and prompt formatting through the family profile.

GPU inference

A Compute host with the GPU image runs inference on the device when the model shape is supported. The engine covers:

• RMSNorm or LayerNorm.
• RoPE or NoPE positional encoding.
• Gated or dense GELU FFN.
• LLaMA/Qwen and GPT-2/BERT-style decoders, within the arena limit.

/health reports the backend and the reason:

"accelerators": [{ "name": "ilgpu", "inference": true, "backend": "CUDA · cuBLAS" }],
"acceleratorReason": "Inference on ilgpu (CUDA · cuBLAS)."

An unsupported shape or a prompt that overruns the arena falls back to the CPU with the reason in /health. It does not fail the chat. Unlike training, GPU inference does not require RMSNorm or a gated FFN.

Compute API

A SharpMind Compute host serves one active model at a time, OpenAI-style. The token is required on every route except /health.

GET  /health
GET  /v1/models
POST /v1/models/activate       {"name":"model.smm"}
PUT  /v1/models/{name}?activate=true
POST /v1/chat/completions      {"stream":true, ...}
PUT  /v1/files/{path}
POST /v1/training/jobs

• One model per process. Activating another unloads the current one.
• Chat is OpenAI-compatible with streaming. Tool calling on the Compute API is not there yet.
• Uploads and training jobs are bearer-token protected; /health is open.
• Runpod's proxy has a 100-second response timeout; training returns 202 and is polled, so long jobs are fine.

Local, host or OpenRouter

Problems

• Model won't fit: use a smaller quantisation, a shorter context, or a host with more memory. The badge is advisory, not a hard stop.
• GGUF loads but behaves poorly: convert to .smm; the converter verifies the package and keeps the metadata.
• Wrong chat template: set the model family in Settings; "raw" output is usually a missing template.
• 401 from a host: the Studio token and container token differ. See tokens.
• Host model list is empty: upload a model, or activate one already in the host's Models folder.