LLM on dgx01 — guide

Getting started

1. Log in to the console with your HPC cluster credentials (you must be in the Hpc.Users group).
2. Check whether the model you need is already running: shared models can be used by anyone, without starting another one.
3. If not, use the picker ("Which model?"): choose the type of work and start the first suggestion.
4. Use it from the console, or from your own programs with an API key (see Use from programs).
5. When you are done, stop it: GPUs are few and shared.

Single address: — compatible with OpenAI (/v1), Anthropic (/v1/messages) and Jev (/v1/systemone).

Concepts

GPU quota

dgx01 has 7 Tesla V100 32 GB GPUs, shared with the whole cluster. The LLM service can use at most 2 at a time in total (not 2 per user: there is a single quota, because all jobs run as the service user hpc.cluster). If the quota is full, a new request waits in the queue in SLURM until a GPU becomes free.

Dedicated job and pod

• Dedicated job: one model with its 1 or 2 GPUs all to itself. This is the choice for large models and for "exclusive" engines.
• Pod: a job with 1 or 2 GPUs in which several models are started, even by different users, as long as there is GPU memory. Example: Qwen3.6 on llama.cpp (22 GB) + d1-3B (6 GB) on the same V100, using a single GPU of quota. Starting a model inside an existing pod does not use quota.
• A pod left without models for 5 minutes shuts down on its own and releases its GPUs.
• The physical GPU of a pod or dedicated job is chosen by SLURM among the free ones; inside pods you choose the pod and GPU, or "automatic" (the fullest GPU the model still fits in, to keep the others free).
• Exclusive engines (vLLM, strata, ds4) take all the free memory on the card: they must go on an empty GPU. llama.cpp, ollama, colibri and the decision models share.

Shared or private

A started model is visible to everyone. If it is shared (the default), anyone who is logged in can use it; if you start it as private, only you can use it (and the administrators); a private model can use at most 1 GPU. Only the person who started it, or an administrator, can stop it.

Duration (walltime)

Started during the day (08–20): at most 4 hours. Started in the evening or at night: at most 12 hours. When the time runs out the job ends and the model shuts down.

Which model to use

Speeds measured on dgx01 with one user at a time: "prefill" is reading the prompt, "decode" is writing the answer. A token is about ¾ of an English word, less in Italian.

Model cards

Log in to the console to see the current catalog here.

Engines

The same model can run on different engines, with very different performance (e.g. Qwen3.6-35B: prefill 3386 tokens/s on strata, 638 on llama.cpp, 17 on colibri). The profile chosen at startup determines the engine.

Use from programs

Create an API key in the console (section "Use them from programs" → New API key) or with llm-serve key create <name>. The key is shown only once: save it. It does not expire with logins; you revoke it from the console.

curl

Python (openai library)

opencode

The model field is the id of the running model (e.g. qwen3.8-27b), or id@profile for a specific instance, or auto (the best active chat model). GET /v1/models lists the ones you can use. The certificate is self-signed for now: clients must accept it (e.g. curl -k, verify=False).

From a terminal on the cluster

/opt/common/tools/llm-serve/bin/llm-serve login        # HPC cluster credentials
llm-serve gpus                 # the 7 GPUs, who is using them, the pods
llm-serve pick documents       # what to use for long documents
llm-serve start gemma-4-26b-a4b            # automatic: in a pod if there is room, otherwise a dedicated job
llm-serve start d1-3b --pod <id> --gpu 3   # in a specific pod/GPU
llm-serve start qwen3.8-27b --dedicated --private
llm-serve status ; llm-serve stop <job> ; llm-serve logs <job>
eval $(llm-serve env)          # OPENAI_BASE_URL and OPENAI_API_KEY for other programs

Decision models (FDM)

d1, GLiNER2.5-Decide and Laya do not write text: they receive a state (text, JSON, images for d1) and typed questions, and answer in a single pass with calibrated probabilities. Ideal for classifying, routing, triage and extracting fields. The API is POST /v1/systemone, the same as Jev (TypeSafe): a Jev client works by changing only the base URL.

Prefill and decode

Every answer has two phases: prefill (the model reads the whole prompt; this matters for long documents and RAG) and decode (it writes the answer one token at a time; this matters for perceived speed). Engines are better or worse at each phase.

Can two different engines be used, one per phase?

Not on the same request: the internal state produced by prefill (the "KV cache") has a different format for each engine and cannot be passed from one to the other. The alternatives, in order of usefulness on dgx01:

1. One engine that is good at both phases. strata with Qwen3.6-35B has the best prefill (3386 tokens/s) and a decode almost equal to the best (98 versus 106 for llama.cpp). For long documents this is the choice to make: profile strata-1gpu.
2. strata with prefill and decode on two GPUs (--split-serve PREFILL,DECODE): the same engine starts two processes, one reads the prompt on one card and the other writes the answer on the other; the session passes from one to the other through RAM, automatically. It uses both GPUs of the quota for a single model. Profile strata-split-2gpu of Qwen3.6-35B: being tested.
3. Two instances of the same model and automatic routing. If the same model is running twice (e.g. Qwen3.6 on strata and on llama.cpp), the gateway sends requests with long prompts (over ~4000 tokens) to the instance with the faster prefill and the others to the one with the faster decode. Each request is served entirely by one engine. How to do it: start the same model with two different profiles and use as model the id without a profile (qwen3.6-35b-a3b). It costs 2 GPUs of quota, or 1 GPU + CPU if the second instance is llamacpp-cpu.
4. Speculative decoding: a small model "proposes" tokens and the large one verifies them in groups; it speeds up decode without changing the result. It is already active, automatically, on strata (MTP) for Qwen3.6 and Qwen3.8-Flash-Next. llama.cpp (-md <draft-model.gguf>) and vLLM support it but are not yet configured in the catalog.

Speculative decoding

A small model (or the MTP heads already inside the model) proposes a few tokens; the large model checks them all in one pass and keeps the right ones. The answer is identical to the one without speculation: only the writing speed (decode) changes. Prefill does not change.

Average over four texts (code, summary, free text, translation), one user. The -mtp profiles use 1–5 GB more VRAM. The other techniques tried help little on the V100s: EAGLE3 +10% on gpt-oss-20b and −10% on gpt-oss-120b (experts in RAM); n-gram helps only when the answer repeats the prompt; vLLM 0.8.5 with a draft model +27% on average but very variable. Details: BENCHMARK-dgx01.md §14 and the tests in the catalog (click the model).

With several users at once (MTP, 16 slots): the gain shrinks as load grows. Qwen3.6 +46% with 1 user, +4% with 4, −38% with 16; Gemma +52% → +9% → −27%; Qwen3.8-27B (dense) +88% → +33% → +25% with 8 users (with 16 users and MTP it runs out of GPU memory). So: -mtp profiles for private use or few users (up to ~4); for a heavily shared model the profile without speculation is better.

Subtypes

In the catalog the Speculative column shows the subtype on in each profile.

How the data is measured

All measurements were taken on dgx01 between 27 October and 2 November 2026, with the commands and profiles in the catalog. The full details (exact commands, dates, SLURM jobs) are in the file BENCHMARK-dgx01.md in /opt/common/tools/llm-serve/.

Long documents (16k–256k tokens)

Each model was tested with prompts of 16k, 32k, 64k, 128k tokens and up to its maximum: the results are in the drop-down that opens when you click on the model name in the catalog ("long context" row). In short: the longer the text, the slower the reading (a 128k document takes 1.5 to 6 minutes before the answer); for texts over 64k the best are Qwen3.8-Flash-Next on strata (over 1100 tokens/s of reading up to 256k) and Lingshu / Qwen2.5-VL on vLLM with 2 GPUs (up to 128k); for Qwen3.6 over 32k, llama.cpp is the better option. On a single GPU the vLLM models reach ~32k, the d1 decision models 8k (d1-3B) and 32k (d1-2B).

Context window

Each model has a maximum window (read from its metadata: from 32k to 1M tokens) and each profile a configured window, i.e. how many tokens (prompt + answer) a request can use in that service: usually 32k, because larger windows take memory away from the GPU. In the catalog the "Context" column shows configured / maximum. A request longer than the configured window is rejected by the engine.

Cold start, warm start and first token

"Cold" is used with two different meanings; it is best to keep them apart.

1. The model is not running: it has to be started

If the model you need is not already active, it must be started before you can use it (from the console or with llm-serve start): this takes from a few seconds to a few minutes. The service does not start models on its own at the first request: a call to a stopped model receives an error telling you to start it. If instead the model is already active (even if started by a colleague), the request starts immediately and only the time to first token counts.

2. "Cold" or "warm" start: where the model files are

In the catalog, the two startup columns refer to this:

• Cold start: the model files are not in the node's memory (RAM) and must be read from the BeeGFS storage. This happens when the model has not been used for a while, or when the RAM has been needed for something else.
• Warm start: the model was used recently, so the operating system still keeps its files in RAM and reads them again from there, without going through the storage. This happens automatically; the files stay in RAM until the memory is needed by other programs.

What happens during startup

Measured result: on dgx01 cold and warm differ by a few seconds, and in some tests the warm start was even slower, because of the variable load on the node. The reason is that the BeeGFS storage is fast and its servers have their own cache, while the long part of startup is engine initialization, which does not depend on where the files are. The difference becomes visible only with very large models (80–90 GB).

Time to first token (TTFT)

With the model already active, a request first waits for the prompt to be read (prefill) and then receives the answer one piece at a time. TTFT is the time until the first piece: with a short question it is under one second for almost all models; with a 2000-token document it ranges from 0.4 s (vLLM) and 0.7 s (strata) to 3–4 s (large models on llama.cpp). The values are in the "Startup" and "TTFT" columns of the catalog.

In practice: it is best to keep frequently used models active and shared (no startup, only TTFT); a model stopped and restarted on the same day usually starts "warm"; startup times matter mostly for vLLM (2–3 minutes) and for the largest models.

Model loading and cache

Models are read directly from BeeGFS (/beegfs/datasets/models), without copying to the local disk. Measurements on dgx01:

A cache on the local disk would save at most a few seconds (Qwen3.6, 23 GB: ~3 s on a 25–35 s startup) and would require copying in advance (~80 s for 23 GB). That is why there is none: the effective cache already exists and is automatic — the node RAM (503 GB) keeps recently used models and frees the ones unused for the longest time on its own, and the BeeGFS servers do the same.

SWE benchmarks (code quality)

Two public leaderboards measure how well a model can solve real software development tasks:

• SWE-bench Verified (llm-stats.com): percentage of 500 real GitHub issues solved; the values are reported by the model makers, not verified.
• DeepSWE (deepswe.datacurve.ai): pass@1 on 113 long tasks, all with the same agent (mini-swe-agent).

In the catalog the "SWE %" column shows SWE-bench Verified / DeepSWE. As checked on 2 November 2026, the two leaderboards list only Qwen3.6-35B-A3B (73.4 / n/a), DeepSeek-V4-Flash (79.0 / 53) and GLM-5.3-Flash (n/a / 63); the other models in the pool do not appear (the pages show only part of the entries). Note: these are the results of the original full-precision models with the maker's agents and settings, not measurements taken on dgx01 or on the quantized versions we use.

Loading and testing a new model (LOAD/TEST)

In the console, section Load / test a model: paste the HuggingFace address of the model (for GGUF repositories you can specify the quantization, e.g. *Q4_K_M*; gated models need a HuggingFace token, used only for the download and not stored).

1. Analyze (without downloading anything): reads the format, architecture, size, quantization and maximum context from the metadata, and for each engine says whether it can work on the V100s and why (architectures supported by vLLM 0.8.5 and by llama.cpp, unsupported quantizations, memory required).
2. Download and test: downloads the model into the pool, runs on each candidate engine the same test and benchmark used for the catalog (one SLURM job per engine, up to 45 minutes) and shows the outcome, prefill, decode, TTFT, startup, VRAM and any errors.
3. Profiles that pass the test are added to the catalog (marked "user") and the model can be started like the others. The scores per type of use are provisional until an administrator reviews them.

Limits: at most 2 tests at a time; models over 200 GB for administrators only; tests use the same 2-GPU quota as the models in service (if it is full they wait in the queue). vLLM needs the original files (safetensors); llama.cpp and strata need a GGUF repository.

Technology

• Website and API: a Python service (FastAPI + uvicorn, HTTPS) running as hpc.cluster on dgx01; pages in HTML and JavaScript without frameworks. HPC cluster login via PAM/AD.
• Engines: all in Singularity containers (/beegfs/datasets/llm/images/*.sif), started as SLURM jobs: vLLM and ollama from official images, llama.cpp, strata, colibri, DwarfStar and the decision model server recompiled for the V100s (CUDA 12.4, sm_70). No engine is installed on the system.
• Models: on BeeGFS, /beegfs/datasets/models/{llm,fdm}. Catalog: /opt/common/tools/llm-serve/catalog.yaml (+ catalog.d/ for models added with LOAD/TEST).

V100 limitations

• No hardware bf16 or FP8: models run in fp16 or quantized; FP8/NVFP4 weights (e.g. Qwen3.6-27B-FP8) cannot be used.
• Recent vLLM, SGLang and TGI no longer support these GPUs: vLLM is stuck at 0.8.5, so the newest architectures run in GGUF (llama.cpp, strata).
• Very large models (DeepSeek-V4, GLM-5.3) run only experimentally and very slowly.

Frequently asked questions

I see a model started by a colleague: can I use it?

Yes, if it is shared (not marked "private"): just use its id. You cannot stop it.

Can I load my own model on the GPU where another one is already running?

Yes, if it is inside a pod and there is memory: the "Start" window shows the pod GPUs the model fits in. Otherwise you can use another GPU (a new pod or a dedicated job), if the 2-GPU quota is not already full.

Where is my API key?

You create it in the console (or with llm-serve key create) and you see it only at that moment. The service keeps only a fingerprint (hash) of it, so it cannot show it to you again: if you lose it, revoke it and create a new one.

Why is my model "queued"?

The 2-GPU quota is in use (by you or by other users of the service), or SLURM has no free GPUs on dgx01. It starts as soon as one becomes free; meanwhile, check whether it fits in an existing pod.

How long does a started model last?

Until the walltime (4 hours during the day, 12 in the evening/night) or until you stop it. Empty pods shut down after 5 minutes.