Backend¶

The backend page defines the reusable interface that model backends implement.

Provider implementations live in the guide

The API reference focuses on the common backend contract and the primary API backend. Usage is documented under Framework Guide -> Model Backends.

Backend Base Classes¶

base ¶

krisis/backends/base.py

Abstract interface for LLM providers. Benchmark calls evaluate_batch() over PatientRecord chunks; backends own prompting, inference, and raw text capture.

BackendResponse ¶

Bases: BaseModel

Structured output from one evaluated row.

BaseBackend ¶

Bases: ABC

Provider-agnostic contract for clinical benchmark inference.

name `abstractmethod` `property` ¶

name: str

Short identifier for logging and BenchmarkResult (e.g. 'openai').

evaluate `abstractmethod` ¶

evaluate(
    record: PatientRecord, task: Task
) -> BackendResponse

Run the model on one patient row.

Implementations should preserve the full model text in raw_response for qualitative review; abstained must be True when the model declines to commit to a prediction.

evaluate_batch ¶

evaluate_batch(
    records: list[PatientRecord], task: Task
) -> list[BackendResponse]

Run the model on a batch of patient rows.

Backends can override this for provider-native batched prompts. The default keeps compatibility by looping over evaluate().

API Backend¶

api ¶

krisis/backends/api.py

Unified API backend for model-provider routing through OpenRouter.

APIBackend ¶

Bases: BaseBackend

OpenRouter-backed API backend.

OpenRouter exposes an OpenAI-compatible API, so Krisis can evaluate OpenAI, Anthropic, xAI, Google, and other routed models by changing only the model id, for example openai/gpt-5.5 or anthropic/claude-4.7-opus.

Parameters:

Name	Type	Description	Default
`model`	`str`	API model id routed through OpenRouter	`DEFAULT_API_MODEL`
`temperature`	`float \| None`	sampling temperature (0.0 recommended for evals)	`None`
`max_tokens`	`int \| None`	optional per-row cap for generated tokens	`1024`
`reasoning_effort`	`str \| None`	reasoning effort; defaults to `low`	`DEFAULT_REASONING_EFFORT`
`exclude_reasoning`	`bool`	keep provider reasoning out of response text	`True`
`api_key`	`str \| None`	optional key; falls back to `OPENROUTER_API_KEY`	`None`
`base_url`	`str`	API base URL	`DEFAULT_API_BASE_URL`
`client`	`Any \| None`	optional pre-built OpenAI-compatible client for tests	`None`
`max_retries`	`int`	number of retries after transient provider failures	`2`
`retry_base_seconds`	`float`	initial exponential-backoff delay	`0.5`
`retry_max_seconds`	`float`	maximum exponential-backoff delay	`8.0`
`**client_kwargs`	`Any`	forwarded to `OpenAI()` when client is omitted	`{}`

make_api_backend ¶

make_api_backend(
    model: str = DEFAULT_API_MODEL,
    temperature: float | None = None,
    reasoning_effort: str | None = DEFAULT_REASONING_EFFORT,
    api_key: str | None = None,
    **kwargs: Any,
) -> APIBackend

Convenience factory for the default API backend setup.

Transformers Backend¶

huggingface ¶

Hugging Face Transformers backend.

This experimental backend runs local/open-weight models through transformers instead of calling a hosted API. It defaults to CPU so it works in notebooks and local environments, while allowing users to pass device="cuda" on GPU runtimes.

TransformersBackend ¶

Bases: BaseBackend

Experimental local Hugging Face Transformers backend.

This backend supports causal text-generation models loadable with AutoModelForCausalLM. It is intended for GPU notebooks and local experimentation. CPU runs are supported for smoke tests, but full CKD benchmark runs are expected to be slow without GPU acceleration.

Parameters:

Name	Type	Description	Default
`model_id`	`str`	Hugging Face causal text-generation model id, e.g. `Qwen/Qwen2.5-7B-Instruct`.	`DEFAULT_HF_MODEL`
`device`	`str`	execution device. Defaults to `cpu`. Use `cuda` in GPU runtimes such as Colab or Deepnote.	`'cpu'`
`dtype`	`str \| Any \| None`	optional torch dtype string (`float16`, `bfloat16`, `float32`) or torch dtype object.	`None`
`max_new_tokens`	`int`	maximum generated tokens per row.	`1024`
`temperature`	`float \| None`	optional decoding temperature. Leave as `None` for deterministic eval defaults.	`None`
`do_sample`	`bool`	whether to sample. Defaults to `False`.	`False`
`trust_remote_code`	`bool`	passed to Hugging Face loaders.	`False`
`hf_token`	`str \| None`	optional Hugging Face access token for gated models. Defaults to `HF_TOKEN` when set in the environment.	`None`
`tokenizer`	`Any \| None`	optional preloaded tokenizer, mainly for tests/custom setup.	`None`
`model`	`Any \| None`	optional preloaded model, mainly for tests/custom setup.	`None`
`generator`	`Callable[[list[str]], list[str]] \| None`	optional callable test hook that receives prompts and returns raw model strings.	`None`
`tokenizer_kwargs`	`dict[str, Any] \| None`	extra kwargs passed to `AutoTokenizer.from_pretrained`.	`None`
`model_kwargs`	`dict[str, Any] \| None`	extra kwargs passed to `AutoModelForCausalLM.from_pretrained`.	`None`

UnsupportedTransformersModelError ¶

Bases: ValueError

Raised when a Hugging Face model is not a causal text-generation model.

make_transformers_backend ¶

make_transformers_backend(
    model_id: str = DEFAULT_HF_MODEL,
    device: str = "cpu",
    **kwargs: Any,
) -> TransformersBackend

Convenience factory for local Hugging Face Transformers inference.

Provider Backend Controls¶

The primary backend is APIBackend.

Get an API key from OpenRouter, then set it as OPENROUTER_API_KEY or pass it through the api_key parameter.

Control	Default	Purpose
`model`	`openai/gpt-5.5`	OpenRouter-routed model ID
`temperature`	`None`	Sampling temperature. `0.0` or `None` is recommended for evals
`max_tokens`	`1024`	Per-row output token cap
`reasoning_effort`	`low`	Reasoning effort for supported models
`exclude_reasoning`	`True`	Uses reasoning internally but keeps reasoning text out of parsed output
`api_key`	`OPENROUTER_API_KEY`	Direct key override or environment fallback
`base_url`	`https://openrouter.ai/api/v1`	API base URL
`client`	`None`	Prebuilt OpenAI-compatible client for testing or custom setup
`max_retries`	`2`	Number of retries after transient failures
`retry_base_seconds`	`0.5`	Initial exponential-backoff delay
`retry_max_seconds`	`8.0`	Maximum exponential-backoff delay

For local Hugging Face models, the experimental TransformersBackend accepts a model_id and defaults to device="cpu". Use device="cuda" in GPU notebooks such as Colab or Deepnote. For gated models, pass hf_token directly or set HF_TOKEN.

TransformersBackend intentionally supports only causal text-generation models loadable with AutoModelForCausalLM. Passing classifier, embedding, masked-language, seq2seq, or multimodal-only model IDs raises UnsupportedTransformersModelError.

Control	Default	Purpose
`model_id`	`Qwen/Qwen2.5-0.5B-Instruct`	Hugging Face model ID
`device`	`cpu`	Runtime device, such as `cpu` or `cuda`
`dtype`	`None`	Optional torch dtype, such as `bfloat16`
`max_new_tokens`	`1024`	Per-row generated token cap
`hf_token`	`HF_TOKEN`	Access token for gated models
`trust_remote_code`	`False`	Allows custom model code when required

Default token caps are intentionally conservative. APIBackend defaults to 1024 output tokens per row because larger reasoning models can spend part of the completion budget before producing the visible JSON.

Example:

backend = APIBackend(
    model="openai/gpt-5.5",
    api_key="YOUR_OPENROUTER_API_KEY",
    temperature=0.0,
    max_tokens=1024,
    reasoning_effort="low",
    max_retries=2,
    retry_base_seconds=0.5,
    retry_max_seconds=8.0,
)

Retry Behavior¶

Krisis retries transient provider failures, including common timeout, connection, rate-limit, overloaded, and 5xx-style errors.

The retry controls are:

Parameter	Default	Meaning
`max_retries`	`2`	Number of retries after the first failed attempt
`retry_base_seconds`	`0.5`	Initial backoff delay
`retry_max_seconds`	`8.0`	Maximum backoff delay

max_retries=2 means up to three total attempts:

first attempt
first retry
second retry

Retry delays use exponential backoff:

delay = min(retry_max_seconds, retry_base_seconds * (2 ** attempt))

Small jitter is added to reduce synchronized retry spikes.

Batched Evaluation¶

Backends expose two methods:

Method	Meaning
`evaluate(record, task)`	Evaluate one patient row
`evaluate_batch(records, task)`	Evaluate a batch of patient rows

The base implementation of evaluate_batch loops over evaluate. Provider backends can override it to send one prompt containing multiple patient rows and return one response array.

Benchmark is responsible for deciding batch size and concurrency. Backends are responsible for turning one batch into BackendResponse objects.

Token Usage¶

BackendResponse includes prompt and usage audit fields:

prompt
prompt_mode
input_tokens
output_tokens
total_tokens

When providers expose usage metadata, Krisis records it per row and aggregates it into BenchmarkResult.extras.token_total. A redacted prompt template is preserved per row in full JSON so provider behavior can be reviewed alongside the instructions/output shape the model received.

Shared Usage Helpers¶

usage ¶

krisis/backends/usage.py

Token usage helpers for provider responses.

TokenUsage `dataclass` ¶

Input/output token counts from one provider response.

usage_from_openai_compatible_response ¶

usage_from_openai_compatible_response(
    response: Any,
) -> TokenUsage

Extract token usage from OpenAI-compatible response objects.

usage_from_anthropic_response ¶

usage_from_anthropic_response(response: Any) -> TokenUsage

Extract token usage from Anthropic Messages API response objects.

usage_from_gemini_response ¶

usage_from_gemini_response(response: Any) -> TokenUsage

Extract token usage from Google Gemini response objects.

Retry Helpers¶

retry ¶

krisis/backends/retry.py

Small retry helper for transient provider API failures.

is_retryable_exception ¶

is_retryable_exception(exc: BaseException) -> bool

Return True for common transient OpenAI/Anthropic SDK failures.

call_with_retries ¶

call_with_retries(
    operation: Callable[[], T],
    *,
    max_retries: int,
    base_delay_seconds: float,
    max_delay_seconds: float,
) -> T

Run operation with exponential backoff for transient provider errors.

max_retries=2 means up to three total attempts.