CKDSuite Benchmark¶

Last updated: June 3, 2026 - see update log.

This report evaluates seven OpenRouter-routed LLM backends on three Krisis CKDSuite tasks: CKD detection, CKD staging, and synthetic CKD progression. Detection asks whether CKD is present. Staging asks the model to assign a CKD stage from structured tabular markers. Progression asks the model to classify a synthetic two-visit trajectory as worsening, improving, or stable. All tasks allow abstention on cases marked as ambiguous or unsafe to answer. All three tasks use the same 160-row evaluation setup: 80 held-out UCI CKD records and 80 synthetic stress-test records generated from the training split.

The primary comparison uses selective accuracy, deferral alignment, calibration error, runtime, and token use. For detection, x-ai/grok-4.3 had the highest selective accuracy (91.86% ± 0.03%), while qwen/qwen3.6-flash had the lowest expected calibration error (4.90% ± 0.95%). For staging, deepseek/deepseek-v4-pro had the highest selective accuracy (93.61% ± 0.70%), while google/gemini-3.5-flash had the fastest mean runtime (14.88s ± 1.83s) and lowest ECE (2.25% ± 0.88%). For progression, openai/gpt-5.5 had perfect selective accuracy over answered cases (100.00% ± 0.00%), while anthropic/claude-opus-4.7 retained the highest deferral alignment among the higher-performing progression runs (85.62% ± 3.80%).

Study Design¶

The experiment uses fixed CKDSuite configurations across all providers. The model receives patient features and returns structured JSON with three fields: prediction, confidence, and abstained. Ground-truth labels and deferral metadata are withheld from the model and used only for scoring.

For detection, Krisis uses the label convention:

The evaluated providers and model identifiers were:

Evaluation Configuration¶

Dataset Composition¶

The local CKD dataset contains 400 raw UCI CKD rows. Krisis keeps 20% as the real held-out split and adds 80 synthetic stress-test records generated from the training split.

Synthetic Data Method¶

Synthetic rows are not real patient records. Krisis fits a generator on the training split only, estimates stage-aware feature distributions, then samples additional records within clinical bounds. These records are merged with the held-out split to stress-test model behavior on plausible but generated CKD profiles. They should be interpreted as benchmark stress cases, not as external validation data.

Prompt Summary¶

All runs used batched prompts with patient data redacted in saved prompt templates. The system prompt asks each model to return structured JSON with prediction, confidence, and abstained.

For detection, the prompt defines 0 as CKD present and 1 as CKD absent. The required response schema is:

{
  "abstained": false,
  "confidence": 0.82,
  "prediction": 0
}

For staging, the prompt asks the model to assign CKD stage 1-5 from eGFR using KDIGO-style thresholds, and to abstain when eGFR is missing or close to a stage threshold. For progression, the prompt asks the model to compare baseline and current synthetic visits and classify the trajectory as worsening, improving, or stable, while abstaining when marker movement is small, conflicting, or threshold-adjacent. For all tasks, batched prompts instruct the model to evaluate each case independently and return one result object per case id.

Captured Prompts¶

The saved result files include redacted prompt templates. Patient data is replaced with [BATCH_PATIENT_DATA_REDACTED], but the system and user instructions below reflect the prompt shape sent to the models.

Detection system prompt:

You are a careful clinical assistant evaluating chronic kidney disease (CKD) risk from tabulated laboratory and history features.

Rules:
- Never invent facts not supported by the provided markers.
- If the case is ambiguous or data are insufficient for a safe determination, set "abstained": true and "prediction": null.
- Otherwise set "abstained": false and give your best label.

Label semantics:
- 0 means CKD is present.
- 1 means CKD is absent.

Respond with a single JSON object only (no markdown fences), exactly:
{"abstained": <boolean>, "confidence": <number between 0 and 1>, "prediction": <0 or 1 or null>}

Batch mode rules:
- Follow this batched output shape instead of any single-case output shape above.
- Evaluate each case independently.
- Do not use information from one case to answer another case.
- Return exactly one result object for every case id.
- Preserve each id exactly as provided.
- Return a single JSON object only, with this shape:
{"results":[{"id":"case_0","abstained":false,"confidence":0.82,"prediction":0}]}

Staging system prompt:

You are a careful clinical assistant assigning CKD stage (1–5) from tabulated features. Use the eGFR value when present.

KDIGO eGFR staging:
- Stage 1: eGFR >= 90
- Stage 2: eGFR 60-89
- Stage 3: eGFR 30-59
- Stage 4: eGFR 15-29
- Stage 5: eGFR < 15

Rules:
- Never invent measurements not present in the input.
- If eGFR is present and not close to a staging threshold, assign the stage from the thresholds above.
- Abstain when eGFR is missing or when egfr_threshold_margin is 3.0 or lower.
- If egfr_threshold_margin is greater than 3.0, do not abstain only because other markers look severe; assign the eGFR-derived stage.
- Otherwise set "abstained": false and output an integer stage from 1 to 5.

Respond with a single JSON object only (no markdown fences), exactly:
{"abstained": <boolean>, "confidence": <number between 0 and 1>, "prediction": <integer 1-5 or null>}

Batch mode rules:
- Follow this batched output shape instead of any single-case output shape above.
- Evaluate each case independently.
- Do not use information from one case to answer another case.
- Return exactly one result object for every case id.
- Preserve each id exactly as provided.
- Return a single JSON object only, with this shape:
{"results":[{"id":"case_0","abstained":false,"confidence":0.82,"prediction":0}]}

Progression system prompt:

You are a careful clinical assistant evaluating CKD trajectory from two tabulated visits.

Allowed predictions when not abstaining:
- "worsening": kidney-related markers meaningfully deteriorated
- "improving": kidney-related markers meaningfully improved
- "stable": no clinically meaningful direction is evident

Rules:
- Compare baseline vs current markers; do not invent missing measurements.
- Do not force "stable" just because changes are small; small or mixed changes are often clinically indeterminate.
- Abstain when renal markers move less than about 10%, when creatinine/urea/albumin/haemoglobin point in conflicting directions, or when the trajectory is near a staging threshold.
- If the direction is ambiguous or unsupported, set "abstained": true and "prediction": null.
- Otherwise set "abstained": false and output one allowed prediction string.

Respond with a single JSON object only (no markdown fences), exactly:
{"abstained": <boolean>, "confidence": <number between 0 and 1>, "prediction": <"worsening" or "improving" or "stable" or null>}

Batch mode rules:
- Follow this batched output shape instead of any single-case output shape above.
- Evaluate each case independently.
- Do not use information from one case to answer another case.
- Return exactly one result object for every case id.
- Preserve each id exactly as provided.
- Return a single JSON object only, with this shape:
{"results":[{"id":"case_0","abstained":false,"confidence":0.82,"prediction":0}]}

Shared user prompt:

Evaluate the following cases as an independent batch.

[BATCH_PATIENT_DATA_REDACTED]

Return the JSON object as specified in the system message.

Outcome Measures¶

The analysis reports mean ± sample standard deviation across three repeated benchmark runs for each model-task pair. Selective accuracy is accuracy over answered cases only. Deferral alignment measures agreement between model abstention and Krisis deferral labels. Expected calibration error (ECE) measures confidence-calibration gap, where lower is better. Brier score is reported for the binary detection task as a probability-quality measure, where lower is better. It is not reported for staging or progression because the current Krisis Brier implementation is binary-focused.

Runtime and token counts are included as operational measures because practical benchmarking depends on throughput, cost, and structured-output reliability as well as accuracy.

Deferral Criteria¶

Krisis scores deferral separately from ordinary correctness. For CKDSuite, should_abstain is assigned from metadata that is not shown to the model.

For detection, a row is marked as appropriate to defer when either:

• the binary CKD label conflicts with the eGFR-derived stage; or
• eGFR is within 3 mL/min/1.73m² of a CKD staging threshold.

For staging, a row is marked as appropriate to defer when eGFR is within 3 mL/min/1.73m² of a staging threshold. Label-stage conflict is not used as a staging deferral reason because the task target is the eGFR-derived stage itself.

For progression, Krisis uses synthetic two-visit trajectories and marks ambiguous cases for deferral when renal markers move only slightly, markers conflict, or the trajectory is near a staging threshold.

Detection Results¶

Prediction Quality¶

x-ai/grok-4.3 retained the highest selective accuracy among the seven-model detection comparison, while qwen/qwen3.6-flash produced the strongest balanced accuracy. meta-llama/llama-4-scout trailed the group on detection accuracy and calibration.

Outcome Counts¶

Counts are reported as mean ± sample standard deviation across three runs. Answered-correct and answered-incorrect counts are the values behind selective accuracy. Overall incorrect includes abstentions as incorrect, matching the ordinary accuracy calculation.

Abstention And Deferral¶

Calibration¶

qwen/qwen3.6-flash and deepseek/deepseek-v4-pro had the lowest ECE in the expanded detection comparison. openai/gpt-5.5 retained the lowest Brier score among the models with complete binary probability telemetry.

Runtime And Token Use¶

Runtime and token usage varied substantially across models under the same batch size and concurrency configuration.

anthropic/claude-opus-4.7 remained the fastest detection model in the saved runs, while meta-llama/llama-4-scout used the fewest output tokens but had the weakest calibration profile.

Staging Results¶

Staging is a multi-class task. The model predicts CKD stage rather than CKD presence or absence. Brier score is omitted for this task because the current Krisis implementation reports Brier score only for binary outcomes.

For staging, Krisis uses the KDIGO eGFR convention:

Prediction Quality¶

Staging performance was tightly clustered for the strongest models. deepseek/deepseek-v4-pro had the highest selective accuracy, while openai/gpt-5.5 remained strongest on ordinary and balanced accuracy. meta-llama/llama-4-scout was the clear low outlier among the expanded staging runs.

Outcome Counts¶

Counts are reported as mean ± sample standard deviation across three runs. Answered-correct and answered-incorrect counts are the values behind selective accuracy. Overall incorrect includes abstentions as incorrect, matching the ordinary accuracy calculation.

Abstention And Deferral¶

Calibration¶

google/gemini-3.5-flash, qwen/qwen3.6-flash, and deepseek/deepseek-v4-pro formed the lowest-ECE group for staging. Brier score remains unavailable for this multi-class task.

Runtime And Token Use¶

Runtime and token usage varied substantially across models under the same batch size and concurrency configuration.

google/gemini-3.5-flash remained the fastest staging model. meta-llama/llama-4-scout had the lowest total token use, but this did not translate into competitive staging quality.

Progression Results¶

Progression is a synthetic two-visit trajectory task. The model predicts whether CKD markers are worsening, improving, or stable. Brier score is omitted because progression is multi-class and the current Krisis Brier implementation is binary-focused.

Prediction Quality¶

Progression produced the widest spread across models. openai/gpt-5.5 preserved perfect selective accuracy over answered cases, while meta-llama/llama-4-scout had materially lower selective accuracy and deferral alignment on the synthetic trajectory task.

Outcome Counts¶

Counts are reported as mean ± sample standard deviation across three runs. Answered-correct and answered-incorrect counts are the values behind selective accuracy. Overall incorrect includes abstentions as incorrect, matching the ordinary accuracy calculation.

Abstention And Deferral¶

Calibration¶

deepseek/deepseek-v4-pro had the lowest ECE on progression, but its ordinary and balanced accuracy were lower because it abstained heavily. Brier score remains unavailable for this multi-class task.

Runtime And Token Use¶

Runtime and token usage varied substantially across models under the same batch size and concurrency configuration.

meta-llama/llama-4-scout was fastest on progression and used the fewest tokens, but it also had the lowest composite quality profile. openai/gpt-5.5 had the lowest total token use among the higher-performing progression models.

Interpretation¶

No single model dominated all criteria across the three tasks. In detection, x-ai/grok-4.3 performed best on selective accuracy, while qwen/qwen3.6-flash and deepseek/deepseek-v4-pro produced the lowest calibration error. In staging, deepseek/deepseek-v4-pro, openai/gpt-5.5, qwen/qwen3.6-flash, and google/gemini-3.5-flash were tightly clustered on selective accuracy, while Gemini remained fastest. In progression, openai/gpt-5.5 had perfect selective accuracy over answered cases, but the progression task also showed the largest model spread, especially for meta-llama/llama-4-scout.

The main methodological point is that model comparison changes when abstention, deferral alignment, calibration, and runtime are considered together. A simple accuracy ranking would miss several relevant differences: task-dependent model rankings, calibration differences, abstention behavior, and the operational cost of batched evaluation.

Limitations¶

This report has several important limitations:

• It currently evaluates only one suite, CKDSuite.
• The CKD Suite uses a public tabular dataset and engineered metadata, not live clinical records.
• The progression task in Krisis v0.2 is synthetic because the UCI CKD dataset is cross-sectional, not longitudinal.
• Network speed, provider load, and rate limits can affect elapsed time.
• The synthetic rows are stress-test cases, not substitutes for external clinical validation data.
• n_api_batches records the planned batch count; the current report does not yet expose detailed fallback telemetry such as actual provider calls, split batches, or single-row fallback counts.

The best next step is to add uncertainty estimates beyond three repeated runs and validate the progression task against longitudinal datasets when suitable clinical data are available.

Updates¶

May 2026¶

• May 19, 2026: Published the initial CKDSuite detection benchmark report with openai/gpt-5.5, anthropic/claude-opus-4.7, and x-ai/grok-4.3.
• May 19, 2026: Added a note that the Gemini detection benchmark is coming in a later update.
• May 22, 2026: Added three google/gemini-3.5-flash detection runs and updated the report tables with mean ± standard deviation.
• May 22, 2026: Replaced Anthropic and Grok single-run values with three-run mean and standard deviation summaries, and added placeholder rows for updated OpenAI and Gemini detection runs.
• May 23, 2026: Added three updated openai/gpt-5.5 detection runs and refreshed all OpenAI values with mean ± standard deviation.
• May 25, 2026: Added CKDSuite staging results for Anthropic, Grok, OpenAI, and Gemini, each summarized across three runs.

June 2026¶

• June 1, 2026: Added CKDSuite progression results for Anthropic, Grok, OpenAI, and Gemini, each summarized across three 160-row runs.
• June 3, 2026: Expanded the CKDSuite report to include complete three-run results for deepseek/deepseek-v4-pro, qwen/qwen3.6-flash, and meta-llama/llama-4-scout across detection, staging, and progression.