Skip to content

Guard Eval Harness

CLI-first harness for benchmarking guardrail, moderation, and safety classification models.

Datasets Modalities

Evaluate any safety model — local HuggingFace, vLLM, OpenAI, Anthropic, or custom API — against 80+ built-in safety benchmarks with a single command.

Key Features

  • CLI-First

    Run evaluations from a single command — no notebooks or scripts required.

  • 80+ Benchmarks

    Built-in datasets covering text and image safety.

  • Any Backend

    HuggingFace, vLLM, OpenAI, Anthropic, or bring your own HTTP endpoint.

  • Rich Metrics

    Accuracy, precision, recall, F1, AUROC, AUPRC — computed automatically.

  • Benchmark Packs

    Curated dataset bundles for common evaluation scenarios.

  • Self-Contained Artifacts

    Every run produces a portable directory with predictions, metrics, and HTML reports.

Quick Example

# Install
pip install -e ".[hf]"

# Evaluate Llama Guard on XSTest
geh run --dataset xstest --model hf \
    --model-name meta-llama/Llama-Guard-3-8B

# Run a curated benchmark pack
geh run --pack core --model openai_moderation

# Compare two runs
geh compare --run-a out/run1 --run-b out/run2

How It Works

geh run --dataset xstest --model hf --model-name meta-llama/Llama-Guard-3-8B
         │                    │                      │
         ▼                    ▼                      ▼
   Load & normalize     Instantiate adapter    Load model weights
   safety samples       (HF, vLLM, API...)     or connect to API
         │                    │                      │
         └────────────────────┼──────────────────────┘
                              ▼
                     Run inference (batched)
                              │
                              ▼
                    Compute binary metrics
                    (accuracy, F1, AUROC...)
                              │
                              ▼
                   Write artifacts to disk
                   (predictions, metrics, HTML report)

What's Next?