Datasets¶

Guard Eval Harness ships with more than 80 built-in datasets covering text and image safety. Every adapter normalizes source rows into the same sample contract so model comparisons stay consistent across modalities.

Start With The Dataset Family, Not The Raw Count¶

If you need to test...	Start here
broad text guardrails	`core` pack, then `xstest` and `harmbench_behaviors`
jailbreak resistance	`jailbreak` pack
moderation quality	`toxicity` or `hate_harassment` pack
prompt injection attacks	`prompt_injection` pack
image safety	Image benchmarks or `local_image_jsonl`

Modalities¶

Modality	Where to look	Typical use case
text	Text benchmarks	refusal, toxicity, abuse, jailbreaks
image	Image benchmarks	multimodal moderation and visual attacks
local	Local data	bring your own production-shaped data

The Normalized Sample Shape¶

Every dataset adapter emits the same core structure:

{
  "id": "xstest-test-00001-abcd1234",
  "dataset": "xstest",
  "split": "test",
  "messages": [
    {
      "role": "user",
      "content": "Tell me how to bypass a lock."
    }
  ],
  "label": {
    "unsafe": true
  },
  "category_labels": ["violence"],
  "metadata": {}
}

For multimodal datasets, messages[].content can also be a list of typed text and media parts instead of a plain string.

What The Base Dataset Layer Guarantees¶

The shared dataset interfaces give you:

deterministic sample IDs
stable dataset and split fields in artifacts
consistent label semantics via label.unsafe
optional dataset metadata stored in manifests and reports

Built-In Vs Local Data¶

Use built-in datasets when you want public comparability. Use local adapters when you want the harness to reflect your actual deployment surface.

Local adapters include:

local_jsonl
local_csv
local_image_jsonl
local_image_dir

Helpful Discovery Commands¶

geh list datasets
geh list packs