Benchmark RadarPoolBench
PoolBench isolates pooling strategies as the experimental variable in concept representation evaluation for decoder-only LLMs. It covers 17 concepts, 19 pooling strategies, and 3 open-weight models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B) on a corpus of 37,693 real-text passages, with primary axis linear separability (AUROC) and diagnostic axes for steering and disentanglement.
- Released
- 2026-05-26
- Readiness
- Paper only
- Primary field
- General AI
Why it matters
Pooling is a consequential but under-examined design choice in concept representation work, yet no shared protocol exists for comparing pooling rules across concepts, models, and tasks. PoolBench provides a controlled protocol with released corpus, pre-extracted activations, and scoring code to enable principled comparison.
Motivation
Pooling is a consequential but under-examined design choice in decoder-only concept representation work: practitioners must collapse token-level hidden states into a passage-level vector, yet no shared protocol exists for comparing this choice across concepts, models, and tasks.
Primary resources
Benchmark Radar records only publicly supported details and links back to primary sources for verification.