Frontier Question Bank
Questions from working scientists that no contender could answer, each with criteria a third party can grade against.
Why this exists
A benchmark made of questions somebody already knew the answer to measures recall. This one is made of questions a scientist needed answered and could not get answered, which is the thing everyone actually wants to measure. Half the bank is held back: the private split never leaves our infrastructure, so a contender cannot be trained on it and then tested against it.
Admission
- Nominated by the scientist at resolution.
- Both contenders lost to the reference answer.
- Publication-matched nominator, or a panel endorsement.
- Verification criteria a third party can apply.
The bank is empty
Nothing has been admitted yet. A question gets here by being asked in earnest, beating both contenders, and coming with a way to check the answer.
Prompt hashes are published for a reason
Every admitted question's exact prompt text is hashed here. A lab that wants to prove it excluded the bank from training can check its corpus against these hashes without ever seeing the questions. Download every hash, both splits, as a text file a build script can diff.