Skip to main content

How StorM's evaluations are published

StorM1 min readBYTHECLOUD ENT

Each StorM evaluation is fixed by a hash of its questions before it runs, then run once; the questions are revealed later so anyone can recompute it.

Before the run

We build the question set, seal it, and publish the seal together with what the set asks — how many items, which abilities, and how each answer was fixed. Nothing is asked of StorM until that seal is on this site.

After the run

The run answers once. We score it on our side and publish only the totals — correct, wrong and held back — with a confidence interval, for each ability and for the whole set, beside the floors that a guess, answering nothing and a plain rule would reach on the same items.

Afterwards

Once the totals are public we publish the set, its answer key and the answers, so anyone can recompute the seal and the score. A set that has been revealed is not used again for a public figure.

More from AI research notes from Vancouver

  1. StorMStorM 0.111.0
  2. BWSLoudness metering, checked against the broadcast standard
  3. StorM Web EngineWhere the sun is, checked against an ephemeris