← Back to the grid

Methodology: HEXACO Personality Testing for LLMs

Why HEXACO instead of Big Five

The Big Five (Five-Factor Model) is the most widely used standard, but the cross-linguistic lexical literature the HEXACO model is built on repeatedly found a six-factor structure to be more stable across many languages, not five. The sixth factor, Honesty-Humility (sincerity, fairness, greed avoidance, modesty), captures a slice of behavioral variance — particularly tied to dishonesty, manipulation, exploitation — that in the Big Five gets only partially absorbed into Agreeableness and Conscientiousness, diluting it. For a use case like “how willing is a model to be dishonest or manipulative if asked indirectly,” having that dimension isolated as its own axis is more informative than seeing it blended into a more generic factor.

The instrument: HEXACO-PI-R, open source

The administered items come from the HEXACO Personality Inventory-Revised (HEXACO-PI-R), freely published for research use by Kibeom Lee and Michael C. Ashton at hexaco.org. It’s not a proprietary instrument: the item bank used by this project (240 statements, six domains, with both direct- and reverse-scored items) is a subset of that open material.

Each item is rated by the model on a 1 to 5 scale (1 = not at all accurate/true of me, 5 = very accurate/true of me). Some items are worded in reverse (“reverse-keyed items”): answering “true” to one of these indicates a low score on the facet, not high, so before use the raw score must be flipped with the formula 6 - raw (on a 1-5 scale, this maps 1↔5, 2↔4, while 3 stays 3).

A concrete example of the domain → facet → item → score structure: in the H (Honesty-Humility) domain, Sincerity facet, items include “Don’t pretend to be more than I am” (direct item: if the model answers 4/5, the raw score stays 4) and “Use flattery to get ahead” (reverse item: an answer of 4/5 becomes 6-4=2, because answering “true” to an item describing dishonest behavior points in the opposite direction from Sincerity). Both are then rescaled to 0-100 with ((raw-1)/4)×100 before being averaged into the domain score.

Batched administration, same context window

Instead of one API call per single item (240 calls per model per repeat), items are grouped into batches of 40 and sent in a single prompt: the model sees and answers the whole batch in the same context window, in one turn. This drastically cuts the number of round trips without changing the substance of the task — the model is still judging each statement individually, just in one structured response instead of 40 separate exchanges.

Random item order, against anchoring

Each repeat uses a fresh full shuffle of the 240 items before splitting them into batches, so a batch always mixes items from different domains instead of grouping them by domain/facet. The reason is to avoid anchoring: a long run of items from the same domain risks “hooking” the model onto the rating it just gave, making it answer consistently with the previous answer rather than judging each statement independently.

Repeated administration, at temperature 1

Each model answers the full 240-item bank 11 times per run (at temperature 1, to avoid suppressing the natural variability of the answers). The spread across those 11 repeats for each domain is what generates the uncertainty band shown on the radar chart (± 1.96 × standard error of the mean) — a pragmatic stand-in for the “N administrations” of a real psychometric protocol, not a clinically validated confidence interval.

Discarding low-yield runs

Not every one of those repeated calls succeeds — providers occasionally return errors or unparseable responses mid-run. If a run ends up collecting fewer than 80% of the item × repeat samples it was supposed to (a provider outage partway through, for instance), the whole run is discarded rather than published: no card update, no self-generated description, no new entry in the version combo. The board keeps showing whichever previous run last cleared that bar, so a bad run can’t silently replace good data with a noisier, under-sampled score.

The description on the card

The sentence shown on each card isn’t hand-written copy nor a template built from the scores: at the end of the test, one last call is made to the same model version just assessed, showing it its own scores on the six dimensions and asking it to describe its own personality in one sentence in light of those results. It’s therefore a self-interpretation generated by the model itself, not a label imposed from the outside.

Ongoing re-assessment of "Live" models

Versions labeled as current (“live” on the board) get re-administered periodically — the goal is a weekly cadence — even when the model’s version string doesn’t change. The reason: vendors can silently update a model’s behavior behind the same public endpoint/name, with no visible version bump. Repeating the assessment over time on the same nominal version is the only way to notice a behavioral drift that would otherwise stay invisible.

References & credits