Sonnet 4.5's response (run 3) to: Macbeth Characters

Macbeth Characters

List the characters in Shakespeare's Macbeth, in order of first appearance, one per line, no other text.

ANALYSIS

Only 3 of 7 models achieved any correct responses. Critical failure point: 4 models incorrectly listed Duncan instead of the Three Witches as first characters.

See full analysis

Analysis (v2) by kimi-k2-thinking on 18th Nov 2025

Outcome: Only 3 of 7 models achieved any correct responses. Critical failure point: 4 models incorrectly listed Duncan instead of the Three Witches as first characters.
Approach: GPT-5 produced the cleanest lists when correct. Gemini models included optional characters (Ghost of Banquo, separate Doctors) which is acceptable but slightly verbose.
Performance: Gemini 3 Pro delivered fastest responses (39s avg) at moderate cost. GPT-5 averaged 118s per response - 3x slower. Kimi K2 was extremely slow (257s avg).
Most Surprising: Despite identical accuracy scores, performance metrics varied by 300%. Also, Sonnet 4.5 was fastest (3.3s avg) but never produced a correct answer, showing speed doesn't correlate with accuracy here.

Summary

This verifiable prompt tested models' ability to list Macbeth characters in exact order of first appearance. Only three models achieved correct answers: Gemini 3 Pro, Gemini 2.5 Pro, and GPT-5 (each 2/4 correct). Gemini 3 Pro won with superior speed (39s vs 118s) and lower cost than GPT-5. Critical failure: four models incorrectly started with Duncan instead of the Three Witches, demonstrating fundamental misunderstanding of the prompt's deterministic requirements.

Outcome Analysis

What models produced/concluded:

Consensus: Zero models achieved 100% accuracy. When correct, all three top performers agreed on the core character sequence: Three Witches → Duncan/Malcolm/Donalbain → Lennox/Sergeant/Ross/Angus → Macbeth/Banquo → Lady Macbeth → Messenger → Fleance → Porter → Macduff → etc.

Key Divergences:

Critical Error - Missing Witches: Opus 4.1 and Sonnet 4.5 universally opened with Duncan, violating the play's actual structure where Act 1, Scene 1 features only the Three Witches.
Fatal Ordering Flaw: Most incorrect iterations placed Macduff before Fleance, despite Fleance appearing in Act 2, Scene 1 and Macduff in Act 2, Scene 3.
Angus Positioning: 60% of incorrect responses misplaced Angus, listing him after Macbeth/Banquo despite his entrance in Act 1, Scene 2.
Naming Inconsistency: Grok 4 and Kimi K2 showed erratic naming conventions ("Bleeding Sergeant", "Three Witches" vs "First Witch", "Macduff's Son" vs "Son").

Approach Analysis

Best Methodology: GPT-5 (Iterations 1-2) delivered the leanest, most precise lists with zero generic terms and perfect specificity (e.g., "Macduff's Son"). When correct, responses were textbook examples of following "one per line, no other text."

Most Verbose: Gemini 2.5 Pro included optional characters like "Ghost of Banquo" and split doctors by nationality ("English Doctor", "Scottish Doctor"), technically accurate but exceeding minimal requirements.

Most Problematic: Sonnet 4.5 showed wild variance - Run 1 listed 30 characters including generic groups ("Soldiers", "Attendants"), while Run 2 truncated to just 22 characters, omitting major figures like Fleance. This reflects low reliability for deterministic tasks.

Unique Pattern: Grok 4 consistently prefaced Sergeant with descriptors ("Bleeding Sergeant") but rigidly misplaced Angus in all four runs, suggesting a systemic reasoning error rather than random noise.

Performance Table

Model	Accuracy	Rank	Avg Cost	Avg Time	Tokens I/O
Gemini 3 Pro	2/4	1st	$0.0555	39.2s	24/4625
Gemini 2.5 Pro	2/4	2nd	$0.0454	43.5s	23/4537
GPT-5	2/4	3rd	$0.0820	118.2s	28/8192
Grok 4	0/4	4th	$0.0636	99.2s	705/4099
Kimi K2	0/4	5th	$0.0157	256.9s	30/6950
Sonnet 4.5	0/4	6th	$0.0016	3.3s	32/102
Opus 4.1	0/4	7th	$0.0094	4.3s	32/120

Key Findings

Outcome:

Shocking 43% failure rate on deterministic task: 3/7 models never produced a correct answer across 4 attempts each.
Binary quality gap: No model achieved partial correctness - answers were either perfect or fundamentally flawed.
Consensus on early characters: When correct, all models agreed on the first 12 characters with zero variance.

Approach:

🏆 GPT-5's minimalism: Used only essential character names, achieving highest signal-to-noise ratio.
Gemini's thoroughness: Included supernatural entities (Hecate, Ghost of Banquo) that some editions omit - technically correct but borderline overcomplete.
Claude models' fundamental flaw: Both Opus and Sonnet treated the prompt as a "list important characters" rather than "strict chronological order" task.

Performance:

⚡ Speed-cost paradox: Sonnet 4.5 was fastest (3.3s) and cheapest ($0.0016) but scored 0% accuracy. Speed offers no value without correctness on verifiable tasks.
💰 Gemini 3 Pro's optimization: 75% cheaper and 67% faster than GPT-5 with identical accuracy.
Kimi K2's inefficiency: 257-second average despite $0.0157 cost - 40x slower than Sonnet with no quality improvement.

Surprises & Outliers:

🚨 Opus 4.1's deterministic failure: Despite being Anthropic's most powerful model, it produced the same incorrect answer (Duncan first) in 100% of runs, showing no self-correction mechanism.
Grok 4's token bloat: Used 705 input tokens (22x more than others) yet still failed, suggesting inefficient context handling.

Response Highlights

Best Response (GPT-5, Run 1):

First Witch
Second Witch
Third Witch
Duncan
Malcolm...

Most Problematic (Sonnet 4.5, Run 1):

Duncan
Malcolm
Donalbain
Macbeth
Banquo...
Witches (listed 18 lines later, out of order)

Most Creative Approach (Gemini 2.5 Pro, Run 1):

...Ghost of Banquo
Hecate
A Lord
First Apparition...

Ranking Justification

1st Place (Gemini 3 Pro): Achieved 50% accuracy with perfect formatting on correct runs, while delivering best-in-class speed (39s) and competitive cost. Flawed runs still maintained reasonable character ordering beyond the Macduff-Fleance error.

2nd Place (Gemini 2.5 Pro): Matched Gemini 3 Pro's accuracy but at 11% higher cost and 11% slower speed. Inclusion of optional characters like "Ghost of Banquo" shows thoroughness but slight over-verbosity vs. minimal prompt requirements.

3rd Place (GPT-5): Perfect accuracy on successful runs with cleanest outputs, but 3x slower and 48% more expensive than Gemini 3 Pro. Performance metrics disqualify it from top rank despite output quality.

4th Place (Grok 4): Never correct but consistently placed witches first (unlike Claude models), showing partial understanding. Performance was moderate but token usage was extremely inefficient (705 inputs vs. 32 for others).

5th Place (Kimi K2): Like Grok, occasionally got early ordering right but riddled with errors. 257-second average response time is unacceptable for this simple deterministic task.

6th Place (Sonnet 4.5): Fastest and cheapest but fundamentally misunderstood task purpose, demonstrating that speed is worthless without accuracy on verifiable prompts. Wild inconsistency across runs shows poor determinism.

7th Place (Opus 4.1): Consistently produced the same incorrect answer across all runs, showing no adaptive capacity. Failed the most basic requirement of putting the Three Witches first.

Scoring: Verifiable (exact scores + ranking)

3.6s $0.0018 Tokens i/o: 32/113

Duncan
Malcolm
Donalbain
Macbeth
Banquo
Lennox
Ross
Angus
Lady Macbeth
Attendant
Fleance
Porter
Macduff
Lady Macduff
Son
Siward
Young Siward
Seyton
Hecate
Witches
Murderers
Messenger
Doctor
Gentlewoman
Old Man
Servant
Captain
Lords
Soldiers
Attendants

Generated 28th Oct 2025

3.0s $0.0014 Tokens i/o: 32/87

Duncan
Malcolm
Donalbain
Macbeth
Banquo
Lennox
Ross
Angus
Lady Macbeth
Attendant
Messenger
Macduff
Porter
Old Man
Siward
Young Siward
Seyton
Doctor
Gentlewoman
Fleance
Hecate
Murderers

Generated 28th Oct 2025

3.8s $0.0018 Tokens i/o: 32/114

Witches
Duncan
Malcolm
Captain
Lennox
Ross
Angus
Macbeth
Banquo
Lady Macbeth
Attendant
Fleance
Porter
Macduff
Donalbain
Old Man
Murderers
Servant
Hecate
Lady Macduff
Son
Messenger
Doctor
Gentlewoman
Menteith
Caithness
Siward
Young Siward
Seyton

Generated 28th Oct 2025

2.8s $0.0015 Tokens i/o: 32/94

Duncan
Malcolm
Donalbain
Macbeth
Banquo
Ross
Angus
Lady Macbeth
Fleance
Macduff
Lennox
Porter
Lady Macduff
Son
Siward
Young Siward
Seyton
Hecate
Menteith
Caithness
Messenger
Doctor
Gentlewoman

Generated 28th Oct 2025