Most of the business documents that reach my desk are not clean. They contain partial numbers, overlapping customer descriptions that do not quite match, strategic recommendations that sit at a slight angle to the earlier constraints, and at least one important detail that appears only in a footnote or an email thread. When I evaluate a model I therefore prefer to start with material that already looks like the real thing. Clean prompts produce clean stories. Messy prompts produce diagnostic information. This field test uses a deliberately imperfect business brief to examine how Opus 5.2 behaves when the input is incomplete and internally inconsistent. The goal is not theater. The goal is a clearer map of where the model stays useful and where human review becomes non-negotiable.
Why a Messy Brief Is a Better Diagnostic Than a Polished One
Clean Inputs Conceal the Actual Failure Modes
A well-structured prompt with complete data and consistent goals tends to elicit fluent, well-organized output. That fluency is real and often helpful. It does not reveal how the model handles missing causal links, conflicting segment descriptions, or recommendations that drift from stated constraints. Those conditions are common in professional work. A test that avoids them produces an incomplete picture of reliability.
Messiness Forces Specific Error Patterns Into the Open
Incomplete quantitative data invites invention of plausible bridging numbers. Conflicting qualitative descriptions invite over-smoothing into a single coherent narrative. Ambiguous priorities invite confident ranking that is not fully supported by the source. Each of these patterns is easier to observe—and therefore easier to plan around—when the source material is allowed to remain imperfect.
The Test Is Designed to Be Diagnostic Rather Than Dramatic
I am not trying to generate a viral success story or a dramatic takedown. I am trying to record, under controlled conditions, which parts of the task the model handles with acceptable stability and which parts require mandatory human intervention. The value lies in the boundary information, not in the emotional valence of the result.
Test Design and Materials
Construction of the Synthetic Brief
I prepared a synthetic business brief of moderate length, roughly the scale of an internal strategy memo. It contained a stated growth objective, partial financial figures for two consecutive periods, two customer-segment descriptions whose attributes overlapped but did not fully align, a set of operational constraints, and a concluding strategic recommendation that sat at a mild angle to those constraints. No claim is made that this brief mirrors any actual company. It was constructed only to embed realistic forms of incompleteness and mild contradiction.
Task Instructions Given to the Model
The model was asked to produce four elements: a short situation summary grounded only in the provided text, an explicit list of internal contradictions or missing data, a prioritized set of open questions that a human team should resolve next, and a cautious recommendation that remained inside the available evidence. The instructions emphasized flagging uncertainty rather than resolving it. No additional clarifying context was supplied during the runs.
Controlled Conditions and Scoring Approach
The identical brief and instructions were used across multiple independent runs. Outputs were scored on four dimensions: fidelity to the source material, completeness of contradiction and gap detection, resistance to invented detail, and structural stability across runs. Human review time required to bring each output to an acceptable risk level was also recorded. The following table summarizes the scoring dimensions:
Scoring Dimension | What Was Measured | Pass Threshold Used in This Test |
|---|---|---|
Source Fidelity | Presence of invented numbers, claims, or causal links | No unsupported quantitative or causal statements |
Gap and Contradiction Detection | Explicit identification of conflicts and missing data | At least two major conflicts or gaps named |
Invention Resistance | Frequency of plausible but unsupported bridging detail | Zero unflagged inventions in core sections |
Cross-Run Stability | Consistency of structure and key findings | Material findings appear in majority of runs |
Observed Behavior Across the Runs
Situation Summaries Remained Mostly Anchored
In the majority of runs the high-level situation summary correctly reflected the stated growth objective and the main quantitative claims that actually appeared in the brief. The prose was fluent and the tone remained professional. This portion of the task showed relatively high source fidelity and did not require heavy rewriting.
Contradiction Detection Varied Noticeably
Some runs explicitly listed the mismatched customer-segment attributes and the mild misalignment between the final recommendation and the earlier operational constraints. Other runs softened or omitted one of the conflicts, producing a smoother narrative at the cost of completeness. The variance itself is useful information: contradiction detection was possible but not automatic under the tested conditions.
Invented Bridging Detail Appeared When Causal Links Were Absent
When the brief left a causal connection unspecified, a subset of runs supplied a plausible explanation that was not present in the source. The invented link was rarely marked as speculative. This pattern is consistent with behavior I have observed on other incomplete inputs and must be treated as a known risk rather than a surprise.
Open-Question Lists Provided Consistent Practical Value
The request for prioritized open questions produced more stable and more immediately useful output than the request for a single integrated recommendation. The question lists surfaced genuine gaps and gave a human reviewer clear next investigative steps. In several runs the open-question section was the portion I would be most willing to hand to a client team with only light editing.
Human Review Requirements Made Visible by the Test
Causal and Recommendatory Claims Require Source-Level Checking
Any statement that linked cause to effect or that advanced a recommendation beyond the explicit evidence required direct comparison with the original brief. Fluency of expression was not a reliable signal of grounding. In practice this means the recommendation section cannot be treated as low-review content.
Uncertainty Labeling Still Needs Human Reinforcement
Even when the instructions asked the model to flag uncertainty, some outputs presented inferred connections with the same surface confidence as directly supported claims. Reviewers must therefore apply their own uncertainty markers rather than relying solely on the model’s self-reporting.
Review Load Scales With the Degree of Messiness
Cleaner inputs can support lighter review. Highly incomplete or internally conflicting briefs demand heavier fact-checking and more frequent returns to the source. The review-time numbers recorded in this test were higher than those I typically see with well-structured source material. That difference should be expected and budgeted for.
Practical Takeaways for Similar Professional Tasks
Prefer Explicit Gap Analysis Over Forced Synthesis
When source material is known to be messy, instructing the model to surface contradictions and missing data often yields more usable output than asking it to produce a polished, fully integrated view. The gap-focused output gives the human team a clearer agenda and reduces the risk of silently resolved conflicts.
Estimate Review Time Proportionally to Input Quality
A workflow that looks efficient on clean documents can become net-negative on messy ones once verification time is included. Teams should adjust their expectations and their time estimates according to the quality and consistency of the incoming material rather than applying a single productivity multiplier.
Treat Cross-Run Stability as a Gate for Lower-Supervision Use
If repeated runs on the same messy brief produce materially different contradiction lists or divergent recommendations, the workflow is not yet stable enough for low-supervision deployment. Stability is a stronger signal of readiness than any single impressive output.
The following table captures the practical decision rules I now apply after this class of test:
Observed Pattern | Immediate Practical Response |
|---|---|
Strong situation summary, weak gap detection | Use model for first-pass overview only; human owns gap list |
Frequent unflagged inventions | Require source citation or explicit “not in source” tags |
High variance across runs | Do not reduce human review intensity yet |
Stable open-question lists | Candidate for light-edit handoff to team |
What This Field Test Does Not Claim
It Does Not Establish Performance Across All Business Document Types
One synthetic brief, however carefully constructed, remains a single data point. Other lengths, domains, conflict densities, and stakeholder contexts will surface different behavior. Local validation is still required.
It Does Not Provide Cost, Pricing, or Interface Details
No token counts, dollar costs, rate limits, or interface behaviors are asserted here. Those factors matter for adoption decisions and must be measured under the actual account and tooling conditions of the team considering the workflow.
It Does Not Replace Context-Specific Risk Assessment
Teams operating under regulatory constraints, high reputational risk, or strict audit requirements will correctly apply heavier review standards than teams handling internal exploratory work. The test supplies method and observed patterns; it does not supply a universal approval.
Closing the Field Test
A messy business brief is closer to the documents that arrive in ordinary professional life than a perfectly curated prompt. Under those conditions Opus 5.2 produced fluent and mostly grounded situation summaries together with useful open-question lists, while still showing uneven contradiction detection and occasional invented bridging detail.
The practical response is neither blanket rejection nor uncritical adoption. It is a workflow that leans on the model for gap analysis and first-pass orientation, insists on source-backed human review for any causal or recommendatory claim, and treats cross-run stability as a requirement rather than a luxury. Useful remains a higher bar than impressive. This test is one more calibrated data point on the way to clearing that bar with eyes open.
