Project Red Team - audit memo
Hawthorne Arena Sample Audit
Release status
not release-safe
At least one check failed. The headline claims in this document should not be repeated in a public decision until the findings below are answered. This is a finding about what the document shows, not a finding that its numbers are wrong.
not release-safe: at least one check failed and the headline claims should not travel yet.
A check is one of the 9 tests this audit runs against a submitted document. An evidence item is one disclosure a check looked for and did not match, so 4 failed checks can account for 5 evidence items. 1 further item is a limit on how much of the file we could read rather than a finding about the study, and is counted under Source Safety below; the audit's own register lists all 6 items together.
Every count above and below is produced by a text detector reading the document. Its measured limits are stated next, and they apply to all of them.
What Every Count In This Memo Is Worth
Every count in this memo comes from a text detector, not from a person reading the study. Measured on three studies it had not seen, it found 76% of the passages it looks for, and about 4 in 10 of the sentences it flagged turned out not to be what it was looking for. Recall on individual documents ran 67% to 87%, and those ranges overlap, so we do not claim it reads one firm's writing better than another's. Treat every count here as approximate.
A count in this pack may be set against another study's count only where three things hold: both were extracted on the same basis, the detector's measured error is printed beside both, and the ordering survives resampling. Where the ordering does not survive, we publish the band and the reason for it instead of a rank.
The Three Failed Checks This Memo Leads With
These are the three largest. One further check also failed and is listed in full below; closing these three would not clear the document.
1 Whether the tax figure is a net gain to the public purse
Fiscal revenue is detected, but no cost-side or net-fiscal treatment was detected.
What follows from it: Do not release fiscal benefit language as net benefit.
Basis: the audit did not find it. That is a miss, not an absence.
2 Whether the survey behind the spending figure carries the weight put on it
Survey evidence is detected, but response coverage was not detected.
What follows from it: Survey-based claims should carry a data-quality penalty.
Basis: the audit did not find it. That is a miss, not an absence.
3 Whether the study says which year's model and which year's dollars it used
The model is named, but model year and/or dollar year is missing.
What follows from it: Do not release year-sensitive claims without vintage disclosure.
Basis: the audit did not find it. That is a miss, not an absence.
Not Found In The Document As Submitted
Everything in this section is a miss by our detector, not an absence in the study. The detector reads for known phrasings and studies phrase things in ways no list anticipates. Read each line as "we did not find it", and close it by pointing us at the passage.
| Subject | What the audit found | Why it matters | Weight |
|---|---|---|---|
| Whether the tax figure is a net gain to the public purse | Not found in the document as submitted: Public service costs, abatements, incentives, or other cost-side fiscal evidence. Tax or fiscal revenue appears without a detected public cost-side ledger. | Gross tax revenue may be mistaken for net fiscal benefit. | blocking |
| Whether the survey behind the spending figure carries the weight put on it | Not found in the document as submitted: Survey response rate, sample frame, and usable response count. Survey language appears without response-rate or sample-size disclosure. | Survey-derived spending or input assumptions cannot be weighted for coverage risk. | blocking |
| Whether the study says which year's model and which year's dollars it used | Not found in the document as submitted: Model year or data year. IMPLAN is named, but no model/data year was detected. | Model structure may not match the spending period. | blocking |
| Whether the study says which year's model and which year's dollars it used | Not found in the document as submitted: Dollar year or currency year. IMPLAN is named, but no dollar/currency year was detected. | Nominal and real dollar claims may be mixed. | blocking |
| Whether the tax figure is gross collections or money this jurisdiction keeps | Not found in the document as submitted: Gross-vs-net tax treatment and retention/abatement assumptions. Tax revenue appears without gross/net or tax-retention language. | A gross tax collection can be mistaken for fiscal benefit to the decision maker. | blocking |
Counted by detector: 76% recall on held-out studies, about 4 in 10 flags not what was sought, and per-document recall observed across 67% to 87% with overlapping intervals. Full limits are stated above.
Contradicted By The Study's Own Disclosures
Everything in this section rests on text the study does contain. These are not misses: the document's own disclosure works against the claim attached to it, so they survive the detector's recall limit and a reader cannot close them by finding a passage we overlooked.
No finding of this kind was recorded: the detector matched nothing in the document working against a claim the document makes. At 76% measured recall that is a result about the detector, not proof that there was nothing to find. Every finding above is a gap in what we could confirm, not evidence that the study is wrong.
Claims A Reader May Quote Only With Support
These are the figures a reader is most likely to repeat back in a public meeting. Each one is quotable only with the support named beside it and the findings above attached to it.
| What is claimed | Figure as stated | Where it was found | The sentence it came from |
|---|---|---|---|
| Jobs | 1,240 jobs | Stated in the sentence quoted | Executive summary: the arena will create 1,240 jobs and generate $4.6 million in annual state and local tax revenue. |
| Tax and fiscal revenue | $4.6 million | Stated in the sentence quoted | Executive summary: the arena will create 1,240 jobs and generate $4.6 million in annual state and local tax revenue. |
| Visitor spending | $18.2 million | Stated in the sentence quoted | The arena attracts 310,000 visitors whose $18.2 million of spending is counted in the model. |
| Return on investment or benefit-cost ratio | 7.0:1 ratio | Stated in the sentence quoted | A $4.0 million city grant returns $28.0 million, a 7.0:1 ROI. |
Source Safety
audit-safe: text; plain-text extraction; 15 text lines; 0 tables; confidence 100% on the source-extraction scale
This section is about our reading of the file you sent us, not about the study's contents. A thin extraction lowers what any finding below is worth, in both directions.
| What the audit could not read | What that costs | Weight |
|---|---|---|
| Not found in the file as extracted: Tables or structured figures that can reconcile prose claims. | Prose figures cannot be checked against tabulated support. | caution |
Counted by detector: 76% recall on held-out studies, about 4 in 10 flags not what was sought, and per-document recall observed across 67% to 87% with overlapping intervals. Full limits are stated above.
What Human Review Would Change
This audit reads a document. It does not re-run the study's model, and it cannot tell a defensible number from an indefensible one. What it can do is tell a reader exactly which passages to go and find. Every line in the not-found column closes the moment somebody points at the passage.
5 evidence items, across 4 checks that failed, hold this document at not release-safe. 5 of those items close if a reader finds the passage the audit missed. Every one of them closes that way.
| Subject | What the audit found | What would close it |
|---|---|---|
| Whether the tax figure is a net gain to the public purse | Not found in the document as submitted: Public service costs, abatements, incentives, or other cost-side fiscal evidence. Tax or fiscal revenue appears without a detected public cost-side ledger. | Point the audit at public service costs, abatements, incentives, or other cost-side fiscal evidence in the full report. |
| Whether the survey behind the spending figure carries the weight put on it | Not found in the document as submitted: Survey response rate, sample frame, and usable response count. Survey language appears without response-rate or sample-size disclosure. | Point the audit at survey response rate, sample frame, and usable response count in the full report. |
| Whether the study says which year's model and which year's dollars it used | Not found in the document as submitted: Model year or data year. IMPLAN is named, but no model/data year was detected. | Point the audit at model year or data year in the full report. |
| Whether the study says which year's model and which year's dollars it used | Not found in the document as submitted: Dollar year or currency year. IMPLAN is named, but no dollar/currency year was detected. | Point the audit at dollar year or currency year in the full report. |
| Whether the tax figure is gross collections or money this jurisdiction keeps | Not found in the document as submitted: Gross-vs-net tax treatment and retention/abatement assumptions. Tax revenue appears without gross/net or tax-retention language. | Point the audit at gross-vs-net tax treatment and retention/abatement assumptions in the full report. |
Audit reading: release status not release-safe; 4 claims inventoried; 5 evidence items not found; 0 findings contradicted by the document; 1 reading limit on the file; 13 evidence observations.
This memo audits the document as submitted. It does not prove the underlying model is correct; it shows what the document claims, what support was detected, and what evidence was not found. A finding here is a reason to ask a question, not a finding of error.