Project Red Team - audit memo
Hawthorne Arena Sample Audit
Release status
not release-safe
At least one check failed. The headline claims in this document should not be repeated in a public decision until the findings below are answered. This is a finding about what the document shows, not a finding that its numbers are wrong.
not release-safe: at least one check failed and the headline claims should not travel yet.
A check is one of the 9 tests this audit runs against a submitted document. An evidence item is one disclosure a check looked for and did not match, so 4 failed checks can account for 5 evidence items. 1 further item is a limit on how much of the file we could read rather than a finding about the study, and is counted under Source Safety below; the audit's own register lists all 6 items together.
Every count above and below is produced by a text detector reading the document. Its measured limits are stated next, and they apply to all of them.
What Every Count In This Memo Is Worth
Every count in this memo comes from a text detector, not from a person reading the study. Measured on three studies it had not seen, it found 76% of the passages it looks for, and about 4 in 10 of the sentences it flagged turned out not to be what it was looking for. Recall on individual documents ran 67% to 87%, and those ranges overlap, so we do not claim it reads one firm's writing better than another's. Treat every count here as approximate.
A count in this pack may be set against another study's count only where three things hold: both were extracted on the same basis, the detector's measured error is printed beside both, and the ordering survives resampling. Where the ordering does not survive, we publish the band and the reason for it instead of a rank.
The Three Failed Checks This Memo Leads With, On Support Our Detector Did Not Match
These are the three largest. One further check also failed and is listed in full below; closing these three would not clear the document.
1 Whether the tax figure is a net gain to the public purse
Fiscal revenue is detected, but no cost-side or net-fiscal treatment was detected.
What follows from it: Do not release fiscal benefit language as net benefit.
Basis: our detector did not match the support in the document as submitted. That is a miss, not an absence.
2 Whether the survey behind the spending figure carries the weight put on it
Survey evidence is detected, but response coverage was not detected.
What follows from it: Survey-based claims should carry a data-quality penalty.
Basis: our detector did not match the support in the document as submitted. That is a miss, not an absence.
3 Whether the study says which year's model and which year's dollars it used
IMPLAN is named, but neither a model year nor a dollar year was detected.
What follows from it: Do not release year-sensitive claims without vintage disclosure.
Basis: our detector did not match the support in the document as submitted. That is a miss, not an absence.
Not Found In The Document As Submitted
Everything in this section is a miss by our detector, not an absence in the study. The detector reads for known phrasings and studies phrase things in ways no list anticipates. Read each line as "we did not find it", and close it by pointing us at the passage.
| Subject | What the audit found | Why it matters | Weight |
|---|---|---|---|
| Whether the tax figure is a net gain to the public purse | Our detector did not match this in the document as submitted: Public service costs, abatements, incentives, or other cost-side fiscal evidence. What the detector read: tax or fiscal revenue appears without a detected public cost-side ledger. | Gross tax revenue may be mistaken for net fiscal benefit. | blocking |
| Whether the survey behind the spending figure carries the weight put on it | Our detector did not match this in the document as submitted: Survey response rate, sample frame, and usable response count. What the detector read: survey language appears without response-rate or sample-size disclosure. | Survey-derived spending or input assumptions cannot be weighted for coverage risk. | blocking |
| Whether the study says which year's model and which year's dollars it used | Our detector did not match this in the document as submitted: Model year or data year. What the detector read: IMPLAN is named, but no model/data year was detected. | Model structure may not match the spending period. | blocking |
| Whether the study says which year's model and which year's dollars it used | Our detector did not match this in the document as submitted: Dollar year or currency year. What the detector read: IMPLAN is named, but no dollar/currency year was detected. | Nominal and real dollar claims may be mixed. | blocking |
| Whether the tax figure is gross collections or money this jurisdiction keeps | Our detector did not match this in the document as submitted: Gross-vs-net tax treatment and retention/abatement assumptions. What the detector read: tax revenue appears without gross/net or tax-retention language. | A gross tax collection can be mistaken for fiscal benefit to the decision maker. | blocking |
Counted by detector: 76% recall on held-out studies, about 4 in 10 flags not what was sought, and per-document recall observed across 67% to 87% with overlapping intervals. Full limits are stated above.
Where These Checks Have Already Missed A Real Disclosure
These are misses this detector has actually made on published economic impact studies. Each was found afterwards by hand, in text this same extraction had already recovered. They are printed beside the rows above so a reader can see what a not-found row is worth before acting on one. The studies are unnamed because the error in each case was ours.
| Check | What it reads for | What a published study wrote instead | What that made this pack say |
|---|---|---|---|
| Whether the survey behind the spending figure carries the weight put on it | A percentage, or a count set against its frame in the form “116 of 442”, or the words “usable responses”. | “administered an online survey to 442 craft brewers who are members of the organization. Approximately 116 brewers responded to the survey.” That is a sample frame, a usable response count and an implied 26% rate, written in prose on the second page. | This pack listed the response rate as not found. The disclosure was in the document and in the text this audit extracted from it. The detector reads for a form the study did not use. |
| Whether the tax figure is gross collections or money this jurisdiction keeps | Gross-versus-net or retention language in the same sentence as a tax figure. | “fiscal benefits (in the form of gross tax receipts) for Bexar County”, with the labelling clause in one sentence and the dollar figure in the next. | This check accepts the words “gross tax” and failed the document anyway, because only the sentence carrying the dollar figure falls inside the window it reads. |
| Whether the tax figure is a net gain to the public purse | Cost-side or net-fiscal language near a tax claim. | A table headed “Detailed Net Fiscal Impacts by Level of Gov.”, with a net position by level of government tabulated under it. | This pack listed the cost side as not found. The net fiscal impacts were tabulated under a heading the tax-claim window does not reach. |
Contradicted By The Study's Own Disclosures
Everything in this section rests on text the study does contain. These are not misses: the document's own disclosure works against the claim attached to it, so they survive the detector's recall limit and a reader cannot close them by finding a passage we overlooked.
No finding of this kind was recorded: the detector matched nothing in the document working against a claim the document makes. At 76% measured recall that is a result about the detector, not proof that there was nothing to find. Every finding above is a gap in what we could confirm, not evidence that the study is wrong.
Claims A Reader May Quote Only With Support
These are the figures a reader is most likely to repeat back in a public meeting. Each one is quotable only with the support named beside it and the findings above attached to it.
| What is claimed | Figure as stated | Where it was found | The sentence it came from |
|---|---|---|---|
| Jobs | 1,240 jobs | Stated in the sentence quoted | Executive summary: the arena will create 1,240 jobs and generate $4.6 million in annual state and local tax revenue. |
| Tax and fiscal revenue | $4.6 million | Stated in the sentence quoted | Executive summary: the arena will create 1,240 jobs and generate $4.6 million in annual state and local tax revenue. |
| Visitor spending | $18.2 million | Stated in the sentence quoted | The arena attracts 310,000 visitors whose $18.2 million of spending is counted in the model. |
| Return on investment or benefit-cost ratio | 7.0:1 ratio | Stated in the sentence quoted | A $4.0 million city grant returns $28.0 million, a 7.0:1 ROI. |
Source Safety
audit-safe: plain text; plain-text extraction; 15 text lines; 0 tables; 100% text extraction quality; 100% source readiness
This source arrived as text and its text was read in full. Nothing stood between the audit and the document, so no finding here is limited by how the file was laid out.
This section is about our reading of the file you sent us, not about the study's contents. A thin extraction lowers what any finding below is worth, in both directions.
| What limited our reading of the file | What that costs | Weight |
|---|---|---|
| Our reconstruction of the submitted file did not recover this: Tables or structured figures that can reconcile prose claims. | Prose figures cannot be checked against tabulated support. | caution |
Counted by detector: 76% recall on held-out studies, about 4 in 10 flags not what was sought, and per-document recall observed across 67% to 87% with overlapping intervals. Full limits are stated above.
What Human Review Would Change
This audit reads a document. It does not re-run the study's model, and it cannot tell a defensible number from an indefensible one. What it can do is tell a reader exactly which passages to go and find. Every line in the not-found column closes the moment somebody points at the passage.
5 evidence items, across 4 checks that failed, hold this document at not release-safe. 5 of those items close if a reader finds the passage the audit missed. Every one of them closes that way.
| Subject | What the audit found | What would close it |
|---|---|---|
| Whether the tax figure is a net gain to the public purse | Our detector did not match this in the document as submitted: Public service costs, abatements, incentives, or other cost-side fiscal evidence. What the detector read: tax or fiscal revenue appears without a detected public cost-side ledger. | Point the audit at public service costs, abatements, incentives, or other cost-side fiscal evidence in the full report. |
| Whether the survey behind the spending figure carries the weight put on it | Our detector did not match this in the document as submitted: Survey response rate, sample frame, and usable response count. What the detector read: survey language appears without response-rate or sample-size disclosure. | Point the audit at survey response rate, sample frame, and usable response count in the full report. |
| Whether the study says which year's model and which year's dollars it used | Our detector did not match this in the document as submitted: Model year or data year. What the detector read: IMPLAN is named, but no model/data year was detected. | Point the audit at model year or data year in the full report. |
| Whether the study says which year's model and which year's dollars it used | Our detector did not match this in the document as submitted: Dollar year or currency year. What the detector read: IMPLAN is named, but no dollar/currency year was detected. | Point the audit at dollar year or currency year in the full report. |
| Whether the tax figure is gross collections or money this jurisdiction keeps | Our detector did not match this in the document as submitted: Gross-vs-net tax treatment and retention/abatement assumptions. What the detector read: tax revenue appears without gross/net or tax-retention language. | Point the audit at gross-vs-net tax treatment and retention/abatement assumptions in the full report. |
What This Audit Can Still Be Used For
What this audit matched is usable now. The contradicted findings rest on text the document contains, the claims inventory records what the document asserts, and the source-safety reading is a measurement of the file itself. What the detector did not match settles nothing in either direction. A miss is a miss: it does not show that the document lacks a disclosure, and it does not show that it carries one.
Audit reading: release status not release-safe; 4 claims inventoried; 5 evidence items not found; 0 findings contradicted by the document; 1 reading limit on the file; 13 evidence observations.
This memo audits the document as submitted. It does not prove the underlying model is correct; it shows what the document claims, what support was detected, and what evidence was not found. A finding here is a reason to ask a question, not a finding of error.