← Insights

How to read an impact study before you stake money on it

You do not have to be an economist to catch a study that will not hold up. The few things to check before you quote its number or stake money on it.

You are the one who has to defend it

When a study lands on your desk, whether you are a commissioner, a council member, a lender, or a funder, the number in it is about to become your number. You will quote it, fund against it, or vote on it, and if it does not hold up, that is your problem, not the consultant's. The good news is that you do not need a degree in economics to catch the most common ways a study goes wrong. You need to know where to look.

Is it impact, or is it contribution

The first check is what the study is actually measuring. An impact study estimates new activity a project creates. A contribution study describes activity that already exists. Both are legitimate, and they answer different questions, so a contribution number presented as impact will always look enormous. If the headline figure seems too large for the project, this is usually why.

Is the spending new, or just moved

The next check is whether the study separated genuinely new money from spending that was already happening locally. A project that pulls customers from other local businesses moves dollars around without growing the regional economy. If the study counts all the activity as new, without ever asking what would have happened anyway, the number is inflated by design.

Does every big claim have a source

Follow the largest numbers to their support. A defensible study shows where its key figures come from, whether from federal data, a real survey, or a documented assumption. If the biggest claims trace back to nothing you can check, or to a survey too small to mean anything, treat the number as a guess dressed up as a finding.

Does it admit what could be wrong

An honest study tells you what it does not know, what it assumed, and what would change the answer. One that reports a single confident number with no range, no assumptions, and no discussion of risk is selling certainty it does not have. That is the study most likely to embarrass whoever quotes it.

When it is worth a second set of eyes

If a study is going to justify real money, a public vote, or a loan, it is worth having someone inventory its claims and check the support before you rely on it. That is the whole idea behind a red team: find the weak assumption at your desk, not after the money is committed.