Research history

Inert source text, including its original formatting. It is not executed as page markup.

A **counterfactual** is what would have happened otherwise. The counterfactual impact of a gift is the difference between the world with the gift and the world without it. A program’s results and a gift’s results are not the same thing. If the program would have been funded anyway, the gift’s real effect lands somewhere else.

## Three ways the counterfactual changes the picture

**Funding gaps get filled.** Benjamin Todd of 80,000 Hours warns against treating a charity’s funding gap as if you were the only donor. If other donors would have filled it, your gift does less than it seems. The opposite mistake is also possible: assuming your gift does nothing because others would give. Filling the gap frees those donors for other work and saves the charity fundraising time, so the honest answer depends on what everyone else does.

**Other funders respond.** GiveWell uses two words for this. *Leverage* is a charity’s influence on how others, such as governments, allocate their money. *Funging* is when spending on a program crowds out money that would have come from somewhere else. Since late 2017 GiveWell has estimated both for each charity it recommends, and it says the estimates involve substantial guesswork. Its test of “room for more funding” asks the same question from the charity’s side: what would extra money achieve, beyond what the charity would raise anyway?

**The comparison group decides.** A program can look like a success until you see what happened to similar people without it. A Cochrane review of nine trials of “Scared Straight” programs, which organize prison visits for young people at risk of offending, found that they increased offending compared with doing nothing at all. See [the case study](/record/case-studies/scared-straight/).

## Shared credit

Most outcomes have many causes. Todd points out that when two groups are each necessary for a result, both can be counterfactually responsible for all of it, so claimed impacts can add up to more than 100 percent. That is not a logical error. The practical risk is that funders, each counting the whole result, collectively pay more than it is worth.

## Why it is hard

The counterfactual is never observed. Randomized trials estimate it for the people in a study. Ordinary gifts get no estimate at all, which is part of why the [feedback loop](/diagnosis/missing-feedback-loop/) is missing. Producing counterfactual estimates cheaply and at scale is [Open Problem 4](/open-problems/the-counterfactual-problem/).

{
  "adapterVersion": "research-encyclopedia-v1",
  "authorship": {
    "actorIds": [],
    "complete": false,
    "refs": [],
    "state": "unknown"
  },
  "hash": "eec7fac2e5e67320558ffe4121312521997b552abe4cd3f61812bf97fc24b400",
  "id": "version:concept-counterfactual-impact-v1",
  "previousId": null,
  "recordId": "concept:counterfactual-impact",
  "sources": [
    {
      "selector": {
        "kind": "entry"
      },
      "sourceId": "source:encyclopedia-counterfactual-impact-v1"
    }
  ],
  "subject": {
    "body": "\nA **counterfactual** is what would have happened otherwise. The counterfactual impact of a gift is the difference between the world with the gift and the world without it. A program’s results and a gift’s results are not the same thing. If the program would have been funded anyway, the gift’s real effect lands somewhere else.\n\n## Three ways the counterfactual changes the picture\n\n**Funding gaps get filled.** Benjamin Todd of 80,000 Hours warns against treating a charity’s funding gap as if you were the only donor. If other donors would have filled it, your gift does less than it seems. The opposite mistake is also possible: assuming your gift does nothing because others would give. Filling the gap frees those donors for other work and saves the charity fundraising time, so the honest answer depends on what everyone else does.\n\n**Other funders respond.** GiveWell uses two words for this. *Leverage* is a charity’s influence on how others, such as governments, allocate their money. *Funging* is when spending on a program crowds out money that would have come from somewhere else. Since late 2017 GiveWell has estimated both for each charity it recommends, and it says the estimates involve substantial guesswork. Its test of “room for more funding” asks the same question from the charity’s side: what would extra money achieve, beyond what the charity would raise anyway?\n\n**The comparison group decides.** A program can look like a success until you see what happened to similar people without it. A Cochrane review of nine trials of “Scared Straight” programs, which organize prison visits for young people at risk of offending, found that they increased offending compared with doing nothing at all. See [the case study](/record/case-studies/scared-straight/).\n\n## Shared credit\n\nMost outcomes have many causes. Todd points out that when two groups are each necessary for a result, both can be counterfactually responsible for all of it, so claimed impacts can add up to more than 100 percent. That is not a logical error. The practical risk is that funders, each counting the whole result, collectively pay more than it is worth.\n\n## Why it is hard\n\nThe counterfactual is never observed. Randomized trials estimate it for the people in a study. Ordinary gifts get no estimate at all, which is part of why the [feedback loop](/diagnosis/missing-feedback-loop/) is missing. Producing counterfactual estimates cheaply and at scale is [Open Problem 4](/open-problems/the-counterfactual-problem/).\n",
    "collection": "encyclopedia",
    "frontmatter": {
      "aliases": [
        "counterfactual",
        "additionality",
        "replaceability",
        "funging"
      ],
      "conditions": [
        {
          "collection": "conditions",
          "id": "missing-feedback-loop"
        },
        {
          "collection": "conditions",
          "id": "measurement-gap"
        }
      ],
      "related": [
        "/open-problems/the-counterfactual-problem/",
        "/record/encyclopedia/neglectedness/",
        "/record/principles/05-counterfactuals-and-neglect/",
        "/record/lexicon/#counterfactual",
        "/tools/impact-calculator/"
      ],
      "sources": [
        {
          "note": null,
          "publisher": "Benjamin Todd, 80,000 Hours",
          "title": "Counterfactuals and how they change our view of what does good",
          "url": "https://80000hours.org/articles/counterfactuals/",
          "year": 2021
        },
        {
          "note": null,
          "publisher": "Benjamin Todd, 80,000 Hours",
          "title": "Doing good together: how to coordinate effectively and avoid single-player thinking",
          "url": "https://80000hours.org/articles/coordination/",
          "year": 2018
        },
        {
          "note": null,
          "publisher": "James Snowden, The GiveWell Blog",
          "title": "Revisiting leverage",
          "url": "https://blog.givewell.org/2018/02/13/revisiting-leverage/",
          "year": 2018
        },
        {
          "note": null,
          "publisher": "GiveWell",
          "title": "Our Criteria",
          "url": "https://www.givewell.org/how-we-work/criteria",
          "year": null
        },
        {
          "note": null,
          "publisher": "Anthony Petrosino and colleagues, Cochrane Database of Systematic Reviews",
          "title": "“Scared Straight” and other juvenile awareness programs for preventing juvenile delinquency",
          "url": "https://www.cochrane.org/evidence/CD002796_scared-straight-and-other-juvenile-awareness-programs-preventing-juvenile-delinquency",
          "year": 2022
        }
      ],
      "summary": "The real effect of a gift is the difference between what happens with it and what would have happened without it. Estimating that difference is one of the hardest and most important problems in giving.",
      "title": "Counterfactual impact"
    },
    "type": "entry"
  },
  "subjectHash": "8c2e85325f2d4b5512fc2aa93c8f0a83158c6cd3785ccf7a21a8811f22a7a742",
  "version": 1
}