Capstone: Replicate, Extend, Defend

The course converges on one project. You choose a published paper that estimates a local projection, reproduce one of its results from the authors’ replication package, add one motivated extension, and hand the whole package to another team who tries to reproduce it in a clean directory. The audit comes back to you; you answer it.

A novel research contribution is welcome but not required. A null result can receive full credit. An unsupported causal claim cannot.

What the extension may be

The extension changes exactly one of the following, and you must say which of the estimand, the identifying assumptions, the precision, or the interpretation it changes:

  • the outcome variable;
  • a defensible sample restriction;
  • the shock or instrument measure;
  • the inference procedure; or
  • one pre-specified dimension of heterogeneity.

An extension that changes the estimand is not wrong, but it is a different question, and the write-up must say so.

Milestones

Checkpoint Submission
After Lecture 4 Research question, candidate paper, proposed estimand, and replication target (figure or table component, paper version, data vintage).
After Lecture 7 Working baseline replication and a discrepancy record.
After Lecture 10 One pre-specified extension with its identifying assumptions written before estimation.
After Lecture 13 Complete reproducibility package, ready for peer audit.
Lecture 14 Final submission, audit response, and defense.

The package

Your package is a directory that another person can run without asking you a question:

capstone-<surname>/
├── README.md              # entry point, software version, expected runtime
├── master.do              # rebuilds every output from data/raw
├── data/
│   ├── raw/               # untouched, with PROVENANCE.md
│   └── derived/           # generated by master.do
├── code/
├── output/                # figures, tables, saved estimates, logs
├── design-memo.md         # question, estimand, identification, extension
├── discrepancy-log.md     # every difference from the published result
└── report.md              # methods, results, limitations, audit responses

Redistribute published data only when the package’s license permits it. Otherwise document the acquisition steps exactly and ship a script that performs them.

The design memo

Write the memo before the first regression and revise it as you learn. It answers, in this order:

  1. Question. What does a shock do to what, over what horizon, for whom?
  2. Estimand. \beta_h of which regression, in which units, normalized how, accumulated or not.
  3. Identification. Why s_t is orthogonal to u_{t,h} given \mathbf w_t; which assumption (A1–A3 of Lecture 3, or the instrument conditions of Lecture 4, or the parallel-trends and no-anticipation conditions of Lecture 12) carries the weight; what would violate it.
  4. Inference. Which procedure, matched to which claim (Lectures 5–6).
  5. Replication target. Paper version, figure or table, data vintage, tolerance.
  6. Extension. What changes, and which of estimand, assumptions, precision, or interpretation it affects.

The discrepancy log

Every difference between your reproduction and the published result is a row:

Field Content
Object Which coefficient, standard error, sample size, or figure feature
Published The published value
Reproduced Your value
Class data / specification / software / inference / presentation
Diagnosis What you believe caused it, and the evidence
Status resolved / explained / open

A log with no rows is suspicious. A log whose open rows are the most consequential ones is honest.

The audit

The auditor receives the package, not the author. In a clean directory, they run master.do, compare outputs with the published target and with the author’s reported values, and produce a replication audit:

  1. Did the package run as documented? Where did it need intervention?
  2. Which published numbers were reproduced within tolerance?
  3. For each discrepancy, is the author’s classification and diagnosis supported?
  4. What is the single most consequential unresolved issue, and does it change the paper’s conclusion?
  5. Does the extension’s write-up correctly state what it changes?

The audit concerns the published benchmark, not only the extension. The author then responds in writing and updates the discrepancy log.

Grading

Component Weight
Stata problem sets 1–13 40%
Replication records 1–13 25%
Browser-lab mastery checks 10%
Integrated Lecture 14 submission (package, audit, response, defense) 25%

The mastery-based alternative assesses conceptual understanding, technical implementation, and research judgment separately, with reassessment on fresh examples. Three gates apply in that version: correct timing, transformations, units, and sample construction before any empirical extension; a defensible baseline identification and inference strategy before any heterogeneous design; and a successful audit of an unfamiliar analysis before completion.