01 / INFERENCECorrelation is not causation
If umbrella sales and wet pavements rise together, umbrellas need not cause rain. Ask about reverse causation, common causes, and what intervention would change the outcome.
Try: What would happen if we changed X while holding other causes stable?
02 / DESIGNConfounding
A third factor can affect both exposure and outcome. People who voluntarily use a study tool may already be more motivated. Random assignment, careful design and sensitivity analyses help, but each has limits.
Try: Name one variable that predicts both the choice and the result.
03 / MEASUREMENTMeasurement error
“Motivation” cannot be observed directly like temperature. A survey, click count or diary samples only part of the construct. Reliability is consistency; validity asks whether the measure captures the intended meaning.
Try: What could this measure miss, distort or incentivize?
04 / REPRODUCTIONReplicability
One striking finding may depend on chance, analytic choices or context. Repeating a method and obtaining comparable results strengthens confidence; a failed replication also needs careful interpretation.
Try: Could a stranger reconstruct the analysis from your documentation?
05 / PROBABILITYBase rates
A test with 90% sensitivity and 90% specificity sounds decisive. But if only 1% of cases have the condition, among 1,000 cases you expect about 9 true positives and 99 false positives. Most positive results would be false positives in this illustrative scenario.
Try: Start with prevalence, then apply test accuracy.
06 / RESPONSIBILITYEthics
A clever design can still be unethical. Consent, privacy, power differences, possible harm and the right to withdraw belong in the method, not an appendix. Do not collect sensitive personal data without a sound purpose and protections.
Try: Who bears the risk, and who gets the benefit?
07 / GENERALIZATIONSampling
Who was reachable, who opted in, and who dropped out? A large convenience sample can still be biased. Sampling design matters more than impressive sample size alone when generalizing to a population.
Try: Who is missing from the data, and why?