goodthingswithdata.com

Data in the MMWR

Conclusions (6 of 12)

Data in the MMWR (§5.5): Draw conclusions under incomplete information
Posted to LinkedIn

2026-01-20

NotePreviously published

Posted to LinkedIn on 2026-01-20. The text below is the post as published.

Read the original on LinkedIn

Links to works now on this site point to their pages here; the words are as published. Where the post showed a web address, the icon stands in for it (hover over it to see the address as posted).

Data in the MMWR (§5.5): Draw conclusions under incomplete information On this site (opens in a new tab)
[2026-01-20]

Principles: Professional organizations have commented extensively on the use and interpretation of P-values and confidence intervals (full citations in the main review): “Scientific conclusions … should not be based only on whether a p-value passes a specific threshold. … Proper inference requires full reporting and transparency.” “Distinguish between clinical and statistical significance.” Appeals to nonoverlapping CIs “should not be used for formal significance testing unless the data analyst is aware of its deficiencies and unless the information needed to carry out a more appropriate procedure is unavailable.”

Observations: The reports in this review consistently state the intended level for statistical significance (usually 0.05) and coverage probability for confidence intervals (usually 95%). The report text usually gives information from which a reader can determine how inferential procedures were applied. One table contains cell values marked “NS” with no further quantitative content. Another reports P-values as either “<0.05” or as a number greater than 0.05 formatted to 4 decimal places. When the slope of a line is not statistically significantly different from 0, It is often not appropriate to interpret no change over time. One report incorrectly describes the Kruskal-Wallis test as comparing medians. At least 3 reports appeal to nonoverlapping CIs. Several reports include dozens or hundreds of simultaneous confidence intervals or significance tests.

Recommendations: Avoid emphasizing binary conclusions. It is acceptable to report both a CI and a P-value. Don’t conclude absence of an effect when it is not “significant” (e.g., “stable” for slope not significantly different from 0), unless the conclusion also appeals to adequate statistical power or other justification. Mention the CI construction procedure or testing procedure by the correct, unambiguous name on the first occurrence. State the confidence level. A P-value should typically be reported to 2 significant digits unless its value is less than 0.001; avoid “P < 0.05” or that a result was “nonsignificant”. If a result is described as statistically (non)significant, then state the type 1 error. For example, “X has a P-value of 0.030, which is statistically significantly different from 0 at the 0.05 level.” Never describe a test result whose P-value slightly exceeds the type 1 value as a “trend” or “approaching significance”. A CI or test for a comparison should be applied directly to the comparative quantity itself (e.g., difference or ratio). Appeals to nonoverlapping CIs should be reserved for exceedingly rare cases where such appeals are the only option. When a report includes a large number of significance test results or CIs, address the potential for spurious significance.

Next topic: Tables (§5.6)

Data in the MMWR