# Statistical validation report

## The validation and stress-testing package (from 1.2.0-beta.10)

Counts quoted below as "at 1.2.0-beta.10" are a historical snapshot. The counts
for the deployed release are generated from `validation/results/summary.json`
at release time and published at https://plotvera.app/validation/; the release
they belong to is named in https://plotvera.app/evidence/release.json.

Everything below this section describes the evidence built up before October
2026 and is still true. This section describes the package that now runs on
every release and every night, and that produces the public page
**https://plotvera.app/validation/** for each release.

**Rule: no expected value is typed in by hand.** Every expected number is
either computed by an R script in this repository (`validation/r/`, with the
R and package versions written into each output file) or is a certified value
downloaded from its publisher (NIST StRD, with the URL of every file in
`validation/reference/nist/SOURCES.txt`).

### 1. Numerical validation (`src/validation/numerical.*.test.ts`)

- **Catalogue** (`src/validation/catalogue.ts`): all 86 methods Plotvera
  offers: the 46 analyses in the analysis menu, the 20 built-in curve models plus
  the straight line, and 19 options and sub-methods (post-tests, intervals,
  corrections). A test fails if a new analysis or model appears without an
  entry, so nothing can ship uncatalogued. Status at 1.2.0-beta.10: 74 covered,
  9 partial, 3 not covered, each gap with its reason on the page.
- **Reference datasets**: classic datasets as shipped with R (Student's sleep
  data, PlantGrowth, ToothGrowth, warpbreaks, InsectSprays, chickwts, CO2,
  airquality, mtcars, women, anscombe, Puromycin, Indometh, DNase,
  survival::lung, aml, ovarian, veteran and more) written out as the exact CSV
  inputs in `validation/reference/data/`; and NIST StRD certified datasets for
  univariate statistics (PiDigits, Lottery, Lew, Mavro, Michelso, NumAcc1-4),
  one-way ANOVA (SiRstv, SmLs01-09, AtmWtAg), linear regression (Norris,
  Pontius, NoInt1-2, Filip, Longley, Wampler1-5) and nonlinear regression (the
  27 problems whose model Plotvera can express, each from both NIST starting
  points and from Plotvera's own automatic start).
- **Expected values**: R 4.3.3 (`stats`, `survival`, `MASS`, `drc`,
  `multcomp`, `mvtnorm`, `lme4`/`lmerTest`, `emmeans`, `car`), regenerated
  with `validation/r/run_all.sh` (about 80 s), or the NIST certified values.
- **Tolerances**: typically 1e-9 relative for closed-form results (t, ANOVA,
  regression, survival), 1e-6 for iterative fits against R `nls`, 1e-6 on NIST
  certified parameters, looser only where documented in the case itself (for
  example Dunnett P against R's Monte-Carlo `mvtnorm` integral at 5e-5
  absolute; P values limited by jStat's ~2e-7 relative accuracy).
- **Results at 1.2.0-beta.10**: 247 numerical cases; 208 pass. The largest
  relative errors among them are 1.2e-4 on NIST SmLs07-09 (data with 13
  significant digits at 0.1 resolution, which double precision cannot hold;
  Plotvera matches exact arithmetic on the stored values to 1e-12, and R's
  `lm` does worse on SmLs09), 5e-5 for Dunnett (R's reference is itself a
  Monte-Carlo integral), 2e-5 for NIST nonlinear fits, and 1e-8 or better
  for t tests, ANOVA, regression and survival. 39 NIST curve-fitting
  cases do not pass and are listed as **known limitations** with their reasons
  (below; 8 of them are problems Plotvera does not attempt, because they exceed
its in-browser work budget or use a model it cannot express). The suite
asserts that list exactly: a known limitation that starts
  passing, or any new failure, fails the test run and so blocks a release.
- **Methodological differences** are compared against the matching R option,
  not hidden: log-scale CIs for positive-constrained curve parameters, the
  sample odds ratio ad/bc (R reports the conditional MLE), the first-crossing
  median survival rule, and the 1986 Bland-Altman limits.

**Known limitations (39 NIST cases).** From NIST's own starting values Plotvera
solves 15 of 25 nonlinear problems from Start 1 and 20 of 25 from Start 2;
from its automatic start, 9 of 25 (6 are refused by the custom-fit work budget,
7 report non-convergence). Where it does not solve a problem it says the fit
did not converge rather than report a wrong answer. Of the linear NIST
problems entered as custom equations, NoInt1 is withheld (negative centred R
squared for a no-intercept model), Wampler2-5 stop early or are withheld on
exact-fit or extremely ill-conditioned data, and Filip is refused by the work
budget. Nelson (two predictors) and Roszman1 (arctan) cannot be expressed in
the equation editor.

**Engine bugs this suite found** (fixed in statistics engine 1.12.0, see
CHANGELOG): the curve fitter could report "Converged: Yes" on a poor fit; the
two-way repeated-measures Huynh-Feldt epsilon used N - 1 instead of N - g; the
normality battery refused every n = 3 sample; correlation CIs used 1.96
instead of the exact quantile.

**Not yet independently referenced**: the KS-Lilliefors P value and the
Baptista-Pike odds-ratio interval (their R packages are not installed here),
Grubbs (checked against the published formulas only), frequency-distribution
bin edges (a Plotvera display rule), repeated-measures designs with repeated
rows or both factors repeated, the AUC wizard options, paired and
nonparametric multiple t tests, standard-curve weighting, the mixed model's
per-time contrasts and the replicate row-means option. GraphPad's own worked
examples are not used.

### 2. Reproducibility (`src/validation/reproducibility.test.ts`)

Same input and same version give the same numbers: each representative
analysis (t tests, ANOVA, regression, a 4PL curve fit, Kaplan-Meier and
log-rank, Cox) is run twice and on a deep copy and must give byte-identical
results; reordering rows where order is not part of the design must give the
same result within 1e-12 (curve fits: 1e-6, because the optimiser's path
depends on row order at about 3e-8). See `docs/REPRODUCIBILITY.md`.

### 3. Import fidelity (`src/validation/importFidelity.test.ts`)

Every fixture (Prism `.prism` and `.pzfx`, PowerPoint with embedded Prism
projects, Excel, CSV, JSON, `.plotvera`, SPSS / Stata / SAS, lab-instrument
exports: 61 variants) goes import -> export -> re-import by four routes
(`.plotvera`, `.pzfx`, `.prism`, CSV): 244 round trips. Every cell is compared
exactly (numbers with `Object.is`, blanks must stay blank, never 0), with
table names, types, titles and replicate structure. Five data-loss bugs found
this way are fixed (CHANGELOG). Limits by design: CSV cannot carry table type
or replicate layout; Prism fixes some of its own titles; `.pzfx` parsing in
Node uses `@xmldom/xmldom`, and real-browser parsing is covered by
`test/import_fidelity_browser.mjs`.

### 4. Rendering regression (`src/validation/rendering.test.ts`)

126 approved SVG pictures: all 23 chart types and the key options (log axes,
every error-bar kind, legends, palettes, dark backgrounds, tick rotation,
split axes, fitted curves, survival options). Each rendering is normalised
(ids renumbered, coordinates rounded to 0.01 px, the version comment removed)
and compared with its golden; any element that differs fails the test, and
the gate is itself tested on deliberate moves, label, colour and element
changes. Goldens change only through
`node scripts/approve-goldens.mjs "<reason>"`, which prints what changed and
records the date, reason and cases in `test/golden/render/APPROVALS.md`.

### 5. Stress and fuzz testing

- **Property-based tests** (`src/validation/property.*.test.ts`, fast-check,
  fixed seeds): 77 properties over every parser (CSV/TSV, clipboard, Excel,
  JSON, `.plotvera`, Prism, PowerPoint, SPSS/Stata/SAS, share links) and every
  statistic (permutation, group-swap and unit invariance; monotonicity;
  P in [0, 1]; blanks and NaN never counted as 0; 1e300, 1e-300, subnormals,
  ties, n = 1 and 2, constant data, n = 100,000). Eleven bugs found; ten fixed
  with regression tests (CHANGELOG). The nightly run uses 10 times as many
  generated cases.
- **Large data** (`test/stress_large_data_browser.mjs`): 100,000 rows and 500
  columns are beyond the documented project limits (20,000 rows, 256 columns,
  250,000 cells; `docs/PROJECT_SCHEMA.md`) and must be refused within 3 s with
  a clear message; the largest accepted tables (20,000 x 12, 900 x 256) and a
  project with 20 tables, 200 graphs and 50 layouts must import, draw, analyse,
  save and reopen within time budgets, with the JavaScript heap measured.
- **Long session** (`test/stress_soak_browser.mjs`): scripted realistic use for
  2 hours nightly (20 minutes when first measured: heap 14.7 -> 17.2 MB, no
  growth trend, no errors).
- **Two tabs, autosave** (`test/stress_concurrency_browser.mjs`), **offline and
  flaky network** (`test/stress_offline_browser.mjs`), and **Chromium, Firefox
  and WebKit plus phone and tablet widths** (`test/browser_matrix.mjs`).

### Running it

- `pnpm validate` runs the validation suites and prints the summary per suite.
- `npx vitest run` runs everything (the validation package is part of it).
- `validation/r/run_all.sh` regenerates every R reference value.
- Releases (`scripts/release.sh`) and the nightly run publish the results;
  the nightly run also re-runs the R scripts and warns if any reference value
  moved. See `docs/RELEASING.md`.


## Scope and status

Plotvera’s automated validation suite is an engineering control for the
statistics engine. It is not a certification, a clinical-device validation, or
proof that an analysis is appropriate for a particular experiment.

Current automated gates:

- full Vitest suite (`pnpm test`);
- golden reference suite (`pnpm golden`);
- production dependency audit plus executable backport regressions
  (`pnpm security:audit` and `src/lib/dependencySecurity.test.ts`). The audit
  permits only the two exact `image-size@2.0.2` advisory paths covered by the
  locked local backport; any other high-severity advisory fails;
- ESLint (`pnpm lint`);
- browser-app and edge-API TypeScript checks plus production build (`pnpm build`);
- a regression check that no test module can be discovered as a Vercel API
  function;
- standalone SVG and PowerPoint artifact-structure tests plus publication
  PNG/PDF allocation-boundary tests within the full Vitest suite;
- headless-Chromium release smoke (`pnpm smoke`) against the built app;
- continuous testing on the VPS on every push to `main` and nightly
  (GitHub Actions is not available on this account; see `docs/RELEASING.md`).

The browser smoke checks a recommended desktop dose-response analysis,
convergence and provenance, accessible graph semantics and minimum rendered
size, and Assistant reopen behaviour. It also verifies that a quoted-delimiter
CSV preview can be cancelled without mutating the project and then confirmed
with quoted fields intact. Cloud sharing must remain disabled until explicit
consent, while local sharing must create a fragment link without an API
request. Project navigation and graph layout must work without horizontal
overflow at a 390 px mobile viewport. These are semantic assertions, not
screenshot comparisons.

The export tests cover standalone SVG XML metadata, namespace, and Unicode
handling; exact normal/boundary/over-limit PNG/PDF geometry; fail-closed canvas
and page-allocation limits; and an unzipped PowerPoint package with an editable
embedded SVG, bounded data slide, and no hidden project link. The browser smoke
also parses the downloaded PNG resolution chunks and vector PDF page geometry,
embedded font/Unicode maps, and absence of raster image objects.

The statistical golden corpus covers representative values and edge cases across
distributions, descriptive statistics, t tests, nonparametric methods,
correlation, one-/two-way and repeated-measures analyses, multiple comparisons,
regression, logistic regression, survival, diagnostics, multivariable methods,
transformations, and nonlinear-fit failure states. Project/import boundaries and
untrusted-file handling are covered separately by the full automated suite.
Reference comments identify exact identities, hand calculations, canonical
datasets, or an independent implementation where used.

The normality corpus includes an alpha-crossing KS-Lilliefors fixture against
the published Dallal-Wilkinson upper-tail equation and a separate R
`nortest::lillie.test` fixture for the Stephens modified-statistic branch used
when that equation is outside its stated P < 0.1 domain. Direct, combined-battery,
dispatcher, and provenance assertions prevent either reference domain from
being replaced by an unlabelled approximation.

The versioned machine-readable registry at `validation-manifest.json` maps all
advertised analysis keys to executable evidence, an explicit support status,
an input/minimum-sample/missing-data/design-unit profile, shared persistence and
provenance fixtures, and published limitations statements where evidence is partial.
All built-in nonlinear model keys are registered separately as `covered`,
`partial`, or `unvalidated`. A `partial` or `unvalidated` entry is intentionally
visible in the product but is not a claim that the full evidence standard below
has been met. The shared infrastructure fixtures are not treated as per-method
proof.

The beta.3 candidate retains machine-checked lifecycle dossiers for
selected methods. The one-sample t and Pearson dossiers register executable
input-contract, independent-reference, boundary/failure, recomputation,
project/share, and provenance evidence. That is six executable categories, not
the complete seven-part promotion standard: the exact limitations still require
product/statistics-owner review. Both methods therefore remain `partial`, and a
passing dossier test must not be presented as owner approval or independent
external validation. The new one-sample Wilcoxon path likewise remains partial
while its independent reference breadth, lifecycle evidence, and owner review
are incomplete.

Engine 1.4.0 adds a NIST Longley reference plus translation/unit-invariance and
near-collinearity refusal fixtures for multiple linear regression. Hill-model
coverage now includes exact and noisy parameter recovery, finite uncertainty,
positive-parameter provenance, domain/identifiability/stationary-failure paths,
and response-unit invariance from 10^-6 through 10^6. This invariance claim is
historically Hill-specific in the 1.4.0 evidence; Engine 1.7.0 broadens the
executable contract to all registered built-ins. Kaplan-Meier
chart evidence checks Greenwood complementary-log-log confidence limits at
ordinary, all-censored, and terminal-zero boundaries, verifies that exported
SVG contains its number-at-risk table, and exercises compact-canvas omission
disclosures. These additions strengthen the named paths without promoting the
parent analyses beyond their manifest status or substituting for owner/external
review.

Engine 1.5.0 separates the later inferential contract from 1.4.0: Shapiro-Wilk
uses the full-precision R/Royston coefficient construction and correct
finite-sample P-value branch; corrected Greenhouse-Geisser probabilities drive
repeated-measures decisions and provenance; unsupported fully repeated
interaction inference is withheld; Dunn's independent-group follow-up accepts
unequal sample sizes; and graph-only independent parametric comparisons use
Welch's test deterministically. Exact alpha-crossing, coefficient-sensitive and
unequal-group fixtures prevent these changes from sharing an older engine
identifier.

Engine 1.6.0 adds the previously withheld fully repeated two-way interaction
inference. For an interaction with multiple contrast degrees of freedom, each
subject's cell profile is projected into an orthonormal A x B contrast space;
the covariance of those scores supplies Greenhouse-Geisser epsilon, and the
corrected effect/error degrees of freedom supply the primary interaction P
value. An independent 3 by 2 repeated-measures reference pins F(2, 8) =
5.505618, uncorrected P = 0.031356, GG epsilon = 0.559314 and corrected P =
0.070452, so a sphericity-assumed false-positive decision cannot reappear.
The uncorrected and Huynh-Feldt values remain labelled sensitivity outputs.

Engine 1.7.0 adds three scale/decision integrity families. First, ordinary and
repeated-measures one-way plus Type-III two-way ANOVA centre finite responses
before their sums-of-squares, covariance/sphericity, or full/reduced OLS
calculations; exact alpha-boundary fixtures remain byte-identical after adding
a common 10^15 response offset. Tukey and Dunnett follow-up likewise use
centred means and pooled variance, preserving a pinned adjusted P = 0.051731
decision. Ordinary linear regression likewise centres
predictor/response covariance and residual evaluation; its slope inference is
identical after a common 10^15 X origin, Y origin, or both, while the reported
intercept retains its source-axis meaning. Balanced nested t and nested ANOVA
use stable finite-midrange subject summaries/decompositions and retain their
near-alpha inference after a common 10^15 response origin. One-sample contrasts
and unpaired t calculations are evaluated in centred coordinates; the exact
Analyze/graph Welch fixture remains significant at P = 0.0487 after a common
10^15 response origin. Pearson correlation likewise removes common X/Y origins
before normalisation, preserving the exact P = 0.00997 and two-star tier after
10^15 offsets. The derived-sheet z-score contract in
`src/stats/descriptives2.transforms.test.ts` pins byte-identical serialization
for an exact seven-value column before and after a common 10^15 response origin.
Second, all 17 built-in nonlinear models declare every
response-dimensional parameter and
are exercised on exact and noisy responses at 10^-6, 1 and 10^6 scales. The
tests pin normalized parameters, standard errors, confidence intervals,
covariance, predictions, SSR and Sy.x. Named exponential families additionally
use positive rate/directional coordinates, while Michaelis-Menten Km, Gaussian
SD and sine wavelength use positive coordinates matching their definitions.
Michaelis-Menten and linear-dose log-logistic families refuse negative
concentrations while permitting a zero-dose control, and inverse interpolation
withholds negative concentration solutions. Gaussian, polynomial and sine
models withhold arbitrary roots for multivalued inverse problems, while the
positive-amplitude/rate two-phase decay retains its unique numeric inverse.
Exact opposite-direction curves are
refused, R² < 0 candidates are non-citable, and chart overlays share the same
eligibility decision with an accessible model/status disclosure. User-defined
equations remain deliberately outside automatic scaling because their parameter
dimensions cannot be inferred safely. Third, offline and online normality
guidance routes through the supported Shapiro-Wilk/battery evidence, treats
departures as diagnostic sensitivity information, and does not replace the
researcher-declared parametric family with a rank test automatically.

Engine 1.8.0 extends the numeric-origin contract to feature-wise testing and
logistic regression. A pinned Welch feature row remains P = 0.0487 with one
planned-family discovery before and after a common 10^15 response offset.
Simple logistic slope estimates, standard errors, Wald statistics and P values
remain identical after a 10^15 predictor origin or unit rescale because fitting
uses centred/sample-SD coordinates and returns coefficients to source units.
The suite separately proves that a numerically zero slope withholds only its
undefined X-at-50% inverse, while separation cannot emit a citable significance
claim. Multiple logistic fitting uses the same stable predictor coordinates and
diagnostic-only status contract. Bland–Altman, Grubbs and MAD fixtures exercise
finite same-sign values near 10^308 whose naive intermediate sums overflow;
the finite results now survive while genuinely nonfinite derived quantities
still fail closed. Their public numeric arrays and the portable project codec
also reject sparse JavaScript slots before a hole can disappear from a result
or serialize as `null`. Finally, a perfect reverse ROC fixture pins AUC = 0 as
inverse discrimination and prevents the declared larger-is-positive cutoff
from being presented as an inverse-direction optimum. Correlation-matrix
results and provenance identify their single listwise complete-row set and
their P values as unadjusted exploratory outputs.

Engine 1.9.0 extends the numeric-origin contract to the remaining standardised
multivariate workflows. Correlation matrices, correlation-matrix PCA, and
k-means now anchor each complete variable to an observed source value when its
finite differences are representable, scale those differences into bounded
coordinates, and form its mean and sample SD there. If a finite difference
itself overflows, an extreme-range fallback first scales the source values from
zero. These are algebraically equivalent affine changes of coordinates, but
avoid losing low-order information by summing large raw origins. The exact
five-row matrix fixture `A = [0,-11,19,2,10]`, `B = [13,16,5,15,11]` remains
`r = -0.923`, `P = 0.0254` after adding `8 × 10^15` to every cell; the previous
path returned `r = -0.865`, `P = 0.0583` and reversed the displayed alpha
decision. A separate six-row, two-variable fixture retains the identical
`{1,6}|{2,3,4,5}` k-means partition and within-cluster `SSE = 3.973` after the
same exact translation, while PCA eigenvalues, loadings and scores are
byte-identical. The fallback preserves the exact correlation/PCA structure of
finite opposite-extreme cells instead of failing on an avoidable intermediate
overflow. Constant columns and genuinely
non-finite derived results still fail closed; k-means standardisation or
source-centroid overflow now sets the explicit unavailable flag. Provenance
names the common-origin-stable coordinates, and the manifest records each
identity without promoting these partial methods or claiming broad external
validation.

Engine 1.10.0 corrects the KS-Lilliefors reference calculation. Within the
Dallal-Wilkinson upper-tail equation's stated domain, the implementation now
evaluates the published expression directly; when its preliminary P value is
above 0.1, the Stephens modified-statistic polynomial used by R `nortest` is
used instead. The five-point alpha-crossing fixture
`[0.01311955,-0.12719983,0.58471993,-1.45560984,0.04402]` has
`D = 0.3320397`, `P = 0.0749618` and Pass, where the prior unrelated rational
expression returned `P = 0.00398` and Fail. A separate `n = 1000` fixture pins
the exact original-n/original-D Stephens fallback used by `nortest`, while
invalid direct inputs are explicitly unavailable rather than displayed as a
normality failure. The same engine contract forms KS-Lilliefors,
Anderson-Darling, skewness and kurtosis in bounded affine coordinates before
returning the displayed mean and SD to source units. Exact `8 × 10^15`
translation fixtures preserve a formerly reversed KS decision (`D = 0.3149`,
`P = 0.0637`, Pass) and a formerly reversed Anderson-Darling decision
(`A*² = 0.7485`, `P = 0.0514`, Pass), while genuinely unrepresentable
source-unit spread remains explicitly unavailable.

Engine 1.11.0 is the next statistics candidate. Iterative Grubbs inference now
forms its mean, SD and standardised residuals in bounded affine coordinates
while retaining source-unit summaries and row identity. For the exact fixture
`[-31,-45,-33]`, `G = 1.1446` remains below the two-sided alpha-0.05 critical
value `1.1543` after adding exactly `8 × 10^15` to every safe-integer value;
the former raw-origin path changed `G` to `1.291` and falsely excluded source
row 2. Direct and dispatcher assertions pin the statistic, SD, first-pass
decision and selected row. The same candidate makes a Friedman design with
every within-subject row completely tied explicitly unavailable: its tie
correction is exactly zero, so neither the omnibus path nor a Dunn follow-up
may replace the undefined rank statistic with `P = 1`, `W = 0`, or an `ns`
decision. These additions strengthen the named partial methods without
promoting their support status.

## Evidence standard for a supported analysis

Before an analysis may be labelled supported, its manifest must include:

1. input contract, minimum sample size, missing-data policy, and design units;
2. equation/algorithm and every user-visible option;
3. representative independent reference values for estimates, standard errors,
   degrees of freedom, test statistics, P values, confidence intervals, and
   effect sizes where applicable;
4. ties, zero variance, singularity, nonconvergence, invalid-domain, and other
   relevant failure fixtures;
5. deterministic recomputation and save/open/share round-trip coverage;
6. provenance text that names the method actually used;
7. an owner-reviewed limitations entry.

## Known validation gaps

- The machine-readable manifest now registers every advertised analysis and
  every built-in nonlinear model, but it does not yet tie every individual
  output to an independent R/SciPy/statsmodels fixture. Every built-in has
  exact/noisy generating-model and response-scale evidence; `covered` still
  applies only to each entry's named expectations and is not external review or
  full support of the parent curve-fit analysis.
- Structural artifact tests do not replace pixel/vector goldens or visual checks
  in external SVG, PDF, PowerPoint, and PNG viewers for every chart type,
  long-label case, scale, and output format.
- The core semantic browser smoke is automated, but full keyboard and
  screen-reader testing, contrast and zoom checks, visual regression, and the
  supported browser/OS matrix remain manual release gates.
- Cross-version project migration has limited historical fixtures.
- Some advanced methods are implemented but remain “partial” in the capability
  matrix because option depth or diagnostics are incomplete.
- No independent external statistical review has been published.
- Product/statistics-owner review of the exact limitations remains pending for
  the candidate lifecycle dossiers; no analysis is promoted solely from
  executable evidence.

## Release rule

Any change that can alter a numeric result requires an `ENGINE_VERSION` bump,
an explicit changelog entry, and updated golden evidence. UI/import/chart-only
changes bump the app version but not the engine. A failing or unexplained
reference comparison blocks release; it must never be waived merely to deploy.

The exact passing test count and build SHA must be recorded in the release notes
from a clean release candidate, not hard-coded here before that run occurs.