GC-torture testing — surfacing latent SEXP-protection bugs

R provides gctorture(TRUE) and gctorture2(step, wait, inhibit_release) to make every allocation trigger a full GC. Any SEXP that’s reachable through Rust state but not rooted in R’s protect mechanism gets collected on the next allocation, surfacing use-after-free that would otherwise hide for thousands of test runs (or only manifest under stricter allocators like glibc R 4.6 release on CI).

This page documents the harness pattern that works in this codebase, common pitfalls, and a recipe for running it across the full rpkg/ test suite.

🔗When to use

• A new unsafe block reads or writes an SEXP cached in Rust state.
• A new entry-point allocates SEXPs that linger in Vec<SEXP> / Box<dyn Any> / similar containers across .Call boundaries, ALTREP callbacks, or background threads.
• CI fails with malloc(): unsorted double linked list corrupted, *** caught segfault ***, or address 0x... 'invalid permissions', particularly on Linux R release but not on devel/oldrel — that’s the strict-allocator signature for a UAF that other runtimes tolerate.
• Before merging anything that touches serde/columnar.rs, ALTREP code, ExternalPtr storage, or the worker-thread serializer.

🔗The two pitfalls

🔗1. Don’t enable gctorture before library(miniextendr)

gctorture(TRUE) is so aggressive that ordinary package-load paths (initMethodDispatch, S4 method registration, loadNamespace hooks) hit unprotected SEXPs in unrelated CRAN packages and crash before your tests start. The crash signature is the giveaway — traceback ends in library / loadNamespace / runHook(".onLoad", ...).

Always load the package first, then flip gctorture on:

library(miniextendr)
library(testthat)
gctorture(TRUE)
# ... call your code ...
gctorture(FALSE)

🔗2. test_dir loads more packages

testthat::test_dir may attach extra packages (rlang, withr, dplyr, …) the first time it runs. Each .onLoad hook is a gctorture-amplified hazard you don’t want to debug. For targeted testing, call the fixture directly via get(name, envir = getNamespace("miniextendr"))() rather than going through test_dir. Keep test_dir for the final whole-suite sweep, after you’ve loaded everything you need.

🔗Recipe — per-function sweep

The gctorture/gc_stress fixtures are deliberately unexported (noexport — see audit A7) so they don’t leak into the public API. Enumerate and call them through the package namespace, not the attached-exports search path (ls("package:miniextendr") only sees exports and silently returns zero matches for these fixtures):

library(miniextendr)
gctorture(TRUE)

ns <- getNamespace("miniextendr")
funs <- ls(ns, pattern = "^test_columnar_")  # or any prefix, e.g. "^gc_stress_"
ok <- 0L
fail <- character(0)
for (f in funs) {
  res <- tryCatch({ get(f, envir = ns)(); "ok" }, error = function(e) conditionMessage(e))
  if (identical(res, "ok")) ok <- ok + 1L
  else fail <- c(fail, sprintf("%s: %s", f, res))
}
gctorture(FALSE)
cat(sprintf("%d/%d ok\n", ok, length(funs)))
if (length(fail)) cat(fail, sep = "\n")

This is the cheapest mode: ~1 minute per simple function under gctorture(TRUE). Catches every Rf_allocVector → unprotected-SEXP-read sequence inside the call.

Crash interpretation:

• exit=0 with truncated output — the script SIGSEGV’d silently. The last printed function header is the one that crashed. Re-run that single function in isolation to confirm.
• Explicit *** caught segfault *** with traceback — R’s signal handler caught it. Read the traceback for the immediate cause; the root cause is usually one frame up (the allocation that triggered GC, not the one that segfaulted reading freed memory).
• Rscript exits with backtrace and address 0x... low — wild pointer from a freed SEXP, often from Rf_eval or R_NilValue-adjacent reads.

🔗Recipe — full rpkg/ sweep (slow)

For the full suite, use gctorture2(step = N) with N > 1 to keep the run tractable. step = 100 is ~100× slower than baseline (vs. ~1000× for gctorture(TRUE)) and still surfaces the vast majority of UAF bugs.

library(miniextendr)
library(testthat)
setTimeLimit(Inf, Inf, transient = FALSE)
options(timeout = Inf)
# test_dir() does not assume "not on CRAN" the way test_local() does, and the
# gctorture blocks are opt-in (see "How CI runs the gctorture tests" below).
Sys.setenv(NOT_CRAN = "true", MINIEXTENDR_STRESS = "true")

gctorture2(step = 100, wait = 0, inhibit_release = FALSE)
# `package` + `load_package = "installed"` run the tests in a child of the
# namespace, so tests that call unexported functions find them.
res <- test_dir(
  "rpkg/tests/testthat",
  package = "miniextendr",
  load_package = "installed",
  reporter = ProgressReporter,
  stop_on_failure = FALSE
)
gctorture2(step = 0)  # disable
print(res)

Run it as Rscript /path/to/script.R 2>&1 > /tmp/gctorture-full.log, in the background, and watch the log. Expect hours, not minutes — the full suite at step = 100 typically takes 30–90 minutes locally and substantially longer in CI.

For a tighter feedback loop while bisecting:

• step = 10 covers most bugs in <30s/test.
• step = 1 matches gctorture(TRUE) and only makes sense for single-function bisects.

🔗What gctorture cannot find

• Stack-discipline violations on the protect stack — Rf_protect / Rf_unprotect imbalances surface as panics from R_PPStackTop overflow, not gctorture firings.
• Cross-.Call lifetime bugs that depend on R-side caching (e.g., an ExternalPtr that survives one call only because R’s namespace cached the wrapping closure).
• Thread-affine bugs — gctorture is single-threaded and won’t reproduce data races across the worker thread.
• ALTREP materialization order issues — gctorture stresses GC, not lazy-eval ordering.

🔗Project-specific reference

• Root cause example for this technique: PR #424 (issue #307) — ColumnBuffer::Generic held raw Vec<Option<SEXP>> across allocations. The fix was a ProtectScope opened in from_rows and threaded through ColumnFiller. CI surfaced the bug only on Linux R 4.6 release; gctorture surfaced it deterministically in a 30-second local run.
• Helpers in miniextendr-api/src/gc_protect.rs — OwnedProtect, ProtectScope, ProtectIndex, ReprotectSlot, Root. See module docs for trade-offs.
• Pool variant for any-order release: miniextendr-api/src/protect_pool.rs (ProtectPool — VECSXP-backed, generational keys).

🔗How CI runs the gctorture tests (opt-in per PR)

The gctorture blocks in the testthat suite (mostly in test-gc-stress-fixtures.R, test-externalptr-self-root.R, test-iter-to-dataframe.R and test-dataframe-deserialize.R) are ~31 of the suite’s ~34 minutes, so they are opt-in: each block calls skip_gc_stress_if_disabled() from rpkg/tests/testthat/helper-gc-stress.R and runs only when MINIEXTENDR_STRESS is true. The cheap structure / value assertions in the same files always run. The plain suite (just devtools-test, the R CMD check legs, the CRAN-like check, the r-tests job and its heap-check rounds, the feature legs, the webR smoke) leaves the variable unset and skips the stress blocks.

Run them yourself when a change adds a path holding SEXPs across allocations:

just devtools-test-stress                     # the whole suite, stress blocks included
just devtools-test-stress gc-stress-fixtures  # FILTER is positional

or use the per-function harness above for a faster loop.

In CI, only the sharded r-stress-tests job and the nightly sweep set MINIEXTENDR_STRESS. r-stress-tests runs unconditionally on push-to-main, the weekly cron and workflow_dispatch; on a pull request it runs only when the PR carries the gc-stress label (the same opt-in shape as heap-check for the MALLOC_CHECK_ rounds). Label a PR that adds or changes a path holding SEXPs across allocations — Vec<SEXP>, sidecar fields, generic-list buffers, a new gc_stress_* fixture — and re-label after a force-push if the label was removed. An unlabeled PR skips the job, and a skipped job passes ci-success; a labeled PR whose shards fail blocks the merge. Two env vars, both handled by helper-gc-stress.R, orchestrate the split:

• MINIEXTENDR_STRESS — turns the torture blocks on. It is parsed with as.logical(), so it accepts true / TRUE / True / T; anything else, including 1, leaves them off. The helper also calls skip_on_cran() first: a half-hour suite is far beyond real CRAN’s check budget. Note skip_on_cran() alone can’t gate CI — r-lib’s setup actions export NOT_CRAN=true into every job.
• MINIEXTENDR_STRESS_SHARD=k/n — splits the dynamic no-arg fixture sweep in test-gc-stress-fixtures.R across the parallel r-stress-tests shards by fixture index. Unset locally, so just devtools-test-stress runs the whole sweep.

Main-branch coverage is unchanged by this layout — same fixtures, same iteration counts, run once per merge instead of five times. PR coverage is the author’s call via the label; the nightly sweep below is the backstop for anything that slipped through unlabeled.

🔗Adding to CI (nightly deep sweep)

A nightly gctorture nightly job invokes the full-suite recipe at step = 100 on the Linux R-release runner, catching this class of bug before it reaches a release. It is wired up at .github/workflows/gctorture-nightly.yml (#1024):

• Scheduled at 04:00 UTC daily, plus workflow_dispatch for on-demand runs. It is deliberately not on the pull_request / push matrix — the full sweep needs ~11 hours of runner time, far too expensive for a PR gate, so it fans out over a six-shard matrix (MINIEXTENDR_GCTORTURE_SHARD=k/n partitions the test files).
• timeout-minutes: 300 per shard (~2 hours of sweeping each, plus the build and uneven file weights). The harness also calls setTimeLimit(Inf) / options(timeout = Inf) so R’s own wall-clock limits never abort the (deliberately slow) run.
• The torture logic lives in just gctorture-full (→ scripts/gctorture-full-sweep.R), so it is testable locally: just gctorture-full runs the same sweep, and just gctorture-full 10 is the faster bisect mode (STEP is positional). On failure the workflow surfaces the offending test file(s) in the GitHub job summary.
• The recipe sets MINIEXTENDR_STRESS=true (the workflow sets it too), so the sweep includes the opt-in gctorture blocks, and NOT_CRAN=true, which test_dir() does not assume, so a local run covers the same skip_on_cran() tests as CI.