Home / Manual / GC-torture testing — surfacing latent SEXP-protection bugs

GC-torture testing — surfacing latent SEXP-protection bugs

R provides gctorture(TRUE) and gctorture2(step, wait, inhibit_release) to make every allocation trigger a full GC. Any SEXP that's reachable through Rust state but not rooted in R's protect mechanism gets collected on the next allocation, surfacing use-after-free that would otherwise hide for thousands of test runs (or only manifest under stricter allocators like glibc R 4.6 release on CI).

5 min read 911 words

R provides gctorture(TRUE) and gctorture2(step, wait, inhibit_release) to make every allocation trigger a full GC. Any SEXP that’s reachable through Rust state but not rooted in R’s protect mechanism gets collected on the next allocation, surfacing use-after-free that would otherwise hide for thousands of test runs (or only manifest under stricter allocators like glibc R 4.6 release on CI).

This page documents the harness pattern that works in this codebase, common pitfalls, and a recipe for running it across the full rpkg/ test suite.

🔗When to use

  • A new unsafe block reads or writes an SEXP cached in Rust state.
  • A new entry-point allocates SEXPs that linger in Vec<SEXP> / Box<dyn Any> / similar containers across .Call boundaries, ALTREP callbacks, or background threads.
  • CI fails with malloc(): unsorted double linked list corrupted, *** caught segfault ***, or address 0x... 'invalid permissions', particularly on Linux R release but not on devel/oldrel — that’s the strict-allocator signature for a UAF that other runtimes tolerate.
  • Before merging anything that touches serde/columnar.rs, ALTREP code, ExternalPtr storage, or the worker-thread serializer.

🔗The two pitfalls

🔗1. Don’t enable gctorture before library(miniextendr)

gctorture(TRUE) is so aggressive that ordinary package-load paths (initMethodDispatch, S4 method registration, loadNamespace hooks) hit unprotected SEXPs in unrelated CRAN packages and crash before your tests start. The crash signature is the giveaway — traceback ends in library / loadNamespace / runHook(".onLoad", ...).

Always load the package first, then flip gctorture on:

library(miniextendr)
library(testthat)
gctorture(TRUE)
# ... call your code ...
gctorture(FALSE)

🔗2. test_dir loads more packages

testthat::test_dir may attach extra packages (rlang, withr, dplyr, …) the first time it runs. Each .onLoad hook is a gctorture-amplified hazard you don’t want to debug. For targeted testing, call the fixture directly via get(name, envir = getNamespace("miniextendr"))() rather than going through test_dir. Keep test_dir for the final whole-suite sweep, after you’ve loaded everything you need.

🔗Recipe — per-function sweep

The gctorture/gc_stress fixtures are deliberately unexported (noexport — see audit A7) so they don’t leak into the public API. Enumerate and call them through the package namespace, not the attached-exports search path (ls("package:miniextendr") only sees exports and silently returns zero matches for these fixtures):

library(miniextendr)
gctorture(TRUE)

ns <- getNamespace("miniextendr")
funs <- ls(ns, pattern = "^test_columnar_")  # or any prefix, e.g. "^gc_stress_"
ok <- 0L
fail <- character(0)
for (f in funs) {
  res <- tryCatch({ get(f, envir = ns)(); "ok" }, error = function(e) conditionMessage(e))
  if (identical(res, "ok")) ok <- ok + 1L
  else fail <- c(fail, sprintf("%s: %s", f, res))
}
gctorture(FALSE)
cat(sprintf("%d/%d ok\n", ok, length(funs)))
if (length(fail)) cat(fail, sep = "\n")

This is the cheapest mode: ~1 minute per simple function under gctorture(TRUE). Catches every Rf_allocVector → unprotected-SEXP-read sequence inside the call.

Crash interpretation:

  • exit=0 with truncated output — the script SIGSEGV’d silently. The last printed function header is the one that crashed. Re-run that single function in isolation to confirm.
  • Explicit *** caught segfault *** with traceback — R’s signal handler caught it. Read the traceback for the immediate cause; the root cause is usually one frame up (the allocation that triggered GC, not the one that segfaulted reading freed memory).
  • Rscript exits with backtrace and address 0x... low — wild pointer from a freed SEXP, often from Rf_eval or R_NilValue-adjacent reads.

🔗Recipe — full rpkg/ sweep (slow)

For the full suite, use gctorture2(step = N) with N > 1 to keep the run tractable. step = 100 is ~100× slower than baseline (vs. ~1000× for gctorture(TRUE)) and still surfaces the vast majority of UAF bugs.

library(miniextendr)
library(testthat)
setTimeLimit(Inf, Inf, transient = FALSE)
options(timeout = Inf)

gctorture2(step = 100, wait = 0, inhibit_release = FALSE)
res <- test_dir(
  "rpkg/tests/testthat",
  reporter = ProgressReporter,
  stop_on_failure = FALSE
)
gctorture2(step = 0)  # disable
print(res)

Run it as Rscript /path/to/script.R 2>&1 > /tmp/gctorture-full.log, in the background, and watch the log. Expect hours, not minutes — the full suite at step = 100 typically takes 30–90 minutes locally and substantially longer in CI.

For a tighter feedback loop while bisecting:

  • step = 10 covers most bugs in <30s/test.
  • step = 1 matches gctorture(TRUE) and only makes sense for single-function bisects.

🔗What gctorture cannot find

  • Stack-discipline violations on the protect stack — Rf_protect / Rf_unprotect imbalances surface as panics from R_PPStackTop overflow, not gctorture firings.
  • Cross-.Call lifetime bugs that depend on R-side caching (e.g., an ExternalPtr that survives one call only because R’s namespace cached the wrapping closure).
  • Thread-affine bugs — gctorture is single-threaded and won’t reproduce data races across the worker thread.
  • ALTREP materialization order issues — gctorture stresses GC, not lazy-eval ordering.

🔗Project-specific reference

  • Root cause example for this technique: PR #424 (issue #307) — ColumnBuffer::Generic held raw Vec<Option<SEXP>> across allocations. The fix was a ProtectScope opened in from_rows and threaded through ColumnFiller. CI surfaced the bug only on Linux R 4.6 release; gctorture surfaced it deterministically in a 30-second local run.
  • Helpers in miniextendr-api/src/gc_protect.rs — OwnedProtect, ProtectScope, ProtectIndex, ReprotectSlot, Root. See module docs for trade-offs.
  • Pool variant for any-order release: miniextendr-api/src/protect_pool.rs (ProtectPool — VECSXP-backed, generational keys).

🔗Adding to CI

A nightly gctorture nightly job invokes the full-suite recipe at step = 100 on the Linux R-release runner, catching this class of bug before it reaches a release. It is wired up at .github/workflows/gctorture-nightly.yml (#1024):

  • Scheduled at 04:00 UTC daily, plus workflow_dispatch for on-demand runs. It is deliberately not on the pull_request / push matrix — the sweep takes 30–90 minutes, too expensive for a PR gate.
  • timeout-minutes: 120 — ~2× the r-check-linux job’s 60-minute budget, since the step=100 sweep is ~100× slower than a baseline test run. The harness also calls setTimeLimit(Inf) / options(timeout = Inf) so R’s own wall-clock limits never abort the (deliberately slow) run.
  • The torture logic lives in just gctorture-full (→ scripts/gctorture-full-sweep.R), so it is testable locally: just gctorture-full runs the same sweep, and just gctorture-full step=10 is the faster bisect mode. On failure the workflow surfaces the offending test file(s) in the GitHub job summary.
Back to Manual