Home / Manual / ALTREP in miniextendr

ALTREP in miniextendr

ALTREP (Alternative Representations) is R's system for creating custom vector implementations. miniextendr provides a powerful, safe abstraction for creating ALTREP vectors from Rust.

37 min read 7203 words

ALTREP (Alternative Representations) is R’s system for creating custom vector implementations. miniextendr provides a powerful, safe abstraction for creating ALTREP vectors from Rust.

Additional Resources:

🔗What is ALTREP?

ALTREP allows you to create R vectors with custom internal representations. Instead of storing data in R’s native format, you can:

  • Compute elements on demand (lazy sequences)
  • Reference external data without copying (zero-copy views)
  • Use compact representations (constant vectors, arithmetic sequences)
  • Provide optimized operations (O(1) sum for arithmetic sequences)

🔗Quick Start

Here’s a minimal ALTREP example - a constant integer vector using the field-based derive (simplest approach):

use miniextendr_api::prelude::*;

// 1. Define your data type with derive - generates everything
#[derive(AltrepInteger)]
#[altrep(len = "len", elt = "value", class = "ConstantInt")]
pub struct ConstantIntData {
    value: i32,
    len: usize,
}

// 2. Export a constructor
#[miniextendr]
pub fn constant_int(value: i32, n: i32) -> SEXP {
    let data = ConstantIntData { value, len: n as usize };
    data.into_sexp()
}

Usage in R:

x <- constant_int(42L, 1000000L)  # Creates 1M-element vector using O(1) memory
x[1]     # 42
x[500]   # 42
sum(x)   # 42000000 (uses default R sum)

🔗Choosing ALTREP vs Regular Conversion

miniextendr offers two conversion paths for Rust data:

🔗Regular Conversion (IntoR) - Copy to R

#[miniextendr]
fn get_data() -> Vec<i32> {
    vec![1, 2, 3, 4, 5]
}
// Or explicitly: vec.into_sexp()

Behavior:

  • Data is copied to R’s heap
  • Original Vec is dropped
  • R owns a regular integer vector (INTSXP)
  • O(n) memory copy, O(n) memory allocation

Best for:

  • Small data (<1000 elements)
  • Data R will modify
  • Temporary results
  • When simplicity matters

🔗ALTREP Conversion (IntoRAltrep) - Zero-Copy

use miniextendr_api::prelude::*;

#[miniextendr]
fn get_data() -> SEXP {
    let vec = vec![1, 2, 3, 4, 5];
    vec.into_sexp_altrep()
}
// Or: Altrep(vec).into_sexp()

Behavior:

  • Data stays in Rust (ExternalPtr wrapper)
  • No copying, no duplication
  • R accesses via ALTREP callbacks
  • O(1) creation, ~10ns per element overhead

Best for:

  • Large vectors (>1000 elements)
  • Lazy evaluation (compute on access)
  • External data (files, APIs, databases)
  • Zero-copy requirements

🔗Performance Comparison (Measured)

Pure Creation (No Access):

SizeCopyALTREPSpeedup
1000.33 ms0.42 ms0.8x (copy faster)
1,0000.43 ms0.50 ms0.9x (similar)
100,0000.44 ms0.42 ms1.0x (similar)
1,000,0000.44 ms0.20 ms2.2x faster
10,000,0004.16 ms1.90 ms2.2x faster

Partial Access (Create 1M, Access First 10):

SizeCopyALTREPSpeedup
10,0000.02 ms0.02 ms1.0x
100,0000.06 ms0.02 ms3.0x faster
1,000,0000.42 ms0.20 ms2.1x faster
10,000,0004.28 ms0.08 ms53.5x faster

Memory:

  • Copy (1M elements): R heap +3.8 MB
  • ALTREP (1M elements): R heap +0.0 MB (data in Rust heap)

Benchmarks run on Apple M-series, R 4.5. Your results may vary.

🔗Decision Guide

Is your data > 1000 elements?
├─ Yes → Use .into_sexp_altrep()
└─ No
   └─ Will R modify it?
      ├─ Yes → Use .into_sexp() (copy)
      └─ No → Either works, .into_sexp() is simpler

🔗Examples

use miniextendr_api::prelude::*;

// Small data - copy is fine
#[miniextendr]
fn get_config() -> Vec<i32> {
    vec![1, 2, 3]  // Automatically copies via IntoR
}

// Large data - use ALTREP
#[miniextendr]
fn get_large_data() -> SEXP {
    let data = vec![0; 1_000_000];
    data.into_sexp_altrep()  // Zero-copy!
}

// Lazy computation - definitely ALTREP
#[miniextendr]
fn fibonacci_seq(n: i32) -> SEXP {
    (0..n as usize)
        .map(|i| fibonacci(i))
        .collect::<Vec<i32>>()
        .into_sexp_altrep()
}

// Range - already lazy, use ALTREP
#[miniextendr]
fn int_range(from: i32, to: i32) -> SEXP {
    (from..to)
        .collect::<Vec<_>>()
        .into_sexp_altrep()
}

🔗Migration from Altrep(...) to .into_sexp_altrep()

Both forms are equivalent and compile to identical code:

// Old style (still works!)
return Altrep(vec).into_sexp();

// New style (more explicit)
return vec.into_sexp_altrep();

// Both are valid - use whichever is clearer

🔗Architecture Overview

miniextendr’s ALTREP system uses a single-struct pattern with two derive paths:

┌─────────────────────────────────────────────────────────────────┐
│  Path A: Field-based derive (simplest)                          │
│  #[derive(AltrepInteger)] + #[altrep(len, elt, class)]         │
│  Generates EVERYTHING: traits, registration, IntoR, Ref/Mut    │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  Path B: Manual data traits (still one derive)                  │
│  #[derive(AltrepInteger)] + #[altrep(manual, class = "...")]    │
│  Generates registration (TypedExternal, RegisterAltrep, IntoR)  │
│  AND the low-level bridge. YOU implement AltrepLen, Alt*Data.   │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│  High-Level Data Traits (you implement, or derive generates)    │
│  AltrepLen, AltIntegerData, AltRealData, etc.                   │
│  Safe, idiomatic Rust - no raw SEXP handling                    │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│  Low-Level Traits (auto-generated unless #[altrep(no_lowlevel)])│
│  Implements Altrep, AltVec, AltInteger traits                   │
└─────────────────────────────────────────────────────────────────┘

No wrapper struct is needed. The data struct registers directly with R.

Path B used to be spelled with the generic #[derive(Altrep)] plus an explicit impl_alt*_from_data!(MyType) call. That form still exists (it is what #[altrep(no_lowlevel)] falls back to — see Low-Level Trait Macros), but every example below now uses the per-family derive (AltrepInteger, AltrepReal, …) with #[altrep(manual)] instead: same registration, but the derive also auto-emits the low-level bridge, so you never write the macro call yourself.

🔗R defaults and the miniextendr bridge

The defaults below were checked against R-devel’s src/main/altrep.c on 2026-10-10. They are the defaults installed by R, before miniextendr registers its callbacks.

Callback / classR default
All classes: LengthError: the class must supply a length.
All classes: Serialized_state, Duplicate, CoerceC null pointer, requesting the caller’s ordinary-vector fallback. Unserialize itself errors.
All classes: InspectFalse, requesting ordinary inspection.
Vector DataptrError. R does not provide an automatic materializer here.
Dataptr_or_null, Extract_subsetC null pointer. A declined subset must not return the R NULL object.
Integer, real, logical, raw, complex: Elt / Get_regionElt reads through DATAPTR; Get_region loops through element access. Supplying a lazy Elt avoids forcing a pointer for region access.
Integer, real, logical, string: sortedness / missingnessUnknown sortedness and no guarantee of being NA-free. Raw and complex have no such callbacks.
Integer and real: Sum, Min, Max; logical: SumC null pointer, requesting ordinary reduction.
String and list: Elt / Set_eltError. Registering an element reader does not install a writer. Lists also have a separate erroring Dataptr default.

Miniextendr’s standard atomic bridge installs a materializing Dataptr callback. When the backing value cannot supply a pointer, the bridge fills an R vector and roots it in data2. Subsequent element, region, length, reduction and serialization paths must agree with that cache. This is a framework contract, not an R default; see Bridge cache contract.

🔗Containers, ownership, and views

Rust representationSuitable R exposureRequired ownership rule
Owned Vec<T> / Box<[T]>, native element layoutExplicit ALTREP can retain the allocation; ordinary IntoR copies.Once a pointer escapes, keep its allocation valid and preserve R aliasing semantics.
Cow<[T]>Borrow for reads, own on writes through to_mut().Borrowed storage must remain rooted/alive. Deserialization must own its decoded data unless it explicitly retains the serialized state.
IteratorA concrete registered ALTREP wrapper over an iterator state, with a declared length and access/cache policy.R requests random element/region access; define replay, prefix caching or skipping, and serialization behavior. A data-trait implementation alone is not a registered class.
HashMap<K, V>A named list or indexed ALTREP view, with an explicit order snapshot.Hash buckets are not a contiguous native R buffer. Root R values held by any cached index or sidecar.
Read / Write / SeekA connection for stream semantics.Preserve EOF, failure and position semantics; a stream is not inherently a random-access R vector.

A Vec can grow before exposing its buffer, and a later callback can retrieve the new pointer from the owning value. A replacement function that re-enters Rust can grow the owned buffer; subsequent Length, Elt and Dataptr callbacks can then expose the current allocation. This requires a coordinated Duplicate policy: R can duplicate before invoking the setter, and the default duplication may materialize an ordinary R vector. A live-view class can deliberately preserve its owner, while a value class can duplicate its ALTREP state and detach the storage on write. The latter must keep an existing alias unchanged.

The derives offer #[altrep(duplicate = "clone")] for value semantics. It requires Clone, clones the data1 ExternalPtr owner, and copies the authoritative data2 cache, including a partial string cache. The copy stays ALTREP, and R’s default DuplicateEX copies the attributes onto it, so y <- x; y[1] <- 0L leaves x unchanged. Omitting the option keeps R’s default. It composes with dataptr, subset and serialize; no_lowlevel leaves the policy to your own Altrep impl.

A custom Clone must detach buffers exposed for R writes. Vec and owned Cow clones do this; simply sharing an Rc allocation is not sufficient.

Derive-generated string and list classes take R’s element writes (x[[i]] <- v) without a Dataptr. The first write copies the values into a plain vector in data2; from then on element reads, Duplicate and serialization use it, and the Rust data stays as it was. So under duplicate = "clone" a copy that R writes to stays ALTREP and leaves the original unchanged, and a list or string saved after a write restores as an ordinary vector holding the written values.

There is no derive option that returns the original object from Duplicate. R’s DuplicateEX then leaves the attributes alone, so every copy R makes is the same SEXP: y[1] <- 0L writes into x, names(y) <- ... and class(y) <- ... rename and reclass x, as.vector(y) strips x’s names, and a function that modifies its argument modifies the caller’s vector. A hand-written no_lowlevel Duplicate can still do this, as the live view in tests/altrep_assignment.rs does, but it gives up R’s value semantics. A live view that keeps attributes per copy is tracked in #2051.

AltrepBuffer<T> is a native owner that can grow after R has a pointer to it. Return Altrep(AltrepBuffer::new(values)) for i32, f64, u8, Rcomplex or RLogical. Its edit(|vec| ...) method gives the closure the Vec. Once R has requested a pointer, an edit first clones the values and keeps the old Vec until the owner is dropped, so the old pointer stays valid and later callbacks read the edited values. Before that, edits reuse the allocation. The built-in buffer classes clone on R duplication and serialize the current values. A custom class can delegate its native data traits to the buffer and pick its own duplication policy.

R has no notification for a pointer it no longer uses, so every edit after an exposure keeps one more allocation until the owner is dropped (#2052). An edit also changes the R object in place: every binding of that object sees it, since R only duplicates when it writes through a binding itself.

Both duplication hooks are available today: implement Altrep::duplicate with HAS_DUPLICATE = true, or duplicate_ex with HAS_DUPLICATE_EX = true. The bridge registers those callbacks conditionally. R’s default DuplicateEX delegates to Duplicate and handles attributes on a returned copy; overriding DuplicateEX means handling those attributes yourself. With a per-family derive, #[altrep(manual, no_lowlevel)] leaves the low-level policy to your implementation.

R’s own wrapper classes coordinate duplication with a data lock once a pointer has been exposed (see wrapper_Duplicate and wrapper_Dataptr). This is a useful model for ordinary value semantics: preserve the old object’s buffer and mutate a detached replacement. A live-view class has explicit reference semantics and needs a more restrictive pointer/borrow protocol.

An already returned pointer still refers to the old allocation. End Rust borrows before the replacement call and reacquire pointers afterwards; do not keep a slice or C pointer across a boundary that may resize its owner. Pointer refresh at the callback boundary and preservation of outstanding borrows are distinct requirements.

Current design follow-ups include composing container duplication and mutation policies #1924, native-newtype vector conversion coherence #1684, lifetimes tied to rooted R inputs #1427, and access to sidecar handles from ALTREP callbacks #1880.

🔗Sidecars in callbacks

An ALTREP struct can hold R values in #[r_data] Sidecar<T> fields, as an ExternalPtr struct can. The values live in the protection list of the data1 owner, so the ALTREP vector keeps them alive. The derive emits only the Rust accessors: T::field(&ptr), and T::set_field(&mut ptr, value) with mut. An ALTREP vector has no R field methods, so #[r_data] takes only ref and mut.

The high-level data traits (AltrepLen, Alt*Data) receive &self, which has no way back to its handle, so they can’t read sidecars (#1880). A callback that needs a sidecar uses #[altrep(manual, no_lowlevel)] and gets the handle from the ALTREP SEXP:

#[derive(AltrepInteger)]
#[altrep(manual, no_lowlevel)]
struct SidecarVector {
    #[r_data(ref)]
    values: Sidecar<SEXP>,
}

impl AltInteger for SidecarVector {
    const HAS_ELT: bool = true;
    fn elt(x: SEXP, i: R_xlen_t) -> i32 {
        // An existing data2 cache takes precedence after materialization.
        if let Some(cache) = unsafe { materialized_altrep_data2(x) } {
            return i32::elt(cache, i);
        }
        let owner = unsafe { altrep_data1_as_unchecked::<Self>(x) }.unwrap();
        i32::elt(Self::values(&owner), i)
    }
}

rpkg/src/rust/altrep_sidecar_tests.rs has the complete class, with Altrep::length, AltVec and impl_inferbase_integer!. Its Dataptr copies into data2 instead of handing R a writable pointer into the sidecar vector, which other R code may also reference. Once data2 exists, Elt and length read it. With no serialize or duplicate method, R serializes and duplicates the vector as an ordinary one. The default RUnwind guard covers the sidecar conversions and allocations.

A class whose callbacks read only Rust fields can still use a sidecar to keep an R value alive. A duplicate = "clone" copy clones the handle, so the copy holds the same R values. Serialization never includes sidecars: R saves the class’s serialized state, or else the values.

🔗High-Level Data Traits

🔗Core Trait: AltrepLen

Every type using the high-level bridge implements AltrepLen:

impl AltrepLen for MyData {
    fn altrep_len(&self) -> usize {
        // Return the vector length
    }
}

🔗Type-Specific Traits

R Vector TypeRust TraitRequired Method
integerAltIntegerDatafn elt(&self, i: usize) -> i32
numericAltRealDatafn elt(&self, i: usize) -> f64
logicalAltLogicalDatafn elt(&self, i: usize) -> Logical
rawAltRawDatafn elt(&self, i: usize) -> u8
characterAltStringDatafn elt(&self, i: usize) -> Option<&str>
complexAltComplexDatafn elt(&self, i: usize) -> Rcomplex
listAltListDatafn elt(&self, i: usize) -> SEXP

🔗Optional Methods

Each trait provides optional methods you can override:

impl AltIntegerData for MyData {
    fn elt(&self, i: usize) -> i32 {
        // Required: element access
    }

    fn no_na(&self) -> Option<bool> {
        // Optional: NA hint (enables optimizations)
        Some(true)  // No NAs in this vector
    }

    fn is_sorted(&self) -> Option<Sortedness> {
        // Optional: sortedness hint
        Some(Sortedness::Increasing)
    }

    fn sum(&self, na_rm: bool) -> Option<i64> {
        // Optional: O(1) sum (instead of element-by-element)
        Some(self.formula_based_sum())
    }

    fn min(&self, na_rm: bool) -> Option<i32> {
        // Optional: O(1) min
        Some(self.known_minimum())
    }

    fn max(&self, na_rm: bool) -> Option<i32> {
        // Optional: O(1) max
        Some(self.known_maximum())
    }

    fn get_region(&self, start: usize, len: usize, buf: &mut [i32]) -> usize {
        // Optional: bulk element access. The default fills from elt() in a
        // loop (shown here); override when a block can be filled cheaper
        // than n independent elt() calls. Clamp, fill, and return the count
        // written — see ALTREP_GET_REGION.md for the contract and patterns.
        let n = len.min(buf.len()).min(self.altrep_len().saturating_sub(start));
        for (k, slot) in buf[..n].iter_mut().enumerate() {
            *slot = self.elt(start + k);
        }
        n
    }
}

🔗Example: Arithmetic Sequence

A lazy arithmetic sequence that computes elements on demand. This uses the manual data-traits pattern (#[altrep(manual)] on the AltrepReal derive) since we need custom trait implementations with optimization hints:

#[derive(miniextendr_api::AltrepReal)]
#[altrep(manual, class = "ArithSeq")]
pub struct ArithSeqData {
    start: f64,
    step: f64,
    len: usize,
}

impl AltrepLen for ArithSeqData {
    fn altrep_len(&self) -> usize {
        self.len
    }
}

impl AltRealData for ArithSeqData {
    fn elt(&self, i: usize) -> f64 {
        self.start + (i as f64) * self.step
    }

    fn no_na(&self) -> Option<bool> {
        Some(true)  // Arithmetic sequences never produce NA
    }

    fn is_sorted(&self) -> Option<Sortedness> {
        if self.step < 0.0 {
            Some(Sortedness::Decreasing)
        } else {
            Some(Sortedness::Increasing)
        }
    }

    fn sum(&self, _na_rm: bool) -> Option<f64> {
        // O(1) sum using arithmetic series formula: n*(first+last)/2
        let last = self.start + (self.len - 1) as f64 * self.step;
        Some(self.len as f64 * (self.start + last) / 2.0)
    }
}

// No explicit `impl_altreal_from_data!` call needed — `#[altrep(manual)]`
// on the `AltrepReal` derive auto-emits the low-level bridge.

#[miniextendr]
pub fn arith_seq(from: f64, to: f64, length_out: i32) -> SEXP {
    let len = length_out as usize;
    let step = if len > 1 { (to - from) / (len - 1) as f64 } else { 0.0 };
    ArithSeqData { start: from, step, len }.into_sexp()
}

🔗Lazy Materialization

For cases where you want lazy computation but also need to support DATAPTR:

use miniextendr_api::prelude::*;
use miniextendr_api::altrep_data::AltrepDataptr;

#[derive(AltrepInteger)]
#[altrep(manual, dataptr, class = "LazyIntSeq")]
pub struct LazyIntSeqData {
    start: i32,
    step: i32,
    len: usize,
    materialized: Option<Vec<i32>>,  // Lazily allocated
}

impl AltrepLen for LazyIntSeqData {
    fn altrep_len(&self) -> usize { self.len }
}

impl AltIntegerData for LazyIntSeqData {
    fn elt(&self, i: usize) -> i32 {
        // Compute on-the-fly (no materialization needed)
        self.start.saturating_add((i as i32).saturating_mul(self.step))
    }
}

impl AltrepDataptr<i32> for LazyIntSeqData {
    fn dataptr(&mut self, _writable: bool) -> Option<*mut i32> {
        // Materialize on first DATAPTR access
        if self.materialized.is_none() {
            let data: Vec<i32> = (0..self.len)
                .map(|i| self.start.saturating_add((i as i32).saturating_mul(self.step)))
                .collect();
            self.materialized = Some(data);
        }
        self.materialized.as_mut().map(|v| v.as_mut_ptr())
    }

    fn dataptr_or_null(&self) -> Option<*const i32> {
        // Return pointer only if already materialized
        // Returning None tells R to use elt() instead
        self.materialized.as_ref().map(|v| v.as_ptr())
    }
}

// No explicit `impl_altinteger_from_data!` call — the `dataptr` key on
// `#[altrep(manual, dataptr, ...)]` auto-emits the low-level bridge wired to
// the `AltrepDataptr<i32>` impl above.

Key behaviors:

  • elt() always works, no allocation needed
  • dataptr_or_null() returns None until materialized
  • dataptr() allocates on first call, caches result
  • Operations like x + y trigger dataptr(), causing materialization

🔗Serialization Support

To make ALTREP objects serializable (for saveRDS/readRDS), add serialize to the derive attribute from the previous section — #[altrep(manual, dataptr, serialize, class = "LazyIntSeq")] — and implement AltrepSerialize:

use miniextendr_api::altrep_data::AltrepSerialize;

impl AltrepSerialize for LazyIntSeqData {
    fn serialized_state(&self) -> SEXP {
        // Return a serializable representation (typically a simple vector)
        unsafe {
            use miniextendr_api::SEXPTYPE;
            use miniextendr_api::sys::{Rf_allocVector, SET_INTEGER_ELT};
            let state = Rf_allocVector(SEXPTYPE::INTSXP, 3);
            SET_INTEGER_ELT(state, 0, self.start);
            SET_INTEGER_ELT(state, 1, self.step);
            SET_INTEGER_ELT(state, 2, self.len as i32);
            state
        }
    }

    fn unserialize(state: SEXP) -> Option<Self> {
        unsafe {
            use miniextendr_api::sys::INTEGER_ELT;
            Some(LazyIntSeqData {
                start: INTEGER_ELT(state, 0),
                step: INTEGER_ELT(state, 1),
                len: INTEGER_ELT(state, 2) as usize,
                materialized: None,  // Fresh - not materialized
            })
        }
    }
}

No explicit macro call here either — the derive reads the serialize key the same way it read dataptr.

🔗Class Registration and Cross-Session readRDS

For saveRDS/readRDS to work across R sessions, R must be able to find the ALTREP class by name when unserializing. This requires two things:

🔗1. DllInfo – associating the class with a package

R_make_alt*_class(class_name, pkg_name, dll_info) takes a DllInfo* that tells R which package owns the class. The serialized stream stores the class name and package name. On readRDS, R looks up the class by (class_name, pkg_name) – this lookup requires the DllInfo to have been provided at registration time.

miniextendr stores the DllInfo from package_init in a global and passes it to all R_make_alt*_class calls:

// In init.rs -- during R_init_<pkg>:
crate::set_altrep_dll_info(dll);

// In registration code -- when creating the class:
let dll = $crate::altrep_dll_info();
let cls = R_make_altreal_class(class_name, pkg_name, dll);

Without DllInfo (NULL), R can’t find the class during deserialization, even if it’s registered. This was a bug: all classes were registered with NULL.

🔗2. Eager registration: classes must exist before readRDS runs

ALTREP classes are registered in two ways:

Derive-generated classes (user #[derive(Altrep)] / #[derive(AltrepInteger)] structs) register via linkme’s #[distributed_slice]. Each ALTREP struct emits an entry that’s called during R_init:

// Generated by proc-macro:
#[distributed_slice(MX_ALTREP_REGISTRATIONS)]
fn register_my_class() {
    MyType::get_or_init_class();
}

// Called during R_init:
for reg_fn in MX_ALTREP_REGISTRATIONS.iter() {
    reg_fn();
}

Built-in classes (Vec<f64>, Box<[i32]>, Arrow arrays, etc.) use OnceLock inside RegisterAltrep::get_or_init_class(). These are lazy, the class is created on first use (e.g., when .into_sexp_altrep() is called). This is a problem for readRDS: R tries to find the class during deserialization, before any miniextendr code has called into_sexp_altrep.

The fix: register_builtin_altrep_classes() is called during R_init and eagerly calls get_or_init_class() for every built-in type:

// In registry.rs - during R_init:
register_builtin_altrep_classes();  // Vec, Box, Range
#[cfg(feature = "arrow")]
register_arrow_altrep_classes();    // Float64Array, Int32Array, etc.
pub(crate) fn register_builtin_altrep_classes() {
    use crate::altrep::RegisterAltrep;
    Vec::<i32>::get_or_init_class();
    Vec::<f64>::get_or_init_class();
    Vec::<bool>::get_or_init_class();
    Vec::<u8>::get_or_init_class();
    Vec::<String>::get_or_init_class();
    Vec::<Option<String>>::get_or_init_class();
    // ... all built-in types
}

🔗What happens during readRDS

Session A: saveRDS(altrep_vec, "data.rds")
  → ALTREP serialize hook fires
  → serialized_state() materializes data to plain R vector
  → Stream contains: class_name="miniextendr_Vec_f64", pkg_name="miniextendr", state=<REALSXP>

Session B: library(miniextendr); readRDS("data.rds")
  → R_init_miniextendr runs → registers all ALTREP classes (with DllInfo)
  → readRDS parses stream → finds class "miniextendr_Vec_f64" in package "miniextendr"
  → R calls unserialize(class, state) → reconstructs Vec<f64> from the REALSXP
  → Returns a live ALTREP vector backed by Rust data

Session C: readRDS("data.rds")  # WITHOUT library(miniextendr)
  → ALTREP class not registered → R falls back to the serialized state
  → Returns a plain R numeric vector (the materialized data)
  → Works correctly (just not an ALTREP anymore)

🔗Adding serialization to new types

When you add serialization support to a new ALTREP type:

  1. Implement AltrepSerialize for the type
  2. Enable it — user types add serialize to the derive attribute (#[altrep(manual, serialize, ...)]); built-in types (in miniextendr-api itself, which don’t go through the derive) call the macro directly: impl_altreal_from_data!(MyType, serialize);
  3. If it’s a built-in type, add it to register_builtin_altrep_classes() so it’s eagerly registered at init

User types don’t need step 3. The proc-macro generates #[distributed_slice] entries automatically.

🔗Class names must be unique within the package

R_make_alt*_class expects a unique (class_name, pkg_name) pair. Two registrations that pick the same name silently clobber each other. miniextendr catches this at package init by tracking every registered class name and panicking on the first duplicate:

miniextendr: duplicate ALTREP class name "MyClass" - each ALTREP type must have
a unique class name within the package

If you hit this, check every #[altrep(class = "...")] string in your crate (derive-generated and manual paths combined). Two types share the same name. Built-in classes (Vec<i32>, Box<[f64]>, etc.) are registered with names the framework guarantees unique, so the collision is always in your own code.

🔗Mutable Vectors

Derive-generated string and list classes already take R’s element writes: the first one copies the values into data2, as described under Containers, ownership, and views. Write set_elt yourself, on the low-level AltString/AltList traits, only when R’s writes should reach your Rust data instead.

Important: Only String and List types support set_elt. Numeric vectors (Integer, Real, Logical, Raw, Complex) cannot be mutated through ALTREP.

set_elt is not a data-trait method and not an impl_alt*_from_data! macro option — AltStringData/AltListData (the high-level traits the family derives generate from) have no set_elt method at all, so there is no derive attribute key for a custom one either. It lives one layer down, on the low-level AltString/AltList traits (miniextendr_api::altrep_traits) as an optional method gated by const HAS_SET_ELT: bool. Getting there means opting out of both derive conveniences with #[altrep(manual, no_lowlevel)]:

  • manual skips generating AltrepLen/Alt*Data — irrelevant here, since a mutable type doesn’t build its low-level traits from those anyway.
  • no_lowlevel skips the derive’s automatic impl_alt*_from_data! call. That macro can’t help you here — it’s generated from AltStringData/AltListData, neither of which has set_elt.

What’s left after both flags: Altrep, AltVec, and the family trait (AltString/AltList) written by hand, operating directly on SEXP (no &self). The derive still emits registration (TypedExternal, RegisterAltrep, IntoR, the linkme entry) and the AltrepExtract blanket impl that gets you &Self/&mut Self from the SEXP — you only write the three traits below plus a one-line impl_inferbase_*! call (registration needs an InferBase impl, which is otherwise emitted as part of the suppressed low-level bridge).

🔗Mutable String Vectors

use miniextendr_api::R_xlen_t;
use miniextendr_api::altrep_traits::{Altrep as LowLevelAltrep, AltVec, AltString};
use miniextendr_api::altrep_data::AltrepExtract;
use miniextendr_api::prelude::*;
use std::cell::RefCell;
use std::ffi::CStr;

#[derive(AltrepString)]
#[altrep(manual, no_lowlevel, class = "MutableString")]
pub struct MutableStringData {
    strings: RefCell<Vec<Option<String>>>,
}

impl LowLevelAltrep for MutableStringData {
    fn length(x: SEXP) -> R_xlen_t {
        let data = unsafe { MutableStringData::altrep_extract_ref(x) };
        data.strings.borrow().len() as R_xlen_t
    }
}

impl AltVec for MutableStringData {}

impl AltString for MutableStringData {
    fn elt(x: SEXP, i: R_xlen_t) -> SEXP {
        let data = unsafe { MutableStringData::altrep_extract_ref(x) };
        match data.strings.borrow().get(i as usize).and_then(|s| s.as_deref()) {
            Some(s) => SEXP::charsxp(s),
            None => SEXP::na_string(),
        }
    }

    // Enable mutation
    const HAS_SET_ELT: bool = true;
    fn set_elt(x: SEXP, i: R_xlen_t, v: SEXP) {
        let data = unsafe { MutableStringData::altrep_extract_mut(x) };
        if let Some(slot) = data.strings.borrow_mut().get_mut(i as usize) {
            *slot = if v.is_na_string() {
                None
            } else {
                // `v` is itself the CHARSXP being assigned — read it with
                // `R_CHAR`, not `SexpExt::string_elt_str` (which indexes a
                // STRSXP vector; `v` isn't one).
                let cstr = unsafe { CStr::from_ptr(miniextendr_api::sys::R_CHAR(v)) };
                Some(cstr.to_string_lossy().into_owned())
            };
        }
    }
}

// Registration still needs `InferBase`; `no_lowlevel` only suppresses the
// bridge that would otherwise emit it, so provide it directly.
miniextendr_api::impl_inferbase_string!(MutableStringData);

Unlike the earlier (*ptr).get(i)-through-a-raw-pointer approach, elt here never aliases a live &mut — it borrows the RefCell immutably for the duration of the match, and returns an owned SEXP (SEXP::charsxp/ SEXP::na_string) rather than a borrowed &str, so there’s no lifetime trick and no dangerous_implicit_autorefs risk.

🔗Mutable List Vectors

A list’s elements are R objects, and R’s garbage collector does not look inside Rust memory, data1 included. So keep them in an R list that the class roots, not in a Vec<SEXP>:

use miniextendr_api::R_xlen_t;
use miniextendr_api::altrep_traits::{Altrep as LowLevelAltrep, AltVec, AltList};
use miniextendr_api::altrep_data::AltrepExtract;
use miniextendr_api::prelude::*;

#[derive(AltrepList)]
#[altrep(manual, no_lowlevel, class = "MutableList")]
pub struct MutableListData {
    // An R list rooted for as long as the class holds it.
    elements: Preserved,
}

impl LowLevelAltrep for MutableListData {
    fn length(x: SEXP) -> R_xlen_t {
        let data = unsafe { MutableListData::altrep_extract_ref(x) };
        data.elements.xlength()
    }
}

impl AltVec for MutableListData {}

impl AltList for MutableListData {
    fn elt(x: SEXP, i: R_xlen_t) -> SEXP {
        let data = unsafe { MutableListData::altrep_extract_ref(x) };
        data.elements.vector_elt(i)
    }

    // Enable mutation
    const HAS_SET_ELT: bool = true;
    fn set_elt(x: SEXP, i: R_xlen_t, v: SEXP) {
        let data = unsafe { MutableListData::altrep_extract_ref(x) };
        data.elements.set_vector_elt(i, v);
    }
}

miniextendr_api::impl_inferbase_list!(MutableListData);

🔗Safety Considerations

1. R’s Copy-on-Write: R may copy your vector before calling set_elt, so mutations may not affect the original vector reference.

2. GC Protection: When storing SEXPs in mutable lists:

  • A SEXP stored in Rust memory, data1 included, is not protected; store it in an R list the class roots (a Preserved, or data2)
  • If you create new SEXPs, ensure they’re returned to R immediately
  • Don’t store raw SEXP pointers that outlive their protection

3. Thread Safety:

  • ALTREP callbacks run on R’s main thread
  • Use RefCell (not Mutex) for interior mutability
  • No async/threading allowed inside ALTREP methods

4. Materialization:

  • R may materialize (copy to regular vector) when it needs a dataptr
  • After materialization, mutations go to the copy, not your ALTREP

🔗When to Use Mutable ALTREP

Good use cases:

  • Lazy evaluation with caching
  • Proxying to external mutable data sources
  • Implementing special data structures (e.g., sparse vectors)

Avoid for:

  • Regular data storage (use Vec<T> instead)
  • Situations where you need dataptr (forces materialization)
  • Performance-critical code (mutations have overhead)

🔗Standard Type Support

miniextendr provides built-in ALTREP support for common Rust types via .into_sexp_altrep():

🔗Vec<T> (Owned Data)

#[miniextendr]
pub fn simple_vec_int(values: Vec<i32>) -> SEXP {
    values.into_sexp_altrep()
}

🔗Owned unsized strings

Box<str>, Rc<str> and Arc<str> convert to/from character scalars. Their Option and Vec<Option<...>> forms preserve NA; the plain forms follow String and map incoming NA to an empty string. Cow<str> keeps its existing scalar conversion.

Altrep<Vec<Box<str>>>, Altrep<Vec<Rc<str>>> and Altrep<Vec<Arc<str>>>, including optional elements, retain their string owners and create CHARSXPs on demand. R writes materialize the authoritative STRSXP cache. The UTF-8 bytes of a single str are not exposed as a mutable R character-vector buffer.

🔗Shared slices: Rc<[T]> and Arc<[T]>

Altrep(Rc::<[i32]>::from(vec![1, 2, 3])) and its Arc equivalent keep the shared owner in the R vector. Supported elements are i32, f64, u8, Rcomplex, RLogical, bool and String.

Element and region reads use the shared allocation. A pointer to the whole buffer is different, because R may write through it:

  • Dataptr_or_null returns the buffer only when the vector owns it alone (one strong reference, no weak ones). Otherwise R falls back to element and region reads.
  • Dataptr, read-only included, calls make_mut first. If other Rc/Arc handles share the allocation, this copies the whole buffer: any C code that calls DATAPTR_RO(), INTEGER() or REAL() on the vector makes R’s copy, and the other handles keep the old allocation and never see R’s writes. A unique owner is used in place. The copy happens once; the read-only case copies too, so that a pointer R already holds stays valid if R later asks to write.
  • Duplicate shares an owner that is still shared, since no pointer to it has been handed out. It copies a unique owner, which may have handed one out. That also copies a unique owner that never did; no extra state is kept.

bool and String vectors keep R’s values in data2 (a logical vector, or the string element cache). Duplicate copies data2’s current values, NA included; a string cache is copied slot by slot, and the copy computes the slots not yet read from its own owner. Native owners serialize their values and come back as the same class; once R has written to a bool or String vector, it is saved as an ordinary vector.

The owner is only safe to share while R can’t write through it. Don’t clone or downgrade it through unsafe extraction (altrep_extract_ref / altrep_extract_mut) once R may hold its buffer: the Rust clone would see R’s writes, and Duplicate, finding a shared owner, would give the new R vector the same buffer, so a write to one vector changes the other. An Arc doesn’t make writing to that buffer from another thread safe either.

🔗[T; N] (Owned Fixed-Size Array)

Altrep([1_i32, 2, 3]) retains the array itself in an external pointer. Arrays of i32, f64, u8, Rcomplex, RLogical, bool and String work for any length, including zero. Native layouts expose that array’s buffer; bool and string arrays materialize R storage when needed, and subsequent callbacks read the materialized values, including R-written NA.

Each element type and length gets a separate class on first conversion. These classes use R’s default duplicate and serialization behaviour: an ordinary R vector with the current values. Serialized data therefore does not depend on registering a particular array length in a fresh session.

Altrep<T> also allows T: ?Sized for borrowed views such as &Altrep<[i32]>. Returning a value to R still requires an owned backing type; both Altrep<Box<[i32]>> and Box<Altrep<[i32]>> transfer the boxed slice without copying its allocation. The latter can be obtained by unsizing Box::new(Altrep([1_i32, 2, 3])).

Altrep::into_boxed_inner keeps the metadata of a custom trailing DST or a trait object too. Such an owner still needs its own ALTREP class and data traits. AltrepExtract accepts an unsized view. AltrepSerialize needs a sized type with a working unserialize, because a class that saves its state must be able to load it again. A type that can’t rebuild itself leaves serialization to R, which saves an ordinary vector.

A custom trailing DST works the same way:

struct Tagged<T: ?Sized> {
    tag: u32,
    values: T,
}

let value: Box<Altrep<Tagged<[i32]>>> =
    Box::new(Altrep(Tagged { tag: 7, values: [1, 2, 3] }));

The sized Box<Tagged<[i32]>> is the owner: it implements TypedExternal, the data traits and RegisterAltrep. An AltrepExtract impl on Tagged<[i32]> gives the callbacks the unsized view, which is valid only while the R vector is. miniextendr-api/tests/altrep_unsized.rs has the whole example.

🔗Box<[T]> (Immutable Owned Slice)

#[miniextendr]
pub fn boxed_ints(n: i32) -> SEXP {
    let data: Box<[i32]> = (1..=n).collect();
    data.into_sexp_altrep()
}

🔗Static Slices (&'static [T])

static MY_DATA: &[i32] = &[10, 20, 30, 40, 50];

#[miniextendr]
pub fn static_ints() -> SEXP {
    MY_DATA.into_sexp_altrep()
}

Note: Static ALTREPs are read-only and cannot support writable DATAPTR.

🔗Complex Numbers

use miniextendr_api::prelude::*;
use miniextendr_api::Rcomplex;

#[derive(AltrepComplex)]
#[altrep(manual, class = "UnitCircle")]
pub struct UnitCircleData {
    n: usize,  // Number of points on unit circle
}

impl AltrepLen for UnitCircleData {
    fn altrep_len(&self) -> usize { self.n }
}

impl AltComplexData for UnitCircleData {
    fn elt(&self, i: usize) -> Rcomplex {
        let theta = 2.0 * std::f64::consts::PI * (i as f64) / (self.n as f64);
        Rcomplex { r: theta.cos(), i: theta.sin() }
    }
}

#[miniextendr]
pub fn unit_circle(n: i32) -> SEXP {
    UnitCircleData { n: n as usize }.into_sexp()
}

🔗Logical Vectors

Use the Logical enum for proper NA handling:

use miniextendr_api::prelude::*;
use miniextendr_api::altrep_data::Logical;

#[derive(AltrepLogical)]
#[altrep(manual, class = "LogicalVec")]
pub struct LogicalVecData {
    data: Vec<Logical>,
}

impl AltrepLen for LogicalVecData {
    fn altrep_len(&self) -> usize { self.data.len() }
}

impl AltLogicalData for LogicalVecData {
    fn elt(&self, i: usize) -> Logical {
        self.data[i]
    }

    fn no_na(&self) -> Option<bool> {
        Some(!self.data.iter().any(|v| matches!(v, Logical::Na)))
    }

    fn sum(&self, na_rm: bool) -> Option<i64> {
        // Sum = count of TRUE values
        let mut total = 0i64;
        for v in &self.data {
            match v {
                Logical::True => total += 1,
                Logical::False => {}
                Logical::Na => if !na_rm { return None; }
            }
        }
        Some(total)
    }
}

🔗String Vectors

String ALTREPs return Option<&str> where None represents NA:

use miniextendr_api::prelude::*;

#[derive(AltrepString)]
#[altrep(manual, class = "StringVec")]
pub struct StringVecData {
    data: Vec<Option<String>>,
}

impl AltrepLen for StringVecData {
    fn altrep_len(&self) -> usize { self.data.len() }
}

impl AltStringData for StringVecData {
    fn elt(&self, i: usize) -> Option<&str> {
        self.data[i].as_deref()  // None = NA
    }

    fn no_na(&self) -> Option<bool> {
        Some(!self.data.iter().any(|v| v.is_none()))
    }
}

🔗Raw Vectors

use miniextendr_api::prelude::*;

#[derive(AltrepRaw)]
#[altrep(manual, class = "RepeatingRaw")]
pub struct RepeatingRawData {
    pattern: Vec<u8>,
    total_len: usize,
}

impl AltrepLen for RepeatingRawData {
    fn altrep_len(&self) -> usize { self.total_len }
}

impl AltRawData for RepeatingRawData {
    fn elt(&self, i: usize) -> u8 {
        self.pattern[i % self.pattern.len()]
    }
}

🔗List Vectors

List vectors (R’s list type / VECSXP) can contain any R objects. The AltListData trait allows you to create lists that compute or fetch elements on demand.

use miniextendr_api::prelude::*;
use miniextendr_api::sys::Rf_ScalarInteger;

#[derive(AltrepList)]
#[altrep(manual, class = "IntegerSequenceList")]
pub struct IntegerSequenceListData {
    n: usize,  // Number of elements in the list
}

impl AltrepLen for IntegerSequenceListData {
    fn altrep_len(&self) -> usize {
        self.n
    }
}

impl AltListData for IntegerSequenceListData {
    fn elt(&self, i: usize) -> SEXP {
        // Each element is a scalar integer equal to its index
        unsafe { Rf_ScalarInteger((i + 1) as i32) }
    }
}

#[miniextendr]
pub fn int_seq_list(n: i32) -> SEXP {
    let data = IntegerSequenceListData { n: n as usize };
    data.into_sexp()
}

Usage in R:

lst <- int_seq_list(5L)
length(lst)  # 5
lst[[1]]     # 1L
lst[[3]]     # 3L
lst[[5]]     # 5L

🔗List Safety Considerations

Important: List elements are SEXPs that must be properly protected from garbage collection. When implementing AltListData::elt():

  1. Return stored SEXPs only if they are rooted: R’s garbage collector does not see SEXPs kept in Rust memory, data1 included. Keep them in an R list the class roots (a Preserved), or use IterListData, which roots the elements its iterator yields
  2. Create new SEXPs: If you create SEXPs on-the-fly (like Rf_ScalarInteger), return them straight away; R protects the value Elt returns
  3. Avoid raw pointers: Don’t store raw SEXP pointers that might become invalid

🔗Practical List Examples

Example 1: Repeating Element

#[derive(miniextendr_api::AltrepList)]
#[altrep(manual, class = "RepeatedList")]
pub struct RepeatedListData {
    element: SEXP,  // Stored in data1 slot (protected)
    n: usize,
}

impl AltrepLen for RepeatedListData {
    fn altrep_len(&self) -> usize {
        self.n
    }
}

impl AltListData for RepeatedListData {
    fn elt(&self, _i: usize) -> SEXP {
        self.element  // Same element for all indices
    }
}

Example 2: List of Named Lists

impl AltListData for NamedListGenerator {
    fn elt(&self, i: usize) -> SEXP {
        // Create a named list for each element
        let names = vec!["x", "y"];
        let values = vec![
            unsafe { Rf_ScalarInteger(i as i32) },
            unsafe { Rf_ScalarReal(i as f64) },
        ];
        // Use miniextendr's list builder
        miniextendr_api::list::named_list(&names, &values).into_sexp()
    }
}

🔗Reference Types

When you need to pass an ALTREP back to Rust functions:

// The ALTREP derives generate these automatically:
// - ConstantIntDataRef: immutable reference to ALTREP data
// - ConstantIntDataMut: mutable reference to ALTREP data

#[miniextendr]
pub fn inspect_constant_int(x: ConstantIntDataRef) -> String {
    format!("value={}, len={}", x.value, x.len)
}

#[miniextendr]
pub fn double_constant_int(mut x: ConstantIntDataMut) {
    x.value *= 2;
}

🔗Low-Level Trait Macros

Every example above passes dataptr / serialize / subset as keys on the #[altrep(manual, ...)] derive attribute, and the derive auto-emits the matching impl_alt*_from_data! call for you. You do not need to call these macros yourself in the normal flow.

The macros are still there as the escape hatch for #[altrep(no_lowlevel)] (when you write the low-level Altrep/AltVec/Alt* traits by hand and only want the macro for a specific family, or you’re calling it from outside a derive entirely — see Mutable Vectors for a case where you skip the macro altogether and hand-write the low-level traits). They accept the same options as the derive keys:

// Basic (element access only)
miniextendr_api::impl_altinteger_from_data!(MyType);

// With dataptr support (enables DATAPTR method)
miniextendr_api::impl_altinteger_from_data!(MyType, dataptr);

// With serialization (enables saveRDS/readRDS)
miniextendr_api::impl_altinteger_from_data!(MyType, serialize);

// With subset optimization (enables optimized x[i] for index vectors)
miniextendr_api::impl_altinteger_from_data!(MyType, subset);

// Combinations (canonical alphabetical order; `dataptr` and `subset` are
// mutually exclusive):
miniextendr_api::impl_altinteger_from_data!(MyType, dataptr, serialize);
miniextendr_api::impl_altinteger_from_data!(MyType, subset, serialize);

// Generic types use the `_generic!` sibling. The first brace group declares
// parameters; the second supplies the impl's where-clause.
miniextendr_api::impl_altinteger_from_data_generic!(
    {T} GenericInt<T> {T: Copy + Into<i32> + 'static}
);

Every family has a matching _generic! macro. Prefer the derive path for concrete types; use this explicit form when a low-level implementation itself must stay generic.

OptionWhat it doesRequires
dataptrEnables DATAPTR methodimpl AltrepDataptr<T>
serializeEnables serializationimpl AltrepSerialize
subsetEnables optimized subsettingimpl AltrepSubset

🔗Sortedness and NA Hints

Providing hints enables R to optimize operations:

use miniextendr_api::altrep_data::Sortedness;

impl AltIntegerData for MyData {
    fn is_sorted(&self) -> Option<Sortedness> {
        match self.ordering {
            Ordering::Ascending => Some(Sortedness::Increasing),
            Ordering::Descending => Some(Sortedness::Decreasing),
            Ordering::Unknown => None,  // Don't know
        }
    }

    fn no_na(&self) -> Option<bool> {
        Some(true)  // Enables R to skip NA checks
    }
}

🔗Subsetting Optimization (Extract_subset)

The extract_subset() method allows you to optimize R’s subsetting operations (x[indices]). Instead of R extracting elements one-by-one, you can return a new ALTREP object or optimized representation.

🔗When R Calls Extract_subset

R calls extract_subset(x, indices, call) when:

  • User writes x[c(1, 3, 5)] - integer vector indices
  • User writes x[condition] - logical vector indices
  • Subsetting with names: x[c("a", "b")]

Note: Single element access x[i] or x[[i]] uses elt(), not extract_subset().

🔗Example: Range Subsetting via #[altrep(subset)]

Add subset to the derive attribute (AltIntegerData/AltComplexData only — subset and dataptr are mutually exclusive, and List rejects subset entirely) and implement the high-level AltrepExtractSubset trait. The bridge already validated indices as an integer vector and converted it to a 1-based &[i32] before calling you; return None to fall back to R’s default element-by-element extraction:

use miniextendr_api::prelude::*;
use miniextendr_api::altrep_data::AltrepExtractSubset;

#[derive(AltrepInteger)]
#[altrep(manual, subset, class = "RangeInt")]
pub struct RangeIntData {
    start: i32,
    len: usize,
}

impl AltrepLen for RangeIntData {
    fn altrep_len(&self) -> usize {
        self.len
    }
}

impl AltIntegerData for RangeIntData {
    fn elt(&self, i: usize) -> i32 {
        self.start + i as i32
    }
}

impl AltrepExtractSubset for RangeIntData {
    fn extract_subset(&self, indices: &[i32]) -> Option<SEXP> {
        // Only optimize the common case: a contiguous ascending run
        // (e.g. `x[1:10]`). Anything else falls back to R's default,
        // element-by-element extraction.
        let is_contiguous_ascending = indices.windows(2).all(|w| w[1] == w[0] + 1);
        if indices.is_empty() || !is_contiguous_ascending {
            return None;
        }
        let new_start = self.start + (indices[0] - 1);
        Some(RangeIntData { start: new_start, len: indices.len() }.into_sexp())
    }
}

#[miniextendr]
pub fn range_int_altrep(from: i32, len: i32) -> SEXP {
    RangeIntData { start: from, len: len.max(0) as usize }.into_sexp()
}

🔗Performance Benefits

O(1) Subset Creation:

x <- range_int_altrep(1L, 1000000L)  # O(1) - no allocation
y <- x[1:100000]                       # O(1) - returns new Range(1, 100001)

Without extract_subset, R would:

  1. Allocate a 100,000-element vector
  2. Call elt() 100,000 times
  3. Fill the new vector

With extract_subset:

  1. Return a new Range object (few bytes)
  2. No element extraction
  3. Lazy evaluation continues

🔗When to Implement Extract_subset

Good candidates:

  • ✅ Mathematical sequences: Range, arithmetic sequences (subset is another sequence)
  • ✅ Constant vectors: Subset is constant with different length
  • ✅ Views/windows: Subset adjusts the window bounds
  • ✅ External data: Subset delegates to underlying data source
  • ✅ Sparse vectors: Subset maintains sparsity

Not worth it for:

  • ❌ Materialized data (Vec, Box): R’s default is already efficient
  • ❌ Complex computations: Unless subset is much simpler than original
  • ❌ Small vectors: Overhead not worth the optimization

🔗Index Types and the Fallback Strategy

AltrepExtractSubset::extract_subset only ever receives already-validated integer indices (&[i32]) — the bridge checks TYPEOF(indices) == INTSXP before calling you and returns SEXP::null() (R’s “use the default” sentinel) itself for any other index type (real, logical, character indices all coerce to integer before reaching an ALTREP’s Extract_subset method in R’s own subsetting machinery). You never need to branch on index type yourself.

Always provide a fallback: return None from extract_subset for any case you don’t want to special-case, and R falls back to element-by-element extraction via elt():

impl AltrepExtractSubset for MyData {
    fn extract_subset(&self, indices: &[i32]) -> Option<SEXP> {
        // Try an optimized path; return None for anything else.
        self.try_optimized_subset(indices)
    }
}

This ensures correctness even when optimization isn’t possible. If you write the raw low-level AltVec::extract_subset(x: SEXP, indices: SEXP, call: SEXP) -> SEXP yourself (the #[altrep(no_lowlevel)] escape hatch), the same sentinel applies: return SEXP::null() to fall back.

🔗Performance Tips

  1. Implement sum/min/max when you can compute them in O(1)
  2. Use no_na() hint when you know there are no NAs
  3. Use is_sorted() hint for sorted data
  4. Implement get_region() for efficient bulk access — see ALTREP_GET_REGION.md for when R calls it and cache-friendly filling patterns
  5. Delay materialization - prefer elt() over dataptr()
  6. Return None from dataptr_or_null() until actually materialized

🔗Common Patterns

🔗Pattern 1: Constant Vector

struct Constant<T> { value: T, len: usize }
// All elements return the same value
fn elt(&self, _i: usize) -> T { self.value }

🔗Pattern 2: Computed Sequence

struct Sequence { start: T, step: T, len: usize }
// Elements computed from formula
fn elt(&self, i: usize) -> T { self.start + i * self.step }

🔗Pattern 3: External Data View

struct ExternalView<'a> { data: &'a [T] }
// Zero-copy view into external data
fn elt(&self, i: usize) -> T { self.data[i] }

🔗Pattern 4: Lazy Computation with Cache

struct Lazy { params: Params, cache: Option<Vec<T>> }
// Compute and cache on first access
fn dataptr(&mut self) -> *mut T {
    if self.cache.is_none() { self.cache = Some(self.compute()); }
    self.cache.as_mut().unwrap().as_mut_ptr()
}

🔗Advanced Methods

These methods are rarely needed but available for special use cases.

🔗Inspect - Custom Debug Output

The inspect() method customizes the output of .Internal(inspect(x)), R’s internal debugging tool.

impl Altrep for MyData {
    const HAS_INSPECT: bool = true;

    fn inspect(
        x: SEXP,
        pre: i32,
        deep: i32,
        pvec: i32,
        inspect_subtree: Option<unsafe extern "C-unwind" fn(SEXP, i32, i32, i32)>,
    ) -> bool {
        // Print custom information
        eprintln!("  MyData ALTREP");
        eprintln!("  - custom_field: {}", /* access your data */);

        // Optionally inspect child objects
        if let Some(inspect) = inspect_subtree {
            unsafe { inspect(/* child SEXP */, pre, deep, pvec); }
        }

        true  // Return true if inspection succeeded
    }
}

When to use:

  • Debugging complex ALTREP structures
  • Showing internal state in .Internal(inspect())
  • Documenting ALTREP design for users

When to skip:

  • Most use cases (R’s default inspection is fine)
  • Production code (debugging feature)

🔗Duplicate - Custom Object Duplication

The duplicate() and duplicate_ex() methods customize how R duplicates your ALTREP object when copy-on-write semantics require it.

impl Altrep for LazyWithCache {
    const HAS_DUPLICATE: bool = true;

    fn duplicate(x: SEXP, deep: bool) -> SEXP {
        let handle = unsafe { altrep_data1_as::<LazyWithCache>(x) }?;
        let data = handle.borrow();

        if deep {
            // Deep copy: clone cached data too
            let new_data = LazyWithCache {
                params: data.params.clone(),
                cache: RefCell::new(data.cache.borrow().clone()),
            };
            new_data.into_sexp()
        } else {
            // Shallow copy: share cache (default R behavior)
            x  // Return self
        }
    }
}

When to use:

  • Controlling what gets copied (cache vs params)
  • Optimizing duplication for large cached data
  • Implementing copy-on-write semantics
  • Sharing immutable state across copies

When to skip:

  • Default R duplication is correct
  • No shared mutable state
  • No expensive cached data

Note: duplicate_ex() is the newer extended version - prefer it over duplicate() if implementing both.

🔗Coerce - Custom Type Conversion

The coerce() method customizes how R converts your ALTREP to other types (e.g., integer → real, real → integer).

impl Altrep for ArithSeq {
    const HAS_COERCE: bool = true;

    fn coerce(x: SEXP, to_type: SEXPTYPE) -> SEXP {
        use SEXPTYPE::*;

        let handle = unsafe { altrep_data1_as::<ArithSeq>(x) }?;
        let data = handle.borrow();

        match to_type {
            REALSXP => {
                // Convert integer sequence to real sequence
                // Instead of materializing, return a new Real ALTREP
                let real_seq = RealArithSeq {
                    start: data.start as f64,
                    step: data.step as f64,
                    len: data.len,
                };
                real_seq.into_sexp()
            }
            _ => {
                // Let R handle other conversions
                std::ptr::null_mut()
            }
        }
    }
}

When to use:

  • Converting between related ALTREP types (IntSeq → RealSeq)
  • Avoiding materialization during coercion
  • Preserving ALTREP properties after conversion
  • Optimizing common conversion paths

When to skip:

  • Default R coercion is acceptable
  • Conversion requires materialization anyway
  • Rare conversion path

Return values:

  • Return new SEXP: Your custom coercion
  • Return NULL (null_mut()): Let R use default coercion

🔗Materialization and DATAPTR

🔗Understanding Materialization

Materialization is the process of converting your lazy/compact ALTREP representation into a standard R vector with contiguous memory. This happens when R needs direct memory access to your data.

🔗When R Requests DATAPTR

R calls the dataptr() or dataptr_or_null() methods when:

  1. Operations requiring contiguous memory:

    • sort(), order(), unique()
    • .C() or .Fortran() calls passing the vector
    • as.vector() with specific types
    • Some vectorized operations (x + y, x * 2)

  2. Serialization (unless you provide serialize())

  3. Interop with other packages expecting raw pointers

🔗The Three Dataptr Strategies

🔗Strategy 1: No DATAPTR (Lazy Forever)

When to use: Pure lazy evaluation, external data sources, mathematical sequences

// Don't implement AltrepDataptr - only provide elt()

#[derive(miniextendr_api::AltrepInteger)]
#[altrep(manual, class = "LazySequence")]
pub struct LazySequence {
    start: i32,
    step: i32,
    len: usize,
}

impl AltrepLen for LazySequence {
    fn altrep_len(&self) -> usize {
        self.len
    }
}

impl AltIntegerData for LazySequence {
    fn elt(&self, i: usize) -> i32 {
        self.start + (i as i32) * self.step
    }
}

// No `dataptr` key, no explicit macro call — `#[altrep(manual)]` alone
// auto-emits the low-level bridge with a trivial materializing `AltrepDataptr`
// (always returns `None` until R itself forces materialization).

Behavior:

  • ✅ O(1) creation
  • ✅ O(1) element access
  • ❌ Operations needing DATAPTR will materialize to regular R vector
  • ❌ R owns the materialized copy (you lose control)

🔗Strategy 2: Materialization on Demand

When to use: Lazy until needed, then cache the materialized form

use miniextendr_api::altrep_data::AltrepDataptr;
use std::cell::RefCell;

#[derive(miniextendr_api::AltrepInteger)]
#[altrep(manual, dataptr, class = "LazyWithCache")]
pub struct LazyWithCache {
    // Computation parameters
    start: i32,
    step: i32,
    len: usize,

    // Materialized cache (initially None)
    materialized: RefCell<Option<Vec<i32>>>,
}

impl AltrepLen for LazyWithCache {
    fn altrep_len(&self) -> usize {
        self.len
    }
}

impl AltIntegerData for LazyWithCache {
    fn elt(&self, i: usize) -> i32 {
        self.start + (i as i32) * self.step
    }
}

impl AltrepDataptr<i32> for LazyWithCache {
    fn dataptr(&mut self, _writable: bool) -> Option<*mut i32> {
        // Materialize on first call
        let mut mat = self.materialized.borrow_mut();
        if mat.is_none() {
            let vec: Vec<i32> = (0..self.len)
                .map(|i| self.start + (i as i32) * self.step)
                .collect();
            *mat = Some(vec);
        }

        // Return pointer to cached data
        mat.as_mut().map(|v| v.as_mut_ptr())
    }

    fn dataptr_or_null(&self) -> Option<*const i32> {
        // Return None if not yet materialized (saves memory)
        self.materialized
            .borrow()
            .as_ref()
            .map(|v| v.as_ptr())
    }
}

// No explicit macro call — `#[altrep(manual, dataptr, ...)]` wires the
// low-level bridge to the `AltrepDataptr<i32>` impl above.

Behavior:

  • ✅ Lazy until DATAPTR requested
  • ✅ Subsequent DATAPTR calls are O(1)
  • ✅ You control the materialized form
  • ⚠️ Uses memory after materialization

🔗Strategy 3: Pre-Materialized (Vec/Box)

When to use: Data already in memory, just wrapping existing vector

#[derive(miniextendr_api::AltrepInteger)]
#[altrep(manual, dataptr, class = "VecWrapper")]
pub struct VecWrapper {
    data: Vec<i32>,
}

impl AltrepLen for VecWrapper {
    fn altrep_len(&self) -> usize {
        self.data.len()
    }
}

impl AltIntegerData for VecWrapper {
    fn elt(&self, i: usize) -> i32 {
        self.data[i]
    }
}

impl AltrepDataptr<i32> for VecWrapper {
    fn dataptr(&mut self, _writable: bool) -> Option<*mut i32> {
        Some(self.data.as_mut_ptr())
    }

    fn dataptr_or_null(&self) -> Option<*const i32> {
        Some(self.data.as_ptr())
    }
}

// No explicit macro call — the derive auto-emits the low-level bridge.

Behavior:

  • ✅ DATAPTR always available (O(1))
  • ✅ No lazy evaluation overhead
  • ❌ Memory used immediately
  • ❌ No computation savings

🔗Materialization Trade-offs

AspectNo DATAPTROn-DemandPre-Materialized
MemoryMinimalGrows on useFull upfront
SpeedFast elt()Fast after firstFastest DATAPTR
Use caseMath sequencesCachingExisting data
Lazy eval✅ Always✅ Until DATAPTR❌ Never

🔗When to Provide DATAPTR

Provide DATAPTR if:

  • ✅ Your data is already in memory (Vec, Box, slice)
  • ✅ Users will frequently perform operations requiring contiguous memory
  • ✅ You can efficiently materialize when needed
  • ✅ You want to control the materialization process

Skip DATAPTR if:

  • ✅ Data is external (database, file, network)
  • ✅ Pure mathematical sequence (no need to materialize)
  • ✅ Memory is at a premium
  • ✅ R’s default materialization is acceptable

🔗Safety Requirements

When implementing dataptr():

  1. Pointer Validity: Keep the allocation valid for callers using the returned pointer. Reallocating or replacing a Vec invalidates its old pointers; looking up the new pointer on a later callback does not repair an escaped C pointer.

  2. Lifetime: Keep the buffer owned by the ALTREP, either in its data1 ExternalPtr or in an R vector rooted through data2. A borrowed Cow does not itself keep an R source object alive.

  3. Mutability: If writable=true, the pointer must be mutable. R may modify the data.

// ❌ WRONG - pointer becomes invalid
fn dataptr(&mut self, _writable: bool) -> Option<*mut i32> {
    let vec = vec![1, 2, 3];
    Some(vec.as_mut_ptr())  // vec is dropped! Pointer is now invalid!
}

// ✅ CORRECT - pointer remains valid
fn dataptr(&mut self, _writable: bool) -> Option<*mut i32> {
    self.cached_data.as_mut().map(|v| v.as_mut_ptr())
}

🔗Bridge cache contract

The standard bridge reserves data2 for its R vector cache. Before materialization, callbacks read the Rust value in data1. Once an atomic vector is materialized, length, element access and region reads use data2, including writes made through DATAPTR. Sortedness and NA hints become unknown, and summaries and serialization return C NULL so R computes them from the current vector. This is different from R’s NULL value, which is not the fallback sentinel.

Direct native buffers (Altrep<Vec<i32>>, for example) keep their data in the Vec instead: fresh callbacks query its current allocation. Rust mutation through an explicit backing-data accessor must respect outstanding R/C pointers and R aliases. For a materialized class, mutating data1 does not replace the R cache. A shared growable container therefore needs an explicit snapshot or mutation policy; the ALTREP callbacks alone cannot make reallocation transparent.

Built-in Cow deserialization creates owned slices and strings. The temporary serialization state is not retained as a sidecar, so borrowing it would outlive its GC root.

🔗Example: Controlling Materialization

#[derive(miniextendr_api::AltrepInteger)]
#[altrep(manual, dataptr, class = "OptionallyMaterialized")]
pub struct OptionallyMaterialized {
    generator: Box<dyn Fn(usize) -> i32>,
    len: usize,
    cache: RefCell<Option<Vec<i32>>>,
}

// AltrepLen / AltIntegerData / AltrepDataptr<i32> impls as in
// "Strategy 2: Materialization on Demand" above.

impl OptionallyMaterialized {
    pub fn is_materialized(&self) -> bool {
        self.cache.borrow().is_some()
    }

    pub fn force_materialize(&mut self) {
        if self.cache.borrow().is_none() {
            let vec = (0..self.len).map(|i| (self.generator)(i)).collect();
            *self.cache.borrow_mut() = Some(vec);
        }
    }
}

Key Insight: Materialization is a one-way door. Once materialized, you typically stay materialized. Plan your memory strategy accordingly.

🔗Troubleshooting

🔗“Error: could not find function”

🔗Elements return wrong values

  • Check your elt() implementation
  • Verify index bounds handling

🔗R crashes on access

  • Ensure an ALTREP derive (#[derive(AltrepInteger)] etc., or the generic #[derive(Altrep)]) is on your data type
  • Check that into_sexp() is called to create the ALTREP object

🔗Serialization fails

  • Implement AltrepSerialize trait
  • Add serialize to the derive attribute (#[altrep(manual, serialize, ...)]), or the serialize option to impl_alt*_from_data! if you’re on the no_lowlevel escape hatch

🔗DATAPTR operations crash

  • Implement AltrepDataptr trait
  • Add dataptr to the derive attribute (#[altrep(manual, dataptr, ...)]), or the dataptr option to impl_alt*_from_data! if you’re on the no_lowlevel escape hatch
  • Ensure returned pointer is valid for the vector’s lifetime

🔗Native logical containers

Altrep<Vec<RLogical>>, Altrep<Box<[RLogical]>> and Altrep<Cow<'static, [RLogical]>> expose their existing native logical storage. RLogical has R’s i32 layout and represents FALSE, TRUE and NA:

use miniextendr_api::{Altrep, RLogical};

let values = Altrep(vec![RLogical::FALSE, RLogical::NA, RLogical::TRUE]);

Element reads, region reads and pointer access use the same storage, including raw nonzero logical values. Writes are visible to later reads and serialization. Cow::Borrowed stays borrowed during read-only access and detaches on the first writable pointer request. Deserialization always restores owned storage.

Use a truly static slice for borrowed storage, or ensure an R-backed borrow stays rooted for the entire ALTREP lifetime. The Cow itself does not root an R owner.

These classes retain the existing built-in duplication policy: R may materialize a regular vector when duplicating. Rust structural mutations must reacquire the current buffer; they cannot keep previously exposed pointers valid after growth.

🔗Ring-buffer containers

Altrep<VecDeque<T>> supports i32, f64, u8 and Rcomplex:

use std::collections::VecDeque;
use miniextendr_api::{Altrep, miniextendr};

#[miniextendr]
fn queued_values() -> Altrep<VecDeque<i32>> {
    Altrep(VecDeque::from(vec![10, 20, 30]))
}

The deque remains the owner. Element access follows its logical order, and region reads copy directly from its two segments without flattening it. Dataptr_or_null returns a pointer only when all elements occupy one segment. A Dataptr request calls make_contiguous() on the deque and exposes that buffer through a mutable reference; it does not convert the deque to a Vec. Writes through this pointer are visible to later reads and serialization.

As with the other built-in containers, ordinary R duplication uses R’s default policy and may materialize the replacement. A custom live-view or copy-on-write policy still belongs on an explicit ALTREP class. Structural Rust mutations must reacquire the current buffer; previously exposed pointers cannot survive arbitrary growth or ring rearrangement.

🔗Iterator-Backed ALTREP

miniextendr provides two iterator-backed ALTREP variants.

The *Data types below are data adaptors: they implement only the data-level traits (AltrepLen + the matching Alt*Data trait) and cannot back a live R vector by themselves (they do not implement RegisterAltrep). To expose one to R, wrap it in a concrete struct deriving the matching #[derive(Altrep*)] with #[altrep(manual)] and delegate the data-trait methods to the inner adaptor — see the wrapper walkthrough in SPARSE_ITERATOR_ALTREP.md for a complete example.

🔗Constructing from any iterator

AltrepIteratorExt is blanket-implemented for every Iterator. Its into_altrep_data_exact() constructor uses ExactSizeIterator::len(); into_altrep_data(n) declares a finite prefix without consuming the source. AltrepIteratorItem selects the data family for i32, f64, u8, Rcomplex, RLogical, bool and String. Option<T> also works when T: IsNA + AltrepIteratorItem: missing items become T::R_NA lazily. This includes integer, real, native logical and complex items, and excludes raw bytes. Local element types can select an existing coercing adapter or their own data adapter by implementing that trait. Because the item type picks the family, integer literals need a suffix (0..20i32).

use miniextendr_api::{AltrepIteratorExt, AltIntegerData};

let data = (0..20i32).map(|x| x * 2).into_altrep_data_exact();
assert_eq!(data.elt(3), 6);

Coverage follows capabilities rather than a list of iterator type names:

Iterator source or adapterConstruction
Exact-size collection iterators, arrays, supported finite ranges, and adapters retaining ExactSizeIteratorinto_altrep_data_exact()
filter, filter_map, flat_map, flatten, scan, take_while, chain, other unknown-size sourcesinto_altrep_data(n) with a known output prefix length
repeat, repeat_with, cycle, from_fn, successors, random-distribution samplesDeclare a finite prefix; construction never counts an unbounded source
itertools adaptersThe same bounds apply; miniextendr needs no itertools feature or dependency
Borrowing or draining iteratorsUsable locally; an R-owned class still needs an owned, 'static source
Tuples, chunks, results and custom itemsMap to a supported item, supply an item data-family implementation, or consume via RIterator for RefCell<I>

take(n) alone does not make every source implement ExactSizeIterator. DoubleEndedIterator does not imply cheap random access, and Clone does not promise an independently replayable source. Construction does not infer either property. The standard full-cache adapter treats the first None as permanent exhaustion. When the declared length is longer than the source, the numeric and logical iterator and stream adaptors read the elements the source never produced as R’s NA for the type (NA_integer_, NA_real_, NA_complex_, logical NA), never as an ordinary NaN; raw elements read 00. Full materialization (materialize_all(), and so every string adaptor read) panics instead.

String data still materializes all strings on the first element lookup to provide stable borrowed &str storage. Native numeric/logical data caches only the prefix reached so far. None of these constructors registers an R class or supplies a source-restoration policy.

A rand_distr sample iterator works the same way: Normal::new(0.0, 1.0)?.sample_iter(rng).into_altrep_data(n). Its cache gives repeated reads of an index the same value. To stream samples with no fixed length instead, read them through a connection: CONNECTIONS.md.

The iterator data adaptors retain the concrete iterator type. A registered class can use its capabilities and choose its own duplication and serialization policy. Keep that type when operations such as cloning, skipping or restoring the source depend on it.

Serializing a fully evaluated prefix preserves values, but restoring those values does not reconstruct a lazy iterator. A class that needs to stay lazy across serialization must encode enough source state to reconstruct it.

🔗IterState (Prefix Caching)

The default iterator state caches elements as a contiguous prefix. When you access element i, all elements 0..=i are generated and cached.

use miniextendr_api::altrep_data::IterIntData;

// Create from an iterator
let data = IterIntData::from_iter((0..1000).map(|x| x * 2), 1000);

// Access element 100 - generates and caches elements 0-100
let elem = data.elt(100);

// Access element 50 - returns from cache (no computation)
let elem = data.elt(50);

Characteristics:

  • Cache is contiguous Vec<T>
  • All elements up to max accessed index are cached
  • as_slice() available after full materialization
  • Memory usage: O(max_accessed_index)

🔗SparseIterState (Skipping)

For sparse access patterns, use the sparse variants that skip intermediate elements using Iterator::nth():

use miniextendr_api::altrep_data::SparseIterIntData;

// Create from an iterator
let data = SparseIterIntData::from_iter((0..1_000_000).map(|x| x * 2), 1_000_000);

// Access element 999_999 - skips directly there
let elem = data.elt(999_999);  // Only this element is cached

// Element 0 was skipped and is now inaccessible
let first = data.elt(0);  // Returns NA_INTEGER

Characteristics:

  • Cache is sparse BTreeMap<usize, T>
  • Only accessed elements are cached
  • Skipped elements return NA/default forever
  • The first None permanently ends the source, even if it is not fused
  • Completing the declared prefix drops the source without an extra poll
  • as_slice() always returns None
  • Memory usage: O(num_accessed)

🔗Comparison

FeatureIterStateSparseIterState
Cache storageContiguous Vec<T>Sparse BTreeMap<usize, T>
Access patternPrefix (0..=i) cachedOnly accessed indices cached
Skipped elementsAll cachedGone forever (return NA)
Memory for sparse accessO(max_index)O(num_accessed)
as_slice() supportYes (after full materialization)No

🔗Available Types

Prefix caching (IterState):

  • IterIntData<I> - Integer vectors
  • IterRealData<I> - Real (f64) vectors
  • IterLogicalData<I> - Logical (bool) vectors
  • IterRawData<I> - Raw (u8) vectors
  • IterStringData<I> - Character vectors (forces full materialization)
  • IterComplexData<I> - Complex number vectors
  • IterListData<I> - List vectors (SEXP elements)
  • IterIntCoerceData<I, T> - Integer with coercion from other types
  • IterRealCoerceData<I, T> - Real with coercion from other types

Sparse/skipping (SparseIterState):

  • SparseIterIntData<I> - Integer vectors
  • SparseIterRealData<I> - Real (f64) vectors
  • SparseIterLogicalData<I> - Logical (bool) vectors
  • SparseIterRawData<I> - Raw (u8) vectors
  • SparseIterComplexData<I> - Complex number vectors

🔗When to Use Which

Use IterState (prefix caching) when:

  • Access is mostly sequential (0, 1, 2, …)
  • You’ll eventually access most/all elements
  • You need as_slice() or full materialization later

Use SparseIterState (skipping) when:

  • Access is sparse and moves forward (e.g., sorted sampling)
  • Vector is very large but you only need a few elements
  • You don’t need skipped elements ever again
  • Memory is constrained

🔗Serializing a concrete iterator cache

The atomic and string Iter*Data<I> adapters implement AltrepSerialize. Saving reads the whole declared sequence; restoring builds an adapter of the same concrete type that has no iterator and serves the saved values. I needs neither Default nor a trait object. With I: Clone, the adapters implement Clone, which copies the cached prefix and the remaining iterator without advancing either.

A registered class picks its policies with the derive options:

#[derive(Clone, miniextendr_api::AltrepInteger)]
#[altrep(manual, serialize, duplicate = "clone", class = "RangeCache")]
struct RangeCache(IterIntData<std::ops::Range<i32>>);

impl AltrepLen for RangeCache {
    fn altrep_len(&self) -> usize { self.0.altrep_len() }
}
impl AltIntegerData for RangeCache {
    fn elt(&self, i: usize) -> i32 { self.0.elt(i) }
}
impl AltrepSerialize for RangeCache {
    fn serialized_state(&self) -> SEXP { self.0.serialized_state() }
    fn unserialize(state: SEXP) -> Option<Self> {
        IterIntData::unserialize(state).map(Self)
    }
}

After R writes into the vector, the bridge serializes R’s values, not the iterator’s cache. A source that ends before the declared length has no complete state to save: the adapter then declines, and R saves the values the element reads give. String adapters materialize on first access, to hand out stable borrowed strings.

IterListData keeps the elements it has yielded in one R list, allocated on its first in-bounds read. Each yielded SEXP goes into that list before the iterator runs again, so a mapping closure may return freshly allocated R objects. The iterator must still root any objects it keeps between calls. The adapter roots the list itself, in the same pool as owning ExternalPtr handles, so it is safe before the adapter belongs to an ALTREP class, and releases it on drop. That pool releases in O(1) in any order. R’s finalizers drop ALTREP owners newest first, and R_ReleaseObject scans its list from the newest root. So R_PreserveObject roots for caches read in another order would take quadratic time to release: 8 s instead of 0.03 s for 50,000 lists read in reverse. Build, read, clone and drop the adapter on R’s main thread (!Send + !Sync).

A list adapter’s Clone copies its cache list and clones the remaining iterator without advancing it. The copies share their elements, as R’s shallow_duplicate() does: R copies an element that two lists hold before changing it. Its AltrepSerialize saves the realized list; unserialize keeps R’s restored list as the cache, without constructing I. Unlike an atomic adapter, which has no value to save for elements a short source never produced, a list leaves those slots NULL, which is also what its element reads return, so the list is saved as it is. A concrete list class delegates the same three data traits as above, using AltrepList, AltListData and IterListData.

Back to Manual