Data Frame Conversion in miniextendr

miniextendr converts between Rust types and R data frames through a DataFrame view, an owned BuiltDataFrame handle, and two conversion verbs that mirror the scalar/vector surface (IntoR and TryFromSexp). One verb builds a rooted data frame from rows; the other reads an R-owned frame back into rows. Both verbs live on the data, errors are a single DataFrameError, and missing cells round-trip as nullable fields.

29 min read 5672 words

miniextendr converts between Rust types and R data frames through a DataFrame view, an owned BuiltDataFrame handle, and two conversion verbs that mirror the scalar/vector surface (IntoR and TryFromSexp). One verb builds a rooted data frame from rows; the other reads an R-owned frame back into rows. Both verbs live on the data, errors are a single DataFrameError, and missing cells round-trip as nullable fields.

🔗Owned named lists and mixed columns

NamedListBuilder builds a named list from mixed Rust values and existing R values. Its growing R list owns one precious-list root (R_PreserveObject), so it can be moved, stored and dropped independently of other builders without holding protect-stack entries. Use it on R’s main thread; it is !Send + !Sync.

use miniextendr_api::NamedListBuilder;

let mut columns = NamedListBuilder::with_capacity(2);
columns.push("id", vec![1, 2, 3]);
columns.push("estimate", vec![0.5, 1.5, 2.5]);
columns.rename(1, "slope"); // names_mut() also permits editing in place
let frame = columns.into_data_frame()?; // owned BuiltDataFrame

Use push_sexp(name, sexp) for an existing R value: the builder roots it, keeping classes such as factor and Date intact. Values must be live when passed in, but need no separate root after the push. len() counts entries.

Return the builder directly from a #[miniextendr] function for a named list via IntoR. To keep working with the list before returning it, finish with into_list(): the BuiltList holds its own root like BuiltDataFrame, dereferences to the List view, and its set_class(&[..]) / set_attr(name, value) allocate while the list stays rooted, so it can be classed, passed to R code and read back first:

let out = columns.into_list().set_class(&["my_result", "list"]);

There is no bare-List finish. into_data_frame() validates all column lengths against the first, reports every mismatch in one error, and sets compact row names. An empty builder produces names = character(0) for every finish; its zero-column frame is identical to data.frame().

🔗The two verbs

Trait	Method	Direction	Scalar analogue
`IntoDataFrame`	`rows.into_dataframe()? -> BuiltDataFrame`	Rust → R	`IntoR`
`FromDataFrame`	`Vec::<Row>::from_dataframe(&df)? -> Vec<Row>`	R → Rust	`TryFromSexp`

The verbs are implemented on the data (Vec<Row> / &DataFrame), not on a companion type. There is one error type (DataFrameError) and one NA contract: a missing cell maps to None in an Option<T> field, and None maps back to NA.

For the full design rationale and the orphan-rule mechanics behind the blanket impls, see DATAFRAME_INTERFACE.md.

🔗Quick start

#[derive(DataFrameRow)] is the primary path: derive it on a row struct, then return BuiltDataFrame from a #[miniextendr] function.

use miniextendr_api::{
    BuiltDataFrame, DataFrame, DataFrameRow, FromDataFrame, IntoDataFrame, miniextendr,
};

#[derive(Clone, DataFrameRow)]
struct Measurement {
    time: f64,
    value: f64,
    sensor: String,
}

#[miniextendr]
fn get_measurements() -> BuiltDataFrame {
    let rows = vec![
        Measurement { time: 1.0, value: 10.0, sensor: "A".into() },
        Measurement { time: 2.0, value: 20.0, sensor: "B".into() },
        Measurement { time: 3.0, value: 30.0, sensor: "C".into() },
    ];
    rows.into_dataframe().unwrap()
}

The reverse direction reads a DataFrame argument back into rows:

#[miniextendr]
fn round_trip(df: DataFrame) -> BuiltDataFrame {
    let rows: Vec<Measurement> = Vec::<Measurement>::from_dataframe(&df).unwrap();
    rows.into_dataframe().unwrap()
}

DataFrame implements TryFromSexp, so it is the inbound view type. BuiltDataFrame implements IntoR, so a Rust-created frame is handed back to R without exposing an unrooted intermediate. For an optional frame, take Option<DataFrame>: NULL is None, and a data frame is Some of the same view (its class, attributes, row names and NA cells untouched), refused in the same words as a DataFrame parameter ('df' must be NULL or a data frame: got numeric). Option<Either<L, DataFrame>> (feature either) works the same way around an Either.

🔗The `DataFrame` view and `BuiltDataFrame` handle

DataFrame is a validated, cheap Copy view over a bare SEXP. It has no GC root of its own and is sound only while R’s .Call frame, a ProtectScope, or another owning handle keeps the object reachable. BuiltDataFrame owns a precious-list root and dereferences to the view. Every Rust-side constructor returns the handle.

The new-frame producers (drop, select, select_rows, prepend_column, with_column) also return BuiltDataFrame, so constructor → edit chains are rooted at every link and safe across R allocations (#1247). rename / strip_prefix edit the names attribute in place; on a BuiltDataFrame receiver the handle carries through the chain.

select_rows(&[usize]) (0-based row indices, in output order; also what group_by sub-frames are cut with) takes a row of every column, as df[idx, , drop = FALSE] does. A matrix column (I(matrix(...))) is subset along its first dimension, with its row names, and a packed data.frame column is subset recursively; arrays of three or more dimensions are sliced along their first dimension too (vctrs::vec_slice(); base [ subsets those by element). Each column keeps every other attribute, the vctrs::vec_slice() rule. So a POSIXct column keeps its tzone, a difftime its units, and a plain column its label. Base [ drops the last one. Element names are subset with the values.

let df: BuiltDataFrame = rows.into_dataframe()?;

df.nrow();                       // row count
df.ncol();                       // column count
df.names();                      // Vec<String> of column names
df.contains_column("sensor");    // bool

let values: Vec<f64> = df.column("value").unwrap();   // typed column accessor
let raw: SEXP = df.column_raw("sensor").unwrap();      // untyped column SEXP

let df = df
    .rename("value", "reading")  // rename a column (in place)
    .drop("time")                // remove a column
    .select(&["sensor", "reading"]); // keep/reorder a subset — still a rooted BuiltDataFrame

Use DataFrame::from_sexp(sexp) to validate an arbitrary SEXP, and as_sexp() / as_list() to drop down to the raw representation when you need it.

🔗Frame attributes

Every new-frame producer keeps the attributes of the frame itself, not only those of its columns. That is everything except names and row.names: the class, a units map that describes some columns, a package’s metadata. This is the rule of dplyr_reconstruct(), vctrs::vec_slice() and tibble’s [. select_rows gives the result fresh compact row.names (1:length(idx)), where base [ and vctrs::vec_slice() subset character row names (#1703). The column producers (drop, select, prepend_column, with_column) keep the row names they were given.

dplyr’s grouping is the exception. A grouped_df or rowwise_df caches its groups in a groups attribute. That attribute holds each group’s row indices and names the key columns, so it is only copied while it still fits the new frame:

Producer	dplyr `grouped_df` / `rowwise_df` input
`select_rows`	always returned ungrouped: the selected rows no longer match the cached row indices
`drop`, `select`, `prepend_column`	grouping kept while every grouping column stays unchanged; ungrouped when one is removed or replaced
`with_column`, appending	grouping kept (a new column is never a grouping column)

“Ungrouped” means the groups attribute and the grouped_df / rowwise_df class are gone. A dplyr input comes back as a tibble that keeps its other attributes, never as a grouped frame whose groups are stale. dplyr itself recomputes the groups in dplyr::slice(), and its [ regroups by the grouping columns that remain (#1701). To keep a row subset grouped, read group_declaration() from the input and regroup the result by its vars: with group_by_multi in Rust, or with dplyr::grouped_df(out, dplyr::group_vars(df)) in R.

Any other attribute indexed by row is copied as it is. A package that keeps one has to rebuild it after a row subset, as ?dplyr_row_slice asks of its methods. The in-place edits (rename, strip_prefix, the replace path of with_column) change the frame they are given and leave its attributes alone, groups included, even when they rename or overwrite a grouping column (#1702).

🔗`#[derive(DataFrameRow)]` in depth

🔗Heterogeneous types

The derive supports different types in different fields; each field keeps its R type:

#[derive(Clone, DataFrameRow)]
struct Person {
    name: String,      // character in R
    age: i32,          // integer in R
    height: f64,       // numeric in R
    is_student: bool,  // logical in R
}

🔗Collection expansion

Fixed-size arrays [T; N] are automatically expanded into N suffixed columns. #[dataframe(expand)] / #[dataframe(unnest)] request it explicitly, though arrays expand by default.

#[derive(Clone, DataFrameRow)]
struct Point3D {
    label: String,
    coords: [f64; 3],  // → coords_1, coords_2, coords_3
}

For Vec<T>, Box<[T]>, and &[T], two expansion modes are available:

Fixed width (width = N): expands into exactly N columns at compile time.

#[derive(Clone, DataFrameRow)]
struct Scored {
    name: String,
    #[dataframe(width = 3)]
    scores: Vec<f64>,  // → scores_1, scores_2, scores_3 as Option<f64>
}

Shorter vecs: padded with NA.
Longer vecs: truncated to N (extra elements silently dropped).

Auto-expand (expand or unnest): column count determined at runtime from the maximum length across all rows.

#[derive(Clone, DataFrameRow)]
struct Measured {
    name: String,
    #[dataframe(expand)]       // or: #[dataframe(unnest)]
    readings: Vec<f64>,        // → readings_1, readings_2, ... (as many as needed)
}

Shorter vecs: padded with NA.
All elements preserved (no truncation).
If all vecs are empty: no expansion columns produced.

Box<[T]> and &[T] work identically to Vec<T> for all expansion modes. Without width or expand/unnest, Vec<T>, Box<[T]>, and &[T] stay as opaque single columns (list columns in R).

Borrowed fields keep their lifetime parameters on the companion (for example, FooDataFrame<'a>). A borrowed fixed array (&[T; N]) or a slice with width clones its elements into owned columns, so T: Clone is required. A zero-sized marker preserves any lifetime no longer present in those columns. A slice with expand stays borrowed in the companion and clones elements when the R columns are built. This applies to struct and enum rows; borrowed fields have no from-R reader.

🔗Field-level attributes

#[derive(Clone, DataFrameRow)]
struct Row {
    #[dataframe(skip)]           // Omit from data frame
    internal_id: u64,

    #[dataframe(rename = "lbl")] // Custom column name
    label: String,

    #[dataframe(as_list)]        // Suppress expansion (keep as single column)
    coords: [f64; 3],

    #[dataframe(width = 5)]      // Expand Vec to 5 columns
    scores: Vec<f64>,
}

Attribute	Effect	Valid On
`skip`	Omit field from data frame	Any field
`rename = "name"`	Custom column name	Any field
`as_list`	Suppress expansion (keep one list-column)	`[T; N]`, `Vec<T>`, `Box<[T]>`, `&[T]`, maps, wrapper types; a struct or enum type that implements `IntoList`
`expand`	Explicit expansion (default for `[T; N]`; auto-expand for `Vec<T>`/`Box<[T]>`/`&[T]`)	`[T; N]`, `Vec<T>`, `Box<[T]>`, `&[T]`
`unnest`	Alias for `expand`	`[T; N]`, `Vec<T>`, `Box<[T]>`, `&[T]`
`width = N`	Pin expansion width (truncates longer vecs/slices)	`Vec<T>`, `Box<[T]>`, `&[T]`

Conflicts: as_list + expand/unnest, as_list + width are compile errors.

Round-tripping: fixed-width and auto-expanded fields are read by regrouping their suffixed columns. Trailing NA padding is removed, so a ragged Vec<T> round-trips to its original length. Borrowed fields (&[T], &[T; N]) and explicitly opaque as_list fields have no reader because owned R data cannot recreate the borrow or infer the opaque element shape.

🔗Other collection types

Non-expanded collection fields work natively for both struct and enum DataFrameRows:

use std::collections::{HashSet, BTreeSet};

#[derive(Clone, DataFrameRow)]
struct ComplexRow {
    measurements: Vec<f64>,      // opaque list column
    data: Box<[i32]>,            // opaque list column
    tags: HashSet<String>,       // opaque list column
    categories: BTreeSet<i32>,   // opaque list column
}

In struct DataFrameRows the columns land as Vec<C> and convert to a VECSXP list-column. In enum DataFrameRows they land as Vec<Option<C>> with None for variants that don’t carry the field — these convert to a VECSXP list-column with NULL for absent rows. See CONVERSION_MATRIX.md for the full set of supported C.

Each list-column cell is the R vector the field’s own value converts to: a Vec<T> / Box<[T]> / [T; N] field works for every element type with a Vec<T>: IntoR conversion, and a borrowed &[T] field for every T with &[T]: IntoR. The wide integers (usize, u64, i64, isize, u32) choose per row, as a returned Vec<usize> does: a cell is an integer vector when every value fits in R’s integer range and a double vector otherwise, so one column can hold both, and an empty row is integer(0):

#[derive(Clone, DataFrameRow)]
struct Row {
    #[dataframe(as_list)]
    starts: Vec<usize>,
}
// rows [1, 5], [], [3_000_000_000]
// R: df$starts is list(c(1L, 5L), integer(0), 3e9)

List columns have no strict mode. When an out-of-range value must be an error, return the rows from a #[miniextendr(strict)] function as Vec<Vec<usize>> (see STRICT_MODE.md).

HashMap<K, V> / BTreeMap<K, V> variant fields expand to two parallel list-columns (see Map fields below). Struct-typed and nested-enum variant fields are covered in Nested enum fields below.

🔗Map fields: parallel list-column expansion

HashMap<K, V> and BTreeMap<K, V> fields on enum variants expand to two parallel list-columns named <field>_keys and <field>_values. Each cell holds a vector of K and a vector of V respectively, in the same entry order:

#[derive(Clone, DataFrameRow)]
#[dataframe(align, tag = "_type")]
enum Event {
    Tally { label: String, tally: BTreeMap<String, i32> },
    Empty { label: String },
}
// In R (BTreeMap, sorted key order):
//   _type   label  tally_keys    tally_values
//   Tally   "a"    c("a","b")    c(1L, 2L)
//   Empty   "b"    NULL          NULL

Absent-variant rows produce NULL in both columns (not NA). An empty map produces character(0) / integer(0), not NULL.

HashMap ordering: HashMap iteration order is non-deterministic. Keys and values are parallel within a single row, but the key order may differ across rows and runs. Use setequal or sort-based comparison in R tests, never expect_equal on unsorted key vectors.

BTreeMap ordering: keys are always in sorted order per the BTreeMap contract. expect_equal is safe.

as_list opt-out: annotate the field with #[dataframe(as_list)] to keep it as a single opaque named-list column (the pre-expansion behavior). Only use this when the named-list per-row shape is needed directly in R.

Detection caveats: classify_field_type detects HashMap / BTreeMap by matching the last path segment (HashMap or BTreeMap) and requiring exactly two generic type arguments. It also detects struct-typed fields by matching bare path types (single- or multi-segment, e.g. Point or crate::geom::Point) whose last segment has no generic arguments.

🔗Rejected wrapper types

The following shapes produce a compile error (since #484) because they cannot be automatically expanded and would otherwise silently produce a confusing opaque list-column:

Option<T> for a non-scalar T — Option<HashMap<K,V>>, Option<UserStruct>, Option<Vec<T>>, etc. (Option<f64>, Option<String> and the other scalar payloads are the NA contract and are accepted; see Missing data)
Cow<T>, Rc<T>, Arc<T>, RefCell<T>, Cell<T>, Mutex<T>, RwLock<T>

For all of these, use #[dataframe(as_list)] to opt into an explicit opaque list-column, or unwrap to the inner type (e.g. store HashMap<K,V> directly and use an empty map for the absent case):

#[derive(Clone, DataFrameRow)]
struct Row {
    id: i32,
    // `counts: Option<HashMap<String, i32>>` → compile error without `as_list`.
    #[dataframe(as_list)]
    counts: Option<HashMap<String, i32>>,
}

Type aliases are not automatically unwrapped — type Counts = HashMap<String, i32>; field: Counts has Counts as the last segment, so map expansion is not triggered. Use the concrete type directly (field: HashMap<String, i32>), or annotate with #[dataframe(as_list)]. See #604.

Multi-segment paths whose last segment does NOT implement DataFrameRow (e.g. std::ffi::CString) produce a clear compile-time error from the _assert_inner_is_dataframe_row assertion — this is intentional. #[dataframe(as_list)] does not help there unless the type implements IntoList (see List-columns of struct or enum values), which CString doesn’t: store a String instead, or wrap the value in a newtype that implements IntoList.

🔗Nested enum fields: flatten and opt-outs

A variant field whose type is itself a DataFrameRow enum flattens into prefixed columns by default. The inner enum must #[derive(DataFrameRow)]; the outer field’s name acts as a prefix. The inner enum should use #[dataframe(tag = "variant")] so that its discriminant column merges cleanly as <field>_variant:

#[derive(Clone, DataFrameRow)]
#[dataframe(align, tag = "variant")]  // inner enum's own discriminant is "variant"
enum Status { Ok, Err { code: i32 } }

#[derive(Clone, DataFrameRow)]
#[dataframe(align, tag = "_type")]
enum Event {
    Tracked { id: i32, status: Status },
    Other   { id: i32 },
}
// Columns in R:
//   _type          character ("Tracked" / "Other")
//   id             integer
//   status_variant character ("Ok" / "Err" / NA for Other rows)
//   status_code    integer   (NA for Ok rows and Other rows; error code for Err rows)

Absent-variant rows (e.g. Other above, which has no status field) produce NA in all prefixed columns.

Inner tag naming: use #[dataframe(tag = "variant")] on the inner enum — the outer prefix then produces <field>_variant (single underscore). Using #[dataframe(tag = "_variant")] (with leading underscore) produces <field>__variant (double underscore). Avoid leading underscores on inner tags.

🔗`as_factor` — unit-only inner enum

When the inner enum has only unit variants (no payload), annotate the field with #[dataframe(as_factor)] to emit a single R factor column instead of flattening. The inner enum does not need DataFrameRow for this path — only UnitEnumFactor, which is auto-emitted by #[derive(DataFrameRow)] for unit-only enums:

#[derive(Clone, Copy, DataFrameRow)]
#[dataframe(tag = "variant")]
enum Direction { North, South, East, West }

#[derive(Clone, DataFrameRow)]
#[dataframe(align, tag = "_type")]
enum Move {
    Step { id: i32, #[dataframe(as_factor)] dir: Direction },
    Stop { id: i32 },
}
// R column: dir — integer factor with levels c("North","South","East","West")
// Stop rows have NA in dir.

The same attribute works on named and tuple struct fields, including rename. For example:

#[derive(DataFrameRow)]
struct Step {
    id: i32,
    #[dataframe(as_factor, rename = "direction")]
    dir: Direction,
}
// Columns: id (integer), direction (factor), with no direction_variant column.

The companion stores the enum values themselves, so the enum needs neither Clone nor Copy. A struct’s factor column has no missing rows, so its reader checks the factor’s levels and reports every NA or invalid code in one error. The parallel reader delegates to the sequential one, which moves the values, as enum readers do for flattened and factor fields. An empty column still carries every level. Tuple structs and #[dataframe(tag)] structs stay write-only, as for every field type. A struct with a factor field can itself be flattened into an enum variant; rows of the other variants get NA in its factor column.

Factor levels are the variant idents in declaration order. is.factor(df$dir) returns TRUE. Annotating a payload-bearing enum with as_factor is a compile error (missing UnitEnumFactor implementation).

Generic unit enums: #[derive(DataFrameRow)] rejects enums with type or const generic parameters, so it never emits UnitEnumFactor for one. A generic unit enum must implement UnitEnumFactor by hand if as_factor is needed.

🔗`as_list` — opaque list-column

Use #[dataframe(as_list)] to keep an inner enum as a single opaque VECSXP list-column. Each present row gets a list cell; absent-variant rows get NULL:

enum Event {
    Move { id: i32, #[dataframe(as_list)] dir: Direction },
    Stop { id: i32 },
}
// R column: dir — list-column; Move rows have a list cell, Stop rows have NULL.

The cell is the value’s IntoList list, so the inner type must implement IntoList; see List-columns of struct or enum values.

🔗`<field>_variant` collision detection

When a field kind: Inner is flattened, the macro detects a compile-time collision if any sibling field in the same variant produces a column named kind_variant (the name that the inner enum’s discriminant column will receive after prefixing). Rename the colliding field or change the inner enum’s tag:

// ERROR: kind_variant is both the flatten discriminant and a sibling field name.
enum Bad {
    Wrap { kind: Inner, kind_variant: String },
}

// OK: rename sibling field, or change inner tag.
enum Good {
    Wrap { kind: Inner, #[dataframe(rename = "kind_type")] kind_type: String },
}

🔗Enum align mode

Enums build a unified schema where each variant’s fields contribute columns. Fields absent in a variant are filled with None (→ NA in R):

#[derive(Clone, DataFrameRow)]
#[dataframe(tag = "_type")]
enum Event {
    Click { id: i64, x: f64, y: f64 },
    Impression { id: i64, slot: String },
    Error { id: i64, code: i32, message: String },
}

#[miniextendr]
fn get_events() -> BuiltDataFrame {
    let rows = vec![
        Event::Click { id: 1, x: 1.5, y: 2.5 },
        Event::Impression { id: 2, slot: "top_banner".into() },
        Event::Error { id: 3, code: 404, message: "not found".into() },
    ];
    rows.into_dataframe().unwrap()
}

// In R:
//   _type       id    x     y   slot        code  message
//   Click       1     1.5   2.5 NA          NA    NA
//   Impression  2     NA    NA  top_banner  NA    NA
//   Error       3     NA    NA  NA          404   not found

Key points:

All enum columns are Vec<Option<T>> (absent fields get None).
tag = "col" adds a variant discriminator column.
align is implicit for enums (accepted but not required).
Borrowed fields (&'a str, &'a [T]) work in enum variants — the same lifetime propagates through the companion struct. Explicit lifetime params on #[miniextendr] fns/impls are allowed too (only type and const generics are rejected).

🔗Type conflicts across variants

If two variants use the same field name with different types, the derive fails by default. Use conflicts = "string" to coerce all conflicting columns to String:

#[derive(Clone, DataFrameRow)]
#[dataframe(conflicts = "string")]
enum Mixed {
    A { value: f64 },
    B { value: String },  // value column becomes String for all variants
}

🔗Enum field attributes

All field-level attributes (skip, rename, as_list, width) work in enum variants too:

#[derive(Clone, DataFrameRow)]
#[dataframe(tag = "_type")]
enum Observation {
    Point { id: i32, coords: [f64; 2] },          // coords → coords_1, coords_2
    Measurement { id: i32, #[dataframe(width = 3)] readings: Vec<f64> },
}

🔗Enum split mode (`into_dataframe_split`)

Alongside the aligned form (into_dataframe(), which produces a single data frame with NA/NULL fill for variants that don’t carry a field), enums also get rows.into_dataframe_split() — the IntoDataFrameSplit trait, re-exported from miniextendr_api::prelude next to IntoDataFrame/FromDataFrame — which partitions the rows by variant. Each partition is a data frame with only that variant’s own columns — no NA-filled columns from sibling variants. Struct rows have no variants to partition, so the trait is implemented only for enum-derived Vec<Row>. It returns a DataFrameShape (the same type the serde vec_to_dataframe_split returns), so a #[miniextendr] function returns DataFrameShape:

Variants × rows in input	`DataFrameShape` variant	R return
Single-variant enum, any number of rows	`Bare(BuiltDataFrame)`	bare `data.frame`
Multi-variant enum, mixed rows	`PerVariantList(Vec<(String, BuiltDataFrame)>)`	named `list` of data frames, one per variant in `snake_case`

Every partition is a rooted BuiltDataFrame, so the result can be held across R allocations (building another frame, calling back into R) before it is returned; it is not a convert-immediately view (#1750). Match on the variant to read a partition from Rust.

use miniextendr_api::{DataFrameShape, IntoDataFrameSplit};

#[miniextendr]
fn split_events() -> DataFrameShape {
    let rows = vec![
        Event::Click { id: 1, x: 1.5, y: 2.5 },
        Event::Impression { id: 2, slot: "top_banner".into() },
        Event::Error { id: 3, code: 404, message: "not found".into() },
    ];
    rows.into_dataframe_split()
    // In R: list(click = <1-row df with id, x, y>,
    //            impression = <1-row df with id, slot>,
    //            error = <1-row df with id, code, message>)
}

Variants absent from the input still appear in the result as 0-row data frames carrying that variant’s column shape. Unit variants produce a 0-column data frame with the correct row count. Tuple variants name positional columns _0, _1, … . See the cardinality matrix in rpkg/tests/testthat/test-dataframe-enum-payload-matrix.R for the full set of guarantees (PR #463).

🔗Container attributes

#[derive(DataFrameRow)]
#[dataframe(
    name = "Measurements",     // Custom companion type name (default: {StructName}DataFrame)
    tag = "_type",             // Add variant discriminator column (enums)
    align,                     // Unified schema with NA fill (implicit for enums)
    conflicts = "string",      // Coerce type conflicts to String (enums)
)]
struct Measurement { /* ... */ }

🔗Requirements

A row needs no trait besides the derive. Of each field type the derive asks only what turning that field into columns needs, and a missing impl is reported at the field:

Field	What its type needs
scalar or opaque column (`i32`, `String`, `Vec<T>`, `HashMap<K, V>`, …)	its column converts to R (`Vec<T>: IntoR`; in an enum, `Vec<Option<T>>`)
`[T; N]`, or `width` / `expand` over an owned `Vec<T>` / `Box<[T]>`	`T`’s columns convert to R; the elements are moved, so `T` needs no `Clone` or `Copy`
the same over a borrowed `&[T; N]` / `&[T]`	also `T: Clone`: each element is cloned out of the borrow
flattened struct, nested enum	`#[derive(DataFrameRow)]`; no `Clone`: values are moved into their columns, and out of them when an enum is read back
`as_factor`	`UnitEnumFactor`, which `#[derive(DataFrameRow)]` on a unit-only enum and `#[derive(RFactor)]` both provide; no `Clone` or `Copy`
`as_list`	`IntoList` (see below)
`skip`	nothing

The rest of what the derive generates exists only where the field types allow it, so it never adds a requirement:

The {Row}DataFrame companion is Debug / Clone when every column type is (for a Vec<T> column, when T is).
A named-field struct companion without expansion iterates its rows (IntoIterator, ColumnarFrame::into_rows) when every skip field’s type is Default: the skipped fields are rebuilt with Default::default().
With rayon, from_rows_par / into_dataframe_par fill the columns in parallel when the row type is Send and so are the column values, a struct’s placeholder-filled columns (scalar and [T; N] columns) are Default, and flattened struct and as_list values are Clone (they are copied out before the rows move to other threads). Otherwise the same call fills the columns sequentially.

Derive or write IntoList for the row only if you also return one row to R as a list, and then any impl works, derived or hand-written.

🔗List-columns of struct or enum values

A field whose type is a plain struct or enum path (Point, geom::Point, Direction) and carries #[dataframe(as_list)] becomes a list-column whose cells are the values’ IntoList lists. That type must implement IntoList, and a missing impl is reported at the field’s type:

a struct: #[derive(IntoList)], or a hand-written impl;
a unit-only enum: its own #[derive(DataFrameRow)] also emits IntoR (a one-element factor) and IntoList;
a payload enum: a hand-written impl IntoList (#[derive(IntoList)] takes structs only).

#[derive(Clone, IntoList)]
struct Point { x: f64, y: f64 }

#[derive(Clone, DataFrameRow)]
struct Row {
    id: i32,
    #[dataframe(as_list)]
    origin: Point, // df$origin is list(list(x = 1, y = 2), ...)
}

🔗Reading data frames back (`FromDataFrame`)

Vec::<Row>::from_dataframe(&df)? transposes columns back into rows:

let rows: Vec<Measurement> = Vec::<Measurement>::from_dataframe(&df)?;
for m in &rows {
    println!("time {} value {}", m.time, m.value);
}

The reader covers scalar fields, as_factor fields, fixed/auto expansion, nested struct flattening, tagged enums (including flattened payload enums and as_factor unit enums), and enum map columns. Shapes that intentionally discard reconstruction information still return DataFrameError::Conversion: borrowed or skip fields, as_list and other opaque non-scalar collection columns, #[dataframe(tag)] structs, conflict-string enums, and tagless/skip-field enums. See The DataFrame interface for the exact contract.

🔗Group-level iteration

Two rungs, cheapest first.

🔗Typed rows: `group_rows`

Once rows are extracted, grouping is plain Rust. group_rows makes the idiom discoverable and gives NA-able keys a defined home (key on Option<T> — None sorts first — or a custom enum):

let rows: Vec<Measurement> = Vec::<Measurement>::from_dataframe(&df)?;
let by_sensor: BTreeMap<String, Vec<Measurement>> =
    group_rows(rows, |m| m.sensor.clone());

The result is plain Rust data — no SEXP contact — so stepping over groups with rayon is safe:

let summaries: Vec<(String, f64)> = by_sensor
    .into_iter()
    .par_bridge()                       // rayon: rows are Send, no SEXPs held
    .map(|(sensor, ms)| (sensor, ms.iter().map(|m| m.value).sum()))
    .collect();

🔗Untyped frames: `DataFrame::group_by`

For heterogeneous frames where extracting typed rows is wasteful, group_by(col) computes group indices in one pass (main thread, no row copies) and returns a GroupedDataFrame:

let grouped = df.group_by("site")?;

// (key, row-indices) — zero-copy
for (key, rows) in grouped.iter() { /* key: &GroupKey, rows: &[usize] */ }

// per-group sub-frames — root each one as you go (push protects it)
let mut out = NamedListBuilder::with_capacity(grouped.len());
for (key, sub) in grouped.frames() {
    out.push(key.label(), sub);
}
let named_list_of_frames = out; // rooted until IntoR hands it to R

// or: one typed extraction, then an index move-partition (rayon-safe after)
let by_group: Vec<(GroupKey, Vec<Measurement>)> = grouped.extract()?;

Supported key columns: factor (fast path — levels are the keys, level order kept, empty levels included, like split()), character (byte-order sort), integer (numeric sort), logical (FALSE, TRUE), and double, Date and POSIXct included (numeric sort, NaN after every number). List columns and bit64 integer64 columns (int64 bits stored in a double vector) are an error.

Double keys use dplyr’s equality (vctrs’): -0 and 0 are one key, every NaN is one key, and NaN and NA form separate groups — the same groups, in the same order, as dplyr::group_by(). A double key is a GroupKey::Real(RealKey): value() is the (normalised) number — days since 1970-01-01 for a Date, seconds since the epoch for a POSIXct — and the label is R’s as.character() of the key, called with the column’s attributes. That is the text factor() / split() use for level names: "0.1", "1e+05" and "NaN" for plain doubles, "2024-01-02" for a Date, and for a POSIXct the time in the column’s time zone ("2024-01-02 10:30:00", or just the date at midnight). Equality ignores the label.

NA keys form one group, ordered last. This deliberately deviates from R’s split(), which silently drops NA-keyed rows.

🔗Composite keys: `DataFrame::group_by_multi`

group_by_multi(&["a", "b"]) groups on several columns at once, keying each group by a GroupKey::Tuple of the per-column scalar keys (same supported types and errors as group_by):

let grouped = df.group_by_multi(&["site", "day"])?;
for (key, sub) in grouped.frames() {
    out.push(key.label(), sub); // label(): "site.day", "." separator
}

Non-NA groups match split(df, interaction(a, b, …, drop = TRUE)) — the first column varies fastest (R interaction()’s default, lex.order = FALSE), each column ordered as group_by would order it alone. For character keys the match is exact for keys whose byte order coincides with the session’s collation — always true in the C locale; single-case ASCII in practice (en_US.UTF-8 collates a A b B where byte order gives A B a b). This is inherited from group_by’s byte-order sort for character keys (see above). Labels join element labels with . (interaction’s default separator).

NA extends the single-column convention. interaction() maps any row with an NA in any component to NA and split() drops it; group_by_multi instead keeps such rows, forming one trailing group per distinct NA-containing tuple, in first-encounter row order. An empty column slice is an error; a single-column slice delegates to group_by and yields scalar keys (never 1-tuples), so downstream matches on GroupKey never have to special-case them.

🔗Honoring a dplyr `grouped_df`: `DataFrame::group_by_metadata`

group_by / group_by_multi compute grouping from your chosen columns. When a #[miniextendr] function receives a frame that dplyr has already grouped (a grouped_df), group_by_metadata() reads that existing grouping — respecting the caller’s grouping instead of recomputing it:

// df arrived as dplyr::group_by(data, site, day)
let grouped = df.group_by_metadata()?;
for (key, sub) in grouped.frames() {
    out.push(key.label(), sub); // one entry per dplyr group, in dplyr order
}

dplyr stores the grouping in attr(df, "groups"): a data frame whose leading columns are the group-key columns (one row per group, in dplyr’s order) and whose trailing .rows list-column holds each group’s 1-based row indices. group_by_metadata ingests that verbatim into the same GroupedDataFrame the compute verbs return — single key column → scalar GroupKey, multiple → a .-joined GroupKey::Tuple (same supported key types as group_by, double, Date and POSIXct included). It preserves the groups-frame order exactly (no re-sorting, no NA reordering), converts every .rows index from 1-based to 0-based, and keeps .drop = FALSE empty groups as groups with empty index vectors (mirroring group_by’s retention of empty factor levels).

Unlike group_by, this does no recomputation: a frame that does not inherit from grouped_df or rowwise_df is a NotGroupedDataFrame error — callers who want the framework to compute grouping use group_by / group_by_multi. A frame that does inherit from one of them but whose groups attribute is corrupt is reported as corrupt, with the same structural checks as group_declaration below (GroupsNotDataFrame, MissingGroupRows, BadGroupRows; a .rows list of doubles is rejected, as in dplyr). Other errors: UnequalLengths (a key column’s length differs from that of .rows) and UnsupportedGroupColumn (a key column of an unsupported type).

It trusts the cached rows. .rows is dplyr’s cache of the grouping, and group_by_metadata uses it as it is, after checking that it still fits the frame: every index in 1..=nrow (GroupIndexOutOfRange) and every row in exactly one group (GroupRowUncovered, GroupRowDuplicated). These errors say the metadata is stale and how to fix it. The usual cause is subsetting a grouped frame without dplyr loaded — base [ keeps the old groups attribute (with dplyr loaded, [.grouped_df regroups). The rows’ key values are not compared with their group’s key, so a pure reorder done that way (df[c(3, 4, 1, 2), ]) still fits and yields the old row positions (#1687).

Degenerate edge: a groups frame with zero key columns (e.g. a dplyr rowwise_df, whose groups frame is just .rows) yields empty-Tuple keys with empty labels — that shape is outside the documented grouped_df scope, so don’t pass rowwise frames here.

🔗Reading only the declaration: `DataFrame::group_declaration`

A function that groups the frame’s current rows itself only needs the declaration — the grouping variables and the .drop policy. group_declaration() returns exactly what dplyr’s group_vars() and group_by_drop_default() report: Ok(None) when the frame is not grouped, and an error when its grouping is corrupt, where group_vars() errors too. It reads no row index and partitions nothing, so the key-column types and stale metadata never matter:

// df arrived as dplyr::group_by(data, site, day, .drop = FALSE)
if let Some(decl) = df.group_declaration()? {
    // decl.vars == ["site", "day"]; decl.drop == false
    let vars: Vec<&str> = decl.vars.iter().map(String::as_str).collect();
    let grouped = df.group_by_multi(&vars)?; // recomputed from the current rows
}

A frame is grouped when it inherits from grouped_df or rowwise_df, whatever its attributes (dplyr::is_grouped_df() is inherits(x, "grouped_df")); a plain data.frame with a stray groups attribute is not. A grouped frame’s groups attribute must pass the structural checks of dplyr’s validate_grouped_df(), which group_vars() runs on a grouped_df:

`groups` attribute	`group_declaration()`
missing, not a data frame, or no columns	`GroupsNotDataFrame`
last column not called `.rows`	`MissingGroupRows`
`.rows` not a list of integer vectors (a list of doubles is not one)	`BadGroupRows`
`.rows` indices out of bounds for the frame	read as usual: row bounds are not checked

A rowwise_df gets the same checks. There dplyr is looser: group_vars() reads a rowwise_df’s groups attribute without validating it, and validate_rowwise_df() adds row-shape checks (one .rows element per row) that group_declaration skips, as it skips row bounds.

drop follows group_by_drop_default(): for a grouped_df, true unless the groups frame’s .drop attribute is exactly FALSE (identical(attr(groups, ".drop"), FALSE) — a length-1 logical FALSE with no attributes); always true for a rowwise_df, which has no group_by_drop_default() method of its own. group_by_multi is not a dplyr reimplementation: it orders groups like split(interaction()) and applies its own empty-group rule (a single factor key keeps its empty levels, composite keys keep only observed tuples) whatever drop says. For dplyr’s exact groups, regroup in R and read them with group_by_metadata.

🔗Parallel fast paths (`feature = "rayon"`)

Explicit _par variants produce the same BuiltDataFrame / Vec<Row> as the sequential verbs — parallelism is an opt-in method, not a hidden threshold:

let df   = rows.into_dataframe_par()?;             // parallel (column, row-range) fill
let rows = Vec::<Measurement>::from_dataframe_par(&df)?; // off-main-thread row assembly

Dropping _par (building without the rayon feature) degrades cleanly to the sequential call — the verb name is stable across feature sets.

#[miniextendr]
fn big_points() -> BuiltDataFrame {
    let points: Vec<Point> = (0..100_000)
        .map(|i| Point { x: i as f64, y: (i * 2) as f64 })
        .collect();
    points.into_dataframe_par().unwrap()  // explicit parallel fill
}

Parallel fill is most beneficial for large row counts (10k+), wide data frames (many fields), or expensive per-field conversions. For small data frames, prefer the sequential into_dataframe() to avoid rayon overhead.

🔗Heterogeneous columns without a row type

When you are filling columns directly (not transposing a Vec<Row>), use DataFrame::builder(nrow). column::<T> takes a native element type (f64 / i32 / RLogical / u8 / Rcomplex) and a chunk-fill closure; column_str builds a character column from a per-row closure returning Option<String> (None → NA_character_). build() yields a rooted BuiltDataFrame:

let df = DataFrame::builder(nrow)
    .column::<f64>("x", |chunk, offset| {
        for (i, slot) in chunk.iter_mut().enumerate() {
            *slot = (offset + i) as f64;
        }
    })
    .column_str("label", |i| Some(format!("p{i}")))
    .build();

The builder is always available. With the rayon feature, column buffers are filled in parallel over disjoint row ranges; without it, each column is filled serially. Both paths assemble the same rooted data.frame on the R thread.

🔗serde rows

Types that derive serde::Serialize / Deserialize convert through the SerdeRows<T> newtype, which keeps the serde path from colliding with the derive’s concrete Vec<Row> conversions:

use miniextendr_api::serde::SerdeRows;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct LogEntry {
    timestamp: f64,
    level: String,
    message: String,
}

#[miniextendr]
fn get_logs() -> BuiltDataFrame {
    let logs = vec![
        LogEntry { timestamp: 1.0, level: "INFO".into(), message: "started".into() },
        LogEntry { timestamp: 2.0, level: "ERROR".into(), message: "failed".into() },
    ];
    SerdeRows(logs).into_dataframe().unwrap()
}

// Reading back:
//   let logs = SerdeRows::<LogEntry>::from_dataframe(&df)?.into_inner();

Column types are inferred from serde field types:

Rust type	R column
`bool`	logical
`i8` / `i16` / `i32`	integer
`i64` / `u64` / `f32` / `f64`	numeric
`String` / `&str`	character
`Option<T>`	same type, `NA` for `None`

This is useful when you already have serde-serializable types and don’t want to add a DataFrameRow derive. For new types, prefer #[derive(DataFrameRow)], which gives you a typed companion type and the reverse-direction reader.

Requires the serde feature.

🔗Missing data

Use Option<T> for nullable scalar fields. None becomes the typed NA in R (NA_real_, NA_integer_, NA_character_, NA), and NA reads back as None; the column type is unchanged, so an all-None column is still a double / character / logical / integer vector:

#[derive(Clone, DataFrameRow)]
struct Record {
    id: i32,
    value: Option<f64>,  // NA in R when None
}

Accepted payloads: bool, f32, f64, i8..i64, u16..u64, isize, usize, String (the reader covers the same set as bare scalars: u64, isize, usize are write-only). Option around a map, a nested DataFrameRow struct, or a collection is rejected at compile time (see Rejected wrapper types); in enum row types the variant fields are already Option columns (absent for other variants), so write the bare scalar there.

u8 has no NA: R stores it in a raw vector. A column that would need an NA cell can’t hold u8, and the derive rejects it at the field, suggesting u16 or i32. That covers Option<u8>, any u8 in an enum variant (scalar, fixed array, width or expand), and width / expand over a u8 slice in a struct, all of which pad with NA. Struct fields of type u8, [u8; N] and Vec<u8> still work, and so does a plain Vec<u8> variant field, which stays a list-column of raw vectors.

🔗`DataFrameError`

A single error type covers every failure mode of both verbs:

Variant	Meaning
`NotList(actual)`	The SEXP is not a VECSXP; carries the SEXPTYPE it was.
`NotDataFrame`	The object does not inherit from `data.frame`.
`NoNames`	The list has no `names` attribute (columns must be named).
`BadRowNames(msg)`	Could not extract `nrow` from the `row.names` attribute.
`UnequalLengths { expected, columns }`	Every mismatched column and its actual length, compared to the first column.
`UnnamedColumns`	A row could not be turned into named columns.
`NoSuchColumn(name)`	`group_by` referenced a column name that does not exist.
`EmptyGroupColumns`	`group_by_multi` was called with an empty column slice.
`UnsupportedGroupColumn { column, type_of }`	A key column has no grouping semantics (list-column, `integer64`, …).
`NotGroupedDataFrame`	`group_by_metadata` on a frame that does not inherit from `grouped_df` or `rowwise_df`.
`GroupsNotDataFrame`	A `grouped_df` / `rowwise_df` whose `groups` attribute is missing or is not a data frame with at least one column.
`MissingGroupRows`	The last column of the `groups` frame is not called `.rows`.
`BadGroupRows { group, type_of }`	The `.rows` column is not a list (`group: None`), or one of its elements is not an integer vector (`group: Some(g)`; doubles are rejected, as in dplyr).
`GroupIndexOutOfRange { group, value, nrow }`	A `.rows` index is `< 1` or `> nrow`: the grouping metadata is stale.
`GroupRowUncovered { row, nrow }`	No group’s `.rows` holds a row of the frame: the grouping metadata is stale.
`GroupRowDuplicated { row, first_group, second_group }`	Two `.rows` entries hold the same row: the grouping metadata is stale.
`Conversion(msg)`	A serde or other conversion failure, carried as a message (also covers “this shape has no reader”).

It implements std::error::Error and From<RSerdeError>, so ? works in functions that mix serde and data-frame conversions.

As a #[miniextendr] argument, NotList and NotDataFrame become SexpError::Class and are worded 'df' must be a data frame: got list; the other variants keep their own message.

🔗Migration from the legacy surface

The redundant public types below were removed (#781) — there is no backwards-compat shim. If you have older code, map it to the façade:

Was	Now
`DataFrameView`, `convert::DataFrame<T>`	`DataFrame` view + `BuiltDataFrame` owner
`DataFrame::from_rows(rows)` (typed row buffer)	`rows.into_dataframe()?`
`ToDataFrame<Companion>` wrapper + `value.to_data_frame()`	`rows.into_dataframe()?`
`convert::SerializeDataFrame<T>` / `AsSerializeRow<T>` / `from_serialize()`	`serde::SerdeRows(rows).into_dataframe()?`
`impl IntoDataFrame for X { fn into_data_frame(self) -> List }`	derive `DataFrameRow`, or build a `BuiltDataFrame` via `DataFrame::builder(n)`
`Row::try_from_dataframe(sexp)` (bare `String` error)	`Vec::<Row>::from_dataframe(&df)?` (`DataFrameError`)
`RDataFrameBuilder::new(n)`	`DataFrame::builder(n)`
four conversion error types	one `DataFrameError`

The companion type that #[derive(DataFrameRow)] generates ({Name}DataFrame, with to_dataframe / from_rows / from_rows_par / from_dataframe and IntoIterator) still exists as the engine the façade verbs delegate to. It takes the row type’s visibility, so a private row struct gets a private companion. The serde columnar path (serde::vec_to_dataframe and friends) produces the same BuiltDataFrame handle — there is no separate ColumnarDataFrame type; the naming convergence was completed in #783.

🔗Feature flags

Base functionality: no features required.
serde: enables AsSerialize<T> (any Serialize value to R data), the SerdeRows<T> wrapper, and #[derive(Serialize, DataFrameRow)].
rayon: enables the _par verbs and parallelizes DataFrame::builder; the builder itself remains available without this feature.

🔗Examples

See rpkg/src/rust/dataframe_examples.rs for complete working examples.