feat: support FixedSizeList<Struct> #5593

wkalt · 2025-12-29T20:37:00Z

Prior to this commit, FixedSizeList was only supported with primitive element types (e.g., FSL<Float32> for vectors). This adds structural encoding support for FSL<Struct>, enabling use cases like fixed-size arrays of bounding boxes, coordinate tuples, or other structured data.

Key changes:

New FixedSizeListStructuralEncoder that encodes FSL validity to rep/def and delegates child encoding to the struct encoder
New StructuralFixedSizeListScheduler that scales row ranges by the FSL dimension when scheduling reads
New StructuralFixedSizeListDecoder that reconstructs FSL arrays from child data and rep/def validity

A key challenge is "garbage filtering": unlike variable-length lists which can omit children under null entries, FSL children always exist. When an FSL row is null, any nested list-like types within its children contain undefined "garbage" data. The encoder normalizes these to empty null lists before encoding.

wkalt · 2025-12-29T20:38:09Z

sorry, mislabeled this breaking. Should not be breaking.

codecov · 2025-12-29T21:13:50Z

Codecov Report

❌ Patch coverage is 87.63838% with 67 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...-encoding/src/encodings/logical/fixed_size_list.rs	91.21%	28 Missing and 9 partials ⚠️
rust/lance-encoding/src/encoder.rs	58.62%	9 Missing and 3 partials ⚠️
rust/lance-datagen/src/generator.rs	81.96%	8 Missing and 3 partials ⚠️
rust/lance-encoding/src/decoder.rs	54.54%	0 Missing and 5 partials ⚠️
rust/lance-core/src/datatypes/field.rs	90.00%	0 Missing and 1 partial ⚠️
...ust/lance-encoding/src/encodings/logical/struct.rs	85.71%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

wkalt · 2025-12-31T05:27:56Z

I will update this patch once #5591 is merged - there are some related tests I want to add here that currently fail.

westonpace · 2026-01-06T17:04:54Z

Should this encoder require 2.2? Our general rule of thumb has been "any file written by any version of 2.1 should be readable by any version of 2.1" which is perhaps stricter than normal backwards compatibility rules. Since this allows you to create files that old 2.1 readers will not be able to read I think this will need to be a 2.2 feature.

Or maybe you have that check and I just missed it.

Also, does this handle FixedSizeList<List<...>> or is that a todo still?

wkalt · 2026-01-06T17:26:59Z

FixedSizeList<List> is still unimplemented - there is some existing todo note about List<List> somewhere I believe, and the same constraints still apply.

Sounds like this should require 2.2, thanks for catching that. I'll push an update shortly.

wkalt · 2026-01-06T21:05:46Z

@westonpace thanks, this is updated, I think the failures are unrelated.

wjones127 · 2026-01-07T00:14:04Z

I think the failures are unrelated.

👍 #5646

wjones127

Have one concern about handle corrupt schemas, but otherwise looks good.

wjones127 · 2026-01-07T00:15:33Z

rust/lance-core/src/datatypes/field.rs

+                let size: i32 =
+                    lt.0.split(':')
+                        .next_back()
+                        .expect("fixed_size_list:struct logical type missing size suffix")
+                        .parse()
+                        .expect("fixed_size_list:struct logical type has invalid size");


Does this mean we can panic if we read a dataset that has a corrupt schema?

This is the lance schema. I'm not entirely sure users are able to create their own lance schema. So I think the only way this could happen is if there was some kind of corrupt protobuf. Also, there is a significant panic potential down below at lt => DataType::try_from(lt).unwrap().

I suppose it is technically a valid concern but this method has many many callsites and changing it to result returning should probably be a PR on its own or else this one is going to get real confusing.

westonpace

Nicely done. A few minor thoughts but looks good.

westonpace · 2026-01-07T13:51:24Z

rust/lance-datagen/src/generator.rs

 }

+#[derive(Debug)]
+struct RandomMapGenerator {


Can you document this (or the rand_map method). Just a brief comment somewhere to describe that it will randomly generate maps with 0-4 items.

westonpace · 2026-01-07T13:51:58Z

rust/lance-datagen/src/generator.rs

+        let total_entries = lengths.values().iter().sum::<i32>() as u64;
+        let offsets = OffsetBuffer::from_lengths(lengths.values().iter().map(|v| *v as usize));
+
+        let keys = self.keys_gen.generate(RowCount::from(total_entries), rng)?;


Not really relevant but I wonder if keys need to be unique within a map? I guess not.

westonpace · 2026-01-07T13:52:43Z

rust/lance-encoding/src/decoder.rs

+                let child = field
+                    .children
+                    .first()
+                    .expect("FixedSizeList field must have a child");


Can you use expect_ok here so we get a result and not a panic?

westonpace · 2026-01-07T13:53:14Z

rust/lance-encoding/src/encoder.rs

+                    let child = field
+                        .children
+                        .first()
+                        .expect("FixedSizeList should have a child");


westonpace · 2026-01-07T13:58:18Z

rust/lance-encoding/src/encodings/logical/fixed_size_list.rs

+}
+
+/// Filters garbage from nested FSL arrays that contain list-like children.
+fn filter_nested_fsl_garbage(


Nice work here.

westonpace · 2026-01-07T13:59:11Z

rust/lance-encoding/src/encodings/logical/fixed_size_list.rs

+    async fn test_fsl_struct_random(
+        #[case] struct_fields: Fields,
+        #[case] dimension: i32,
+        #[case] min_version: LanceFileVersion,
+        #[values(STRUCTURAL_ENCODING_MINIBLOCK, STRUCTURAL_ENCODING_FULLZIP)]
+        structural_encoding: &str,
+    ) {
+        let data_type = make_fsl_struct_type(struct_fields, dimension);
+        let mut field_metadata = HashMap::new();
+        field_metadata.insert(
+            STRUCTURAL_ENCODING_META_KEY.to_string(),
+            structural_encoding.into(),
+        );
+        let field = Field::new("", data_type, true).with_metadata(field_metadata);
+        let test_cases = TestCases::basic().with_min_file_version(min_version);
+        check_specific_random(field, test_cases).await;
+    }


Nice compact and comprehensive test

westonpace · 2026-01-07T13:59:57Z

rust/lance-encoding/src/encodings/logical/fixed_size_list.rs

+    }
+
+    #[test]
+    #[should_panic(expected = "Unsupported logical type: map")]


Would be nice if unsupported type changed to err instead of panic. I'll make a ticket and we can handle in a follow-up.

Prior to this commit, FixedSizeList was only supported with primitive element types (e.g., FSL<Float32> for vectors). This adds structural encoding support for FSL<Struct>, enabling use cases like fixed-size arrays of bounding boxes, coordinate tuples, or other structured data. Key changes: - New `FixedSizeListStructuralEncoder` that encodes FSL validity to rep/def and delegates child encoding to the struct encoder - New `StructuralFixedSizeListScheduler` that scales row ranges by the FSL dimension when scheduling reads - New `StructuralFixedSizeListDecoder` that reconstructs FSL arrays from child data and rep/def validity A key challenge is "garbage filtering": unlike variable-length lists which can omit children under null entries, FSL children always exist. When an FSL row is null, any nested list-like types within its children contain undefined "garbage" data. The encoder normalizes these to empty null lists before encoding.

github-actions bot added enhancement New feature or request breaking-change labels Dec 29, 2025

wkalt changed the title ~~feat!: support FixedSizeList<Struct>~~ feat: support FixedSizeList<Struct> Dec 29, 2025

wkalt force-pushed the task/complex-fsl branch from 10ef8c2 to 23ba011 Compare January 1, 2026 14:57

wjones127 reviewed Jan 7, 2026

View reviewed changes

westonpace approved these changes Jan 7, 2026

View reviewed changes

wkalt added 6 commits January 7, 2026 20:52

lint fix

5f7daf0

format

d4bea20

Require Lance version 2.2

507efd9

Document random map generation

7736131

expect_ok

3d0064b

wkalt force-pushed the task/complex-fsl branch from 690cac0 to 3d0064b Compare January 8, 2026 05:01

lint

9eb5a9a

feat: support FixedSizeList<Struct> #5593

Are you sure you want to change the base?

feat: support FixedSizeList<Struct> #5593

Uh oh!

Conversation

wkalt commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wkalt commented Dec 29, 2025

Uh oh!

codecov bot commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

wkalt commented Dec 31, 2025

Uh oh!

westonpace commented Jan 6, 2026

Uh oh!

wkalt commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wkalt commented Jan 6, 2026

Uh oh!

wjones127 commented Jan 7, 2026

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wkalt commented Dec 29, 2025 •

edited

Loading

codecov bot commented Dec 29, 2025 •

edited

Loading

wkalt commented Jan 6, 2026 •

edited

Loading