Add option to FlightDataEncoder to always resend batch dictionaries

[alexwilcoxson-rel]

Which issue does this PR close?

Closes #4895

Rationale for this change

see #4895

What changes are included in this PR?

Adds with_send_dictionaries to FlightDataEncoderBuilder to opt in to always sending dictionary messages with each record batch encoded.

Are there any user-facing changes?

with_send_dictionaries added to FlightDataEncoderBuilder and FlightDataEncoder. Doc strings updated.

[alamb]

Thank you for this PR @alexwilcoxson-rel -- I hope to find time to review it properly in the next few days

[alamb]

Thank you @alexwilcoxson-rel -- I think this is a really nice improvement and it could be merged as is.

I was thinking to a future when we might have proper support for resending dictionaries and trying to minimize the API impact as well as adding a nice place for more documentation. What do you think about using an enum instead of a bool like this:

So instead of

pub struct FlightDataEncoder {
...
    send_dictionaries: bool,
...
}

Something like

pub struct FlightDataEncoder {
...
    dictionary_handling: DictionaryHandling,
...
}

/// Defines how a [`FlightDataEncoder`] encodes [`DictionaryArray`]s
pub enum DictionaryHandling {
  /// Expands to the underlying type (default). This likely sends more data over the network
  /// but requires less memory (dictionaries are not tracked) and is  more compatible
  /// with other arrow flight client implementations that may not support `DictionaryEncoding`
  /// see [`hydrate_dictionary`] for more details.
  Hydrate,
  /// Send dictionary FlightData with every RecordBatch that contains a [`DictionaryArray`].
  /// See [`Self::Hydrate`] for more tradeoffs. No attempt is made to skip sending the same (logical)
  /// dictionary values twice. 
  Resend, 
}

[alamb]

👍

[alamb]

👌

[alamb]

BTW it looks like some documentation also needs to be updated, like

arrow-rs/arrow-flight/src/encode.rs

Line 33 in 23db567

/// # Caveats

I can handle doing so, but wanted to mention it in case @alexwilcoxson-rel was going to change this one more

[alexwilcoxson-rel]

BTW it looks like some documentation also needs to be updated, like

arrow-rs/arrow-flight/src/encode.rs

Line 33 in 23db567

/// # Caveats

I can handle doing so, but wanted to mention it in case @alexwilcoxson-rel was going to change this one more

Updated these docs

[alexwilcoxson-rel]

Updated to use Enum per @alamb feedback

[alamb]

Thanks @alexwilcoxson-rel -- this looks great

[alamb]

The CI seems to be failing the doc checks. I'll try and fix that

[alamb]

Here is a proposed PR to your branch for improved documentation relativityone#1 🙏

[alexwilcoxson-rel]

Here is a proposed PR to your branch for improved documentation relativityone#1 🙏

LGTM, merged! please rerun CI when you can.

[alamb]

Thanks again @alexwilcoxson-rel