[Feature][BanyanDB] Support for Tags and Fields TTL

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I suggest adding a new feature to BanyanDB that supports TTL (Time to Live) for tags and fields. This feature would enable BanyanDB to automatically remove specific tag or field data based on the TTL rules defined on a Group.

Currently, BanyanDB only supports TTL for the entire Group, which means that data can accumulate indefinitely, potentially causing storage issues. By implementing TTL for tags and fields, only relevant and recent data will be stored, which optimizes storage usage.

Furthermore, the proposed hot-warm-cold architecture could be utilized to set different TTL for cold, warm, or hot data, respectively.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sksDonni]

I would like to take this up to get myself familiarised with the backend of banyandb. Might need some help along the way.

[hanahmily]

I would like to take this up to get myself familiarised with the backend of banyandb. Might need some help along the way.

Go for it. When you are ready, I would like assign this task to you.

[sksDonni]

Hi @hanahmily, I am ready to take this up now.

[hanahmily]

@sksDonni If you have any questions about this task, please let me know. Before submitting a PR, it's important to write a design document detailing how you plan to implement this feature.

[wu-sheng]

About this, I think we need to update the protocol to support this. The server side can't / shouldn't determine which fields could be removed with time, unless OAP told so. Because in some cases(mostly for streaming), fields and tags are required until TTL reached.