Rust SDK: Implement Stream-Based Pagination Support for List/Collection Endpoints

[iamnamananand996]

Which Fern component?

SDK Generator

How important is this?

P0 - Critical (Blocking work)

What's the feature?

Description:

Implement Stream-based pagination for list endpoints using Rust's async Stream trait for efficient data handling.

Expected Implementation:

Create Paginator<T> that implements Stream<Item = Result<T, ApiError>>:

use futures::Stream;
use std::pin::Pin;
use std::task::{Context, Poll};

pub struct Paginator<T> {
    http_client: HttpClient,
    base_path: String,
    next_cursor: Option<String>,
    current_items: VecDeque<T>,
    params: Option<serde_json::Value>,
    request_options: Option<RequestOptions>,
    exhausted: bool,
}

impl<T> Stream for Paginator<T> 
where 
    T: serde::de::DeserializeOwned + Unpin,
{
    type Item = Result<T, ApiError>;
    
    fn poll_next(
        mut self: Pin<&mut Self>, 
        cx: &mut Context<'_>
    ) -> Poll<Option<Self::Item>> {
        // 1. Return buffered items first
        if let Some(item) = self.current_items.pop_front() {
            return Poll::Ready(Some(Ok(item)));
        }
        
        // 2. If exhausted, return None
        if self.exhausted {
            return Poll::Ready(None);
        }
        
        // 3. Fetch next page
        // Implementation for async page fetching
    }
}

Method Generation:

For endpoints that return lists, generate both standard and paginated versions:

impl UsersClient {
    // Standard method (existing)
    pub async fn list_users(
        &self, 
        params: Option<ListUsersParams>,
        options: Option<RequestOptions>
    ) -> Result<ListUsersResponse, ApiError> {
        // Existing implementation
    }
    
    // New paginated method
    pub async fn list_users_paginated(
        &self,
        params: Option<ListUsersParams>, 
        options: Option<RequestOptions>
    ) -> Result<Paginator<User>, ApiError> {
        Ok(Paginator::new(
            self.http_client.clone(),
            "/users".to_string(),
            params.map(|p| serde_json::to_value(p).unwrap()),
            options,
        ))
    }
}

Pagination Detection:
Identify paginated endpoints by checking IR for:

• Response has next cursor or page info
• Response contains array of items
• Common pagination patterns in endpoint names (list_*, get_*s)

Acceptance Criteria:

• Paginator<T> implements Stream trait correctly
• Automatic page fetching when items exhausted
• Paginated methods generated for list endpoints
• Support for cursor-based and offset-based pagination
• Helper methods: collect_all(), take(n), skip(n)
• Error handling during pagination
• Integration tests demonstrate pagination workflow

Usage Examples:

// Stream usage
let mut paginator = client.users.list_users_paginated(None, None).await?;
while let Some(user) = paginator.next().await.transpose()? {
    println!("User: {}", user.name);
}

// Collect all items
let all_users: Vec<User> = client.users
    .list_users_paginated(None, None)
    .await?
    .collect_all()
    .await?;

// Take first 50 items
let first_50: Vec<User> = client.users
    .list_users_paginated(None, None)
    .await?
    .take(50)
    .try_collect()
    .await?;

Definition of Done:

• Paginator implements Stream trait and works with futures combinators
• Paginated methods generated for appropriate endpoints
• Integration tests show automatic page fetching
• Performance tests confirm efficient memory usage
• Documentation shows usage examples

---

Any alternatives?

No response

Are you interested in contributing this feature?

Yes