Skip to content

Conversation

@zilto
Copy link
Collaborator

@zilto zilto commented Oct 31, 2025

This updates the template with several changes:

  1. more descriptive module-level docstring; removes keyword vibe coding and specifics about dlt commands
  2. fix import path from dlt.sources.rest_api.typing. The previous import was unintended and creates linting errors with pylance (this throws off IDE agents like Cursor)
  3. Remove instructions to set API key in plaintext. This suggests that the LLM should read / write your API key. Instead, added a comment about how dlt.secrets.value works
  4. Remove get_data() function. This scope is unecessary and adds complexity if we intend to run this as a script.
  5. Remove dataset_name kwarg. This simplifies the work of the LLM. Anyways, the name is derived from the pipeline name.
  6. Use dev_mode=True, this avoids schema migrations across runs.
  7. Use refresh="drop_sources" to avoid state conflicts as the agents does multiple iterations

@zilto zilto requested a review from rudolfix October 31, 2025 21:34
@zilto zilto self-assigned this Oct 31, 2025
@zilto zilto added the enhancement New feature or request label Oct 31, 2025
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Oct 31, 2025

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
docs b93a710 Commit Preview URL

Branch Preview URL
Nov 12 2025, 10:04 PM

rudolfix
rudolfix previously approved these changes Nov 2, 2025
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thx for fixing this

rudolfix
rudolfix previously approved these changes Nov 2, 2025
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I've modified AST parser to allow optional dataset names and script import checker to allow pipelines to be instantiated on import (makes sense also for runtime)

pipeline running, iterating resources and source instantiation is not allowed

@zilto zilto marked this pull request as draft November 2, 2025 20:50
@zilto
Copy link
Collaborator Author

zilto commented Nov 2, 2025

Converted to draft to avoid merging. I realized that dev_mode has some counterintuitive behavior #3276 which may be confusing for humans / LLMs.

I think the workspace feature and automated local pipelines_dir is generally helpful for LLMs since it gives the full "project context/state" under the IDE project root.

@zilto zilto marked this pull request as ready for review November 12, 2025 19:19
@zilto zilto force-pushed the feat/update-scaffolding-template branch from e5c8f3a to b93a710 Compare November 12, 2025 21:55
@zilto zilto merged commit 152555a into devel Nov 13, 2025
71 of 72 checks passed
@zilto zilto deleted the feat/update-scaffolding-template branch November 13, 2025 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants