Exec Steps Concurrent in task (task support DAG)

[lugeng]

Feature request

I would exec a tekton task with multi concurrent steps, but it seems cannot support this; (I know the pipeline can handle this feature, but I think it is too heavy to move workspace in different pod);

Use case

In my user case, I hope that the scan step and image build step can execute concurrent. And at last I need a job to collect the scan and build result.

[vdemeester]

@lugeng I think you might be interested by TEP-0044: Decouple Task Composition from Scheduling.

/cc @bobcatfish

[lugeng]

PipelineRun in a Pod, running of Tasks in parallel perfect meet our need !

[ms-lolo]

Hey just wanted to add a use case that I think is not met by being able to run pipelines on a single pod.

I have a single task that generates documentation in a repository. It's a single task because it is the one unit of reusable logic with a single output (a directory of html files).

However, my repository is made up of many python packages, and the documentation site is built by combining some top level documentation pages (a mkdocs site built from markdown files in a /docs folder) with the API documentation of each of the packages, generated by sphinx.

In order to generate API docs, I need to install the development environment for each of the python packages, individually, and then run my documentation command that handles running the appropriate sphinx process everywhere.

This all has to happen in the same Task because there are dependencies between some of these steps, but installing the development environment is a bit slow and should happen in parallel across all the packages in my repository. Right now, I am doing this in a single Step that runs multiple threads of my pipenv install --dev command. But this results in very confusing error logs when one out of the ~15 pipenv commands fails. If we want to get to the point where a Step is a single command, then I need to be able to have these 15 Steps run in parallel.

The output of my Task is a single html directory which I can bundle into a docker image in another task, but it doesn't make sense to have the pipenv install commands be their own Tasks just for the sake of parallelization. They don't represent a reusable unit of work that would be useful anywhere else.

It looks a little like this:

It's worth pointing out that I really cannot run mkdocs or sphinx without all of my packages having been initialized. So there is no way to generate docs for pkg A above without having done pipenv install on all 3 packages. This is what makes turning them into multiple Tasks impossible. Docs belong to my repository (a project), not the individual components (a python package).

Edit: as a general suggestion, I would love for tasks and steps to offer the same concurrency controls. In my mind, the main difference should be the default behavior. It makes sense for Tasks to run in parallel by default, but I can control this by defining dependencies between Tasks. By default, a Task has no dependencies and will run as soon as resources can be allocated on my cluster. It also makes sense for Steps to run sequentially, where it's assumed that a Step depends on the completion of the previous Step. But it would be nice to alter this default DAG and tell my 15 pipenv install Steps that they depend on the clone repo Step instead of the preceding pipenv install Step. Or, optionally, that they don't have a dependency at all and should run as soon as the container becomes ready.

And thanks for the awesome work!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.