Support starting from any coordinates in the DAG

[ninjabear]

Currently Factotum does not allow starting from some points in the DAG - specifically if a dependant task requires a task from "above" the starting task. It would be nice to start from any task - but we need to define a sensible mode of operation in this event.

/cc @alexanderdean

[alexanderdean]

My suggestion is:

• Keep current behavior the same
• Also add support for --start "task 1","task 2","task 3" where those tasks unambiguously define a comprehensive resume point across all parallel branches in the DAG
• The --start command should check that all of the start points are consistent with each other, and raise an error in the case of any ambiguity or conflict

[alexanderdean]

I want to confirm that this error will be fixed by this ticket:

"The job cannot be started from 'mcloudfront-emr-etl-runner' because the job cannot be started here without triggering prior tasks"

[ninjabear]

Yes, that is the error message for this

[alexanderdean]

Cool, can't wait to get this supported

[ninjabear]

had a good conversation with @alexanderdean about this!

Reasoning about where a DAG is paused and then working out the start task needed to resume is simple for trivial DAGs but becomes progressively harder as they become more complex, and multiple things can fail at once.

Just to recap, factotum refuses to run from a start task when this happens:

               A
              / \
             B   C
            / \ /
           D   E 

We cannot start at B because E depends on C, which depends on A (similar for C)    

               A 
              / \
             B   C  
            / \   \ 
           D   E   F 

It's fine to start at B here though, causing B, D, and E to be run 

Essentially, it prevents you resuming tasks which are "shared" by chains of execution - this is as designed, since the parent(s) state is unknown it's safer to not allow it.

The breakthrough idea here from Alex is, instead of offloading the mental gymnastics to the person running the DAG, when a failure occurs factotum should return a code, that you could then pass into a --start argument to resume for you.

For example:

               A
              / \
             B   C
            / \ /
           D   E 

Pretending that B fails, Factotum should say:

Task "B" failed!
...
To resume all incomplete tasks, rerun with code <some alphanumeric code>

Then we could resume doing something like:

factotum run xyz.factfile --start=ABC12345

What do you think @leonmaas / @ihortom? Can you see a situation where this wouldn't be tons better?

[alexanderdean]

Just a note that we also wanted to keep a "raw" command option for more fine-grained resumption, like --start=B,C. Because humans struggle with this, we might also want to let the user itemize all steps to run, like --steps B,C,D,E

[ninjabear]

@alexanderdean RE timings;

I think putting "resume tokens" in the next version of this will take about a day to complete (65% certainty), to 3 days tops (90% certainty).

Also - completely agree, this will be "as well as" rather than "instead of"!

[ninjabear]

Moving back while we noodle the best way to put in resume tokens!

[ihortom]

It's a great idea to provide a suggestion on restart rather than figuring it out on your own. I would like to add an ability to simulate the failure at a step and see the suggested rerun. The reason is the incident could have been closed and it could take some time to find it and see the suggested rerun. Could we also simulate the failed run (using --dry-run) to see the suggestion? Example:

factotum run <factfile> --dry-run --fail-at=<task>

The option --fail-at has a meaning only if factotum runs with --dry-run. Thus, it could be just a different (exclusive) option like:

factotum run <factfile> --dry-run-and-fail=<task>