Prototyping the 8-hr auto-restart mechanism and stream restoration

[sgbaird]

Related:

• YouTube Live Streaming API - minimal working example #202 (focused on getting the API set up and pulling stream URL automatically)
• Download self-owned YouTube videos #212
• Post-processing videos (still frame removal and speedup) #223

Based on discussion with @Neil-YL:

How should the following tasks be handled (e.g., same manager, different managers):

1. Close the current stream and start a new one every 8 hours
2. Log the recent status of each stream into a CSV every 10-15mins
3. Report the latest download link to a database

Or is it better to separate these into different scripts or services?

Maybe the monitor (task 2) as a separate service. Maybe the auto-restart and download links as a schedule-based service (e.g., scheduled task with Prefect) to trigger the workflow: first retrieve the download link for the latest completed video (not the currently active stream), then close the current broadcast and create a new one.

Another idea is to have a monitor create new stream when device_name is not in active stream list, and an 8-hr schedule to end all active streams

8-hr schedule to end all active streams
This one could also restart the stream, to minimize downtime (i.e., loop through each stream one at a time --> [end stream, restart stream]), but maybe this would cause issues if the monitor and this both try to restart the same stream.

It could be one code/service to both end/restart and monitor, when executing ending and restart, try to move out of the loop of the checking and when the restart completed back to the monitor loop.

---

There also needs to be a service that auto-downloads the videos and post-processes them (high bandwidth, so probably don't want to mix with the scripts that ensure uptime). This might be long-running processes on the order of 1+ hrs for each video, where GPUs would significantly accelerate the process. If there are only a few streams, a single service might be able to handle this. If there are 50+ streams, then there would need to be multiple machines doing the processing. This might justify some kind of trigger-based and on-demand mechanism.. (instead of having a few dedicated machines doing the processing). This might still require a schedule and monitor to see if new videos are available in a playlist (or if new entries are available in MongoDB).

https://prefect-community.slack.com/archives/C04DZJC94DC/p1745859761114599?thread_ts=1745858182.153999&cid=C04DZJC94DC

Sterling Baird
42 minutes ago
@Marvin
how do you handle on-demand spinup of resources based on monitoring some kind of condition? (e.g., new entry added to a MongoDB database)

Marvin
APP 41 minutes ago
To handle on-demand resource spinup based on a condition like a new entry in a MongoDB database using Prefect, you can follow these general steps:

1. Monitoring and Triggering: Use Prefect's event-driven triggers to monitor conditions. For instance, you can create a custom trigger that listens for specific events such as changes in your MongoDB database. Prefect allows you to define custom events and triggers that can start flows based on these events. More details can be found in the Prefect custom triggers documentation.
2. Infrastructure Provisioning: Prefect can automatically provision the necessary infrastructure for running your flows. This is done using Prefect's CLI with the --provision-infra flag, which allows for serverless resource provisioning. This requires proper authentication and setup with your cloud provider, such as AWS. You can find more information in the Prefect infrastructure examples documentation.
3. Flow Definition: Define your flow using Prefect's flow decorator, which is part of the core API. Prefect supports imperative programming styles, which should suit your preference.
   For more specific implementations, you may need to integrate with Prefect's Python API to define tasks that interact with MongoDB and trigger flows based on the results.
   If you have more specific requirements or need further clarification, feel free to ask!

[Neil-YL]

While testing the "End device stream and start a new one" function, I noticed that even with "Autostart" enabled, which should automatically start the stream when YouTube receives data from the RTMP stream key, the stream doesn't begin immediately with "Excellent connection".

To clarify: When YouTube API creates a new "Broadcast", it initially appears as "Scheduled", which can then be started either manually or via Autostart.

This could result in losing footage and may also bring many troubles for the monitor set up (but we can simply add "Scheduled" as an active broadcast in the restart condition)

(In this test, the OT-2 stream started in about 5 minutes, but the A1 Mini stream took more than 20 minutes to start after the manager triggered the "End device stream and start a new one")

[Neil-YL]

Sum up for the test yesterday:

I set a 5 hours restart yesterday at around 16:35.

So it will trigger end and restart at 21:35, 2:35, 7:35, we should have 3 video for each device, and a live now.

1. 16:35-21:35
2. 21:35-2:35
3. 2:35-7:35
4. 7:35-now

Now we only has the footage for:

• OT2: 1. 4hrs 56mins; 2. 32mins; 3. no footage; 4. no live;
• A1mini: 1. 4rs56mins; 2. no footage; 3: 1hr52mins; 4. on live;
• Pi4-test: 1. no footage; 2. 32mins; 3. no footage; 4. no live;

The 32 mins records are very likely be triggered by the 2am reboot on the zero2w, but not sure why A1mini doesn't have a similar 32mins video.

Update:

When checking the status of the un-started broadcast, it shows as "ready" but never covert to "active"
YouTube API provides a status transition request, it only applies to broadcasts that are already in the "active" state:

Note that to transition a broadcast to either the testing or live state, the status.streamStatus must be active for the stream that the broadcast is bound to.

Very similar to our situation: https://stackoverflow.com/a/79204910

[sgbaird]

Thanks for collecting the data and the update! If the "already sending data" piece is the main issue and isn't one that can easily be overcome with YouTube's API and restrictions, then we may need to brainstorm a bit more. We could use an AWS Lambda function perhaps that handles the setup and teardown of the streams and call the lambda function from the device directly. The token.pickle file could live on the AWS Lambda instance, and this way the device script can stop sending data, tear down the stream, set it back up, and then start sending data again. Thoughts?

[sgbaird]

As a side note, I think it may be worth changing the stream latency to "low-latency", and possibly adding a delay (unless adding a delay doesn't help). There seems to be missing segments (e.g., when the OT-2 was removed from the office for the event), as noted in a separate issue (I think the OT-2 equipment monitoring issue or the original Zero 2W issue). [cid:d31037e8-050a-4f89-aa50-c024665cd509]

[Neil-YL]

As a side note, I think it may be worth changing the stream latency to "low-latency", and possibly adding a delay (unless adding a delay doesn't help). There seems to be missing segments (e.g., when the OT-2 was removed from the office for the event), as noted in a separate issue (I think the OT-2 equipment monitoring issue or the original Zero 2W issue).
[cid:d31037e8-050a-4f89-aa50-c024665cd509]

I think it is because it happened to have a no data issue during the removal period. I will change the latency to low in next start.

[Neil-YL]

Add enableMonitorStream: false to the broadcast creating codes as suggested in one stackoverflow, seems not help in improving the uptime:

14:00 - 19:00: lose around 30mins
19:00 - 24:00: lose around 1hrs 8/15mins
0:00 - 5:00: lose around 1hrs 40mins and 2 hrs
5:00 - now: never start until I reboot zero2w

I will think about the cross devices end and restart.
Basic scheme:
Manager: End device.py on zero2w --> End YTB broadcast --> Restart YTB broadcast --> Restart device.py on zero2w
(It also could be: End YTB broadcast --> Restart YTB broadcast --> End device.py on zero2w --> Restart device.py on zero2w?)
Monitor: Find which device broadcast is down --> End device.py on zero2w --> Restart YTB broadcast --> Restart device.py on zero2w

[Neil-YL]

Thanks for collecting the data and the update! If the "already sending data" piece is the main issue and isn't one that can easily be overcome with YouTube's API and restrictions, then we may need to brainstorm a bit more. We could use an AWS Lambda function perhaps that handles the setup and teardown of the streams and call the lambda function from the device directly. The token.pickle file could live on the AWS Lambda instance, and this way the device script can stop sending data, tear down the stream, set it back up, and then start sending data again. Thoughts?

[Scheduled reboot via crontab]
↓
[Zero2W reboots]
↓
[systemd auto-starts device.py]
↓
[device.py, before starting P1/P2 stream]
→ Calls Lambda to:
- End current YouTube broadcast
- Start a new one
- Return successful message
↓
[device.py starts stream and enters existing while loop]

I think by doing so, we can limit the changes to our systemd setup and the device.py while loop, helping maintain overall stability.

[sgbaird]

This sounds great! I thing this leverages the scheduling strengths of cron pretty well, and like you said avoids complicating device top high which already took a while to get it to be robust.

[sgbaird]

Per our conversation, I think the decision was to include something in the title (maybe original video id? Or could be timestamp) that we can keep the same between the original and the processed videos, and add some kind of note in the title like: [processed, 16x]. @Neil-YL

Source: #223 (comment)

[sgbaird]

@Neil-YL instructions https://github.com/ACC-HelloWorld/5-data-logging#aws-lambda-function

[Neil-YL]

I finished the setup on AWS Lambda. I will start working on the device.py and the crontab restart.

Here is a test on starting a new broadcast for A1mini

[Neil-YL]

OK I think I have finished the deployment and I put it on the Bambu A1mini stream zero2w for test. I set a reboot at 05:00, 13:00 and 21:00 with crontab.
Now the playlist has 27 video so it would be 30 tomorrow.

If everything works as expected I will create a PR later.

[sgbaird]

Awesome! Funnily, just as I was reading this, I got a yt notification (9:01 pm)

[Neil-YL]

Looks good!