Switching input stream from low quality to high quality after iframe (simulcast?)

[smhc]

The "smart" style h265 encoding saves lots of bandwidth, but it has very low i-frame rates. This makes it difficult to use for on-demand live streaming which this project is aimed at.

Is it possible to define a stream that takes two inputs, a "fast, low quality stream" and a "high latency, low i-frame, HD stream" and migrates to the higher quality stream as soon as an i-frame is received?

It theoretically may be possible using 'exec', multiple processes, pipes and probes for i-frames. Perhaps may need to use the 'overlay' filter as well. But I am not sure about the 'switch-over' of streams. I don't mind how much CPU it uses as I expect it to only be running temporarily on-demand.

The use case for this is to "live-view" a camera using its 'detect' stream and move over to the higher quality 'h265/record/hd' stream once it can be successfully decoded for viewing, which could take 20 seconds or more.

Is this something that is already possible? Or a potential feature to implement? I can imagine it would have plenty of application for users of this project and may be a better solution to some of the other proposals like "pre-loading" etc.

[AlexxIT]

I had no experience working with different qualities within the same stream.
As far as I know, this is best supported in the HLS format. But this is the worst format for real-time streaming.
I've heard that this is supported in WebRTC as well. But there was no experience.
It will also most likely require more complex logic on the client side.

[smhc]

I was thinking of something like the below.. but haven't quite got it working yet:

#!/bin/bash

steam1="rtsp://127.0.0.1:8554/lq_stream"
stream2="rtsp://127.0.0.1:8554/hq_stream"
rm -f /tmp/stream1.pipe /tmp/stream2.pipe

# Array to hold child PIDs
CHILD_PIDS=()

# Function to kill all child processes
cleanup() {
    for pid in "${CHILD_PIDS[@]}"; do
        kill "$pid" 2>/dev/null
    done
    # Remove pipes if desired
    rm -f /tmp/stream1.pipe /tmp/stream2.pipe
    exit 0
}

# Trap SIGINT, SIGTERM, and EXIT to cleanup
trap cleanup SIGINT SIGTERM EXIT


mkfifo -m 666 /tmp/stream1.pipe
mkfifo -m 666 /tmp/stream2.pipe

# Start streaming stream1 to stream1.pipe in the background
(
    ffmpeg -hide_banner -loglevel error -rtsp_transport tcp -i "$steam1" -c:v libx264 -f h264 -y /tmp/stream1.pipe
) &
PID_STREAM1=$!
CHILD_PIDS+=($PID_STREAM1)

# Start streaming stream2 to stream2.pipe in the background
(
    ffmpeg -hide_banner -loglevel error -rtsp_transport tcp -i "$stream2" -c:v libx264 -f h264 -y /tmp/stream2.pipe
) &
PID_STREAM2=$!
CHILD_PIDS+=($PID_STREAM2)

# Forward stream1.pipe to output in the background
(
    # Use dd to ensure binary data is output to stdout without any conversion
    dd if=/tmp/stream1.pipe of=/dev/stdout
) &
PID_CAT=$!
CHILD_PIDS+=($PID_CAT)

(
    # echo "Checking for I-frame in stream2" >&2
    # echo "PID: $PID_CAT" >&2
    # Function to check for I-frame in stream2
    check_iframe() {
        ffprobe -hide_banner -loglevel error -select_streams v:0 -show_frames -show_entries frame=pict_type -of csv /tmp/stream2.pipe 2>/dev/null | grep -m 1 'I'
    }

    # Wait for I-frame in stream2
    while true; do
        if check_iframe; then
            break
        fi
        # Flush current buffer of stream2.pipe 
        dd if=/tmp/stream2.pipe of=/dev/null bs=4096 count=1 2>/dev/null
        sleep 1
    done

    # Once I-frame is found, switch output to stream2
    kill $PID_CAT 2>/dev/null
    kill $PID_STREAM1 2>/dev/null

    # Forward stream2.pipe to output
    dd if=/tmp/stream2.pipe of=/dev/stdout
) &
PID_CHECK_IFRAME=$!
CHILD_PIDS+=($PID_CHECK_IFRAME)
wait $PID_CHECK_IFRAME
exit 0

[AlexxIT]

Forget about input. The whole difficulty is in the output.

The RTSP, WebRTC, MSE, HomeKit and many others protocols will not survive the quality change without special improvements.

[smhc]

I'm happy to take the LQ stream and transcode it into the target HQ stream - if that's even possible. The end result is to watch the HQ stream, the LQ stream is just temporary. Even if it means using the 'overlay' filter then turning it off later - I just don't know that ffmpeg can do something like that.

BlueIris seems to do something like this - but I'm not sure if it has a special client to support it or not.

[AlexxIT]

I get the idea. This is also quite a difficult task - first use one stream with transcoding and then switch to the second stream.

I think it's better to implement the usual switching between different streams. But this is a big task for the distant future.