Frame number of depth frame and color frame does mismatch on linux and pipe return same frame

[heavyman1]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D435
Firmware Version	05.09.13.00
Operating System & Version	Ubuntu 16.04.1
Kernel Version (Linux Only)	4.13.0-36-generic
Platform	Up Board
SDK Version	2.12.0
Language	C++
Segment	others

Frame number of depth frame and color frame does not match on linux and pipe return same frame.

Code :

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <iostream>

int main(int argc, char * argv[])
{
    try
    {
        const int N = 100;

        rs2::config config;
        config.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 30);
        config.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 30);
        rs2::pipeline pipe;
        pipe.start(config);

        double preTimestampDepth = 0;
        double preTimestampColor = 0;
        for (int i = 0; i < N; i++)
        {
            auto fs = pipe.wait_for_frames();
            auto depth_frame = fs.get_depth_frame();
            auto color_frame = fs.get_color_frame();
            double timestampDepth = depth_frame.get_timestamp();
            double timestampColor = color_frame.get_timestamp();
            if (i == 0)
            {
                printf("frame [%02d]. depth : frame# %lld, timestamp diff [%7.3f]. color : frame# %lld, timestamp diff[%7.3f].\n", i, 
                    depth_frame.get_frame_number(), 0.0f,
                    color_frame.get_frame_number(), 0.0f);
            }
            else
            {
                printf("frame [%02d]. depth : frame# %lld, timestamp diff [%7.3f]. color : frame# %lld, timestamp diff[%7.3f].\n", i, 
                    depth_frame.get_frame_number(), timestampDepth - preTimestampDepth,
                    color_frame.get_frame_number(), timestampColor - preTimestampColor);
            }
            preTimestampDepth = timestampDepth;
            preTimestampColor = timestampColor;
        }

        return EXIT_SUCCESS;
    }
    catch (const rs2::error & e)
    {
        std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
        return EXIT_FAILURE;
    }
    catch (const std::exception& e)
    {
        std::cerr << e.what() << std::endl;
        return EXIT_FAILURE;
    }
}

Result :
frame [00]. depth : frame# 8, timestamp diff [ 0.000]. color : frame# 1, timestamp diff[ 0.000].
frame [01]. depth : frame# 9, timestamp diff [ 33.345]. color : frame# 1, timestamp diff[ 0.000].
frame [02]. depth : frame# 9, timestamp diff [ 0.000]. color : frame# 2, timestamp diff[ 84.559].
frame [03]. depth : frame# 10, timestamp diff [ 33.324]. color : frame# 3, timestamp diff[ 79.647].
frame [04]. depth : frame# 11, timestamp diff [ 33.342]. color : frame# 3, timestamp diff[ 0.000].
frame [05]. depth : frame# 12, timestamp diff [ 33.245]. color : frame# 3, timestamp diff[ 0.000].
frame [06]. depth : frame# 13, timestamp diff [ 33.420]. color : frame# 4, timestamp diff[ 80.300].
frame [07]. depth : frame# 14, timestamp diff [ 33.290]. color : frame# 4, timestamp diff[ 0.000].
frame [08]. depth : frame# 15, timestamp diff [ 33.371]. color : frame# 5, timestamp diff[ 77.857].
frame [09]. depth : frame# 16, timestamp diff [ 33.316]. color : frame# 5, timestamp diff[ 0.000].
frame [10]. depth : frame# 17, timestamp diff [ 33.298]. color : frame# 6, timestamp diff[ 79.399].
frame [11]. depth : frame# 18, timestamp diff [ 33.271]. color : frame# 6, timestamp diff[ 0.000].
frame [12]. depth : frame# 19, timestamp diff [ 33.389]. color : frame# 6, timestamp diff[ 0.000].
frame [13]. depth : frame# 20, timestamp diff [ 33.286]. color : frame# 7, timestamp diff[ 80.174].
frame [14]. depth : frame# 21, timestamp diff [ 33.298]. color : frame# 7, timestamp diff[ 0.000].
frame [15]. depth : frame# 22, timestamp diff [ 33.299]. color : frame# 8, timestamp diff[ 77.524].
frame [16]. depth : frame# 23, timestamp diff [ 33.328]. color : frame# 8, timestamp diff[ 0.000].
frame [17]. depth : frame# 24, timestamp diff [ 33.327]. color : frame# 9, timestamp diff[ 78.921].
frame [18]. depth : frame# 26, timestamp diff [ 66.715]. color : frame# 9, timestamp diff[ 0.000].
frame [19]. depth : frame# 26, timestamp diff [ 0.000]. color : frame# 10, timestamp diff[ 79.074].
frame [20]. depth : frame# 27, timestamp diff [ 66.625]. color : frame# 10, timestamp diff[ 0.000].
frame [21]. depth : frame# 28, timestamp diff [ 33.279]. color : frame# 11, timestamp diff[ 77.333].
frame [22]. depth : frame# 29, timestamp diff [ 33.361]. color : frame# 11, timestamp diff[ 0.000].
frame [23]. depth : frame# 30, timestamp diff [ 33.278]. color : frame# 12, timestamp diff[ 77.279].
frame [24]. depth : frame# 32, timestamp diff [ 66.709]. color : frame# 12, timestamp diff[ 0.000].
frame [25]. depth : frame# 32, timestamp diff [ 0.000]. color : frame# 13, timestamp diff[ 79.093].
frame [26]. depth : frame# 33, timestamp diff [ 66.653]. color : frame# 13, timestamp diff[ 0.000].
frame [27]. depth : frame# 34, timestamp diff [ 33.331]. color : frame# 14, timestamp diff[ 77.029].
frame [28]. depth : frame# 35, timestamp diff [ 33.342]. color : frame# 14, timestamp diff[ 0.000].
frame [29]. depth : frame# 36, timestamp diff [ 33.256]. color : frame# 15, timestamp diff[ 77.124].
frame [30]. depth : frame# 38, timestamp diff [ 66.669]. color : frame# 15, timestamp diff[ 0.000].
frame [31]. depth : frame# 38, timestamp diff [ 0.000]. color : frame# 16, timestamp diff[ 78.807].
frame [32]. depth : frame# 39, timestamp diff [ 66.663]. color : frame# 16, timestamp diff[ 0.000].
frame [33]. depth : frame# 40, timestamp diff [ 33.272]. color : frame# 17, timestamp diff[ 76.747].
frame [34]. depth : frame# 41, timestamp diff [ 33.313]. color : frame# 17, timestamp diff[ 0.000].
frame [35]. depth : frame# 42, timestamp diff [ 33.343]. color : frame# 18, timestamp diff[ 77.459].
frame [36]. depth : frame# 44, timestamp diff [ 66.636]. color : frame# 18, timestamp diff[ 0.000].
frame [37]. depth : frame# 44, timestamp diff [ 0.000]. color : frame# 19, timestamp diff[ 78.811].
frame [38]. depth : frame# 45, timestamp diff [ 66.666]. color : frame# 19, timestamp diff[ 0.000].
frame [39]. depth : frame# 46, timestamp diff [ 33.352]. color : frame# 20, timestamp diff[ 76.756].
frame [40]. depth : frame# 47, timestamp diff [ 33.290]. color : frame# 20, timestamp diff[ 0.000].
frame [41]. depth : frame# 48, timestamp diff [ 33.350]. color : frame# 21, timestamp diff[ 77.686].
frame [42]. depth : frame# 50, timestamp diff [ 66.621]. color : frame# 21, timestamp diff[ 0.000].
frame [43]. depth : frame# 50, timestamp diff [ 0.000]. color : frame# 22, timestamp diff[ 79.061].
frame [44]. depth : frame# 51, timestamp diff [ 66.636]. color : frame# 22, timestamp diff[ 0.000].
frame [45]. depth : frame# 52, timestamp diff [ 33.439]. color : frame# 23, timestamp diff[ 79.766].
frame [46]. depth : frame# 54, timestamp diff [ 66.612]. color : frame# 23, timestamp diff[ 0.000].
frame [47]. depth : frame# 55, timestamp diff [ 33.358]. color : frame# 24, timestamp diff[ 79.272].
frame [48]. depth : frame# 56, timestamp diff [ 33.286]. color : frame# 24, timestamp diff[ 0.000].
frame [49]. depth : frame# 57, timestamp diff [ 33.272]. color : frame# 25, timestamp diff[ 77.015].
frame [50]. depth : frame# 58, timestamp diff [ 33.342]. color : frame# 25, timestamp diff[ 0.000].
frame [51]. depth : frame# 59, timestamp diff [ 33.314]. color : frame# 26, timestamp diff[ 78.957].
frame [52]. depth : frame# 61, timestamp diff [ 66.621]. color : frame# 26, timestamp diff[ 0.000].
frame [53]. depth : frame# 62, timestamp diff [ 33.332]. color : frame# 27, timestamp diff[ 79.217].
frame [54]. depth : frame# 63, timestamp diff [ 33.395]. color : frame# 27, timestamp diff[ 0.000].
frame [55]. depth : frame# 64, timestamp diff [ 33.244]. color : frame# 28, timestamp diff[ 77.119].
frame [56]. depth : frame# 65, timestamp diff [ 33.390]. color : frame# 28, timestamp diff[ 0.000].
frame [57]. depth : frame# 66, timestamp diff [ 33.267]. color : frame# 29, timestamp diff[ 79.002].
frame [58]. depth : frame# 68, timestamp diff [ 66.650]. color : frame# 29, timestamp diff[ 0.000].
frame [59]. depth : frame# 69, timestamp diff [ 33.329]. color : frame# 30, timestamp diff[ 79.357].
frame [60]. depth : frame# 70, timestamp diff [ 33.302]. color : frame# 30, timestamp diff[ 0.000].
frame [61]. depth : frame# 71, timestamp diff [ 33.337]. color : frame# 31, timestamp diff[ 77.202].
frame [62]. depth : frame# 72, timestamp diff [ 33.421]. color : frame# 31, timestamp diff[ 0.000].
frame [63]. depth : frame# 73, timestamp diff [ 33.220]. color : frame# 32, timestamp diff[ 80.378].
frame [64]. depth : frame# 75, timestamp diff [ 66.700]. color : frame# 32, timestamp diff[ 0.000].
frame [65]. depth : frame# 75, timestamp diff [ 0.000]. color : frame# 33, timestamp diff[ 79.144].
frame [66]. depth : frame# 76, timestamp diff [ 66.676]. color : frame# 33, timestamp diff[ 0.000].
frame [67]. depth : frame# 77, timestamp diff [ 33.299]. color : frame# 34, timestamp diff[ 76.791].
frame [68]. depth : frame# 78, timestamp diff [ 33.320]. color : frame# 34, timestamp diff[ 0.000].
frame [69]. depth : frame# 79, timestamp diff [ 33.307]. color : frame# 35, timestamp diff[ 77.624].
frame [70]. depth : frame# 81, timestamp diff [ 66.626]. color : frame# 35, timestamp diff[ 0.000].
frame [71]. depth : frame# 81, timestamp diff [ 0.000]. color : frame# 36, timestamp diff[ 79.260].
frame [72]. depth : frame# 82, timestamp diff [ 66.646]. color : frame# 36, timestamp diff[ 0.000].
frame [73]. depth : frame# 83, timestamp diff [ 33.394]. color : frame# 37, timestamp diff[ 77.768].
frame [74]. depth : frame# 85, timestamp diff [ 66.619]. color : frame# 37, timestamp diff[ 0.000].
frame [75]. depth : frame# 86, timestamp diff [ 66.593]. color : frame# 38, timestamp diff[ 88.747].
frame [76]. depth : frame# 87, timestamp diff [ 33.390]. color : frame# 39, timestamp diff[ 85.789].
frame [77]. depth : frame# 89, timestamp diff [ 66.657]. color : frame# 39, timestamp diff[ 0.000].
frame [78]. depth : frame# 90, timestamp diff [ 66.644]. color : frame# 40, timestamp diff[ 85.535].
frame [79]. depth : frame# 91, timestamp diff [ 33.251]. color : frame# 40, timestamp diff[ 0.000].
frame [80]. depth : frame# 92, timestamp diff [ 33.373]. color : frame# 41, timestamp diff[ 82.892].
frame [81]. depth : frame# 93, timestamp diff [ 33.327]. color : frame# 41, timestamp diff[ 0.000].
frame [82]. depth : frame# 94, timestamp diff [ 33.271]. color : frame# 42, timestamp diff[ 84.706].
frame [83]. depth : frame# 96, timestamp diff [ 66.659]. color : frame# 42, timestamp diff[ 0.000].
frame [84]. depth : frame# 97, timestamp diff [ 33.315]. color : frame# 43, timestamp diff[ 85.439].
frame [85]. depth : frame# 98, timestamp diff [ 33.383]. color : frame# 43, timestamp diff[ 0.000].
frame [86]. depth : frame# 99, timestamp diff [ 33.276]. color : frame# 44, timestamp diff[ 85.098].
frame [87]. depth : frame# 101, timestamp diff [ 66.709]. color : frame# 44, timestamp diff[ 0.000].
frame [88]. depth : frame# 102, timestamp diff [ 33.332]. color : frame# 45, timestamp diff[ 84.825].
frame [89]. depth : frame# 103, timestamp diff [ 33.292]. color : frame# 45, timestamp diff[ 0.000].
frame [90]. depth : frame# 104, timestamp diff [ 33.360]. color : frame# 46, timestamp diff[ 86.517].
frame [91]. depth : frame# 106, timestamp diff [ 66.583]. color : frame# 46, timestamp diff[ 0.000].
frame [92]. depth : frame# 107, timestamp diff [ 33.349]. color : frame# 47, timestamp diff[ 79.108].
frame [93]. depth : frame# 108, timestamp diff [ 33.351]. color : frame# 47, timestamp diff[ 0.000].
frame [94]. depth : frame# 109, timestamp diff [ 33.289]. color : frame# 48, timestamp diff[ 77.273].
frame [95]. depth : frame# 110, timestamp diff [ 33.363]. color : frame# 48, timestamp diff[ 0.000].
frame [96]. depth : frame# 111, timestamp diff [ 33.275]. color : frame# 49, timestamp diff[ 78.367].
frame [97]. depth : frame# 113, timestamp diff [ 66.624]. color : frame# 49, timestamp diff[ 0.000].
frame [98]. depth : frame# 114, timestamp diff [ 33.334]. color : frame# 50, timestamp diff[ 79.101].
frame [99]. depth : frame# 115, timestamp diff [ 33.374]. color : frame# 50, timestamp diff[ 0.000].

[dorodnic]

Hi @heavyman1
I think this can be caused by auto-exposure.
Can you try replacing pipe.start(config) in your code with:

auto profile = pipe.start(config);
auto color = profile.get_device().query_sensors()[1];
color.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 0);
color.set_option(RS2_OPTION_EXPOSURE, 100);

[heavyman1]

Result is same.
But even if frame number is different with depth and color, frames from same frameset are acquired at the same time?

[dorodnic]

Here are the rules for Depth-RGB sync:
a. Depth and RGB have to be in same actual FPS. This can be forced by disabling exposure_priority control (this way auto-exposure will not reduce FPS).
b. Depth frame and RGB frame get their timestamps in different parts of the (hardware)pipe. I believe depth in start frame and RGB mid exposure. This means that visually the frames should be the same, but TS will never be exactly the same.

Also, it is a good idea to check frame.get_timestamp_domain() - this should be hardware_clock.

[heavyman1]

frame timestamp domain is system time.
how can I set timestamp domain to hardware_clock?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @heavyman1,

Once the metadata is enabled as the description in below link, I think you should have the timestamp domain based on hardware clock.
https://github.com/IntelRealSense/librealsense/blob/master/doc/frame_metadata.md

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @heavyman1,

Is your issue resolved? Still need any support for this topic?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @heavyman1,

any update?