Slow simGetImages function with Scene and DepthPlanar

[DanieleMarchisotti]

Hi,

I am opening this issue about the time required to get images in Cosys-Airsim.
The simGetImages function goes very slow when saving Scene and DepthPlanar images.

This is the code I use to get images:

scene_requests = [
    airsim.ImageRequest("0", airsim.ImageType.Scene, False, False), # camera_name, image_type, pixels_as_float, compress
    airsim.ImageRequest("0", airsim.ImageType.DepthPlanar, True, False) # camera_name, image_type, pixels_as_float, compress
]
getimages_time_start = time.time()
scene_responses = self.client.simGetImages(scene_requests)
getimages_time_stop = time.time()
print("Time taken to get images: ", getimages_time_stop - getimages_time_start)
for idx, response in enumerate(scene_responses):
    if response.pixels_as_float:
        filename = self.save_dir + "/depth/" + str(response.time_stamp)
        pfm_time_start = time.time()
        self.depth = airsim.get_pfm_array(response)
        pfm_time_end = time.time()
        print("Time taken to get pfm image: ", pfm_time_end - pfm_time_start)
        airsim.write_pfm(os.path.normpath(filename + '.pfm'), self.depth)
    else: #uncompressed array
        filename = self.save_dir + "/rgb/" + str(response.time_stamp)
        rgb_timestamp = response.time_stamp
        rgb_time_start = time.time()
        img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) # get numpy array
        rgb_time_end = time.time()
        print("Time taken to get rgb array: ", rgb_time_end - rgb_time_start)
        img_rgb = img1d.reshape(response.height, response.width, 3)
        if self.ue_ver == 5:
            img_rgb = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2RGB)
        cv2.imwrite(os.path.normpath(filename + '.png'), img_rgb) # write to png

and the performances with CityPark environment are the following, 2-3FPS (similar but slightly worse performance are obtained with ElectricDreams environment):

Captured frame:  0
Time taken to get images:  0.4015023708343506
Time taken to get rgb array:  0.0005230903625488281
Time taken to get pfm image:  0.03378462791442871
Time taken to capture data:  0.5081825256347656
Captured frame:  1
Time taken to get images:  0.56178879737854
Time taken to get rgb array:  0.0004322528839111328
Time taken to get pfm image:  0.03719067573547363
Time taken to capture data:  0.6768307685852051
Captured frame:  2
Time taken to get images:  0.2557053565979004
Time taken to get rgb array:  0.0004165172576904297
Time taken to get pfm image:  0.03760504722595215
Time taken to capture data:  0.36663150787353516

while with blocks environment I get the following:

Captured frame:  0
Time taken to get images:  0.04610753059387207
Time taken to get rgb array:  0.001096963882446289
Time taken to capture data:  0.06566596031188965
Captured frame:  1
Time taken to get images:  0.0488286018371582
Time taken to get rgb array:  0.0011472702026367188
Time taken to capture data:  0.0663454532623291
Captured frame:  2
Time taken to get images:  0.04476642608642578
Time taken to get rgb array:  0.0011594295501708984
Time taken to capture data:  0.06387186050415039

which is better (around 20 FPS).

Do you know how could I increase speed of simGetImages? Is it a matter of Cosys-Airsim or UE environments?

I have seen a similar issue also in Airsim microsoft/AirSim#3566

[WouterJansen]

The more camera layers you request, the slower it will get. This has been a long standing issue for AirSim related projects and not easy to fix I believe.

[hail-linda]

Hi @DanieleMarchisotti @WouterJansen

i have same problem and solved(partly). Here is what i did.

my application is that i need capture six scene in 1000px*1000px from [front,back,up,bottom,left,right] and make them a panarama(UTM).

• in PC windows 11 with cpu:i9-11900k gpu:nvidia RTX 4080
• in UE 5.5
• in cosys-airsim 3.3
• call python api for drone control and scenes capture
• in a complate self-build Level(in UE)

UE will render each frame in my work (also in yours), which is 6 of sceneCpature2d and 1of main view.
but usually we only need few of them. like 2~3from over 30fps.
what i did is ask UE DO NOT render each sceneCapture2d during the play.
i made a blueprint as below:

for each 10 seconds:

• get All actors of Class.
  • For Each Lop (actors)
    • get all componements by class == sceneCapture2d
      • for each loop
        • set bCaptureEveryFrame (False)
        • set bCaptureOnMovement (False)

Result:

• FPS from 10~12 to 58~63
• will stuck when collection

Here is my game Level

Happy for further discussion.