Very slow image grabs due to obsolete pure Python msgpack

[patrickmineault]

Bug report

• AirSim Version/#commit: 1.4.0
• UE/Unity version: tested using UE NH environment from airsim-1.4.0
• autopilot version: -
• OS Version: Windows 10

What's the issue you encountered?

AirSim's python bindings require msgpack-rpc-python, which depends on an obsolete version of msgpack (related issue: #3251). That old version of msgpack doesn't compile, so it falls back to a pure Python implementation which is very slow - in my workflow, more than half the time is spent in msgpack rather than, e.g. rendering.

Settings

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision",
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 112,
        "Height": 112,
        "FOV_Degrees": 120,
        "AutoExposureSpeed": 100
      }
    ]
  }
}

How can the issue be reproduced?

1. Create a new, clean conda environment, e.g. conda create --name test python=3.8
2. Activate it, conda activate test
3. pip install airsim
4. Run the NH sample
5. Create a file test.py with the following code:

import airsim

client = airsim.VehicleClient()
client.confirmConnection()

for i in range(10):
    responses = client.simGetImages([
        airsim.ImageRequest("front_center", airsim.ImageType.Scene, False, False),
    ])

6. Run this file under the profiler: python -m cProfile test.py. Scroll down and you will see many calls to fallback.py, e.g.:

181988/36 0.202 0.000 0.884 0.025 fallback.py:380(_fb_unpack)

fallback.py is the pure Python fallback from msgpack: https://github.com/msgpack/msgpack-python/blob/master/msgpack/fallback.py.

A quick way of figuring out if msgpack is installed with compiled bindings: from msgpack import _cmsgpack. If it's not, things are going to be real slow.

Include full error message in text form

No errors, just slowness

Workaround

I found a nasty workaround. I uninstalled airsim, msgpack-python and mspack-rpc-python. I then pip installed from git this upgraded fork of msgpack-rpc-python: https://github.com/tbelhalfaoui/msgpack-rpc-python/tree/fix-msgpack-dep. Then I git cloned airsim, locally installed in editable mode, and I patched client.py to remove the input and output encodings (those were removed in msgpack 1.0). Now my pipeline is much much faster.

What's better than filing an issue? Filing a pull request :).

[rajat2004]

I tested this and I'm not seeing any fallback.py in the profiler output. See the 2 attached files, the second is sorted by total time -
outputs_unsorted.txt
outputs.txt

Did the _cmsgpack test as well, and it's not present though, from the Readme it says that fallback.py is to be used for Python 2. Note that I'm not using Anaconda but running the system python (3.6.9) with venv

[patrickmineault]

Can you try with Python 3.8? That's the version I used. Regarding Python 2, the README does say that fallback.py may be used in case Python 2 is used; however, there are a larger set of circumstances where fallback.py will be used, including anytime there's an ImportError import _cmsgpack:

https://github.com/msgpack/msgpack-python/blob/02e1f7623cd8d0fcd4763d542fc60e2957ee2046/msgpack/__init__.py#L16

[huyaoyu]

Encountered the same issue that @patrickmineault has found.

On Windows 10 with:

• Unreal Editor 4.26.2 (built from source)
• Unreal Editor 4.25.4 (installed from launcher)
• Python3.9.2 (Anaconda)
• Python2.7.18 (Installed from official site)

Once I make a request through simGetImages() with pixels_as_float set to True, the response time becomes two orders of magnitude slower (0.06s vs. 8s). And also, as discussed by @patrickmineault it seems to be an issue of the msgpack-rpc-python package, because, when debugging the AirSim code, the float image finishes copy the pixel long before the Python client receives the response.

The solution is exactly as @patrickmineault has suggested. First uninstall msgpack-rpc-python and msgpack-python. Then install the latest msgpack with (check out here)

pip install msgpack

Check the installation by from msgpack import _cmsgpack

Then clone https://github.com/tbelhalfaoui/msgpack-rpc-python/tree/fix-msgpack-dep.

Remember to checkout the branch fix-msgpack-dep. Then install the cloned msgpack-rpc-python by python setup.py install.

Then modify the client.py file of AirSim Python client. Change the msgpackrpc.Client() line inside the __init__() of class VehicleClient into

class VehicleClient:
    def __init__(self, ip = "", port = 41451, timeout_value = 3600):
        if (ip == ""):
            ip = "127.0.0.1"
        # self.client = msgpackrpc.Client(msgpackrpc.Address(ip, port), timeout = timeout_value, pack_encoding = 'utf-8', unpack_encoding = 'utf-8')
        self.client = msgpackrpc.Client(msgpackrpc.Address(ip, port), timeout = timeout_value)

Upon these modifications, the execution of the AirSim Python client is mush faster. Basically the same speed with pixels_as_float set to be False.

[rajat2004]

I tested this again with #3889, and am getting the same performance with changing the msgpack library, ~36 FPS with Blocks binary for default camera. Some more details -

Ubuntu 18.04
Python 3.6.9 (System default)
AirSim 1.5.0 release Blocks env

msgpack env (from msgpack import _cmsgpack test passes)-

msgpack                       1.0.2
msgpack-rpc-python            0.4

Normal airsim env (_cmsgpack doesn't work) -

msgpack-python                0.5.6
msgpack-rpc-python            0.4.1

Will need to test further whether fallback.py is getting used somehow

[varadVaidya]

@rajat2004 did you find any solution to speed it up? i am also facing the same issue