Dataflow: pictures & metadata from systems to database

[JonasBchrt]

Hi @JimBacon ,

I thought I just give you a bit of an overview of what we need in the cloud to get data (e.g., a picture with metadata) sent from a device to the cloud.
The way I collect data right now, I can either send data straight to a bare HTTPS endpoint whenever a picture arrives or send it to pre-configured AWS endpoint (see available options in the picture below). I think FDRI are using IoT Analytics, but for us, Lambda may be the way to go (or stick with bare HTTPS endpoint, which is what I am doing right now: https://github.com/NERC-CEH/ami-mini-db).

Either way, all I'd need from "you" to send a picture, is a static URL. Obviously, I can just test this myself since it's not rocket science, but ultimately, we'd need this integrated into the main setup.

Either way, the data would arrive as JSON with some metadata and likely a binary payload (the picture).

You could also imagine receiving certain status events this way, which then wouldn't contain a binary payload.

[JimBacon]

Interesting to see you can send http requests direct to a Lambda function. However, binary payloads have to be base64 encoded which is not so good for performance and bandwidth of a mobile device. I propose we use the API Gateway which supports a binary payload as well as offering numerous other features. https://docs.aws.amazon.com/lambda/latest/dg/furls-http-invoke-decision.html. From your perspective this will look like a bare HTTPS endpoint

[JonasBchrt]

Absolutely, we should send binary and not base64 encoded data. However, right now, I've got another layer between the devices and the input pipeline, mostly for security and simplicity. Having devices in the field talking directly to a hard-coded URL is a bit of a security night-mare. The current setup saves me from having to do key exchanges, encryption, and other low-level stuff myself. So, technically, we are just looking at server-to-server communication rather than device-to-server communication here and I could do (almost) any sort of message transformation on the way in the cloud, e.g., binary -> base64.
While this would technically allow to use AWS Lambda without having to worry much about the bandwidth constraints that you mention, I think a bare HTTPS endpoint is still attractive. (Plus provides flexibility in the future.)
I will send you some example messages as the flow from the device to the cloud (currently just a webhook) soon.

[JonasBchrt]

As an example, @JimBacon , here are six consecutive JSON files coming in wirelessly from a device (and being routed to an HTTPS endpoint). They have a number of fields (which I could add/remove/modify) with payload carrying the base64 encoded binary picture data. The picture is transmitted in chunks of 100-something kilobytes. I.e., after decoding, the payload needs to be concatenated to actually form a picture (the one below to be specific).
The (dummy) hardware device ID would be dev:1234567812345678 here. (It's always dev: followed by 16 digits.)
The body field contains some information that helps reconstruction (probably needs a bit more work).
chunk-0.json
chunk-4.json
chunk-3.json
chunk-2.json
chunk-1.json
chunk-5.json

Here's some (potentially not relevant) Python code to reconstruct the picture from the chunks:

import json
import base64
from pathlib import Path
from PIL import Image
import io

def show_image_from_chunks(folder="data", num_chunks=6):
    combined_bytes = b""

    for i in range(num_chunks):
        file_path = Path(folder) / f"chunk-{i}.json"
        with open(file_path, "r", encoding="utf-8") as f:
            data = json.load(f)
        if "payload" not in data:
            raise KeyError(f"'payload' field missing in {file_path}")
        
        # Decode and append
        combined_bytes += base64.b64decode(data["payload"])

    # Open as image
    image = Image.open(io.BytesIO(combined_bytes))
    image.show()

if __name__ == "__main__":
    show_image_from_chunks()