Struggling to configure Lithops for AWS Lambda & S3

[rsignell]

It's my first time using Lithops, and I'm trying to configure it for AWS lambda & s3.

I've installed lithops via:

pip install lithops[aws]

and I've set my AWS credential pair set as environment vars.

My .lithops_config looks like this:

lithops:
    backend: aws_lambda
    storage: aws_s3

aws:
    region: us-west-2

aws_lambda:
    execution_role: arn:aws:iam::097535060392:role/lambdaLithopsExecutionRole

but when I try running the test command:

lithops hello -b aws_lambda -s aws_s3

I get back:

$ lithops hello -b aws_lambda -s aws_s3
2024-06-09 11:33:39,912 [INFO] config.py:139 -- Lithops v3.4.1 - Python3.11
2024-06-09 11:33:40,014 [INFO] aws_s3.py:59 -- S3 client created - Region: us-west-2
2024-06-09 11:33:41,098 [INFO] aws_lambda.py:97 -- AWS Lambda client created - Region: us-west-2
2024-06-09 11:33:41,099 [INFO] invokers.py:107 -- ExecutorID 807111-0 | JobID A000 - Selected Runtime: default-runtime-v311 - 256MB
2024-06-09 11:33:41,208 [INFO] invokers.py:115 -- Runtime default-runtime-v311 with 256MB is not yet deployed
2024-06-09 11:33:41,209 [INFO] aws_lambda.py:388 -- Deploying runtime: default-runtime-v311 - Memory: 256 - Timeout: 180
2024-06-09 11:33:41,638 [INFO] aws_lambda.py:187 -- Creating lambda layer for runtime default-runtime-v311
Traceback (most recent call last):
File "/home/rsignell/miniforge3/envs/pangeo/bin/lithops", line 8, in
sys.exit(lithops_cli())
^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/scripts/cli.py", line 210, in hello
fexec.call_async(hello, username)
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/executors.py", line 193, in call_async
runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/invokers.py", line 116, in select_runtime
runtime_meta = self.compute_handler.deploy_runtime(self.runtime_name, runtime_memory, runtime_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/serverless.py", line 84, in deploy_runtime
return self.backend.deploy_runtime(runtime_name, memory, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 527, in deploy_runtime
self._deploy_default_runtime(runtime_name, memory, timeout)
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 393, in _deploy_default_runtime
layer_arn = self._create_layer(runtime_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 248, in _create_layer
response = self.lambda_client.publish_layer_version(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the PublishLayerVersion operation: Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist.

What am I missing?

[abourramouss]

Hello!

So I think you took the instructions from here: https://lithops-cloud.github.io/docs/source/compute_config/aws_lambda.html

What I do is create a file under ~/.lithops/config, using the following format:

lithops:
    backend: aws_lambda
    storage: aws_s3

aws:
    access_key_id: <AWS_ACCESS_KEY_ID>
    secret_access_key: <AWS_SECRET_ACCESS_KEY>
    region: <REGION_NAME>

aws_lambda:
    execution_role: <EXECUTION_ROLE_ARN>

That's explained better in these docs: https://lithops-cloud.github.io/docs/source/compute_config/aws_lambda.html and https://lithops-cloud.github.io/docs/source/storage_config/aws_s3.html

I hope you find it helpful!

[rsignell]

@abourramouss , it seems that your config is the same except:

• you are specifying your aws credentials keypair explicitly in the file (I'm specifying mine through environment variables)
• you have your config in ~/.lithops/config, while I have mine in the directory I'm running my lithops command from (and called .lithops_config.

According to the documentation there should be no differences.

And indeed when I tried moving the config to ~/.lithops/config and adding the credentials explicitly, I get the same exact error.

It seems there must be some other problem, do you agree?

[abourramouss]

Yes, I completly agree.

After looking at the log, it looks that authentication might be happening since you're getting a NoSuchKey. S3 Error Message: The specified key does not exist.. Given the error, I think we can discard credentials as an issue.

It looks like there's something going on when building the runtime, although am not too sure of what it might be, we might get some hints if you execute:

lithops hello -b aws_lambda -s aws_s3 --debug

@JosepSampe any ideas of what might be going on?

[rsignell]

Here's the output with debug on:

(pangeo) rsignell@OSC:~/.lithops$ lithops hello -b aws_lambda -s aws_s3 --debug
2024-06-09 17:24:50,725 [INFO] config.py:139 -- Lithops v3.4.1 - Python3.11
2024-06-09 17:24:50,725 [DEBUG] config.py:101 -- Loading configuration from /home/rsignell/.lithops/config
2024-06-09 17:24:50,733 [DEBUG] config.py:179 -- Loading Serverless backend module: aws_lambda
2024-06-09 17:24:50,789 [DEBUG] config.py:220 -- Loading Storage backend module: aws_s3
2024-06-09 17:24:50,791 [DEBUG] aws_s3.py:36 -- Creating AWS S3 Client
2024-06-09 17:24:50,845 [INFO] aws_s3.py:59 -- S3 client created - Region: us-west-2
2024-06-09 17:24:51,429 [DEBUG] aws_lambda.py:53 -- Creating AWS Lambda client
2024-06-09 17:24:51,652 [INFO] aws_lambda.py:97 -- AWS Lambda client created - Region: us-west-2
2024-06-09 17:24:51,654 [DEBUG] invokers.py:93 -- ExecutorID e5c9a4-0 - Invoker initialized. Max workers: 1000
2024-06-09 17:24:51,654 [DEBUG] invokers.py:295 -- ExecutorID e5c9a4-0 - Serverless invoker created
2024-06-09 17:24:51,655 [DEBUG] executors.py:148 -- Function executor for aws_lambda created with ID: e5c9a4-0
2024-06-09 17:24:51,655 [INFO] invokers.py:107 -- ExecutorID e5c9a4-0 | JobID A000 - Selected Runtime: default-runtime-v311 - 256MB
2024-06-09 17:24:51,655 [DEBUG] storage.py:483 -- Runtime metadata not found in local cache. Retrieving it from storage
2024-06-09 17:24:51,655 [DEBUG] storage.py:487 -- Trying to download runtime metadata from: aws_s3://lithops-us-west-2-4o4w/lithops.runtimes/aws_lambda/3.4.1/us-west-2/lithops-worker-4o4w-341-d0be8ec6dc.meta.json
2024-06-09 17:24:51,767 [DEBUG] storage.py:509 -- Runtime metadata not found in storage
2024-06-09 17:24:51,767 [INFO] invokers.py:115 -- Runtime default-runtime-v311 with 256MB is not yet deployed
2024-06-09 17:24:51,768 [INFO] aws_lambda.py:388 -- Deploying runtime: default-runtime-v311 - Memory: 256 - Timeout: 180
2024-06-09 17:24:51,768 [DEBUG] aws_lambda.py:297 -- Listing lambda layers
2024-06-09 17:24:52,135 [DEBUG] aws_lambda.py:301 -- Listed 0 layers
2024-06-09 17:24:52,135 [INFO] aws_lambda.py:187 -- Creating lambda layer for runtime default-runtime-v311
2024-06-09 17:24:52,136 [DEBUG] aws_lambda.py:199 -- Creating "layer builder" function
2024-06-09 17:25:17,720 [DEBUG] aws_lambda.py:155 -- "lithops_v341_4o4w_layer_builder_512MB" function is being deployed... (status: Pending)
2024-06-09 17:25:18,041 [DEBUG] aws_lambda.py:165 -- Ok --> function "lithops_v341_4o4w_layer_builder_512MB" is active
2024-06-09 17:25:18,041 [DEBUG] aws_lambda.py:220 -- OK --> Created "layer builder" function default-runtime-v311
2024-06-09 17:25:18,041 [DEBUG] aws_lambda.py:230 -- Invoking "layer builder" function
2024-06-09 17:25:50,892 [DEBUG] aws_lambda.py:233 -- OK --> Layer lithops_v341_4o4w_default-runtime-v311_layer built
2024-06-09 17:25:50,892 [DEBUG] aws_lambda.py:239 -- Trying to delete "layer builder" function
2024-06-09 17:25:51,414 [DEBUG] aws_lambda.py:247 -- Creating layer lithops_v341_4o4w_default-runtime-v311_layer ...
Traceback (most recent call last):
File "/home/rsignell/miniforge3/envs/pangeo/bin/lithops", line 8, in
sys.exit(lithops_cli())
^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/scripts/cli.py", line 210, in hello
fexec.call_async(hello, username)
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/executors.py", line 193, in call_async
runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/invokers.py", line 116, in select_runtime
runtime_meta = self.compute_handler.deploy_runtime(self.runtime_name, runtime_memory, runtime_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/serverless.py", line 84, in deploy_runtime
return self.backend.deploy_runtime(runtime_name, memory, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 527, in deploy_runtime
self._deploy_default_runtime(runtime_name, memory, timeout)
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 393, in _deploy_default_runtime
layer_arn = self._create_layer(runtime_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 248, in _create_layer
response = self.lambda_client.publish_layer_version(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rsignell/miniforge3/envs/pangeo/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the PublishLayerVersion operation: Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist.

[rsignell]

Okay, this is kind of embarrassing, but it turns out the policy was not attached to the role. :(
Once attached (per the instructions!), the command works fine!

[abourramouss]

Okay, this is kind of embarrassing, but it turns out the policy was not attached to the role. :( Once attached (per the instructions!), the command works fine!

Nothing embarassing about that! Glad you've found the issue.