Set up CI/CD with Chalice for AWS Lambda functions

[sgbaird]

Copying over the discussion from #231 (comment)

---

Difficult to share Lambda function directly on AWS, so instead long-term we may switch to having the lambda function on GitHub and automatically deploy to cloud via AWS Chalice for example, per comment in:

https://stackoverflow.com/questions/58441717/how-to-share-lambda-function-with-another-user-in-organization

If I understand correctly:

• Create a new repo for the Lambda function in GH
• Edit Lambda scripts in this repo
• Set up AWS account credentials on GH secrets
• Commit to trigger GH action using Chalice to deploy new scripts on Lambda(?)

The only thing I need help would be the AWS credentials? I think I need an IAM user credentials within the AC organization?

BTW:
https://stackoverflow.com/questions/58441717/how-to-share-lambda-function-with-another-user-in-organization#comment103249681_58441810

It's atypical to create an account per developer, in my experience (if by developer, you mean a single person). It's common to see prod, dev, and test accounts (possibly one set dedicated to a product, if it's large enough, but often shared). Have added to original answer.

---

I'm ok with either option - pursue Chalice now (the steps you described are what I was thinking of too), or create a shared account like what was mentioned in that issue. I'll double check on AWS credentials. Not sure if I gave you admin permissions.

---

I thought the IAM user credentials is something other than our personal accounts, like an IAM user for the Lambda deployment with Chalice.

---

@Neil-YL - started the process. Looks like AWSLambda_FullAccess might be the necessary policy to attach.

This is where I went for creating an IAM user: https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/users based on instructions in https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html.

Going with access key so it doesn't expire.

Shared the credentials with you privately.

---

Not sure if the IAM user also needs the permit to visit S3 (for the pickle token) or there is a better practice. I will look into this while preparing the other setup for the shared managed Lambda.

---

[sgbaird]

@Neil-YL had an error missing: aws_secret_access_key. @Neil-YL also brought up that we can consider configuring with CLI user case instead of "other", though we're not sure.

[sgbaird]

Various notes:

• Only one delegated admin allowed per organization
• docs for creating admin accounts appear in multiple places such as https://docs.aws.amazon.com/lexv2/latest/dg/gs-account.html and https://docs.aws.amazon.com/streams/latest/dev/setting-up.html (not necessarily a problem)
• There are notes about assigning "programmatic access"

@Neil-YL what do you think about https://stackoverflow.com/a/65495039/13697228? (and to your point seems like CLI access might be the right setup)

You're also welcome to prototype on a personal account/organization, since things are a bit fuzzy in terms of the admin permissions.

[sgbaird]

Just added AmazonS3ReadOnlyAccess to the credentials I shared.

EDIT: I had only sent you the IAM username and secret access key (which I just called access key). I shared the Access key ID just now. For reference, the access key ID is of the form AKIAIOSFODNN7EXAMPLE and the secret access key is of the form wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY, with a corresponding IAM username (e.g., someDescriptiveName) and whatever permissions (e.g., AmazonS3ReadOnlyAccess and AWSLambda_FullAccess)

[Neil-YL]

Already created a repo for the lambda function on my GH account (will move to AC's later)
I will test Chalice with the access key locally then move to GH action.

[Neil-YL]

OK current permit is enough for Chalice, but I need some additional IAM (role) permissions for my IAM user (streamingLambda) to assign role to Lambda.

This is an example json file for the permit that the IAM user needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:*",
        "iam:CreateRole",
        "iam:PutRolePolicy",
        "iam:PassRole",
        "apigateway:*",
        "logs:*",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}

Explanation:

• "lambda:"
  Grants full access to manage AWS Lambda functions, including creating, updating, and invoking them.
• "iam:CreateRole"
  Allows Chalice to create the IAM execution role (not the IAM user) that the Lambda function will use.
• "iam:PutRolePolicy"
  Allows Chalice to attach inline policies to the IAM role (assign role policy to Lambda).
• "iam:PassRole"
  Allows assigning the created IAM role to the Lambda function. This is required so that Lambda can “assume” the role during execution.
• "apigateway:"
  Grants Chalice permission to manage API Gateway resources in case the app is deployed using an API Gateway endpoint instead of Function URL. (Testing purpose, our device.py uses Function URL though)
• "logs:"
  Grants access to create and write CloudWatch logs. This is important for Lambda to log errors or output from the function.
• "s3:GetObject"
  Allows the Lambda function to download specific objects from S3
• "s3:ListBucket"
  Allows the Lambda function to list contents of the bucket, which may be required to verify the token file exists.

[sgbaird]

Added! (via inline policy). Probably some redundancy, but at least these ones are for sure enabled

[Neil-YL]

I’ve completed the local deployment test of Chalice, except for uploading the pickle token file.

Since we don’t plan to upload the token file to GitHub, we will manually upload it to AC’s AWS S3. For this reason, we’ve decided not to grant the profile for Chalice the permission to create buckets or upload files.

[sgbaird]

Setting up following bucket (setting image is low-res but readable). Best to keep the bucket name private, even though it's restricted against public access. Bucket name must be globally unique (across all AWS accounts for a particular region like United States).

GUI makes it pretty easy to make a folder and upload the file. In the end looks like following:

[sgbaird]

Since we ran into [ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'ytb_api_utils', likely we'll need to follow https://aws.github.io/chalice/topics/packaging.html by using a chalicelib folder

[sgbaird]

Updating the lambda function via Chalice is working, but we're having some trouble with the app.py routing maybe.

Used following for testing:

{
    "action": "create",
    "cam_name": "cam-<cam-id>",
    "workflow_name": "newfunction-[undecided]-[undecided]-<cam-id>",
    "privacy_status": "public"
}

Status: Succeeded
Test Event Name: testCreate

Response:
{
  "headers": {},
  "multiValueHeaders": {},
  "statusCode": 500,
  "body": "{\"Code\":\"InternalServerError\",\"Message\":\"Unknown request.\"}"
}

[sgbaird]

Aside: Need to make sure Function URL is set to public access, otherwise we end up with RuntimeError: HTTP error occurred: 403 Client Error: Forbidden for url. On "permissions" sidebar item (e.g., available at https://<region>.console.aws.amazon.com/lambda/home?region=<region>#/functions/<function-url-name>?subtab=permissions&tab=configure):

On "Function URL" sidebar item, should display Your function URL is public. Anyone with the URL can access your function.

[Neil-YL]

Finished.

Repo: https://github.com/AccelerationConsortium/streamingLambda