[receiver/awscloudwatch] Missing log stream events

[AllanOricil]

Component(s)

receiver/awscloudwatch

What happened?

Description

1 - Past log streams aren't exported
2 - Log streams are incomplete

Steps to Reproduce

1 - create a lambda function that logs more than 15 log lines in Cloud Watch
2 - run this function several times and ensure a single log stream has more than 15 log lines
3 - in an EC2 machine, run otel-collector with this receiver

WARNING: don't forget to change NAME_OF_YOUR_LAMBDA_FUNCTION_LOG_GROUP by the name of the log group of your lambda function

 awscloudwatch:
    region: us-east-2
    logs:
      poll_interval: 10s
      max_events_per_request: 1000
      groups:
        autodiscover:
          limit: 100
          prefix: /aws/lambda/NAME_OF_YOUR_LAMBDA_FUNCTION_LOG_GROUP

Don't forget to register the receiver in the logs pipeline

service:
  telemetry:
    logs:
      level: debug
    metrics:
      address: 0.0.0.0:8888
  extensions: [health_check, zpages]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics/internal:
      receivers: [prometheus, hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp, awscloudwatch]
      processors: [batch]
      exporters: [otlp, debug]

4 - verify that logs are processed with no error. You should see something as shown below

2024-04-09T01:32:48.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:32:48.382Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:32:48.468Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 15}
2024-04-09T01:32:48.469Z	info	ResourceLog #0
Resource SchemaURL: 
Resource attributes:
     -> aws.region: Str(us-east-2)
     -> cloudwatch.log.group.name: Str(/aws/lambda/get-instances-api-function)
     -> cloudwatch.log.stream: Str(2024/04/09/[$LATEST]9a79cb34998a4037bf1e3ff5df35fea5)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:41.354 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(INIT_START Runtime Version: nodejs:18.v26	Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9
)
Attributes:
     -> id: Str(38192844104842187460300983962019760926010078374443876352)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #1
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.498 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(START RequestId: 9f7b878f-a62a-47d1-be33-9526adf64f30 Version: $LATEST
)
Attributes:
     -> id: Str(38192844130354239967420016835936622629919803937285472257)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #2
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.571 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"body":null,"headers":{"Accept":"application/json, text/plain, */*","Accept-Encoding":"gzip, deflate, br, zstd","Accept-Language":"en-GB,en-US;q=0.9,en;q=0.8,pt;q=0.7","Authorization":"eyJraWQiOiJvSUdwUUVhaFIxOWlMc05GbWhpVHpsRkNYUEx3eDcxNDN0S3hGaEFUNEJBPSIsImFsZyI6IlJTMjU2In0.eyJhdF9oYXNoIjoiUTZHdUZyLXAtREJvd3ZacElwZW81QSIsInN1YiI6IjgxNGIxNTYwLWIwNTEtNzA0MC1iNThiLWIyMThiNDJjZTU5YiIsImN1c3RvbTpzdWJkb21haW4iOiJhbGxhbm9yaWNpbCIsImNvZ25pdG86Z3JvdXBzIjpbInVzLWVhc3QtMl84QmVYSm42M2RfR29vZ2xlIiwiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIl0sImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzhCZVhKbjYzZCIsImlkZW50aXRpZXMiOlt7ImRhdGVDcmVhdGVkIjoiMTcxMTExNjcwOTcwNyIsInVzZXJJZCI6IjEwNDExNjkxMTE3NDI0NjE1NzU5NSIsInByb3ZpZGVyTmFtZSI6Ikdvb2dsZSIsInByb3ZpZGVyVHlwZSI6Ikdvb2dsZSIsImlzc3VlciI6bnVsbCwicHJpbWFyeSI6InRydWUifV0sImF1dGhfdGltZSI6MTcxMjQzNTE1MywiZXhwIjoxNzEyNzEyNzYwLCJpYXQiOjE3MTI2MjYzNjAsImp0aSI6ImZjNWE3NDkzLTFjOWQtNGUyMC1hMGViLThjYzMzYzlmMTdmMiIsImVtYWlsIjoiYWxsbGFub3JpY2lsQGdtYWlsLmNvbSIsImN1c3RvbTpwc19zdWJzY3JpcHRpb25faWQiOiJzdWJfMU94OHFWSFl3TUVKQjQ2OFFGWlNMeThHIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjdXN0b206cHNfY3VzdG9tZXJfaWQiOiJjdXNfUG1pSFowanV4RnAzTHYiLCJjdXN0b206b3JnYW5pemF0aW9uX2lkIjoiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIiwiY3VzdG9tOnBzX3JlZGlyZWN0X3VybCI6Imh0dHA6XC9cL25vZGUtcmVhZHkuY29tXC9pbnN0YW5jZS1wcm92aXNpb25pbmctam9ic1wvY2FsbGJhY2siLCJjdXN0b206b3JnYW5pemF0aW9uX25hbWUiOiJBbGxhbiBPcmljaWwiLCJjb2duaXRvOnVzZXJuYW1lIjoiR29vZ2xlXzEwNDExNjkxMTE3NDI0NjE1NzU5NSIsImN1c3RvbTpzZXR1cCI6InRydWUiLCJnaXZlbl9uYW1lIjoiQWxsYW4gR29uw6dhbHZlcyBHb21lcyIsIm9yaWdpbl9qdGkiOiIwMzI5NTkwOC1hYjIzLTQ2NzAtODVjNS1mOTQ3NjhiNmQ3OGYiLCJhdWQiOiIxazMxY2Q4NzRoMTRkNHU4aTcyNGlkNDJwbiIsImN1c3RvbTpjcmVhdGVfaW5zdGFuY2UiOiJ0cnVlIiwidG9rZW5fdXNlIjoiaWQiLCJmYW1pbHlfbmFtZSI6Ik9yaWNpbCJ9.Srb-vdOkRbkNcq0ZvHUDYT1cl9GJ-F_XM8est0yzehoOKsfD72e4_qhWcwCRGCp1ZRd9KmFxp7rhQiqzTjccXjNNSORl823c8fM3sDLSpUF6-llLvdf9zDaH5JZxOwMX-wIfJmtRCgWjYmCEJpZatPHG2M3e8jVvAFzB-EAaEs9AfR6Re5YhVtzy5aUcHUxkMw_mCtrEmg2V-zY2SEyIg5m6EMQ6J5AfKsDe3nteM28Sa0IoWOxGf5KhBySUC49RawQCfO0vYNYXC3PVJNfmTaKJ8P2g4kXtCpi11RS7ONMth2ynrPg-eWTiI_I6g9pQdvsAruomontiYrvH4kknNw","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-ASN":"269073","CloudFront-Viewer-Country":"BR","Host":"vjydkvkskl.execute-api.us-east-2.amazonaws.com","Referer":"https://node-ready.com/","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","Via":"2.0 6fe8e2d5db6a80353eb675f61c249810.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"kgg1BBFHKR4861YdGx-Kcf1-149AGIp-EUSUm9_J4MFT5WvC0cH3PA==","X-Amzn-Trace-Id":"Root=1-66149ab8-0a47130e4ff50bbf715025c0","X-Forwarded-For":"138.94.127.146, 15.158.19.244","X-Forwarded-Port":"443","X-Forwarded-Proto":"https","origin":"https://node-ready.com","sec-ch-ua":"\"Brave\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"cross-site","sec-gpc":"1"},"httpMethod":"GET","isBase64Encoded":false,"level":"info","message":"event","multiValueHeaders":{"Accept":["application/json, text/plain, */*"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8,pt;q=0.7"],"Authorization":["eyJraWQiOiJvSUdwUUVhaFIxOWlMc05GbWhpVHpsRkNYUEx3eDcxNDN0S3hGaEFUNEJBPSIsImFsZyI6IlJTMjU2In0.eyJhdF9oYXNoIjoiUTZHdUZyLXAtREJvd3ZacElwZW81QSIsInN1YiI6IjgxNGIxNTYwLWIwNTEtNzA0MC1iNThiLWIyMThiNDJjZTU5YiIsImN1c3RvbTpzdWJkb21haW4iOiJhbGxhbm9yaWNpbCIsImNvZ25pdG86Z3JvdXBzIjpbInVzLWVhc3QtMl84QmVYSm42M2RfR29vZ2xlIiwiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIl0sImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzhCZVhKbjYzZCIsImlkZW50aXRpZXMiOlt7ImRhdGVDcmVhdGVkIjoiMTcxMTExNjcwOTcwNyIsInVzZXJJZCI6IjEwNDExNjkxMTE3NDI0NjE1NzU5NSIsInByb3ZpZGVyTmFtZSI6Ikdvb2dsZSIsInByb3ZpZGVyVHlwZSI6Ikdvb2dsZSIsImlzc3VlciI6bnVsbCwicHJpbWFyeSI6InRydWUifV0sImF1dGhfdGltZSI6MTcxMjQzNTE1MywiZXhwIjoxNzEyNzEyNzYwLCJpYXQiOjE3MTI2MjYzNjAsImp0aSI6ImZjNWE3NDkzLTFjOWQtNGUyMC1hMGViLThjYzMzYzlmMTdmMiIsImVtYWlsIjoiYWxsbGFub3JpY2lsQGdtYWlsLmNvbSIsImN1c3RvbTpwc19zdWJzY3JpcHRpb25faWQiOiJzdWJfMU94OHFWSFl3TUVKQjQ2OFFGWlNMeThHIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjdXN0b206cHNfY3VzdG9tZXJfaWQiOiJjdXNfUG1pSFowanV4RnAzTHYiLCJjdXN0b206b3JnYW5pemF0aW9uX2lkIjoiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIiwiY3VzdG9tOnBzX3JlZGlyZWN0X3VybCI6Imh0dHA6XC9cL25vZGUtcmVhZHkuY29tXC9pbnN0YW5jZS1wcm92aXNpb25pbmctam9ic1wvY2FsbGJhY2siLCJjdXN0b206b3JnYW5pemF0aW9uX25hbWUiOiJBbGxhbiBPcmljaWwiLCJjb2duaXRvOnVzZXJuYW1lIjoiR29vZ2xlXzEwNDExNjkxMTE3NDI0NjE1NzU5NSIsImN1c3RvbTpzZXR1cCI6InRydWUiLCJnaXZlbl9uYW1lIjoiQWxsYW4gR29uw6dhbHZlcyBHb21lcyIsIm9yaWdpbl9qdGkiOiIwMzI5NTkwOC1hYjIzLTQ2NzAtODVjNS1mOTQ3NjhiNmQ3OGYiLCJhdWQiOiIxazMxY2Q4NzRoMTRkNHU4aTcyNGlkNDJwbiIsImN1c3RvbTpjcmVhdGVfaW5zdGFuY2UiOiJ0cnVlIiwidG9rZW5fdXNlIjoiaWQiLCJmYW1pbHlfbmFtZSI6Ik9yaWNpbCJ9.Srb-vdOkRbkNcq0ZvHUDYT1cl9GJ-F_XM8est0yzehoOKsfD72e4_qhWcwCRGCp1ZRd9KmFxp7rhQiqzTjccXjNNSORl823c8fM3sDLSpUF6-llLvdf9zDaH5JZxOwMX-wIfJmtRCgWjYmCEJpZatPHG2M3e8jVvAFzB-EAaEs9AfR6Re5YhVtzy5aUcHUxkMw_mCtrEmg2V-zY2SEyIg5m6EMQ6J5AfKsDe3nteM28Sa0IoWOxGf5KhBySUC49RawQCfO0vYNYXC3PVJNfmTaKJ8P2g4kXtCpi11RS7ONMth2ynrPg-eWTiI_I6g9pQdvsAruomontiYrvH4kknNw"],"CloudFront-Forwarded-Proto":["https"],"CloudFront-Is-Desktop-Viewer":["true"],"CloudFront-Is-Mobile-Viewer":["false"],"CloudFront-Is-SmartTV-Viewer":["false"],"CloudFront-Is-Tablet-Viewer":["false"],"CloudFront-Viewer-ASN":["269073"],"CloudFront-Viewer-Country":["BR"],"Host":["vjydkvkskl.execute-api.us-east-2.amazonaws.com"],"Referer":["https://node-ready.com/"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"],"Via":["2.0 6fe8e2d5db6a80353eb675f61c249810.cloudfront.net (CloudFront)"],"X-Amz-Cf-Id":["kgg1BBFHKR4861YdGx-Kcf1-149AGIp-EUSUm9_J4MFT5WvC0cH3PA=="],"X-Amzn-Trace-Id":["Root=1-66149ab8-0a47130e4ff50bbf715025c0"],"X-Forwarded-For":["138.94.127.146, 15.158.19.244"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"origin":["https://node-ready.com"],"sec-ch-ua":["\"Brave\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""],"sec-ch-ua-mobile":["?0"],"sec-ch-ua-platform":["\"macOS\""],"sec-fetch-dest":["empty"],"sec-fetch-mode":["cors"],"sec-fetch-site":["cross-site"],"sec-gpc":["1"]},"multiValueQueryStringParameters":null,"path":"/instances","pathParameters":null,"queryStringParameters":null,"requestContext":{"accountId":"845044614340","apiId":"vjydkvkskl","authorizer":{"claims":{"at_hash":"Q6GuFr-p-DBowvZpIpeo5A","aud":"1k31cd874h14d4u8i724id42pn","auth_time":"1712435153","cognito:groups":"us-east-2_8BeXJn63d_Google,d1d42950-dabb-43ef-b7cb-06784b8cf45b","cognito:username":"Google_104116911174246157595","custom:create_instance":"true","custom:organization_id":"d1d42950-dabb-43ef-b7cb-06784b8cf45b","custom:organization_name":"Allan Oricil","custom:ps_customer_id":"cus_PmiHZ0juxFp3Lv","custom:ps_redirect_url":"http://node-ready.com/instance-provisioning-jobs/callback","custom:ps_subscription_id":"sub_1Ox8qVHYwMEJB468QFZSLy8G","custom:setup":"true","custom:subdomain":"allanoricil","email":"alllanoricil@gmail.com","email_verified":"false","exp":"Wed Apr 10 01:32:40 UTC 2024","family_name":"Oricil","given_name":"Allan Gonçalves Gomes","iat":"Tue Apr 09 01:32:40 UTC 2024","identities":"{\"dateCreated\":\"1711116709707\",\"userId\":\"104116911174246157595\",\"providerName\":\"Google\",\"providerType\":\"Google\",\"issuer\":null,\"primary\":\"true\"}","iss":"https://cognito-idp.us-east-2.amazonaws.com/us-east-2_8BeXJn63d","jti":"fc5a7493-1c9d-4e20-a0eb-8cc33c9f17f2","origin_jti":"03295908-ab23-4670-85c5-f94768b6d78f","sub":"814b1560-b051-7040-b58b-b218b42ce59b","token_use":"id"}},"deploymentId":"ube8ur","domainName":"vjydkvkskl.execute-api.us-east-2.amazonaws.com","domainPrefix":"vjydkvkskl","extendedRequestId":"V70c7FCmiYcED6Q=","httpMethod":"GET","identity":{"accessKey":null,"accountId":null,"caller":null,"cognitoAuthenticationProvider":null,"cognitoAuthenticationType":null,"cognitoIdentityId":null,"cognitoIdentityPoolId":null,"principalOrgId":null,"sourceIp":"138.94.127.146","user":null,"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","userArn":null},"operationName":"This method can be used to list all instances from an account","path":"/v1/instances","protocol":"HTTP/1.1","requestId":"4b2c442e-a405-4659-8e62-6a58bb578993","requestTime":"09/Apr/2024:01:32:40 +0000","requestTimeEpoch":1712626360898,"resourceId":"t4baq8","resourcePath":"/instances","stage":"v1"},"requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","resource":"/instances","stageVariables":null,"timestamp":"2024-04-09T01:32:42.513Z"}
)
Attributes:
     -> id: Str(38192844131982194366912752325268730063823134327222042626)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #3
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.572 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"------------- headers -------------","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"}
)
Attributes:
     -> id: Str(38192844132004495112111282948410265782095782688728023043)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #4
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.572 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"origin","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"}
)
Attributes:
     -> id: Str(38192844132004495112111282948410265782095782688728023044)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #5
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"0":"http://localhost:8080","1":"https://node-ready.com","2":"https://instance-provisioning-admin-api-doc-bucket.s3.us-east-2.amazonaws.com","3":"https://instance-provisioning-api-doc-bucket.s3.us-east-2.amazonaws.com","level":"debug","message":"allowedOrigins","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003461)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #6
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":{"Access-Control-Allow-Credentials":true,"Access-Control-Allow-Headers":"Content-Type,Authorization,X-Amz-Date,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent","Access-Control-Allow-Methods":"OPTIONS,GET","Access-Control-Allow-Origin":"https://node-ready.com","Content-Security-Policy":"default-src 'none'; connect-src 'self'; img-src 'none'; script-src 'self'; style-src 'none'; object-src 'none'; frame-src 'none'","Content-Type":"application/json","Cross-Origin-Embedder-Policy":"require-corp","Cross-Origin-Opener-Policy":"same-origin","Referrer-Policy":"no-referrer","Strict-Transport-Security":"max-age=63072000; includeSubdomains; preload","X-Content-Type-Options":"nosniff","X-Frame-Options":"DENY","X-Permitted-Cross-Domain-Policies":"none","X-XSS-Protection":"1; mode=block"},"requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003462)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #7
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"------------- headers -------------","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003463)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #8
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"fetching secrets from secrets manager","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003464)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #9
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.433 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"*****","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"}
)
Attributes:
     -> id: Str(38192844151205436728046149473272519214846021945377161225)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #10
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.433 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"creating new prisma client","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"}
)
Attributes:
     -> id: Str(38192844151205436728046149473272519214846021945377161226)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #11
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.434 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"*****","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"}
)
Attributes:
     -> id: Str(38192844151227737473244680096414054933118670306883141643)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #12
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.434 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.434Z"}
)
Attributes:
     -> id: Str(38192844151227737473244680096414054933118670306883141644)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #13
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.594 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"setting up sentry","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.591Z"}
)
Attributes:
     -> id: Str(38192844154795856705009579799059769856742408147840008205)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #14
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.752 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"finding all instances for: 814b1560-b051-7040-b58b-b218b42ce59b","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.752Z"}
)
Attributes:
     -> id: Str(38192844158319374446377418255422413343820849265784913934)
Trace ID: 
Span ID: 
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-04-09T01:32:58.361Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:32:58.381Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}

Expected Result

• All log events from a log stream should be exported
• All log streams of a log group that are still in AWS must be exported

Actual Result

• Just a few log events from a log stream are exported
• Only the most recent log stream of a log group is exported

Collector version

0.97.0

Environment information

Environment

NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
SUPPORT_END="2028-03-15"

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  hostmetrics:
    scrapers:
      cpu:
      disk:
      filesystem:
      load:
      memory:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 5s
          static_configs:
            - targets: [localhost:8888]
  awscloudwatch:
    region: us-east-2
    logs:
      poll_interval: 10s
      max_events_per_request: 1000
      groups:
        autodiscover:
          limit: 100
          prefix: /aws/lambda/get-instances-api-function
processors:
  batch:
    send_batch_size: 10
    timeout: 1s
  # Ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md
  resourcedetection:
    detectors: [env, system] # include ec2 for AWS, gcp for GCP and azure for Azure.
    # Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
    timeout: 2s
    system:
      hostname_sources: [os] # alternatively, use [dns,os] for setting FQDN as host.name and os as fallback
extensions:
  health_check: {}
  zpages: {}
exporters:
  otlp:
    endpoint: "MY_OTLP_GATEWAY_IP_ADDRESS:4317"
    tls:
      insecure: true
  debug:
    # verbosity of the logging export: detailed, normal, basic
    verbosity: detailed
service:
  telemetry:
    logs:
      level: debug
    metrics:
      address: 0.0.0.0:8888
  extensions: [health_check, zpages]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics/internal:
      receivers: [prometheus, hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp, awscloudwatch]
      processors: [batch]
      exporters: [otlp, debug]

Log output

2024-04-09T01:32:28.424Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:32:38.361Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:32:38.386Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:32:48.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:32:48.382Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:32:48.468Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 15}
2024-04-09T01:32:48.469Z	info	ResourceLog #0
Resource SchemaURL: 
Resource attributes:
     -> aws.region: Str(us-east-2)
     -> cloudwatch.log.group.name: Str(/aws/lambda/get-instances-api-function)
     -> cloudwatch.log.stream: Str(2024/04/09/[$LATEST]9a79cb34998a4037bf1e3ff5df35fea5)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:41.354 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(INIT_START Runtime Version: nodejs:18.v26	Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9
)
Attributes:
     -> id: Str(38192844104842187460300983962019760926010078374443876352)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #1
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.498 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(START RequestId: 9f7b878f-a62a-47d1-be33-9526adf64f30 Version: $LATEST
)
Attributes:
     -> id: Str(38192844130354239967420016835936622629919803937285472257)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #2
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.571 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"body":null,"headers":{"Accept":"application/json, text/plain, */*","Accept-Encoding":"gzip, deflate, br, zstd","Accept-Language":"en-GB,en-US;q=0.9,en;q=0.8,pt;q=0.7","Authorization":"eyJraWQiOiJvSUdwUUVhaFIxOWlMc05GbWhpVHpsRkNYUEx3eDcxNDN0S3hGaEFUNEJBPSIsImFsZyI6IlJTMjU2In0.eyJhdF9oYXNoIjoiUTZHdUZyLXAtREJvd3ZacElwZW81QSIsInN1YiI6IjgxNGIxNTYwLWIwNTEtNzA0MC1iNThiLWIyMThiNDJjZTU5YiIsImN1c3RvbTpzdWJkb21haW4iOiJhbGxhbm9yaWNpbCIsImNvZ25pdG86Z3JvdXBzIjpbInVzLWVhc3QtMl84QmVYSm42M2RfR29vZ2xlIiwiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIl0sImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzhCZVhKbjYzZCIsImlkZW50aXRpZXMiOlt7ImRhdGVDcmVhdGVkIjoiMTcxMTExNjcwOTcwNyIsInVzZXJJZCI6IjEwNDExNjkxMTE3NDI0NjE1NzU5NSIsInByb3ZpZGVyTmFtZSI6Ikdvb2dsZSIsInByb3ZpZGVyVHlwZSI6Ikdvb2dsZSIsImlzc3VlciI6bnVsbCwicHJpbWFyeSI6InRydWUifV0sImF1dGhfdGltZSI6MTcxMjQzNTE1MywiZXhwIjoxNzEyNzEyNzYwLCJpYXQiOjE3MTI2MjYzNjAsImp0aSI6ImZjNWE3NDkzLTFjOWQtNGUyMC1hMGViLThjYzMzYzlmMTdmMiIsImVtYWlsIjoiYWxsbGFub3JpY2lsQGdtYWlsLmNvbSIsImN1c3RvbTpwc19zdWJzY3JpcHRpb25faWQiOiJzdWJfMU94OHFWSFl3TUVKQjQ2OFFGWlNMeThHIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjdXN0b206cHNfY3VzdG9tZXJfaWQiOiJjdXNfUG1pSFowanV4RnAzTHYiLCJjdXN0b206b3JnYW5pemF0aW9uX2lkIjoiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIiwiY3VzdG9tOnBzX3JlZGlyZWN0X3VybCI6Imh0dHA6XC9cL25vZGUtcmVhZHkuY29tXC9pbnN0YW5jZS1wcm92aXNpb25pbmctam9ic1wvY2FsbGJhY2siLCJjdXN0b206b3JnYW5pemF0aW9uX25hbWUiOiJBbGxhbiBPcmljaWwiLCJjb2duaXRvOnVzZXJuYW1lIjoiR29vZ2xlXzEwNDExNjkxMTE3NDI0NjE1NzU5NSIsImN1c3RvbTpzZXR1cCI6InRydWUiLCJnaXZlbl9uYW1lIjoiQWxsYW4gR29uw6dhbHZlcyBHb21lcyIsIm9yaWdpbl9qdGkiOiIwMzI5NTkwOC1hYjIzLTQ2NzAtODVjNS1mOTQ3NjhiNmQ3OGYiLCJhdWQiOiIxazMxY2Q4NzRoMTRkNHU4aTcyNGlkNDJwbiIsImN1c3RvbTpjcmVhdGVfaW5zdGFuY2UiOiJ0cnVlIiwidG9rZW5fdXNlIjoiaWQiLCJmYW1pbHlfbmFtZSI6Ik9yaWNpbCJ9.Srb-vdOkRbkNcq0ZvHUDYT1cl9GJ-F_XM8est0yzehoOKsfD72e4_qhWcwCRGCp1ZRd9KmFxp7rhQiqzTjccXjNNSORl823c8fM3sDLSpUF6-llLvdf9zDaH5JZxOwMX-wIfJmtRCgWjYmCEJpZatPHG2M3e8jVvAFzB-EAaEs9AfR6Re5YhVtzy5aUcHUxkMw_mCtrEmg2V-zY2SEyIg5m6EMQ6J5AfKsDe3nteM28Sa0IoWOxGf5KhBySUC49RawQCfO0vYNYXC3PVJNfmTaKJ8P2g4kXtCpi11RS7ONMth2ynrPg-eWTiI_I6g9pQdvsAruomontiYrvH4kknNw","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-ASN":"269073","CloudFront-Viewer-Country":"BR","Host":"vjydkvkskl.execute-api.us-east-2.amazonaws.com","Referer":"https://node-ready.com/","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","Via":"2.0 6fe8e2d5db6a80353eb675f61c249810.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"kgg1BBFHKR4861YdGx-Kcf1-149AGIp-EUSUm9_J4MFT5WvC0cH3PA==","X-Amzn-Trace-Id":"Root=1-66149ab8-0a47130e4ff50bbf715025c0","X-Forwarded-For":"138.94.127.146, 15.158.19.244","X-Forwarded-Port":"443","X-Forwarded-Proto":"https","origin":"https://node-ready.com","sec-ch-ua":"\"Brave\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"cross-site","sec-gpc":"1"},"httpMethod":"GET","isBase64Encoded":false,"level":"info","message":"event","multiValueHeaders":{"Accept":["application/json, text/plain, */*"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8,pt;q=0.7"],"Authorization":["eyJraWQiOiJvSUdwUUVhaFIxOWlMc05GbWhpVHpsRkNYUEx3eDcxNDN0S3hGaEFUNEJBPSIsImFsZyI6IlJTMjU2In0.eyJhdF9oYXNoIjoiUTZHdUZyLXAtREJvd3ZacElwZW81QSIsInN1YiI6IjgxNGIxNTYwLWIwNTEtNzA0MC1iNThiLWIyMThiNDJjZTU5YiIsImN1c3RvbTpzdWJkb21haW4iOiJhbGxhbm9yaWNpbCIsImNvZ25pdG86Z3JvdXBzIjpbInVzLWVhc3QtMl84QmVYSm42M2RfR29vZ2xlIiwiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIl0sImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzhCZVhKbjYzZCIsImlkZW50aXRpZXMiOlt7ImRhdGVDcmVhdGVkIjoiMTcxMTExNjcwOTcwNyIsInVzZXJJZCI6IjEwNDExNjkxMTE3NDI0NjE1NzU5NSIsInByb3ZpZGVyTmFtZSI6Ikdvb2dsZSIsInByb3ZpZGVyVHlwZSI6Ikdvb2dsZSIsImlzc3VlciI6bnVsbCwicHJpbWFyeSI6InRydWUifV0sImF1dGhfdGltZSI6MTcxMjQzNTE1MywiZXhwIjoxNzEyNzEyNzYwLCJpYXQiOjE3MTI2MjYzNjAsImp0aSI6ImZjNWE3NDkzLTFjOWQtNGUyMC1hMGViLThjYzMzYzlmMTdmMiIsImVtYWlsIjoiYWxsbGFub3JpY2lsQGdtYWlsLmNvbSIsImN1c3RvbTpwc19zdWJzY3JpcHRpb25faWQiOiJzdWJfMU94OHFWSFl3TUVKQjQ2OFFGWlNMeThHIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjdXN0b206cHNfY3VzdG9tZXJfaWQiOiJjdXNfUG1pSFowanV4RnAzTHYiLCJjdXN0b206b3JnYW5pemF0aW9uX2lkIjoiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIiwiY3VzdG9tOnBzX3JlZGlyZWN0X3VybCI6Imh0dHA6XC9cL25vZGUtcmVhZHkuY29tXC9pbnN0YW5jZS1wcm92aXNpb25pbmctam9ic1wvY2FsbGJhY2siLCJjdXN0b206b3JnYW5pemF0aW9uX25hbWUiOiJBbGxhbiBPcmljaWwiLCJjb2duaXRvOnVzZXJuYW1lIjoiR29vZ2xlXzEwNDExNjkxMTE3NDI0NjE1NzU5NSIsImN1c3RvbTpzZXR1cCI6InRydWUiLCJnaXZlbl9uYW1lIjoiQWxsYW4gR29uw6dhbHZlcyBHb21lcyIsIm9yaWdpbl9qdGkiOiIwMzI5NTkwOC1hYjIzLTQ2NzAtODVjNS1mOTQ3NjhiNmQ3OGYiLCJhdWQiOiIxazMxY2Q4NzRoMTRkNHU4aTcyNGlkNDJwbiIsImN1c3RvbTpjcmVhdGVfaW5zdGFuY2UiOiJ0cnVlIiwidG9rZW5fdXNlIjoiaWQiLCJmYW1pbHlfbmFtZSI6Ik9yaWNpbCJ9.Srb-vdOkRbkNcq0ZvHUDYT1cl9GJ-F_XM8est0yzehoOKsfD72e4_qhWcwCRGCp1ZRd9KmFxp7rhQiqzTjccXjNNSORl823c8fM3sDLSpUF6-llLvdf9zDaH5JZxOwMX-wIfJmtRCgWjYmCEJpZatPHG2M3e8jVvAFzB-EAaEs9AfR6Re5YhVtzy5aUcHUxkMw_mCtrEmg2V-zY2SEyIg5m6EMQ6J5AfKsDe3nteM28Sa0IoWOxGf5KhBySUC49RawQCfO0vYNYXC3PVJNfmTaKJ8P2g4kXtCpi11RS7ONMth2ynrPg-eWTiI_I6g9pQdvsAruomontiYrvH4kknNw"],"CloudFront-Forwarded-Proto":["https"],"CloudFront-Is-Desktop-Viewer":["true"],"CloudFront-Is-Mobile-Viewer":["false"],"CloudFront-Is-SmartTV-Viewer":["false"],"CloudFront-Is-Tablet-Viewer":["false"],"CloudFront-Viewer-ASN":["269073"],"CloudFront-Viewer-Country":["BR"],"Host":["vjydkvkskl.execute-api.us-east-2.amazonaws.com"],"Referer":["https://node-ready.com/"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"],"Via":["2.0 6fe8e2d5db6a80353eb675f61c249810.cloudfront.net (CloudFront)"],"X-Amz-Cf-Id":["kgg1BBFHKR4861YdGx-Kcf1-149AGIp-EUSUm9_J4MFT5WvC0cH3PA=="],"X-Amzn-Trace-Id":["Root=1-66149ab8-0a47130e4ff50bbf715025c0"],"X-Forwarded-For":["138.94.127.146, 15.158.19.244"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"origin":["https://node-ready.com"],"sec-ch-ua":["\"Brave\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""],"sec-ch-ua-mobile":["?0"],"sec-ch-ua-platform":["\"macOS\""],"sec-fetch-dest":["empty"],"sec-fetch-mode":["cors"],"sec-fetch-site":["cross-site"],"sec-gpc":["1"]},"multiValueQueryStringParameters":null,"path":"/instances","pathParameters":null,"queryStringParameters":null,"requestContext":{"accountId":"845044614340","apiId":"vjydkvkskl","authorizer":{"claims":{"at_hash":"Q6GuFr-p-DBowvZpIpeo5A","aud":"1k31cd874h14d4u8i724id42pn","auth_time":"1712435153","cognito:groups":"us-east-2_8BeXJn63d_Google,d1d42950-dabb-43ef-b7cb-06784b8cf45b","cognito:username":"Google_104116911174246157595","custom:create_instance":"true","custom:organization_id":"d1d42950-dabb-43ef-b7cb-06784b8cf45b","custom:organization_name":"Allan Oricil","custom:ps_customer_id":"cus_PmiHZ0juxFp3Lv","custom:ps_redirect_url":"http://node-ready.com/instance-provisioning-jobs/callback","custom:ps_subscription_id":"sub_1Ox8qVHYwMEJB468QFZSLy8G","custom:setup":"true","custom:subdomain":"allanoricil","email":"alllanoricil@gmail.com","email_verified":"false","exp":"Wed Apr 10 01:32:40 UTC 2024","family_name":"Oricil","given_name":"Allan Gonçalves Gomes","iat":"Tue Apr 09 01:32:40 UTC 2024","identities":"{\"dateCreated\":\"1711116709707\",\"userId\":\"104116911174246157595\",\"providerName\":\"Google\",\"providerType\":\"Google\",\"issuer\":null,\"primary\":\"true\"}","iss":"https://cognito-idp.us-east-2.amazonaws.com/us-east-2_8BeXJn63d","jti":"fc5a7493-1c9d-4e20-a0eb-8cc33c9f17f2","origin_jti":"03295908-ab23-4670-85c5-f94768b6d78f","sub":"814b1560-b051-7040-b58b-b218b42ce59b","token_use":"id"}},"deploymentId":"ube8ur","domainName":"vjydkvkskl.execute-api.us-east-2.amazonaws.com","domainPrefix":"vjydkvkskl","extendedRequestId":"V70c7FCmiYcED6Q=","httpMethod":"GET","identity":{"accessKey":null,"accountId":null,"caller":null,"cognitoAuthenticationProvider":null,"cognitoAuthenticationType":null,"cognitoIdentityId":null,"cognitoIdentityPoolId":null,"principalOrgId":null,"sourceIp":"138.94.127.146","user":null,"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","userArn":null},"operationName":"This method can be used to list all instances from an account","path":"/v1/instances","protocol":"HTTP/1.1","requestId":"4b2c442e-a405-4659-8e62-6a58bb578993","requestTime":"09/Apr/2024:01:32:40 +0000","requestTimeEpoch":1712626360898,"resourceId":"t4baq8","resourcePath":"/instances","stage":"v1"},"requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","resource":"/instances","stageVariables":null,"timestamp":"2024-04-09T01:32:42.513Z"}
)
Attributes:
     -> id: Str(38192844131982194366912752325268730063823134327222042626)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #3
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.572 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"------------- headers -------------","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"}
)
Attributes:
     -> id: Str(38192844132004495112111282948410265782095782688728023043)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #4
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.572 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"origin","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"}
)
Attributes:
     -> id: Str(38192844132004495112111282948410265782095782688728023044)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #5
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"0":"http://localhost:8080","1":"https://node-ready.com","2":"https://instance-provisioning-admin-api-doc-bucket.s3.us-east-2.amazonaws.com","3":"https://instance-provisioning-api-doc-bucket.s3.us-east-2.amazonaws.com","level":"debug","message":"allowedOrigins","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003461)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #6
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":{"Access-Control-Allow-Credentials":true,"Access-Control-Allow-Headers":"Content-Type,Authorization,X-Amz-Date,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent","Access-Control-Allow-Methods":"OPTIONS,GET","Access-Control-Allow-Origin":"https://node-ready.com","Content-Security-Policy":"default-src 'none'; connect-src 'self'; img-src 'none'; script-src 'self'; style-src 'none'; object-src 'none'; frame-src 'none'","Content-Type":"application/json","Cross-Origin-Embedder-Policy":"require-corp","Cross-Origin-Opener-Policy":"same-origin","Referrer-Policy":"no-referrer","Strict-Transport-Security":"max-age=63072000; includeSubdomains; preload","X-Content-Type-Options":"nosniff","X-Frame-Options":"DENY","X-Permitted-Cross-Domain-Policies":"none","X-XSS-Protection":"1; mode=block"},"requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003462)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #7
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"------------- headers -------------","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003463)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #8
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:42.573 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"fetching secrets from secrets manager","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"}
)
Attributes:
     -> id: Str(38192844132026795857309813571551801500368431050234003464)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #9
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.433 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"*****","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"}
)
Attributes:
     -> id: Str(38192844151205436728046149473272519214846021945377161225)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #10
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.433 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"creating new prisma client","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"}
)
Attributes:
     -> id: Str(38192844151205436728046149473272519214846021945377161226)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #11
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.434 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"*****","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"}
)
Attributes:
     -> id: Str(38192844151227737473244680096414054933118670306883141643)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #12
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.434 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"debug","message":"","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.434Z"}
)
Attributes:
     -> id: Str(38192844151227737473244680096414054933118670306883141644)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #13
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.594 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"setting up sentry","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.591Z"}
)
Attributes:
     -> id: Str(38192844154795856705009579799059769856742408147840008205)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #14
ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC
Timestamp: 2024-04-09 01:32:43.752 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"level":"info","message":"finding all instances for: 814b1560-b051-7040-b58b-b218b42ce59b","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.752Z"}
)
Attributes:
     -> id: Str(38192844158319374446377418255422413343820849265784913934)
Trace ID: 
Span ID: 
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-04-09T01:32:58.361Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:32:58.381Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:33:08.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:33:08.382Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:33:18.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:33:18.380Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:33:28.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:33:28.380Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:33:38.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:33:38.379Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:33:48.361Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:33:48.382Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}
2024-04-09T01:33:58.362Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-09T01:33:58.385Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 113613\n}"}

Additional context

Im using Signoz to see my logs

Compare the images to verify that there are missing log events for my log stream named 2024/04/09/[$LATEST]9a79cb34998a4037bf1e3ff5df35fea5

In Signoz, I filtered logs by the log stream name

[github-actions]

Pinging code owners:

• receiver/awscloudwatch: @djaglowski @schmikei

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[AllanOricil]

It is also not iterating over all log groups I have in my aws account. It iterates over and over again in log groups that dont even have log streams anymore, and it also never tries other log groups than these ones shown below.

2024-04-09T17:01:47.395Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/w4r1tjdtbj/v1:*\",\n  CreationTime: 1704593418144,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/w4r1tjdtbj/v1\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/apigateway/w4r1tjdtbj/v1\",\n  MetricFilterCount: 0,\n  StoredBytes: 33873236\n}"}
2024-04-09T17:01:47.395Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/welcome:*\",\n  CreationTime: 1686097976313,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/welcome\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/apigateway/welcome\",\n  MetricFilterCount: 0,\n  StoredBytes: 110\n}"}
2024-04-09T17:01:47.395Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/DockerBuild:*\",\n  CreationTime: 1690469165284,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/DockerBuild\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/DockerBuild\",\n  MetricFilterCount: 0,\n  StoredBytes: 61572\n}"}
2024-04-09T17:01:47.395Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/LLEProjectE0AA5ECD-AICRGHcCk5hn:*\",\n  CreationTime: 1711131852056,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/LLEProjectE0AA5ECD-AICRGHcCk5hn\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/LLEProjectE0AA5ECD-AICRGHcCk5hn\",\n  MetricFilterCount: 0,\n  StoredBytes: 2808\n}"}
2024-04-09T17:01:47.395Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineLintPipelin-ZfBXxNdpfAzC:*\",\n  CreationTime: 1705729018015,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineLintPipelin-ZfBXxNdpfAzC\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyBucketPipelineLintPipelin-ZfBXxNdpfAzC\",\n  MetricFilterCount: 0,\n  StoredBytes: 13570\n}"}
2024-04-09T17:01:47.395Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelinePublishPipe-kmL0P9FPfuzQ:*\",\n  CreationTime: 1705730226062,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelinePublishPipe-kmL0P9FPfuzQ\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyBucketPipelinePublishPipe-kmL0P9FPfuzQ\",\n  MetricFilterCount: 0,\n  StoredBytes: 43284\n}"}
2024-04-09T17:01:47.396Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineTestPipelin-UHa4TfoMAlV3:*\",\n  CreationTime: 1705729172049,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineTestPipelin-UHa4TfoMAlV3\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyBucketPipelineTestPipelin-UHa4TfoMAlV3\",\n  MetricFilterCount: 0,\n  StoredBytes: 367152\n}"}
2024-04-09T17:01:47.396Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyPipeline-selfupdate:*\",\n  CreationTime: 1690237246405,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyPipeline-selfupdate\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyPipeline-selfupdate\",\n  MetricFilterCount: 0,\n  StoredBytes: 17016\n}"}
2024-04-09T17:01:47.396Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo:*\",\n  CreationTime: 1686462158018,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo\",\n  MetricFilterCount: 0,\n  StoredBytes: 11720\n}"}
2024-04-09T17:01:47.396Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-8K3Qo5kQ14Sd:*\",\n  CreationTime: 1686767847677,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-8K3Qo5kQ14Sd\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyProject39F7B0AE-8K3Qo5kQ14Sd\",\n  MetricFilterCount: 0,\n  StoredBytes: 2253\n}"}
2024-04-09T17:01:56.753Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo:*\",\n  CreationTime: 1686462158018,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo\",\n  MetricFilterCount: 0,\n  StoredBytes: 11720\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/w4r1tjdtbj/v1:*\",\n  CreationTime: 1704593418144,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/w4r1tjdtbj/v1\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/apigateway/w4r1tjdtbj/v1\",\n  MetricFilterCount: 0,\n  StoredBytes: 33873236\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/welcome:*\",\n  CreationTime: 1686097976313,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/apigateway/welcome\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/apigateway/welcome\",\n  MetricFilterCount: 0,\n  StoredBytes: 110\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/DockerBuild:*\",\n  CreationTime: 1690469165284,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/DockerBuild\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/DockerBuild\",\n  MetricFilterCount: 0,\n  StoredBytes: 61572\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/LLEProjectE0AA5ECD-AICRGHcCk5hn:*\",\n  CreationTime: 1711131852056,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/LLEProjectE0AA5ECD-AICRGHcCk5hn\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/LLEProjectE0AA5ECD-AICRGHcCk5hn\",\n  MetricFilterCount: 0,\n  StoredBytes: 2808\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineLintPipelin-ZfBXxNdpfAzC:*\",\n  CreationTime: 1705729018015,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineLintPipelin-ZfBXxNdpfAzC\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyBucketPipelineLintPipelin-ZfBXxNdpfAzC\",\n  MetricFilterCount: 0,\n  StoredBytes: 13570\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelinePublishPipe-kmL0P9FPfuzQ:*\",\n  CreationTime: 1705730226062,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelinePublishPipe-kmL0P9FPfuzQ\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyBucketPipelinePublishPipe-kmL0P9FPfuzQ\",\n  MetricFilterCount: 0,\n  StoredBytes: 43284\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineTestPipelin-UHa4TfoMAlV3:*\",\n  CreationTime: 1705729172049,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyBucketPipelineTestPipelin-UHa4TfoMAlV3\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyBucketPipelineTestPipelin-UHa4TfoMAlV3\",\n  MetricFilterCount: 0,\n  StoredBytes: 367152\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyPipeline-selfupdate:*\",\n  CreationTime: 1690237246405,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyPipeline-selfupdate\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyPipeline-selfupdate\",\n  MetricFilterCount: 0,\n  StoredBytes: 17016\n}"}
2024-04-09T17:02:08.930Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch/2", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo:*\",\n  CreationTime: 1686462158018,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/codebuild/MyProject39F7B0AE-1WinSXoew9Qo\",\n  MetricFilterCount: 0,\n  StoredBytes: 11720\n}"}

[AllanOricil]

@djaglowski @schmikei could you guys help me to get my dev env so that I can fix this?

[AllanOricil]

is this #32057 the fix?

[schmikei]

I think it might be related, if you'd be willing to try out the PR feel free to see if it resolves your issue, we'd very much appreciate it!

[AllanOricil]

@schmikei how can I do it? Im using the otel-collector binary

[schmikei]

@schmikei how can I do it? Im using the otel-collector binary

You can checkout the fork and do a make otelcontribcol to build a binary!

[AllanOricil]

@schmikei can I build linux dis on a mac?

[AllanOricil]

never mind @schmikei
I asked that because my t2.micro died while building it. I created a t3.medium, and I'm now trying to build it again. Let's hope it is enought.

[AllanOricil]

@schmikei do you have this error while building? I checkout your branch, and then I run the make otelcontribcol

[AllanOricil]

google/cadvisor#3508

[AllanOricil]

@schmikei I spent the day building this stuff but it did not work :/

I used the same config.yaml with the binary I created from your branch and they had both different results.

Below is the output when using 0.97.0

2024-04-11T04:59:12.112Z	info	prometheusreceiver@v0.97.0/metrics_receiver.go:299	Starting scrape manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-04-11T04:59:21.971Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:59:22.003Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 137470\n}"}
2024-04-11T04:59:31.970Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:59:31.994Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 137470\n}"}
2024-04-11T04:59:41.971Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:59:41.997Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 137470\n}"}

Below you can see the result I got from the new binary

2024-04-11T04:58:36.200Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:58:36.200Z	error	awscloudwatchreceiver@v0.97.0/logs.go:165	unable to perform discovery of log groups	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "error": "unable to list log groups: InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, DescribeLogGroupsInput.NextToken.\n"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver.(*logsReceiver).startPolling
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver@v0.97.0/logs.go:165
2024-04-11T04:58:46.201Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:58:46.201Z	error	awscloudwatchreceiver@v0.97.0/logs.go:165	unable to perform discovery of log groups	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "error": "unable to list log groups: InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, DescribeLogGroupsInput.NextToken.\n"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver.(*logsReceiver).startPolling