Please add support for AWS SDK v2

[xenomachina]

LocalStackContainer uses the AWS SDK for Java v1.

The AWS SDK for Java v2 API has renamed everything. In particular, all of the packages have changed, and so the AWSCredentialsProvider and AwsClientBuilder.EndpointConfiguration returned by LocalStackContainer are not usable as-is in the AWS SDK for Java v2. It is possible to adapt them with some work (and we can share our wrapper if you like, though it is written in Kotlin) however, it would be nice if clients using one version of the AWS SDK for Java did not need to depend on artifacts from the other version, even transitively.

I imagine you'll want to retain support for the v1 SDK at least until AWS deprecates it.

[rnorth]

I've used Testcontainers for testing code that uses AWS SDKv2, and I'd agree that LocalStackContainer's coupling to the v1 SDK makes it a rough fit.

The AWS SDK coupling in general is, I think, fairly important to retain as it removes some pain points that would be exposed if we just provided a thin wrapper around Localstack.

I think that we probably need a separate module for AWS SDKv2 compatibility in this case. Let me think and discuss with @bsideup and @kiview...

[ashleyfrieze]

@rnorth - it might be possible to make two subclasses of a LocalStackContainerBase, one for v1 and one for v2, with provided dependencies. It would be a shame to have to duplicate the code across two modules.

Perhaps some of the issue here is that the local stack container doesn't really need to do a HUGE amount of AWS stuff itself. We only really need the URL to pass to our SDK client library.

If @bsideup and @kiview and yourself can propose how this should be done, I don't mind hacking a PR together. It would be useful in my current project.

[rnorth]

Thanks @ashleyfrieze - sorry for the slow reply.

Perhaps some of the issue here is that the local stack container doesn't really need to do a HUGE amount of AWS stuff itself. We only really need the URL to pass to our SDK client library.

You're quite right; this container only faintly interacts with the AWS SDK, but just enough to cause us a compatibility issue.

We since learned that this is a bit of a mistaken pattern to use for our modules, and now recommend that new modules only expose JDK types in their public APIs. Too late for this module, though 😬.

Thinking about whether this should be within the same module:

Pros:

• We don't have to publish and document a new module (though really, it's not much of a burden for us)
• Users do not have to learn of the existence of a new module when they want to use/migrate to AWSDKv2
• We avoid an embarassingly asymmetric naming situation of having a localstack module and a localstack-awssdk2 (?) module 😁

Cons:

• Code supporting both versions will be on the classpath, but one will be broken depending on which version of the AWS SDK is also present. I feel that this could be very confusing and frustrating for users.

In the long run, IMHO it would be good to have a localstack-awssdk-v1 module and a localstack-awssdk-v2 module. Renaming the current module to localstack-awssdk-v1 would of course be a breaking change, so something that we should really wait to do along with a major version bump for Testcontainers...

But that wouldn't stop us from taking the first step in that direction: creation of localstack-awssdk-v2.

Thoughts?

(Aside: it's amusing that this is a prime example of 'naming things' being the hardest part of the problem 😄)

[oscarvarto]

I've used Testcontainers for testing code that uses AWS SDKv2, and I'd agree that LocalStackContainer's coupling to the v1 SDK makes it a rough fit.

Would you mind explaining your strategy to use Testcontainers (currently coupled to SDK V1) and code written with AWS SDK V2?
That might be helpful for a project I am currently working on.
Thanks in advance @rnorth

@ashleyfrieze, @bsideup, @kiview, and @rnorth I am craving for SDK V2 support in Testcontainers. Any news?

[xenomachina]

@oscarvarto we have something like this (in Kotlin):

import software.amazon.awssdk.awscore.client.builder.AwsClientBuilder
...
fun <T: AwsClientBuilder<*,*>> T.localStack(
    container: LocalStackContainer,
    service: LocalStackContainer.Service
): T {
    endpointOverride(URI(container.getEndpointConfiguration(service).serviceEndpoint))
    credentialsProvider(container.defaultCredentialsProvider?.let { v1Provider ->
        val v1Credentials = v1Provider.credentials
        AwsCredentialsProvider {
            object : AwsCredentials {
                override fun accessKeyId(): String = v1Credentials.awsAccessKeyId
                override fun secretAccessKey(): String = v1Credentials.awsSecretKey
            }
        }
    })
    return this
}

And then build SDK v2 clients that connect to a LocalStackContainer by doing something like:

val client = DynamoDbClient.builder()
    .localStack(localStackContainer, DYNAMODB))
    .build()

[guizmaii]

Scala version of the @xenomachina code:

import java.net.URI

import org.testcontainers.containers.localstack.LocalStackContainer
import software.amazon.awssdk.auth.credentials.{AwsCredentials, AwsCredentialsProvider}
import software.amazon.awssdk.awscore.client.builder.AwsClientBuilder
import software.amazon.awssdk.regions.Region

object AwsClientBuilderOps {

  implicit final class AwsClientBuilderOps[T <: AwsClientBuilder[_, _]](val builder: T) extends AnyVal {
    def localStack(container: LocalStackContainer, service: LocalStackContainer.Service): T = {
      builder.endpointOverride(new URI(container.getEndpointConfiguration(service).getServiceEndpoint))
      builder.region(Region.US_EAST_1)
      builder.credentialsProvider {
        val v1Credentials = container.getDefaultCredentialsProvider.getCredentials
        new AwsCredentialsProvider {
          override def resolveCredentials(): AwsCredentials = new AwsCredentials {
            override def accessKeyId(): String = v1Credentials.getAWSAccessKeyId

            override def secretAccessKey(): String = v1Credentials.getAWSSecretKey
          }
        }
      }
      builder
    }
  }

}

I added the .region because you need it for SQS.

Cheers
😉

[hdurix]

And now a Java version of the temporary solution:

@Bean
public S3Client s3Client(LocalStackContainer localStackContainer) throws URISyntaxException {
  return S3Client
    .builder()
    .endpointOverride(new URI(localStackContainer.getEndpointConfiguration(LocalStackContainer.Service.S3).getServiceEndpoint()))
    .credentialsProvider(new CrossAwsCredentialsProvider(localStackContainer.getDefaultCredentialsProvider()))
    .build();
}

private class CrossAwsCredentialsProvider implements AwsCredentialsProvider {
  private final AWSCredentials credentials;

  public CrossAwsCredentialsProvider(AWSCredentialsProvider provider) {
    this.credentials = provider.getCredentials();
  }

  @Override
  public AwsCredentials resolveCredentials() {
    return AwsBasicCredentials.create(credentials.getAWSAccessKeyId(), credentials.getAWSSecretKey());
  }
}

[dbveeva]

When will the fix committed by @rnorth be released? I am using v1.14.3.

[TylerJaacks]

Is this issue solved because I am on 15 and it seems to still be an issue.

[jakobbraun]

I'am also experiencing issues with version 1.14.3, when using AWS SDK v2:

java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentials
at com.exasol.adapter.document.files.DocumentFilesAdapterS3IT.<init>

The reason for that error is, that the localstack testconainers module internally still uses the AWS SDK v1. While that would be no problem, the issue is, that it declares it as compileOnly dependency (see build.gradle)

A quick fix for this issue is to also provide the AWS SDK in test scop.
For maven you can use:

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.860</version>
            <scope>test</scope>
        </dependency>

I do however no understand, why this dependecy is set to compileOnly. At least some documentation would be very helpful.