Add GCloud module for Google Cloud Datastore, Firestore, PubSub, and Spanner emulators #2690

eddumelendez · 2020-05-09T19:51:49Z

Add support to perform additional commands in the container
Add docs
Add gcloud module

eddumelendez · 2020-05-09T20:08:23Z

It is pending to polish the pubsub test 🙈

bsideup · 2020-05-13T19:40:58Z

modules/gcloud/src/main/java/org/testcontainers/containers/GCloudGenericContainer.java

+		return self();
+	}
+
+	private static ImageFromDockerfile buildImage(String image, String mainCmd, String[] prerequisiteCmds) {


does it really need to be an image? Can't we just use the command for it, since the image is not cached anyways?

At the beginning I was not able to run it using command but that led me to know more about testcontainers. Now, I did it but had to set up a startupTimeout.

Are we sure about this? 😄
It looks like these setup steps will require a non-trivial amount of time to execute - perhaps even the majority of the startup time.

Would it be worth trying to get more attention on GoogleCloudPlatform/cloud-sdk-docker#202 so that we can avoid having these setup steps entirely?

Actually, I missed that we install Java and other things... Yeah, I believe we need to wait for these images :(

/cc @jamesward @saturnism

docs/modules/gcloud.md

modules/gcloud/src/test/java/org/testcontainers/containers/SpannerEmulatorContainerTest.java

modules/gcloud/src/test/java/org/testcontainers/containers/PubSubEmulatorContainerTest.java

modules/gcloud/src/main/java/org/testcontainers/containers/DatastoreEmulatorContainer.java

saturnism · 2020-08-04T14:25:19Z

quick ping on this to see if there are any more work needed, happy to help

eddumelendez · 2020-08-07T19:48:42Z

@saturnism currently, one of the things is the issue GoogleCloudPlatform/cloud-sdk-docker#202 in order to get images for the rest of the emulators. Also, I would appreciate your feedback in the tests. Thanks!

saturnism · 2020-08-08T00:05:12Z

Will it be possible for the first iteration to use the non-alpine image that has emulators pre-installed? e.g., google/cloud-sdk:292.0.0 (and being able to specify version would be great!

eddumelendez · 2020-08-08T00:15:08Z

for the first version, it make sense for me to use that image with components pre-installed. if everyone agree I can work on the changes.

eddumelendez · 2020-08-08T03:08:13Z

@saturnism I was working on the changes but still using image google/cloud-sdk:304.0.0-debian_component_based we will need to install jdk in order to run emulators 😢

saturnism · 2020-08-15T22:42:08Z

@eddumelendez can we use gcr.io/google.com/cloudsdktool/cloud-sdk:305.0.0? it's a larger image but should have the emulators.

however, it seems like firestore and spanner ones are not installed in this.
GoogleCloudPlatform/cloud-sdk-docker#217

saturnism · 2020-08-15T22:48:08Z

added GoogleCloudPlatform/cloud-sdk-docker#218

eddumelendez · 2020-08-16T03:29:55Z

@saturnism currently, I am using google/cloud-sdk:305.0.0 and it is working for datastore and pubsub. Regarding to spanner, in the cloud sdk use docker so would be a DinD approach. But, since the image in already available gcr.io/cloud-spanner-emulator/emulator:1.0.0 I am using it directly, or the PR you already sent will not use docker?. I will wait for firestore to be added so I will run my tests again and update the PR.

Have the feeling that we are close 😃

saturnism · 2020-08-17T19:59:14Z

a quick update, the pr to have additional emulators has been merged, and it's likely it'll be released tomorrow under the 306 version tag. will keep you updated! i'm excited to see a closure on this soon :)

eddumelendez · 2020-08-17T20:01:48Z

saw the merge notification. So excited too! I will be updating the PR tomorrow night and then ask for review

eddumelendez · 2020-08-19T00:09:04Z

very glad to let you know that now that all components are installed in the image provided we can run everything successfully

saturnism · 2020-08-20T17:00:42Z

this looks great! i feel this is almost ready to go?

walles · 2020-08-24T07:22:31Z

modules/gcloud/src/test/java/org/testcontainers/containers/PubSubEmulatorContainerTest.java

+    public PubSubEmulatorContainer emulator = new PubSubEmulatorContainer();
+
+    @Test
+    public void testSimple() throws IOException, ExecutionException, InterruptedException {


It would be nice to have a(nother) test that:

Publishes a message with some trivial contents

Pulls that message from a subscription to that topic

Verifies that the trivial contents was received

@walles thanks for the suggestions. I have added one more commit introducing that change. 👍🏽

Beautiful! 🥇

rnorth · 2020-08-29T09:04:33Z

Thanks! I'll try and have another look at this over the weekend.

For now there's a docs build failure, apparently due to an incorrect codeinclude path:

12:58:58 AM: FileNotFoundError: [Errno 2] No such file or directory: '/opt/build/repo/docs/modules/../../../modules/gcloud/src/test/java/org/testcontainers/containers/DatastoreEmulatorContainerTest.java'

If you want a quicker feedback loop to see/check the generated docs locally, please have a look at: https://www.testcontainers.org/contributing_docs/

eddumelendez · 2020-08-29T23:39:37Z

@rnorth thanks! fixed the docs

eddumelendez · 2020-10-08T02:51:23Z

@bsideup branch rebase with master and proper changes applied :)

modules/gcloud/src/test/java/org/testcontainers/containers/DatastoreEmulatorContainerTest.java

modules/gcloud/src/test/java/org/testcontainers/containers/PubSubEmulatorContainerTest.java

rnorth · 2020-10-11T06:43:25Z

modules/gcloud/src/main/java/org/testcontainers/containers/SpannerEmulatorContainer.java

+    private static final int GRPC_PORT = 9010;
+    private static final int HTTP_PORT = 9020;


I wonder, maybe these should be public constants so that they can be used by tests? If we did that, we could do so for the other container classes in this PR.

WDYT @bsideup, @kiview?

WDYT about not exposing the constants, but getGrpcPort/getHttpPort methods?

looking at some of the other modules, private static final seems like the norm/convention atm. would it make sense to follow up w/ an issue for the broader change to expose these across modules?

I've been persuaded by @bsideup - many of our other modules have some mechanism for getting the mapped ports, address or URL for the running container. It seems like we should do this here, instead of exposing constants, as it's fundamentally more useful for the user.

I'll take the action to do this, as we've asked @eddumelendez to do too much already. I'll raise a quick PR tonight, and will merge this PR now.

rnorth

A few pretty trivial comments from me; otherwise I think we should merge soon!

…astoreEmulatorContainerTest.java Co-authored-by: Richard North <[email protected]>

rnorth · 2020-10-11T08:28:46Z

LGTM but I'll let @bsideup comment too.
Thanks for the PR and the very quick responses @eddumelendez!

rnorth · 2020-10-13T14:51:39Z

Merging now - thanks once again for the contribution @eddumelendez!

* Simplify `KafkaContainerCluster#start` * When an image version is not specified, use `latest` as the default tag (#3313) * Add workflow for Update Gradle Wrapper Action. (#3297) Co-authored-by: Richard North <[email protected]> * Always continue on error for examples CI (#3339) * Bump snakeyaml from 1.25 to 1.27 in /core (#3252) Bumps [snakeyaml](https://bitbucket.org/asomov/snakeyaml) from 1.25 to 1.27. - [Commits](https://bitbucket.org/asomov/snakeyaml/branches/compare/snakeyaml-1.27..snakeyaml-1.25) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump lombok from 1.18.12 to 1.18.14 in /examples (#3322) Bumps [lombok](https://github.com/rzwitserloot/lombok) from 1.18.12 to 1.18.14. - [Release notes](https://github.com/rzwitserloot/lombok/releases) - [Changelog](https://github.com/rzwitserloot/lombok/blob/master/doc/changelog.markdown) - [Commits](projectlombok/lombok@v1.18.12...v1.18.14) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump r2dbc-mariadb from 0.8.3-beta1 to 0.8.4-rc in /modules/mariadb (#3300) Bumps [r2dbc-mariadb](https://github.com/mariadb-corporation/mariadb-connector-r2dbc) from 0.8.3-beta1 to 0.8.4-rc. - [Release notes](https://github.com/mariadb-corporation/mariadb-connector-r2dbc/releases) - [Changelog](https://github.com/mariadb-corporation/mariadb-connector-r2dbc/blob/master/CHANGELOG.md) - [Commits](https://github.com/mariadb-corporation/mariadb-connector-r2dbc/commits) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump testng from 7.2.0 to 7.3.0 in /examples (#3068) Bumps [testng](https://github.com/cbeust/testng) from 7.2.0 to 7.3.0. - [Release notes](https://github.com/cbeust/testng/releases) - [Changelog](https://github.com/cbeust/testng/blob/master/CHANGES.txt) - [Commits](https://github.com/cbeust/testng/commits) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump assertj-core from 3.17.1 to 3.17.2 in /core (#3251) Bumps [assertj-core](https://github.com/joel-costigliola/assertj-core) from 3.17.1 to 3.17.2. - [Release notes](https://github.com/joel-costigliola/assertj-core/releases) - [Commits](assertj/assertj@assertj-core-3.17.1...assertj-core-3.17.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump aws-java-sdk-dynamodb from 1.11.865 to 1.11.880 in /modules/dynalite (#3332) Bumps [aws-java-sdk-dynamodb](https://github.com/aws/aws-sdk-java) from 1.11.865 to 1.11.880. - [Release notes](https://github.com/aws/aws-sdk-java/releases) - [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-java@1.11.865...1.11.880) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump mockito-core from 3.5.11 to 3.5.13 in /core (#3275) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump elasticsearch-rest-client from 7.9.1 to 7.9.2 in /modules/elasticsearch (#3276) Bumps [elasticsearch-rest-client](https://github.com/elastic/elasticsearch) from 7.9.1 to 7.9.2. - [Release notes](https://github.com/elastic/elasticsearch/releases) - [Commits](elastic/elasticsearch@v7.9.1...v7.9.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Increase memory limits used in example (#3340) For improved test stability * Bump mockito-core from 3.5.11 to 3.5.13 in /modules/junit-jupiter (#3283) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump tomcat-jdbc from 9.0.37 to 9.0.39 in /modules/jdbc-test (#3338) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump junit from 4.13 to 4.13.1 in /examples (#3328) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump mariadb-java-client from 2.6.2 to 2.7.0 in /modules/mariadb (#3278) Bumps [mariadb-java-client](https://github.com/mariadb-corporation/mariadb-connector-j) from 2.6.2 to 2.7.0. - [Release notes](https://github.com/mariadb-corporation/mariadb-connector-j/releases) - [Changelog](https://github.com/mariadb-corporation/mariadb-connector-j/blob/master/CHANGELOG.md) - [Commits](mariadb-corporation/mariadb-connector-j@2.6.2...2.7.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump solr-solrj from 8.6.2 to 8.6.3 in /examples (#3321) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump postgresql from 42.2.16 to 42.2.17 in /examples (#3323) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump postgresql from 42.2.16 to 42.2.17 in /modules/junit-jupiter (#3327) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump postgresql from 42.2.16 to 42.2.17 in /modules/spock (#3330) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump tomcat-jdbc from 9.0.37 to 9.0.39 in /modules/jdbc (#3333) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump postgresql from 42.2.16 to 42.2.17 in /modules/postgresql (#3334) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump aws-java-sdk-sqs from 1.11.860 to 1.11.880 in /modules/localstack (#3337) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump httpclient from 4.5.12 to 4.5.13 in /modules/junit-jupiter (#3326) Bumps httpclient from 4.5.12 to 4.5.13. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump httpclient from 4.5.12 to 4.5.13 in /modules/spock (#3329) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump cucumber-junit from 6.7.0 to 6.8.1 in /examples (#3325) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump org.springframework.boot from 2.3.3.RELEASE to 2.3.4.RELEASE in /examples (#3247) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump aws-java-sdk-s3 from 1.11.870 to 1.11.880 in /modules/localstack (#3336) Bumps [aws-java-sdk-s3](https://github.com/aws/aws-sdk-java) from 1.11.870 to 1.11.880. - [Release notes](https://github.com/aws/aws-sdk-java/releases) - [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-java@1.11.870...1.11.880) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump zt-exec from 1.10 to 1.12 in /core (#3253) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Richard North <[email protected]> * Bump s3 from 2.14.21 to 2.15.7 in /modules/localstack (#3335) Bumps s3 from 2.14.21 to 2.15.7. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cucumber-java from 6.6.0 to 6.8.1 in /examples (#3324) Bumps [cucumber-java](https://github.com/cucumber/cucumber-jvm) from 6.6.0 to 6.8.1. - [Release notes](https://github.com/cucumber/cucumber-jvm/releases) - [Changelog](https://github.com/cucumber/cucumber-jvm/blob/main/CHANGELOG.md) - [Commits](cucumber/cucumber-jvm@v6.6.0...v6.8.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove GitHub Actions cache restore keys (#3342) For more specific cache matching * Allow users to specify a MongoDB database name (#2980) Co-authored-by: Richard North <[email protected]> * Add GCloud module for Google Cloud Datastore, Firestore, PubSub, and Spanner emulators (#2690) Co-authored-by: Richard North <[email protected]> * Use a lighter weight image for MultiplePortsExposedTest (#3343) * Use a lighter weight image for MultiplePortsExposedTest * Update helloworld container version * docker-machine: get full remote daemon URL, to allow for use of custom daemon port (#2769) (#3237) Co-authored-by: Vitalii Chura <[email protected]> * Fix remote gradle cache 400 InvalidArgument error (#3346) per hint given in https://www.digitalocean.com/community/questions/node-upload-file-to-s3-error-invalidargument-null * Add gcloud endpoint accessors (#3344) Co-authored-by: Sergei Egorov <[email protected]> Co-authored-by: Richard North <[email protected]> Co-authored-by: Cristian Greco <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: silaev <[email protected]> Co-authored-by: Eddú Meléndez Gonzales <[email protected]> Co-authored-by: vcvitaly <[email protected]> Co-authored-by: Vitalii Chura <[email protected]>

eddumelendez requested review from bsideup, kiview and rnorth as code owners May 9, 2020 19:51

eddumelendez changed the title ~~gcloud~~ Add gcloud module May 9, 2020

bsideup reviewed May 13, 2020

View reviewed changes