Google Cloud Storage support #792

sergseven · 2022-01-27T13:55:17Z

The image used as a base: https://github.com/fsouza/fake-gcs-server

This reverts commit 8f13c9d.

# Conflicts: # testcontainers-spring-boot-bom/pom.xml

Aloren · 2022-01-27T22:12:41Z

What is an issue with taking the following approach https://github.com/googleapis/google-cloud-java/blob/main/TESTING.md#testing-code-that-uses-storage?

sergseven · 2022-01-28T13:38:26Z

@Aloren
That's a good question.
Actually we are using the approaches mentioned at https://github.com/googleapis/google-cloud-java/blob/main/TESTING.md#on-your-machine-2 for low level(unit or low level integration tests) tests, we also have test projects and test buckets to test our integration with the real world GCS service as well as for E2E tests.

Moreover we have some kind of middle level "component" tests which test both integration and functionality independently of the environment outside of the "component". One of the most important requirement for such a test we have is not to replace any part of a component with test double or test implementation, but test configuration close to production as much as possible on this level. So a component should talk with the dependencies making a network call.
Testcontainers works best for this purposes.
Also, since we have junit & spring boot on the board, the embedded testcontainers work best as well.

sergseven · 2022-02-09T10:21:27Z

Heads up!
Any chance to have it reviewed soon? Any feedback is appreciated.

Aloren · 2022-02-23T14:45:56Z

Sorry for the delay.
Is it possible to omit google-cloud-storage from the compile dependencies? Otherwise it might conflict with user's production dependency.
Also I am hesitant to merge the PR since this is not officially supported mock, so as a result end users can face issues that mock is behaving differently than real storage. What is your impression of working with fake google cloud storage? Is it stable enough?

Aloren · 2022-02-23T14:46:28Z

Also can you pls rebase to the latest changes.

…age name

sergseven · 2022-03-21T11:52:50Z

@Aloren
Sorry for the delay too, was out of my mind with all this hassle...

Is it possible to omit google-cloud-storage from the compile dependencies? Otherwise it might conflict with user's production dependency.

sure, made it as a "provided" dependency.

What is your impression of working with fake google cloud storage? Is it stable enough?

I would say it's pretty much mature, it has about 500+ stars on github and is actively supported and developed by the maintainer.
From my prospective It also has much more features supported than LocalStorageHelper suggested for local testing.

Aloren · 2022-03-22T22:11:15Z

embedded-google-storage/pom.xml

+            <groupId>com.google.cloud</groupId>
+            <artifactId>google-cloud-storage</artifactId>
+            <version>${google-cloud-storage.version}</version>
+            <scope>provided</scope>


Please add into the documentation that this module consumes/expects this dependency to be provided by the user.

Aloren · 2022-03-22T22:12:31Z

...e-storage/src/main/java/com/playtika/test/storage/EmbeddedStorageBootstrapConfiguration.java

+
+@Slf4j
+@Configuration
+@ConditionalOnExpression("${embedded.containers.enabled:true}")


Please add
@ConditionalOnClass(com.google.cloud.storage.Storage.class)
to this autoconfiguration, so that is does not startup and fail if user does not have google-cloud-storage on classpath.

makes sense, added

Aloren · 2022-03-22T22:18:23Z

embedded-google-storage/pom.xml

+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <properties>


can you please move these properties just before dependencies section?

sure, moved

Aloren · 2022-03-22T22:19:43Z

embedded-google-storage/src/main/java/com/playtika/test/storage/StorageResourcesGenerator.java

+    @PostConstruct
+    protected void init() {
+        log.info("Creating buckets.");
+        buckets.forEach(this::createBucket);


is it possible to create bucket via cli, so that we do not have dependency on google-cloud-storage?

i've just looked into the transitive dependencies of google-cloud-storage and it looks like it comes with half of the internet :), so if possible please use rest api. for bucket creation, i suppose, this won't be difficult?
https://cloud.google.com/storage/docs/json_api/v1/buckets/insert

Aloren · 2022-03-22T22:28:45Z

embedded-google-storage/src/main/java/com/playtika/test/storage/StorageProperties.java

+    /** location for buckets */
+    private String bucketLocation = "US-CENTRAL1";
+    /** list with predefined buckets to be created */
+    private Collection<String> buckets = emptyList();


Bucket creation operation supports additional parameters, that may be needed by someone in the future, so lets' replace this instead with:

private Collection<BucketProperties> buckets = emptyList(); static class BucketProperties { String name; // other properties if needed can be added later }

makes sense to me, BucketProperties introduced

Aloren · 2022-03-22T22:32:10Z

embedded-google-storage/README.adoc

+* `embedded.google.storage.projectId` `(project id for storage, default is my-project-id)`
+* `embedded.google.storage.bucketLocation` `(location for buckets, default is US-CENTRAL1)`
+
+* Predefined buckets:


Can you please rephrase this part to smth like:
embedded.google.storage.buckets creates buckets on startup

The description changed.

Aloren · 2022-03-22T22:36:14Z

...e-storage/src/main/java/com/playtika/test/storage/EmbeddedStorageBootstrapConfiguration.java

+        GenericContainer<?> container = new GenericContainer<>(ContainerUtils.getDockerImageName(properties))
+            .withExposedPorts(StorageProperties.PORT)
+            .withCreateContainerCmdModifier(cmd -> cmd.withEntrypoint(
+                "/bin/fake-gcs-server",


shouldnt we also provide -backend memory option? is is provided in examples and i don't see what is the default value for it.

the default one is filesystem backend, which was used previously.
from now, setting it as a memory backend.
nice catch!

sergseven · 2022-03-31T13:10:26Z

@Aloren
Thank you for a valuable review!

I would discuss the google-cloud-storage dependency topic before actual substitution implementation.

The com.google.cloud.storage.Storage class intended to be the client to GCS is distributed with google-cloud-storage.
What use cases do you see here for a case when a user won't have this dependency already on the class path?

The main use case I would expect is to add embedded-google-storage as a test-scope dependency, so it won't appear on user's prod classpath. The compile scope should be fine then.
Could you please share other possible use cases?

Also noticing that google-cloud-storage is intended to be used along with the bom, so it's pretty unexpected we would mix up client's dependency versions since they should already be managed by a BOM(s).

Aloren · 2022-04-07T18:46:50Z

@sergseven 2.1.6 was release successfully and is already present in sonatype. Unfortunately Maven central had sync issues on 6th April. I hope that within 48 hours artifacts will be available there as well.
https://status.maven.org/

sergseven · 2022-04-08T09:49:31Z

@sergseven 2.1.6 was release successfully and is already present in sonatype. Unfortunately Maven central had sync issues on 6th April. I hope that within 48 hours artifacts will be available there as well. https://status.maven.org/

Awesome, thank you!
I will try to help to maintain with embedded-gcs support.

Aloren · 2022-04-08T12:31:28Z

@sergseven we've released 2.1.7 and it's already available in maven central, please use it. Thanks

sergseven added 15 commits January 17, 2022 17:56

add embedded-google-storage module draft

8bf9991

fix autoconfig & tests

041b333

fix format

7d4e4c5

clean up

6559be2

add a log for createBucket

ce4be8b

fix format

e60834b

add embedded-google-storage to the spring boot bom

e5aadaa

refactoring - remove scheme, add endpoint, clean up

1c8e898

minor rename

12b6068

add jar deployment to ox GAR

8f13c9d

update documentation

366776a

Revert "add jar deployment to ox GAR"

20ef1e7

This reverts commit 8f13c9d.

Merge branch 'develop' into feature/gcs-support

16fff8a

# Conflicts: # testcontainers-spring-boot-bom/pom.xml

apply formatting

e9467b6

fix parent pom version

a059a56

sergseven marked this pull request as draft January 27, 2022 13:55

replace forked base docker image with the upstream one

3054c76

sergseven marked this pull request as ready for review January 28, 2022 13:38

Merge branch 'develop' into feature/gcs-support

c508b13

Merge branch 'develop' into feature/gcs-support

be70081

sergseven and others added 5 commits March 7, 2022 14:41

Merge branch 'develop' into feature/gcs-support

949e14d

set provided scope for google-cloud-storage dependency

f52f182

implement StorageProperties.getDefaultDockerImage()

76cd0c7

Merge branch 'develop' into feature/gcs-support

675a155

update parent pom version

ab40889

use ContainerUtils.getDockerImageName() to get effective container im…

055ea1c

…age name

sergseven added 2 commits March 21, 2022 22:40

Merge branch 'develop' into feature/gcs-support

8c0efb0

update parent pom version

9356747

Aloren added the feature label Mar 22, 2022

Aloren reviewed Mar 22, 2022

View reviewed changes

sergseven added 4 commits March 25, 2022 19:22

rearrange properties section

56e1804

introduce BucketProperties

089455b

add @ConditionalOnClass for EmbeddedStorageBootstrapConfiguration

291fe97

set fake gcs server backend as "memory"

d6d009a

sergseven and others added 3 commits March 31, 2022 15:15

Merge branch 'develop' into feature/gcs-support

ac9faf1

update parent pom version

0ab823f

Merge branch 'develop' into feature/gcs-support

06f91f4

Aloren approved these changes Apr 5, 2022

View reviewed changes

Aloren enabled auto-merge (squash) April 5, 2022 21:03

Aloren merged commit 5654108 into PlaytikaOSS:develop Apr 5, 2022

Google Cloud Storage support #792

Google Cloud Storage support #792

Uh oh!

Conversation

sergseven commented Jan 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aloren commented Jan 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergseven commented Jan 28, 2022

Uh oh!

sergseven commented Feb 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aloren commented Feb 23, 2022

Uh oh!

Aloren commented Feb 23, 2022

Uh oh!

sergseven commented Mar 21, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergseven commented Mar 31, 2022

Uh oh!

Aloren commented Apr 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergseven commented Apr 8, 2022

Uh oh!

Aloren commented Apr 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sergseven commented Jan 27, 2022 •

edited

Loading

Aloren commented Jan 27, 2022 •

edited

Loading

sergseven commented Feb 9, 2022 •

edited

Loading

Aloren commented Apr 7, 2022 •

edited

Loading