Service instances are not created when using repo sync

[RichardChester]

I have an environment in a different account to proton when I follow the steps to create a new synced service and deploy it to that environment no service instance is created bellow I have attached a step by step of what happens

at this point you can see the files look correct

I then merge in the pr's that get created here

this can be seen above

and again here and although this should lead to instances being created as you can see bellow none are

and just to confirm it does say that the service sync was successful despite there being no instances created. It's also worth noting that the pipeline then fails it's run (because it's looking for instances) but then also updating the service file to add a new instance is also ignored despite the sync registering the push

@nouvionp @rafavallina

[nouvionp]

Thanks for the details, I will have someone look into it. Did you test this using an environment in the same account by chance?

[RichardChester]

Thanks @nouvionp, Yes with the same source and target account (source and target are different accounts)

[racicot-amzn]

Hello @RichardChester, taking a look at this now.

I was not able to find any records of attempted service instance deployments to an environment in a different account for the service qpjb. However I did find many attempts at deploying service instances to an environment development within the same account, many of which failed due to failures in the CodeBuild job.

I do see you were able to successfully able to deploy a service instance qpjb/service-instance/dev as of 8/21.

Are you still facing the deployment issue? And can you confirm you were attempting to deploy a service instance to an environment in a different account?

[RichardChester]

Hello @racicot-amzn thanks for looking into this.

So for the cross account part of this I'll clarify the state Proton is running in a central account (Account A) the environment development is deployed in a different account (account B) when deploying qpjb (a service with a pipeline) the pipeline sits in account A and the actual fargate service gets deployed into B .

So I have now managed to get qpjb running however it doesn't appear to work as intended on the initial service creation. When first creating the service as shown above the result is that a service instance doesn't get created however if I then pull the repo and after making a small change (like changing max instances from 1 to 2) push it a new instance will then be created.

In the CDK example in this repo part of the code build job checks the service for it's instances so that it can update the ecr repo for cross account access but in practice the instance doesn't start being created before the pipeline finishes causing the code build error. To be honest it did occur to me that this could be how proton is meant to work and that it is actually this example that's misleading.

Once the pipeline has failed and I re push the code and the pipeline re-runs it's then fine and works as expected. The code build errors your seeing are largely caused by my modifying this example to work across the account boundary.

[racicot-amzn]

@RichardChester,

After looking into this a bit more, I think I've determined the root issue here.

During the initial deployment when creating a service via ServiceSync, the service is created without any of the instances but will attempt to deploy the pipeline if one is specified.

The problem scenario is:

• ServiceSync is kicked off, starting the deployment of the service and the pipeline.
• The ServiceSync process attempts to create the service instances too quickly, specifically before the creation of the service pipeline.
• The creation of the instance(s) fails. (Because the pipeline is still in progress)
• The pipeline and service deployments complete successfully, explaining your last screenshot showing the successful deployment with an empty list of instances.
• Upon retry, the subsequent instance deployment succeeds because the service and pipeline have previously succeeded to deploy.

So the issues are, the initial creation of the service instances is too quick in our internal ServiceSync process, and the fact that the instance creation failures were not surfaced in the console. Both of which we are tracking and are determining the next steps to fix.

It seems you have been able to find a work around yourself which is good news, but if it helps, you would be able to get the status of the service instance creation manually yourself from calling the GetServiceInstanceSyncStatus API.

[nouvionp]

Just posting an update here that we will try to deploy some UX improvements in about 3 weeks.
We will surface a message when initially creating the service if no service instances were created. We will also include a new table that will expose the data from the GetServiceInstanceSyncStatus so it is easier to see the details (and errors) of each service instance sync.