For an alternative proposal that uses simple CloudFormation YAML patches, see #4209!

I really love this! The only addition I'd have is to take it even further. A lot of the power of the CDK is high level abstractions, so it'd be nice to have a Copilot abstraction that takes care of details like knowing where/how to template include and possibly having opinionated/type-safe ways of getting templated resources.

Hi @RichiCoder1 👋

Thanks for the feedback! Are you thinking of something along these lines:

import * as copilot from 'aws-copilot-lib';

const props = copilot.buildPropsFromManifest('../manifest.yml');
const lbws = new copilot.LoadBalancedWebService(this, 'Web', props);
const service = lbws.getService(); // returns L2 ECS service construct.                  

One benefit that I can think of with this route is that it would allow you to compose CDK apps made out of Copilot abstractions 👍.
Or were you thinking of a different option?

that takes care of details like knowing where/how to template include

For clarification, when running copilot svc/env override, Copilot will scaffold the following piece of code:

constructor (scope: cdk.App, id: string, props?: cdk.StackProps) {
      super(scope, id, props);
       this.template = new cdk.cloudformation_include.CfnInclude(this, 'Template', {
          templateFile: path.join('.build', 'in.yaml'), 
      });

And everytime copilot svc package or copilot deploy is run Copilot will write the CloudFormation template automatically under .build/in.yaml so no user action is needed.

opinionated/type-safe ways of getting templated resources

Similarly, Copilot will scaffold the following line:

const service = this.template.getResource("Service") as ecs.CfnService;

The service variable is now a type-safe L1 CDK construct: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs.CfnService.html

Do these responses mitigate your concerns?

I think this is great! I have a few questions/comments:

• Will there be a beta we can use before the final release of this feature (if it is released)?
• Currently, any generated cloudformation from copilot isn't visible until its actually deployed to AWS. Would there be a synth-like command that could be run from copilot to show us everything that will be deployed (not just the CDK we've written to override/expand services/jobs etc.)
• I think having constructs that represents the abstracted "service" or "job" etc would be very helpful. That way we can work with a single construct as opposed to working individually with each resource that composes the abstraction/pattern. See: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns-readme.html ... I'm not sure of the patterns here track with what copilot releases.

Hello @benjaminpottier!

I can probably answer your second question - inviting @efekarakus for the other two!

Currently, any generated cloudformation from copilot isn't visible until its actually deployed to AWS. Would there be a synth-like command that could be run from copilot to show us everything that will be deployed (not just the CDK we've written to override/expand services/jobs etc.)

You should be able to run copilot svc package to preview the CFN template that would be generated after applying the CDK code changes. In addition, we are thinking about adding --diff flag to copilot svc package for folks to view the difference between the deployed template and the local template. Would these be helpful for you?

Hello @benjaminpottier!

I can probably answer your second question - inviting @efekarakus for the other two!

Currently, any generated cloudformation from copilot isn't visible until its actually deployed to AWS. Would there be a synth-like command that could be run from copilot to show us everything that will be deployed (not just the CDK we've written to override/expand services/jobs etc.)

You should be able to run copilot svc package to preview the CFN template that would be generated after applying the CDK code changes. In addition, we are thinking about adding --diff flag to copilot svc package for folks to view the difference between the deployed template and the local template. Would these be helpful for you?

That answers my question. Thank you! And yes, diff would be helpful.

We are currently planning to use copilot in "compliance environments" (UK Public Sector), where individual development teams have limited access to AWS APIs (i.e. no ability to apply CF from local terminal).

What this looks like is using Copilot strategically as a tool as part of a larger "platform offering" from a centralised platform team. I do wonder how these advanced override features will evolve over time and be suitable in those scenarios.

My current view is that there would either need to be an abstraction around copilot, or some sort of copilot-managed concept that groups "Applications" into a wider platform, and allows cross-cutting changes to be applied to many teams copilot managed services.

Hi @bpottier, @craigjbass !
Apologies on the late reply, I was at re:invent 😎

Will there be a beta we can use before the final release of this feature (if it is released)?

We can happily share a pre-release binary here before submission to get early feedback on the feature, that's a great idea 👍

I think having constructs that represents the abstracted "service" or "job" etc would be very helpful.

That's really interesting, we can definitely create a L3 construct that takes the same manifest file as an object as input. Similar to in this comment:

const lbws = new copilot.LoadBalancedWebService(this, 'Web', {
  http: {
     path: '/',
     targetPort: 80,
  },
  image: {
     build: './Dockerfile',
  },
});

Are you inclined for this abstraction so that you can have all your infrastructure defined in the CDK?

My current view is that there would either need to be an abstraction around copilot, or some sort of copilot-managed concept that groups "Applications" into a wider platform, and allows cross-cutting changes to be applied to many teams copilot managed services.

Interesting! Can I ask some clarification questions @craigjbass 🥺:

1. Do you deem it's okay if this feature is reserved to the platform team rather than the dev teams?
2. Out of curiosity: How will dev teams make changes to their services? Do they have access to the manifest.yml files and can do a git push to trigger the copilot pipeline to apply their changes?
   3.Is the concern applying the same "extension" across multiple manifest files? If so I think a pattern like this can be possible:

   import {customTransform} from '../../common/transform-copilot.ts'; // Import from a common path or a published node module.
   
   constructor (scope: cdk.App, id: string, props?: cdk.StackProps) {
      super(scope, id, props);
       this.template = new cdk.cloudformation_include.CfnInclude(this, 'Template', {
          templateFile: path.join('.build', 'in.yaml'), 
      });
      customTransform(this.template.getResource("Service");  // Apply the transformation here.
   }

1. Do you deem it's okay if this feature is reserved to the platform team rather than the dev teams?

Yes, although I wonder how we'd enforce that without simply not providing the Copilot CLI to teams. I can see a benefit in giving some but not all of the functionality of the Copilot CLI directly to dev teams. (Mostly around operability)

2. Out of curiosity: How will dev teams make changes to their services? Do they have access to the manifest.yml files and can do a git push to trigger the copilot pipeline to apply their changes?

Unclear yet. Ideally, they would just use the copilot manifest, but if we can't put in the relevant controls - they will probably commit a manifest that a custom tool will read to produce the copilot manifest. Examples of areas where this might be beneficial are for applying security groups without referencing a security group by ID.

3.Is the concern applying the same "extension" across multiple manifest files? If so I think a pattern like this can be possible:

Yes but also in restricting the scope of the contract between platform and dev teams.