Consume Aspire Resource Metadata to Influence IaC generation

[ellismg]

Aspire is adding the concept of a metadata bag on each resource, which can be used to smuggle information from the app host into consumers of the manifest.

dotnet/aspire#1619

We'd like to use this as a way to enable a way to make targeted configuration changes to the generated IaC (to save folks the trouble of running infra synth, modifying the bicep or yaml and then keeping it up to date.

We should support this in azd and also understand a set of metadata we define that allows customization of our bicep and yaml generation.

A good end to end here would be the ability in the app host to add metadata to a project resource with keys like: azd.minInstances and azd.maxInstances which would control the min and max instance count in the generated container app.

We may also use this technology to control things like "redis.v0" means "use a production version instead of the ACA dev service".

Let's use this issue to track building the infrastructure needed to do this and one small end to end and then sit and think about the knobs we want to add across the components that continue to exist after we do the bicep.v0 resource work and deprecate support for all the azure. resources.

[davidfowl]

We should support this in azd and also understand a set of metadata we define that allows customization of our bicep and yaml generation.

I was thinking we should look for a generic solution here. For the bicep generation we're moving to bicep resources so that basically out of azd's hand (at least for those resources). That leaves a couple of components where infra synth is required to tweak:

1. The resources that azd creates for the compute environment (resources.bicep):

   • Container registry
   • Managed identity
   • LA workspace
   • Container apps environment

2. The container apps themselves (containerApp.yaml.tmpl)

Invisible resources are a problem because there's no representation for them in the app model at all. I'm not sure we'll have time to model them or not, but we should enable modifying arbitrary container apps properties. I think we want this to be as flexible as possible (following the pattern of the bicep resource).

Since we're on the path with the bicep resource, it seems like it would make sense to model the container app manifest as more bicep. If we went this route, aspire would attach/add the container app bicep resource to the manifest and azd would produce the right scaffolding to make this work.

Still working through a proposal.

[ellismg]

Still working through a proposal.

Yeah - I was looking at this again and in light of where I think we are moving, I'm not sure if we'll need this in the fullness of time.

I think there are three "categories" of resource types floating around today:

1. Cloud Agnostics Resources (e.g. redis.v0, postgres.v0, etc). Today azd does the infrastructure generation for these resource types and infra syth is the customization story.
2. Cloud specific resources (e.g. stuff from Aspire.Hosting.Azure). Today these are special azure.XYZ resources that azd understands, but we are moving to a world where they no longer exist and instead Aspire.Hosting.Azure produces some bicep which we include in our graph.
3. Aspire specific resources like project.v0 which implicitly have cloud resources (like the container app environment and container app) which back them.

For (2) the plan is to move to a world where you have full control over customization (via the C# type system). For (1), I think we are thinking we'd be on the same plan, where you'd be able to write some code in your app host that sets what implementation you want to use at deployment time.

For (3) (which may only be project.v0) resources I'm still not sure what the customization plan is. I think in the fullness of time we want to support a model like (1) where you can control the IaC at the C# layer (e.g. some sort of .WithContainerAppForDeployment(app => { /* party on a type from Azure.ResourceManager.AppContainers or something */}) and this just generates bicep (or the yaml rendering of the resource) and we deploy that.

In that world, I can see how we don't need this at all.

But I'm also not sure if we get the project stuff figured out for GA, so I'm not sure if we want to use this as a way to control metadata on the app itself in the short term.

Invisible resources are a problem because there's no representation for them in the app model at all. I'm not sure we'll have time to model them or not, but we should enable modifying arbitrary container apps properties. I think we want this to be as flexible as possible (following the pattern of the bicep resource).

I think the right way for us to do this is to have metadata on parameters that say "what type" you need and azd tries to wire up the right thing (creating a default one if needed). Then, over time, as more and more of these invisible resources are generated on the C# side of the house, you can just drop the parameters from the .bicep file you generate, and azd can use that as a signal that it need not generate these resources anymore (if nothing requests them).

[rajeshkamal5050]

• Not in scope for Aspire Preview 4. Assigning to the Germanium bucket.
• We can pull this back into milestones when we firm up on the proposal/scope.

@ellismg @davidfowl

[davidfowl]

For (2) the plan is to move to a world where you have full control over customization (via the C# type system). For (1), I think we are thinking we'd be on the same plan, where you'd be able to write some code in your app host that sets what implementation you want to use at deployment time.

Right, we got here in preview4 with the bicep resource. So the abstract resource types will go away and azd will stich things together.

For (3) (which may only be project.v0) resources I'm still not sure what the customization plan is. I think in the fullness of time we want to support a model like (1) where you can control the IaC at the C# layer (e.g. some sort of .WithContainerAppForDeployment(app => { /* party on a type from Azure.ResourceManager.AppContainers or something */}) and this just generates bicep (or the yaml rendering of the resource) and we deploy that.

In that world, I can see how we don't need this at all.

There's a world in the future where we can use the cdk to spit out main.bicep, but we're not there yet.

But I'm also not sure if we get the project stuff figured out for GA, so I'm not sure if we want to use this as a way to control metadata on the app itself in the short term.

Right, I think we want to spend a little bit of time thinking about modeling not just the container apps settings but also the compute environment in the generated resources.bicep (keyvaults + LA workspaces). Even if we don't make it for GA and we have a backup plan (using metadata).

It'll allow us to remove some of the magic inference that azd is doing today. Some of that inference is stuff that we can easily represent via some contract, the rest of it we would keep in azd because it's a little harder to describe.

I think the right way for us to do this is to have metadata on parameters that say "what type" you need and azd tries to wire up the right thing (creating a default one if needed). Then, over time, as more and more of these invisible resources are generated on the C# side of the house, you can just drop the parameters from the .bicep file you generate, and azd can use that as a signal that it need not generate these resources anymore (if nothing requests them).

I think more fine-grained metadata is needed or people will end up doing infra synth way more than expected. Originally I was thinking something like JSON patch via metadata but the more time I spent playing with various approaches, the one I liked best was something like this:

{
  "resources": {
    "storage": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.storage.bicep",
      "params": {
        "principalId": "{mid.outputs.principalId}",
        "principalType": "ServicePrincipal",
        "storageName": "storage"
      }
    },
    "containerAppEnviorment": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.containerAppEnviorment.bicep"
    },
    "blobs": {
      "type": "azure.bicep.v0",
      "connectionString": "{storage.outputs.blobEndpoint}",
      "parent": "storage"
    },
    "api-containerApp": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.containerApp.bicep",
      "params": {
        "containerAppName": "{api.name}",
        "containerAppImage": "{api.containerImage}",
        "containerAppPort": "8080",
        "containerRegistryName": "{containerAppEnviorment.outputs.containerRegistry}",
        "containerRegisteryIdentity" : "{containerAppEnviorment.outputs.containerRegistryIdentity}",
        "minReplicas": "1",
        "maxReplicas": "10",
        "userAssignedIdentities": ["{mid.outputs.principalId}", "{containerAppEnviorment.outputs.containerRegistryIdentity}"]
      }
    },
    "api": {
      "type": "project.v0",
      "path": "../WebApplication1/WebApplication1.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__blobs": "{blobs.connectionString}",
        "AZURE_CLIENT_ID": "{containerAppEnviorment.outputs.clientId}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    }
  }
}

In the above containerAppEnviorment replaces resources.bicep, at least the core of it. To make this work we would need a contract about what this module looked like inputs and outputs wise. This feels achievable but I wouldn't want to make this standard until we better understood AKS (or some other compute environment). This would allow aspire to define the bicep for the compute environment and would allow users to customize it without dropping to infra synth.

The second part is less fleshed out, but there would be a container apps bicep file that references properties from the project itself. If you look at the "api-containerApp" resource, this is just a normal bicep resource that references some magic properties from the project (api.containerImage). In the above pattern, azd would understand the dependency graph here such that these bicep resources get pushed into the deploy phase (they depend on the project outputs).

I haven't figured out how to model secrets or urls in this world as yet. That part is still a bit magical. I feel if we iterated on this we could come up with something reasonable. The container app bicep becomes the way that you break glass to set whatever properties you want to and azd doesn't have to keep exposing properties piecemeal and users don't need to do infra synth to change settings.

My end goal would be that infra synth isn't used as an escape hatch, it's used to store things in source control. You never tweak the outputs, you always tweak the apphost and then see the what if via infra synth.

[DamianEdwards]

What are next steps to make progress on this in preview.5? For your last strawman proposal @davidfowl are you thinking this model requires the user to reference Aspire.Hosting.Azure? What happens to the default story? Does azd instruct the user to add Aspire.Hosting.Azure if the required resources are not in the manifest, or does azd keep some of its magic inference for the vanilla-app scenario?

[mitchdenny]

I think that this is probably a two step process. And we need to see what we can achieve by GA. I think the next click stop is to take over generating these resources but still generate the manifest and get AZD to effectively do the main.bicep piece.

I'd prefer to define these resources via CDK constructs and there are a few missing:

• AppInsights
• Log Workspaces
• Container Apps / Container Environments
• Container Registry

We need to finish the CDK constructs for the existing resource we have for P5. If we want more (like the ones above) we need tdo let the AzSDK team know ASAP as they might need to get more people working on it.

I think we could probably adapt the Azure Provisioner lifecycle hook that we have today to generate the extra resources in the App Model (or just create a new one that injects them on publish mode) - once we have the CDK constructs available).

[mitchdenny]

Oh and we'll need a mechanism to signal to AZD we are working in this mode. I would prefer that we don't break the existing approach until this approach is fully in and working. So maybe some kind of feature flag in AZD?

[davidfowl]

@vhvb1989 and I have been discussing how to take over resources.bicep.

[ellismg]

Oh and we'll need a mechanism to signal to AZD we are working in this mode. I would prefer that we don't break the existing approach until this approach is fully in and working. So maybe some kind of feature flag in AZD?

We can do feature flags - or we can figure out some way to reflect this in the manifest. Part of me is wondering if we should add an azure.aspireRuntime.containerApps.v0 resource or something to the manifest for cases where azd needs to generate the container app environment, acr and supporting infrastructure. If we had that, then it might be easier to signal when we are on the new plan (since it no longer shows up in the graph)? I know long term we wouldn't have this resource (and I suspect in the CDK world it's like an L2 or L3 construct that you add to the builder via a gesture like builder.UseAzureContainerAppsRuntime() (and then later there is a builder.UseAzureKurbernetesServiceRuntime() or something) which ends up just injecting some bicep.v0 resources - but perhaps having it be a first class thing in the manifest until then would be helpful? It would let us start to define the shape of the output contract perhaps?

The second part is less fleshed out, but there would be a container apps bicep file that references properties from the project itself. If you look at the "api-containerApp" resource, this is just a normal bicep resource that references some magic properties from the project (api.containerImage). In the above pattern, azd would understand the dependency graph here such that these bicep resources get pushed into the deploy phase (they depend on the project outputs).

We should consider tying together the project.v0 resource and the bicep that represents the "runtime configuration" somehow. Perhaps this bicep.v0 can have a parent that ties it to the project resource? azd could take that as a signal that it should not do generation?