Skip to content

Better "expected results" in ADA #668

Open
@rgardler-msft

Description

@rgardler-msft

A common failure in ADA is the result of poor result formatting. In my experience we can significantly improve on this by having the generation use a regular expression for validation rather than the similarity. This is because the system will often use truncated results (which is good from a readability perspective). For example:

{
    "agentPoolProfiles": [
        {
            "name": "nodepool1",
            "count": 1,
            "vmSize": "Standard_DS2_v2",
            "maxPods": 110,
            "osType": "Linux"
        }
    ],
    "dnsPrefix": "AKSClusterxxx-dns",
    "provisioningState": "Succeeded",
    "resourceGroup": "AKSLabResourceGroupxxx",
    "name": "AKSClusterxxx",
    "type": "Microsoft.ContainerService/ManagedClusters",
    ...
} 

However, this is not valid JSON and thus the validation step fails.

In this case, and in many like it, what we are actually looking for is "provisioningState": "Succeeded". Therefore for responses like this one we can use:

<!-- expected_similarity=.*Succeeded -->

On a similar note, it seems the tool gives a default similarity value of 0.3 which, for the most part, is too low to be reliable.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions