[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

brijesh-elastic · 2025-06-24T15:13:24Z

Proposed commit message

aws: Update inspector data stream for Cloud Detection and Response (CDR) workflow

Add ECS mappings and latest transform to aws.inspector datastream 
to help with the Cloud Native Vulnerability Management (CNVM)[1] workflow.

Enabled agentless deployment for the `inspector` datastream.
Parse and map the newly introduced fields in the `inspector` datastream.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html

Note

To Reviewers:

Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws directory.
Run the following command to run tests.

elastic-package test

Related issues

…CDR) workflow

elasticmachine · 2025-06-24T15:13:29Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

brijesh-elastic · 2025-06-24T15:38:06Z

@maxcold, I've updated the integration title and all other references of AWS Inspector to Amazon Inspector. See Product Page and Documentation Page.

elastic-vault-github-plugin-prod · 2025-06-24T16:58:37Z

🚀 Benchmarks report

Package `aws` 👍(17) 💚(2) 💔(3)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`inspector`	1652.89	1044.93	-607.96 (-36.78%)	💔
`lambda_logs`	10526.32	8264.46	-2261.86 (-21.49%)	💔
`s3access`	5464.48	3952.57	-1511.91 (-27.67%)	💔

To see the full report comment with /test benchmark fullreport

efd6 · 2025-06-25T06:00:36Z

packages/aws/_dev/build/docs/inspector.md

+     - grant use of transforms, and
+     - grant access to source and destination indices
+   For more details on Transform Setup, refer to the link [here](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup)
+3. Because the latest copy of vulnerabilities is now indexed in two places, i.e., in both source and destination indices, users must anticipate storage requirements accordingly.


Suggested change

3. Because the latest copy of vulnerabilities is now indexed in two places, i.e., in both source and destination indices, users must anticipate storage requirements accordingly.

3. Because the latest copy of vulnerabilities is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.

The number in the first clause is confusing; is it one copy for each of the vulnerabilities (arguably plural) or is it one copy of the set of vulnerabilities (singular but would be helped by inclusion of "set of")?

efd6 · 2025-06-25T06:01:03Z

packages/aws/changelog.yml

+  - description: Enhancements for Guardduty dashboards.
+    type: enhancement
+    link: https://github.com/elastic/integrations/pull/13542


Please revert this change.

efd6 · 2025-06-25T06:01:57Z

packages/aws/data_stream/inspector/_dev/deploy/docker/files/config.yml

@@ -4,4 +4,4 @@ rules:
    responses:
      - status_code: 200
        body: |
-          {"findings":[{"awsAccountId":"123456789","description":"Findins message","findingArn":"arn:aws:s3:::sample","firstObservedAt":"1.663703546405E9","inspectorScore":1.2,"inspectorScoreDetails":{"adjustedCvss":{"adjustments":[{"metric":"Base","reason":"use Base metric"}],"cvssSource":"scope1","score":8.9,"scoreSource":"scope2","scoringVector":"Attack Vector","version":"v3.1"}},"lastObservedAt":"1.663703546405E9","networkReachabilityDetails":{"networkPath":{"steps":[{"componentId":"02ce3860-3126-42af-8ac7-c2a661134129","componentType":"type"}]},"openPortRange":{"begin":1234,"end":4567},"protocol":"TCP"},"packageVulnerabilityDetails":{"cvss":[{"baseScore":1.1,"scoringVector":"Attack Vector","source":"scope3","version":"v3.1"}],"referenceUrls":["https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6111"],"relatedVulnerabilities":["security"],"source":"example","sourceUrl":"https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6111","vendorCreatedAt":"1.663703546405E9","vendorSeverity":"basic","vendorUpdatedAt":"1.663703546405E9","vulnerabilityId":"123456789","vulnerablePackages":[{"arch":"arch","epoch":123,"filePath":"/example","fixedInVersion":"3","name":"example","packageManager":"BUNDLER","release":"release","sourceLayerHash":"50d858e0985ecc7f60418aaf0cc5ab587f42c2570a884095a9e8ccacd0f6545c","version":"2.0"}]},"remediation":{"recommendation":{"text":"example","Url":"https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6111"}},"resources":[{"details":{"awsEc2Instance":{"iamInstanceProfileArn":"arn:aws:s3:::iam","imageId":"123456789","ipV4Addresses":["89.160.20.128","81.2.69.192"],"ipV6Addresses":["2a02:cf40::"],"keyName":"sample","launchedAt":"1.663703546405E9","platform":"EC2","subnetId":"123456","type":"Instance","vpcId":"3265875"},"awsEcrContainerImage":{"architecture":"arch","author":"example","imageHash":"50d858e0985ecc7f60418aaf0cc5ab587f42c2570a884095a9e8ccacd0f6545d","imageTags":["sample"],"platform":"ECR","pushedAt":"1.663703546405E9","registry":"ecr registry","repositoryName":"sample"}},"id":"12345678","partition":"partition","region":"us-east-1","tags":{"string1":"string1","string2":"string2"},"type":"AWS_EC2_INSTANCE"}],"severity":"INFORMATIONAL","status":"ACTIVE","title":"sample findings","type":"NETWORK_REACHABILITY","updatedAt":"1.663703546405E9"}]}
+          {"findings":[{"awsAccountId":"123456789012","description":"The tokenizer incorrectly interprets tags with unquoted attribute values that end with a solidus character (/) as self-closing. When directly using Tokenizer, this can result in such tags incorrectly being marked as self-closing, and when using the Parse functions, this can result in content following such tags as being placed in the wrong scope during DOM construction, but only when tags are in foreign content (e.g. <math>, <svg>, etc contexts).","epss":{"score":0.00024},"exploitAvailable":"NO","findingArn":"arn:aws:inspector2:us-east-2:123456789012:finding/fb6294abcdef0123456789abcdef8123","firstObservedAt":1748539687.919,"fixAvailable":"YES","inspectorScore":6.5,"inspectorScoreDetails":{"adjustedCvss":{"adjustments":[],"cvssSource":"NVD","score":6.5,"scoreSource":"NVD","scoringVector":"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:L/I:L/A:L","version":"3.1"}},"lastObservedAt":1749165796.162,"packageVulnerabilityDetails":{"cvss":[{"baseScore":6.5,"scoringVector":"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:L/I:L/A:L","source":"NVD","version":"3.1"},{"baseScore":6.5,"scoringVector":"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:L/I:L/A:L","source":"NVD","version":"3.1"}],"referenceUrls":["https://groups.google.com/g/golang-announce/c/ezSKR9vqbqA","https://nvd.nist.gov/vuln/detail/CVE-2025-22872","https://alas.aws.amazon.com/AL2023/ALAS-2025-981.html","https://alas.aws.amazon.com/AL2/ALASDOCKER-2025-064.html","https://alas.aws.amazon.com/AL2023/ALAS-2025-980.html","https://alas.aws.amazon.com/AL2/ALASDOCKER-2025-063.html","https://alas.aws.amazon.com/AL2023/ALAS-2025-979.html","https://alas.aws.amazon.com/cve/json/v1/CVE-2025-22872.json","https://alas.aws.amazon.com/AL2/ALAS-2025-2863.html","https://alas.aws.amazon.com/cve/json/v1/CVE-2025-22872.json"],"relatedVulnerabilities":[],"source":"NVD","sourceUrl":"https://nvd.nist.gov/vuln/detail/CVE-2025-22872","vendorCreatedAt":1744827364,"vendorSeverity":"MEDIUM","vendorUpdatedAt":1747437319,"vulnerabilityId":"CVE-2025-22872","vulnerablePackages":[{"epoch":0,"filePath":"vol-0e47545061282cd35:/p1:opt/cni/bin/aws-cni","fixedInVersion":"0.38.0","name":"golang.org/x/net","packageManager":"GOBINARY","version":"v0.1.0"},{"epoch":0,"filePath":"vol-0e47545061282cd35:/p1:etc/eks/image-credential-provider/ecr-credential-provider","fixedInVersion":"0.38.0","name":"golang.org/x/net","packageManager":"GOBINARY","version":"v0.30.0"},{"epoch":0,"filePath":"vol-0e47545061282cd35:/p1:opt/cni/bin/dhcp","fixedInVersion":"0.38.0","name":"golang.org/x/net","packageManager":"GOBINARY","version":"v0.30.0"},{"epoch":0,"filePath":"vol-0e47545061282cd35:/p1:usr/bin/aws-iam-authenticator","fixedInVersion":"0.38.0","name":"golang.org/x/net","packageManager":"GOBINARY","version":"v0.30.0"},{"epoch":0,"filePath":"vol-0e47545061282cd35:/p1:usr/bin/kubelet","fixedInVersion":"0.38.0","name":"golang.org/x/net","packageManager":"GOBINARY","version":"v0.30.0"},{"arch":"X86_64","epoch":0,"fixedInVersion":"0:2.0.5-1.amzn2.0.1","name":"nerdctl","packageManager":"OS","release":"1.amzn2.0.1","remediation":"yum update nerdctl","version":"2.0.4"}]},"remediation":{"recommendation":{"text":"None Provided"}},"resources":[{"details":{"awsEc2Instance":{"iamInstanceProfileArn":"arn:aws:iam::123456789012:instance-profile/eks-0012345a-1234-5678-1234-6c1abcdef012","imageId":"ami-0e0f0123456789abd","ipV4Addresses":["10.90.1.245","10.90.1.45","10.90.1.168","10.90.1.157","1.128.0.1","10.90.1.103","10.90.1.197","10.90.1.220","10.90.1.86","10.90.1.29","10.90.1.18","10.90.1.181","10.90.1.161","10.90.1.229","10.90.1.108","10.90.1.219","10.90.1.9","10.90.1.106","10.90.1.206"],"ipV6Addresses":[],"launchedAt":1748534768,"platform":"AMAZON_LINUX_2","subnetId":"subnet-0ababcdefabcdef8b","type":"t3.medium","vpcId":"vpc-04ab0123456789123"}},"id":"i-0fabcdefabcdef50b","partition":"aws","region":"us-east-2","tags":{"aws:autoscaling:groupName":"eks-sei_demo_prod_linux-00c12345-abcd-1234-5678-601234567896","aws:ec2launchtemplate:version":"6","aws:eks:cluster-name":"sei_demo_prod","eks:cluster-name":"sei_demo_prod","eks:nodegroup-name":"sei_demo_prod_linux","k8s.io/cluster-autoscaler/enabled":"true","k8s.io/cluster-autoscaler/sei_demo_prod":"owned","kubernetes.io/cluster/sei_demo_prod":"owned"},"type":"AWS_EC2_INSTANCE"}],"severity":"MEDIUM","status":"ACTIVE","title":"CVE-2025-22872 - golang.org/x/net, golang.org/x/net and 4 more","type":"PACKAGE_VULNERABILITY","updatedAt":1749165796.162}]}


Pretty print the JSON and use minify_json.

efd6 · 2025-06-25T06:10:46Z

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

+      tag: set_vulnerability_cve
+      copy_from: aws.inspector.package_vulnerability_details.vulnerability_id
+      ignore_empty_value: true
+      if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY' && ctx.aws?.inspector?.package_vulnerability_details?.vulnerability_id.startsWith('CVE')


Suggested change

if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY' && ctx.aws?.inspector?.package_vulnerability_details?.vulnerability_id.startsWith('CVE')

if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY' && ctx.aws?.inspector?.package_vulnerability_details?.vulnerability_id?.startsWith('CVE') == true

(if ctx.aws?.inspector?.package_vulnerability_detail?.vulnerability_id is null, the call will fail and even with the ? protecting the call, it will resolve to null, which is not a boolean, so we cannot use the expression as a boolean)

efd6 · 2025-06-25T06:18:21Z

packages/aws/kibana/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b.json

@@ -551,9 +627,10 @@
                        "description": "",
                        "params": {
                            "fontSize": 13,
-                            "markdown": "[Inspector Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",
+                            "markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",


Suggested change

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

"markdown": "**Inspector Findings Overview Dashboard** | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

We don't need a link to the page we are on.

efd6 · 2025-06-25T06:19:04Z

packages/aws/kibana/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139.json

@@ -328,9 +409,10 @@
                        "description": "",
                        "params": {
                            "fontSize": 13,
-                            "markdown": "[Inspector Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",
+                            "markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",


Suggested change

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | **Inspector Vulnerabilities Dashboard** | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

efd6 · 2025-06-25T06:19:46Z

packages/aws/kibana/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139.json

@@ -359,9 +420,10 @@
                        "description": "",
                        "params": {
                            "fontSize": 13,
-                            "markdown": "[Inspector Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",
+                            "markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",


Suggested change

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | **Inspector Severity Dashboard** | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

efd6 · 2025-06-25T06:20:16Z

packages/aws/kibana/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139.json

@@ -550,9 +665,10 @@
                        "description": "",
                        "params": {
                            "fontSize": 13,
-                            "markdown": "[Inspector Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",
+                            "markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139)  ",


Suggested change

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) | [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) | [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) | **Inspector EC2 and ECR Overview Dashboard**",

brijesh-elastic · 2025-06-25T06:21:26Z

packages/aws/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - aws.inspector.transform_unique_id


@maxcold, I'm using the aws.inspector.transform_unique_id field as the unique key in the transform because if I use individual fields (vulnerability.id, resource.id, package.id, package.version), the transforms behave strangely. The destination index has a higher event count than the source index, which seems to be caused by the fields resource.id, package.id, and package.version being arrays.

@maxcold, here's the logic for calculating aws.inspector.transform_unique_id

integrations/packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

Lines 1150 to 1154 in bbd4c37

- set:

field: aws.inspector.transform_unique_id

tag: set_transform_unique_id

value: '{{vulnerability.id}}|{{resource.id}}|{{package.name}}|{{package.version}}'

if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY'

thanks! I think it makes sense. I wonder when resource.id can be an array. I guess we had similar situation with AWS Security Hub when resource was defined as array in the docs but we couldn't find any example in the real data. Did you see this case with Amazon Inspector in the wild?
anyway, the unique key logic seems reasonable to me!

In Amazon Inspector also we're getting the single element in resources.

guess we had similar situation with AWS Security Hub when resource was defined as array in the docs but we couldn't find any example in the real data. Did you see this case with Amazon Inspector in the wild?

It is similar to AWS SecurityHub.
Added a comment here: https://github.com/elastic/security-team/issues/10753#issuecomment-2921417448

kcreddy · 2025-06-25T08:21:59Z

packages/aws/_dev/build/docs/inspector.md

@@ -1,15 +1,19 @@
 # Inspector

-The [AWS Inspector](https://docs.aws.amazon.com/inspector/) integration collects and parses data from AWS Inspector [Findings](https://docs.aws.amazon.com/inspector/v2/APIReference/API_ListFindings.html) REST APIs.
+The [Amazon Inspector](https://docs.aws.amazon.com/inspector/) integration collects and parses data from Amazon Inspector [Findings](https://docs.aws.amazon.com/inspector/v2/APIReference/API_ListFindings.html) REST APIs.


@jamiehynds @cpascale43 , we are changing AWS Inspector into Amazon Inspector as it should be with this change. We are only changing UI elements and not updating any field names, so the custom fields are still aws.inspector.*.
Let me know if you have any concerns.

kcreddy · 2025-06-25T08:34:42Z

packages/aws/manifest.yml

@maxcold, current kibana version constraint is following:

kibana: version: "~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0"

In Inspector, we don't have missing CVE values for package vulnerabilities. Also, it is not an array and always contain single values. For example: vulnerability.id: CVE-2025-38000

Do we still need to bump the minimum versions here? Related to multiple/empty CVEs: #14079 (comment)

@kcreddy thanks for highlighting it! I need to test the integration with 8.18 and 9.0 to check that. Right now I'm testing with the latest 9.1 snapshot, will play with 8.18/9.0 version later

When testing 8.18 I realised that we don't support multiple packages in this version and also have some other rough edges. I think we will need to bump to ^8.19.0 || ^9.1.0" but let's wait for product feedback as well

aws integration has shared ownership. If we are completely removing support for 8.17 and 8.18, we need everyone's approval on this.
Is it possible to backport this support multiple packages to 8.17.9 and 8.18.4?

kcreddy · 2025-06-25T08:39:08Z

packages/aws/data_stream/inspector/fields/package.yml

+    - name: architecture
+      type: keyword
+      external: ecs


This is existing ECS field. Does the build or tests fail if its not defined manually?
Similar for other ECS fields.

kcreddy · 2025-06-25T08:40:02Z

packages/aws/data_stream/inspector/fields/vulnerability.yml

+    - name: cve
+      type: keyword
+    - name: published_date
+      type: date
+    - name: title
+      type: keyword


Add descriptions per CDR guide.

kcreddy · 2025-06-25T08:40:20Z

packages/aws/data_stream/inspector/fields/resource.yml

+- name: resource
+  type: group
+  fields:
+    - name: id


Add description per CDR guide

kcreddy · 2025-06-25T08:43:26Z

packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: observer.vendor
+      tag: set_observer_vendor
+      value: Amazon Inspector
+  - set:
+      field: vulnerability.scanner.vendor
+      tag: set_vulnerability_scanner_vendor
+      value: Amazon Inspector


@maxcold, this is set to Amazon Inspector as opposed to AWS Inspector as discussed before. FYI.
https://docs.aws.amazon.com/inspector/. Let me know if you have any concern.

I think it works for CDR usecases!

kcreddy · 2025-06-25T08:45:36Z

packages/aws/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - aws.inspector.transform_unique_id


@maxcold, here's the logic for calculating aws.inspector.transform_unique_id

integrations/packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

Lines 1150 to 1154 in bbd4c37

- set:

field: aws.inspector.transform_unique_id

tag: set_transform_unique_id

value: '{{vulnerability.id}}|{{resource.id}}|{{package.name}}|{{package.version}}'

if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY'

maxcold · 2025-06-25T08:53:20Z

@kcreddy @brijesh-elastic While testing I see that we don't have the resource.name , I guess it was discussed during the mapping, but I realised that for EC2 instances we can rely on the Name tag (it might not be present, but when it is it represents the instance name). Wdyt about using it as resource.name?

kcreddy · 2025-06-26T10:16:56Z

@kcreddy @brijesh-elastic While testing I see that we don't have the resource.name , I guess it was discussed during the mapping, but I realised that for EC2 instances we can rely on the Name tag (it might not be present, but when it is it represents the instance name). Wdyt about using it as resource.name?

@maxcold, that sounds reasonable. We will also populate host.name based on this tag (only if present) as we don't have that as well.

brijesh-elastic · 2025-06-27T07:21:03Z

@kcreddy @brijesh-elastic While testing I see that we don't have the resource.name , I guess it was discussed during the mapping, but I realised that for EC2 instances we can rely on the Name tag (it might not be present, but when it is it represents the instance name). Wdyt about using it as resource.name?

@maxcold, that sounds reasonable. We will also populate host.name based on this tag (only if present) as we don't have that as well.

Populated resource.name and host.name in the 34a8fd6

elasticmachine · 2025-06-27T08:53:14Z

💚 Build Succeeded

Buildkite Build
Commit: b9b2494

History

💚 Build #27693 succeeded 671472d
💚 Build #27603 succeeded bbd4c37

cc @brijesh-elastic

elastic-sonarqube · 2025-06-27T08:53:21Z

Quality Gate failed

Failed conditions
65.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

Update inspector datastream to support Cloud Detection and Response (…

9ff387c

…CDR) workflow

brijesh-elastic self-assigned this Jun 24, 2025

brijesh-elastic added the Integration:aws AWS label Jun 24, 2025

brijesh-elastic requested review from a team as code owners June 24, 2025 15:13

Update changelog entry

bbd4c37

efd6 reviewed Jun 25, 2025

View reviewed changes

brijesh-elastic commented Jun 25, 2025

View reviewed changes

kcreddy requested a review from a team June 25, 2025 08:18

kcreddy reviewed Jun 25, 2025

View reviewed changes

address pr comment

671472d

brijesh-elastic requested review from efd6 and kcreddy June 26, 2025 09:59

brijesh-elastic and others added 2 commits June 27, 2025 12:48

Populate resource.name and host.name when tags[].Name present

34a8fd6

Merge branch 'main' into aws-4.0.0

b9b2494

brijesh-elastic requested a review from maxcold June 27, 2025 07:26

	3. Because the latest copy of vulnerabilities is now indexed in two places, i.e., in both source and destination indices, users must anticipate storage requirements accordingly.
	3. Because the latest copy of vulnerabilities is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.

	if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY' && ctx.aws?.inspector?.package_vulnerability_details?.vulnerability_id.startsWith('CVE')
	if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY' && ctx.aws?.inspector?.package_vulnerability_details?.vulnerability_id?.startsWith('CVE') == true

	"markdown": "[Inspector Findings Overview Dashboard](#/dashboard/aws-131a1550-5a0b-11ed-a807-bd2da8f2e79b) \| [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) \| [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) \| [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",
	"markdown": "Inspector Findings Overview Dashboard \| [Inspector Severity Dashboard](#/dashboard/aws-60881ab0-63e0-11ed-be08-4b4db5223139) \| [Inspector Vulnerabilities Dashboard](#/dashboard/aws-383d4630-63df-11ed-be08-4b4db5223139) \| [Inspector EC2 and ECR Overview Dashboard](#/dashboard/aws-63984b70-63e1-11ed-be08-4b4db5223139) ",

	- set:
	field: aws.inspector.transform_unique_id
	tag: set_transform_unique_id
	value: '{{vulnerability.id}}\|{{resource.id}}\|{{package.name}}\|{{package.version}}'
	if: ctx.aws?.inspector?.type == 'PACKAGE_VULNERABILITY'

[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

Are you sure you want to change the base?

[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

Conversation

brijesh-elastic commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jun 24, 2025

Uh oh!

brijesh-elastic commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package aws 👍(17) 💚(2) 💔(3)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold commented Jun 25, 2025

Uh oh!

kcreddy commented Jun 26, 2025

brijesh-elastic commented Jun 24, 2025 •

edited

Loading

brijesh-elastic commented Jun 24, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Jun 24, 2025 •

edited

Loading

Package `aws` 👍(17) 💚(2) 💔(3)

kcreddy Jun 25, 2025 •

edited

Loading

kcreddy Jun 25, 2025 •

edited

Loading

kcreddy Jun 26, 2025 •

edited

Loading

kcreddy Jun 25, 2025 •

edited

Loading