How to read entire JSON file and apply remap ?

[sankyfunky]

Discussed in #21261

^{Originally posted by sankyfunky September 10, 2024}
I have the use case where I have the valid JSON files are being generated in a folder.

I want to parse those file and extract events from it.

file01.json and other files are all same format like this.

{
    "scan_data": {
        "version": 1,
        "ip_info": "",
        "hostname": "examplehost.example.com"
    },
    "vulnerabilities": [
        {
            "id": 1001,
            "status": "",
            "secure": false,
            "details": "some details",
            "title": "ICMP Responses Logged"
        },
        {
            "id": 1002,
            "status": "",
            "secure": false,
            "details": "some details",
            "title": "Resolved DNS Name"
        }
    ]
}

events needs to be generated
event01

 {
        "version": 1,
        "ip_info": "",
        "hostname": "examplehost.example.com",
        "id": 1001,
        "status": "",
        "secure": false,
        "details": "some details",
        "title": ""title": "Resolved DNS Name""
}

event02

{
        "version": 1,
        "ip_info": "",
        "hostname": "examplehost.example.com",
        "id": 1002,
        "status": "",
        "secure": false,
        "details": "some details",
        "title": "ICMP Responses Logged"
}

Just like that. I want all events to be streamed.

how should I write my source and transform config.

this is what i have but it reads line by line and I can't parse json. Is it something available out of the box ?

data_dir = "/Users/folder/vector_data/"

[sources.file_source]
  type = "file"
  include = [ "/Users/folder/hs_logs/result/*.json" ]
  read_from = "beginning"

[transforms.asep_transform]
  type = "remap"
  inputs = ["file_source"]
  source = '''
    . = parse_json!(.message)
  '''

[sinks.console]
  type = "console"
  inputs = ["asep_transform"]
  target = "stdout"
  encoding.codec = "json"

[jszwedko]

Closing since this is a discussion rather than an issue.