This processor can parse CSV data and stores it as individual fields. This filter can also parse data with any separator, not just commas.
Elastic introduced in the version 7.6 an official CSV processor for this reason I will not maintain anymore this project.
In this version the processor handles multiple patterns: it will consider the first one matching.
There is also an optional field, key_field
, that allows to store the key value of the matching pattern.
PUT _ingest/pipeline/csv-pipeline
{
"description": "A pipeline to do whatever",
"processors": [
{
"csv" : {
"field" : "my_field",
"columns" : {
"csv_pattern1" : [ "a","b","c" ],
"csv_pattern2" : [ "a","b" ],
"csv_pattern3" : [ "a","b","c","d" ]
}
}
}
]
}
PUT /my-index/my-type/1?pipeline=csv-pipeline
{
"my_field" : "a_value,b_value"
}
GET /my-index/my-type/1
{
"my_field" : "a_value,b_value",
"a" : "a_value",
"b" : "b_value"
}
Parameter | Use | Required |
---|---|---|
field | Field name of where to read the content from. | Yes |
columns | Define multiple maps, each one with a list of column names. | Yes |
quote_char | Define the character used to quote CSV fields. If this is not specified the default is a double quote " . |
No |
separator | Define the column separator value. If this is not specified, the default is a comma , . |
No |
key_field | Define the field where to save the key value of the matching pattern in columns (for example csv_pattern1 ,csv_pattern2 or csv_pattern3 in the example). If not specified, this field will not be saved. |
No |
max_chars | Define the max number of characters that column can contain. Default is 32766 . |
No |
In order to install this plugin, you need to create a zip distribution first by running
gradle clean check
This will produce a zip file in build/distributions
.
After building the zip file, you can install it like this
bin/plugin install file:///path/to/ingest-csv/build/distribution/ingest-csv-0.0.1.zip
If you need to work on a different version, you can try to change the setting in gradle.properties
before compiling. Use at your own risk!
- Need more test, like using separator and quote_char