Add http connector to quickstart setup (#509)

* add file http_pipeline.yml to integrate http connector
---------

Co-authored-by: ekneg54 <[email protected]>

README.md

@@ -394,11 +394,11 @@ Opensearch Dashboards. Following services are available after start up:
 The example rules that are used in the docker instance of Logprep can be found
 in `quickstart/exampledata/rules`.
 Example events that trigger for the example rules can be found in
-`quickstart/exampledata/input_logdata/test_input.jsonl`.
+`quickstart/exampledata/input_logdata/logclass/test_input.jsonl`.
 These events can be added to Kafka with the following command:
 
 ```bash
-(docker exec -i kafka kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic consumer) < exampledata/input_logdata/test_input.jsonl
+(docker exec -i kafka kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic consumer) < exampledata/input_logdata/logclass/test_input.jsonl
 ```
 
 Once the events have been processed for the first time, the new indices *processed*, *sre*

quickstart/exampledata/config/http_pipeline.yml

@@ -0,0 +1,25 @@
+version: 1
+
+metrics:
+  enabled: true
+  port: 8000
+
+input:
+  httpinput:
+      type: http_input
+      uvicorn_config:
+          host: 0.0.0.0
+          port: 9000
+      endpoints:
+          /lab/123/ABC/auditlog: jsonl
+output:
+  kafka:
+    type: confluentkafka_output
+    topic: consumer
+    error_topic: errors
+    flush_timeout: 300
+    send_timeout: 0
+    kafka_config:
+      bootstrap.servers: 127.0.0.1:9092
+      compression.type: gzip
+      statistics.interval.ms: "60000"

quickstart/exampledata/input_logdata/logclass/config.yaml

@@ -0,0 +1,7 @@
+target_path: /lab/123/ABC/auditlog
+timestamps:
+  - key: TIMESTAMP_FIELD_1
+    format: "%Y%m%d"
+  - key: TIMESTAMP_FIELD_1
+    format: "%H%M%S"
+    time_shift: "+0200"