Raw event 검색 성능 개선

[syncpark]

Issue

It takes 12 seconds to search events with 2 filtering option like orig_addr and orig_port in 200 million HTTP events.
It's required to improve the search performance.

Idea

Make new query for faster search.

Current filtering options are:

• Time range
• Source
• Protocol
• Original/response address range
• Original/response port range

Try to make Index table with key like following:

• Type 1: SOURCE + ORIG_ADDR + ORIG_PORT + RESP_ADDR + RESP_PORT + TIME
  • TIME: YYYYMMDDHH
  • ORIG_ADDR, RESP_ADDR: u32(IPv4) or u128(IPv6)
  • ORIG_PORT, RESP_PORT: u16
• Type 2: SOURCE + IP_ADDR + PORT + TIME
  • IP_ADDR: ORIG_ADDR or RESP_ADDR
  • PORT: ORIG_PORT or RESP_PORT

This index table have key values of other Protocol database.

Benefit

• Search performance can be improved.

Defeat

• Current pagination could be impossible for this query.
• The burden increases when saving.