Separating CLI from elasticsearch-file-importer.py

[peter279k]

As title, to make issue #2 working friendly, I suggest we can consider let the CLI be separated from elasticsearch-file-importer.py.

That is, the following Python codes can be on new Python file name called es-importer.py:

if __name__ == '__main__':

    PARSER = argparse.ArgumentParser(
        description='Read a data from a variety of file formats and post the data to Elasticsearch'
        )
    
    SUBPARSERS = PARSER.add_subparsers(title="data_type", description="Supported data format", help="Choose one of the supported data formats.")
    CSV_PARSER = SUBPARSERS.add_parser("CSV", help="Import a CSV file into Elasticsearch")
    CSV_PARSER.add_argument('csvFile', help='Path to the CSV file to read')
    CSV_PARSER.add_argument('esIndex', help='Name of the Elasticsearch index mapping')
    CSV_PARSER.add_argument('--stopWordsFile', help='Path to a file of stopwords')
    CSV_PARSER.set_defaults(func=process_report)

    LOG_PARSER = SUBPARSERS.add_parser("Logs", help="Import a log into ElasticSearch")
    LOG_PARSER.add_argument("logFile", help="Path to the log file to read")
    LOG_PARSER.add_argument("formatFile", help="Path to file containing log format regex string.")
    LOG_PARSER.add_argument("esIndex", help="Name of the Elasticsearch index mapping")
    LOG_PARSER.set_defaults(func=process_log)

    JSON_PARSER = SUBPARSERS.add_parser("JSON", help="Import a JSON file into Elasticsearch")
    JSON_PARSER.add_argument("jsonFile", help="Path to JSON file to read")
    JSON_PARSER.add_argument("esIndex", help="Name of the Elasticsearch index mapping")
    JSON_PARSER.set_defaults(func=process_json)

    ARGS = PARSER.parse_args()
    ARGS.func(ARGS)

[jadonn]

That sounds like a great idea! Would it be good to split the other functions into their own files? That might be unnecessary, but it might also make the code easier to organize.

@peter279k If you want to submit a pull request with any contributions, I would be happy to review it and get it merged in. I'm going to try to work on the issues you have listed, but any help would be great, too!

[jadonn]

I finally got a chance to finish splitting off the CLI portion into its own script. I merged in the changes in #10. It seems like the script still works ok so far. I am going to work on splitting off the rest of the functions into their own scripts for each type of file - CSV, log, and JSON, and start working on writing automated tests. I am going to close this issue. If there are any more improvements or changes that need to be made for the CLI, we can reopen this issue or file a new one.