Developed a code/file generator in Python that reduced the time needed to ingest new source systems into the data lake from 4-6 weeks to just 1 week. Migrated over 500 tables from an outdated system using the code/file generator, resulting in cost savings of approximately 600K €. Implemented data ingestion on Google Cloud Platform. Migrated large amounts of data from IBM Bigsql to the Hadoop Data Platform. Created data flows using Apache NiFi and Apache Kafka. Built data pipelines using Apache Spark, Scala, and Apache Kafka. Developed metrics and Grafana Monitoring Dashboards for enhanced data monitoring. Conducted thorough testing of end results to ensure accuracy. Implemented test automation for efficient and streamlined testing processes. Established continuous integration and continuous delivery using TeamCity and XLDeploy.