Migeng 371 avoid messages duplication issues #70
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* WINDUP-2398 changed mimetype and created sample available json * changed kafka topic * changed kafka topic and category * refactored code * replaced seda by direct as it seemed not to work * Converted calculator into Bean Added Sonar + Jacoco execution * Created first test for a simple route in order to know how to properly test Camel * Created unit test for AnalysticsCalculator Added JUnit 5 and AssertJ to pom Aplied some clean-code suggestions * Removed sonar token from pom * WINDUP-2398 Added support for JUnit 5 on old maven versions Added constructors on model Fixed AnalyticsCalculatorTest test Added Sonar info to README * WINDUP-2398 Added lombok file to add Generated annotation that will be avoided by JaCoCo Added sample file to test multipart Added test for the multipart Removed accessors/modifiers for cloudforms classes Cleaned CustomizedMultipartDataFormat * WINDUP-2398 Renamed test methods * WINDUP-2398 Renamed test methods * WINDUP-2398 Moved back to JUnit 4 as Camel Test support doesnt support JUnit 5 yet * WINDUP-2398 Added very simple test for direct:store * WINDUP-2398 Added test for direct-upload with individual files and zip file * WINDUP-2398 Added test for direct:insights route * WINDUP-2398 minor change on xmlroutestest * WINDUP-2398 Applied changes from travis PR * WNDUP-2398 added test for kafka listener route * WNDUP-2398 added a test for the direct-download route ( not finished ) * WNDUP-2398 fixed test for direct:download route * WNDUP-2398 added test for direct:calculate added cloudforms export file * WNDUP-2398 testing the rest endpoint * WNDUP-2398 testing the rest-upload endpoint * WNDUP-2398 testing the rest report endpoint * WNDUP-2398 switched to regular endpoints for rest, as rest DSL has some "issues" * WNDUP-2398 renamed kie helper methods with no Random, and removed unused code * WNDUP-2404 testing call-kie-extract-reports route removed testing route * WNDUP-2398 testing the direct:decisionserver endpoint * WNDUP-2404 changed kie endpoint from http4 to http as a ClientProtocolException appears * WINDUP-2404 adjusted the tests to consider the new context-path * windup-2398 considering non file params as headers for all messages * windup-2398 testing the param from multi part form * WINDUP-2398 removed customerid from the url on the rest endpoint * WINDUP-2398 removed customerid from the test of rest endpoint * WINDUP-2398 modified tests to work with dynamic metadata properties and filtering for messages having all those properties * MIGENG-23 added new version of Cloudforms file, refactored Calculator * MIGENG-23 replaced the object model to be sent to JMS with the one in xavier-analytics * MIGENG-23 adapted tests to new report model * MIGENG-23 changed cloudforms model types Integer to Long * MIGENG-23 changed the way we obtain RequestId as before it was always the same * MIGENG-23 fixed tests according to the new model * MIGENG-23 replaced cloudforms test files * MIGENG-23 made host.type filter configurable with a property * MIGENG-23 added tar and gz support to the route * MIGENG-23 added TAR.GZ support and added test for that * MIGENG-23 added exception handling when the json crashes on the unmarshal * MIGENG-23 fixed mocking log endpoint in test * MIGENG-23 fixed attributes and filter names * MIGENG-23 fixed tests with the new "MA_metadata" header now we use the RHIdentity object passed as x-rh-identity header if it exists and we populate our params * MIGENG-106 Modified the routes in order to do the unzip after receiving the message from kafka and not in the upload * MIGENG-106 Fixed tests in order to work with the unzipping after the kafka message * MIGENG-106 Removed all structure for CloudForms Modified test and added paths to properties * MIGENG-106 Small refactors * MIGENG-106 Renamed test * end2end added test to allow an End To End test using TestContainers and real layers using WireMock to simulate Rest InsightsUpload service * end2end working on issue with Ryuk and test containers * MIGENG-106 code prepared to use the version on the manifest * MIGENG-106 added test to check versioning fallback * MIGENG-106 fixed tests according the new payload taxonomy * MIGENG-106 added tar.gz file for only 1 json inside * MIGENG-106 modified getRHIdentity to modify only the identity.internal element * MIGENG-106 fixes to issues during deployment and test * MIGENG-106 fixed the calculation method for the number of hypervisors * MIGENG-106 fixing and adding tests to changes on getRHIdentity and extractMAmetadata * MIGENG-106 replaced artemis with activemq Improvements in the E2E test with the containers * end2end added hoverfly to simulate insights upload service, firsts attempts * end2end typo fix * end2end added logging for testing, and changed to mock-server to mock rest API * end2end Fixed the test with correct multipart sending, correct jpa retrieving, and kie sample files as response * end2end Fix the End2End test with correct version of TestContainers Fixed test for Kie Service with proper expectation Fixed test for Direct Download for considering the proper endpoint having query params * end2end Removed analytics dependency * end2end Removed Debug LOG lines * end2end Removed hoverfly logging * MIGENG-150 Creation of initial classes and routes for VMWorkloadInventory * end2end renamed the EndToEnd to a regular Test * end2end fixing the filtering to camel headers , and switching back to previous false state * end2end making the identity of the ..with-targz.json file the full one * end2end making the identity of the platform.upload.xavier.json.gz and .json file the full one * end2end fixing base64 identity to have the same fix the type of the file loaded to be consistent with the one in the base64 identity * end2end fixed test dependencies * end2end removed Spring Boot ActiveMQ dependency * MIGENG-150 Added Test for the VMWorkloadInventoryCalculator Changed Ids for second host in sample file (.json) Added multicast route to send message to both calculators Renamed cost savings route Added paths to obtain workload inventory properties * MIGENG-150 Fixed test * MIGENG-150 removed parallel processing * MIGENG-150 added test for direct:calculate-vmworkloadinventory * MIGENG-150 adapted route using decisionServer to use the right methods * MIGENG-150 set right values for kie command and session for WKI * MIGENG-150 fixed value for test * MIGENG-150 fixed multicast and split issue as seen on CI deployment * MIGENG-150 changed the method call from setWorkload ... to AddWorkLoad * MIGENG-150 changed the test for calculate vmworkload to consider individual elements return instead of a collection * MIGENG-150 added Serializable to the WorkloadInventory Model object * MIGENG-150 moved class VMWorkloadinventory to another package * MIGENG-150 fixed depending classes after moved class VMWorkloadinventory to another package * end2end added info about Ryuk and Testcontainers property * MIGENG-151 adding methods to get the collection fields * MIGENG-151 added methods to set values for the collection fields added info to the test payload * MIGENG-150 working on the Files map retrieval * MIGENG-150 fixed the path for the files map fixed the lambda to get the files map value pairs * MIGENG-151 working on the Files map retrieval * MIGENG-151 fixed the path for the files map fixed the lambda to get the files map value pairs * MIGENG-151 changed expected values considering also the collection fields for the workloadinventory * MIGENG-155 saving and persisting the entity * MIGENG-155 adding methods on the analysisService to get and update the analysismodel * MIGENG-155 modified WLI route to use the new method in analysisService modified test on DirectStore * MIGENG-155 renamed resources for inticial cost savings report * end2end Added S3 client bean to customise its endpoint url Added PostgreSQL testcontainers container Added Localstack testcontainers container Fixed analysisid in json file as response from kafka * end2end Added reports files for demolab payload added tests for the 3 reports added test for S3 added performance test * end2end Added demolab payloads files added report file for demolab * end2end Added Drools and KIE containers Added Ingress image build and container creation * end2end reordered code * end2end Modified KAFKA estrategy to earliest to take all pending messages Added properties for kieserver to connect to local container Added properties for KIE container to connect to DROOLS container Added docker-compose containers for Ingress Removed mocks Added commands to import projecto into drools * end2end added more env vars added bash script file with docker and curl commands to simulate the infrastructure deployment * end2end add env vars * end2end adjusted testcontainers env vars added sleeps in the bash script * end2end using shell commands to execute docker images for drools and kie * end2end added travis stuff for docker changed the way we get test files from FS * end2end fixed ocd analytics-template version * end2end fixed ScanRunModel accessors of Type * end2end fixed date value for summary report scan run model test file * end2end fixed Spring Boot error for Port 8080 already in use modified folder on ignored test for S3 added live replacement on ingress docker-compose to make it work increased performance test timeout to 1 minute * end2end addded changes to run Testcontainers in Travis * end2end added again H2 as test database configured the env properties for End2End Test with postgresql * end2end changes to travis to fix the issue with Testcontainers * end2end setting log level to debug on tests to check on travis * end2end log level back to info * end2end increasing time to wait until initialcostsavings report is generated * end2end increased time for CSR to 3 minutes, and used awaitillity to continue earlier if its finished * end2end added log level to hibernate to check connections to DB * end2end added commands to upgrade docker version at Travis removed testcontainers kafka dependency as it's not used * end2end making kafka to fail in the connection * end2end added print out of docker logs added explicit naming of kafka container * end2end using master branch of ingress at 20191115 * end2end renamed folder of downloaded insights as it causes issues with docker-compose * end2end added docker logs in travis * end2end added logger output to critical containers * end2end added time before the test starts * end2end added logger to activemq switched back to old version of ingress * end2end added more logs to containers and increased times * end2end added advertising for kafka * end2end enabled again ryuk * end2end removed -DtestLogRootLevel=warn -Pcoverage from the test script * end2end adding log and more time to start the test * end2end added zookeeper logs * end2end moved docker compose containers to Testcontainers added temporary hacks in the download-file route to test * end2end enabled the download of ingress-go project from github * end2end printing ScanRunModels and using sameDay as comparison between dates in ScanRunModel * end2end not considering scanrunmodel.date in the test * end2end removing tracing because Travis kills the job if we exceed * end2end removed log of docker containers in travis yaml * end2end fixed unit tests set all tests to be run in travis refactored comparisson of sets * end2end reduced log level to reduce logs in Travis * end2end changed hardcoded values set the logging level in the travis pipeline removed unused files * end2end re-added Profile Coverage to mvn execution * end2end some PR clean tasks - fixed Testcontainers name in Readme - removed unused files - refactored ...DirectCalculateVMWorkloadInventoryModelTest assert on scanRunModel - fixed typo on test properties - removed comments on ScanRunModel * end2end added minio.start * end2end replaced timeouts with something way shorter (in milliseconds) * end2end added Github action to build project and red hat repositories for dependencies * Update maven.yml * MIGENG-365 removed github pipeline * MIGENG-365 cleanup downloaded files * MIGENG-365 added timouts to the test properties and increased the value fixed not available minio.host prop on runtime * MIGENG-365 as we are not specifying profile, moved the regular SpringConfiguration to any profile except test * MIGENG-365 added delay between Minio and MINIOMC containers * MIGENG-365 removed test code from the production route, and moved to the test * MIGENG-365 cleaned the route on the download-file side, and moved parts to the test * change `type` by `smartStateEnabled` * restore `type` by `smartStateEnabled` since it will modify the database * removed unused import * fix test * MIGENG-365 [PR] removed unused jmsTemplate * MIGENG-365 [PR] moved Docker only test files to subfolder * MIGENG-365 [PR] renamed minio shell script * end2end removed minio-mc commands setting policies for the buckets as they are wrong and apparently not needed Co-authored-by: Carlos E. Feria Vila <[email protected]>
added test after stoping camel context commented out the idempotentConsumer added debug info on localstack
added test after stoping camel context commented out the idempotentConsumer added debug info on localstack
…ent no need to use it
…ent no need to use it
added 8 commits
January 22, 2020 15:46
…between local and Travis
…between local and Travis
refactored duplication tests
refactored duplication tests
Codecov Report
@@ Coverage Diff @@
## master #70 +/- ##
============================================
+ Coverage 78.99% 79.36% +0.37%
- Complexity 733 744 +11
============================================
Files 71 72 +1
Lines 2038 2094 +56
Branches 93 94 +1
============================================
+ Hits 1610 1662 +52
- Misses 371 374 +3
- Partials 57 58 +1
Continue to review full report at Codecov.
|
jonathanvila
force-pushed
the
MIGENG-371-avoid-messages-duplication-issues
branch
from
01c8eb4
to
fad6692
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.redhat.com/browse/MIGENG-371
This code basically consists on tests as the code change is very small ( setting autoOffsetReset=latest from the kafka consumer in camel ). So, the work is on trying to reproduce the same situation on the tests reconnecting the client to the Kafka broker and see how many messages are re-read.