Skip to content

Latest commit

 

History

History
279 lines (279 loc) · 19.5 KB

release-notes.md

File metadata and controls

279 lines (279 loc) · 19.5 KB
Raw

Garmadon release notes

v1.4.0 - 2019-07-03

  • Add tracing of Hdfs class based on AbstractFileSystem (#143) e79221e (Nicolas Fraison)
  • Bump dependencies (#147) 5a126e3 (Nicolas Fraison)
  • Convert Double.NEGATIVE_INFINITY value on map to Double max negative value (#146) 5e77fd0 (Nicolas Fraison)
  • Revert "Discard messages from Agent when protocol version is incorrect instead of closing the connection (#20)" (#148) 019cd99 (Nicolas Fraison)
  • Update Flink dashboard 0156249 (n.fraison)
  • NullPointer exception can still happen time to time on APP_EVENT creation 52299a3 (n.fraison)
  • Report top 10 users on compute dashboard based on top consumption 3a50896 (n.fraison)
  • Resolve null pointer exception leading to generate partial APPLICATION_EVENT from RM 8672464 (n.fraison)
  • Add a leading dot to checkpoint files to exclude them from the Parquet table 2dbf9bc (Xavier NOELLE)
  • Add hadoop 3 to docker test platform (#137) 835fbfc (Nicolas Fraison)
  • HDFS reader indicates when it's done with a given day 0ef11c2 (Xavier NOELLE)
  • [HDFS EXPORTER] Commit pending files when revoking a partition, instead of dropping them 6150ccc (Xavier NOELLE)
  • Framework enum renamed to align with ones reported by RM (#139) d7b78dc (Nicolas Fraison)
  • Set header id on all events (#136) b790682 (Nicolas Fraison)
  • Avoid huge logging in FileHeuristics 3b72a3b (Mathieu Chataigner)
  • Enrich events related to application running on the cluster on ES reader (#132) 9990b36 (Nicolas Fraison)
  • Support Flink framework on Garmadon (#117) e953189 (Nicolas Fraison)
  • Add index doc param making checkstyle test failing (#133) e1d1d74 (Nicolas Fraison)
  • Bump dependencies (#135) 7bc04fa (Nicolas Fraison)
  • Change compression codec of parquet file from SNAPPY to GZIP 493dbe4 (n.fraison)
  • Correct disk write query for server dashboard 41b8b88 (n.fraison)
  • Improve hdfs reader logging aa80396 (n.fraison)
  • Ensure final folder is ccreated before to resolve path 83d08f7 (n.fraison)
  • Failed if not able to close consumer after 5 times 6c77129 (n.fraison)
  • Update bump_release.sh script to remove release branch and tag on the 'update release note' commit 5e5d357 (William Montaz)
  • Add FsTracer on RM and NM + some dashboard improvment (#127) c765065 (Nicolas Fraison)
  • Add support of prestosql fork to PrestoModule bfd66ec (Raphael CLAUDE)
  • Add more JVM monitoring (#126) 295943a (William Montaz)
  • Second MR to implement merge of parquet file on hdfs reader (#120) bd27d82 (Nicolas Fraison)
  • Apply fixes to spark dashboards: (#125) f067e8c (William Montaz)
  • Add fs event status (#122) 2414bc2 (William Montaz)
  • Apply miscellaneaous checkstyle cc97287 (n.fraison)
  • Fix vulnerability CVE-2019-9658 (#123) d870d21 (William Montaz)
  • First MR to implement merge of parquet file on hdfs reader (#119) 4cdbafc (Nicolas Fraison)
  • Remove temporary hdfs folder at reader shutdown (#118) 9bde400 (Nicolas Fraison)
  • [HDFS EXPORTER] Output the exception when failing to close a consumer d08fec6 (Xavier NOELLE)
  • Bump dependencies (#113) 418ca26 (Nicolas Fraison)
  • Add docs about supported frameworks and how it get metrics from it (#109) 5a0dbad (Nicolas Fraison)
  • fix javadoc error ac48350 (Mathieu Chataigner)
  • Ensure reported metric for memory/vcore usage is greater than 0 (#112) b1b646b (Nicolas Fraison)
  • [HDFS exporter] Only poll last 2 days of logs when computing offsets 548aff2 (Xavier NOELLE)
  • Add back GC_EVENTS to ES and display GC Evengts to yarn and standalone dashboards 53e40b9 (n.fraison)
  • Support kafka cluster migration in reader-hdfs (#110) 0d25561 (Mathieu Chataigner)
  • Bump kafka version to 2.0.1 9d3aa7e (Mathieu Chataigner)
  • Standardize readers configuration (#107) f46da12 (Nicolas Fraison)
  • Flink (#95) 75156c7 (Vivian OLIVERES)
  • add consul connection support for garmadon agent 8ce547a (Mathieu Chataigner)
  • grafana garmadon-yarn-application: datasource var fix d685278 (kockockockoc)
  • normalize uri, src_path and dst_path in reader-common 02235f2 (Mathieu Chataigner)
  • Add tags mechanism on agent (#98) da75ed2 (Nicolas Fraison)
  • Retreive real hdfs user performing hdfs call (#84) 2f8b00c (Nicolas Fraison)
  • Add log in case of exception during startup 83ff226 (Mathieu Chataigner)
  • Update elasticsearch template documentation (#101) 6ebd551 (Nicolas Fraison)
  • Add datasource name in link from spark dashboard to yarn app and other spark dashboards 16ffa93 (n.fraison)
  • Update ConMarkSweep legend in standalone GC dashboard 788a9d9 (n.fraison)
  • Update typo and improve filter in hdfs dashboard 1ae8753 (n.fraison)
  • Add vcores metrics on compute dahboards and remove useless framework component variables a009501 (n.fraison)
  • Provide config to set listening host on forwarder 84ab155 (n.fraison)
  • [HDFS exporter] Kill the app when failing to close output file 37f3bbc (Xavier NOELLE)
  • Add mchataigner as garmadon developper 74cb8a5 (Mathieu Chataigner)
  • [reader] serialize timestamp with along header and body 62fff12 (Mathieu Chataigner)
  • change es template to index method_duration_millis 182fb8b (Dhia MOAKHAR)
  • [hdfs-exporter] Use message timestamp instead of current time 0264a53 (Mathieu Chataigner)
  • Fix remaining Javadoc errors 09aa6a8 (Mathieu Chataigner)
  • Fix Javadoc errors to resume auto-deployment 6923b00 (Xavier NOELLE)
  • [HDFS exporter] Batch HDFS calls to get initial offset 89f5169 (Xavier NOELLE)

v1.3.0 - 2019-01-15

  • Add hdfs reader to docker test platform a86cecf (n.fraison)
  • Remove useless dependency and maven plugin 8b64e0b (n.fraison)
  • Bump maven plugins 0cc26a7 (n.fraison)
  • Bump dependencies b6e2406 (n.fraison)
  • [HDFS exporter] Clean logging (#80) 041e456 (Xavier NOELLE)
  • Compute severity based on number of created files (#27) 7152728 (Meriam Lachkar)
  • Auto-commit for lag monitoring 81420c7 (Xavier NOELLE)
  • [HDFS exporter] Log successfully opened files 4f85045 (Xavier NOELLE)
  • [HDFS exporter] Export current running offset per event 552c525 (Xavier NOELLE)
  • Add back kafka CLIENT_ID_CONFIG ConsumerConfig properties and solve hostname static order (#78) 6b75d1e (Nicolas Fraison)
  • Add Presto module based on StandaloneModule (#65) 304bd82 (Nicolas Fraison)
  • [HDFS exporter] Add Prometheus metrics & align behavior with specs (#70) d7c0369 (Xavier NOELLE)
  • Update garmadon elasticsearch template at reader startup c764d7b (n.fraison)
  • Display hdfs latency in ms bbe83bd (n.fraison)
  • Add back normalization on FS_EVENT removed when migrating to ProtoConcatenator concatToMap function (#74) 96ade16 (Nicolas Fraison)
  • Correct network graph in server dashboard 030b353 (n.fraison)
  • Implement common coding problems checkstyle 645ddba (n.fraison)
  • Implement Modifier Checks checkstyle 5d3e962 (n.fraison)
  • Implement Size violation checkstyle 625d393 (n.fraison)
  • Implement Annotation checkstyle de46f68 (n.fraison)
  • [HDFS exporter] Allow setting Proto Parquet writer's "block size" 8dd864e (Xavier NOELLE)
  • Change forwarders ports to not use ones from ephemeral range port 1848d18 (n.fraison)
  • Add start_time field as date in es template b25f19d (n.fraison)
  • [HDFS exporter] Don't close ParquetWriter again when already closed (#64) 9fcefbb (Xavier NOELLE)
  • Ensure javadoc plugin is executed at each build and correct remaining javadoc issues fb27131 (n.fraison)
  • Fix Javadoc style errors 7f650b5 (Xavier Noëlle)

v1.2.0 - 2018-12-12

  • Fetch tags from remote (#46) 037167f (William Montaz)
  • [HDFS EXPORTER] Decrease number of output files (#61) ae2ff0a (Xavier NOELLE)
  • [HDFS exporter] Main parameters are configurable through command line 7dae6f8 (Xavier NOELLE)
  • [HDFS exporter] Ensure final file's parent directory exists 16a5b60 (Xavier NOELLE)
  • Add packets & errors information for network (#38) d1ac0f0 (jpbempel)
  • [HDFS exporter] Set Hadoop dependencies as provided 25e7ca0 (Xavier NOELLE)
  • [HDFS exporter] Fix ConcurrentModificationException 135c756 (Xavier NOELLE)
  • Consumer field must be public to be accessible by resourcemanager classes (#51) d9aad9c (Nicolas Fraison)
  • Migrate ES reader to ProtoConcatenator concatToMap function (#52) 1532c80 (Nicolas Fraison)
  • [HDFS reader] Build a shaded JAR (#49) 64fd3cc (Xavier NOELLE)
  • Add APPLICATION_EVENT to hdfs reader b384fe8 (n.fraison)
  • Get canonical hostname to ensure having same hostname format c6d3a8c (n.fraison)
  • Improve ES reader (#44) 440f449 (Nicolas Fraison)
  • Implement whitespace checkstyle 8e00ce4 (n.fraison)
  • Implement imports checkstyle 3c9e11b (n.fraison)
  • Implement Class Design checkstyle 1d147b7 (n.fraison)
  • Implementing blocks checkstyle ff966d4 (n.fraison)
  • Implement naming convention checkstyles 257b1db (n.fraison)
  • Adding checkstyle to build phase b5e122c (n.fraison)
  • Add HDFS exporter 385a068 (Xavier NOELLE)

v1.1.0 - 2018-11-22

  • Correct bump_release.sh for OSX (#43) 5ceccd1 (William Montaz)
  • GarmadonReader#Builder accepts a KafkaConsumer for its #stream method 6b21bdf (Xavier NOELLE)
  • Add ProtoConcatenator class for Protobuf & JSON ca1c978 (Xavier NOELLE)
  • Create script to easily bump release 14784f5 (n.fraison)
  • Update readme (#37) 2fe4f5c (William Montaz)
  • Allow running Docker launch script from any directory c2b0d72 (Xavier Noëlle)
  • Correct predefBlacklist list to avoid trying to instrument garmadon classes a4ed4ec (n.fraison)
  • Rename jvm statisics package to align with default package naming on other module cdc5071 (n.fraison)
  • Merge jvm-statistics sub-modules to garmadon-jvm-statistics module d41e04e (n.fraison)
  • Move jvm-statistics proto schema to garmadon-schema module c60eb45 (n.fraison)
  • Correct flaky jvm statistics test f967b53 (n.fraison)
  • Remove setConnectionRequestTimeout on ES reader as it lead to false timeout 9cc5335 (n.fraison)
  • JVMStatistics should scheduleAtFixedInterval instead of scheduleAtFixedRate (#29) ed63e6f (William Montaz)
  • Compute the size of any message sent to kafka (#31) a1cb957 (William Montaz)
  • [ES template] Add keyword field for paths e4b57b6 (Dhia MOAKHAR)
  • Correct flaky prometheusHttp test (#22) 12c0ae9 (Nicolas Fraison)
  • Discard messages from Agent when protocol version is incorrect instead of closing the connection (#20) 63f891f (William Montaz)
  • Get AppName from RM b364218 (n.fraison)
  • Relocate agent tracers in different packages a6733f2 (n.fraison)
  • Ensure we close the socket if there is an issue during handshake 089b138 (n.fraison)
  • Add prometheus monitoring client on readers 94a716b (n.fraison)
  • Add method latency on HDFS traces mlethod c28046f (n.fraison)
  • Merge ContainerMetricsTracer and ContainerResourceMonitoringTracer 1c70fb2 (n.fraison)
  • Add tracer on listStatus, getContentSummary and addBlock hdfs call c77bd22 (n.fraison)

v1.0.0 - 2018-11-09

  • Update garmadon release name to not contain dot 7de1a4a (n.fraison)
  • Protect reader from bad event or non garmadon event 327df10 (n.fraison)
  • Change log level to debug for garmadon message not deserializable d0469fb (n.fraison)
  • Update build so MANIFEST.MF contains entry Implementation-Version 7643e51 (n.fraison)
  • Update reader to not crash if header is malformed 1449058 (n.fraison)
  • [BREAKING] - Provide timestamp in Garmadon's ProtocolMessage 07d5bdc (n.fraison)
  • [Breaking] - Bump protobuf from 2.5.0 to 3.6.1 d310ff1 (n.fraison)
  • Deduplicate executor_id field between Spark events and header (#8) 5907296 (Xavier NOELLE)
  • Bump spark dependency to spark 2.3.2 5b11bcb (n.fraison)
  • Update ES schema to align fields type to their java type e74b1a8 (n.fraison)
  • Bump garmadon dependency d06cc24 (n.fraison)
  • On standalone JVM dashboard filter on username for main class and pid f759142 (n.fraison)
  • Update Server Overview dashboard removing useless tags 74087e8 (n.fraison)
  • Push to maven central 9a78328 (n.fraison)
  • Remove jmoab dependency fad08c4 (n.fraison)
  • Add GPG keys and SONATYPE_PASSWORD secrets 3f5d2b4 (Maxime Brugidou)
  • Add main class to the header c917cb1 (n.fraison)
  • [BREAKING] - Remove usage of StateEvent in Spark state events 3c5dd7f (n.fraison)
  • Add Travis CI configuration af7a0d5 (n.fraison)
  • Add Apache License 4c91260 (n.fraison)
  • Use -d when running docker-compose up (instead of the long option) 1efa960 (Xavier Noëlle)
  • Ensure garmadon forwarder doesn't hang when prometheus http server can't start 30662f4 (n.fraison)
  • Use datasource variable on grafana dashboards a654ef9 (n.fraison)
  • Update log level of logs in agent to avoid polluting logs with garmadon ones 8c33f8b (n.fraison)
  • Change log level of jvm-statistics logs and retry indefinitely to register stats 46dc3a6 (n.fraison)
  • Remove grafana dashboards and es template from resources b0180a4 (n.fraison)
  • Remove CRITEO env usage on es reader 41f0d02 (n.fraison)
  • Remove Criteo's references 440aca4 (n.fraison)
  • Update garmadon dashboards a340e09 (n.fraison)
  • Relocate jar dependencies and slf4j one to be able to run on standalone not hadoop JVM 72da500 (n.fraison)
  • Bump Dependency Release b577554 (n.fraison)
  • Update all dashboards 152a52d (n.fraison)
  • Remove useless value field and rename vaue_float to value 620cfc3 (n.fraison)
  • Add support of Yarn client mode for spark metrics 2773550 (n.fraison)
  • Add container Vcore usage and limit from nodemanager 74ce86d (n.fraison)
  • Move kafka from application id to generic id 755ec8f (n.fraison)
  • [Risky] - Add management of Standalone JVM e3966b5 (n.fraison)
  • Avoid NPE issue on the Garmadon Spark Listener bfa12b9 (n.fraison)
  • Add metaspace used, compile and safepoints information on yarn application dashboard 1ff12b0 (n.fraison)
  • [Risky] - Remove tag field on header b794212 (n.fraison)
  • Correct typo in server.properties file f8db417 (n.fraison)
  • Moving tag to a list of tags 98728c4 (n.fraison)
  • Migrate to hourly index 1417425 (n.fraison)
  • Add Spark executor_id in header 8767974 (n.fraison)
  • Revert "Add application name from RM" 51c68eb (n.fraison)
  • Update grafana dashboards b510a6e (n.fraison)
  • Update yarn application grafana dashboard to display avg, max and limit memory on containers b9e3953 (n.fraison)
  • Sort stage_id, stage_attempt_id and executor_id as Number in grafana ab0ffbe (n.fraison)
  • Integrate jvm-statistics module to garmadon f8a274b (n.fraison)
  • Add application name from RM a73e460 (n.fraison)
  • Add Spark tasks events ce23d1e (n.fraison)
  • Remove CUTTLE_APP_MASTER detection 4160f9b (n.fraison)
  • Move readers in a common readers sub-module 3de72fa (n.fraison)
  • Add Spark Executor Events b7edf09 (n.fraison)
  • Add Spark Stage State events 959c093 (n.fraison)
  • Move Framework and Component to Schema project fe0f373 (n.fraison)
  • Use proto classes instead of intermediate events classes a4352df (n.fraison)
  • Treat only Spark and MapReduce events on dr-elephant heuristics 3308750 (n.fraison)
  • Implement spark listener 3d5e9c5 (n.fraison)
  • Correct typo on pid field and remove search filter on username on yarn dashboard c2a6505 (n.fraison)
  • Remove useless tests folder 7e01cae (n.fraison)
  • Implement a way using docker-compose to test locally the whole platform 7962e2a (n.fraison)
  • Add grafana dashboards in ES reader submodule d7655ff (n.fraison)
  • Create specific template file for elasticsearch index template c472a9d (n.fraison)
  • Bump release c88aa29 (n.fraison)
  • Activate eager_global_ordinals to speed up terms aggregation 7d0b7d7 (n.fraison)
  • Bump netty and byte buddy 1c686a9 (n.fraison)
  • Bump to ES 6.3.1 to have better Exception management 6709b80 (n.fraison)
  • Update process_written and read field to long 6f7b4b6 (n.fraison)
  • Update process_read field to double type instead of integer b6a1eda (n.fraison)
  • Increase number of shards from 20 to 30 fc5d550 (n.fraison)
  • Expose kafka producer metrics on Prometheus endpoint of Forwarder 9a83074 (n.fraison)
  • [forwarder] Add 75th,90th,99th quantiles to event size summary cb15b54 (William Montaz)
  • Update elasticsearch reader to use event type name c284156 (n.fraison)
  • Add help support for Heuristics 7632977 (Jean-Philippe Bempel)
  • Add event size reporting via prometheus from forwarder f6a5f5f (William Montaz)
  • Correct garmadon ES index template 4027339 (n.fraison)
  • Increase HostStatistics interval from 10 to 30s af07281 (n.fraison)
  • Do not load GC events in ES e66a0cb (n.fraison)
  • Do not load MapReduceModule by default on Container 62c52d3 (n.fraison)
  • Add framework and component header c8c5879 (n.fraison)
  • Uniformize uri 1c195f0 (n.fraison)
  • Remove uri from src_path and dst_path c24c72a (n.fraison)
  • Manage disks and network as independant metrics e8e3073 (n.fraison)
  • Add append FsEvent 9ae4386 (William Montaz)
  • Add two filters to DSL 4acba87 (William Montaz)
  • Use global app completion detection mechanism on FileHeuristic 0ac3032 (William Montaz)
  • Ensure search on path use keywork analyzer d936c3c (n.fraison)
  • Add attemptId on heuristics 0aa4cd4 (n.fraison)
  • Update uri field to be a keyword d1a940e (n.fraison)
  • Migrate from pipeline feature to direct index access 8c0b656 (n.fraison)
  • Add pid on header from containers d0b64b1 (n.fraison)
  • Add type for all event send to ES e0cf996 (n.fraison)
  • Align way we set Elasticsearch Reader config with the HeuristicsReader b96746b (n.fraison)
  • Bump ES to current ES cluster release 6.3.0 e54ea6a (n.fraison)
  • Remove redundant callback 5aed310 (William Montaz)
  • Add kafka group and autocommit on Heuristics reader e2e3c32 (n.fraison)
  • Fix bug preventing setting a specific groupIf for a reader 9045bc9 (William Montaz)
  • Reduce ES reader verbosity bc8348f (n.fraison)
  • Add default logging config + log4j2 to slf4j on elasticsearch reader fbfce67 (n.fraison)
  • Aggregate Heuristic results per application instead of container f8e71d2 (Jean-Philippe Bempel)
  • Update ealsticsearch reader and index value type 502d1bf (n.fraison)
  • Create elasticsearch sub module 2349bfd (n.fraison)
  • Remove useless Metric protobuf schema 1993de0 (n.fraison)
  • Few refacto on logging usage f090fb1 (n.fraison)
  • Remove useless main method in agent ae6dd60 (n.fraison)
  • Add agent release in debug logs of the agent 110697b (n.fraison)
  • Link FileHeuristic to reader 867708b (William Montaz)
  • FileHeuristic manages END state for containers 1f1783f (William Montaz)
  • Init FileHeuristic with simple counters 449dda3 (William Montaz)
  • Add ready column to CREATE_YARN_APP_HEURISTIC_RESULT_SQL d2cd83a (William Montaz)
  • Possibility to define kafka properties in reader builder a4be2d1 (William Montaz)
  • Fix instrumentation problems on spark hive 6e0ab9d (William Montaz)
  • Preparation commit for GarmadonHeuristics 8cda554 (William Montaz)
  • Combine whitelist and blacklist mechanism a01d83f (William Montaz)
  • MapReduceModule use isSubTypeOf without requiring to load the class 3d3720f (William Montaz)
  • Add a package exclusion mechanism for instrumentation e97e959 (William Montaz)
  • Add specific teragen with flamegraph sampling mechanism be263ca (n.fraison)
  • Rename prometheus instance field to hostname and ensure it is set to FQDN df1441f (n.fraison)
  • Use hadoop-parent pom 4e072fa (n.fraison)
  • Add forJoin metrics to prometheus forwarder ee532ac (n.fraison)
  • Improve Agent startup removing REDEFINITION of already loaded classes and Waiting max 3sec 4f0f072 (n.fraison)
  • Avoid loading classes when defining intrumentation matching for MapReduceModule 63204c5 (William Montaz)
  • Remove useless metrics dependency 21a41ff (n.fraison)
  • Bump dependency 01de9c4 (n.fraison)
  • Remove LoggingHandler from netty f984c74 (n.fraison)
  • Add ES reader 5507898 (n.fraison)
  • Get Container memory usage from NodeManagers e09042d (n.fraison)
  • Migrate to slf4j logging a53df9e (n.fraison)
  • Start modules asynchronously on Container 109f635 (n.fraison)
  • Expose metrics to prometheus HTTP server 906623a (n.fraison)
  • Lowercase garmadon topic to be align with topic created by rivers 67c3dd0 (n.fraison)
  • Remove kafka version to rely on java parent one ac41750 (n.fraison)
  • Initial commit de108c4 (n.fraison)
  • Adding OWNERS file 9c535c6 (qabot)
  • Apply java template 8dd59dc (qabot)