Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding autocompletion + scripts + standard deviation in 'mapper time' heuristic #275

Open
wants to merge 288 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
288 commits
Select commit Hold shift + click to select a range
65ae460
HADOOP-8096: Property 'username' is converted to lowercase before sup…
Dec 13, 2014
bced339
[HADOOP-7814] Prepare to add metrics library to dr-elephant
mcvsubbu Dec 10, 2014
467794b
[HADOOP-7814] Publish counters via dr-elephant
mcvsubbu Nov 17, 2014
6fc595c
[HADOOP-8640] Fix hang in dr-elephant due to h1 and h2 incompatibility
mcvsubbu Jan 16, 2015
be47552
[HADOOP-8648]Updating dr-elephant release ID to 0.6.4
mcvsubbu Jan 15, 2015
b6ddadd
[HADOOP-8648] Reset version to 0.6.5-SNAPSHOT
mcvsubbu Jan 21, 2015
194dee4
HADOOP-8294: Improved slow performance of join queries
Jan 22, 2015
45e3ca2
HADOOP-5369: Paginate search results of Dr. Elephant
Feb 2, 2015
f7562cc
HADOOP-7948: Make Dr. Elephant's Reducer Time Moderate Severity cut-o…
Feb 3, 2015
f6a2a95
HADOOP-8320: Add detailed suggestions in Dr. Elephant help page
Feb 3, 2015
8272cf0
HADOOP-8320: Add detailed suggestions in Dr. Elephant help page
Feb 3, 2015
138693a
HADOOP-8856: Fix a broken Dr.Elephant test case for reducer time heur…
Feb 3, 2015
81a27f7
HADOOP-8859: Add ideal task time suggestion in Dr. Elephant 's reduce…
Feb 4, 2015
6beb02d
HADOOP-8846: Release Dr. Elephant v0.6.5
Feb 4, 2015
12408ba
HADOOP-8929: Update Dr.Elephant version number to 0.6.6-SNAPSHOT
Feb 9, 2015
28d1283
HADOOP-9685: Dr. Elephant fails to fetch task counter data on hadoop2
Mar 27, 2015
335198a
HADOOP-8941: Further improve search performance for Dr. Elephant
Mar 16, 2015
adf1cc2
HADOOP-9714: Add Pluggable jobtype to Dr. Elephant
Mar 31, 2015
c16a050
HADOOP-9716: Dr. Elephant rest interface needs search endpoint
Apr 2, 2015
d73bc48
HADOOP-9853: Release Dr. Elephant v0.6.6
Apr 2, 2015
d163a69
HADOOP-9886: Fix for making Dr. Elephant's pagination thread safe
Apr 6, 2015
0fcaa7b
HADOOP-10289: Update Dr. Elephant version number to 0.6.7-SNAPSHOT
Apr 20, 2015
9e72289
HADOOP-10089: Dr. Elephant misses on type of Voldemort bnp job
Apr 20, 2015
872b20e
HADOOP-10301: Dr. Elephant occasionally misses a job
Apr 20, 2015
877ae89
HADOOP-9900: Dr. Elephant Mapper Input Size Help Page should suggest …
Apr 21, 2015
71a7fc5
HADOOP-10290: Release Dr. Elephant v0.6.7
Apr 21, 2015
9009da5
HADOOP-10314: Dr. Elephant should not mention deprecated properties i…
Apr 21, 2015
1bab26e
[HADOOP-10405] Fix one-off error in getting cluster name from NN host…
mcvsubbu Apr 23, 2015
478b2d5
HADOOP-11768: Update Dr. Elephant version to v1.0.0-SNAPSHOT
Jun 12, 2015
df3f7a6
OFFREL-234: Adding Spark log analysers into Dr. Elephant
timyitong May 27, 2015
5845765
Addressing naming problems, code cleaness and added more help pages
timyitong Jun 2, 2015
1e12e8c
Renaming Mapreduce -> MapReduce, Classic -> Hadoop1, Yarn -> Hadoop2
timyitong Jun 2, 2015
5e74010
Addressing major feedback from Fangshi: 1.bind application type with …
timyitong Jun 3, 2015
0b0ea85
Modify Spark's dependency configuration, some more logging and a fix …
timyitong Jun 4, 2015
5a7e1dd
A tmp commit trying to test Nertz Spades deployment
timyitong Jun 4, 2015
9cf450a
Addressing some renaming feedback, logging
timyitong Jun 11, 2015
5be797a
Code cleaness and styles improvements
timyitong Jun 12, 2015
9808e79
HADOOP-11876: Separate Dr. Elephant configs from deployable zip
Jun 17, 2015
0f36e0c
HDOOP-11456: Change Dr. Elephant's heuristic from MapperInputSize to …
Jun 17, 2015
9d55569
HADOOP-11922: Fix Dr. Elephant search by heuristic field
Jun 18, 2015
e180bea
HADOOP-11912: Release Dr. Elephant v1.0.0
Jun 18, 2015
5e61c3a
HADOOP-12013: Dr. Elephant has a corner case to analyze duplicate ent…
Jun 22, 2015
5f81420
HADOOP-12247 - Update Dr. Elephant version to v1.0.1-SNAPSHOT
Jun 29, 2015
49a80d8
HADOOP-11913: Fix Dr. Elephant broken tests
Jun 29, 2015
31eae86
HADOOP-12067: Dr. Elephant heuristic is too lenient to jobs with long…
Jun 29, 2015
8e8ef38
HADOOP-12023: Dr. Elephagent new heuristics: memory util ratio
Jul 1, 2015
e818c35
HADOOP-12593: Release Dr. Elephant v1.0.1
Jul 10, 2015
de5a0dc
HADOOP-12632: Enable Dr. Elephant 's mapper spill heuristics
Jul 13, 2015
b9998f7
HADOOP-12669: Dr. Elephant's mapper spill heuristic is too harsh
Jul 14, 2015
e2e887a
HADOOP-12674: Release Dr. Elephant v1.0.2
Jul 14, 2015
f5e4d23
HADOOP-12603: Dr. Elephant has a failed test in Mapper Speed Heuristic
Jul 14, 2015
48e5848
HADOOP-12603: Update Mapper Spill Heuristic help page
Jul 14, 2015
4f4ee66
HADOOP-12063: fix a potential devide by 0 exception in mapper spill h…
Jul 15, 2015
0178364
HADOOP-12688: Dr. Elephant needs task time info in Memory Heuristic
Jul 15, 2015
6de0b0a
HADOOP-12709: Dr. Elephant has a corner case for flagging 0 tasks in …
Jul 15, 2015
a7d6d3b
LIHADOOP-13019, LIHADOOP-13020: Upgrade to Spark 1.4, and add event l…
timyitong Jul 22, 2015
b8bf46b
LIHADOOP-13018: Dr. Elephant 's Spill heuristic should be lenient to …
Jul 28, 2015
ec46343
LIHADOOP-13094: Fix Dr. Elephant build on Jenkins after spark1.4 upgrade
Jul 29, 2015
d8ab3c8
LIHADOOP-13016: Release Dr. Elephant v1.0.3
Jul 29, 2015
4d839ca
Adjust StageRuntime heuristics so that it's the real duration time in…
timyitong Aug 20, 2015
8320332
LIHADOOP-13747 Update Spark driver name to be consistent with Spark 1.4.
Victsm Aug 20, 2015
14d06fc
LIHADOOP-13953 The start script should take arguments to enable evolu…
Aug 31, 2015
a0850b0
LIHADOOP-5177 Update documentation of Dr. Elephant
Aug 27, 2015
1f6837a
LIHADOOP-12802 Occasional Concurrent Modification Exception during Dr…
Sep 9, 2015
475cdef
LIHADOOP-8114 [Opensource Dr. E] Apache License and LinkedIn Copyrigh…
akshayrai Aug 24, 2015
fd8e408
Minor formatting change in Readme License
Sep 11, 2015
01d057b
LIHADOOP-14305: Dr. Elephant goes OOM while displaying other executio…
Sep 14, 2015
8429542
LIHADOOP-14319: Dr. Elephant's should append not overwrite log during…
Sep 14, 2015
1ba1857
LIHADOOP-14321: Release Dr. Elephant v1.0.4
Sep 14, 2015
c9bb9a7
LIHADOOP-14322 Dr. Elephant feature request. The ability to compare r…
Sep 28, 2015
1ca9c89
LIHADOOP-14722: Adjust some Spark heuristic parameters of Dr Elephant
timyitong Sep 29, 2015
a8a9a52
LIHADOOP-14688 Adding a timestamp to Dr. Elephant search result page
Sep 30, 2015
54f1f7f
LIHADOOP-13819: Dr. Elephant heuristic request. GC efficiency
Oct 14, 2015
b35ea5d
LIHADOOP-13711: Dr. Elephant should add virtual memory info in its me…
Sep 3, 2015
2aa93da
LIHADOOP-15134: Dr. Elephant should provide exception stacktrace from…
Oct 14, 2015
f80f7b9
LIHADOOP-14342: Dr. Elephant's memory heuristic has a corner case to …
Oct 14, 2015
ace7607
LIHADOOP-15136: Release Dr. Elephant v1.0.5
Oct 14, 2015
1d44f48
LIHADOOP-15080 Clean up and organize the views in Dr. Elephant
Oct 12, 2015
6b66c86
LIHADOOP-15252: Update Dr. Elephant version to v1.0.6-SNAPSHOT
Oct 22, 2015
850a17f
LIHADOOP-15372 Update the Heuristic Conf for helpGC with the proper p…
akshayrai Oct 26, 2015
2569a66
LIHADOOP-15377 Job execution URLs on Dr. Elephant's Flow Execution pa…
akshayrai Oct 27, 2015
bc186e2
LIHADOOP-16125 Other executions of a job should list the recent jobs …
Dec 8, 2015
1a2a498
LIHADOOP-15217 Implement comparison of historical flows/jobs in Dr. E…
Dec 11, 2015
4471ae9
LIHADOOP-16102 Discuss and remove Hadoop 1 support from Dr. Elephant
Dec 20, 2015
ce4abb4
LIHADOOP-9004 Support Hadoop 2 properties in Dr. Elephant
Jan 4, 2016
ba92a86
LIHADOOP-16687 Update Dr. Elephant to use the new Hadoop API
Jan 6, 2016
b9feaeb
LIHADOOP-16576 Make severity threshold values configurable for the he…
Dec 21, 2015
6ce11ca
LIHADOOP-17131 Prepare Readme for Dr. Elephant
akshayrai Jan 29, 2016
0f371c3
LIHADOOP-16587 Dr. Elephant should provide search by start time
akshayrai Jan 22, 2016
8fddb19
LIHADOOP-17485: Refactor and cleanup Dr. Elephant code
akshayrai Jan 30, 2016
e0bb36c
LIHADOOP-17502: Improve Dr. Elephant's history graph and tooltips
akshayrai Feb 14, 2016
96b7b5e
LIHADOOP-16100: Dr. Elephant should show timestamp specific to timezo…
akshayrai Feb 16, 2016
30ca499
LIHADOOP-15268: Task sampling in Dr. Elephant
Nov 16, 2015
cee14d3
LIHADOOP-17669: Update Notice and copyright year on Dr. Elephant
akshayrai Feb 25, 2016
581798c
LIHADOOP-17715: Remove Publish Metrics Logic from Dr. Elephant
akshayrai Feb 26, 2016
5cfd449
LIHADOOP-17537: Replace bootstrap-datepicker library with a copy from…
akshayrai Feb 25, 2016
c467979
LIHADOOP-17538: Code changes to support new schema
akshayrai Feb 15, 2016
a95fdf6
LIHADOOP-17783: Update mysql-connector-java version to 5.1.36
akshayrai Feb 26, 2016
c7856b8
LIHADOOP-15398: Release Dr. Elephant v2.0.0
akshayrai Feb 26, 2016
8e33311
LIHADOOP-15398: Release Dr. Elephant v2.0.1 (The jenkins build failed…
akshayrai Mar 1, 2016
cb3bf02
LIHADOOP-17890: Set default fields when spark fails to get the job conf
akshayrai Mar 1, 2016
71c66b8
LIHADOOP-16101: Review the evolution scripts in Dr. Elephant
akshayrai Mar 2, 2016
6f127ba
LIHADOOP-16103 Check and Separate Linkedin Specific Code from Dr. Ele…
akshayrai Mar 1, 2016
8afdffa
LIHADOOP-18028: Dr. Elephant 's spark parser on Nertz is not working
akshayrai Mar 8, 2016
808a58e
LIHADOOP-17996: Fix potential issues in Dr. Elephant v2.0.1
akshayrai Mar 7, 2016
6844372
LIHADOOP-18056: Release Dr. Elephant v2.0.2
akshayrai Mar 8, 2016
23312f6
LIHADOOP-16126: Easy integration of new schedulers to Dr. Elephant
akshayrai Mar 13, 2016
cffc26b
LIHADOOP-18124: Complete Open sourcing of Dr. Elephant
akshayrai Mar 13, 2016
f142fd9
LIHADOOP-18232: Include a resolver template file in the configuration
akshayrai Mar 15, 2016
2237777
Merge pull request #1 from akshayrai/master
akshayrai Mar 15, 2016
6f0fde6
LIHADOOP-18234: Add wiki documentation images to code base
akshayrai Mar 15, 2016
64e52e2
Merge pull request #2 from akshayrai/addImages
akshayrai Mar 15, 2016
c354ec9
LIHADOOP-18434: Add index on start time and allow querying by start t…
akshayrai Apr 2, 2016
bfd218f
Merge pull request #3 from akshayrai/revertToBigint
akshayrai Apr 7, 2016
313c99d
LIHADOOP-18918: Dr. Elephant should support all play versions
akshayrai Apr 8, 2016
7753b26
Merge pull request #4 from akshayrai/Support
akshayrai Apr 8, 2016
4075d07
LIHADOOP-18919: Investigate and fix why Dr. Elephant configurations a…
akshayrai Apr 8, 2016
d71e1da
Merge pull request #5 from akshayrai/confDirFix
akshayrai Apr 8, 2016
902d3ea
LIHADOOP-18977: Dr. Elephant doesn't compile with spark 1.5.0+
akshayrai Apr 11, 2016
56d322c
Merge pull request #9 from akshayrai/sparkTypeMismatch
akshayrai Apr 11, 2016
7d9e34e
Get the active resource manager URL when HA is enabled
Apr 19, 2016
9068957
fix start script to redirect output to a file instead of /dev/null
Apr 19, 2016
69a4108
addressed code review comments on RM HA support PR
Apr 19, 2016
3d7ead5
Merge pull request #15 from krishnap/kp_fix_start_script
akshayrai Apr 19, 2016
c3cb9ef
Merge pull request #14 from krishnap/kp_fix_rm_issue
akshayrai Apr 19, 2016
b3a4576
LIHADOOP-19325: [DR-Elephant] Add integration tests for the rest API'…
rajagopr Apr 22, 2016
fa2d249
Added a link to the mailing list.
akshayrai Apr 26, 2016
63aa982
Adding commons codec to the dependency (#26)
tglstory Apr 27, 2016
2d86f06
Spark dependency resolution based on version (#25)
tglstory Apr 27, 2016
04f02b2
LIHADOOP-18925: Expose additional metrics through Dr Elephant ( Job E…
nntnag17 Apr 19, 2016
8455ce4
LIHADOOP-19491: [DR-Elephant] Add code coverage to Dr. Elephant
rajagopr Apr 28, 2016
27d8c77
LIHADOOP-19495 Make sampling configurable in Dr elephant
nntnag17 Apr 28, 2016
5fa4785
LIHADOOP-19610: Metrics should not be shown if runtime and resourceus…
nntnag17 May 2, 2016
8411862
LIHADOOP-19609: Make Spark Event Log Directory configurable
akshayrai May 2, 2016
68d74d3
Fix an cast error in SparkDataCollection.scala (#37)
stiga-huang May 5, 2016
af70800
Separate configuration logic from RM address update logic (#38)
tglstory May 6, 2016
97014ac
LIHADOOP-19850: Test for AnalyticJob (#42)
rajagopr May 10, 2016
a215c9a
LIHADOOP-20088: Add organization name to Dr. Elephant to generate ivy…
rajagopr May 18, 2016
19935f4
LIHADOOP-18924: Improve Job History to show runtime, waittime, resour…
nntnag17 May 4, 2016
fcf8839
LIHADOOP-20170: Write help pages for new metrics in Dr. Elephant
nntnag17 May 23, 2016
24ef482
LIHADOOP-19666: totalDelay is negative for zero tasks
nntnag17 May 26, 2016
2364c26
LIHADOOP-19841: Enable Analyzing Spark jobs for Namenode HA
nntnag17 May 25, 2016
6d384ec
Merge remote-tracking branch 'origin/master' into AggregatedMetrics
nntnag17 May 26, 2016
5bc5f11
Minor fixup on AnalyticJobTest
nntnag17 May 26, 2016
d8baeb5
LIHADOOP-20350: Release dr-elephant v. 2.0.3
nntnag17 May 26, 2016
1ff6ece
LIHADOOP-20390: Disable AnalyticTest in Dr. Elephant
nntnag17 May 27, 2016
dfdf644
LIHADOOP-20347: Fix Mapper Speed Heuristic's Disk Speed Severity Thre…
akshayrai May 27, 2016
12ea974
LIHADOOP-20392: NullPointerException on zero mappers or reducers
nntnag17 May 27, 2016
9229aee
Merge remote-tracking branch 'upstream/master' into AggregatedMetrics
nntnag17 May 27, 2016
70b03b7
Added engineering blog link
akshayrai May 28, 2016
9d93de1
LIHADOOP-20476: Generating code coverage report in xml format too. Th…
rajagopr Jun 1, 2016
b4ca1b9
LIHADOOP-20582: Dr Elephant shows all the analyzed jobs on the dashbo…
nntnag17 Jun 6, 2016
fc78ebc
LIHADOOP-20577: Dr Elephant shows more than 100% waittime for some jobs
nntnag17 Jun 6, 2016
04bbe3b
LIHADOOP-20626 Ignore slow start for considering reducer delay
nntnag17 Jun 7, 2016
e65615b
LIHADOOP-20629 Minor UI fixes for Dr. Elephant
nntnag17 Jun 7, 2016
bf2a378
Merge branch 'master' of https://github.com/linkedin/dr-elephant into…
nntnag17 Jun 7, 2016
b34652d
LIHADOOP-20630: Release Dr.Elephant v. 2.0.4
nntnag17 Jun 7, 2016
4ce5f18
Removing some un-necessary logic. (#70)
rajagopr Jun 8, 2016
cfb720a
Added support to search by queue-name (#40)
chetnachaudhari Jun 9, 2016
ee80526
Try alternate resource manager if one is down (#78)
paulbramsen Jun 18, 2016
cb1ce83
Ignore and remove .DS_Store files (#80)
paulbramsen Jun 18, 2016
14decef
Add GeneralConf.xml to make number of executors configurable (#75)
stiga-huang Jun 22, 2016
1bd8f98
Add spark log ext conf param (#79)
paulbramsen Jun 23, 2016
c4d688a
Fix fallback to default `SPARK_LOG_EXT` value (#82)
ljank Jun 23, 2016
4814818
Add Travis CI support (#86)
paulbramsen Jun 25, 2016
ab40207
Add Airflow scheduler support to Dr. Elephant. Includes Tests. (#83)
paulbramsen Jul 1, 2016
baf9024
Make default container mem configurable (#90)
paulbramsen Jul 1, 2016
80568ec
Web analytics for Dr. Elephant. (#94)
rajagopr Jul 5, 2016
ed61aa0
Fix job history page tab highlighting (#98)
paulbramsen Jul 7, 2016
69e37af
dr-elephant#95: copy conf files to zip package automatically (#96)
YannByron Jul 7, 2016
dcf0990
Add Gitter badge (#102)
gitter-badger Jul 14, 2016
7297e28
Put build and gitter badges on the same line (#105)
paulbramsen Jul 14, 2016
ddaba87
Update wiki images for aggregated metrics feature.
nntnag17 Jul 15, 2016
9086515
Merge remote-tracking branch 'origin/AggregatedMetrics'
nntnag17 Jul 15, 2016
27bcaf6
Add 3.sql and revert modifications in 1.sql
nntnag17 Jul 18, 2016
600f330
Merge pull request #109 from nntnag17/master
nntnag17 Jul 20, 2016
f22440f
Make flow search more powerful (#99)
paulbramsen Jul 21, 2016
a8b2f8a
Add thread to all log output for better debugging (#108)
paulbramsen Jul 21, 2016
84919e2
Fix Flow/Job ID/Url mixups (#100)
paulbramsen Jul 22, 2016
b876c02
fix a log error when throttling spark event log (#114)
stiga-huang Jul 25, 2016
b56b2c4
LIHADOOP-21568: Add Dropwizard metrics to Dr. Elephant. This change e…
rajagopr Jul 27, 2016
189acb4
Add default for boolean unboxing (#119)
paulbramsen Jul 28, 2016
3d6bbc9
Fix sync issue in info extractor (#118)
paulbramsen Aug 1, 2016
3f7981a
Modify main run loop to be inline with thread pool executor paradigms…
paulbramsen Aug 1, 2016
32bc429
Faster fetcher for MapReduce apps (#67)
stiga-huang Aug 1, 2016
71a69e7
Show result count in search (#120)
paulbramsen Aug 2, 2016
6c1e86d
LIHADOOP-21657: Release Dr. Elephant v. 2.0.5 (#117)
rajagopr Aug 2, 2016
5783a9f
Add rest interface for getting resource usage by user (#125)
shankar37 Aug 8, 2016
65e0ed8
Improved metrics (#126)
paulbramsen Aug 8, 2016
91a6f64
Add Metrics feature for Spark Jobs (#135)
shankar37 Aug 29, 2016
dad905c
Changes to load an agent jar along with Dr. Elephant. Current use-cas…
rajagopr Sep 14, 2016
e41025e
Add rest end point for new user interface
nntnag17 Sep 23, 2016
113c9d3
LIHADOOP-23923: Add rest end points for search results and search opt…
nntnag17 Oct 3, 2016
f5ea207
LIHADOOP-23967: Setup JobHistory, FlowHistory and help pages for new …
nntnag17 Oct 13, 2016
ff59bd9
LIHADOOP-23924 Create dashboard of Dr Elephant using ember
nntnag17 Oct 7, 2016
81e1cd9
Merge pull request #150 from nntnag17/UIHistoryHelp
nntnag17 Oct 18, 2016
7f1cb88
LIHADOOP-24102 Create Workflow, Job and Application pages in ember
nntnag17 Oct 18, 2016
39888ff
Added License to all the new files
nntnag17 Oct 19, 2016
a8ec16d
Integration of the ember app with play framework
nntnag17 Oct 24, 2016
fa66fa4
Merge pull request #157 from nntnag17/UIHistoryHelp
nntnag17 Oct 26, 2016
f38f3bc
LIHADOOP-24211 Implement search page and panel
nntnag17 Oct 20, 2016
6c6a492
Return the resource Usage per user as GBHours instead of MBSeconds (#…
shankar37 Nov 7, 2016
d7daee5
Multiple fixes on Dr. Elephant: (#159)
nntnag17 Nov 7, 2016
d36ddd5
Increment version to 2.0.6 (#160)
nntnag17 Nov 7, 2016
aa510fe
Multiple fixes: (#161)
nntnag17 Nov 9, 2016
c8a9af6
Oozie scheduler support (#152)
NoamShaish Nov 17, 2016
08377da
Add null checks around update methods (#164)
babak-altiscale Nov 23, 2016
5f056c1
logger actually returned number of application types, not job types (…
Dec 12, 2016
ca08fce
add exclude_jobs_filter to heuristics that is checked in AnalyticJob …
Dec 12, 2016
09c5971
fix superParentId - now also returns the Coordinator id for subworkfl…
Dec 12, 2016
134bab1
use an ordered map to preserve order of jobs (#170)
Dec 12, 2016
28f4025
Rewrite Spark fetcher/heuristics. (#162)
rayortigas Dec 13, 2016
e3b1096
LIHADOOP-24958: Investigate the cause of Skipped Jobs Alert in Dr. El…
akshayrai Dec 13, 2016
cf5576b
Post Refactoring of Spark Fetcher, we have removed Java 6 support
akshayrai Dec 13, 2016
42a9be7
Resources used (#173)
bpitman Jan 5, 2017
e0c5935
Removes dependencies on Play to load classes (#179)
shkhrgpt Jan 5, 2017
416cf90
Post Refactoring of Spark Fetcher, we have removed Java 6 support (#172)
akshayrai Jan 5, 2017
9a13165
Add JAVA_OPTS env var export to start.sh (#116)
paulbramsen Jan 5, 2017
e16a192
LIHADOOP-25142: Build the user-summary page for Dr. Elephant (#174)
nntnag17 Jan 6, 2017
e8e23e5
Removes play dependencies to read config file and read system propert…
shkhrgpt Jan 8, 2017
2f687cd
LIHADOOP-25809: Update Dr. Elephant README to point to Quick Setup an…
akshayrai Jan 13, 2017
22ae13d
Scheduler-less support for flow and job history (#181)
vaxvms Jan 15, 2017
1d0350b
Show exceptions for failed workflows (#188)
nntnag17 Jan 24, 2017
4df9ba9
Added new heuristic DistributedCacheLimit heuristic. (#187)
rajagopr Feb 6, 2017
2a84735
Cleanes up MapReduceTaskData class by removing unnecessary constructo…
shkhrgpt Feb 7, 2017
e93d431
Fixes Spark REST fetcher for client mode applications (#193)
shkhrgpt Feb 7, 2017
dd7a458
Fix for null pointers in TaskList returned by MapReduceFSFetcherHadoo…
stiga-huang Feb 9, 2017
d3c90d5
Fix #162 with the right calculation for resourceswasted and add miss…
Feb 22, 2017
da7983c
Fix Exception thrown when JAVA_EXTRA_OPTIONS is not present (#210)
Feb 23, 2017
0d668ab
Adds an option to fetch recently finished apps from RM (#212)
shkhrgpt Feb 28, 2017
f6274b1
Fixes issue caused by http in history server config property (#217)
shkhrgpt Feb 28, 2017
965cba3
add config for timezone of job history server (#214)
stiga-huang Feb 28, 2017
6b80614
Include reference to the weekly meeting
akshayrai Mar 4, 2017
5a98701
Fixes MapReduce aggregator and heuristic to correctly handle task dat…
shkhrgpt Mar 28, 2017
8e4a094
Spark fetcher is now able to fetch event logs via REST API (#225)
superbobry Apr 12, 2017
c8a7009
Update #224 (credits: rayortigas) to add FSFetcher as a standalone fe…
Apr 14, 2017
b7e04ab
Spark metrics aggregator fix (#237)
Apr 18, 2017
a1f866a
Minor bug fixes in exception and UI (#238)
nntnag17 Apr 19, 2017
7c373d4
Updates Spark configuration heuristic severity calculations (#229)
shkhrgpt Apr 20, 2017
1ca2676
Enables SparkFetcher to only get eventLog via rest and process it loc…
shkhrgpt May 3, 2017
cae79c7
Refactor statusapiv1 to trait and implement for ease of creation of t…
May 23, 2017
cdf680b
MRfetcher ignores failed tasks (#249)
May 30, 2017
f77886a
Add index on severity, finish_time to speed up welcome page display (…
ashangit Jun 7, 2017
54a16fd
Add pinball scheduler to dr-elephant (#253)
georgewu2 Jun 12, 2017
1d6f3f6
add s3, s3a, s3n bytes read and bytes written, and update heuristics …
georgewu2 Jun 21, 2017
9a65e0e
Add custom flowtime per scheduler (#268)
Jul 28, 2017
7230038
Add filtering on Job Definition Id in the Search view (#269)
Jul 28, 2017
752a94b
added logic for map reduce time-skew heuristic (#267)
skakker Jul 28, 2017
62a116f
update.sh and update.py scripts for automatic adding new heuristic + …
Aug 10, 2017
9190667
Standard deviation in mapper time heuristic
Aug 10, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
24 changes: 24 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
#
# Copyright 2016 LinkedIn Corp.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy of
# the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under
# the License.
#

# general
*~
*.log
Expand Down Expand Up @@ -32,5 +48,13 @@ target
logs
RUNNING_PID

# macOS
.DS_Store

*.jar

#web
public/assets/ember/
public/assets/fonts/
web/bower_components/
web/node_modules/
11 changes: 11 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
language: scala
sudo: false
jdk:
- oraclejdk8
- oraclejdk7

# only build PRs and master (not all branch pushes)
branches:
only:
- master

202 changes: 202 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@

Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.

"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:

(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.

You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
115 changes: 115 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
Copyright 2016 LinkedIn Corp.

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
------------------------------------------------------------------------------

This product automatically loads third party code from an external repository(configurable) using Play's build system.
Such third party code is subject to other license terms than as set forth above. Please review the complete list of
dependencies for applicable license terms.

In addition, such third party code may also depend on and load multiple tiers of dependencies. Please review the
applicable licenses of the additional dependencies.

------------------------------------------------------------------------------

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/)

This product includes/uses Apache Commons IO (https://commons.apache.org/proper/commons-io/)
Copyright (c) 2002-2016 The Apache Software Foundation
License: Apache 2.0

This product includes/uses Gson (https://github.com/google/gson)
Copyright (C) 2008 Google Inc.
License: Apache 2.0

This product includes/uses Jsoup (https://github.com/jhy/jsoup)
Copyright (c) 2009 - 2015 Jonathan Hedley ([email protected])
License: The MIT License (https://github.com/jhy/jsoup/blob/master/LICENSE)

This product includes/uses Guava (https://github.com/google/guava)
License: Apache 2.0

This product includes/uses Jackson Databind (http://github.com/FasterXML/jackson)
License: Apache 2.0

This product includes/uses Apache Hadoop (http://hadoop.apache.org/)
Notice: https://github.com/apache/hadoop/blob/trunk/NOTICE.txt
License: Apache 2.0

This product includes/uses Apache Spark (http://spark.apache.org/)
Notice: https://github.com/apache/spark/blob/master/NOTICE
License: Apache 2.0 (https://github.com/apache/spark/blob/master/LICENSE)

This product includes/uses Play Framework (https://www.playframework.com/)
Copyright (C) 2009-2016 Typesafe Inc. (http://www.typesafe.com)
License: Apache 2.0

This product requires the following play dependencies in addition to the core Play library
javaEbean, the Ebean plugin for Java.
javaJdbc, the Java database API.

This product includes/uses JMockit (http://jmockit.org/)
Notice: https://github.com/jmockit/jmockit1/blob/master/NOTICE.txt
License: The MIT License (https://github.com/jmockit/jmockit1/blob/master/LICENSE.txt)

This product includes/uses Mockito (http://mockito.org)
Copyright (c) 2007 Mockito contributors
License: The MIT License (https://github.com/mockito/mockito/blob/master/LICENSE)

This product includes/uses Metrics (http://metrics.dropwizard.io/)
Copyright 2010-2014, Coda Hale, Yammer Inc.. Created using Sphinx 1.4.1.
License: Apache 2.0 (https://github.com/dropwizard/metrics/blob/master/LICENSE)

This product includes/uses ember-cli (https://ember-cli.com/)
Copyright (c) 2013-2016 Stefan Penner, Robert Jackson and ember-cli contributors
License: The MIT License (MIT) (https://github.com/ember-cli/ember-cli/blob/master/LICENSE)

This product includes/uses bower (https://bower.io/)
Copyright (c) 2016 Twitter and other contributors (https://github.com/bower/bower/blob/master/LICENSE)

This product includes/uses broccoli-asset-rev
Copyright (c) 2014 Rick Harrison
The MIT License (MIT) (https://github.com/rickharrison/broccoli-asset-rev/blob/master/LICENSE)

------------------------------------------------------------------------------
Attribution for JavaScript Libraries
------------------------------------------------------------------------------

jQuery (http://jquery.com/)
Copyright 2005, 2014 jQuery Foundation, Inc. and other contributors
License: MIT License (http://jquery.org/license)

jQuery Deserialize (https://github.com/kflorence/jquery-deserialize/)
Copyright (c) 2015 Kyle Florence
License: Dual licensed under the MIT and GPLv2 licenses.

Bootstrap (http://getbootstrap.com/)
Copyright 2011-2016 Twitter, Inc.
License: MIT License (https://github.com/twbs/bootstrap/blob/master/LICENSE)

Bootstrap Datepicker (https://github.com/eternicode/bootstrap-datepicker)
Copyright 2012 Stefan Petre, Improvements by Andrew Rowls
License: Apache 2.0

D3.js (http://d3js.org)
Copyright (c) 2010-2016, Michael Bostock
License: BSD 3-Clause (https://github.com/mbostock/d3/blob/master/LICENSE)

ember.js ( http://emberjs.com/)
Copyright (c) 2016 Yehuda Katz, Tom Dale and Ember.js contributors
License: (https://github.com/emberjs/ember.js/blob/master/LICENSE)

------------------------------------------------------------------------------


Loading