diff --git a/LICENSE b/LICENSE index 57ee58ca8..f6acfa3b2 100644 --- a/LICENSE +++ b/LICENSE @@ -204,156 +204,6 @@ MIT License ----------- -Boostrap V4.6.1 - bootstrap.bundle.min.js - bootstrap.min.css - - The MIT License (MIT) - - Copyright (c) 2011-2019 Twitter, Inc. - Copyright (c) 2011-2019 The Bootstrap Authors - - Permission is hereby granted, free of charge, to any person obtaining a copy - of this software and associated documentation files (the "Software"), to deal - in the Software without restriction, including without limitation the rights - to use, copy, modify, merge, publish, distribute, sublicense, and/or sell - copies of the Software, and to permit persons to whom the Software is - furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in - all copies or substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE - AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER - LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN - THE SOFTWARE. - - -DataTables - DataTables-1.12.0/jquery.dataTables.min.js - DataTables-1.12.0/dataTables.bootstrap4.min.css - DataTables-1.12.0/dataTables.bootstrap4.min.js - searchpanes-2.0.1/searchPanes.bootstrap4.min.css - searchpanes-2.0.1/searchPanes.bootstrap4.min.js - searchpanes-2.0.1/dataTables.searchPanes.min.js - select-1.4.0/select.bootstrap4.min.css - select-1.4.0/dataTables.select.min.js - buttons-2.2.3/buttons.bootstrap4.min.css - buttons-2.2.3/buttons.bootstrap4.min.js - buttons-2.2.3/buttons.html5.min.js - buttons-2.2.3/dataTables.buttons.min.js - responsive-2.3.0/dataTables.responsive.min.js - responsive-2.3.0/responsive.bootstrap4.min.css - responsive-2.3.0/responsive.bootstrap4.min.js - - The MIT License (MIT) - - Copyright (C) 2008-2022, SpryMedia Ltd. - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software - and associated documentation files (the "Software"), to deal in the Software without - restriction, including without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the - Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or - substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING - BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, - DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -jQuery V3.6.0 - jquery.min.js - - Copyright JS Foundation and other contributors, https://js.foundation/ - - This software consists of voluntary contributions made by many - individuals. For exact contribution history, see the revision history - available at https://github.com/jquery/sizzle - - The following license applies to all parts of this software except as - documented below: - - ==== - - Permission is hereby granted, free of charge, to any person obtaining - a copy of this software and associated documentation files (the - "Software"), to deal in the Software without restriction, including - without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to - permit persons to whom the Software is furnished to do so, subject to - the following conditions: - - The above copyright notice and this permission notice shall be - included in all copies or substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF - MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE - LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION - OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION - WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - ==== - - All files located in the node_modules and external directories are - externally maintained libraries used by this software which have their - own licenses; we recommend you read them, as their terms may differ from - the terms above. - -Mustache.js V4.10 - mustache.min.js - - The MIT License - - Copyright (c) 2009 Chris Wanstrath (Ruby) - Copyright (c) 2010-2014 Jan Lehnardt (JavaScript) - Copyright (c) 2010-2015 The mustache.js community - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software - and associated documentation files (the "Software"), to deal in the Software without - restriction, including without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the - Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or - substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING - BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, - DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -Spur Dashboard V1.1.0 - spur.min.css - - The MIT License (MIT) - - Copyright 2018 Alexander Rechsteiner - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software - and associated documentation files (the "Software"), to deal in the Software without - restriction, including without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the - Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or - substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING - BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, - DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - jsoup License The MIT License diff --git a/NOTICE-binary b/NOTICE-binary index a27d0356a..fc2e7b4f0 100644 --- a/NOTICE-binary +++ b/NOTICE-binary @@ -1,5 +1,5 @@ Spark Rapids Tools -Copyright (c) 2023, NVIDIA CORPORATION +Copyright (c) 2023-2024, NVIDIA CORPORATION // ------------------------------------------------------------------ // NOTICE file corresponding to the section 4d of The Apache License, @@ -17,29 +17,6 @@ snakeyaml --------------------------------------------------------------------- This product bundles various third-party components under other open source licenses. -Boostrap V4.6.1 - The MIT License (MIT) - License Text ( https://github.com/twbs/bootstrap/blob/v4.6.1/LICENSE ) - Copyright (c) 2011-2021 Twitter, Inc. - Copyright (c) 2011-2021 The Bootstrap Authors - -DataTablesSrc V1.12 - The MIT License (MIT) - License Text ( https://datatables.net/license/ ) - Copyright (C) 2008-present, SpryMedia Ltd. - -jQuery V3.6.0 - The MIT License (MIT) - License Text ( https://jquery.org/license/ ) - Copyright OpenJS Foundation and other contributors, https://openjsf.org/ - -Mustache.js V4.10 - The MIT License (MIT) - License Text ( https://github.com/janl/mustache.js/blob/master/LICENSE ) - Copyright (c) 2009 Chris Wanstrath (Ruby) - Copyright (c) 2010-2014 Jan Lehnardt (JavaScript) - Copyright (c) 2010-2015 The mustache.js community - -Spur Dashboard V1.1.0 - The MIT License (MIT) - License Text ( https://github.com/HackerThemes/spur-template/blob/master/LICENSE ) - Copyright 2016 - 2019 Alexander Rechsteiner - jsoup - The MIT License (MIT) License Text ( https://jsoup.org/license ) Copyright (c) 2009 - 2023 Jonathan Hedley (https://jsoup.org/) diff --git a/core/.gitignore b/core/.gitignore index a3b961d01..2cfd7e010 100644 --- a/core/.gitignore +++ b/core/.gitignore @@ -2,9 +2,5 @@ *.iml target event_log_profiling.log -## ignore qual-ui static files -**/ui/assets/ -**/ui-dependencies-cache/ -**/ui/js/data-output.js ## ignore output folders of the test scripts **/dev/qualification-output/ diff --git a/core/pom.xml b/core/pom.xml index 5fb33b01d..9cbffc469 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -472,10 +472,6 @@ 3.0.5 spark${buildver} ${spark.version.classifier} - ${project.build.directory}/ui-dependencies-cache - src/main/resources/ui - ${project.basedir}/${ui.resources.relative} - assets 3.2.0 3.0.0 3.9.0 @@ -786,40 +782,6 @@ false - - download-ui-dependencies - generate-sources - - run - - - - - - - - - - - - - - - - - - - - - - copy-notice @@ -887,31 +849,6 @@ - - - org.apache.maven.plugins - maven-clean-plugin - ${maven.clean.plugin.version} - - - clean-qual-ui-dependencies - clean - - clean - - - - - ${ui.resources.relative} - - ${ui.resources.external.folder}/ - - - - - - - org.apache.maven.plugins maven-jar-plugin diff --git a/core/prepare-ui-libraries.xml b/core/prepare-ui-libraries.xml deleted file mode 100644 index a65a2a19f..000000000 --- a/core/prepare-ui-libraries.xml +++ /dev/null @@ -1,267 +0,0 @@ - - - - - Ant task to download dependency files from internet and extract during maven package. - Those files are used to render the Qualification UI report file. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Checksum error - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/core/src/main/resources/ui/css/rapids-dashboard.css b/core/src/main/resources/ui/css/rapids-dashboard.css deleted file mode 100644 index 3f5b2bad8..000000000 --- a/core/src/main/resources/ui/css/rapids-dashboard.css +++ /dev/null @@ -1,86 +0,0 @@ -/*! - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* Customised Styling for the RAPIDS dashboard */ - -.arrow-open { - width: 0; - height: 0; - border-left: 5px solid transparent; - border-right: 5px solid transparent; - border-top: 5px solid #08c; - display: inline-block; - margin-bottom: 2px; -} - -.arrow-closed { - width: 0; - height: 0; - border-top: 5px solid transparent; - border-bottom: 5px solid transparent; - border-left: 5px solid #08c; - display: inline-block; - margin-left: 2px; - margin-right: 3px; -} - -table.dataTable tr.dtrg-group.dtrg-level-0 td { - background-color: #f0f1f7; -} - -.badge-strong-recommended { - color: #fff; - background-color: #1DAB47; } -a.badge-strong-recommended:hover, a.badge-strong-recommended:focus { - color: #fff; - background-color: #167f35; } -a.badge-strong-recommended:focus, a.badge-strong-recommended.focus { - outline: 0; - -webkit-box-shadow: 0 0 0 0.2rem rgba(29, 171, 71, 0.5); - box-shadow: 0 0 0 0.2rem rgba(29, 171, 71, 0.5); } - -.badge-recommended { - color: #fff; - background-color: rgba(10, 102, 35, 0.98); } -a.badge-recommended:hover, a.badge-recommended:focus { - color: #fff; - background-color: #0a6623; } -a.badge-recommended:focus, a.badge-recommended.focus { - outline: 0; - -webkit-box-shadow: 0 0 0 0.2rem rgba(29, 171, 71, 0.5); - box-shadow: 0 0 0 0.2rem rgba(29, 171, 71, 0.5); } - -.badge-not-recommended { - color: #212529; - background-color: #FCAE3F; } -a.badge-not-recommended:hover, a.badge-not-recommended:focus { - color: #212529; - background-color: #fb990d; } -a.badge-not-recommended:focus, a.badge-not-recommended.focus { - outline: 0; - -webkit-box-shadow: 0 0 0 0.2rem rgba(252, 174, 63, 0.5); - box-shadow: 0 0 0 0.2rem rgba(252, 174, 63, 0.5); } - -.badge-not-applicable { - color: #fff; - background-color: #727F94; } -a.badge-not-applicable:hover, a.badge-not-applicable:focus { - color: #fff; - background-color: #5b6678; } -a.badge-not-applicable:focus, a.badge-not-applicable.focus { - outline: 0; - -webkit-box-shadow: 0 0 0 0.2rem rgba(114, 127, 148, 0.5); - box-shadow: 0 0 0 0.2rem rgba(114, 127, 148, 0.5); } \ No newline at end of file diff --git a/core/src/main/resources/ui/html/application.html b/core/src/main/resources/ui/html/application.html deleted file mode 100644 index 42969ab21..000000000 --- a/core/src/main/resources/ui/html/application.html +++ /dev/null @@ -1,212 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Spark Qualification tool - Application Report - - - -
-
-
- Qualification Tool -
- -
-
-
- - - - -
- GitHub -
-
-
-
-
- -
-
-
-
-
- - - - -
-
Application Details
-
-
-
-
-
-
-
- -
-
-
-
-
- - - -
-
Stage Details
-
-
-
-
-
-
-
- -
-
-
-
-
- - - -
-
Execs Details
-
-
-
-
-
-
-
- -
-
-
-
-
- - - -
-
SQL Details
-
-
-
-
-
-
-
-
-
-
-
- - - - - - - - - diff --git a/core/src/main/resources/ui/html/index.html b/core/src/main/resources/ui/html/index.html deleted file mode 100644 index 13ff4656c..000000000 --- a/core/src/main/resources/ui/html/index.html +++ /dev/null @@ -1,192 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Qualification Tool Dashboard - - - -
-
-
- Qualification Tool -
- -
-
-
- - - - -
- GitHub -
-
-
-

Qualification Tool

-
- -
-
- -
-
-
-
-
-
-
-
-
- - - - -
-
Apps GPU Recommendations Table
-
-
- - -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- - - - - - - - diff --git a/core/src/main/resources/ui/html/raw.html b/core/src/main/resources/ui/html/raw.html deleted file mode 100644 index 10000e470..000000000 --- a/core/src/main/resources/ui/html/raw.html +++ /dev/null @@ -1,174 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Qualification Tool Dashboard – Raw Data - - - -
-
-
- Qualification Tool -
- -
-
-
- - - - -
- GitHub -
-
-
-
-

Full Applications Report

-
-
- -
-
-
-
-
-
-
- - - - -
-
Raw Data of all Applications
-
-
-
-
-
-
-
-
-
-
-
- - - - - - - - - diff --git a/core/src/main/resources/ui/html/sql-recommendation.html b/core/src/main/resources/ui/html/sql-recommendation.html deleted file mode 100644 index f31fbf50e..000000000 --- a/core/src/main/resources/ui/html/sql-recommendation.html +++ /dev/null @@ -1,174 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Qualification Tool Dashboard – Per-SQL GPU Recommendation - - - -
-
-
- Qualification Tool -
- -
-
-
- - - - -
- GitHub -
-
-
-
-

Per-SQL GPU Recommendation

-
-
- -
-
-
-
-
-
-
- - - - -
-
Per-SQL GPU Recommendation Table
-
-
-
-
-
-
-
-
-
-
-
- - - - - - - - - diff --git a/core/src/main/resources/ui/js/app-report.js b/core/src/main/resources/ui/js/app-report.js deleted file mode 100644 index aa5a6ec7e..000000000 --- a/core/src/main/resources/ui/js/app-report.js +++ /dev/null @@ -1,341 +0,0 @@ -/* - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* globals $, jQuery, Mustache, qualificationRecords, qualReportSummary */ - -let applicationUIRecord = null; - -function fetchApplicationData(appsArray, id) { - return appsArray.find(app => app.appId === id); -} - -// Uses DFS to get flat representations of the execsInfo -function dpExecInfo(queue, resultArray) { - while (!queue.isEmpty()) { - let execInfo = queue.dequeue(); - resultArray.push(execInfo); - // extract execName into the map to build the search Panes - let execName = extractExecName(execInfo.exec); - execInfo["execName"] = execName; - execNames.set(execName, true); - for (let sqlInd in execInfo.children) { - queue.enqueue(execInfo.children[sqlInd]); - } - } -} - -function getAppExecArray(appRecord) { - let execInfos = []; - let execBuffer = new Queue(); - for (let ind in appRecord.planInfo) { - let currPlan = appRecord.planInfo[ind]; - - if (currPlan.execInfo.length > 0) { - // concatenate all elements adding appID - for (let sqlInd in currPlan.execInfo) { - let execInfoRec = currPlan.execInfo[sqlInd]; - execBuffer.enqueue(execInfoRec); - } - } - dpExecInfo(execBuffer, execInfos) - } - return execInfos; -} - -function getAppStagesArray(appRecord) { - return appStagesMap.addAppRec(appRecord); -} - -function getAppReportPageHeaderTemplate() { - let content = ` -

App Details: {{appInfo.appName}} {{{extension.badgeWrapper}}}

-
-
- -
-
-
-
-
-

App Duration

-
-
- - - -
-
-
{{appInfo.timing.appDuration}}
-
- {{appInfo.timing.startTime}} - Start -
-
- {{appInfo.timing.endTime}} - End -
-
-
-
-
-
-
-

GPU Opportunity

-
-
- - - - - - - - - - - -
-
-
{{appInfo.timing.gpuOpportunity}}
-
- {{appInfo.timing.sqlDFDuration}} - SQL DF Duration -
-
- {{appInfo.taskSpeedupFactor}} - Task Speed-up Factor -
-
-
-
-
-
-
-

Estimated GPU Duration

-
-
- - - - - - - -
-
-
{{appInfo.timing.estimatedGPUDuration}}
-
- {{appInfo.timing.estimatedGPUTimeSaved}} - Time Saved -
-
- {{appInfo.estimatedSpeedup}} - App Speed-up -
-
-
-
-
-
- `; - return content; -} - -function getAppStagesDetailsTableTemplate() { - let content = ` -
- - - - {{#displayCol_appID}} - - {{/displayCol_appID}} - - - - - - - -
App IDStage IDAverage Speedup FactorStage Task DurationUnsupported Task DurationStage Estimated
-
- `; - return content; -} - -function getAppExecsDetailsTableTemplate() { - let content = ` -
- - - - {{#displayCol_appID}} - - {{/displayCol_appID}} - - - - - - - - - - - - - -
App IDSQL IDExec NameExpression NameExec Is SupportedSpeed-up FactorExec DurationSQL Node IdStagesExec ChildrenExec Children Node IdsIs Removed
-
- `; - return content; -} - -$(document).ready(function() { - const queryString = window.location.search; - const urlParams = new URLSearchParams(queryString); - const appID = urlParams.get('app_id') - // get the appData - let rawAppRecord = fetchApplicationData(qualificationRecords, appID); - let attemptsArray = [rawAppRecord]; - applicationUIRecord = processRawData(attemptsArray)[0]; - let execsArray = getAppExecArray(applicationUIRecord); - let sqlIDsArray = getAppSqlArray(applicationUIRecord); - // - // set the statistics cards - // - let appHeaderInfoRec = { - "appInfo": { - appName: applicationUIRecord.appName, - timing: { - appDuration: formatDuration(applicationUIRecord.appDuration), - startTime: formatTimeMillis(applicationUIRecord.startTime), - endTime: formatTimeMillis(applicationUIRecord.startTime + applicationUIRecord.appDuration), - gpuOpportunity: applicationUIRecord.durationCollection.gpuOpportunity, - sqlDFDuration: applicationUIRecord.durationCollection.sqlDFDuration, - estimatedGPUDuration: applicationUIRecord.durationCollection.estimatedGPUDuration, - estimatedGPUTimeSaved: applicationUIRecord.durationCollection.gpuTimeSaved, - }, - estimatedSpeedup: applicationUIRecord.totalSpeedup_display, - taskSpeedupFactor: applicationUIRecord.taskSpeedupFactor_display, - execs: { - totalExecutors: execsArray.length - } - }, - "extension": { - badgeWrapper: getAppBadgeHTMLWrapper(applicationUIRecord) - } - }; - let headerContent = Mustache.render(getAppReportPageHeaderTemplate(), appHeaderInfoRec); - $("#app-report-page-header").html(jQuery.parseHTML(headerContent, false)); - - // - // set the app details table - // - let appDetailsDataTable = constructDataTableFromHTMLTemplate( - [applicationUIRecord], - "singleAppView", - createAppDetailedTableConf, - { - tableId: "appDetails", - appId: appID, - dataTableTemplate: getAppDetailsTableTemplate(), - datatableContainerID: '#app-details-data-container', - tableDivId: '#all-apps-raw-data-table', - } - ); - - // - // set the stage details table - // - let appStagesDataTable = constructDataTableFromHTMLTemplate( - getAppStagesArray(applicationUIRecord), - "singleAppView", - createAppDetailsStagesTableConf, - { - tableId: "appStages", - appId: appID, - dataTableTemplate: getAppStagesDetailsTableTemplate(), - datatableContainerID: '#app-stages-details-data-container', - tableDivId: '#app-stages-raw-data-table', - } - ); - - // - // set the exec details table - // - let appExecDetailsTable = constructDataTableFromHTMLTemplate( - execsArray, - "singleAppView", - createAppDetailsExecsTableConf, - { - tableId: "appExecs", - appId: appID, - dataTableTemplate: getAppExecsDetailsTableTemplate(), - datatableContainerID: '#app-execs-details-data-container', - tableDivId: '#app-execs-raw-data-table', - } - ); - - // - // set the sqlID details table - // - let appSQLsDetailsTable = constructDataTableFromHTMLTemplate( - sqlIDsArray, - "singleAppView", - createAppDetailsSQLsTableConf, - { - tableId: "appSQLs", - appId: appID, - dataTableTemplate: getAppSQLsDetailsTableTemplate(), - datatableContainerID: '#app-sqls-details-data-container', - tableDivId: '#app-sqls-raw-data-table', - replaceTableIfEmptyData: { - enabled: true, - text: "No Data to display in the table" - } - } - ); - - setupToolTipForTableCells(); - setupNavigation(); -}); diff --git a/core/src/main/resources/ui/js/per-sql-report.js b/core/src/main/resources/ui/js/per-sql-report.js deleted file mode 100644 index 75bc9f4c0..000000000 --- a/core/src/main/resources/ui/js/per-sql-report.js +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* global $, Mustache, formatDuration, jQuery, qualificationRecords */ - -function processSQLRawData(rawAppRecords) { - let sqlArray = []; - for (let ind in rawAppRecords) { - let currAppRec = rawAppRecords[ind]; - sqlArray.push(...getAppSqlArray(currAppRec)); - } - return sqlArray; -} - -$(document).ready(function() { - let sqlArray = processSQLRawData(qualificationRecords); - - // set the per-sql details table - let appSQLDetailsDataTable = constructDataTableFromHTMLTemplate( - sqlArray, - "listAppsView", - createAppDetailsSQLsTableConf, - { - tableId: "appSQLs", - dataTableTemplate: getAppSQLsDetailsTableTemplate(), - datatableContainerID: '#per-sql-details-data-container', - tableDivId: '#app-sqls-raw-data-table', - replaceTableIfEmptyData: { - enabled: true, - text: "No Data to display in the table" - } - } - ); - - setupToolTipForTableCells(); - setupNavigation(); -}); diff --git a/core/src/main/resources/ui/js/qual-report.js b/core/src/main/resources/ui/js/qual-report.js deleted file mode 100644 index 934ab5b99..000000000 --- a/core/src/main/resources/ui/js/qual-report.js +++ /dev/null @@ -1,450 +0,0 @@ -/* - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* global $, Mustache, formatDuration, jQuery, qualificationRecords, qualReportSummary */ - -/* - * HTML template used to render the application details in the collapsible - * rows of the GPURecommendationTable. - */ -function getExpandedAppDetails() { - let fullDetailsContent = - '
' + - ' Go To Full App Details' + - '
'; - let tableContent = - '' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - ' ' + - '
#ValueDescription
Estimated Speed-up {{totalSpeedup_display}} ' + toolTipsValues.gpuRecommendations.details.totalSpeedup + '
App Duration {{durationCollection.appDuration}} ' + toolTipsValues.gpuRecommendations["App Duration"] + '
Estimated GPU Duration {{durationCollection.estimatedDurationWallClock}} ' + toolTipsValues.gpuRecommendations.details.estimatedDuration + '
SQL Duration {{durationCollection.sqlDFDuration}} ' + toolTipsValues.gpuRecommendations.details.sqlDFDuration + '
GPU Opportunity {{durationCollection.gpuOpportunity}} ' + toolTipsValues.gpuRecommendations.details.gpuOpportunity + '
GPU Time Saved {{durationCollection.gpuTimeSaved}} ' + toolTipsValues.gpuRecommendations.details.gpuTimeSaved + '
'; - - if (UIConfig.fullAppView.enabled) { - return tableContent + fullDetailsContent; - } - return tableContent ; -} - -function formatAppGPURecommendation ( rowData) { - return Mustache.render(getExpandedAppDetails(), rowData); -} - -let definedDataTables = {}; -let gpuRecommendationTableID = "gpuRecommendations"; - -function expandAllGpuRowEntries() { - expandAllGpuRows(definedDataTables[gpuRecommendationTableID]); -} - -function collapseAllGpuRowEntries() { - collapseAllGpuRows(definedDataTables[gpuRecommendationTableID]); -} - -function expandAllGpuRows(gpuTable) { - // Enumerate all rows - gpuTable.rows().every(function(){ - // If row has details collapsed - if (!this.child.isShown()){ - // Open this row - this.child(formatAppGPURecommendation(this.data())).show(); - $(this.node()).addClass('shown'); - } - }); - gpuTable.draw(false); -} - -function collapseAllGpuRows(gpuTable) { - // Enumerate all rows - gpuTable.rows().every(function(){ - // If row has details expanded - if(this.child.isShown()){ - // Collapse row details - this.child.hide(); - $(this.node()).removeClass('shown'); - } - }); - gpuTable.draw(false); -} - -function createGPURecommendationTableConf( - attemptArray, - tableViewType, - mustacheRecord, - extraFunctionArgs) { - let initGpuRecommendationConf = UIConfig.datatables[extraFunctionArgs.tableId]; - let initGpuRecommendationCustomConf = initGpuRecommendationConf[tableViewType]; - // Start implementation of GPU Recommendations Apps - let recommendGPUColName = "recommendation" - let totalSpeedupColumnName = "totalSpeedup" - let gpuRecommendationConf = { - info: true, - paging: (attemptArray.length > defaultPageLength), - pageLength: defaultPageLength, - lengthMenu: defaultLengthMenu, - stripeClasses: [], - "data": attemptArray, - "columns": [ - { - "className": 'dt-control', - "orderable": false, - "data": null, - "defaultContent": '' - }, - { - name: "appName", - data: "appName" - }, - { - name: "appId", - data: "appId", - render: (appId, type, row) => { - if (type === 'display' || type === 'filter') { - if (UIConfig.fullAppView.enabled) { - return `${appId}` - } - } - return appId; - } - }, - { - name: recommendGPUColName, - data: 'gpuCategory', - render: function (data, type, row) { - if (type === 'display') { - return getAppBadgeHTMLWrapper(row); - } - return data; - }, - fnCreatedCell: (nTd, sData, oData, _ignored_iRow, _ignored_iCol) => { - let recommendGroup = recommendationsMap.get(sData); - let toolTipVal = recommendGroup.description; - $(nTd).attr('data-toggle', "tooltip"); - $(nTd).attr('data-placement', "top"); - $(nTd).attr('html', "true"); - $(nTd).attr('data-html', "true"); - $(nTd).attr('title', toolTipVal); - } - }, - { - name: totalSpeedupColumnName, - data: 'totalSpeedup', - searchable: false, - type: 'numeric', - render: function (data, type, row) { - if (type === 'display') { - return row.totalSpeedup_display - } - return data; - }, - }, - { - name: 'appDuration', - data: 'appDuration', - type: 'numeric', - searchable: false, - render: function (data, type, row) { - if (type === 'display') { - return formatDuration(data) - } - return data; - }, - fnCreatedCell: (nTd, sData, oData, _ignored_iRow, _ignored_iCol) => { - if (oData.endDurationEstimated) { - $(nTd).css('color', 'blue'); - } - } - }, - ], - //dom with search panes - dom: initGpuRecommendationCustomConf.Dom, - initComplete: function(settings, json) { - // Add custom Tool Tip to the headers of the table - let thLabel = extraFunctionArgs.tableDivId + ' thead th'; - let dataTableToolTip = toolTipsValues[initGpuRecommendationCustomConf.toolTipID]; - $(thLabel).each(function () { - let $td = $(this); - let toolTipVal = dataTableToolTip[$td.text().trim()]; - $td.attr('data-toggle', "tooltip"); - $td.attr('data-placement', "top"); - $td.attr('html', "true"); - $td.attr('data-html', "true"); - $td.attr('title', toolTipVal); - }); - } - }; - - processDatableColumns( - gpuRecommendationConf, - initGpuRecommendationConf, - initGpuRecommendationCustomConf, - mustacheRecord); - - - // set searchpanes - let optionGeneratorsFunctionsMap = new Map(); - optionGeneratorsFunctionsMap.set("recommendation", function() { - let categoryOptions = []; - for (let i in recommendationContainer) { - let currOption = { - label: recommendationContainer[i].displayName, - value: function(rowData, rowIdx) { - return (rowData["gpuCategory"] === recommendationContainer[i].displayName); - } - } - categoryOptions.push(currOption); - } - return categoryOptions; - }); - - optionGeneratorsFunctionsMap.set("users", function() { - let sparkUserOptions = []; - sparkUsers.forEach((data, userName) => { - let currOption = { - label: userName, - value: function(rowData, rowIdx) { - // get spark user - return (rowData["user"] === userName); - }, - } - sparkUserOptions.push(currOption); - }); - return sparkUserOptions; - }); - - setDataTableSearchPanes( - gpuRecommendationConf, - initGpuRecommendationConf, - initGpuRecommendationCustomConf, - optionGeneratorsFunctionsMap); - - // add buttons if enabled - setDataTableButtons( - gpuRecommendationConf, - initGpuRecommendationConf, - initGpuRecommendationCustomConf); - - return gpuRecommendationConf; -} - -function getGPURecommendationTableTemplate() { - return ` -
- - - - - - - - - - - -
- App Name - - App ID - - Recommendation - - Estimated Speed-up - - App Duration -
-
- `; -} - -function getGlobalStatisticsTemplate() { - return ` -
-
-
-

{{totalApps.header}}

-
-
- - - -
-
-
{{totalApps.numeric}}
-
- {{totalApps.totalAppsDurations}} - {{totalApps.totalAppsDurationLabel}} -
-
-
-
-
-
-
-

{{candidates.header}}

-
-
- - - -
-
-
{{candidates.numeric}}
-
- {{candidates.statsPercentage}} - {{candidates.statsTimeFrame}} -
-
-
-
-
-
-
-

{{speedups.header}}

-
-
- - - - - - - - - - - -
-
-
{{speedups.numeric}}
-
- {{speedups.totalSqlDataframeTaskDuration}} - {{speedups.totalSqlDFDurationsLabel}} -
-
- {{speedups.statsPercentage}} - {{speedups.statsTimeFrame}} -
-
-
-
-
-
- `; -} - -$(document).ready(function(){ - // do the required filtering here - let attemptArray = processRawData(qualificationRecords); - - let gpuRecommendationTable = constructDataTableFromHTMLTemplate( - attemptArray, - "listAppsView", - createGPURecommendationTableConf, - { - tableId: "gpuRecommendations", - datatableContainerID: '#app-recommendations-data-container', - dataTableTemplate: getGPURecommendationTableTemplate(), - tableDivId: '#gpu-recommendation-table', - } - ); - - definedDataTables[gpuRecommendationTableID] = gpuRecommendationTable; - - // Add event listener for opening and closing details - $('#gpu-recommendation-table tbody').on('click', 'td.dt-control', function () { - let tr = $(this).closest('tr'); - let row = gpuRecommendationTable.row( tr ); - - if ( row.child.isShown() ) { - // This row is already open - close it - row.child.hide(); - tr.removeClass('shown'); - } - else { - // Open this row - row.child( formatAppGPURecommendation(row.data()) ).show(); - tr.addClass('shown'); - } - }); - - // Handle click on "Expand All" button - $('#btn-show-all-children').on('click', function() { - expandAllGpuRows(gpuRecommendationTable); - }); - - // Handle click on "Collapse All" button - $('#btn-hide-all-children').on('click', function() { - collapseAllGpuRows(gpuRecommendationTable); - }); - - // set the template of the report qualReportSummary - let text = Mustache.render(getGlobalStatisticsTemplate(), qualReportSummary); - $("#qual-report-summary").html(jQuery.parseHTML(text, false)); - - setupToolTipForTableCells(); - setupNavigation(); -}); diff --git a/core/src/main/resources/ui/js/raw-report.js b/core/src/main/resources/ui/js/raw-report.js deleted file mode 100644 index 2973ec6d2..000000000 --- a/core/src/main/resources/ui/js/raw-report.js +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* global $, Mustache, formatDuration, jQuery, qualificationRecords */ - -$(document).ready(function() { - let attemptArray = processRawData(qualificationRecords); - // - // set the app details table - // - let appDetailsDataTable = constructDataTableFromHTMLTemplate( - attemptArray, - "listAppsView", - createAppDetailedTableConf, - { - tableId: "appDetails", - dataTableTemplate: getAppDetailsTableTemplate(), - datatableContainerID: '#app-details-data-container', - tableDivId: '#all-apps-raw-data-table', - } - ); - - setupToolTipForTableCells(); - setupNavigation(); -}); diff --git a/core/src/main/resources/ui/js/ui-config.js b/core/src/main/resources/ui/js/ui-config.js deleted file mode 100644 index 0f5c9cc85..000000000 --- a/core/src/main/resources/ui/js/ui-config.js +++ /dev/null @@ -1,556 +0,0 @@ -/* - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -let qualReportSummary = { - "config": { - "showTLCSummary": false - }, - "totalApps": { - "numeric": 0, - "header": "Total Applications", - "statsPercentage": "%", - "statsTimeFrame": "Apps with Estimated End Time", - "totalAppsDurations": "0 ms", - "totalAppsDurationLabel": "Total Run Durations", - }, - "candidates": { - "numeric": 0, - "header": "RAPIDS Candidates", - "statsPercentage": "%", - "statsTimeFrame": "Fit for GPU acceleration", - }, - "speedups": { - "numeric": "N/A", - "header": "GPU Opportunity", - "statsPercentage": "%", - "statsTimeFrame": "Supported SQL DF Durations", - "totalSqlDataframeTaskDuration" : "0 ms", - "totalSqlDFDurationsLabel" : "Total SqlDF Durations", - }, - "tlc": { - "numeric": 0, - "header": "Apps that need TLC", - "statsPercentage": "% Needs more information", - "statsTimeFrame": "We found apps with potential problems", - }, -}; - -let toolTipsValues = { - "gpuRecommendations": { - "App Name": "Name of the application", - "App ID": "An application is referenced by its application ID, \app-id\<\/em\>. " + - "\ When running on YARN, each application may have multiple attempts, but there are " + - "attempt IDs only for applications in cluster mode, not applications in client mode. " + - "Applications in YARN cluster mode can be identified by their \attempt-id\<\/em\>.", - "App Duration": "Wall-Clock time measured since the application starts till it is completed. " + - "If an app is not completed an estimated completion time would be computed.", - "GPU Opportunity": "Wall-Clock time that shows how much of the SQL duration and ML functions(if applicable) can be accelerated on the GPU.", - "Recommendation": "Recommendation based on \Estimated Speed-up Factor\<\/em\>.", - "Estimated Speed-up": "Speed-up factor estimated for the app. Calculated as the ratio between \App Duration\<\/em\> and \Estimated GPU Duration\<\/em\>", - "details": { - // Chrome does not support MathML. For simple mathematical formulas, use HTML tags as workaround. - // "totalSpeedup": (App DurationGPU Estimated Duration)", - "totalSpeedup": - "Speed-up factor estimated for the app. Calculated as (App Duration⁄Estimated GPU Duration)", - "nonSqlTaskDurationAndOverhead": "total duration of the app not involving SQL", - "estimatedDuration": "Predicted runtime of the app if it was run on GPU", - "unsupportedDuration": "An estimate total duration of SQL operations that are not supported on GPU", - "sqlDFDuration": "Time duration that includes only SQL-Dataframe queries.", - "gpuOpportunity": "Wall-Clock time that shows how much of the SQL duration and ML functions(if applicable) can be accelerated on the GPU.", - "gpuTimeSaved": "Estimated Wall-Clock time saved if it was run on the GPU." - } - }, - "rawTable": { - "App Name": "Name of the application", - "App ID": "An application is referenced by its application ID, \app-id\<\/em\>. " + - "\ When running on YARN, each application may have multiple attempts, but there are " + - "attempt IDs only for applications in cluster mode, not applications in client mode. " + - "Applications in YARN cluster mode can be identified by their \attempt-id\<\/em\>.", - "App Duration": "Wall-Clock time measured since the application starts till it is completed. " + - "If an app is not completed an estimated completion time would be computed.", - "Estimated GPU Duration": "Predicted runtime of the app if it was run on GPU", - "SQL DF Duration": "Wall-Clock time duration that includes only SQL-Dataframe queries.", - "SQL Dataframe Task Duration": "Sum of the task time that includes parallel SQL-Dataframe queries.", - "Executor CPU Time Percent": - "This is an estimate at how much time the tasks spent doing processing on the CPU vs waiting on IO. Shaded red when it is below 40%", - "Unsupported Task Duration": "Sum of task durations for any unsupported operators.", - "GPU Opportunity": "Wall-Clock time that shows how much of the SQL duration can be accelerated on the GPU.", - "Estimated GPU Speedup": - "Speed-up factor estimated for the app. Calculated as (App Duration⁄GPU Estimated Duration)", - "NONSQL Task Duration Plus Overhead": "Time duration that does not span any running SQL task.", - "Unsupported Read File Formats and Types": "Looks at the Read Schema and reports the file formats along with types " + - "which may not be fully supported. Example: \Parquet[decimal], JDBC[*]\<\/em\>. Note that this is based on the current " + - "version of the plugin and future versions may add support for more file formats and types.", - "Unsupported Write Data Format": "Reports the data format which we currently don’t support, i.e. if the result " + - "is written in JSON or CSV format.", - "Recommendation": "Recommendation based on \Estimated Speed-up Factor\<\/em\>.", - "Estimated GPU Time Saved": "Estimated Wall-Clock time saved if it was run on the GPU", - "Supported SQL DF Task Duration": "Sum of task durations that are supported by RAPIDS GPU acceleration.", - "App Duration Estimated": "Flag set to true when the application end time was \estimated\<\/em\> based on the application progress", - "Task Speed-up Factor": "The average speed-up of all stages." - }, - "appStages": { - "Average Speedup Factor": "The average estimated speed-up of all the operators in the given stage", - "Stage Task Duration": "Amount of time spent in tasks of SQL Dataframe operations for the given stage.", - "Unsupported Task Duration": "Sum of task durations for the unsupported operators.", - "Stage Estimated": "True or False indicates if we had to estimate the stage duration.", - }, - "appExecs": { - "Speed-up Factor": "It is simply the average acceleration of the operators based on the " + - "original CPU duration of the operator divided by the GPU duration. The tool uses historical " + - "queries and benchmarks to estimate a speed-up at an individual operator level to calculate how much a specific " + - "operator would accelerate on GPU.", - "Exec Duration": "Wall-Clock time measured since the operator starts till it is completed.", - "Exec Is Supported": "Whether the Exec is supported by RAPIDS or not.", - "Is Removed": "Whether the Op is removed from the migrated plan." - }, - "appSQLs": { - "GPU Opportunity": "Wall-Clock time that shows how much of the SQL duration can be accelerated on the GPU.", - "Recommendation": "Recommendation based on \Estimated Speed-up Factor\<\/em\>.", - "Estimated GPU Speedup": - "Speed-up factor estimated for the app. Calculated as (SQL Duration⁄GPU Estimated Duration)", - "Estimated GPU Duration": "Predicted runtime of the SQL if it was run on GPU", - "Estimated GPU Time Saved": "Estimated Wall-Clock time saved if it was run on the GPU", - "SQL DF Duration": "Wall-Clock time duration of the query.", - "App ID": "An application is referenced by its application ID, \app-id\<\/em\>. " + - "\ When running on YARN, each application may have multiple attempts, but there are " + - "attempt IDs only for applications in cluster mode, not applications in client mode. " + - "Applications in YARN cluster mode can be identified by their \attempt-id\<\/em\>.", - }, -} - -let UIConfig = { - "dataProcessing": { - // name of the column used to decide on the category of the app - // total SpeedUp is a factor between 1.0 and 10.0 - "gpuRecommendation.appColumn": "estimatedInfo.estimatedGpuSpeedup", - // when set to true, the JS will generate random value for recommendations - "simulateRecommendation": false - }, - "datatables": { - "gpuRecommendations": { - "listAppsView": { - Dom : 'frtlip', - skipColumns: [], - sortTable: true, - sortColumns: [ - {colName: "recommendation", order: "desc"}, - {colName: "totalSpeedup", order:"desc"} - ], - hideColumns: [], - searchableColumns: ["appName", "appId", "recommendation"], - buttons: { - enabled: true, - buttons: [ - { - extend: 'csv', - title: 'rapids_4_spark_qualification_output_ui_apps_recommendations', - text: 'Export' - } - ], - }, - enabledPanes: ["recommendation", "users"], - toolTipID: "gpuRecommendations", - }, - "searchPanes": { - enabled: true, - "dtConfigurations": { - initCollapsed: true, - viewTotal: true, - // Note that there is a bug in cascading that breaks paging of the table - cascadePanes: true, - show: false, - }, - "panes": { - "recommendation": { - header: "Recommendations", - dtOpts: { - searching: true, - }, - order: [[0, 'desc']], - combiner: 'and', - }, - "users":{ - header: "Spark User", - dtOpts: { - searching: true, - }, - combiner: 'and', - } - } - }, - "Dom" : { - default: 'frtlip', - }, - }, - "appDetails": { - "colEnabledPrefix": "displayCol_", - "singleAppView": { // This is a single record. No need to show table information or search - Dom : 'Brt', - skipColumns: ["appName"], - sortTable: false, - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_raw_data_app', - hideColumns: [ - "appName", "sparkUser", "startTime", "longestSqlDuration", - "nonSqlTaskDurationAndOverhead", "endDurationEstimated", - "failedSQLIds", "potentialProblems", "readFileFormatAndTypesNotSupported", - "writeDataFormat", "complexTypes", "nestedComplexTypes", "readFileFormats" - ], - toolTipID: "rawTable" - }, - "listAppsView": { - Dom : 'Bfrtlip', - skipColumns: [], - sortTable: true, - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_raw_data', - sortColumns: [ - {colName: "gpuRecommendation", order: "desc"}, - {colName: "totalSpeedupFactor", order:"desc"} - ], - searchableColumns: [ - "appName", "appId", "sparkUser", "gpuRecommendation", "readFileFormats", - "nestedComplexTypes", "complexTypes", "readFileFormatAndTypesNotSupported", - "writeDataFormat", "potentialProblems" - ], - hideColumns: [ - "appName", "sparkUser", "startTime", "longestSqlDuration", - "nonSqlTaskDurationAndOverhead", "endDurationEstimated", - "failedSQLIds", "potentialProblems", "readFileFormatAndTypesNotSupported", - "writeDataFormat", "complexTypes", "nestedComplexTypes", "readFileFormats" - ], - toolTipID: "rawTable", - } - }, - "appStages": { - "colEnabledPrefix": "displayCol_", - "singleAppView": { // This is a single record. No need to filter because there is no text data - Dom : 'Brtlip', - skipColumns: ["appID"], - sortTable: true, - sortColumns: [{colName: "stageId", order:"asc"}], - searchableColumns: [], - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_stages_data_app', - hideColumns: [], - enabledPanes: ["stageEstimated", "speedupFactor", "taskTypes"], - toolTipID: "appStages", - }, - "listAppsView": { - Dom : 'Bfrtlip', - skipColumns: [], - sortTable: true, - sortColumns: [{colName: "stageId", order:"asc"}], - searchableColumns: [], - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_stages_data', - hideColumns: [], - enabledPanes: ["stageEstimated", "speedupFactor", "taskTypes"], - toolTipID: "appStages", - }, - "searchPanes": { - enabled: true, - "dtConfigurations": { - initCollapsed: true, - viewTotal: true, - // Note that there is a bug in cascading that breaks paging of the table - cascadePanes: true, - show: false, - }, - "panes": { - "stageEstimated": { - header: "Is Stage Estimated", - dtOpts: { - searching: false, - }, - combiner: 'and', - options: [ - { - label: 'Estimated', - value: function(rowData, rowIdx) { - return rowData["estimated"]; - } - }, - { - label: 'Not Estimated', - value: function(rowData, rowIdx) { - return !rowData["estimated"]; - } - } - ] - }, - "taskTypes": { - header: "Tasks GPU Support", - dtOpts: { - searching: false, - }, - combiner: 'and', - options: [ - { - label: 'Fully Supported', - value: function(rowData, rowIdx) { - return rowData["unsupportedTaskDur"] <= 0.0; - } - }, - { - label: 'Partially Supported', - value: function(rowData, rowIdx) { - return rowData["unsupportedTaskDur"] > 0; - } - } - ] - }, - "speedupFactor":{ - header: "Speed-up", - dtOpts: { - searching: false, - }, - combiner: 'and', - options: [ - { - label: '1.0 (No Speed-up)', - value: function(rowData, rowIdx) { - return rowData["averageSpeedup"] <= 1.0001; - } - }, - { - label: '1.0 to 1.3', - value: function(rowData, rowIdx) { - return rowData["averageSpeedup"] > 1.00 && rowData["averageSpeedup"] < 1.3; - } - }, - { - label: '1.3 to 2.5', - value: function(rowData, rowIdx) { - return rowData["averageSpeedup"] >= 1.3 && rowData["averageSpeedup"] < 2.5; - } - }, - { - label: '2.5 to 5', - value: function(rowData, rowIdx) { - return rowData["averageSpeedup"] >= 2.5 && rowData["averageSpeedup"] < 5; - } - }, - { - label: '5 or More', - value: function(rowData, rowIdx) { - return rowData["averageSpeedup"] >= 5; - } - }, - ], - }, - } - } - }, - "appExecs": { - "colEnabledPrefix": "displayCol_", - "singleAppView": { - Dom : 'Bfrtlip', - skipColumns: ["appID"], - sortTable: true, - sortColumns: [{colName: "sqlID", order:"asc"}], - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_execs_data_app', - searchableColumns: ["exec", "expr"], - hideColumns: ["appID", "isRemoved"], - enabledPanes: ["execName", "speedupFactor", "execSupport", "stages", "shouldRemove"], - toolTipID: "appExecs", - }, - "listAppsView": { - Dom : 'Bfrtlip', - skipColumns: [], - sortTable: true, - sortColumns: [ - {colName: "appID", order: "asc"}, - {colName: "sqlID", order:"asc"} - ], - searchableColumns: ["appID", "exec", "expr"], - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_execs_data', - hideColumns: ["appID", "isRemoved"], - enabledPanes: ["execName", "speedupFactor", "execSupport", "shouldRemove"], - toolTipID: "appExecs", - }, - "searchPanes": { - enabled: true, - "dtConfigurations": { - initCollapsed: true, - viewTotal: true, - // Note that there is a bug in cascading that breaks paging of the table - cascadePanes: true, - show: false, - }, - "panes": { - "execSupport": { - header: "GPU Support", - dtOpts: { - "searching": false - }, - combiner: 'and', - options: [ - { - label: 'Yes', - value: function(rowData, rowIdx) { - return rowData["isSupported"]; - } - }, - { - label: 'No', - value: function(rowData, rowIdx) { - return !rowData["isSupported"]; - } - } - ] - }, - "shouldRemove": { - header: "Is Exec Removed", - dtOpts: { - "searching": false - }, - combiner: 'and', - options: [ - { - label: 'Yes', - value: function(rowData, rowIdx) { - return rowData["shouldRemove"]; - } - }, - { - label: 'No', - value: function(rowData, rowIdx) { - return !rowData["shouldRemove"]; - } - } - ] - }, - "execName": { - "header": "Exec", - dtOpts: { - "searching": true - }, - combiner: 'and', - }, - "speedupFactor": { - header: "Speed-up", - dtOpts: { - "searching": false - }, - combiner: 'and', - options: [ - { - label: '1.0 (No Speed-up)', - value: function(rowData, rowIdx) { - return rowData["speedupFactor"] <= 1.0001; - } - }, - { - label: '1.0 to 1.3', - value: function(rowData, rowIdx) { - return rowData["speedupFactor"] > 1.00 && rowData["speedupFactor"] < 1.3; - } - }, - { - label: '1.3 to 2.5', - value: function(rowData, rowIdx) { - return rowData["speedupFactor"] >= 1.3 && rowData["speedupFactor"] < 2.5; - } - }, - { - label: '2.5 to 5', - value: function(rowData, rowIdx) { - return rowData["speedupFactor"] >= 2.5 && rowData["speedupFactor"] < 5; - } - }, - { - label: '5 or More', - value: function(rowData, rowIdx) { - return rowData["speedupFactor"] >= 5; - } - }, - ], - }, - "stages": { - header: "By Stage ID", - dtOpts: { - "searching": true - }, - combiner: 'and', - } - } - } - }, - "appSQLs": { - "colEnabledPrefix": "displayCol_", - "singleAppView": { - Dom : 'Bfrtlip', - skipColumns: ["appID"], - sortTable: true, - sortColumns: [ - {colName: "gpuRecommendation", order: "desc"}, - {colName: "estimatedGpuSpeedup", order:"desc"}, - {colName: "sqlID", order:"asc"} - ], - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_sqls_data_app', - searchableColumns: ["sqlDescription", "gpuRecommendation"], - hideColumns: [], - enabledPanes: ["recommendation"], - toolTipID: "appSQLs", - }, - "listAppsView": { - Dom : 'Bfrtlip', - skipColumns: [], - sortTable: true, - sortColumns: [ - {colName: "gpuRecommendation", order: "desc"}, - {colName: "estimatedGpuSpeedup", order:"desc"}, - {colName: "appID", order:"asc"}, - {colName: "sqlID", order:"asc"} - ], - searchableColumns: ["sqlDescription", "gpuRecommendation", "appID"], - fileExportPrefix: 'rapids_4_spark_qualification_output_ui_sqls_data', - hideColumns: [], - enabledPanes: ["recommendation", "apps"], - toolTipID: "appSQLs", - }, - "searchPanes": { - enabled: true, - "dtConfigurations": { - initCollapsed: true, - viewTotal: true, - // Note that there is a bug in cascading that breaks paging of the table - cascadePanes: true, - show: false, - }, - "panes": { - "recommendation": { - header: "By Recommendation", - dtOpts: { - searching: true, - }, - order: [[0, 'desc']], - combiner: 'and', - }, - "apps": { - header: "By App ID", - dtOpts: { - "searching": true - }, - combiner: 'and', - } - }, - - } - }, - }, - "fullAppView": { - enabled: true - } -}; diff --git a/core/src/main/resources/ui/js/uiutils.js b/core/src/main/resources/ui/js/uiutils.js deleted file mode 100644 index c69748a6f..000000000 --- a/core/src/main/resources/ui/js/uiutils.js +++ /dev/null @@ -1,1531 +0,0 @@ -/* - * Copyright (c) 2022, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* globals $, Mustache, jQuery, qualReportSummary */ - -class Queue { - constructor() { - this.items = []; - } - enqueue(element) { - // adding element to the queue - this.items.push(element); - } - dequeue() { - if(this.isEmpty()) - return "Underflow"; - return this.items.shift(); - } - peak() { - if(this.isEmpty()) - return "No elements in Queue"; - return this.items[0]; - } - isEmpty() { - return this.items.length == 0; - } -} - -class AppsToStagesMap { - constructor() { - this.appsHashMap = new Map(); - } - addAppRec(appRecord) { - if (this.appsHashMap.has(appRecord.appId)) { - // app already exists - return this.appsHashMap.get(appRecord.appId); - } - let appStages = new Set(); - if (Array.isArray(appRecord.stageInfo)) { - appRecord.stageInfo.forEach(stageInfoRec => { - appStages.add(stageInfoRec.stageId); - }); - } - appStages.add("N/A"); - this.appsHashMap.set(appRecord.appId, appStages) - return appRecord.stageInfo; - } - getAllStages(appRecordId) { - return this.appsHashMap.get(appRecordId); - } -} - -let appStagesMap = new AppsToStagesMap(); - -function padZeroes(num) { - return ("0" + num).slice(-2); -} - -/* eslint-disable no-unused-vars */ -function formatTimeMillis(timeMillis) { - if (timeMillis <= 0) { - return "-"; - } else { - let dt = new Date(timeMillis); - return formatDateString(dt); - } -} - -/* eslint-enable no-unused-vars */ - -function formatDateString(dt) { - return dt.getFullYear() + "-" + - padZeroes(dt.getMonth() + 1) + "-" + - padZeroes(dt.getDate()) + " " + - padZeroes(dt.getHours()) + ":" + - padZeroes(dt.getMinutes()) + ":" + - padZeroes(dt.getSeconds()); -} - -function formatDuration(milliseconds) { - if (milliseconds < 100) { - return parseInt(milliseconds).toFixed(1) + " ms"; - } - let seconds = milliseconds * 1.0 / 1000; - if (seconds < 1) { - return seconds.toFixed(2) + " s"; - } - if (seconds < 60) { - return seconds.toFixed(1) + " s"; - } - let minutes = seconds / 60; - if (minutes < 10) { - return minutes.toFixed(2) + " min"; - } else if (minutes < 60) { - return minutes.toFixed(1) + " min"; - } - let hours = minutes / 60; - return hours.toFixed(2) + " h"; -} - -function formatFloatingPoints(numArg) { - return Math.floor(parseFloat(numArg) * 100) / 100; -} - -function getColumnIndex(columns, columnName) { - for (let i = 0; i < columns.length; i++) { - if (columns[i].name === columnName) - return i; - } - return -1; -} - -function removeColumnByName(columns, columnName) { - return columns.filter(function(col) {return col.name != columnName}) -} - -/** calculations of CPU Processor **/ - -let CPUPercentThreshold = 40.0; - -function totalCPUPercentageStyle(cpuPercent) { - // Red if GC time over GCTimePercent of total time - return (cpuPercent < CPUPercentThreshold) ? - ("hsl(0, 100%, 50%, " + totalCPUPercentageAlpha(CPUPercentThreshold - cpuPercent) + ")") : ""; -} - -function totalCPUPercentageAlpha(actualCPUPercentage) { - return actualCPUPercentage >= 0 ? - (Math.min(actualCPUPercentage / 40.0 + 0.4, 1)) : 1; -} - -function totalCPUPercentageColor(cpuPercent) { - return (cpuPercent < CPUPercentThreshold) ? "white" : "black"; -} - -function escapeHtml(unsafe) { - return unsafe - .replaceAll(/&/g, "&") - .replaceAll(//g, ">") - .replaceAll(/"/g, """) - .replaceAll(/'/g, "'"); -} - -function insertSpacePostCommas(arrData) { - if (arrData.length > 0) { - return arrData.replaceAll(',', ', ') - } - return ''; -} - -class GpuRecommendationCategory { - constructor(id, relRate, printName, descr, displayClass, initCollapsed = false) { - this.id = id; - this.displayName = printName; - this.collapsed = initCollapsed; - this.description = descr; - this.rate = relRate; - this.badgeDisplay = displayClass; - } - - // Method - isGroupOf(row) { - return row.recommendation === this.displayName; - } - - toggleCollapsed() { - this.collapsed = !this.collapsed; - } - - getBadgeDisplay(row) { - return this.badgeDisplay; - } -} - -let recommendationContainer = [ - new GpuRecommendationCategory("A", 5, - "Strongly Recommended", - "Spark Rapids is expected to speedup the App", - "badge badge-pill badge-strong-recommended"), - new GpuRecommendationCategory("B", 4, - "Recommended", - "Using Spark RAPIDS expected to give a moderate speedup.", - "badge badge-pill badge-recommended"), - new GpuRecommendationCategory("C", 3, - "Not Recommended", - "[Not-Recommended]: It is not likely that GPU Acceleration will be tangible", - "badge badge-pill badge-not-recommended"), - new GpuRecommendationCategory("D", 2, - "Not Applicable", - "[Not-Applicable]: The application has job or stage failures.", - "badge badge-pill badge-not-applicable") -]; - -function createRecommendationGroups(recommendationsArr) { - let map = new Map() - recommendationsArr.forEach(object => { - map.set(object.displayName, object); - }); - return map; -} - -let recommendationsMap = new Map(createRecommendationGroups(recommendationContainer)); - -let sparkUsers = new Map(); -let execNames = new Map(); - -/* define constants for the tables configurations */ -let defaultPageLength = 20; -let defaultLengthMenu = [[20, 40, 60, 100, -1], [20, 40, 60, 100, "All"]]; - -// bind the raw data top the GPU recommendations -function setGPURecommendations(appsArray) { - for (let i in appsArray) { - let appCategory = recommendationContainer.find(grp => grp.isGroupOf(appsArray[i])) - appsArray[i]["gpuCategory"] = appCategory.displayName; - } -} - -function setAppInfoRecord(appRecord) { - // set default values - sparkUsers.set(appRecord["user"], true); -} - -function getAppBadgeHTMLWrapper(appRecord) { - let recommendGroup = recommendationsMap.get(appRecord.gpuCategory); - return `` + recommendGroup.displayName + ``; -} - -function setAppTaskDuration(appRec) { - // appTaskDuration = nonSql + sqlTask Durations - appRec["appTaskDuration"] = - appRec["sqlDataframeTaskDuration"] - + appRec["nonSqlTaskDurationAndOverhead"] -} - -// Quick workaround to map names generated by scala to the name in UI -function mapFieldsToUI(rawAppRecord) { - rawAppRecord["cpuPercent"] = rawAppRecord["executorCPUPercent"]; - // set default longestSqlDuration for backward compatibility - if (!rawAppRecord.hasOwnProperty("longestSqlDuration")) { - rawAppRecord["longestSqlDuration"] = 0; - } - rawAppRecord["recommendation"] = rawAppRecord.estimatedInfo.recommendation; - rawAppRecord["estimatedGPUDuration"] = parseFloat(rawAppRecord.estimatedInfo.estimatedGpuDur); - rawAppRecord["totalSpeedup"] = rawAppRecord.estimatedInfo.estimatedGpuSpeedup; - rawAppRecord["appDuration"] = rawAppRecord.estimatedInfo.appDur; - rawAppRecord["sqlDataFrameDuration"] = rawAppRecord.estimatedInfo.sqlDfDuration; - rawAppRecord["gpuTimeSaved"] = parseFloat(rawAppRecord.estimatedInfo.estimatedGpuTimeSaved); - rawAppRecord["gpuOpportunity"] = rawAppRecord.estimatedInfo.gpuOpportunity; - // escape html characters for data formatted fields - rawAppRecord["readFileFormats_html_safe"] = - rawAppRecord.readFileFormats.map(elem => { - return escapeHtml(elem); - }); - rawAppRecord["readFileFormatAndTypesNotSupported_html_safe"] = - rawAppRecord.readFileFormatAndTypesNotSupported.map(elem => { - return escapeHtml(elem); - }); - rawAppRecord["writeDataFormat_html_safe"] = - rawAppRecord.writeDataFormat.map(elem => { - return escapeHtml(elem); - }); - rawAppRecord["complexTypes_html_safe"] = - rawAppRecord.complexTypes.map(elem => { - return escapeHtml(elem); - }); - rawAppRecord["nestedComplexTypes_html_safe"] = - rawAppRecord.nestedComplexTypes.map(elem => { - return escapeHtml(elem); - }); - rawAppRecord["writeDataFormat_html_safe"] = - rawAppRecord.writeDataFormat.map(elem => { - return escapeHtml(elem).toUpperCase(); - }); - rawAppRecord["potentialProblems_html_safe"] = - rawAppRecord.potentialProblems.map(elem => { - return escapeHtml(elem).toUpperCase(); - }); -} - -function processRawData(rawRecords) { - let processedRecords = []; - let maxOpportunity = 0; - for (let i in rawRecords) { - let appRecord = JSON.parse(JSON.stringify(rawRecords[i])); - mapFieldsToUI(appRecord) - - // Set duration fields - appRecord["durationCollection"] = { - "appDuration": formatDuration(appRecord["appDuration"]), - "sqlDFDuration": formatDuration(appRecord["sqlDataFrameDuration"]), - "sqlDFTaskDuration": formatDuration(appRecord["sqlDataframeTaskDuration"]), - "sqlDurationProblems": formatDuration(appRecord["sqlDurationForProblematic"]), - "supportedSQLTaskDuration": formatDuration(appRecord["supportedSQLTaskDuration"]), - "nonSqlTaskDurationAndOverhead": formatDuration(appRecord["nonSqlTaskDurationAndOverhead"]), - "estimatedGPUDuration": formatDuration(appRecord["estimatedGPUDuration"]), - "estimatedDurationWallClock": - formatDuration(appRecord.estimatedGPUDuration), - "gpuOpportunity": formatDuration(appRecord.gpuOpportunity), - "unsupportedSQLTaskDuration": formatDuration(appRecord["unsupportedSQLTaskDuration"]), - "longestSqlDuration": formatDuration(appRecord["longestSqlDuration"]), - "gpuTimeSaved": formatDuration(appRecord.gpuTimeSaved), - } - - // Set numeric fields for display - appRecord["totalSpeedup_display"] = formatFloatingPoints(appRecord["totalSpeedup"]); - appRecord["taskSpeedupFactor_display"] = formatFloatingPoints(appRecord["taskSpeedupFactor"]); - - setAppInfoRecord(appRecord); - maxOpportunity = - (maxOpportunity < appRecord["gpuRecommendation"]) - ? appRecord["gpuRecommendation"] : maxOpportunity; - if (UIConfig.fullAppView.enabled) { - appRecord["attemptDetailsURL"] = "application.html?app_id=" + appRecord.appId; - } else { - appRecord["attemptDetailsURL"] = "#!" - } - - setAppTaskDuration(appRecord); - - processedRecords.push(appRecord) - } - setGPURecommendations(processedRecords); - setGlobalReportSummary(processedRecords); - return processedRecords; -} - -// get the sqlInfo items from an apprecord -function getAppSqlArray(appRecord) { - let sqlInfos = []; - if (appRecord.hasOwnProperty('perSQLEstimatedInfo')) { - for (let ind in appRecord.perSQLEstimatedInfo) { - let sqlInfo = appRecord.perSQLEstimatedInfo[ind]; - sqlInfo["recommendation"] = sqlInfo.info.recommendation; - if (UIConfig.fullAppView.enabled) { - sqlInfo["attemptDetailsURL"] = "application.html?app_id=" + sqlInfo.info.appId; - } else { - sqlInfo["attemptDetailsURL"] = "#!" - } - sqlInfos.push(sqlInfo); - } - setGPURecommendations(sqlInfos); - } - - return sqlInfos; -} - -function setGlobalReportSummary(processedApps) { - let totalEstimatedApps = 0; - let recommendedCnt = 0; - let tlcCount = 0; - let totalDurations = 0; - let totalSqlDataframeDuration = 0; - // only count apps that are recommended - let totalGPUOpportunityDurations = 0; - for (let i in processedApps) { - // check if completedTime is estimated - if (processedApps[i]["endDurationEstimated"]) { - totalEstimatedApps += 1; - } - totalDurations += processedApps[i].appDuration; - totalSqlDataframeDuration += processedApps[i].sqlDataFrameDuration; - // check if the app is recommended or needs more information - let recommendedGroup = recommendationsMap.get(processedApps[i]["gpuCategory"]) - if (recommendedGroup.id < "C") { - // this is a recommended app - // aggregate for GPU recommendation box - recommendedCnt += 1; - totalGPUOpportunityDurations += processedApps[i]["gpuOpportunity"] - } else { - if (recommendedGroup.id === "D") { - tlcCount += 1; - } - } - } - - let estimatedPercentage = 0.0; - let gpuPercent = 0.0; - let tlcPercent = 0.0; - let speedUpPercent = 0.0; - - if (processedApps.length != 0) { - // calculate percentage of estimatedEndTime; - estimatedPercentage = (totalEstimatedApps * 100.0) / processedApps.length; - // calculate percentage of recommended GPUs - gpuPercent = (100.0 * recommendedCnt) / processedApps.length; - // percent of apps missing information - tlcPercent = (100.0 * tlcCount) / processedApps.length; - speedUpPercent = (100.0 * totalGPUOpportunityDurations) / totalSqlDataframeDuration; - } - qualReportSummary.totalApps.numeric = processedApps.length; - qualReportSummary.totalApps.totalAppsDurations = formatDuration(totalDurations); - // speedups - qualReportSummary.speedups.numeric = - formatDuration(totalGPUOpportunityDurations); - qualReportSummary.speedups.totalSqlDataframeTaskDuration = - formatDuration(totalSqlDataframeDuration); - qualReportSummary.speedups.statsPercentage = formatFloatingPoints(speedUpPercent) - + qualReportSummary.speedups.statsPercentage; - - // candidates - qualReportSummary.candidates.numeric = recommendedCnt; - qualReportSummary.tlc.numeric = tlcCount; - qualReportSummary.totalApps.statsPercentage = - formatFloatingPoints(estimatedPercentage) - + qualReportSummary.totalApps.statsPercentage; - qualReportSummary.candidates.statsPercentage = - formatFloatingPoints(gpuPercent) - + qualReportSummary.candidates.statsPercentage; - qualReportSummary.tlc.statsPercentage = - formatFloatingPoints(tlcPercent) - + qualReportSummary.tlc.statsPercentage; -} - -function createAppIDLinkEnabled(tableViewType) { - return UIConfig.fullAppView.enabled && tableViewType === "listAppsView" -} - -function createAppDetailedTableConf( - appRecords, - tableViewType, - mustacheRecord, - extraFunctionArgs) { - let totalSpeedupColumnName = "totalSpeedupFactor" - let recommendGPUColName = "gpuRecommendation" - let appDetailsBaseParams = UIConfig.datatables[extraFunctionArgs.tableId]; - let appDetailsCustomParams = appDetailsBaseParams[tableViewType]; - let fileExportName = appDetailsCustomParams.fileExportPrefix; - if (tableViewType === 'singleAppView') { - fileExportName = appDetailsCustomParams.fileExportPrefix + "_" + extraFunctionArgs.appId; - } - let rawDataTableConf = { - paging: (appRecords.length > defaultPageLength), - pageLength: defaultPageLength, - lengthMenu: defaultLengthMenu, - info: true, - data: appRecords, - columns: [ - { - name: "appName", - data: "appName", - }, - { - name: "appId", - data: "appId", - className: "all", - render: (appId, type, row) => { - if (type === 'display') { - if (createAppIDLinkEnabled(tableViewType)) { - return `${appId}` - } - } - return appId; - } - }, - { - name: "sparkUser", - data: "user", - }, - { - name: "startTime", - data: "startTime", - type: 'numeric', - render: function (data, type, row) { - if (type === 'display') { - return formatTimeMillis(data) - } - return data; - }, - }, - { - name: recommendGPUColName, - data: 'gpuCategory', - className: "all", - render: function (data, type, row) { - if (type === 'display') { - return getAppBadgeHTMLWrapper(row); - } - return data; - }, - fnCreatedCell: (nTd, sData, oData, _ignored_iRow, _ignored_iCol) => { - let recommendGroup = recommendationsMap.get(sData); - let toolTipVal = recommendGroup.description; - $(nTd).attr('data-toggle', "tooltip"); - $(nTd).attr('data-placement', "top"); - $(nTd).attr('html', "true"); - $(nTd).attr('data-html', "true"); - $(nTd).attr('title', toolTipVal); - } - }, - { - name: totalSpeedupColumnName, - data: "totalSpeedup", - type: 'numeric', - className: "all", - render: function (data, type, row) { - if (type === 'display') { - return row.totalSpeedup_display - } - return data; - }, - }, - { - name: 'appDuration', - data: 'appDuration', - type: 'numeric', - render: function (data, type, row) { - if (type === 'display' || type === 'filter') { - return formatDuration(data) - } - return data; - }, - fnCreatedCell: (nTd, sData, oData, _ignored_iRow, _ignored_iCol) => { - if (oData.endDurationEstimated) { - $(nTd).css('color', 'blue'); - } - } - }, - { - name: "estimatedGPUDuration", - data: "estimatedGPUDuration", - type: 'numeric', - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.estimatedDurationWallClock; - } - return data; - }, - }, - { - name: "gpuTimeSaved", - data: "gpuTimeSaved", - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.gpuTimeSaved - } - return data; - }, - }, - { - name: "taskSpeedupFactor", - data: "taskSpeedupFactor", - render: function (data, type, row) { - if (type === 'display') { - return row.taskSpeedupFactor_display; - } - return data; - }, - }, - { - name: 'sqlDataFrameDuration', - data: 'sqlDataFrameDuration', - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.sqlDFDuration - } - return data; - }, - }, - { - name: 'gpuOpportunity', - data: "gpuOpportunity", - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.gpuOpportunity - } - return data; - }, - }, - { - name: 'unsupportedSQLTaskDuration', - data: "unsupportedSQLTaskDuration", - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.unsupportedSQLTaskDuration - } - return data; - }, - }, - { - name: 'supportedSQLTaskDuration', - data: "supportedSQLTaskDuration", - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.supportedSQLTaskDuration - } - return data; - }, - }, - { - name: 'sqlDataframeTaskDuration', - data: 'sqlDataframeTaskDuration', - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.sqlDFTaskDuration - } - return data; - }, - }, - { - name: "executorCpuTimePercent", - data: "executorCpuTimePercent", - fnCreatedCell: (nTd, sData, oData, _ignored_iRow, _ignored_iCol) => { - if (oData.executorCpuTimePercent >= 0) { - $(nTd).css('color', totalCPUPercentageColor(oData.executorCpuTimePercent)); - $(nTd).css('background', totalCPUPercentageStyle(oData.executorCpuTimePercent)); - } - } - }, - { - name: "longestSqlDuration", - data: "longestSqlDuration", - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.longestSqlDuration - } - return data; - }, - }, - { - name: "nonSqlTaskDurationAndOverhead", - data: "nonSqlTaskDurationAndOverhead", - render: function (data, type, row) { - if (type === 'display') { - return row.durationCollection.nonSqlTaskDurationAndOverhead - } - return data; - }, - }, - { - data: "endDurationEstimated", - name: "endDurationEstimated", - orderable: false, - }, - { - name: "failedSQLIds", - data: "failedSQLIds[, ]", - }, - { - name: "potentialProblems", - data: "potentialProblems_html_safe[
]", - render: function (data, type, row) { - if (type === 'display') { - return insertSpacePostCommas(data); - } - return data; - }, - }, - { - name: "readFileFormatAndTypesNotSupported", - data: "readFileFormatAndTypesNotSupported_html_safe[
]", - render: function (data, type, row) { - if (type === 'display') { - return insertSpacePostCommas(data); - } - return data; - }, - }, - { - data: "writeDataFormat_html_safe[, ]", - name: "writeDataFormat", - orderable: false, - }, - { - data: "complexTypes_html_safe[
]", - name: "complexTypes", - orderable: false, - render: function (data, type, row) { - if (type === 'display') { - return insertSpacePostCommas(data); - } - return data; - }, - }, - { - data: "nestedComplexTypes_html_safe[
]", - name: "nestedComplexTypes", - orderable: false, - render: function (data, type, row) { - if (type === 'display') { - return insertSpacePostCommas(data); - } - return data; - }, - }, - { - data: "readFileFormats_html_safe[
]", - name: "readFileFormats", - orderable: false, - render: function (data, type, row) { - if (type === 'display') { - return insertSpacePostCommas(data); - } - return data; - }, - }, - ], - responsive: { - details: { - renderer: function ( api, rowIdx, columns ) { - let data = $.map( columns, function ( col, i ) { - let dataTableToolTip = toolTipsValues[appDetailsCustomParams.toolTipID]; - if (!col.hidden) { - return ''; - } - let returnStr = ''+ - ''; - if (dataTableToolTip[col.title]) { - returnStr += ''+col.title+':'+ - ''; - } else { - returnStr += col.title; - } - returnStr += ' '+col.data+' '; - return returnStr; - } ).join(''); - - return data ? - $('').append( data ) : - false; - } - } - }, - dom: appDetailsCustomParams.Dom, - buttons: [{ - extend: 'csv', - title: fileExportName, - text: 'Export' - }], - initComplete: function(settings, json) { - // Add custom Tool Tip to the headers of the table - // Add custom Tool Tip to the headers of the table - let thLabel = extraFunctionArgs.tableDivId + ' thead th'; - let dataTableToolTip = toolTipsValues[appDetailsCustomParams.toolTipID]; - $(thLabel).each(function () { - let $td = $(this); - let toolTipVal = dataTableToolTip[$td.text().trim()]; - $td.attr('data-toggle', "tooltip"); - $td.attr('data-placement', "top"); - $td.attr('html', "true"); - $td.attr('data-html', "true"); - $td.attr('title', toolTipVal); - }); - } - }; - - processDatableColumns( - rawDataTableConf, - appDetailsBaseParams, - appDetailsCustomParams, - mustacheRecord) - - return rawDataTableConf; -} - -function extractExecName(rawExecName) { - let parenthInd = rawExecName.indexOf(" ("); - if (parenthInd == -1) { - return rawExecName; - } - return rawExecName.substring(0, parenthInd); -} - -function createSearchPane(allPanesConfigs, confID, optionsGeneratorFunc = null) { - let execNamePaneConf = allPanesConfigs[confID]; - if (optionsGeneratorFunc === null) { - return execNamePaneConf; - } - execNamePaneConf.options = optionsGeneratorFunc(); - return execNamePaneConf; -} - -function processDatableColumns( - dataTableConf, - itemDetailsBaseParams, - itemDetailsCustomParams, - dataTableMustacheRecord) { - // set Columns defaults - for (let i in dataTableConf.columns) { - let propName = itemDetailsBaseParams.colEnabledPrefix + dataTableConf.columns[i].name; - // set all display column to true by default - dataTableMustacheRecord[propName] = true; - // disable searchable for each column - dataTableConf.columns[i].searchable = false; - } - // hide columns with classname set to none - if (itemDetailsCustomParams.hideColumns.length > 0) { - for (let i in itemDetailsCustomParams.hideColumns) { - let colInd = - getColumnIndex(dataTableConf.columns, itemDetailsCustomParams.hideColumns[i]); - dataTableConf.columns[colInd].className = "none"; - } - } - // enable searchable columns - for (let ind in itemDetailsCustomParams.searchableColumns) { - let dtColumnInd = - getColumnIndex(dataTableConf.columns, itemDetailsCustomParams.searchableColumns[ind]) - dataTableConf.columns[dtColumnInd].searchable = true - } - // remove skipped columns - if (itemDetailsCustomParams.skipColumns.length > 0) { - for (let i in itemDetailsCustomParams.skipColumns) { - let propName = itemDetailsBaseParams.colEnabledPrefix + itemDetailsCustomParams.skipColumns[i]; - dataTableConf.columns = - removeColumnByName(dataTableConf.columns, itemDetailsCustomParams.skipColumns[i]); - dataTableMustacheRecord[propName] = false; - } - } - // order columns - if (itemDetailsCustomParams.sortTable) { - dataTableConf.order = []; - for (let ind in itemDetailsCustomParams.sortColumns) { - let dtColumnInd = - getColumnIndex(dataTableConf.columns, itemDetailsCustomParams.sortColumns[ind].colName) - dataTableConf.order.push([dtColumnInd, itemDetailsCustomParams.sortColumns[ind].order]); - } - } -} - -function setDataTableButtons( - dataTableConf, - itemDetailsBaseParams, - itemDetailsCustomParams, - buttonsArgs) { - - // add buttons if enabled in the customView - if (itemDetailsCustomParams.hasOwnProperty('buttons')) { - let buttonsConf = itemDetailsCustomParams['buttons']; - if (buttonsConf["enabled"]) { - dataTableConf["buttons"] = buttonsConf.buttons - dataTableConf.dom = 'B' + dataTableConf.dom - } - return; - } -} - -function setDataTableSearchPanes( - dataTableConf, - itemDetailsBaseParams, - itemDetailsCustomParams, - optionGeneratorsFunctionsMap = new Map()) { - if (itemDetailsBaseParams.hasOwnProperty('searchPanes')) { - let searchPanesConf = itemDetailsBaseParams['searchPanes']; - if (searchPanesConf.enabled) { - // disable searchpanes on default columns - dataTableConf.columnDefs = [{ - "searchPanes": { - show: false, - }, - "targets": ['_all'] - }]; - } - // add custom panes - let enabledPanes = []; - if (itemDetailsCustomParams.hasOwnProperty("enabledPanes")) { - let panesConfigurations = searchPanesConf["panes"]; - itemDetailsCustomParams.enabledPanes.forEach(searchPaneID => { - let optionFunc = null; - if (optionGeneratorsFunctionsMap.has(searchPaneID)) { - optionFunc = optionGeneratorsFunctionsMap.get(searchPaneID); - } - let currentSearchPane = createSearchPane(panesConfigurations, searchPaneID, optionFunc); - enabledPanes.push(currentSearchPane); - }); - } - - if (enabledPanes.length > 0) { - dataTableConf.searchPanes = searchPanesConf["dtConfigurations"]; - // limit the number of filters to 3 by default - if (enabledPanes.length > 3) { - dataTableConf.searchPanes.layout = 'columns-3'; - } - dataTableConf.dom = 'P' + dataTableConf.dom; - dataTableConf.searchPanes.panes = enabledPanes; - } - } -} - -function constructDataTableFromHTMLTemplate( - dataArray, - viewType, - confInitializerFunc, - dataTableArgs = {}) { - if (dataTableArgs.hasOwnProperty('replaceTableIfEmptyData')) { - if (dataTableArgs.replaceTableIfEmptyData.enabled) { - if (dataArray.length == 0) { - $(dataTableArgs.datatableContainerID).html(jQuery.parseHTML( - dataTableArgs.replaceTableIfEmptyData.text, false)); - return null; - } - } - } - let htmlMustacheRec = {}; - let dataTableConf = confInitializerFunc(dataArray, viewType, htmlMustacheRec, dataTableArgs); - let dataTableContainerContent = Mustache.render(dataTableArgs.dataTableTemplate, htmlMustacheRec); - $(dataTableArgs.datatableContainerID).html(jQuery.parseHTML(dataTableContainerContent, false)); - return $(dataTableArgs.tableDivId).DataTable(dataTableConf); -} - -function createAppDetailsExecsTableConf( - execAppRecords, - tableViewType, - mustacheRecord, - extraFunctionArgs) { - let appExecDetailsBaseParams = UIConfig.datatables[extraFunctionArgs.tableId]; - let appExecDetailsCustomParams = appExecDetailsBaseParams[tableViewType]; - let fileExportName = appExecDetailsCustomParams.fileExportPrefix; - if (tableViewType === 'singleAppView') { - fileExportName = appExecDetailsCustomParams.fileExportPrefix + "_" + extraFunctionArgs.appId; - } - - let appExecDataTableConf = { - paging: (execAppRecords.length > defaultPageLength), - pageLength: defaultPageLength, - lengthMenu: defaultLengthMenu, - info: true, - data: execAppRecords, - columns: [ - { - name: "appID", - data: "appID", - }, - { - name: "sqlID", - data: "sqlID", - className: "all", - }, - { - name: "exec", - data: "exec", - className: "all", - }, - { - name: "expr", - data: "expr", - className: "all", - }, - { - name: "isSupported", - data: "isSupported", - className: "all", - }, - { - name: "speedupFactor", - data: "speedupFactor", - render: function (data, type, row) { - if (data && type === 'display') { - return formatFloatingPoints(data); - } - return data; - }, - }, - { - name: "duration", - data: "duration", - "defaultContent":"", - className: "all", - render: function (data, type, row) { - if (data && type === 'display') { - return formatDuration(data); - } - return data; - }, - }, - { - name: "nodeId", - data: "nodeId", - }, - { - name: "stages", - data: "stages[, ]", - }, - { - name: "children", - "defaultContent":[], - data: "children", - render: "[, ].exec", - }, - { - name: "childrenNodeIDs", - "defaultContent":[], - data: "children", - render: "[, ].nodeId", - }, - { - name: "isRemoved", - data: "shouldRemove", - }, - ], - responsive: { - details: { - renderer: function ( api, rowIdx, columns ) { - let data = $.map( columns, function ( col, i ) { - let dataTableToolTip = toolTipsValues[appExecDetailsCustomParams.toolTipID]; - return col.hidden ? - ''+ - ' '+ - ''+ - '' : - ''; - } ).join(''); - - return data ? - $('
'+col.title+':'+ - ''+col.data+'
').append( data ) : - false; - } - } - }, - dom: appExecDetailsCustomParams.Dom, - buttons: [{ - extend: 'csv', - title: fileExportName, - text: 'Export' - }], - initComplete: function(settings, json) { - // Add custom Tool Tip to the headers of the table - // Add custom Tool Tip to the headers of the table - let thLabel = extraFunctionArgs.tableDivId + ' thead th'; - let dataTableToolTip = toolTipsValues[appExecDetailsCustomParams.toolTipID]; - $(thLabel).each(function () { - let $td = $(this); - let toolTipVal = dataTableToolTip[$td.text().trim()]; - $td.attr('data-toggle', "tooltip"); - $td.attr('data-placement', "top"); - $td.attr('html', "true"); - $td.attr('data-html', "true"); - $td.attr('title', toolTipVal); - }); - } - }; - - processDatableColumns( - appExecDataTableConf, - appExecDetailsBaseParams, - appExecDetailsCustomParams, - mustacheRecord); - - // set searchpanes - let optionGeneratorsFunctionsMap = new Map(); - optionGeneratorsFunctionsMap.set("execName", function() { - let execNameOptions = []; - execNames.forEach((data, execName) => { - let currOption = { - label: execName, - value: function(rowData, rowIdx) { - // get spark user - return (rowData["execName"] === execName); - }, - } - execNameOptions.push(currOption); - }); - return execNameOptions; - }); - optionGeneratorsFunctionsMap.set("stages", function() { - let stageIdOptions = []; - appStagesMap.getAllStages(extraFunctionArgs.appId).forEach(stageID => { - let currOption = { - label: stageID, - value: function(rowData, rowIdx) { - if (Array.isArray(rowData["stages"]) && rowData["stages"].length > 0) { - return rowData["stages"].some(stageNum => (stageNum === stageID)); - } - return stageID === "N/A"; - }, - }; - stageIdOptions.push(currOption); - }); - return stageIdOptions; - }); - - setDataTableSearchPanes( - appExecDataTableConf, - appExecDetailsBaseParams, - appExecDetailsCustomParams, - optionGeneratorsFunctionsMap); - - return appExecDataTableConf; -} - - -function createAppDetailsStagesTableConf( - execAppRecords, - tableViewType, - mustacheRecord, - extraFunctionArgs) { - let appStageDetailsBaseParams = UIConfig.datatables[extraFunctionArgs.tableId]; - let appStageDetailsCustomParams = appStageDetailsBaseParams[tableViewType]; - let fileExportName = appStageDetailsCustomParams.fileExportPrefix; - if (tableViewType === 'singleAppView') { - fileExportName = appStageDetailsCustomParams.fileExportPrefix + "_" + extraFunctionArgs.appId; - } - - let stagesDataTableConf = { - paging: (execAppRecords.length > defaultPageLength), - pageLength: defaultPageLength, - lengthMenu: defaultLengthMenu, - info: true, - data: execAppRecords, - columns: [ - { - name: "appID", - data: "appID", - }, - { - name: "stageId", - data: "stageId", - }, - { - name: "averageSpeedup", - data: "averageSpeedup", - render: function (data, type, row) { - if (data && type === 'display') { - return formatFloatingPoints(data); - } - return data; - }, - }, - { - name: "stageTaskTime", - data: "stageTaskTime", - render: function (data, type, row) { - if (data && type === 'display') { - return formatDuration(data); - } - return data; - }, - }, - { - name: "unsupportedTaskDur", - data: "unsupportedTaskDur", - render: function (data, type, row) { - if (data && type === 'display') { - return formatDuration(data); - } - return data; - }, - }, - { - name: "estimated", - data: "estimated", - }, - ], - responsive: { - details: { - renderer: function ( api, rowIdx, columns ) { - let data = $.map( columns, function ( col, i ) { - let dataTableToolTip = toolTipsValues[appStageDetailsCustomParams.toolTipID]; - return col.hidden ? - ''+ - ' '+ - ''+ - '' : - ''; - } ).join(''); - - return data ? - $('
'+col.title+':'+ - ''+col.data+'
').append( data ) : - false; - } - } - }, - dom: appStageDetailsCustomParams.Dom, - buttons: [{ - extend: 'csv', - title: fileExportName, - text: 'Export' - }], - initComplete: function(settings, json) { - // Add custom Tool Tip to the headers of the table - let thLabel = extraFunctionArgs.tableDivId + ' thead th'; - $(thLabel).each(function () { - let dataTableToolTip = toolTipsValues[appStageDetailsCustomParams.toolTipID]; - let $td = $(this); - let toolTipVal = dataTableToolTip[$td.text().trim()]; - $td.attr('data-toggle', "tooltip"); - $td.attr('data-placement', "top"); - $td.attr('html', "true"); - $td.attr('data-html', "true"); - $td.attr('title', toolTipVal); - }); - } - }; - - processDatableColumns( - stagesDataTableConf, - appStageDetailsBaseParams, - appStageDetailsCustomParams, - mustacheRecord); - - // set searchpanes - setDataTableSearchPanes( - stagesDataTableConf, - appStageDetailsBaseParams, - appStageDetailsCustomParams); - - return stagesDataTableConf; -} - - -function createAppDetailsSQLsTableConf( - appSQLsRecords, - tableViewType, - mustacheRecord, - extraFunctionArgs) { - let appSQLsDetailsBaseParams = UIConfig.datatables[extraFunctionArgs.tableId]; - let appSQLsDetailsCustomParams = appSQLsDetailsBaseParams[tableViewType]; - let fileExportName = appSQLsDetailsCustomParams.fileExportPrefix; - if (tableViewType === 'singleAppView') { - fileExportName = appSQLsDetailsCustomParams.fileExportPrefix + "_" + extraFunctionArgs.appId; - } - let recommendGPUColName = "gpuRecommendation" - - let appSQLsDataTableConf = { - paging: (appSQLsRecords.length > defaultPageLength), - pageLength: defaultPageLength, - lengthMenu: defaultLengthMenu, - info: true, - data: appSQLsRecords, - columns: [ - { - name: "appID", - data: "info.appId", - className: "all", - render: (appID, type, row) => { - if (type === 'display') { - if (createAppIDLinkEnabled(tableViewType)) { - return `${appID}` - } - } - return appID; - } - }, - { - name: "sqlID", - data: "sqlID", - className: "all", - }, - { - name: recommendGPUColName, - data: 'gpuCategory', - className: "all", - render: function (data, type, row) { - if (type === 'display') { - return getAppBadgeHTMLWrapper(row); - } - return data; - }, - fnCreatedCell: (nTd, sData, oData, _ignored_iRow, _ignored_iCol) => { - let recommendGroup = recommendationsMap.get(sData); - let toolTipVal = recommendGroup.description; - $(nTd).attr('data-toggle', "tooltip"); - $(nTd).attr('data-placement', "top"); - $(nTd).attr('html', "true"); - $(nTd).attr('data-html', "true"); - $(nTd).attr('title', toolTipVal); - } - }, - { - name: "sqlDescription", - data: "sqlDesc", - className: "all", - }, - { - name: 'estimatedGpuSpeedup', - data: 'info.estimatedGpuSpeedup', - render: function (data, type, row) { - if (data && type === 'display') { - return formatFloatingPoints(data); - } - return data; - }, - }, - { - name: 'sqlDataFrameDuration', - data: 'info.sqlDfDuration', - render: function (data, type, row) { - if (type === 'display') { - return formatDuration(data) - } - return data; - }, - }, - { - name: 'gpuOpportunity', - data: 'info.gpuOpportunity', - render: function (data, type, row) { - if (type === 'display') { - return formatDuration(data) - } - return data; - }, - }, - { - name: 'estimatedGpuDur', - data: 'info.estimatedGpuDur', - render: function (data, type, row) { - if (type === 'display') { - return formatDuration(data) - } - return data; - }, - }, - { - name: 'estimatedGpuTimeSaved', - data: 'info.estimatedGpuTimeSaved', - render: function (data, type, row) { - if (type === 'display') { - return formatDuration(data) - } - return data; - }, - }, - ], - responsive: { - details: { - renderer: function ( api, rowIdx, columns ) { - let data = $.map( columns, function ( col, i ) { - let dataTableToolTip = toolTipsValues[appSQLsDetailsCustomParams.toolTipID]; - return col.hidden ? - ''+ - ' '+ - ''+ - '' : - ''; - } ).join(''); - - return data ? - $('
'+col.title+':'+ - ''+col.data+'
').append( data ) : - false; - } - } - }, - dom: appSQLsDetailsCustomParams.Dom, - buttons: [{ - extend: 'csv', - title: fileExportName, - text: 'Export' - }], - initComplete: function(settings, json) { - // Add custom Tool Tip to the headers of the table - let thLabel = extraFunctionArgs.tableDivId + ' thead th'; - let dataTableToolTip = toolTipsValues[appSQLsDetailsCustomParams.toolTipID]; - $(thLabel).each(function () { - let $td = $(this); - let toolTipVal = dataTableToolTip[$td.text().trim()]; - $td.attr('data-toggle', "tooltip"); - $td.attr('data-placement', "top"); - $td.attr('html', "true"); - $td.attr('data-html', "true"); - $td.attr('title', toolTipVal); - }); - } - }; - - processDatableColumns( - appSQLsDataTableConf, - appSQLsDetailsBaseParams, - appSQLsDetailsCustomParams, - mustacheRecord); - - // set searchpanes - let optionGeneratorsFunctionsMap = new Map(); - optionGeneratorsFunctionsMap.set("recommendation", function() { - let categoryOptions = []; - for (let i in recommendationContainer) { - let currOption = { - label: recommendationContainer[i].displayName, - value: function(rowData, rowIdx) { - return (rowData["gpuCategory"] === recommendationContainer[i].displayName); - } - } - categoryOptions.push(currOption); - } - return categoryOptions; - }); - optionGeneratorsFunctionsMap.set("apps", function() { - let stageIdOptions = []; - qualificationRecords.forEach(appRecord => { - let currOption = { - label: appRecord.appId, - value: function(rowData, rowIdx) { - return (rowData.info.appId === appRecord.appId); - }, - }; - stageIdOptions.push(currOption); - }); - return stageIdOptions; - }); - - setDataTableSearchPanes( - appSQLsDataTableConf, - appSQLsDetailsBaseParams, - appSQLsDetailsCustomParams, - optionGeneratorsFunctionsMap); - - return appSQLsDataTableConf; -} - -function getAppDetailsTableTemplate() { - return ` -
-
- - - {{#displayCol_appName}} - - {{/displayCol_appName}} - - - - - - - - - - - - - - - - - - - - - - - - - - - -
App NameApp IDUserStart TimeRecommendationEstimated GPU SpeedupApp DurationEstimated GPU DurationEstimated GPU Time SavedTask Speed-up FactorSQL DF DurationGPU OpportunityUnsupported Task DurationSupported SQL DF Task DurationSQL Dataframe Task DurationExecutor CPU Time PercentLongest SQL DurationNONSQL Task Duration Plus OverheadApp Duration EstimatedSQL Ids with FailuresPotential ProblemsUnsupported Read File Formats and TypesUnsupported Write Data FormatComplex TypesNested Complex TypesRead Schema
- - `; -} - -function getAppSQLsDetailsTableTemplate() { - let content = ` -
- - - - {{#displayCol_appID}} - - {{/displayCol_appID}} - - - - - - - - - - -
App IDSQL IDRecommendationSQL DescriptionEstimated GPU SpeedupSQL DF DurationGPU OpportunityEstimated GPU DurationEstimated GPU Time Saved
-
- `; - return content; -} - -function setupNavigation() { - $(".dash-nav-dropdown-toggle").click(function () { - $(this).closest(".dash-nav-dropdown") - .toggleClass("show") - .find(".dash-nav-dropdown") - .removeClass("show"); - - $(this).parent() - .siblings() - .removeClass("show"); - }); - - $(".menu-toggle").click(function () { - $(".dash").toggleClass("dash-compact"); - }); -} - -function setupToolTipForTableCells() { - // Set the tootTips for the table header. - $('thead th[title]').tooltip({ - container: 'body', "html": true - }); - - // Set tooltips for the three tables. - // Note that we should always use method-2 - // - // method-1: - // using datatables. This method has limitations because datatable removes nodes from - // the DOM, therefore events applied with a static event listener might not be able to - // bind themselves to all nodes in the table. - // $('#app-execs-details-data-container [data-toggle="tooltip"]').tooltip({ - // container: 'body', - // html: true, - // animation: true, - // placement:"bottom",}); - // - // method-2: - // Using jQuery delegated event listener options which overcomes the limitations in method-1 - $('tbody').on('mouseover', 'td', function () { - $('[data-toggle="tooltip"]').tooltip({ - trigger: 'hover', - html: true - }); - }); -} diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala index bc0010af7..f21568bed 100644 --- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala +++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala @@ -28,13 +28,13 @@ import org.apache.hadoop.conf.Configuration import org.apache.spark.sql.rapids.tool.FailureApp import org.apache.spark.sql.rapids.tool.qualification._ -import org.apache.spark.sql.rapids.tool.ui.{ConsoleProgressBar, QualificationReportGenerator} +import org.apache.spark.sql.rapids.tool.ui.ConsoleProgressBar import org.apache.spark.sql.rapids.tool.util._ class Qualification(outputPath: String, numRows: Int, hadoopConf: Configuration, timeout: Option[Long], nThreads: Int, order: String, pluginTypeChecker: PluginTypeChecker, reportReadSchema: Boolean, - printStdout: Boolean, uiEnabled: Boolean, enablePB: Boolean, + printStdout: Boolean, enablePB: Boolean, reportSqlLevel: Boolean, maxSQLDescLength: Int, mlOpsEnabled:Boolean, penalizeTransitions: Boolean, tunerContext: Option[TunerContext], clusterReport: Boolean) extends ToolBase(timeout) { @@ -245,9 +245,6 @@ class Qualification(outputPath: String, numRows: Int, hadoopConf: Configuration, logWarning(s"Eventlogs doesn't contain any ML functions") } } - if (uiEnabled) { - QualificationReportGenerator.generateDashBoard(outputDir, allAppsSum) - } if (clusterReport) { qWriter.writeClusterReport(allAppsSum) qWriter.writeClusterReportCsv(allAppsSum) diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala index edf484459..ce322a0dc 100644 --- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala +++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala @@ -167,12 +167,6 @@ Usage: java -cp rapids-4-spark-tools_2.12-.jar:$SPARK_HOME/jars/* val userName: ScallopOption[String] = opt[String](required = false, descr = "Applications which a particular user has submitted." ) - val htmlReport : ScallopOption[Boolean] = - toggle("html-report", - default = Some(false), - prefix = "no-", - descrYes = "Generates an HTML Report. Disabled by default.", - descrNo = "Disables generating the HTML report.") val perSql : ScallopOption[Boolean] = opt[Boolean](required = false, descr = "Report at the individual SQL query level.") diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala index e451526e0..6ed646787 100644 --- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala +++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala @@ -62,7 +62,6 @@ object QualificationMain extends Logging { val timeout = appArgs.timeout.toOption val reportReadSchema = appArgs.reportReadSchema.getOrElse(false) val order = appArgs.order.getOrElse("desc") - val uiEnabled = appArgs.htmlReport.getOrElse(false) val reportSqlLevel = appArgs.perSql.getOrElse(false) val mlOpsEnabled = appArgs.mlFunctions.getOrElse(false) val penalizeTransitions = appArgs.penalizeTransitions.getOrElse(true) @@ -115,7 +114,7 @@ object QualificationMain extends Logging { None } val qual = new Qualification(outputDirectory, numOutputRows, hadoopConf, timeout, - nThreads, order, pluginTypeChecker, reportReadSchema, printStdout, uiEnabled, + nThreads, order, pluginTypeChecker, reportReadSchema, printStdout, enablePB, reportSqlLevel, maxSQLDescLength, mlOpsEnabled, penalizeTransitions, tunerContext, appArgs.clusterReport()) val res = qual.qualifyApps(filteredLogs) diff --git a/core/src/main/scala/org/apache/spark/sql/rapids/tool/ui/QualificationReportGenerator.scala b/core/src/main/scala/org/apache/spark/sql/rapids/tool/ui/QualificationReportGenerator.scala deleted file mode 100644 index 09874e0ba..000000000 --- a/core/src/main/scala/org/apache/spark/sql/rapids/tool/ui/QualificationReportGenerator.scala +++ /dev/null @@ -1,169 +0,0 @@ -/* - * Copyright (c) 2022-2024, NVIDIA CORPORATION. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.spark.sql.rapids.tool.ui - -import java.nio.charset.StandardCharsets -import java.nio.file -import java.nio.file.{Files, FileSystems, Paths} - -import scala.collection.JavaConverters.mapAsJavaMapConverter - -import org.apache.hadoop.fs.{FSDataOutputStream, Path} -import org.json4s.DefaultFormats -import org.json4s.jackson.Serialization - -import org.apache.spark.internal.Logging -import org.apache.spark.sql.rapids.tool.qualification.QualificationSummaryInfo -import org.apache.spark.sql.rapids.tool.util.FSUtils.getFSForPath -import org.apache.spark.sql.rapids.tool.util.RapidsToolsConfUtil -import org.apache.spark.util.Utils - -class QualificationReportGenerator(outputDir: String, - sumArr: Seq[QualificationSummaryInfo]) extends Logging { - - import QualificationReportGenerator._ - implicit val formats = DefaultFormats - - val outputWorkPath = new Path(outputDir) - lazy val fs = Some(getFSForPath(outputWorkPath, RapidsToolsConfUtil.newHadoopConf)) - - def launch(): Unit = { - val uiRootPath = getPathForResource(RAPIDS_UI_ASSETS_DIR) - logDebug(s"Generating UI files into... ${outputWorkPath.toUri}") - copyAssetFolderRecursively(uiRootPath, outputWorkPath) - } - - def copyAssetFolderRecursively(srcFolderPath: java.nio.file.Path, dstPath: Path): Unit = { - logDebug(s"UI code generator: Copying ... ${srcFolderPath.toUri}") - if (Files.isDirectory(srcFolderPath)) { - val destinationPath = new Path(dstPath, srcFolderPath.getFileName.toString) - fs.map { dstFileSys => - dstFileSys.mkdirs(destinationPath) - Files.list(srcFolderPath).forEach { childPath => - if (Files.isDirectory(childPath)) { - copyAssetFolderRecursively(childPath, destinationPath) - } else { - tryCopyAssetFile(childPath, new Path(destinationPath, childPath.getFileName.toString)) - } - } - } - } - } - - def generateJSFiles(): Unit = { - // Serializing the entire list of sums may stress the memory. - // Serializing one record at a time is slower but it would reduce the memory peak consumption. - val outputPath = new Path(outputWorkPath, RAPIDS_UI_JS_DATA) - val mainIndexPath = new Path(outputWorkPath, RAPIDS_UI_INDEX_PATH) - logInfo(s"Generating UI data in ${mainIndexPath.toUri}") - val fileHeader = - s""" - |let qualificationRecords = [ - """.stripMargin - val fileFooter = - s"""|]; - """.stripMargin - fs.foreach { dfs => - val outFile = dfs.create(outputPath) - Utils.tryWithSafeFinally { - outFile.writeBytes(fileHeader) - if (sumArr.nonEmpty) { - if (sumArr.size > 1) { - for (ind <- 0.until(sumArr.size - 1)) { - writeAppRecord(sumArr(ind), outFile) - } - } - writeAppRecord(sumArr.last, outFile, "") - } - outFile.writeBytes(fileFooter) - } { - outFile.flush() - outFile.close() - } - } - } - - private def writeAppRecord(appRec: QualificationSummaryInfo, - outStream: FSDataOutputStream, sep: String =","): Unit = { - val sumRec = "\t" + Serialization.write(appRec) + sep - outStream.write(sumRec.getBytes(StandardCharsets.UTF_8)) - } - - def tryCopyAssetFile(srcFilePath: java.nio.file.Path, dstPath: Path) : Unit = { - logDebug(s"Copying UI assets: ${srcFilePath.toUri.toString} to ${dstPath.toUri.toString}") - fs.foreach { dstFileSys => - Utils.tryWithResource(Files.newInputStream(srcFilePath)) { in => - val out = dstFileSys.create(dstPath) - Utils.tryWithSafeFinally { - val buffer = new Array[Byte](130 * 1024) - Iterator.continually(in.read(buffer)).takeWhile(_ != -1).foreach { bCount => - out.write(buffer, 0, bCount) - } - } { - out.flush() - out.close() - } - } - } - } - - def close(): Unit = { - jarFS.foreach { jFS => - jFS.close() - } - } -} - -object QualificationReportGenerator extends Logging { - val RAPIDS_UI_ASSETS_DIR = "/ui" - val RAPIDS_UI_JS_DATA = s"ui/js/data-output.js" - val RAPIDS_UI_INDEX_PATH = s"ui/html/index.html" - var jarFS : Option[file.FileSystem] = None - - private def getPathForResource(filename: String): java.nio.file.Path = { - val url = getClass.getResource(filename) - if (url.getPath.contains("jar")) { // this is a jar resource - val jFs = jarFS.getOrElse(setJarFileSystem(filename)) - jFs.getPath(filename) - } else { - Paths.get(url.toURI) - } - } - - private def setJarFileSystem(fileName: String): file.FileSystem = { - val jFileSys = FileSystems.newFileSystem(getClass.getResource(fileName).toURI, - Map[String, String]().asJava) - jarFS = Some(jFileSys) - jFileSys - } - - def generateDashBoard(outDir: String, sumArr: Seq[QualificationSummaryInfo]) : Unit = { - val generatorOp = Some(new QualificationReportGenerator(outDir, sumArr)) - var startTime = 0L; - generatorOp.foreach { generator => - Utils.tryWithSafeFinally { - startTime = System.currentTimeMillis() - generator.launch() - generator.generateJSFiles() - } { - generator.close() - val endTime = System.currentTimeMillis() - logInfo(s"Took ${endTime - startTime}ms to process ") - } - } - } -} diff --git a/user_tools/LICENSE b/user_tools/LICENSE index 57ee58ca8..f6acfa3b2 100644 --- a/user_tools/LICENSE +++ b/user_tools/LICENSE @@ -204,156 +204,6 @@ MIT License ----------- -Boostrap V4.6.1 - bootstrap.bundle.min.js - bootstrap.min.css - - The MIT License (MIT) - - Copyright (c) 2011-2019 Twitter, Inc. - Copyright (c) 2011-2019 The Bootstrap Authors - - Permission is hereby granted, free of charge, to any person obtaining a copy - of this software and associated documentation files (the "Software"), to deal - in the Software without restriction, including without limitation the rights - to use, copy, modify, merge, publish, distribute, sublicense, and/or sell - copies of the Software, and to permit persons to whom the Software is - furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in - all copies or substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE - AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER - LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN - THE SOFTWARE. - - -DataTables - DataTables-1.12.0/jquery.dataTables.min.js - DataTables-1.12.0/dataTables.bootstrap4.min.css - DataTables-1.12.0/dataTables.bootstrap4.min.js - searchpanes-2.0.1/searchPanes.bootstrap4.min.css - searchpanes-2.0.1/searchPanes.bootstrap4.min.js - searchpanes-2.0.1/dataTables.searchPanes.min.js - select-1.4.0/select.bootstrap4.min.css - select-1.4.0/dataTables.select.min.js - buttons-2.2.3/buttons.bootstrap4.min.css - buttons-2.2.3/buttons.bootstrap4.min.js - buttons-2.2.3/buttons.html5.min.js - buttons-2.2.3/dataTables.buttons.min.js - responsive-2.3.0/dataTables.responsive.min.js - responsive-2.3.0/responsive.bootstrap4.min.css - responsive-2.3.0/responsive.bootstrap4.min.js - - The MIT License (MIT) - - Copyright (C) 2008-2022, SpryMedia Ltd. - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software - and associated documentation files (the "Software"), to deal in the Software without - restriction, including without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the - Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or - substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING - BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, - DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -jQuery V3.6.0 - jquery.min.js - - Copyright JS Foundation and other contributors, https://js.foundation/ - - This software consists of voluntary contributions made by many - individuals. For exact contribution history, see the revision history - available at https://github.com/jquery/sizzle - - The following license applies to all parts of this software except as - documented below: - - ==== - - Permission is hereby granted, free of charge, to any person obtaining - a copy of this software and associated documentation files (the - "Software"), to deal in the Software without restriction, including - without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to - permit persons to whom the Software is furnished to do so, subject to - the following conditions: - - The above copyright notice and this permission notice shall be - included in all copies or substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF - MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE - LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION - OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION - WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - ==== - - All files located in the node_modules and external directories are - externally maintained libraries used by this software which have their - own licenses; we recommend you read them, as their terms may differ from - the terms above. - -Mustache.js V4.10 - mustache.min.js - - The MIT License - - Copyright (c) 2009 Chris Wanstrath (Ruby) - Copyright (c) 2010-2014 Jan Lehnardt (JavaScript) - Copyright (c) 2010-2015 The mustache.js community - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software - and associated documentation files (the "Software"), to deal in the Software without - restriction, including without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the - Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or - substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING - BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, - DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - -Spur Dashboard V1.1.0 - spur.min.css - - The MIT License (MIT) - - Copyright 2018 Alexander Rechsteiner - - Permission is hereby granted, free of charge, to any person obtaining a copy of this software - and associated documentation files (the "Software"), to deal in the Software without - restriction, including without limitation the rights to use, copy, modify, merge, publish, - distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the - Software is furnished to do so, subject to the following conditions: - - The above copyright notice and this permission notice shall be included in all copies or - substantial portions of the Software. - - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING - BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, - DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - jsoup License The MIT License