Skip to content

MttNewFeaturesIdeas

Jump to bottom
Jeff Squyres edited this page Sep 3, 2014 · 1 revision

Google Summer of Code Project Ideas for MTT

The MPI Testing Tool (MTT) is a flexible framework for designed to test Message Passing Interface (MPI) implementations across multiple environments and globally distributed participating organizations. Each participating organization runs a large number of unit tests locally using the MTT client. The MTT client submits testing results to a central server which stores the data for later analysis. The MTT Reporter is a web front end to the database providing aggregation and analysis capabilities for daily state of health assessments.

For more information about the structure of the MTT project check out the paper on the project or the presentation from SC08:

Below is a listing, by sub-component, of specific ideas for improvement.

Server-side improvements

The server side of MTT consists of two primary parts: the database used to store results, and the MTT Reporter used to query the data.

The MTT database is a PostgreSQL database. Currently the database is 76 GB representing 100 million test results accumulated since Nov. 2006. Over time we have been steadily increasing the amount of testing performed on a nightly basis. We are currently accumulating approximately 500,000 test results every night, which are analyzed using the MTT Reporter every morning. Most of the work needed on the database side focuses on scalability and performance of storing and mining a huge amount of data.

The MTT Reporter is an interactive web portal used by developers to mine the database in order to assess the state of health of the project on a daily basis. It allows developers to drill down into the testing data and pin point failing tests. It also allows developers to analyze not only correctness problems (represented by failing unit tests), but also performance problems. Most of the work needed on the MTT Reporter side focuses on performance of querying and rendering data, and intuitive representations of the testing data.

MTT Database improvements

Programming Experience:: Experience with SQL a necessity. Experience working with PostgreSQL databases is useful. Scripting experience with Perl and/or shell may also be useful.

Investigate Alternative Storage/Access Techniques for Testing Data:: The PostgreSQL database struggles when analyzing large time ranges of data due to the huge volume of data that must be accessed and aggregated. For this project we are interested in looking into alternative techniques and technologies that may help address these performance issues. Below are a few ideas:

  • Google App Engine: Host all of the MTT data in the Google Cloud and use their scalable data storage and use the accessor APIs for the MTT Reporter.
  • Linux bult-in (http://www.danga.com/memcached/ memcached)

Improve Integrity Checking of Submitted Data:: The data submitted by clients should be checked to make sure it is complete before being pushed into the database. This work will focus on both the submission page (PHP based form), and Insertion/Update checks in the database.

Results Tagging:: Add the ability to associate "tags" with specific test results. This would allow a developer to create/edit/delete tags using the MTT Reporter (e.g., lookup some specific results, and tag them)

Command-Line Utility to Query MTT Data:: Such a tool could be used to generate a report/graph at the end of an MTT client run that compares this run to previous/historical runs with data from the database.

MTT Reporter improvements

Programming Experience:: Experience with PHP will be useful. Experience writing SQL queries also useful. Ability to write HTML important.

Convert MTT Reporter from PHP to JavaScript/AJAX-like Alternative:: Currently the MTT Reporter is based in PHP making it 100% server-side driven. We would like reduce the load on the server by exploring alternative technologies such as JavaScript or AJAX. We would also be interested in exploring the Google App Engine APIs for accessing and rendering data (Related to the Alternative Storage idea above).

Visual Depiction of Testing Results Timeline:: Visually depict test results as a graphical timeline of events -- show each step as it happens.

Visual Depiction of Testing Phase Relationships:: Visually depict relationships between different testing phases (e.g., this test result is linked to this test build, which is linked to this MPI install), allowing clickable/re-querying on each phase

Improved Performance Graphs:: Better visualization of bandwidth/latency and other performance graphs

Per-User Personalization of MTT Reporter:: Allow per-user personalization of the web reporter

Durable Off-line Reports:: Export reports to a durable/off-line format (e.g., a series of HTML files, a PDF, ...?).

Automated Script to Alert Developers of Under-tested Configurations:: Create an automated script to query the database and issue an alert when specific configurations have not been tested in a while.

Create Stored Standard Queries:: Create some standard queries that generate reports that can be sent out on a regular basis (perhaps even having the ability to combine these queries):

  • Create a query that only shows "new" test failures (i.e., failures that happened today that have not happened within the last week; ticket trac ticket 70).
  • Create a query that only shows failures on a specific platform, OS, etc. (ticket trac ticket 71).
  • Related Tickets: trac ticket 70 trac ticket 71

Client-side improvements

The MTT Client is a Perl script that participating organizations use to build software and test suites, run unit tests in various configurations, and report the information back to the MTT Database. The MTT Client must be able to work on a variety of different platforms (currently we support Linux, Mac OS X, Solaris, and Windows with Cygwin), and across organizational boundaries.

MTT Client improvements

Programming Experience:: Experience programming in Perl is a must.

Preserve corefile backtrace with failed tests:: When a test fails and generates a corefile, it would be useful to get the backtrace from the corefile and associate the backtrace (not the whole corefile) with the testing data that is submitted to the database.

MTT Functlet to Query the MTT Database:: Create an MTT funclet that gives access to MTT database results

Automatic Parallelization of Testing:: Automatically parallelize all the testing within a single INI file for maximum throughput on specific resources.

Centralized Storage of Testing Configurations:: Create a service that would allow each participating organization to store/update/delete testing configurations for their systems. These configurations can assist in debugging MTT configuration issues, and provide examples for other organizations setting up MTT on a new platform.

Clone this wiki locally