Add ability to email code maintainers when there is a test failure

[jairideout]

It would be cool if the maintainer of the module that has failing tests would be notified via a separate email. Currently if there is a failure, we have to look through the unit test output file, locate the failed test, and then send an email to the maintainer or the person who committed the changes that broke the test.

This feature is not trivial to add as the testing system is very generic: it does not attempt to parse unit test output files or code files (this would be necessary to locate the failure and then look up the maintainer name and email address in the corresponding module).

The best way I can see handling this would to be create some sort of plugin/hook system to support executing arbitrary code/commands to do this extra work. This way, the system can stay generic (which is a strength so that it can be used in a wide variety of contexts/projects), but also can be extended/customized for users' particular needs.

[wasade]

I was looking at something similar the other day. I think this could be
done if the run() method of TestCase is overridden, such that the failure
can actually be trapped. Introspection could be performed to check if
email is in the environment as well. This is all easier said than done,
but I think only a single method needs to be modified. It could stay
generic in that a) could be designed so the functionality can be disabled
at run time b) designed so that the variable holding the email address can
be modified at run time

On Tue, Nov 6, 2012 at 9:23 AM, Jai Ram Rideout [email protected]:

It would be cool if the maintainer of the module that has failing tests
would be notified via a separate email. Currently if there is a failure, we
have to look through the unit test output file, locate the failed test, and
then send an email to the maintainer or the person who committed the
changes that broke the test.

This feature is not trivial to add as the testing system is very generic:
it does not attempt to parse unit test output files or code files (this
would be necessary to locate the failure and then look up the maintainer
name and email address in the corresponding module).

The best way I can see handling this would to be create some sort of
plugin/hook system to support executing arbitrary code/commands to do this
extra work. This way, the system can stay generic (which is a strength so
that it can be used in a wide variety of contexts/projects), but also can
be extended/customized for users' particular needs.

—
Reply to this email directly or view it on GitHubhttps://github.com/qiime/automated_testing/issues/4.

[jairideout]

This is a great idea. I was thinking of solving this problem via the automated testing system (i.e. it parses the failure log and figures out who to email), but maybe it's better to let each project (e.g. QIIME) handle this in their own unit test suite.

@gregcaporaso and I have chatted about adding the ability to send emails from within QIIME (in the context of workflows finishing on EC2 so that the user could be notified when a long-running job finished). If we were to add this capability to QIIME, we'd need a separate email config file (optional, of course) that would contain SMTP settings. It would be kept separate from the QIIME config file for security purposes (e.g. so that users could put different permissions on their email config file than their QIIME config) since a password would need to be stored.

If we decide to go this route, I can create an issue on the QIIME repo for this feature (as it is not really part of the automated testing system anymore).