Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to customize the output of WSGI app from within a testcase? #71

Open
benoitbryon opened this issue May 20, 2014 · 9 comments
Labels

Comments

@benoitbryon
Copy link
Member

See https://twitter.com/bartstroeken/status/467564038025932800

@benoitbryon
Copy link
Member Author

What do you expect?
What would you like to customize? The error messages? The JSON structure? Something else?
Can you give an example?

@bartee
Copy link

bartee commented May 20, 2014

Hi Benoit,

Thanks for responding, I thought about overwriting the TestResultResponse class, but there might be a better way.
I'd like to be able to annotate the details of a testcase, for some extra info like the host where that specific test was executed on - in case of an external service or server.

By adding that, you can interconnect different hosts running hospital tests for example.

{
    "status": "pass",
    "details": [
        {
            "test": "test_http_200 (healthchecks.exampletest.DocumentationHealthCheck)",
            "status": "pass",
            "domain": "http://api.nu.nl", 
        },
        {
            "test": "test_connection (healthchecks.mysqltest.MySQLTestCase)",
            "status": "pass",
            "host": "192.168.1.44"
        },
        {
            "test": "test_ping (healthchecks.redistest.RedisHealthCheck)",
            "status": "pass",
            "host": "192.168.1.55"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 3,
        "expected_failure": 0,
        "error": 0,
        "fail": 0,
        "total": 3,
        "unexpected_success": 0
    }
}

@benoitbryon
Copy link
Member Author

There may be some improvements around the context data passed to the AssertionError.
As an example, at the moment, with a fail we get an additional "context" list:

{
    "status": "fail",
    "details": [
        {
            "test": "test_false (foo.bar)",
            "status": "fail",
            "context": "(<type 'exceptions.AssertionError'>, AssertionError(\"Failed to fetch URL http://localhost:8000/.\\nException was: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: / (Caused by <class 'socket.error'>: [Errno 111] Connection refused)\",), <traceback object at 0x7f34ac5bde18>)"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 0,
        "expected_failure": 0,
        "error": 0,
        "fail": 1,
        "total": 1,
        "unexpected_success": 0
    }
}

Note: the traceback in "context" is not really readable, see #72.
But here is one idea:

  • on one side, we could improve the way hospital displays the "context"
  • on the other side, developers could pass context data to assertions.

... but the fact is I'm not sure how we can easily do this. Using the assertion message? assert True is False, "Here some information, but not easy to convert as a dictionary :("

If you had a (nice) way to populate such a context, would it be ok for you?

@benoitbryon
Copy link
Member Author

A second workaround could be related to the execution context running the healthchecks.
I mean, in your use case, the "host" information is shared by all the healthchecks run on a machine.
At the moment, I have no nice idea about how we could do it. Perhaps using logging, but I'm not sure it would be easy to have relationships between an AssertionError and a log that was emitted before.
What about working on #22 first, then see if it could be used or adapted to your use case?

@benoitbryon
Copy link
Member Author

you can interconnect different hosts running hospital tests for example.

This is a story on its own!
At the moment, I would tend to do something like that:

  • have several machines, each one running a healthcheck service
  • have a "supervisor" machine, which collects and aggregates healthchecks on other machines

With such an architecture, on the "supervisor" (how would you name it?) you could easily have a result like that:

{
    "status": "pass",
    "details": [
        {
            "test": "GET http://api.nu.nl/healthchecks returns HTTP 200 OK",
            "status": "pass"
        },
        {
             "test": "GET http://192.168.1.44:1515 returns HTTP 200 OK",
            "status": "pass"
        },
        {
            "test": "GET http://192.168.1.55 returns HTTP 200 OK",
            "status": "pass"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 3,
        "expected_failure": 0,
        "error": 0,
        "fail": 0,
        "total": 3,
        "unexpected_success": 0
    }
}

If we had some hospital.assert_external_hospital(url) method, we could do it easily and have a better message in case of error, because it could take advantage of the JSON output. And it could highlight the information about external host (IP, port...).

Would this fit your use case?

That said, I also think the behaviour of such a "supervisor" should be:

  • run "smoketests" on each node, i.e. fail fast.
  • in case of a fail, optionally start a diagnosis, in a second process, i.e. run complete healthcheck on the host which failed, but as another action.

I think we should have at least two levels of feedback:

  • smoketests => runs fast, basically gives fail/pass feedback
  • diagnosis => runs slower, provides detailed feedback about the errors

With this idea in mind, I am not sure, at the moment, that a "supervisor" has to be able to concatenate results of every nodes, i.e. display details of each node. Whereas I think it has to be able to summarize results of each node, i.e. display fail/pass status of each node. If you need details about one node, isn't asking the node enough?

@benoitbryon
Copy link
Member Author

And also, notice that we already can attach a custom message to assertions, but it is not easy to parse in the JSON output (it is a string, not a dictionary).
assert expression, "your custom message".
Perhaps it is enough, at least until we have better options.

@benoitbryon
Copy link
Member Author

@bartee: the comments/questions above are meant to have a better idea about your needs, so that we can focus on what really matters ;)
Tell me if I am misunderstanding or forgetting some point!

@bartee
Copy link

bartee commented May 30, 2014

Wooah, you've been busy! 👍

Interesting thoughts! After reading all this, I've gotten my hands dirty myself, and solved it for my usecase. First, I modified my testcase a bit, providing details:

@hospital.healthcheck
class RedisHealthCheck(unittest.TestCase):
    """
    Test the current state of Redis
    """
    def test_ping(self):
        r = redis.Redis(host=redis_host, port=redis_port,password=redis_password)
        try:
            r.ping()
        except redis.ConnectionError, e:
            self.fail(e)

    @property
    def details(self):
        return {
            'host': redis_host,
            'port': redis_port,
            'password': redis_password
        }

After that, I've added two lines to the details-method of the HealthCheckApp- class:

class HealthCheckApp(HealthCheckProgram):

    def details(self, result):
        """Generate report details dict."""
        for status, test, context in result.results:
            report = {
                'test': self.get_test_title(test),
                'status': status,
            }
            if context is not None:
                report['context'] = u'{context}'.format(context=context)

            if hasattr(test,'details'):
                report['details'] = test.details

            yield report

That works like a charm. In that way, the supervisor can use those details.details of a test case to interconnect different hosts. And you don't have to change the signature of existing output (or parse a string into something you can work with :-))

Only downside I can see: I can imagine details.details being a bit of a confusing name. But on the other hand, there might be use cases out there where other kinds of test specs are necessary.

I've already built a little supervisor-project myself. It's config based, checks a configured set of hosts for hospital-based output and puts that output into Redis. I yet have to plug it into my hosts-dashboard. Somewhere soon I'll put that little thing on my github.

@bartee
Copy link

bartee commented May 30, 2014

About that supervisor: https://github.com/bartee/hospital_supervisor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants