counit: to run time/IO related unit tests faster using Swoole

This package helps to run time/IO related unit tests (e.g., sleep function calls, database queries, API calls, etc) faster using Swoole.

Table of Contents

• How Does It Work
• Installation
• Use "counit" in Your Project
• Examples
  • Setup Test Environment
  • The "global" Style
  • The "case by case" Style
  • Comparisons
• Additional Notes
• Local Development
• Alternatives
• TODOs
• License

How Does It Work

Package counit allows running multiple time/IO related tests concurrently within a single PHP process using Swoole. Counit is compatible with PHPUnit, which means:

1. Test cases can be written in the same way as those for PHPUnit.
2. Test cases can run directly under PHPUnit.

A typical test case of counit looks like this:

use Deminy\Counit\TestCase; // Here is the only change made for counit, comparing to test cases for PHPUnit.

class SleepTest extends TestCase
{
  public function testSleep(): void
  {
    $startTime = time();
    sleep(3);
    $endTime = time();

    self::assertEqualsWithDelta(3, ($endTime - $startTime), 1, 'The sleep() function call takes about 3 seconds to finish.');
  }
}

Comparing to PHPUnit, counit could make your test cases faster. Here is a comparison when running the same test suite using PHPUnit and counit for a real project. In the test suite, many tests make calls to method \Deminy\Counit\Counit::sleep() to wait something to happen (e.g., wait data to expire).

	# of Tests	# of Assertions	Time to Finish
counit (without Swoole), or PHPUnit	44	1148	9 minutes and 18 seconds
counit (with Swoole enabled)			19 seconds

Installation

The package can be installed using Composer:

composer require deminy/counit --dev

Or, in your composer.json file, make sure to have package deminy/counit included:

{
  "require-dev": {
    "deminy/counit": "~0.2.0"
  }
}

Use "counit" in Your Project

• Write unit tests in the same way as those for PHPUnit. However, to make those tests faster, please write those time/IO related tests in one of the following two styles (details will be discussed in the next sections):
  • The global style (recommended): Use class Deminy\Counit\TestCase instead of PHPUnit\Framework\TestCase as the base class.
  • The case-by-case style: Wrap each test case inside the callback function for method Deminy\Counit\Counit::create(), and use method Deminy\Counit\Counit::sleep() instead of the PHP function sleep().
• Use the binary executable ./vendor/bin/counit instead of ./vendor/bin/phpunit when running unit tests.
• Have the Swoole extension installed. If not installed, counit will work exactly same as PHPUnit (in blocking mode).
• Optional steps:
  • use PHPUnit extension Deminy\Counit\CounitExtension as shown in file phpunit.xml.dist. This is to wait the whole test suite to finish before printing out the summary information at the end.

Examples

Folder ./tests/unit/global and ./tests/unit/case-by-case contain some sample tests, where we have following time-related tests included:

• Test slow HTTP requests.
• Test long-running MySQL queries.
• Test data expiration in Redis.
• Test sleep() function calls in PHP.

Setup Test Environment

To run the sample tests, please start the Docker containers and install Composer packages first:

docker-compose up -d
docker compose exec -ti swoole composer install -n

There are five containers started: a PHP container, a Swoole container, a Redis container, a MySQL container, and a web server. The PHP container doesn't have the Swoole extension installed, while the Swoole container has it installed and enabled.

As said previously, test cases can be written in the same way as those for PHPUnit. However, to run time/IO related tests faster with counit, we need to make some adjustments when writing those test cases; these adjustments can be made in two different styles.

The "global" Style (recommended)

In this style, each test case runs in a separate coroutine automatically.

For test cases written in this style, the only change to make on your existing test cases is to use class Deminy\Counit\TestCase instead of PHPUnit\Framework\TestCase as the base class.

A typical test case of the global style looks like this:

use Deminy\Counit\TestCase; // Here is the only change made for counit, comparing to test cases for PHPUnit.

class SleepTest extends TestCase
{
  public function testSleep(): void
  {
    $startTime = time();
    sleep(3);
    $endTime = time();

    self::assertEqualsWithDelta(3, ($endTime - $startTime), 1, 'The sleep() function call takes about 3 seconds to finish.');
  }
}

When customized method setUpBeforeClass() and tearDownAfterClass() are defined in the test cases, please make sure to call their parent methods accordingly in these customized methods.

This style assumes there is no immediate assertions in test cases, nor assertions before a sleep() function call or a coroutine-friendly IO operation. Test cases like following still work, but they will trigger some warning messages when tested:

class GlobalTest extends Deminy\Counit\TestCase
{
  public function testAssertionSuppression(): void
  {
    self::assertTrue(true, 'Trigger an immediate assertion.');
    // ......
  }
}

We can rewrite this test class using the "case by case" style (discussed in the next section) to eliminate the warning messages.

To find more tests written in this style, please check tests under folder ./tests/unit/global (test suite "global").

The "case by case" Style

In this style, you make changes directly on a test case to make it work asynchronously.

For test cases written in this style, we need to use class Deminy\Counit\Counit accordingly in the test cases where we need to wait for PHP execution or to perform IO operations. Typically, following method calls will be used:

• Use method Deminy\Counit\Counit::create() to wrap the test case.
• Use method Deminy\Counit\Counit::sleep() instead of the PHP function sleep() to wait for PHP execution. You will need some knowledge on Swoole if you want to make other IO related tests run asynchronously.

A typical test case of the case-by-case style looks like this:

use Deminy\Counit\Counit;
use PHPUnit\Framework\TestCase;

class SleepTest extends TestCase
{
  public function testSleep(): void
  {
    Counit::create(function () { // To create a new coroutine manually to run the test case.
      $startTime = time();
      Counit::sleep(3); // Call this method instead of PHP function sleep().
      $endTime = time();

      self::assertEqualsWithDelta(3, ($endTime - $startTime), 1, 'The sleep() function call takes about 3 seconds to finish.');
    });
  }
}

In case you need to suppress warning message "This test did not perform any assertions" or to make the number of assertions match, you can include a 2nd parameter when creating the new coroutine:

use Deminy\Counit\Counit;
use PHPUnit\Framework\TestCase;

class SleepTest extends TestCase
{
  public function testSleep(): void
  {
    Counit::create( // To create a new coroutine manually to run the test case.
      function () {
        $startTime = time();
        Counit::sleep(3); // Call this method instead of PHP function sleep().
        $endTime = time();

        self::assertEqualsWithDelta(3, ($endTime - $startTime), 1, 'The sleep() function call takes about 3 seconds to finish.');
      },
      1 // Optional. To suppress warning message "This test did not perform any assertions", and to make the counters match.
    );
  }
}

To find more tests written in this style, please check tests under folder ./tests/unit/case-by-case (test suite "case-by-case").

Comparisons

Here we will run the tests under different environments, with or without Swoole.

#1 Run the test suites using PHPUnit:

# To run test suite "global":
docker compose exec -ti php    ./vendor/bin/phpunit --testsuite global
# or,
docker compose exec -ti swoole ./vendor/bin/phpunit --testsuite global

# To run test suite "case-by-case":
docker compose exec -ti php    ./vendor/bin/phpunit --testsuite case-by-case
# or,
docker compose exec -ti swoole ./vendor/bin/phpunit --testsuite case-by-case

#2 Run the test suites using counit (without Swoole):

# To run test suite "global":
docker compose exec -ti php    ./counit --testsuite global

# To run test suite "case-by-case":
docker compose exec -ti php    ./counit --testsuite case-by-case

#3 Run the test suites using counit (with extension Swoole enabled):

# To run test suite "global":
docker compose exec -ti swoole ./counit --testsuite global

# To run test suite "case-by-case":
docker compose exec -ti swoole ./counit --testsuite case-by-case

The first two sets of commands take about same amount of time to finish. The last set of commands uses counit and runs in the Swoole container (where the Swoole extension is enabled); thus it's faster than the others:

	Style	# of Tests	# of Assertions	Time to Finish
counit (without Swoole), or PHPUnit	global	16	24	48 seconds
	case by case			48 seconds
counit (with Swoole enabled)	global			7 seconds
	case by case			7 seconds

Additional Notes

Since this package allows running multiple tests simultaneously, we should not use same resources in different tests; otherwise, racing conditions could happen. For example, if multiple tests use the same Redis key, some of them could fail occasionally. In this case, we should use different Redis keys in different test cases. Method \Deminy\Counit\Helper::getNewKey() and \Deminy\Counit\Helper::getNewKeys() can be used to generate random and unique test keys.

The package works best for tests that have function call sleep() in use; It can also help to run some IO related tests faster, with limitations apply. Here is a list of limitations of this package:

• The package makes tests running faster by performing time/IO operations simultaneously. For functions/extensions that work in blocking mode only, this package can't make their function calls faster. Here are some extensions that work in blocking mode only: MongoDB, Couchbase, and some ODBC drivers.
• The package doesn't work exactly the same as when running under PHPUnit:
  • Tests may not have yet finished even it's marked as finished (by PHPUnit). Because of that, a test marked as "passed" (by PHPUnit) could still fail at a later time under counit. Because of this, the most reliable way to check if all test cases have passed or not is to check the exit code of counit.
  • The # of assertions reported could be different from PHPUnit.
  • Some exceptions/errors are not handled/reported the same.

Local Development

There are pre-built images deminy/counit for running the sample tests. Here are the commands to build the images:

docker build -t deminy/counit:php-only       -f ./dockerfiles/php/Dockerfile    .
docker build -t deminy/counit:swoole-enabled -f ./dockerfiles/swoole/Dockerfile .

Alternatives

This package allows to use Swoole to run multiple time/IO related tests without multiprocessing, which means all tests can run within a single PHP process. To understand how exactly it works, I'd recommend checking this free online talk: CSP Programming in PHP (and here are the slides).

In the PHP ecosystem, there are other options to run unit tests in parallel, most end up using multiprocessing:

• Process isolation in PHPUnit. This allows to run tests in separate PHP processes.
• Package brianium/paratest
• Package pestphp/pest

TODOs

• Better integration with PHPUnit.
  • Deal with annotation @doesNotPerformAssertions in the global style.
  • Make # of assertions consistent with the one reported from PHPUnit.
• Better error/exception handling.

License

MIT license.