ParaTest

:computer: Parallel testing for PHPUnit

Code Quality Rank: L4

Programming language: PHP

License: MIT License

Tags: Testing     Concurrent     Parallel     Phpunit    

Latest version: v6.1.1

README

ParaTest

The objective of ParaTest is to support parallel testing in PHPUnit. Provided you have well-written PHPUnit tests, you can drop paratest in your project and start using it with no additional bootstrap or configurations!

Benefits:

• Zero configuration. After the installation, run with vendor/bin/paratest. That's it!
• Code Coverage report combining. Run your tests in N parallel processes and all the code coverage output will be combined into one report.
• Flexible. Isolate test files in separate processes or take advantage of WrapperRunner for even faster runs.

Installation

To install with composer run the following command:

composer require --dev brianium/paratest

Versions

Only the latest version of PHPUnit is supported, and thus only the latest version of ParaTest is actively maintained.

This is because of the following reasons:

1. To reduce bugs, code duplication and incompatibilities with PHPUnit, from version 5 ParaTest heavily relies on PHPUnit @internal classes
2. The fast pace both PHP and PHPUnit have taken recently adds too much maintenance burden, which we can only afford for the latest versions to stay up-to-date

Usage

After installation, the binary can be found at vendor/bin/paratest. Run it with --help option to see a complete list of the available options.

Optimizing Speed

To get the most out of ParaTest, you have to adjust the parameters carefully.

1. Use the WrapperRunner if possible

   The default Runner for PHPUnit spawns a new process for each testcase (or method in functional mode). This provides the highest compatibility but comes with the cost of many spawned processes and a bootstrapping for each process. Especially when you have a slow bootstrapping in your tests (like a database setup) you should try the WrapperRunner with --runner WrapperRunner. It spawns one "worker"-process for each parallel process (-p), executes the bootstrapping once and reuses these processes for each test executed. That way the overhead of process spawning and bootstrapping is reduced to the minimum.

2. Adjust the number of processes with -p

   To allow full usage of your cpu cores, you should have at least one process per core. More processes allow better resource usage but keep in mind that each process has its own costs for spawning. The default is auto, which means the number of logical CPU cores is set as the number of processes. You might try something like logical CPU cores * 2 (e.g. if you have 8 logical cores, you might try 16), but keep in mind that each process generates a little bit of overhead as well.

3. Choose between per-testcase- and per-testmethod-parallelization with -f

   Given you have few testcases (classes) with many long running methods, you should use the -f option to enable the functional mode and allow different methods of the same class to be executed in parallel. Keep in mind that the default is per-testcase-parallelization to address inter-testmethod dependencies. Note that in most projects, using -f is slower since each test method will need to be bootstrapped separately.

4. Tune batch max size --max-batch-size

   Batch size will affect the max amount of atomic tests which will be used for a single test method. One atomic test will be either one test method from test class if no data provider available for method or will be only one item from dataset for method. Increase this value to reduce per-process overhead and in most cases it will also reduce parallel efficiency. Decrease this value to increase per-process overhead and in most cases it will also increase parallel efficiency. If the amount of all tests is less than the max batch size then everything will be processed in one process thread so ParaTest is completely useless in that case. The best way to find the most effective batch size is to test with different batch size values and select best. Max batch size = 0 means that grouping in batches will not be used and one batch will equal all method tests (one or all from data provider). Max batch size = 1 means that each batch will contain only one test from the data provider or one method if the data provider is not used. Bigger max batch size can significantly increase phpunit command line length so the process can fail. Decrease max batch size to reduce command line length. Windows has a limit around 32k, Linux - 2048k, Mac OS X - 256k.

Test token

The TEST_TOKEN environment variable is guaranteed to have a value that is different from every other currently running test. This is useful to e.g. use a different database for each test:

if (getenv('TEST_TOKEN') !== false) {  // Using ParaTest
    $dbname = 'testdb_' . getenv('TEST_TOKEN');
} else {
    $dbname = 'testdb';
}

A UNIQUE_TEST_TOKEN environment variable is also available and guaranteed to have a value that is unique both per run and per process.

Code coverage

Beginning from PHPUnit 9.3.4, it is strongly advised to set a coverage cache directory, see PHPUnit Changelog @ 9.3.4.

The cache is always warmed up by ParaTest before executing the test suite.

PCOV

If you have installed pcov but need to enable it only while running tests, you have to pass thru the needed PHP binary option:

php -d pcov.enabled=1 vendor/bin/paratest --passthru-php="'-d' 'pcov.enabled=1'"

xDebug

If you have xDebug installed, activating it by the environment variable is enough to have it running even in the subprocesses:

XDEBUG_MODE=coverage vendor/bin/paratest

PHPDBG

PHPDBG is automatically detected and used in the subprocesses if it's the running binary of the main process:

phpdbg vendor/bin/paratest

Initial setup for all tests

Because ParaTest runs multiple processes in parallel, each with their own instance of the PHP interpreter, techniques used to perform an initialization step exactly once for each test work different from PHPUnit. The following pattern will not work as expected - run the initialization exactly once - and instead run the initialization once per process:

private static bool $initialized = false;

public function setUp(): void
{
    if (! self::$initialized) {
         self::initialize();
         self::$initialized = true;
    }
}

This is because static variables persist during the execution of a single process. In parallel testing each process has a separate instance of $initialized. You can use the following pattern to ensure your initialization runs exactly once for the entire test invocation:

static bool $initialized = false;

public function setUp(): void
{
    if (! self::$initialized) {
        // We utilize the filesystem as shared mutable state to coordinate between processes
        touch('/tmp/test-initialization-lock-file');
        $lockFile = fopen('/tmp/test-initialization-lock-file', 'r');

        // Attempt to get an exclusive lock - first process wins
        if (flock($lockFile, LOCK_EX | LOCK_NB)) {
            // Since we are the single process that has an exclusive lock, we run the initialization
            self::initialize();
        } else {
            // If no exclusive lock is available, block until the first process is done with initialization
            flock($lockFile, LOCK_SH);
        }

        self::$initialized = true;
    }
}

Troubleshooting

If you run into problems with paratest, try to get more information about the issue by enabling debug output via --verbose --debug.

When a sub-process fails, the originating command is given in the output and can then be copy-pasted in the terminal to be run and debugged. All internal commands run with --printer [...]\NullPhpunitPrinter which silence the original PHPUnit output: during a debugging run remove that option to restore the output and see what PHPUnit is doing.

Windows

Windows users be sure to use the appropriate batch files.

An example being:

vendor\bin\paratest.bat ...

ParaTest assumes PSR-0 for loading tests.

For convenience, ParaTest for Windows uses 79 columns mode to prevent blank lines in the standard 80x25 windows console.

Caveats

1. Constants, static methods, static variables and everything exposed by test classes consumed by other test classes (including Reflection) are not supported. This is due to a limitation of the current implementation of WrapperRunner and how PHPUnit searches for classes. The fix is to put shared code into classes which are not tests themselves.

Integration with PHPStorm

ParaTest provides a dedicated binary to work with PHPStorm; follow these steps to have ParaTest working within it:

1. Be sure you have PHPUnit already configured in PHPStorm: https://www.jetbrains.com/help/phpstorm/using-phpunit-framework.html#php_test_frameworks_phpunit_integrate
2. Go to Run -> Edit configurations...
3. Select Add new Configuration, select the PHPUnit type and name it ParaTest
4. In the Command Line -> Interpreter options add ./vendor/bin/paratest_for_phpstorm
5. Any additional ParaTest options you want to pass to ParaTest should go within the Test runner -> Test runner options section

You should now have a ParaTest run within your configurations list. It should natively work with the Rerun failed tests and Toggle auto-test buttons of the Run overlay.

Run with Coverage

Coverage with one of the available coverage engines must already be configured in PHPStorm and working when running tests sequentially in order for the helper binary to correctly handle code coverage

For Contributors: testing ParaTest itself

Before creating a Pull Request be sure to run all the necessary checks with make command.