Test parallelism

The build job runs across multiple parallel containers (2 by default) to speed up test execution. Since each container runs the full build and provision steps, the test workload is distributed to make the best use of each container.

Code linting runs in a separate lint job and is not affected by test parallelism settings.

What runs where

Task	First container	Other containers
Jest tests	✓	-
PHPUnit tests	✓	-
Code coverage check and PR comment	✓	-
Single Directory Component validation	✓	-
Behat tests	✓ (profile p0)	✓ (profile p1, p2, ...)

Everything except Behat runs exclusively on the first container to avoid duplicate work. Behat tests run on all containers using profile-based distribution.

Choosing which container runs what

Each tool reads a CI_IS_<TOOL>_RUNNER variable that decides whether it runs on the current container. All of them are declared together at the top of the build job, so the whole distribution is visible and editable in one place:

GitHub Actions

.github/workflows/build-test-deploy.yml

env:
  CI_RUNNER_INDEX: ${{ strategy.job-index }}
  CI_RUNNER_TOTAL: ${{ strategy.job-total }}
  CI_IS_JEST_RUNNER: ${{ matrix.instance == 0 || strategy.job-total == 1 }}
  CI_IS_PHPUNIT_RUNNER: ${{ matrix.instance == 0 || strategy.job-total == 1 }}
  CI_IS_SDC_DEVEL_RUNNER: ${{ matrix.instance == 0 || strategy.job-total == 1 }}
  CI_IS_BEHAT_RUNNER: true

Each tool's step then reads its own flag:

- name: Test with PHPUnit
  if: ${{ env.CI_IS_PHPUNIT_RUNNER == 'true' }}

CircleCI

.circleci/config.yml

- run:
    name: Set test runner roles
    command: |
      {
        echo "export CI_RUNNER_INDEX=${CIRCLE_NODE_INDEX:-0}"
        echo "export CI_RUNNER_TOTAL=${CIRCLE_NODE_TOTAL:-1}"
        echo "export CI_IS_JEST_RUNNER=$([ "${CIRCLE_NODE_INDEX:-0}" -eq 0 ] && echo 1 || echo 0)"
        echo "export CI_IS_PHPUNIT_RUNNER=$([ "${CIRCLE_NODE_INDEX:-0}" -eq 0 ] && echo 1 || echo 0)"
        echo "export CI_IS_SDC_DEVEL_RUNNER=$([ "${CIRCLE_NODE_INDEX:-0}" -eq 0 ] && echo 1 || echo 0)"
        echo "export CI_IS_BEHAT_RUNNER=1"
      } >> "${BASH_ENV}"

Each tool's step then reads its own flag as its first line:

- run:
    name: Test with PHPUnit
    command: |
      [ "${CI_IS_PHPUNIT_RUNNER:-1}" = "1" ] || exit 0

CI_RUNNER_INDEX and CI_RUNNER_TOTAL carry the current container's index and the container count under the same names on both providers.

To run a tool of your own on a specific container, add one more flag alongside the others and reference it from your step:

GitHub Actions

.github/workflows/build-test-deploy.yml

CI_IS_CYPRESS_RUNNER: ${{ matrix.instance == 1 }}

CircleCI

.circleci/config.yml, inside 'Set test runner roles'

echo "export CI_IS_CYPRESS_RUNNER=$([ "${CIRCLE_NODE_INDEX:-0}" -eq 1 ] && echo 1 || echo 0)"

Giving a tool its own container

Start by adding a container for the tool to move onto - the default configuration has containers 0 and 1 only. Then point the tool's flag at the new container and exclude that container from Behat. With a third container added, that is (GitHub Actions shown - mirror the conditions in CircleCI's Set test runner roles step):

CI_IS_PHPUNIT_RUNNER: ${{ matrix.instance == 2 }}
CI_IS_BEHAT_RUNNER: ${{ matrix.instance != 2 }}

warning

Point the flag at a container that exists. A flag whose condition matches no container disables the tool everywhere, and nothing reports it - the steps are skipped and the job still passes.

warning

Use the last container for this, never the first. Behat derives its profile name from the container index, and p0 is the catch-all that runs every scenario without a @pX tag. Excluding container 0 from Behat means p0 never runs and those scenarios silently stop being tested.

Balancing Behat tests

Because the first container handles Jest, PHPUnit and coverage in addition to Behat tests, it has more work to do than the other containers. To keep the overall build time low, assign more Behat scenarios to the non-first containers.

Behat scenarios are assigned to containers using profile tags. A scenario without a profile tag runs on the first container; tag it @p1 to move it to the second container; tag it @smoke to run it on every container:

Scenario: Quick smoke test (untagged, stays on the first container)
  Given I go to the homepage
  Then I should see "Welcome"

@p1
Scenario: Full content workflow (runs on the second container)
  Given I am logged in as a content editor
  ...

When only one container is available, all scenarios run there regardless of tags.

tip

As a rule of thumb, keep lightweight scenarios on the first container and move heavier or more numerous scenarios to additional containers (@p1, @p2, etc.). This keeps the total build time closer to the duration of the longest single container rather than the sum of all tests.

Note that @smoke scenarios - used to check that Behat itself is configured and works correctly - run on every runner by design, so keep them few and fast.

See Behat > Parallel runs for how the shipped profiles map tags to runners.

Adding more containers

Raising the container count takes two changes that must stay in step - the container count itself, and a matching Behat profile for every new container.

1. Increase the container count. See the provider-specific pages:

   • GitHub Actions > Change test parallelism
   • CircleCI > Change test parallelism

2. Add a profile to behat.yml for each new container that runs Behat. Vortex ships with p0 and p1 only, and Behat fails with profile 'p2' does not exist if a container running Behat has no profile named after its index. A container excluded from Behat needs no profile:

   behat.yml

   p2:
     gherkin:
       cache: '/tmp/behat_gherkin_cache'
       filters:
         tags: '@smoke,@p2&&~@skipped'

3. Exclude the new tag from the p0 catch-all, so its scenarios do not also run on the first container:

   behat.yml

   p0:
     gherkin:
       cache: '/tmp/behat_gherkin_cache'
       filters:
         tags: '@smoke,~@p1&&~@p2&&~@skipped'

4. Tag scenarios with @p2 to move them onto the new container.

Scenarios tagged @smoke run on every container by design, so they stay out of the balancing arithmetic.