At Ny Media we’re quality-oriented and our main objective is to provide secure and reliable solutions, giving our clients all tools they need to run successful online projects. To ensure the quality of solutions and meeting the client’s acceptance criteria we’re developing test cases to cover all critical parts of their business logic. In order to achieve this, we are facilitating different testing frameworks, just to name a few:

• PHPUnit - for unit testing
• PHPStan - for static code analysis
• Behat - for user-story testing and acceptance criteria

This is not a complete list (that’s material for a separate blogpost) but should at least provide some perspective on what is our testing stack which varies between different projects.

Travis CI

For many years we were using travis-ci.com, which is a paid version of travis-ci.org - the popular among FOSS (free and open-source software) maintainers, a continuous integration platform, that allows running customized test cases on your private Github repositories. For us, one of the reasons to become paying customer of Travis is to support the company that was promoting FOSS. Many of us, developers here at Ny Media, were using Travis on daily basis for our own open-source side-projects, so integrating Travis into our company workflow was the only logical thing at that time.
Unfortunately, lately, Travis became highly unreliable both to their customers but also their own employees. They have also made it impossible for FOSS maintainers to keep using their services. Therefore we decided it’s time for a change.

Github Actions

Since we’re using Github as our main remote git repository, the natural choice for us was to explore Github CI (aka Github Actions) capabilities. Github has been working hard the last couple of years since the initial announcement of Github Actions to provide everyone with access to their product.

I’d like to name a couple of features that immediately caught our eyes:

1. GitHub-hosted runners
   You don’t need a custom infrastructure to run your tests - Github can provide it for you at a reasonable price. All our tests are running on linux-based platforms and the price, at the time this blog post was created, is 0.008 USD per minute. There is some package of minutes included in your Github plan. Moreover, all public repositories are free to run Github Actions!
2. Self-hosted runners
   You can run your tests on Github infrastructure or use your own infrastructure - either physical or virtual. And the best part is - you don’t need to pay extra for utilizing that infrastructure as a Github test runner
3. .“Unlimited” concurrency
   The number varies depends on your plan but it is quite generous even for Free accounts (20 concurrent runners). At the time of writing this blogpost Travis charges 249 USD for 5 concurrent runners.
4. Powerful infrastructure
   A few years back Github was acquired by Microsoft, and as a side effect of this acquisition, it got access to cloud infrastructure - Azure. Therefore they can provide a quite powerful infrastructure at reasonable prices making it hard to beat.
5. Exceptional documentation and support
   Github is quite great at documenting all features regarding the new platform. Even if the moment of doubts I’ve been able to reach out to the support team and get my answers within the same day.

Migration

Since our tests on Travis were running within a dockerized environment and not directly on the worker instance, we were able to finish the migration within one day. It required the following steps few steps.

Create .github/workflows/test.yaml file

Here’s the example file you can use as a template:

name: Run tests
on:
  push:
    branches:
      - develop
      - master
  pull_request:
    branches:
      - develop
      - master

jobs:
  build:
    name: Run tests
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v2
      - run: /bin/bash run-tests.sh

In the example above run-tests.sh script represents all the steps your test suite requires to execute. It may need to build dockerized environment, it may need to download all dependencies, run test scripts, it’s up to you. In our case it all the above.

I want to highlight 2 things in the template above. This workflow will only be triggered when you push to branches master and develop or if you open a Pull Request against those 2 branches and push some commits to the branch associated with such Pull Request.
Here’s what you’ll be seeing in your Pull Requests

You can see 2 steps being run as a part of this job. Please mind the difference betwee keywords used for each of those steps. The run marks what script or sequence of commands should be run on your runner host.  The uses provides a way of utilizing from the whole marketplace of pre-cooked actions. The one used in the template - actions/checkout@v2 - is provided directly by the Github team. It allows your project to be cloned in a certain directory (by default the current one) and the commit which triggered the build will be checked out. Very simple, yet powerful - more about it in our future blogposts.

Make sure checks passed before merging the Pull Request

Do that by setting up the protected branch in repository Settings -> Branches

Disable Travis

Now that your tests are running using Github CI you can safely remove all references to Travis in 3 easy steps:

1. Cancel your plan. If you’re paying for Travis just go to https://travis-ci.com/plans and switch to plan Free.
2. Go to https://github.com/organizations/yourorganization/settings/installations and remove the Travis App
3. Delete .travis.yml from all repositories that you used to test with Travis. You don’t need that anymore.

Summary

Are we happy with the migration? Yes, very much so. Without making any change to how we run tests (initially, we started with 1:1 test migration from Travis to Github CI) our test times were decreased by ~20% due to much more powerful test runners on Github vs. Travis. In addition, Travis was limiting us to a certain amount of runners that can run concurrently (depending on pricing plan). That restriction was increased by the order of magnitude when we migrated to Github CI, giving our developers much quicker feedback. On top of that, despite the increased performance, we’re actually paying less for using Github CI compared to Travis.

But this is not the end of this story. We have barely scratched the surface of possibilities when it comes to utilizing Github CI API. In the next blog post, we’ll talk about what can you do to further improve your workflow and optimize time spent on testing. Stay tuned.