Selenium Screenshots in Continuous Integration

There is nothing worse that when your tests fail only on your continuous integration server. This non-deterministic behaviour can be a pain to debug. This is compounded when testing with Selenium. If you can't see the state of the browser when a test fails how can you fix the issue?

This week we had an issue with a test that kept failing only on CI. In order to fix the issue, I ended up writing a jUnit test watcher that will take a screenshot of the failure and email it out to our team.

I've talked about our automated acceptance testing before. http://broonix-rants.ghost.io/automated-acceptance-tests/. Lubos, from our development team, wrote a nice how-to that replicates a minimalistic version of our setup. https://lkrnac.net/blog/2016/01/run-selenium-tests-on-travisci/. The code below builds on this setup.

Our setup is pretty standard. Our app (Java/React) is built with Gradle on Travis-ci. While this how to will focus on our setup. It should be adaptable to any other build system.

I'm going to use Lubos' code from the link above as a starting point. Here is a basic test class that uses Selenium.

We'll implement screenshot capture using a jUnit test watcher. Our watcher will look to see if we are running on Travis-ci and if the failure is Selenium related. If it is a screenshot will be taken (if possible) then emailed out to the team.

So far this has proven a rather useful addition to our test setup.