Running Acceptance Tests Multiple Times to Help Diagnose Intermittents

Testing asynchronous systems is hard

Most websites nowadays are asynchronous in some way, usually through the use of AJAX to load data. Single page web apps like our Companion mobile web app for events are very hard to acceptance test because of this.

We have built up a library of Selenium based code over time that really helps us test these kinds of systems. This is mainly about making sure that you always wait for something to happen rather than just quickly check for something once.

However, with any complex asynchronous system, you can intermittent failures sometimes where it'll work in one environment and not in another due to speed or timing issues. This could be just 1 in 100 times. But if you have 1000 tests that all behaved like that you'd never get a clean build as you'd have 10 different tests failing in every build!

You will often just run a single acceptance test multiple times whilst developing that piece of functionality, but once it is passing, it is all too easy to just consider it done. Through Eclipse, it is very hard and time consuming to press the rerun test button 100 times, so we need an easier way to detect intermittents.

Annotations and custom JUnit runner to the rescue

Borrowing some ideas from the tempus-fugit library, we were able to come up with an annotation and a custom JUnit test runner that meets our needs.

We can now add a few annotations whilst developing an acceptance test to help iron out these problems:

@RunWith(RepeatingTestRunner.class)
public class MyAsyncPageAcceptanceTest {

    @Test
    @Repeating(count=100,stopOnFailure=true)
    public void someIntermittentTest() {
       // test some stuff
    }

    @Test
    public void testThatDoesNotNeedRepeating() {
       // test some other stuff
    }
}

This is particularly useful when you have a test class with a lot of tests in and you just want to run one of them lots of times. Other options like the parameterised runs in junit force you to run the whole test class a number of times, not just the one test. Also, this method will run the tests for up to x times, but will stop when you get a failure, allowing you to then see the details of that failure.

Note that we'd only do this whilst developing the acceptance test as we don't want to spend the time running tests 100 times in our continuous integration environments.

As you can see, it's easy to just run a single test a lot of times locally, spending just a bit of extra time up front to save everyone in the future from battling to detect an intermittent test that could have been discovered early on.

The code

package uk.co.blackpepper.serenity;

import static java.lang.annotation.ElementType.*;
import static java.lang.annotation.RetentionPolicy.RUNTIME;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Target({METHOD, TYPE})
@Retention(RUNTIME)
public @interface Repeating {
    int count() default 50;
    boolean stopOnFailure() default true;
}
package uk.co.blackpepper.serenity;

import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;

public class RepeatingTestRunner extends BlockJUnit4ClassRunner {
    private final Class<?> type;

    private boolean stop = false;

    public RepeatingTestRunner(final Class<?> type) throws InitializationError {
        super(type);
        this.type = type;
    }

    @Override
    protected void runChild(final FrameworkMethod method, final RunNotifier notifier) {

        if (shouldStopOnFailure(method)) {
            notifier.addListener(new FailTriggerRunListener());
        }

        for (int i = 0; i < repeatCount(method); i++) {
            if (stop) {
                return;
            }
            super.runChild(method, notifier);
        }
    }

    private int repeatCount(final FrameworkMethod method) {
        if (intermittent(type)) {
            return repetition(type);
        }
        if (intermittent(method)) {
            return repetition(method);
        }
        return 1;
    }

    private boolean shouldStopOnFailure(final FrameworkMethod method) {
        if (intermittent(type)) {
            return stopOnFailure(type);
        }
        if (intermittent(method)) {
            return stopOnFailure(method);
        }
        return false;
    }

    private static boolean intermittent(final FrameworkMethod method) {
        return method.getAnnotation(Repeating.class) != null;
    }

    private static boolean intermittent(final Class<?> type) {
        return type.getAnnotation(Repeating.class) != null;
    }

    private static int repetition(final FrameworkMethod method) {
        return method.getAnnotation(Repeating.class).count();
    }

    private static int repetition(final Class<?> type) {
        return type.getAnnotation(Repeating.class).count();
    }

    private static boolean stopOnFailure(final FrameworkMethod method) {
        return method.getAnnotation(Repeating.class).stopOnFailure();
    }

    private static boolean stopOnFailure(final Class<?> type) {
        return type.getAnnotation(Repeating.class).stopOnFailure();
    }

    private class FailTriggerRunListener extends RunListener {

        @Override
        public void testFailure(final Failure failure) throws Exception {
            super.testFailure(failure);
            stop = true;
        }

    }

}

This site uses cookies. Continue to use the site as normal if you are happy with this, or read more about cookies and how to manage them.

X