Run Selenium tests at scale using AWS Fargate

This article demonstrates an approach for running Selenium tests at scale for low cost by utilizing AWS Fargate Spot to run tests without having to manage and orchestrate their containers.

Selenium framework

Integration tests, as defined by Martin Fowler, “determine if independently developed units of software work correctly when they are connected to each other.” The Selenium framework is an open source suite of automation testing tools based on the JavaScript framework. It can run integration tests directly on different target browsers, such as Chrome or Firefox, and drive the interactions on the web page without any manual inputs.

Tools such as Selenium eliminate repetitive manual testing that consumes lots of time and effort. Automated testing conforms to the ideas of Agile and DevOps, which endorse a continuous delivery workflow. Selenium has become a popular tool for testing, because it is open source and meets the requirement of quick and reliable testing, which helps enterprises save time and money.

What does a testing workflow look like?

Let us consider a scenario where a developer has added a new feature or made a bug fix to an existing product and checked in the changes to Git. As part of the CI/CD pipeline, the following manual and automated steps would typically occur:

• Unit tests are run and then the new code gets built.
• The built code gets deployed in a QA or staging environment.
• Once the deployment is healthy the QA team will run the integration and regression test cases to certify the build.

In a typical case, the number of test cases may vary between a handful to a few thousand depending on the application. Executing these test cases can take lots of time, which slows the process of getting features deployed in production.

The bottleneck when using automated testing is hardware resources, such as virtual machines (VMs) and vCPU, because you can run as many tests as your hardware will support. The traditional ways to accelerate this test case execution are:

• Start more of these test case executions in parallel from multiple machines, monitor them, and collect results.
• Add more physical machines or VMs, so you have enough resources for the browser agents to run these tests.

These methods are not scalable and cost both money and time. The approach outlined in this blog post using Fargate Spot to add more resources in parallel to overcomes these specific problems.

Proposed architecture

The following shows a high-level view of all the components:

Selenium Hub is the central point in Selenium that routes the JSON test commands to Fargate and is used to parallelize and distribute the load between the browser agents for test case execution. The whole infrastructure gets deployed in an ECS cluster with the following strategy for the default capacity provider:

• FARGATE -> Base: 4, Weight: 1
• FARGATE_SPOT -> Weight: 4

Note: Out of five instances, four of them will get provisioned as SPOT, and one of them will get provisioned as ON_DEMAND.

In this setup, Selenium Hub, Chrome Node (Selenium Docker image with headless Chrome), and Firefox Node (Selenium Docker image with headless Firefox) are deployed as ECS services. All ECS services have autoscaling enabled, with the following scale-in and scale-out policies:

• Add one instance if max(CPUUtilization) >= 70 in last 1 minute.
• Remove one instance if max(CPUUtilization) <= 30 in last 1 minute.

The Selenium Hub is backed by an application load balancer to which WebDriver clients connect to run the tests, and CloudWatch logs enable observability for any errors. The code in this repository is written as an AWS Cloud Development Kit (AWS CDK) construct.

Note: Constructs are the basic building blocks of AWS CDK apps. A construct represents a “cloud component” and encapsulates everything AWS CloudFormation needs to create the component.

Benefits of using Selenium on Fargate

Key benefits of this approach are:

• Depending upon the number of concurrent execution, ChromeNode and FirefoxNode will scale out and in automatically (as it impacts the CPUUtilization metrics), so customers pay only for the duration they use (no standing cost).
• Customers can deploy different browsers or different versions of the same browser depending upon their business need without thinking about the infrastructure or cost.
• Based on the capacity provider strategy, most of the instances get provisioned as FARGATE_SPOT, which costs less money, to facilitate quicker execution and quicker scaling with more nodes.
• With this approach, customers can now run the regression and integration test cases as part of their nightly builds, which enables a daily release cycle to support growing business needs.

Headless browsers

A headless browser refers to a web browser without a user interface. In other words, a headless browser can access the web page, but the GUI is hidden from the user. Otherwise, headless browsers are just like other browsers, and we use them to run WebDriver tests. Although headless browsers have many advantages, such as faster page loading, they also have limitations, including:

• Due to its faster page loading ability, sometimes it is difficult to debug issues.
• Real browser testing includes performing test cases in the presence of GUI. These tests are performed in front of the user, so the user can interact with the team, referring to the GUI and discussing where changes or corrections are required. In these cases, headless browsers cannot be used.
• Because headless browsers don’t have a GUI, it is troublesome to report errors with the help of screenshots. A conventional browser helps present defects by generating screenshots, which are a must in testing.
• In the case where a lot of browser debugging is required, the use of headless browsers can be challenging.

Test case execution

Here is the sequence of events that happens when we execute a test case using this architecture:

In this setup, the WebDriver client can talk to Selenium Hub with test cases results via the application load balancer URL. Once the request is received, Selenium Hub will direct the request to the Firefox Node or Chrome Node running as an ECS Fargate task to process the request. The browser node will then launch the browser in headless mode and execute the tests.

Build and deploy

The following prerequisites will be needed:

• AWS CDK should be installed for testing. You can read more about it in the Getting started with the AWS CDK documentation.
• Yarn needs to be installed; you can check the installation status by running the following command:

  npm install -g aws-cdk
  cdk --version 1.87.1

• If Yarn is not installed, run the following command:

  npm install -g yarn
  yarn version
  >1.22.10

• An AWS account and console access are also required.

Deploy

Check out the code from this repository using this command:

git clone https://github.com/aws-samples/run-selenium-tests-at-scale-using-ecs-fargate
cd scaled-test-execution/

Because the code is created as an AWS CDK construct, the following parameters can be customized as part of the deployment:

Next, run the following command to start the deployment:

cdk deploy --require-approval never

Note: Once the deployment is successful, you should see the Selenium-Hub-DNS in the CfnOutput.

The complete Selenium Hub load balancer URL will look like:

http://<>:4444/wb/hub

Testing

Unit testing

Unit test cases can be executed by running the following command from the root directory:

yarn test

Output

$ npx projen test
? test | rm -fr lib/
? test » test:compile | tsc --noEmit --project tsconfig.jest.json
? test | jest --passWithNoTests --all --updateSnapshot
PASS  test/hello.test.ts
✓ create app (730 ms)

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files |     100 |       72 |     100 |     100 |
 index.ts |     100 |       72 |     100 |     100 | 61-70
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        5.163 s
Ran all test suites.
? test » eslint | eslint --ext .ts,.tsx --fix --no-error-on-unmatched-pattern src test build-tools .projenrc.js
✨  Done in 17.45s.

Integration testing (using WebDriver)

A sample WebDriver test case can be found under sample-test-function folder, and we can run the following commands to build and execute the tests against the Selenium Hub load balancer URL:

cd sample-test-function && npm install
npx wdio --hostname <>

The test case will navigate to google.com and perform a search.

Output

All test cases should successfully pass, and the results should look like:

[chrome 87.0.4280.88 linux #0-0] Running: chrome (v87.0.4280.88) on linux
[chrome 87.0.4280.88 linux #0-0] Session ID: 80329f4643463a93a4628a35de4aaab4
[chrome 87.0.4280.88 linux #0-0]
[chrome 87.0.4280.88 linux #0-0] Play with google
[chrome 87.0.4280.88 linux #0-0]    ✓ navigate to the site
[chrome 87.0.4280.88 linux #0-0]    ✓ start a search
[chrome 87.0.4280.88 linux #0-0]
[chrome 87.0.4280.88 linux #0-0] 2 passing (8.2s)


Spec Files:     1 passed, 1 total (100% completed) in 00:00:10

Load testing (using webdriver)

Scale up

To simulate load testing, we ran the above mentioned test case in parallel (closer to 10 concurrent session), which made the CPUUtilization go above 70 percent, resulting in autoscaling.

The following screenshots were captured using Container Insights and AWS ECS console.

AWS ECS console (ECS tasks):

AWS ECS console (ECS service):

Container Insights (ECS cluster):

Container Insights (ECS tasks):

Container Insights (Map view):

Scale down

After successful test case execution, the cluster will automatically scaling down when the CPUUtilization goes below 30 percent with a cool-down interval of 180 seconds. The following shows a preview of ECS services running with just one instance after scaling down to the desired capacity:

Cleanup

Run the following command from the root directory to delete the stack:

cdk destroy

Conclusion

We have shown how a combination of AWS Fargate and Spot instances can help customers run tests in scale with low cost. This approach also allows customers to innovate quickly, fail fast, and reduce their release cycles to introduce new features into their product.

Resources

• Selenium
• Selenium Grid Using Docker
• Docker Selenium
• AWS CDK constructs