The basic goal for testing any software is to isolate errors or bugs that will impact the quality or will compromise the security framework. In the case of legacy waterfall development cycles, the release cycle was long, and the software tester / QA would spend time manually simulating and testing various scenarios. However, as organizations are adopting DevOps ways of working, the frequency of release is changing from months to every hour and it would be inefficient as well as a humongous task to manually test each time. Also, manual testing might not cover all the corner cases, leaving scope for production defects.
Continuous testing in DevOps is an uninterrupted process of testing throughout the SDLC where test cases are included in every stage in the CI/CD pipeline. Right from code commit, to build, to provisioning, to deployment, and everything that happens in between—all the steps are tested automatically. In case there are any bugs encountered in the process, instant feedback is provided to developers ensuring rapid fixes of failure before the final release.
Continuous testing is beneficial in more than one way. Following are some of the benefits of CI/CD testing:
Test automation and continuous testing are often used interchangeably. However, these two are not the same.
Wikipedia defines Continuous Testing as “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” However, this definition is not accepted by many.
Lisa Crispin, an influential Agile Testing Professional, highlights that automated test is an integral part of continuous testing framework but the terms cannot be used interchangeably as continuous testing in CI/CD is not just about test automation.
Continuous testing in DevOps encompasses and is a collaborative effort involving processes, teams, and tools along with automation. It is an evolution of test automation and aims to provide validation and feedback at every step of the software delivery pipeline - right from design to production.
In a typical scenario, a developer writes a piece of code and commits it. The code then goes into version control. This is the first step in DevOps.
In the next phase, Continuous Integration, the code is merged into the main branch and then Jenkins / Git is used to build and test. Following this, as part of continuous testing in CI/CD, the QA writes automated test scripts and executes them.
As part of continuous testing in DevOps, test cases are introduced even before the developer starts writing the piece of code, i.e. during the design phase, and until delivery. If the test fails, a failure notification is sent to the developers. If the test passes, the code is pushed for Continuous Deployment.
In order to incorporate Continuous Testing into a CI/CD pipeline, utilize the steps outlined below:
Multiple types of tests should be integrated into the pipeline as it is important to have coverage on all the software aspects. Some of these tests are listed below:
These are short tests written from a developer’s perspective to verify the performance of single classes and functions i.e. the smallest individual unit of the code.
These tests are written to check if multiple classes and functions are working together as expected.
These tests are written from the user’s perspective to ensure that all software functionalities are behaving as the user expects them to.
Also known as workflow tests, these tests check the entire functionality of the application from backend to frontend. Given the large area they cover, naturally these tests tend to be fairly large in volume.
Bear in mind that the list above is not comprehensive. It touches on key testing types that are mandatory to most CI/CD pipelines. Other frequently used tests comprise API Testing, Performance / Load Testing, Stress testing, regression testing, security testing, UI testing, etc. The types of tests included in the Continuous Testing phase depend fundamentally on the needs of the application under test.
For tests to run smoothly in a Continuous Integration (CI) environment, it is important to run them independently. Let’s say that you are building an application to help find restaurants in the city. Some of the important workflows for this application would be - users should be able to search the restaurants, login into their accounts, and explore different restaurants based on selected criteria.
So while creating the test workflow, focus on creating independent test cases to verify each of the features. This will help reduce testing time and help identify failures at a feature level.
In serial CI/CD testing, tests are run in a sequence one after the other. However, in parallel testing, multiple tests run simultaneously, thus reducing overall execution time.
Keep in mind, however, that for a parallel test execution suite to run smoothly, factors like the number of processors and the capability of the processors play a very important role.
Identifying the types of tests to be included and automated in a CI/CD pipeline can be difficult. However, Agile Testing Quadrants are helpful for the teams to identify and plan the tests to be included, and the resources required for implementation.
There are four quadrants as depicted in the picture below. The bottom of this matrix is tech-focused, while the top is business-facing. The left-hand side focuses on tests that guide product development whereas the right-hand side critiques the product.
Quadrant 1 (Lower left corner) - All the technology-facing tests that guide the programming team are part of this quadrant like unit testing, integration testing, and component testing. These tests are automated and provide regular feedback to the developers on the code quality and if the software is behaving as expected. The test cases for CI/CD Pipeline Automation Testing are written alongside development to confirm that the functionality being built is moving in the right direction.
Quadrant 2 (Upper left corner) - Partially automated and partially manual, this quadrant is business focused and the tests are performed to validate product specifications, simulate or build a prototype. Automated functional testing which is done to ensure the product is aligned with the business specification is part of this bracket.
Quadrant 3 (Upper right corner) - This quadrant is focused on business-facing tests that critique the product. It primarily includes manual testing like exploratory testing, A/B Testing, User Acceptance Testing (UAT), and usability testing. These tests are more user-oriented and are helpful to assess aspects like ease of use, flaws in application workflow, etc.
Quadrant 4 (Lower right corner) - This quadrant is for technology-facing tests like performance testing, load testing, and security testing of the application. The tests in this quadrant should be prioritized based on the application-specific need.
For example, for reporting software containing non-sensitive data type information, fast refresh and loading of a page is of utmost importance. Therefore, implementation of performance testing would add value.
Assess and leverage the test or the quadrant which is relevant for your specific use case. These quadrants should be used more like a guiding framework so that the whole team understands the importance of testing and starts thinking about it right from the start. Don’t use them as a preconceived blueprint. Rather, view them as a skeleton on which to build your unique testing framework.
Created by Mike Cohn, the testing pyramid is a test automation framework that ranks the order and sequence of testing types as well as the ideal composition of each of them for better coverage.
The above pyramid calls for automation testing at three different levels. Unit testing is the foundation of automation test strategy followed by service-level testing and user interface testing. Service here refers to testing the application’s response based on the set of inputs.
Much like the Quadrant model, the pyramid should be considered a guide and not a structure set in stone. Use it to start planning your test sequences, but make sure to include tests and organize them in accordance with your specific requirements.
Like all stages on the CI/CD pipeline, Continuous Testing requires the proper tools to be implemented effectively. The keyword here is implementing test automation.
It is important to involve the complete team in the CI/CD testing process. Business stakeholders can help classify high-priority features, tests can help developers identify edge cases, and developers can help write test scripts. Additionally, keep the automation suite lean and create a strategy on how much automation is possible and what tools will help achieve that.
In keeping with DevOps principles, Continuous Testing in DevOps requires team-wide collaboration and ownership of product outcomes. After defining your automation strategy and deciding what tests to incorporate, the next step is to select the automation tools. As a first step while selecting tools, start with answering two integral questions:
1. What tests need to be implemented?
2. Which is the preferred language in which test cases should be written?
Once you have clarity on the above, start looking for automated tools that best suit team and project requirements. Once again, making this decision in tandem with the whole team so as to promote cross-functional collaboration between testers, developers, and business stakeholders.
Opsera integrates with a plethora of popular testing tools aiming to orchestrate quick, easy and effective testing into the CI/CD process. With the ability to integrate Selenium, JMeter, JUnit, Cypress, NUnit and more, Opsera makes CI/CD testing convenient and minimum-effort for developers and testers alike.
A few widely used tools ideal for Continuous Testing (all offered as integrations with Opsera) are:
Selenium is an open-source end-to-end testing framework with a user-friendly UI. It supports parallel testing and the test cases prepared can be tested using any OS. The framework also allows the testers to write complex test cases in RUBY or HTML.
Selenium is compatible with most of the popular programming languages, including Java, JavaScript, C#, Python, Perl, and PHP.
Apache JMeter™ is an open-source application that is used primarily for load and performance testing. It can be used on not only static but even dynamic resources, including Web dynamic applications.
The tool can also be used to perform automated functional testing. It also facilitates stress testing which helps identify the point where the application can fail. JMeter supports multiple plugins like basic graph and Response Times Over Time Listener.
This tool is designed on a multithreading framework, allowing concurrent sampling on multiple threads and simultaneous testing of varied functions on separate threads. You can also visualize the test results in multiple different ways like tree views, XML, graphs, JSON, etc.
Apart from the ones listed above, there are many other Devops continuous testing tools available. Notable ones among them are Nunit for C#, JUnit and TestNG for Java, and Embunit for C++ development. Choose the tool that best supports your continuous testing requirements.
Read More: Top 25 DevOps tools for 2021
While Continuous Testing can be undertaken manually, doing so would take enormous amounts of time and effort, and be heavily error-prone. Automation solves this testing faster and with greater accuracy. Automate tests as much as possible to achieve faster releases of comprehensively tested software.
Use the right metrics to track success and failure rates of tests. Continuous Testing provides immediate feedback on software performance. By keeping track of bugs, defects, the number of passed and failed tests, the team gets hard data on how the application functions at a particular stage in development.
Ensure that the team communicates openly across functions so that the test pipeline does not scatter in siloes. Without adequate communication, bottlenecks and conflicts occur, preventing effortless progression of the development process.
Each team member has to move in tandem with their teammates, aligning their individual functions with one’s own. This is only possible through active, consistent communication.
Tests must be shaped with a multi-layered approach so that all features and functions can be covered. Be it unit, integration, database, performance, load testing or more, automation must be structured to allow multiple QA processes to run. With applications becoming more complex, a multi-layer testing approach is indispensable towards handling said complexity.
Tests should be written as early as possible in the development cycles. From the very first sprint, testers should be included in brainstorming sessions so that they understand the nature of the application and write tests accordingly. Much like the source code, tests must evolve as the software is built and more features and functions are added. This is only possible if QA, as a function, is incorporated into development every stage of the way.
Moving to a Continuous Testing model isn’t exactly an easy task. Teams unaccustomed to the model might take a while to adjust to the newer methodologies, especially with regard to increased automation. The transition, however, can be facilitated with the guidelines below:
Continuous testing in CI/CD helps isolate failures and provide a critical feedback loop to move fast and build effective software solutions. Most importantly, it helps build a quality CI/CD pipeline and reduces the risk and wastage in the software development lifecycle.
There is no easier way to build and maintain a flexible and high-quality pipeline than Opsera’s Continuous Orchestration. Build drag-and-drop declarative pipelines, with security and quality piped in. Your toolchain, your preferred vendors,, with no coding or manual integrations. Unified insights help drive business decisions with aggregated and contextualized logs, personalized dashboards, and over 85+ KPIs to help measure your success.
Sign up for a demo to learn how Opsera can help you provision and manage your toolchain with no code.
