Software testing services that drive product quality & customer satisfaction

Unit test - A test verifying methods of a single class. Any dependencies external to the class are ignored or mocked out. Note that some single class tests also qualify as feature tests in a few cases, depending on the scope of the “feature” under test.

The reason we write unit tests (in general) is to build greater confidence in the code we write. This can be used as a tool to drive the design of our code, a-la-TDD, or at least it can be used to ensure that the code we've written returns an expected output for some input.

This also gives us a much greater confidence in performing refactorings on the existing code base, as broken test cases can help us catch any change in class/method APIs, as well as potentially breaking expected return types.

We don’t consider Unit tests as 'testing-costs. We think that Unit tests should be part of 'core' engineering & a part of development. Not a task that's added to testing costs. If you aren't writing unit tests (irrespective of whether its TDD or not), you are not developing/engineering your product right. You are only building a stack of cards _hoping_ that it wouldn't collapse at some point in the future.

Integration (Feature) test - The meaning of Integration testing is quite straightforward- Integrate/combine the unit tested module one by one and test the behavior as a combined unit. A test covering many classes and verifying that they work together.

We normally do Integration testing after “Unit testing”. Once all the individual units are created and tested, we start combining those “Unit Tested” modules and start doing the integrated testing.

The end purpose of feature tests is generally much clearer than individual unit tests.

A safety net for refactoring -
Properly designed feature tests provide comprehensive code coverage and don’t need to be rewritten because they only use public APIs. Attempting to refactor a system that only has single class tests is often painful, because developers usually have to completely refactor the test suite at the same time, invalidating the safety net. This incentivizes hacking, creating tech debt

Testing from customer point-of-view - Leads to better user APIs

Test end-to-end behavior - With only single class tests, the test suite may pass but the feature may be broken, if a failure occurs in the interface between modules. Feature tests will verify end-to-end feature behavior and catch these bugs.

Write fewer tests - A feature test typically covers a larger volume of your system than a single class test.

Service as pluggable library - If setup correctly, feature tests lead towards a service design in which the service module itself is embeddable in other applications.

Test remote service failure and recovery - It’s much easier to verify major failure conditions and recovery in feature tests, by invoking API calls and checking the response.

Our approach to testing

(Unit, Integration (Feature), End-to-End)

As indicated here, the pyramid shows from bottom to top: Unit, Integration(Feature), E2E. As you can see from the test pyramid, Unit tests are the fastest and cheapest way to ensure software quality. Integration tests are slower but have higher impact on the quality of features delivered.

One thing that it doesn't show though is that as you move up the pyramid, the confidence quotient of each form of testing increases. You get more bang for your buck. So while E2E tests may be slower and more expensive than unit tests, they bring you much more confidence that your application is working as intended.

Most teams settle for the 70/20/10 rule as promoted by Google Testing which splits testing to 70% unit tests, 20% integration tests, and 10% end-to-end tests.

As a team, we want to be able to answer questions like is our application working the way we want it to, or more specifically:

Can users log in?

Can users send messages?

Can users receive notifications?

The problem with the 70/20/10 strategy which focuses on unit tests is that it doesn’t answer these questions and many other important high level questions.

Our new strategy was to split our automated tests into 40% unit tests, 40% integration tests and 20% end-to-end tests. Our approach is to write more integration tests than Unit tests so that we can focus more on feature quality. Of course, it depends on the solution and our test plan.

TDD stands for Test Driven Development approach. So why TDD? The short answer is “because it is the simplest way to achieve both good quality code and good test coverage”.

TDD starts with writing the tests, which fail until the app is written. Provided the tests are accurate, you write the app so that the tests pass.

For a simple calculator you will ultimately have one of four (or more) functions, at it's simplest you need add, subtract, divide by and multiply by. The point of TDD is to identify these four functions, not to test them. If you cannot break down your program into these simple one purpose functions then perhaps you need to either rethink your acceptance criteria or you need clearer requirements.

TDD's goal isn't *just* to ensure you start with tests, but also (mainly) to simplify your program and keep it on topic. The goal of tests isn't just to find bugs in newly-written code. It's to defend against regressions, and a regression is even harder to localize, given the team may not even be well-familiar with the failing code.

Over the years, we’ve learnt to test "what a thing is supposed to do, not how it does it". Usually this would mean to write high level tests or at least to draft up what using the API might end up looking like (be it a REST API or just a library).

This approach comes with the benefit that you can focus on designing a nice to use and modular API without worrying on how to solve it from the start. And it tends to produce designs with less leaky abstractions.

We prefer to start by writing a final API candidate and its integration tests and only write/derive specific lower level components/unit tests as they become required to advance on the integration test. Our criticism on starting with the bottom-up is that you may end up leaking implementation details in your tests and API because you already defined how the low level components work.

So, what is TDD about?It is about defining some steps or a procedure to make sure from day one that your project will abide to automated unit testing concepts and implementations. In TDD developers

- Write only enough of a unit test to fail.
- Write only enough production code to make the failing unit test pass.

What is RGR or Red-Green-Refactor in TDD?

The red, green, refactor approach helps developers compartmentalize their focus into three phases:

Red — think about what you want to develop. Create a unit test that fails.
Green — think about how to make your tests pass. Write production code that makes test pass.
Refactor — think about how to improve your existing implementation. Clean up the code.

This cycle is typically executed once for every complete unit test, or once every dozen or so cycles of the three laws. The rules of this cycle are simple.

Create a unit tests that fails

Write production code that makes that test pass.

Clean up the mess you just made.

The RGR cycle tells us to first focus on making the software work correctly; and then, and only then, to focus on giving that working software a long-term survivable structure.

Whenever developers change or modify their software, even a small tweak can have unexpected consequences. Regression testing is testing existing software applications to make sure that a change or addition hasn’t broken any existing functionality. Its purpose is to catch bugs that may have been accidentally introduced into a new build or release candidate, and to ensure that previously eradicated bugs continue to stay dead.

Usually regression testing is done repeatedly, any time a change is made, which makes it a good candidate for automated testing.

Two-level approach

In agile development, regression should be happening throughout the cycle as part of automated and unit tests, and continuously expanded to cover any new issues that arise as well as new steel-thread stories.

We follow and recommend this regression cycle -

1. Iteration regression.
The team perform iteration regression at the end of each sprint. Iteration regression specifically focuses on features and changes made in the iteration and areas of the application that could be affected.

2. Full regression.
Test engineers run full regression before releases and project milestones to ensure that the application works as planned.

Regression testing is a risk management game. Testing everything would be too costly, so we need to focus our efforts.

-Identify highest priority test cases - each sprint, prioritize all your test cases, and flag those with the highest priority. These get added to your regression test suite and prioritized in context with the other tests there.

-Start the sprint with regression - as testers, we'll typically have little in the way of new work to test at the start of a sprint. In addition to using this time to plan the testing for the current sprint, use it to run regression test cases for previous sprints. If you're starting sprint 10, you would run regression test cases for sprints 1 - 8 (because there's no changes to sprint 9 code yet). In sprint 11 you run test cases for sprints 1 - 9. And so on.

-Prioritize aggressively - If the work for the current sprint means we can only cover the ten highest priority regression test cases, then that's what we cover

-Document your regression - make sure that part of each sprint review includes what you did not regression-test and why.

- Treat your test suites as a backlog - and groom them aggressively. If a test case isn't relevant anymore, don't hesitate to retire it. If one needs updating, update it.

- Give your test cases execution time estimates - We estimate how long it takes to run each test case and plan accordingly.

- Plan test cases as part of the sprint planning - Your regression test cases are as much a part of the sprint planning as any other activity. Build them into the sprint planning session if at all possible.

Having an automated test suite with good coverage is definitely very helpful. However, we wouldn't recommend relying entirely on automated test for regression, as there are some types of bugs that automated tests aren't particularly good at detecting. We start adding automated tests for basic smoke tests flows and build up to the flows for the acceptance tests or even some functional ones in the sensible areas.

A time-efficient way of complimenting the automation effort with manual testing is to pick out a subset of manual tests based on a risk analysis of the system. Have a think about the areas of the system that are most likely to be impacted by the changes for the sprint, and only target those areas with your manual testing.

We have found this approach results in a much shorter regression testing time towards the end of each sprint, and it has been enough for the level of quality required for our projects. However, it does leave a little room for minor to trivial bugs to slip by in low-risk areas because the low-risk areas are entirely dependent on the automated tests to detect bugs. This may not be acceptable for some project standards, so just be aware of that risk.

There are a few approaches to consider.

Keep all your test cases, but alternate when you run them. Perhaps run some cases every other release, others get run with each release candidate.

Focus regression testing in areas potentially impacted by new feature development.

Metrics should determine when you're done. Test until the frequency of finding bugs falls below a given threshold.

Regression manual tests just roundup acceptance tests + functional + a few negative ones. Depending on what you are testing, maybe you would need performance & load tests also.

For best results, we usually use the manual test cases for those at-risk areas as a guide for my exploratory testing.

Acceptance Test Driven Development (ATDD) aims to help a project team flesh out user stories into detailed Acceptance Tests that, when executed, will confirm whether the intended functionality exists.

“By continuously testing for the existence of a given functionality, and writing code to introduce functionality that can pass the Acceptance Tests, developers’ effort is optimised to the point of just meeting the requirement.”

ATDD is like BDD in that it requires tests to be created first and calls for the code to be written to pass those tests. However, unlike in TDD where the tests are typically technical-facing unit tests, in ATDD the tests are typically customer-facing acceptance tests.

The idea behind ATDD is that user perception of the product is just as important as functionality, so this perception should drive product performance in order to help increase adoption. To bring this idea to life, ATDD collects input from customers, uses that input to develop acceptance criteria, translates that criteria into manual or automated acceptance tests and then develops code against those tests. Like TDD and BDD, ATDD is a test-first methodology, not a requirements driven process.

Also like the TDD and BDD methodologies, ATDD helps eliminate potential areas for misunderstanding by removing the need for developers to interpret how the product will be used. ATDD goes one step further than TDD and BDD though because it goes directly to the source (aka the customer) to understand how the product will be used. Ideally, this direct connection should help minimize the need to re-design features in new releases.

How is ATDD different from TDD?

ATDD borrows from the spirit of Test Driven Development (TDD) in that both techniques allow test cases to be written and executed (and hence fail) before even a single line of code is written.

The main difference is that ATDD focuses on testing for business user functionality, while TDD has been traditionally used to run/automate unit tests. In general, TDD is the pioneer that ATDD emulates to fulfil functional testing – however, both the techniques have the same aim: write just enough code, reduce developer efforts, build to detailed requirements and continuously test the product to ensure it meets business user expectations.

How is it different from standard Waterfall testing? ATDD is different from standard Waterfall testing because it is a test-first methodology. Standard Waterfall testing calls for test cases to be written upfront based on requirements, whereas ATDD is not a requirements driven testing process.

What are best practices? Best practices for testers following an ATDD Agile methodology include:

- Interacting closely with customers, for example through focus groups, in order to determine expectations
- Leaning on customer-facing team members, such as sales representative, customer service agents and account managers, to understand customer expectations
- Developing acceptance criteria based on customer expectations
- Prioritizing two questions:
- Will customers use the system if it does X?
- How can we validate if the system does X?

We automate tests for repeatability. We automate a test because we need to execute the same tests over and over again. Would you want to automate a test if you were only going to run it once and forget about it? Of course not! The time and effort that you spend on automating the test, you could have executed it manually.

Implementing a robust automation testing solution is no mean task and proves challenging for many companies – our dynamic and highly experienced team is amongst the top in automation testing services. With holistic focus on your business, we strategically design test processes, set up robust automated scripts, create QA automation framework and run Selenium & Mobile Apps automated test scripts for consistent and reliable coverage overall.

We use automated tests -

Manual Testing Takes Too Long

Manual Processes Are Error Prone

Automation Frees People to Do Their Best Work

Improves Test Coverage

Automation Testing Gives Feedback, Early, and Often

Saves Time

ROI and Payback

Functional testing is a process of verifying that a system performs as expected when its features are exercised by another system or directly by a user. This means that it lends itself nicely to test case and usage case definitions that can provide a stable, repeatable basis for evaluating the progress of system development.

The entire range of the development process comes under the purview of functionality verification.

Unit tests should start at the very beginning to ensure that each block of code performs its intended manipulation of inputs into desired outputs for the next module.
Integration tests assure that the unit modules connect to each other as expected and convey data and commands throughout the system per the specifications to which it was built.
Sanity checks verify that modifications and fixes applied to the code body don’t have unexpected side effects in, apparently, unrelated parts of the system.
Regression tests verify that later feature additions and bug fixes don’t undo previous efforts or interact with them to cause wholly new problems.
Usability acceptance is the actual operation of the system in the context in which it was designed to be used and is the gateway to deployment.

Exploratory Testing

For functional testing in an agile environment, we rely a lot on testing methodology called Exploratory Testing. Testers set out a certain amount of time to “explore” the software. Although this is also a form of manual testing, exploratory testing deviates from a strict workflow of planning, designing, and executing test case steps.

During this session, testers aim to understand how the software works and identify different tests to run based on that understanding. Since exploratory testing is not scripted, it often mirrors how users will interact with software in real life.

This testing method is complementary to context-driven testing, which contends that there is no “one best way” to conduct testing. Rather, it argues that testing needs to be handled differently for each project based on the context in which the software will be used.

Exploratory testing is not scripted. Rather, it’s about developing the best tests based on each unique piece of software. Because of its unscripted approach, exploratory testing often mimics how users will interact with the software in real life.

Across the board, exploratory testing follows four key principles:

1. Parallel test planning, test design and test execution

2. Specific yet flexible

3. Aligned toward investigation of potential opportunities

4. Knowledge sharing

When should you use exploratory testing?

The best instances in which to use exploratory testing include those when you are under a time constraint, as they require minimal preparation and allow for fast feedback, those when you do not have any specifications from developers, those when you need help determining what types of tests to run, and those when you want to conduct a good conscience catch-all test to make sure you didn’t miss anything when executing previous tests.

Our experience has been that automated functional tests are excellent and discovering that "something" is wrong, but less good at finding what is actually broken.

Say "log in" test fails: OK, I know auth is broken... but what part of it? Is there some JavaScript hijacking the form submission? Is the login view busted? Do we have an error in the authorization system itself? Is the LDAP server we are authorizing against down? Has someone changed the schema? Etc., etc., etc.

We do find functional tests to be a pretty valuable part of our testing toolkit, but unit tests seem to get us a bigger "bang for my buck", so functional tests tend to be something we don't spend a ton of time working on and automating or writing code for and reply on Exploratory and manual testing for that.

1. Continuous integration -> CI

Continuous Integration, shortly called ‘CI’ in DevOps is an important process or a set of processes which is defined and carried out as a part of a pipeline called ‘Build Pipeline’ or ‘CI Pipeline’.

The CI process includes,

1. Merging of all the Developers code to the main line,

2. Triggering a build,

3. Compiling the code and making a build

4. Carrying out the unit test.

So, Continuous Integration is a process of merging all the developer’s code to a central location and validating each one of their merges with an automated build and test.

What happens during CI is,

There will be a server for continuous integration which hosts the CI tool, which keeps watching the version control tool for the code check-in and as soon as, a check-in is found, it triggers the automated compilation, builds and runs unit testing along with static code analysis and a basic level of automated security testing.

The various tools to carry out the automated testing, like Jenkins, TestNG, NUnit to carry out unit testing, Sonar to carry out static code analysis, and fortify to carry out the security testing, all of these tools will be integrated with the CI pipeline.

So, the complete CI pipeline is an automated process without any manual intervention and runs within a few seconds or minutes.

So, the major benefit of the CI is the rapid feedback that the developers get within no time.

1. The CI runs after developer checks in the code and throws out the results in seconds. So, it allows the developers to know immediately if his or her code has successfully built or broken.

2. It also lets the developer know if his code has successfully integrated with the other’s code or broken, that something that another team member has done to a different part of the code base. Hence, CI does the quicker code analysis and makes the later merges simpler and error free.

What are the Benefits of CI?

Continuous Integration aims to have a drastic drop in the degree of errors during software development through feedback mechanisms, automation, and quick bug fix turnaround.

Although it may seem too ambitious for a process to achieve all of this, it can certainly be a reality with some of the continuous integration best practices described below:

#1) Shared repository to maintain code: With Agile evolving rapidly, it is a given that there are multiple developers working on different or same features of a product. It is therefore absolutely necessary to have one repository that will be able to capture the timeline of changes that all developers are making.

The entire CI process is an automated process and hence minimizes the human error by reducing the long, bug-inducing manual merges.

#2) Frequent code-commits: Any number of people can check in their code, any no of times in a day, without waiting for others to complete their coding, wait till they finish their check-in and later check-in. So, CI removes dependency or removes the waiting time of others checks in.

Thus, team members need not have to wait for the other team members to finish their check-in and hence allows to work in parallel.

#3) Quick Build time: The entire process of compiling, building and testing runs in few seconds and hence quite quicker and faster and saves a lot of time and hence helps to achieve the DevOps objective of delivering faster over a period of few hours.

The feedback on individuals’ code is very quick and we need not have to run around to find out whose code has broken the build or induced the defect, as with every check-in it gives the success or failure output indicating the area of failure if there is a failure.

#4) Staging builds: In order to expedite the build process, the build pipeline could be broken down into smaller chunks and executed in parallel.

#5) Create a duplicate of Production System: As the testers, we’re so familiar with environment-related defects. Production systems have their own configurations in terms of database levels, Operating system, OS patches, libraries, networking, storage, etc.

It is a good practice to have the test environment as close as possible to prod system, if not an exact replica. Any major discrepancies and risks can be easily identified this way before it actually hits production systems.

#6) Automating Deployment: In order to get to the point of running different kinds of tests in a CI model, the test setup has to be done as a prelude. As a best practice, you could have scripts that automatically setup the needed runtime environments for testing.

#2) Continuous Delivery

Continuous Delivery is the next step after Continuous integration. The goal of Continuous Delivery is to push the application build into production as quickly as possible. During this process, it goes through various stages in the lifecycle of delivery i.e. QA, Staging, Production environments etc.

This process of regularly delivering the applications built into various stages is known as Continuous Delivery.

Continuous delivery helps in quicker time to market when compared to traditional methods, lesser risk, lowering the cost by encouraging more automation in the release process and most importantly getting faster feedback from the end users to produce a quality product.

In the above diagram, you can look at different environments available and so this provisioning of the infrastructure for environments can also be automated during this continuous delivery process.

#3) Continuous Testing

Continuous Testing is the process of running various types of automated tests starting with CI process till the time the application is finally deployed to production.

So one can note here that continuous testing as an activity starts from the CI stage itself and is a very mandatory step throughout the continuous delivery process.

Continuous delivery also involves certain manual tests and gates, wherein certain tests are carried out manually, before pushing into production.

These intermediate quality gates at every stage of testing and increases the confidence in the code.

So, the continuous testing pipeline as such includes unit testing along with preliminary automated security verifications. Then gets into an integration level of testing, where automated integration tests are run, then on to a system level where system-level scenarios are automated and run.

Here even certain performance test scenarios are also carried out.

Then goes to the ‘Acceptance testing’ which basically includes the automated site acceptance test cases and then finally on to the ‘User Acceptance Testing’ which could be a manual execution and includes end-user participation to carry out the tests and this will be a kind of final sign off on the product or a feature, where manual gate is invoked and finally deployed on to the production site.

Continuous testing requires integrating automation framework with the version control and CI tool and the various automated tools to carry out the functional and non-functional testing across different phases of testing, like:

Sonar for static code analysis,

Fortify for secure code analysis,

Selenium for functional testing,

Load runner for load testing etc.,

#4) Continuous Monitoring

As the application or changes are deployed to the production environment the operations team will look to monitor the application and environment from an up-time, stability, availability point of view. This process is known as Continuous monitoring.

The operations teams will have their own software to monitor the environment but will also need to play their part to monitor the applications deployed for any issues. For this, they would need to work with the development teams in order to build certain tools for analyzing the application issues.

So infrastructure, environment, and applications issues are all that monitored in the process of continuous monitoring.

Microservices attempt to streamline the software architecture of an application by breaking it down into smaller units surrounding the business needs of the application. The benefits that are expected out of doing so include creating systems that are more resilient, easily scalable, flexible, and can be quickly and independently developed by individual sets of smaller teams.

This results in a number of benefits over a traditional monolithic architecture such as independent deployability, language, platform and technology independence for different components, distinct axes of scalability and increased architectural flexibility.

The challenges of testing microservices

Testing microservices is hard. More specifically, end-to-end testing is hard, and that’s something we’ll discuss in greater detail in this article.

One important issue we have to keep in mind while working with microservices is API stability and API versioning. To avoid breaking applications depending on a service, we need to make sure we have a solid set of integration tests for microservice APIs and, in case of a breaking change, we have to provide a backwards-compatible way for clients to migrate to a new version at their own pace to avoid large cross-service API change rollouts.

Here are a few key challenges associated with testing microservices:

Availability: Since different teams may be managing their own microservices, securing the availability of a microservice (or, worse yet, trying to find a time when all microservices are available at once), is tough.

Fragmented and holistic testing: Microservices are built to work alone, and together with other loosely coupled services. That means developers need to test every component in isolation, as well as testing everything together.

Complexity - There are many microservices that communicate with each other. We need to ensure that every one of them is working properly and is resistant to slow responses or failures from other microservices

Performance - Since there are many independent services, it is important to test the whole architecture under traffic close to production.

How we test Microservices -

Components Tests With Hoverfly

In combination with building a service outside-in we also work on component-level testing. This differs from the integration testing in that component testing operates via the public API and tests an entire slice of business functionality. Typically the first wave of component tests utilise the acceptance test scripts, and these assert that we have implemented the business functionality correctly within this service

Hoverfly's simulation mode may be especially useful for building component tests. During component tests, we verify the whole microservice without communication over a network with other microservices or external datastores. The following picture shows how such a test is performed for our sample microservice.

Contract Tests With Pact

The next type of test strategy usually implemented for microservices-based architecture is consumer-driven contract testing. In fact, there are some tools especially dedicated for this type of tests. One of them is Pact. Contract testing is a way to ensure that services can communicate with each other without implementing integration tests. A contract is signed between two sides of communication: consumer and provider. Pact assumes that contract code is generated and published on the consumer side, and then verified by the provider.

Pact provides a tool that can store and share the contracts between consumers and providers. It is called Pact Broker. It exposes a simple RESTful API for publishing and retrieving pacts, and an embedded web dashboard for navigating the API. We can easily run Pact Broker on the local machine using its Docker image.

Performance Tests With Gatling

An important step of testing microservices before deploying them to production is performance testing.

Performance testing of a series of core happy paths offered by the service. We typically use JMeter(often triggered via the Jenkins Performance Plugin) or Gatling

Gatling is a highly capable load testing tool written in Scala. That means that we also have to use Scala DSL in order to build test scenarios.

Testing - Selenium, Saucelabs, Appium, Mocha, Katalon, SoapUI, Gatling, JMeter, Hoverfly

DevOps - Jenkins, CircleCI, TravisCI, Codeship, Gradle