This article is part of Pentera’s Engineering Series – a behind-the-scenes look at the technologies we develop to keep companies secure. In this piece, we look at the testing processes that we use to QA our platform and deliver a high-quality solution. 

It almost goes without saying that testing is a critical part of the software development process. However, many models (see figure 1) only bring testing into play near the end of the development lifecycle.

Each phase in the old model was executed according to its definition, meaning no testing at all was done along the way. The first time the QA teams were introduced to a new feature was in the fifth formal ‘testing’ phase. By then, the bulk of the work was long done.

Figure 1: Old model phases

Involving the QA team as early as possible leads to a higher quality product. (This should be true nearly anywhere; there’s no doubt that the QA teams in every company know the system thoroughly and widely, so why wouldn’t it make sense to involve them as soon as possible?)

This question led us to introduce the idea of a ‘Shift-Left’ practice. testing as soon as possible in the development life cycle lets us prevent defects in our products and improve the final quality at the end of the road.

How did we do it?

Design Meetings (Design Phase)

Every life cycle starts with a design (see figure 2).

Typically, the personnel taking part in design meetings includes the product team (Product, UX/UI, etc.) and on some occasions, some C-level executives.

Once the initial design is ready, the product team sits with the relevant developers to understand if their idea can be achieved and get a perspective on how complex it will be to create. Very seldom is QA part of these discussions.

At Pentera, our Shift-Left approach has the QA team take part in these critical early discussions. We’ve classified these meetings as TL level meetings – meaning all team leaders from the RDP (RND and Product) department are to take part. Raising defects in the design phase ensures that we get fewer defects, questions, and design changes along the other phases of the process.

Figure 2: Milestones along the life cycle

Hand-Off from Product to Dev (Development Phase)

After the design phase is done, the Feature moves to the backlog.

We work in sprints. During sprint planning, stories are selected and moved from the backlog to development; a feature owner is assigned to each feature.

Before we start working on a new feature, the feature owner will schedule a Kick-Off meeting. The participants in those meetings are those who will handle the feature in each team: Front, Back, QA and Product.

The goal of this process is to identify any defects that we didn’t encounter in the previous phase and update the feature accordingly before the developers start working on it. The more “baked” the feature is when it moves to the developers, the lower the chance it will have major bugs.

Hand-Off from Dev to QA (Testing Phase)

When the development phase is done, we conduct a hand-off event from Dev to QA. Called a “demo”, this meeting involves Front End, Back End, QA, and Product representatives.

During the demo, the feature owner will present the feature and show it working end to end.

The goal here is to get an approval from the product team and to move the feature, working as designed to QA. The Developer should verify that the system is working, but should not care as much about side effects. We take the approach that side effects and defects may be present in the surrounding, but that a feature, working according to the design, is enough to move it to QA.

From there, the QA feature owner will go deep inside the feature and its surroundings to confirm that it is working and that nothing was broken in the development phase. Then, the feature can be merged to our stable branch.

If bugs were found, the feature will go back to the feature owner. We’ll restart the development phase and go through the whole process again until all bugs are fixed and the feature is verified. 

We never merge a feature to our stable branch before it is verified by QA; this way, we keep our stable branch ready for production.

QA along the process

Not only do we continually QA our products as we develop them; we use automated tests to verify that we are always “Production Ready”. This means that we can deploy a version to production on demand at any time.

The QA responsibilities are divided along the process:

• After the ‘Kick-Off’ meeting (Hand-Off from Product to Dev), we start writing our tests files. There is nothing visual to test at this time, but when the requirements (Stories) are well-thought-out and specific, it is more than enough. Our QA team puts together an initial test plan, or STP (Software Testing Plan) immediately following this meeting. This serves as our roadmap for how to test the feature.

• When the new feature is ready for testing and has successfully been demo’ed, we will start writing the final test file, or STD (Software Test Description). This will later be automated by our automation team and integrated to our regression.

Additionally, we run a full system tests cycle every day on our “Production Ready” branch.

The Results

The Shift-Left process has dramatically increased the stability and quality of our new features and versions. It has also reduced the amount of time we spend handling defects-related issues (e.g., creating dev fixes, reproducing bugs, and answering support tickets) all around the RND.