The requirements to a quality assurance (QA) engineer are sometimes difficult to define. The scope of their focuses are usually across multiple areas. As the name suggests, anything related to quality is part of their responsibility. The nature of Internet applications adds several sources of complication. For example, agile development, continuous rollout, unforeseeable traffic, just to name a few. There are a lot of things for a new QA engineer to learn, in order to be productive. It might not be difficult to start, but the path could be long, and full of confusions. This is a non exhaustive list of topics that a GOP QA engineer would learn in the journey. We hope this could be a general reference to help new QA engineers in the team, and those who have similar backgrounds.

1. Know Nothing

This is usually what happens to new QA engineers, especially those who are totally new to the industry. The best part of being a QA engineer is that no particular prior knowledge or skill is needed to start with. People are naturally talented at spotting others’ mistakes. This is good enough to kick start a QA journey. In many circumstances, QA engineer is the first user to a new product, or a new feature. As long as the purpose of the new product or feature is clearly defined, a QA engineer could act as a normal user and start testing the basic functionality. This is usually part of the “black-box testing”, where the tester assumes no prior knowledge about the system. The test cases to execute should come intuitively. For example, in GOP login services, a user would expect the following:

• Keying in a correct pair of username and password shall login successfully
• Using an invalid username or incorrect password shall fail
• Logging out and logging in again shall get back to the same account profile
• …

2. Know the Design

Although black-box testing is an essential part of the test plan, it is far from enough. In reality, a good product design would take into consideration many situations that a normal user would seldom encounter. For example, a captcha mechanism that will only be triggered after a user keys in wrong password for more than 5 times, or a block logic that disallow users to make transaction with credit cards issued in another country. To have a good coverage of the test plan, we usually need to study the product design. We need to understand the details in the design, in order to prepare and execute relevant test cases. Experienced QA engineers would also identify potential issues in the product design, and give useful suggestions on any necessary improvement.

3. Know the Implementation

The ability of a QA engineer to read and analyse the product implementation (i.e. the source code) is increasing emphasized nowadays. Many Internet applications use some sorts of client-server architecture. For example, a website has its frontend that interacts with users, collects user requests and send to the backend through APIs; while the backend handles the actual logic and returns results to be displayed by the frontend. Knowing the implementation allows a QA engineer to perform tests on the client and server separately in greater precision. With such skills, a QA engineer could proceed further from the traditional “end-to-end” testing. Many advanced testing techniques, such as integration testing and load testing, are all built upon the understanding on the source code of the application. This is also usually the base where test automation is made possible.
This is not an easy topic. The code base of a product could be huge with vast amount of details (usually business logics). This could be a challenge to QA engineers new to the product (also to new developers) at the beginning. Nevertheless, after a QA engineer has managed to gain solid understanding of product design and implementation, he would be able to operate independently in most QA tasks.

4. Know the Dependencies

Typically, an Internet application is built on top of one or more dependencies. For example, databases to store the user profiles, cache services to store temporary data, or message queue services for processing sequential events. These dependencies are very fundamental things, usually industrial solutions trusted by many companies, and decoupled with the implementation of business logics. More often than not, they are regarded as foolproof and bug-free. QA engineers rarely have to design or perform tests specifically on such dependencies. These low level services usually do not impact the business logics. However, they are the backbone of the entire service(s), and these are places where we usually find the system bottlenecks. Furthermore, the knowledge on the dependencies are general, and applicable to many other applications. So pay extra attention in case any problem regarding the dependencies is encountered.

5. Know the Relations

As the industry embraces microservices, we are going to see more Internet applications being re-factored into a collection of inter-related smaller ones. Some of such smaller applications will become shared internal services that support other applications in the same company / team. For example, GOP manages dozens of inter-related services at the moment, and we are still in the process of breaking down bulky services. In a microservices framework, developing a new feature could involve multiple services. QA engineers would have to understand what are the services involved, and how are they related to one another. This would then allow us to identify the scope of testing, and validate that everything goes as expected in each individual services. In fact, microservices are making QA engineers’ life better, because smaller systems improves testability. Thus bugs are easier to be located.

6. Know the Hacks

The definition of hacks here refers to a range of things that we do to help us efficiently test the target application. It includes preparation of test data, modification in the dependencies, creation of mocks, removal of certain connections, and anything that could effectively simulate various (sometimes extreme) scenarios for the application to handle. We need this capability to increase coverage of our test cases, in order to assess the behaviours of the application under abnormal situations. It could significantly boosts productivity, as we find short-cuts to replace some previously tedious test steps.
This is, arguably, a major milestone to new QA engineers. Knowing the hacks is an indication that a QA engineer has mastered all the topics 1 to 5 related to this particular task. Moreover, it usually also indicates that the engineer is capable of handling incidents to the application. It is likely that one who knows how to hack also knows how to rectify dirty data and wrong status.

7. Know the Legacy

Legacy logics and features are common in many applications. They are designs and logics implemented in previous versions of the application, and have been changed or deprecated. But some records and status are carried over and need to be handled specially in the current and future versions. Nevertheless, the legacy handling could be easily cornered in the iterations, and conveniently overlooked in later developments, as they are likely to have diverted significantly from the current direction. Nobody likes maintaining legacy code. However, these are places where production issues are frequently found. So it is very important to stay vigilant, and evaluate carefully if special adjustment is needed to adapt the legacy cases. It is sometimes difficult for new QA engineers to get hands on the legacy logics without referring to old documentations, or consulting experienced colleagues.

8. Know the Deployment

Trouble shooting deployment issues are usually not a responsibility of QA engineers. But it is good to know how the application is running on the servers. Knowledge on the deployment of the entire system allows QA engineers to quickly identify broken links in the application during the testing process. For example, on the development and test environments, it is common that some services become unstable at times. Knowing the deployment would allow us to bring them online quickly, or at least determine where to look for assistance. Moreover, although we do not manage deployment of the applications, we do maintain QA specific tools such as load testing services and test automation scripts. Some level of deployment knowledge is required for upgrading and troubleshooting these tools.

9. Know the Limitations

This is not supposed to be an independent topic. Instead, it is more regarded as a collection of issues gathered through the understanding of the previous topics. There could be limitations in the design, the implementation, the architecture, the legacy, etc. Sometimes a limitation could be the result of a combination of several factors in different areas. So understanding of the system’s limitations is usually built on the comprehensive knowledge of the entire system. In most cases, the limitations are not really critical, otherwise they should have been properly handled. We seldom run into the system limitations in our daily work. However, knowing the limitations allows us to foresee potential issues even before we start testing, and give feedback for optimizations at an earlier stage.
Topics 7-9 are usually very unique to a specific product, a specific team, or even a specific moment. The knowledge on such topics are usually obtained through extended involvements and experience. As a result, when an engineer has acquired substantial understanding in these topics, he shall have known almost everything related to the target application.

10. Know the Risk

The last few topics are not about technical issues, instead they are more about economics. Nevertheless, we believe they are useful to QA engineers in their everyday work. Practically, there is no test plan that could achieve 100% coverage. No test execution that is able to identify all bugs and issues in a major change. On one hand, more tests always make us more confident in the quality of the application, but never 100% confident. On the other hand, we only have limited QA resources, mostly human time, to be allocated into different tasks / products. Risk should be a primary consideration in the allocation. Risk could be defined in various forms, such as amount of monetary loss, or the number of users queries turned down due to a potential bug. The definition is highly coupled with the nature of the application itself. Though it is not clearly quantifiable, as a general guideline, the allocation shall favour the tasks with higher risk.

11. Know the Cost

Cost refers to the amount of QA resources needed in executing one test case or a set of test cases. It is another primary factor to be considered together with risk. First of all, cost does not increase linearly with coverage of tests. Their relation loosely follows the 80/20 rule, i.e. 20% of test efforts provides some 80% of coverage. After that, the cost goes up very fast as we examine into the obscure corner cases. In addition, tasks with higher risk does not necessarily imply higher cost to achieve the same level of test coverage. For example, a service could be considered having high risk just because it is processing sensitive data, while the service logic could be rather simple to test with. Again, it is not necessary and not really possible to quantify the cost to a very detailed level. But we have to have a rough judgement of the risk of each product/feature and the cost of testing it. So a group of QA tasks could be managed concurrently with a resonably efficient plan.

12. Know the Team

GOP QA engineers work in a cross-functional agile team with product managers, developers, designers, etc. The importance of collaboration and communication could not be emphasized enough. Knowing the people we are working with is the key. After we know the expertise and strength of every member, we will be able to quickly identify who to approach for help in times of need. Communications could also happen more effectively, as we can make our points clear to the teammate with sufficient (but not too many) details, and in a way he is comfortable with. We believe this is an essential skill that would help QA engineers get productive quickly at the beginning, and achieve personal growth along the journey.

Conclusion

We note down this list of topics with the intention of helping QA engineers new to the team to quickly get involved. Take it as the “getting started” manual for GOP QA engineers. The topics are organised in a way that knowledge from some earlier topics is needed for understanding later ones. However, there is no strict dependencies, and learning never occurs sequentially. Again, this list is by no means complete or comprehensive. As what we have pointed out at the beginning of this post, the interest of QA engineers covers multiple areas. So stay curious, and explore beyond the list.