Errors and Bugs
When software do not behave correctly we say that we found an error or bug. This is not only when the program crash and does no longer work. If the program works but produce bad results or behave unexpectedly that is an error. So an error is considered when the program do not follow the specification. Or if there is no specification the program has errors if do not satisfy the customer requirements.
In the past when computers use to have contacts an IBM operator has found a mysterious bug between the contacts. This is how the first Bug was discovered. Since then every time an error is found in a program we say “there was a bug”. It is just an expression. A software bug do not have legs. It is just another name for a hidden defect found or not found. A program has bugs if do not behave properly.
Manual testing: The most simple and direct method of testing. This is also the most expensive. It is a reliable method if the team has good testers. That’s right there are persons that do not do anything else in a team but perform testing. If a team is Agile team then the testing is performed by developers. The manual testing can be based on a testing plan.
Automatic testing: Is less popular but much more efficient than the manual testing. For this a team can use a tool that can run jobs at regular time interval. Each job perform a specific task that represents one or more test cases. The task is usually based on input data or initial state of the system. After a series of actions run automatic the final result is evaluated. If the result is unexpected we have found a defect.
A software system is designed to perform functions or has features. When we use the system in a predefined way we have a workflow or a series of actions we perform to reach a result or system state. This series of actions is called use-case. So a system is designed to cover one or more use-cases. When we test one of the use-cases we say that we have a test case. So this is a test case: testing a set of system features.
A test suite is a series of test cases performed in a specific order for one time or multiple times. The advantage of a test suite is that we have a name for it and a description. So all developers and testers can talk about the same set of use-cases. In other words a test suite is a collection of test-cases.
When a test is executed we must record some data. The name of the test suite or the code, the date/time of test start and the date/time of test end. We must also record if test is completed normally or it crash unexpectedly. A good test usually performs all the test cases and reports the number of cases who pass and the number of cases that failed.
It will be kind of difficult to find a test error if we can’t trace the error back to the exact location and situation leading to the error. For this we need very precise logs with each use-case. How much time it took to execute? What is the exact case code that failed? How hard was the failure: error, warning, fatal error? maybe other important data like: program name and line number.
Saving test results
Every time a test run we must save the logs and the results in organized system. Sometimes we store this data for a month or even more then a month. We need to know if a particular test keep failing. Having this data can reveal a reoccurring bug. We can check when a bug was first found and when it was fixed. For short we can do data mining to analyze past testing results.
It is an automatic testing paradigm. Continuous integration is happening immediately after one developer has modified the program in any way. This is done on CI server. On this servers one or more jobs will be triggered when a check-in is executed. Usually a tool like Jenkins is managing this test server. Jenkins can organize multple test suites and schedule test jobs.
Dry-runs and smoke tests
A dry-run is inspired from the car test. We just start the engine and observe. If the engine sound right and there is no smoke out anywhere then we can continue with the next test. We can switch the gears and start moving the car. If there is smoke we can’t continue any test, we have to turn off the car and check the engine. The smoke test can also apply to repairs of electric appliances or home electronics. If we smell smoke when we turn on the appliance we stop doing other test.
What a dry-run means for software: We just run the software with no data to process. For example we start the software and stop the software. Or for example we create an empty database and we run the software with no data. We execute several use-cases that are considered dry-run. The program should not crash just because there is no data present.
A smoke test is very similar to a dry-run. Smoke test has some data maybe one or two records in each table. The smoke test and also the dry run must be very fast. This test is run very often for continuous integration.
Stress test & nightly test
This kind of test is executed maybe once every day at a particular hour when there is silence. For example during the night when no developer will commit changes. Sometimes this test can be also known as nightly test. During this test we can use a significant data-set that can be useful to measure the performance of the system. The test results of nightly test are used as baseline for next day to fix bugs and performance issues.
A fixture is defined for a good test. This is the initial state of the system or the initial data-set. We must be able to establish initial fixture before we can run a test. For example a database can be back-up before test start. We run the test-cases, store the logs and keep this database active for developers to check. Next time we create a new database using the same fixture. We run again and see the differences between the new database and the previous database.
Negative test cases
We can have two kind of tests: Positive and Negative. When we create a test plan we can design both cases. A positive test case is based on good data and expect good results. The negative test case is based on bad data and system must not crash but elegantly detect the bad situation and graciously report the errors and continue. If the system do not detect the bad case we have found an error.
Unit testing and TDD
The unit testing is done using a function of the application. Unit testing require software developers to create test functionality inside the application. So each unit can be tested if we create a unit test. The unit test is implemented in the application. For each small unit or function we create an additional program that can test this function. So the test and the application compile together. The test will remain in the application. This technique is called Test Driven Development (TDD).
The functional test is more general then the unit testing. For functional testing we focus attention on the functionality of the application. We verify user interface and check if all user actions have the right effect or result. This is harder to achieve then the unit testing. Usually the functional testing is not automatic and require user interaction.
End to End Testing
Usually every software component is individually tested. This is good but the components must also work together. We can test this with a larger test suite that simulate a full workflow from start to end. This is also known as integration test. It is used to verify that all components collaborate well and there is no communication issue between compoents.
Congratulations now you know what is software testing 🙂