In terms of timing and frequency, there are two general approaches to integration: late and one-time, early and frequent.

Late and one-time: wait till all components are completed and integrate all finished components near the end of the project.

Early and frequent: integrate early and evolve each part in parallel, in small steps, re-integrating frequently.

Big-bang integration: integrate all components at the same time.

Incremental integration: integrate few components at a time. This approach is better than the big-bang integration because it surfaces integration problems in a more manageable way.

Based on the order in which components are integrated, incremental integration can be done in three ways.

Top-down integration: higher-level components are integrated before bringing in the lower-level components. One advantage of this approach is that higher-level problems can be discovered early. One disadvantage is that this requires the use of stubs in place of lower level components until the real lower-level components are integrated to the system. Otherwise, higher-level components cannot function as they depend on lower level ones.

Bottom-up integration: the reverse of top-down integration. Note that when integrating lower level components, drivers may be needed to test the integrated components  because the UI may not be integrated yet, just like top-down integration needs stubs.

Sandwich integration: a mix of the top-down and the bottom-up approaches. The idea is to do both top-down and bottom-up so as to 'meet' in the middle.

Unit testing : testing individual units (methods, classes, subsystems, ...) to ensure each piece works correctly.

In OOP code, it is common to write one or more unit tests for each public method of a class.

class Foo{
    String read(){
        //...
    }
    
    void write(String input){
        //...
    }
    
}

class FooTest{
    
    @Test
    void read(){
        //a unit test for Foo#read() method
    }
    
    @Test
    void write_emptyInput_exceptionThrown(){
        //a unit tests for Foo#write(String) method
    }  
    
    @Test
    void write_normalInput_writtenCorrectly(){
        //another unit tests for Foo#write(String) method
    }
}

import unittest

class Foo:
  def read(self):
      # ...
  
  def write(self, input):
      # ...


class FooTest(unittest.TestCase):
  
  def test_read(sefl):
      # a unit test for read() method
  
  def test_write_emptyIntput_ignored(self):
      # a unit tests for write(string) method
  
  def test_write_normalInput_writtenCorrectly(self):
      # another unit tests for write(string) method

A proper unit test requires the unit to be tested in isolation so that bugs in the dependencies cannot influence the test  i.e. bugs outside of the unit should not affect the unit tests.

Stubs can isolate the SUT from its dependencies.

class Logic {
    Storage s;

    Logic(Storage s) {
        this.s = s;
    }

    String getName(int index) {
        return "Name: " + s.getName(index);
    }
}

interface Storage {
    String getName(int index);
}

class DatabaseStorage implements Storage {

    @Override
    public String getName(int index) {
        return readValueFromDatabase(index);
    }

    private String readValueFromDatabase(int index) {
        // retrieve name from the database
    }
}

Logic logic = new Logic(new DatabaseStorage());
String name = logic.getName(23);

@Test
void getName() {
    Logic logic = new Logic(new DatabaseStorage());
    assertEquals("Name: John", logic.getName(5));
}

class StorageStub implements Storage {

    @Override
    public String getName(int index) {
        if(index == 5) {
            return "Adam";
        } else {
            throw new UnsupportedOperationException();
        }
    }
}

@Test
void getName() {
    Logic logic = new Logic(new StorageStub());
    assertEquals("Name: Adam", logic.getName(5));
}

In addition to Stubs, there are other type of replacements you can use during testing. E.g. Mocks, Fakes, Dummies, Spies.

Dependency injection is the process of 'injecting' objects to replace current dependencies with a different object. This is often used to inject stubs to isolate the SUT from its dependencies so that it can be tested in isolation.

class Logic {
    Storage s;

    Logic(Storage s) {
        this.s = s;
    }

    String getName(int index) {
        return "Name: " + s.getName(index);
    }
}

interface Storage {
    String getName(int index);
}

class DatabaseStorage implements Storage {

    @Override
    public String getName(int index) {
        return readValueFromDatabase(index);
    }

    private String readValueFromDatabase(int index) {
        // retrieve name from the database
    }
}

Logic logic = new Logic(new DatabaseStorage());
String name = logic.getName(23);

@Test
void getName() {
    Logic logic = new Logic(new DatabaseStorage());
    assertEquals("Name: John", logic.getName(5));
}

class StorageStub implements Storage {

    @Override
    public String getName(int index) {
        if(index == 5) {
            return "Adam";
        } else {
            throw new UnsupportedOperationException();
        }
    }
}

@Test
void getName() {
    Logic logic = new Logic(new StorageStub());
    assertEquals("Name: Adam", logic.getName(5));
}

A Foo object normally depends on a Bar object, but we can inject a BarStub object so that the Foo object no longer depends on a Bar object. Now we can test the Foo object in isolation from the Bar object.

Polymorphism can be used to implement dependency injection, as can be seen in the example given in [Quality Assurance → Testing → Unit Testing → Stubs] where a stub is injected to replace a dependency.

class Logic {
    Storage s;

    Logic(Storage s) {
        this.s = s;
    }

    String getName(int index) {
        return "Name: " + s.getName(index);
    }
}

interface Storage {
    String getName(int index);
}

class DatabaseStorage implements Storage {

    @Override
    public String getName(int index) {
        return readValueFromDatabase(index);
    }

    private String readValueFromDatabase(int index) {
        // retrieve name from the database
    }
}

Logic logic = new Logic(new DatabaseStorage());
String name = logic.getName(23);

@Test
void getName() {
    Logic logic = new Logic(new DatabaseStorage());
    assertEquals("Name: John", logic.getName(5));
}

class StorageStub implements Storage {

    @Override
    public String getName(int index) {
        if(index == 5) {
            return "Adam";
        } else {
            throw new UnsupportedOperationException();
        }
    }
}

@Test
void getName() {
    Logic logic = new Logic(new StorageStub());
    assertEquals("Name: Adam", logic.getName(5));
}

class Payroll {
    private SalaryManager manager = new SalaryManager();
    private String[] employees;

    void setEmployees(String[] employees) {
        this.employees = employees;
    }

    void setSalaryManager(SalaryManager sm) {
       this. manager = sm;
    }

    double totalSalary() {
        double total = 0;
        for(int i = 0;i < employees.length; i++){
            total += manager.getSalaryForEmployee(employees[i]);
        }
        return total;
    }
}


class SalaryManager {
    double getSalaryForEmployee(String empID){
        //code to access employee’s salary history
        //code to calculate total salary paid and return it
    }
}

class PayrollTest {
    public static void main(String[] args) {
        //test setup
        Payroll p = new Payroll();
        p.setSalaryManager(new SalaryManagerStub()); //dependency injection
        //test case 1
        p.setEmployees(new String[]{"E001", "E002"});
        assertEquals(2500.0, p.totalSalary());
        //test case 2
        p.setEmployees(new String[]{"E001"});
        assertEquals(1000.0, p.totalSalary());
        //more tests ...
    }
}


class SalaryManagerStub extends SalaryManager {
    /** Returns hard coded values used for testing */
    double getSalaryForEmployee(String empID) {
        if(empID.equals("E001")) {
            return 1000.0;
        } else if(empID.equals("E002")) {
            return 1500.0;
        } else {
            throw new Error("unknown id");
        }
    }
}

Integration testing : testing whether different parts of the software work together (i.e. integrates) as expected. Integration tests aim to discover bugs in the 'glue code' related to how components interact with each other. These bugs are often the result of misunderstanding of what the parts are supposed to do vs what the parts are actually doing.

Integration testing is not simply a repetition of the unit test cases but run using the actual dependencies (instead of the stubs used in unit testing). Instead, integration tests are additional test cases that focus on the interactions between the parts.

In practice, developers often use a hybrid of unit+integration tests to minimize the need for stubs.

System testing: take the whole system and test it against the system specification.

System testing is typically done by a testing team (also called a QA team).

System test cases are based on the specified external behavior of the system. Sometimes, system tests go beyond the bounds defined in the specification. This is useful when testing that the system fails 'gracefully' having pushed beyond its limits.

Test case: load a web page that is too big
* Input: load a web page containing more than 5000 characters. 
* Expected behavior: abort the loading of the page and show a meaningful error message. 

System testing includes testing against non-functional requirements too. Here are some examples.

• Performance testing – to ensure the system responds quickly.
• Load testing (also called stress testing or scalability testing) – to ensure the system can work under heavy load.
• Security testing – to test how secure the system is.
• Compatibility testing, interoperability testing – to check whether the system can work with other systems.
• Usability testing – to test how easy it is to use the system.
• Portability testing – to test whether the system works on different platforms.

If a software product has a GUI component, all product-level testing (i.e. the types of testing mentioned above) need to be done using the GUI. However, testing the GUI is much harder than testing the CLI (command line interface) or API, for the following reasons:

• Most GUIs can support a large number of different operations, many of which can be performed in any arbitrary order.
• GUI operations are more difficult to automate than API testing. Reliably automating GUI operations and automatically verifying whether the GUI behaves as expected is harder than calling an operation and comparing its return value with an expected value. Therefore, automated regression testing of GUIs is rather difficult.
• The appearance of a GUI (and sometimes even behavior) can be different across platforms and even environments. For example, a GUI can behave differently based on whether it is minimized or maximized, in focus or out of focus, and in a high resolution display or a low resolution display.

One approach to overcome the challenges of testing GUIs is to minimize logic aspects in the GUI. Then, bypass the GUI to test the rest of the system using automated API testing. While this still requires the GUI to be tested manually, the number of such manual test cases can be reduced as most of the system has been tested using automated API testing.

There are testing tools that can automate GUI testing.

Acceptance testing (aka User Acceptance Testing (UAT)): test the delivered system to ensure it meets the user requirements.

Acceptance tests give an assurance to the customer that the system does what it is intended to do. Acceptance test cases are often defined at the beginning of the project, usually based on the use case specification. Successful completion of UAT is often a prerequisite to the project sign-off.

Acceptance testing comes after system testing. Similar to system testing, acceptance testing involves testing the whole system.

Some differences between system testing and acceptance testing:

Note: negative test cases: cases where the SUT is not expected to work normally e.g. incorrect inputs; positive test cases: cases where the SUT is expected to work normally

Passing system tests does not necessarily mean passing acceptance testing. Some examples:

• The system might work on the testbed environments but might not work the same way in the deployment environment, due to subtle differences between the two environments.
• The system might conform to the system specification but could fail to solve the problem it was supposed to solve for the user, due to flaws in the system design.

Alpha testing is performed by the users, under controlled conditions set by the software development team.

Beta testing is performed by a selected subset of target users of the system in their natural work setting.

An open beta release is the release of not-yet-production-quality-but-almost-there software to the general population. For example, Google’s Gmail was in 'beta' for many years before the label was finally removed.

Testability is an indication of how easy it is to test an SUT. As testability depends a lot on the design and implementation. You should try to increase the testability when you design and implement a software. The higher the testability, the easier it is to achieve a better quality software.

Test coverage is a metric used to measure the extent to which testing exercises the code i.e., how much of the code is 'covered' by the tests.

Here are some examples of different coverage criteria:

• Function/method coverage : based on functions executed  e.g., testing executed 90 out of 100 functions.
• Statement coverage : based on the number of line of code executed  e.g., testing executed 23k out of 25k LOC.
• Decision/branch coverage : based on the decision points exercised  e.g., an if statement evaluated to both true and false with separate test cases during testing is considered 'covered'.
• Condition coverage : based on the boolean sub-expressions, each evaluated to both true and false with different test cases. Condition coverage is not the same as the decision coverage.

• Path coverage measures coverage in terms of possible paths through a given part of the code executed. 100% path coverage means all possible paths have been executed. A commonly used notation for path analysis is called the Control Flow Graph (CFG).
• Entry/exit coverage measures coverage in terms of possible calls to and exits from the operations in the SUT.

Measuring coverage is often done using coverage analysis tools. Most IDEs have inbuilt support for measuring test coverage, or at least have plugins that can measure test coverage.

Coverage analysis can be useful in improving the quality of testing  e.g., if a set of test cases does not achieve 100% branch coverage, more test cases can be added to cover missed branches.

Some intermediate JUnit techniques that may be useful:

• It is possible for a JUnit test case to verify if the SUT throws the right exception.
• JUnit Rules are a way to add additional behavior to a test. e.g. to make a test case use a temporary folder for storing files needed for (or generated by) the test.
• It is possible to write methods thar are automatically run before/after a test method/class. These are useful to do pre/post cleanups for example.
• Testing private methods is possible, although not always necessray

Test-Driven Development(TDD)_ advocates writing the tests before writing the SUT, while evolving functionality and tests in small increments. In TDD you first define the precise behavior of the SUT using test cases, and then write the SUT to match the specified behavior. While TDD has its fair share of detractors, there are many who consider it a good way to reduce defects. One big advantage of TDD is that it guarantees the code is testable.

Except for trivial SUTs, exhaustive testing is not practical because such testing often requires a massive/infinite number of test cases.

Program testing can be used to show the presence of bugs, but never to show their absence!
_{--Edsger Dijkstra}

Here are two alternative approaches to testing a software: Scripted testing and Exploratory testing

1. Scripted testing: First write a set of test cases based on the expected behavior of the SUT, and then perform testing based on that set of test cases.

2. Exploratory testing: Devise test cases on-the-fly, creating new test cases based on the results of the past test cases.

Exploratory testing is ‘the simultaneous learning, test design, and test execution’ [source: bach-et-explained] whereby the nature of the follow-up test case is decided based on the behavior of the previous test cases. In other words, running the system and trying out various operations. It is called exploratory testing because testing is driven by observations during testing. Exploratory testing usually starts with areas identified as error-prone, based on the tester’s past experience with similar systems. One tends to conduct more tests for those operations where more faults are found.

Which approach is better – scripted or exploratory? A mix is better.

The success of exploratory testing depends on the tester’s prior experience and intuition. Exploratory testing should be done by experienced testers, using a clear strategy/plan/framework. Ad-hoc exploratory testing by unskilled or inexperienced testers without a clear strategy is not recommended for real-world non-trivial systems. While exploratory testing may allow us to detect some problems in a relatively short time, it is not prudent to use exploratory testing as the sole means of testing a critical system.

Scripted testing is more systematic, and hence, likely to discover more bugs given sufficient time, while exploratory testing would aid in quick error discovery, especially if the tester has a lot of experience in testing similar systems.

In some contexts, you will achieve your testing mission better through a more scripted approach; in other contexts, your mission will benefit more from the ability to create and improve tests as you execute them. I find that most situations benefit from a mix of scripted and exploratory approaches. --[source: bach-et-explained]

A positive test case is when the test is designed to produce an expected/valid behavior. A negative test case is designed to produce a behavior that indicates an invalid/unexpected situation, such as an error message.

Test case design can be of three types, based on how much of SUT internal details are considered when designing test cases:

• Black-box (aka specification-based or responsibility-based) approach: test cases are designed exclusively based on the SUT’s specified external behavior.

• White-box (aka glass-box or structured or implementation-based) approach: test cases are designed based on what is known about the SUT’s implementation, i.e. the code.

• Gray-box approach: test case design uses some important information about the implementation. For example, if the implementation of a sort operation uses different algorithms to sort lists shorter than 1000 items and lists longer than 1000 items, more meaningful test cases can then be added to verify the correctness of both algorithms.

Use cases can be used for system testing and acceptance testing. For example, the main success scenario can be one test case while each variation (due to extensions) can form another test case. However, note that use cases do not specify the exact data entered into the system. Instead, it might say something like user enters his personal data into the system. Therefore, the tester has to choose data by considering equivalence partitions and boundary values. The combinations of these could result in one use case producing many test cases.

To increase E&E of testing, high-priority use cases are given more attention. For example, a scripted approach can be used to test high priority test cases, while an exploratory approach is used to test other areas of concern that could emerge during testing.

Consider the testing of the following operation.

It is inefficient and impractical to test this method for all integer values [-MIN_INT to MAX_INT]. Fortunately, there is no need to test all possible input values. For example, if the input value 233 failed to produce the correct result, the input 234 is likely to fail too; there is no need to test both.

In general, most SUTs do not treat each input in a unique way. Instead, they process all possible inputs in a small number of distinct ways. That means a range of inputs is treated the same way inside the SUT. Equivalence partitioning (EP) is a test case design technique that uses the above observation to improve the E&E of testing.

By dividing possible inputs into equivalence partitions we can,

• avoid testing too many inputs from one partition. Testing too many inputs from the same partition is unlikely to find new bugs. This increases the efficiency of testing by reducing redundant test cases.
• ensure all partitions are tested. Missing partitions can result in bugs going unnoticed. This increases the effectiveness of testing by increasing the chance of finding bugs.

Equivalence partitions (EPs) are usually derived from the specifications of the SUT.

When the SUT has multiple inputs, you should identify EPs for each input.

An EP may not have adjacent values.

Some inputs have only a small number of possible values and a potentially unique behavior for each value. In those cases we have to consider each value as a partition by itself.

Note that the EP technique is merely a heuristic and not an exact science, especially when applied manually (as opposed to using an automated program analysis tool to derive EPs). The partitions derived depend on how one ‘speculates’ the SUT to behave internally. Applying EP under a glass-box or gray-box approach can yield more precise partitions.

When deciding EPs of OOP methods, we need to identify EPs of all data participants that can potentially influence the behaviour of the method, such as,

• the target object of the method call
• input parameters of the method call
• other data/objects accessed by the method such as global variables. This category may not be applicable if using the black box approach (because the test case designer using the black box approach will not know how the method is implemented)

Boundary Value Analysis (BVA) is test case design heuristic that is based on the observation that bugs often result from incorrect handling of boundaries of equivalence partitions. This is not surprising, as the end points of the boundary are often used in branching instructions etc. where the programmer can make mistakes.

BVA suggests that when picking test inputs from an equivalence partition, values near boundaries (i.e. boundary values) are more likely to find bugs.

Boundary values are sometimes called corner cases.

Typically, we choose three values around the boundary to test: one value from the boundary, one value just below the boundary, and one value just above the boundary. The number of values to pick depends on other factors, such as the cost of each test case.

An SUT can take multiple inputs. You can select values for each input (using equivalence partitioning, boundary value analysis, or some other technique).

Testing all possible combinations is effective but not efficient. If you test all possible combinations for the above example, you need to test 6x5x2x6=360 cases. Doing so has a higher chance of discovering bugs (i.e. effective) but the number of test cases can be too high (i.e. not efficient). Therefore, we need smarter ways to combine test inputs that are both effective and efficient.

Given below are some basic strategies for generating a set of test cases by combining multiple test input combination strategies.

The all combinations strategy generates test cases for each unique combination of test inputs.

The at least once strategy includes each test input at least once.

The all pairs strategy creates test cases so that for any given pair of inputs, all combinations between them are tested. It is based on the observations that a bug is rarely the result of more than two interacting factors. The resulting number of test cases is lower than the all combinations strategy, but higher than the at least once approach.

A variation of this strategy is to test all pairs of inputs but only for inputs that could influence each other.

The random strategy generates test cases using one of the other strategies and then pick a subset randomly (presumably because the original set of test cases is too big).

There are other strategies that can be used too.

Consider the following scenario.

Suppose these are the test cases being considered.

It looks like the test cases were created using the at least once strategy. After running these tests can we confirm that square-format label printing is done correctly?

• Answer: No.
• Reason: Cherry -- the only input that can produce a square-format label -- is in a negative test case which produces an error message instead of a label. If there is a bug in the code that prints labels in square-format, these tests cases will not trigger that bug.

In this case a useful heuristic to apply is each valid input must appear at least once in a positive test case. Cherry is a valid test input and we must ensure that it appears at least once in a positive test case. Here are the updated test cases after applying that heuristic.

Consider the test cases designed in [Heuristic: each valid input at least once in a positive test case].

After running these test cases can you be sure that the error message “invalid price” is shown for negative prices?

• Answer: No.
• Reason: -1 -- the only input that is a negative price -– is in a test case that produces the error message “invalid fruit”.

In this case a useful heuristic to apply is no more than one invalid input in a test case. After applying that, we get the following test cases.

Consider the calculateGrade scenario given below:

To get the first cut of test cases, let’s apply the at least once strategy.

Next, let’s apply the each valid input at least once in a positive test case heuristic. Test case 5 has a valid value for projectGrade=F that doesn't appear in any other positive test case. Let's replace test case 5 with 5.1 and 5.2 to rectify that.

Next, we apply the no more than one invalid input in a test case heuristic. Test cases 5.2 and 6 don't follow that heuristic. Let's rectify the situation as follows:

Next, let us assume that there is a dependency between the inputs examScore and isAbsent such that an absent student can only have examScore=0. To cater for the hidden invalid case arising from this, we can add a new test case where isAbsent=true and examScore!=0. In addition, test cases 3-6.2 should have isAbsent=false so that the input remains valid.

Software Quality Assurance (QA) is the process of ensuring that the software being built has the required levels of quality.

While testing is the most common activity used in QA, there are other complementary techniques such as static analysis, code reviews, and formal verification.

Quality Assurance = Validation + Verification

QA involves checking two aspects:

1. Validation: are we building the right system i.e., are the requirements correct?
2. Verification: are we building the system right i.e., are the requirements implemented correctly?

Whether something belongs under validation or verification is not that important. What is more important is both are done, instead of limiting to verification (i.e., remember that the requirements can be wrong too).

Code review is the systematic examination code with the intention of finding where the code can be improved.

Reviews can be done in various forms. Some examples below:

• In pair programming

  • As pair programming involves two programmers working on the same code at the same time, there is an implicit review of the code by the other member of the pair.

• Pull Request reviews

  • Project Management Platforms such as GitHub and BitBucket allows the new code to be proposed as Pull Requests and provides the ability for others to review the code in the PR.

• Formal inspections

  • Inspections involve a group of people systematically examining a project artifacts to discover defects. Members of the inspection team play various roles during the process, such as:

    • the author - the creator of the artifact
    • the moderator - the planner and executor of the inspection meeting
    • the secretary - the recorder of the findings of the inspection
    • the inspector/reviewer - the one who inspects/reviews the artifact.

Advantages of code reviews over testing:

• It can detect functionality defects as well as other problems such as coding standard violations.
• Can verify non-code artifacts and incomplete code
• Do not require test drivers or stubs.

Disadvantages:

• It is a manual process and therefore, error prone.

Static analysis of code can find useful information such unused variables, unhandled exceptions, style errors, and statistics. Most modern IDEs come with some inbuilt static analysis capabilities. For example, an IDE can highlight unused variables as you type the code into the editor.

Higher-end static analyzer tools can perform for more complex analysis such as locating potential bugs, memory leaks, inefficient code structures etc.

Linters are a subset of static analyzers that specifically aim to locate areas where the code can be made 'cleaner'.

Formal verification uses mathematical techniques to prove the correctness of a program.

Advantages:

• Formal verification can be used to prove the absence of errors. In contrast, testing can only prove the presence of error, not their absence.

Disadvantages:

• It only proves the compliance with the specification, but not the actual utility of the software.
• It requires highly specialized notations and knowledge which makes it an expensive technique to administer. Therefore, formal verifications are more commonly used in safety-critical software such as flight control systems.