Introduction to Testing

• This is chapter 26 of the book. Read it.
• I believe that you now have sufficient tools to truly start testing software
  • You understand modular compilation.
    • You can put small "units" of code in a file.
    • You can compile those units together to produce executables.
  • You should have a basic understanding of a build system
    • This is make.
    • Do a quick review of the Makefile for this section.
• The problem with what you have been given for testing (IMHO)
  • The plan is static.
  • You need to implement the plan by hand.
  • It is in no way close to modern testing standards.
• In chapter 26, and with supplemental material we will work on a better solution.
• He says "A programmer has overcome a major hurdle in her career when she realizes that testing is a part of the software development process. Bugs are not an occasional occurrence. They are found in every project of significant size. .. Your responsibility as a programmer is to write code that works, and to write tests to prove its correctness."
  • "Program testing can be used to show the presence of bugs, but never to show their absence!" (Dijkstra)
• White box testing:
  • The inside of the code is an open box.
  • You know the code and test for problems based on that.
  • Testing based on coding knowledge.
• Black box testing:
  • The code is a closed box.
  • You don't know what is inside.
  • Probably more "user" driven.
• The chapter focuses on white box testing.
• Unit testing:
  • A unit test is a piece of code that exercise specific functionality in a class or subsystem.
  • These are the finest grained tests that you can write.
  • He believes that one or more unit tests should exist for every low-level task that your code can perform.
• An example: You are creating software that supports addition of arbitrary precision numbers. What should you test?
  • Test simple addition
  • Test addition of two large positive numbers
  • Test addition of two large negative numbers
  • Test addition of a small and a large positive number
  • Test addition of a large and small positive number (not the same as previous case).
  • Test addition of a small and a large negative number
  • Test addition of a large and small negative number
  • Test addition of a large positive and large negative number
  • Test addition of a large negative and large positive number
  • Test the addition of 0 to all types of numbers (SL-PN)
  • Test adding (SL-PN) to additive inverse.
  • Test the commutative property (a+b == b+a) for various numbers (perhaps even large random test here).
• He says that well written unit tests
  1. Provides evidence that your code works. (He says proves but ...)
     • Until you test, or a client does, you have no way to know this.
  2. Provides an indication that new code has broken your product
  3. When used in the development process, force the developer to fix problems from the start.
     • Some systems will not let you "commit" code until it passes the tests.
  4. Lets you try your code before the client exists.
  5. Provides as example of how the code should be used.
• He thinks it is hard to go wrong using unit tests.
• But when do you create the tests?
  • Extreme Programming is a style that suggests
    • If a little testing is good, a lot of testing is better.
    • You should create tests before you write code.
    • You should automate your tests.
    • No code is acceptable unless it passes ALL of the unit tests.
  • Think of it this way.
    • When do you compile your code?
    • Why?
    • When do you debug your code?
  • Advantages of writing unit tests BEFORE your code
    • Helps to understand the requirements for the code.
    • Helps to give a measure of when the code is complete.
  • He suggests that many of the benefits can be derived from designing the test code but not implementing it until needed.
  • It is important to achieve maximum code coverage
    • Have you tested all paths through the code?
    • He suggests that ideally you should test all paths through the code.
    • We will discuss gprof at least briefly
      • This is a tool that can compute the level of code coverage for a test suite run.
• The unit testing process
  • He suggests that keeping tests in mind during the design process is a good thing.
    • It might lead to more testable software.
    • And again, it will help clarify requirements.
  • But how do you design your tests?
  • He suggests keeping granularity in mind.
    • Look at the table on page 916
    • These are ideas for testing a database interface.
    • 
    • He suggests that you get finer grain as you implement and revise code.
    • Since you are striving for 100% coverage, you will probably add tests to your code as you go along.
• But what to test?
  • What are the things your code was written to do?
  • What are the typical ways this code will be called?
  • What preconditions could be violated?
  • How could the methods be misused?
  • What data to you expect for input?
  • What data do you not expect for input?
  • What are the edge cases?
  • What are the exceptional conditions?
• What next?
  • Create the input/output BEFORE you run the code.
  • You should be able to predict what your code should produce.
  • DO NOT design your output based on your code
  • This can be hard work!
• Write the tests
  • Use a framework
    • We will be looking at one next time (or so)
  • Make sure that you test one thing each test.
    • This way you will know what failed.
    • I find this hard sometimes, so I work up.
      • A simple test of something basic.
      • A more complex test assuming the basic test worked fine.
      • ...
  • Be specific and not what failed.
    • You need this to be able to fix the problem.
    • I tend to have a problem with this.
  • Document extensively what failed.
    • Your future you will thank you.
  • If you have a code reviewer, have them review the unit tests as well.
• Running the tests
  • You should expect them all to fail.
  • But why?
    • Is your code bad?
    • Is the test code bad?
    • Is the test case bad?
  • You might be debugging your test cases.