Finishing Testing

• When we left off last time, we had developed a reasonable set of tests for the constructor and a few simple tests for the Insert function.
  • We tested inserting a single item.
  • We tested inserting multiple items at the front.
• Do a quick review of DLCL_Multiple_FRONT_Insert_Test
• What else should we test in an insert
  • The specification gives three different cases.
    • Insert into an empty list, we have this one.
    • Insert at the front of a list, we probably have this as well
    • Insert elsewhere in the list.
  • I think we discussed this briefly, but let's look again.
    • We generally insert at the current position.
    • If the list is ...a b c ... where current points at b.
    • The list should become ...a n b c... and current points at n.
    • How can we test this?
      • We know what the results of an insert should be.
      • So we maintain a vector in the proper order.
      • We really should check to see if current is pointing at the right thing.
      • Remember, DataTest checks for a value at a given position.
      • So after the insert, check this first.
      • Then do a list compare.
• I find random testing is also effective
  • The literature is mixed on this,
  • But no-one says it is completely ineffective.
  • So I created a random insert test.
• What have we tested?
  • Constructor
  • Insert
  • Home, Left, Right, Data, Size
• How would you test other methods?
  • InsertAfter?
  • Delete?
  • Copy Constructor?
  • Assignment Operator?
  • Destructor?
• Would valgrind play a role in this?
• Code for this section of the discussion.
• How do we know if we have tested all of the code?
  • Any python, javascript, ... people out there?
    • Did you ever have a long use project that suddenly gets an unexpected input and fails?
    • You covered it, but your logic was wrong.
    • And you never tested it.
• I have some nasty code from a recent programming contest.
  • Look at Doorman
  • I am NOT proud of this code.
    • I would deduct at least 20% for duplicated code.
    • Do you see it?
    • How can we fix it?
  • I see real potential for bugs in this code.
    • If statements nested at least three deep
    • Compound conditions.
    • Duplicated code.
  • But let's build a framework to test it.
    • Made a header file and implementation file for the parts I wanted to test.
    • Copied the test case 0 from the previous example
  • I built a simple test.
  • Then decided to build a file based test.
• Well it passed, but how well did I test the routine?
  gcov is a a coverage testing too.
  • reference
  • also man gcov
  • This is for GCC only.
    • With other compilers you will need a different too.
    • But is suspect ALL decent compilers have one.
  • This tool will tell you how often lines of code are executed
    • You can optimize the lines executed most frequently.
    • You can also tell what liens have not been executed
  • When added to a test suite, you can make sure that you have cases to exercise all of your code.
    • This is probably best done at development time as there are many MANY lines of code in a major project.
  • This is a line oriented tool
    • We use the practice of one statement per line of code, but that is important.
    • We will need to change our compile flags for this, as they want the optimizer turned off.
    • We will also need to compile with the flag --coverage
  • Modification of the makefile is fairly simple
    • I added a target called coverage
      • This is not real, so I also added it to .PHONY
      • I added a line that redefines CXXFLAGS for coverage
      • I added a line to build coverage
        • It relies on Test
        • But you need to rebuild Test with the modified flags.
        • So it relies on clean as well.
      • This extra a line
        • Runs the test code.
        • Then runs gcov.
    • Notice this makefile had extra things to remove in clean
      • This is because gcov produces a number of different files.
  • Running gcov
    • gcov takes a list of the files you want reports generated for.
      • In this case we are only interested in doorman.cpp
      • It will produce a file called doorman.cpp.gcov
        • This is a human readable file.
        • But it can be changed for other tools as well.
    • While there are many flags, for our usage we only need a few.
      • We only want reports on code in this directory so use the -r flag.
      • I like the colorize option so use the -k flag.
      • I also added -fprofile-arcs because I think it will be useful
  • The report gives the number of times a line is executed.
    • - means the line is not executable
    • ###### means the line is not executed.
      • But this is changed for colorized output.
    • This can be switched to percentages if you wish.
  • So after you make coverage look at the report
    • What code has not been tested?
  • I added this back to into the DList test suite.
  • This has been a basic look at gcov.
    • There are quite a few command line arguments.
      • This means you can do more things with it.
    • And I don't have that much experience with this tool.
  • So what is the testing strategy?
    • Build unit tests for all modules developed
    • Test from low level routines to high level routines.
    • Use the requirements to build tests to assure the requirements are met.
    • Use boundary condition like tests to test the edge cases.
    • If you can, use a randomized test to test large cases.
    • Use gcov to see that all lines of code are exercised in your tests.
    • Use good software engineering practices to minimize errors in your test cases.
    • Make sure you know what your test cases should procure BEFORE you execute the tests.
    • Automate the testing process and do it frequently
  • He discusses other levels of testing in the chapter.
    • But those don't seem to be relevant to this course.
  • One last word
    • This is big.
    • You see testing required or desired in many job descriptions
    • You see unit test software built in various tools
      • I spent a few weeks working on testing with git this summer.
      • You should have seen this except for a collapse in our systems administration
      • But you can build a system where submitted code must pass unit tests BEFORE you are allowed to submit it.