Henriette's Notes

Risk Based Testing

A question that is asked regularly in testing circles is: “When should you stop testing?”. With an infinite budget it may be reasonably easy to answer this question. When the budget is severely constrained, this question becomes more difficult to answer. An even more difficult situation to address is when the project starts out with a given budget, but during the project lifetime the budget gets significantly reduced (i.e. due to economic downturn). This forces us be able to provide the highest level of quality for the least amount of money.

 

How Testing Fails

A mistake that is often made is that the testing effort is distributed equally across the system – both critical and less critical portions of the system are tested equally. This results in critical parts of the system not being tested sufficiently and less critical parts being tested to the point of diminishing returns.

 
A further mistaken mindset of developers is that there is no such thing as a useless test, which may entice developers to add tests for the sake of adding tests. In actual fact every test (unit-, integration- or systems test) has to earn its place in the codebase. If there no good motivation for a test, the test must be deleted from the codebase. Why is that? It adds to volume of code that developers have to master to be productive members of the team. It adds to the volume of code that has to be maintained. As such tests adds to the overall cost of maintenance of a system. Remember: The most cost effective code to maintain is the code that has never been written.

 

Risk Based Testing

Risk based testing is a testing approach that is helpful in addressing these concerns. The main advantage of risk based testing is to enable prioritization of the testing effort. The heart of risk based testing is to provide a set of criteria for evaluating risk. Some criteria for determining risk can be found in [1]. The criteria may differ depending on the project. Ideally the set of criteria needs to be negotiated with the project owners. Another important aspect is to decide on the level at which risk will be determined. For example, risk may be determined at business process, component or use case levels.

 

Some general criteria are provided below:

  • How frequently is this use case used?
  • What will be the cost of getting this use case wrong?
  • What is the complexity of this use case?
  • How often will the use case be changed?

 

A value from 1 to 5 is assigned for each of these criteria and the product determined. This product provides a way for ranking the risk of each use case. A higher risk value will indicate a use case with a higher risk.

 

Ideally determining risk should not be left to developers. Developers need to be guided by the project owner, the architect and the business analyst on their project.

 

Advantages/Disadvantages of Risk Based Testing

Advantages of this approach are:

  • The highest risk items can be developed and tested first which reduces the overall risk on the project.
  • If testing has to be watered down, a guideline exists for deciding what to test and what not to test.
  • At any given time an indication can be given of the risk of the project by considering the highest risk use cases that has not been tested.

 

A possible disadvantage of risk based testing is that not everything will be tested, but as stated earlier, if a testcase cannot be motivated, it may be more cost effective to not have the test as part of the codebase.

 

Conclusion

Risk based testing provides a means via which one can estimate how well, or how poorly, a system has been tested.

 

 

Why TDD can be Dangerous to your Project

Some developers who are passionate about testing are sometimes also fanatical advocates of Test Driven Development (TDD), to the point that they believe TDD is the only correct way to test applications. Strict adherence to TDD requires a test to be created first before any code has been written. As such proponents of TDD tend to see TDD as a design mechanism rather than as a test mechanism. Irrespective of whether TDD is used as a design and/or test mechanism, it has a number of pitfalls.

In this post I will:

  1. give a brief overview of the TDD cycle,
  2. explain why TDD as software design mechanism is inadequate,
  3. explain why TDD as software test mechanism is inadequate,
  4. explain the conditions under which TDD can be applied successfully to projects.

TDD

In TDD the steps to add new code are as follows [1]:

  1. Add a (failing) test that serves as a specification for the new functionality to be added.
  2. Run all the tests and confirm that the newly added test fails.
  3. Implement the new functionality.
  4. Run all tests to confirm that they all succeed.
  5. Refactor the code to remove any duplication.

The aim of TDD is to provide a simple way to grow the design of complex software one decision at a time.

TDD as a Design Mechanism

From a design perspective TDD emphasizes mirco level design rather than macro level design [2]. The weaknesses of this approach are:

  • TDD forces a developer to necessarily focus on a single interface. This often neglects the interaction between interfaces and leads to poor abstraction. Bertrand Meyer gives a balanced review of the challenges regarding TDD and Agile in this regard [3].
  • Quality attributes (like performance, scalability, integrability, security, etc.) are easily overlooked with TDD. In the context of Agile, where an upfront architecture effort is typically frowned upon, TDD is particularly dangerous due to poor consideration of quality attributes.

TDD as a Testing Mechanism

From a testing perspective TDD has the following drawbacks:

  • Interfaces/classes may be polluted in order to make them testable. A typical example is that a private method is made public in order to test it. This obfuscates the intended use of the class which will cause developers to more easily digress from the intended use of the class. In the long term this creates an unmaintainable system.
  • Often when TDD is used on projects, unit testing is used to exclusion, with limited or no regard for the broad spectrum of testing, which should at least include integration testing and systems testing.
  • TDD has no appreciation for the prioritization of the testing effort: Equal amounts of effort are expended to test all code irrespective of the associated risk profile. Typically TDD expects all further development to be blocked until all tests pass. This ignores the reality that some functionality has greater business value than others.

Guidelines for using TDD Effectively on Projects

TDD can be used with no side effect if the following process is adhered to:

  1. The architecture has to be complete, which should include details as to how testability of the system at all levels (unit-, integration- and systems testing) will be achieved.
  2. The risk profile of the sub system (module/use case) has been established, which informs the testing effort that must be expended on the sub system.
  3. The design for the sub system (module/use case) should be complete in adherence to the architecture and risk profile. Since testability is considered as part of the architecture, and the design is informed by the architecture, the design should be by definition testable.
  4. Only at this point is the developer now free to follow a TDD approach in implementing the code.

 

Conclusion

The very positive thing that TDD emphasizes is the need for testing. However, to naively embrace TDD, is to do it at the peril of your project.

 

References

  1. Kent Beck, Test-Driven Development by Example, The Addison-Wesley Signature Series, Addison-Wesley, 2003.
  2. Cédric Beust and Hani Suleiman, Next Generation Java Testing : TestNG and Advanced Concepts, Addison-Wesley, Upper Saddle River, NJ, 2008.
  3. Bertrand Meyer, Agile!: The Good, the Hype and the Ugly, Springer, 2014.

DBPedia Extraction Framework and Eclipse Quick Start

I recently treid to compile the DBPedia Extraction Framework. What was not immediately clear to me is whether I have to have Scala installed. It turns out that having Scala installed natively is not necessary, seeing as the scala-maven-plugin is sufficient.

The steps to compile DBPedia Extraction Framework from the command line are:

  1. Ensure you have the JDK 1.8.x installed.
  2. Ensure Maven 3.x is installed.
  3. mvn package

Steps to compile DBPedia Extraction Framework from the Scala IDE (which can be downloaded from Scala-ide.org) are:

  1. Ensure you have the JDK 1.8.x installed.
  2. Ensure you have the Scala IDE installed.
  3. mvn eclipse:eclipse
  4. mvn package
  5. Import existing Maven project into Scala IDE.
  6. Run mvn clean install from within the IDE.

Associations between Classes

This far we have only considered UML classes where the attributes are primitive types rather than classes. Here we will consider UML classes that have classes as attributes. Assume we want to model projects. Assume a project must have one name, one sponsor that must be a manager and it must have a team of between 3 and 10 employees. In UML this can be stated using attributes (see Fig.1(a)) or associations (see Fig. 1(b)). For interest sake Wazlawick [1] suggests using attribute notation for data types and associations for classes. His motivation is that associations makes dependencies between classes more apparent. I usually follow this guideline myself.

Fig. 1

Fig. 1

The OWL representation for these 2 class diagrams is given in Fig. 2. The first thing to notice is that we use ObjectProperty instead of DataProperty to represent the sponsor attribute/association. Similar for the team attribute/association. Our property definitions also now have Domain and Range restrictions. When we say that Susan is the sponsor for ABC, we can infer that Susan is a manager and ABC is project. This information can be captured through Domain and Range restrictions. For the purpose of finding modeling errors in it is preferable to add Domain and Range restrictions.

Association between Classes Manchester

Fig. 2

To limit the number of employees on a team to between 3 and 10 employees we use the property cardinality restrictions team min 3 owl:Thing and team max 10 owl:Thing. It may seem strange that we use team max 10 owl:Thing rather than team max 10 Employee. Surely we want to restrict team members to employees? Well true, but that is achieved through our range restriction on the team object property. Here we restricting our team to 10 whatever classes and the range restriction will infer that the team must be of type Employee.

References

1. R. S. Wazlawick, Object-oriented Analysis and Design for Information Systems: Modeling with UML, OCL and IFML, Morgan Kaufmann, 2014.

 

fast.ai: A Fresh take on Learning and Teaching ML

In this video Rachel Thomas provides an interesting take on learning ML: instead of promoting the typical bottom-up approach, fast.ai promotes a top-down approach. From a pedagogical perspective this seems counter intuitive. Surely you need to know the building blocks before you can move on to the theory that builds on the building blocks?  Indeed, that is how traditional education proceeds. However,  when consultants provide feedback to executives they tend to take a top-down approach. Why is that?

The main reason for taking a top-down approach when writing up/presenting technical findings is that you can provide a roadmap for where you are heading. This means that when you step into the details, the stakeholders can, because they now have a map of where you are heading, know how the details relate to the bigger picture. This is precisely why I think the fast.ai approach to learning ML can be effective.  Rachel Thomas provides further motivation for their approach in their video: How to Learn Deep Learning (when you’re not a computer science PhD)

Inheritance

In this post we will look at how different types of inheritance can be translated to OWL. We consider the case where Person is specialized by Employee and Client (Fig. 1). In a UML class diagram if inheritance is not annotated the default annotation {incomplete, disjoint} is assumed. incomplete means there are instances of Person which are neither of type Employee nor Client. disjoint means there is no instance of Person that is both of type Employee and of type Client. The set representation is given in Fig. 2 and the OWL translation in Fig. 3.

InheritanceDefault

Fig. 1

InheritanceDefaultSet

Fig. 2

InheritanceDefaultOWL

Fig. 3

The annotation {complete, disjoint} means every instance of Person is either a instance of Employee or an instance of Client(Fig. 4). The corresponding Venn diagram is  given in Fig. 5 and the OWL translation in Fig. 6.

InheritanceCompleteDisjoint

Fig. 4

InheritanceCompleteDisjointSet

Fig. 5

InheritanceCompleteDisjointOWL

Fig. 6

When overlapping is used rather than disjoint it means an instance of Person may be both of type Employee and of type Client.  Fig. 7 – 9 provides a UML class diagram, Venn diagram and OWL translation as example for the annotation {incomplete, overlapping}. Fig. 10 – 12 provides a UML class diagram, Venn diagram and OWL translation as example for the annotation {complete, overlapping}.

InheritanceIncompleteOverlapping

Fig. 7

InheritanceIncompleteOverlappingSet

Fig. 8

InheritanceIncompleteOverlappingOWL

Fig. 9

InheritanceCompleteOverlapping

Fig. 10

InheritanceCompleteOverlappingSet

Fig. 11

InheritanceCompleteOverlappingOWL

Fig. 12

A Brief Introduction to Protégé and Reasoners

A question you rightfully may be pondering is: Why translate object oriented classes into OWL? The answer is that it can help you to find logical inconsistencies in your class designs. In this post I will introduce the tools that will eventually enable you to find logical inconsistencies in your class designs.

The tool we will use is called Protégé. Download and installations instructions for Protégé can be found at https://protegewiki.stanford.edu/wiki/Install_Protege5.

In this post I will provide two screencasts:

  1. In the first screencast I will show you how to enter the OWL representation of the Person class introduced in the previous post.
  2. In the second screencast I will show you how to run a reasoner and how an inconsistency can arise.

On to the first screencast:

  1. Create a Person class.
  2. Create the data properties.
    1. name
    2. surname
    3. age
  3. Through sub-classing state that the Person class necessarily have a
    1. name,
    2. surname and
    3. age.
  4. If we run the reasoner on this ontology, no inconsistencies will be found.

In the second screencast I show how an inconsistency can arise. The steps are as follows:

  1. Create an individual called sarah of type Person.
  2. Run the reasoner. You will see the reasoner give no errors (nothing happened). This may come as a surprise to you since we have not set the name, surname or age data properties for the individual called sarah. In OWL this behaviour is expected due to what is called the open world assumption. OWL makes no assumption with regards to knowledge that is not stated explicitly. Since we did not state that the sarah individual does not have, for example, a name, the reasoner found no error in our ontology. This is different from typical database behaviour where absence of information is often assumed to indicate that the information does not exist, which is referred to as the closed world assumption.
  3. Now let us change our sarah individual to state that it does not have a name. This is achieved by stating that the sarah individual is of type name max 0 xsd:string. This states that the sarah individual can have a maximum of 0 name data properties of type xsd:string.
    SarahDoesNotHaveName
  4. If we run the reasoner now it shows that we have an inconsistency. We can ask Protégé to explain the inconsistency.ExplainSarahInconsistency
  5. The explanation states that sarah is of type Person and of type name max 0 xsd:string. But Person is a subclass of name some xsd:string. This states that individuals of type Person must have at least 1 name property of type xsd:string. Hence, the reason for the inconsistency.

 

Admittedly this example is contrived: there is not much sense in creating a Person class which we state must have a name and then create an individual of type Person which we then state does not have a name. But this was done here to show you how to use a reasoner to find inconsistencies in your ontology and to show you what information you can expect when your ontology is inconsistent.