Henriette's Notes

Home » Posts tagged 'RDF'

Tag Archives: RDF

Using SHACL validation with Ontotext GraphDB

Today I have 1 of those moments where I am absolutely sure if I do not write this down, I will forget how to do this next time. For one of the projects I am working on, we need to do SHACL validation of RDF data that will be stored in Ontotext GraphDB. Here are the 10 things I needed to learn in doing this. Some of these are rather obvious, but some were less than obvious to me.

Number 1: To be able to do SHACL validation, your repository needs to be configured for SHACL when you create your repository. This cannot be done after the fact.

Number 2: It seems to be better to import your ontology (or ontologies) and data into different graphs. This is useful when you want to re-import your ontology (or ontologies) or your data, because then you can replace a specific named graph completely. This was very useful for me while prototyping. Screenshot below:

Number 3: SHACL shapes are imported into this named graph

http://rdf4j.org/schema/rdf4j#SHACLShapeGraph

by default. At configuration time you can provide a different named graph or graphs for your SHACL shapes.

Number 4: To find the named graphs in your repository, you can do the following SPARQL query:

select distinct ?g 
where {
  graph ?g {?s ?p ?o }
}

You can then query a specific named graph as follows:

select * 
from <myNamedGraph>
where { 
	?s ?p ?o .
}

Number 5: However, getting the named graphs does not return the SHACL named graph. On StackOverflow someone suggested SHACL shapes can be retrieved using:

http://address:7200/repositories/myRepo/rdf-graphs/service?graph=http://rdf4j.org/schema/rdf4j#SHACLShapeGraph

However, this did not work for me. Instead, the following code worked reliably:

import org.eclipse.rdf4j.model.Model;
import org.eclipse.rdf4j.model.impl.LinkedHashModel;
import org.eclipse.rdf4j.model.vocabulary.RDF4J;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;
import org.eclipse.rdf4j.rio.RDFFormat;
import org.eclipse.rdf4j.rio.Rio;
import org.eclipse.rdf4j.rio.WriterConfig;
import org.eclipse.rdf4j.rio.helpers.BasicWriterSettings;

import java.util.stream.Collectors;

public class RetrieveShaclShapes {
public static void main(String[] args) {
String address = args[0]; /* i.e. http://localhost/ */
String repositoryName = args[1]; /* i.e. myRepo */

HTTPRepository repository = new HTTPRepository(address, repositoryName);
try (RepositoryConnection connection = repository.getConnection()) {
Model statementsCollector = new LinkedHashModel(
connection.getStatements(null, null,null, RDF4J.SHACL_SHAPE_GRAPH)
.stream()
.collect(Collectors.toList()));
Rio.write(statementsCollector, System.out, RDFFormat.TURTLE, new WriterConfig().set(
BasicWriterSettings.INLINE_BLANK_NODES, true));
} catch (Throwable t) {
t.printStackTrace();
}
}
}

using the following dependencies in the pom.xml with

${rdf4j.version} = 4.2.3: 
    <dependency>
        <groupId>org.eclipse.rdf4j</groupId>
        <artifactId>rdf4j-client</artifactId>
        <version>${rdf4j.version}</version>
        <type>pom</type>
    </dependency>

Number 6: Getting the above code to run was not obvious since I opted to using a fat jar. I encountered an “org.eclipse.rdf4j.rio.UnsupportedRDFormatException: Did not recognise RDF format object” error. RFD4J uses the Java Service Provider Interface (SPI) which uses a file in the META-INF/services of the jar to register parser implementations. The maven-assembly-plugin I used, to generate the fat jar, causes different jars to overwrite META-INF/services thereby loosing registration information. The solution is to use the maven-shade-plugin which merge META-INF/services rather overwrite it. In your pom you need to add the following to your plugins configuration:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.4.1</version>
        <executions>
          <execution>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>

You can avoid this problem by using the separate jars rather than a single fat jar.

Number 7: Importing a new shape into the SHACL shape graph will cause new shape information to be appended. It will not replace the existing graph even when you have both the

  • “Enable replacement of existing data” and
  • “I understand that data in the replaced graphs will be cleared before importing new data.”

options enabled as seen in the next screenshot:

To replace the SHACL named graph you need to clear it explicitly by running the following SPARQL command:

clear graph <http://rdf4j.org/schema/rdf4j#SHACLShapeGraph>

For myself I found it easier to update the SHACL shapes programmatically. Note that I made use of the default SHACL named graph:

import org.eclipse.rdf4j.model.vocabulary.RDF4J;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;
import org.eclipse.rdf4j.rio.RDFFormat;

import java.io.File;

public class UpdateShacl {
    public static void main(String[] args)  {
        String address = args[0]; /* i.e. http://localhost/ */
        String repositoryName = args[1]; /* i.e. myRepo */
        String shacl = args[2];
        File shaclFile = new File(shacl);

        HTTPRepository repository = new HTTPRepository(address, repositoryName);
        try (RepositoryConnection connection = repository.getConnection()) {
            connection.begin();
            connection.clear(RDF4J.SHACL_SHAPE_GRAPH);
            connection.add(shaclFile, RDFFormat.TURTLE, RDF4J.SHACL_SHAPE_GRAPH);
            connection.commit();
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }
}

Number 8: Programmatically you can delete a named graph using this code and the same maven dependency as we used above:

import org.eclipse.rdf4j.model.IRI;
import org.eclipse.rdf4j.model.ValueFactory;
import org.eclipse.rdf4j.model.impl.SimpleValueFactory;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;

public class ClearGraph {
    public static void main(String[] args)  {
        String address = args[0]; /* i.e. http://localhost/ */
        String repositoryName = args[1]; /* i.e. myRepo */
        String graph = args[2]; /* i.e. http://rdf4j.org/schema/rdf4j#SHACLShapeGraph */

        ValueFactory valueFactory = SimpleValueFactory.getInstance();
        IRI graphIRI = valueFactory.createIRI(graph);
        
        HTTPRepository repository = new HTTPRepository(address, repositoryName);
        try (RepositoryConnection connection = repository.getConnection()) {
            connection.begin();
            connection.clear(graphIRI);
            connection.commit();
        }
    }
}

Number 9: If you update the shape graph with constraints that are violated by your existing data, you will need to first fix your data before you can upload your new shape definition.

Number 10: When uploading SHACL shapes, unsupported features fails silently. I had this idea to add human readable information to the shape definition to make it easier for users to understand validation errors. Unfortunately “sh:name” and “sh:description” are not supported by GraphDB version 10.0.2. and 10.2.0. Moreover, it fails silently. In the Workbench it will show that it loaded successfully as seen in the next screenshot:

However, in the logs I have noticed the following warnings:

As these are logged as warnings, I was expecting my shape to have loaded fine, except that triples pertaining to “sh:name” and “sh:description” are skipped. However, my shape did not load at all.

You find the list of supported SHACL features here.

Conclusion

This post may come across as being critical of GraphDB. However, this is not the intention. I think it is rather a case of growing pains that are still experienced around SHACL (and Shex, I suspect) adoption. Resources that have been helpful for me in resolving issues are:

Creating Custom Rule Primitives for Jena

In this post I will show you

  1. how to add your own custom rule primitive,
  2. how to inform Jena of your custom rule primitive, and
  3. I will discuss things you have to keep in mind when writing a custom
    primitive.

Adding a Custom Rule Primitive

A powerful feature of Jena is that it allows you to create your own custom builtin rule primitives. Building on our student example of the previous post, assume we want to calculate the final mark for a student given their test result, exam result and project result. We assume we have the following data

:Peet :hasTestResult 77 .
:Peet :hasExamResult 86 .
:Peet :hasProjectResult 91 .

for which we add the following rule

[calcStudentFinalMarkRule: 
  (?student :hasTestResult ?testResult) 
  (?student :hasExamResult ?examResult) 
  (?student :hasProjectResult ?projectResult) 
  calcFinalMark(?testResult, ?examResult, 
   ?projectResult, ?finalMark)
      -> (?student :hasFinalMark ?finalMark)]

The meat of the implementation is the doUserRequiredAction(args, length,
context)
method, which consists of the following steps:

  1. check that we have the correct number of parameters,
  2. retrieve the input parameters,
  3. verify the typing of input parameters,
  4. doing the actual calculation,
  5. creating a node for the output parameter, and
  6. binding the node to the output parameter.
private boolean doUserRequiredAction(Node[] args, 
  int length, RuleContext context) {
  
  // Check we received the correct number of parameters
  checkArgs(length, context);

  boolean success = false;
  
  // Retrieve the input arguments
  Node studentTestResult = getArg(0, args, context);
  Node studentExamResult = getArg(1, args, context);
  Node studentProjectResult = getArg(2, args, context);

  // Verify the typing of the parameters
  if (studentTestResult.isLiteral() && 
    studentExamResult.isLiteral() && 
    studentProjectResult.isLiteral()) {
    Node finalMark = null;
    if (studentTestResult.getLiteralValue() 
        instanceof Number && 
      studentExamResult.getLiteralValue() 
        instanceof Number &&
      studentProjectResult.getIndexingValue() 
        instanceof Number) {
    
      Number nvStudentTestResult = 
        (Number)studentTestResult.getLiteralValue();
      Number nvStudentExamResult = 
        (Number)studentExamResult.getLiteralValue();
      Number nvStudentProjectResult = 
        (Number)studentProjectResult.getLiteralValue();
    
      // Doing the calculation
      int nFinalMark = 
        (nvStudentTestResult.intValue() * 20)/100 + 
        (nvStudentExamResult.intValue() * 50)/100 +
        (nvStudentProjectResult.intValue() * 30)/100;
      
      // Creating a node for the output parameter
      finalMark = Util.makeIntNode(nFinalMark);  
      
      // Binding the output parameter to the node
      BindingEnvironment env = context.getEnv();    
      success = env.bind(args[3], finalMark);
    } 
  }   
  return success;
}

Registering a Custom Primitive with Jena

Our code for load our rules and activating it is similar to my previous post, except that you have to make a call to register the custom primitive:

// Load RDF data
String data = path.toFile().getAbsolutePath() + 
  "/src/main/resources/data2.ttl";
Model model = ModelFactory.createDefaultModel();
model.read(data);
      
// Register custom primitive
BuiltinRegistry.theRegistry.register(new CalcFinalMark());
      
// Load rules
String rules = path.toFile().getAbsolutePath() + 
  "/src/main/resources/student2.rules";
Reasoner reasoner = new GenericRuleReasoner(Rule.rulesFromURL(rules));
     
InfModel infModel = ModelFactory.createInfModel(reasoner, model);  
infModel.rebind();   

Things to Keep in Mind

There are two main things I think one needs to keep in mind with Jena custom rule primitives:

  1. A primitive is suppose to be a elementary building block. Being able to create your own primitives may tempt you to add all sorts of interesting processing besides the manipulation of triples, but I strongly advice against that. Arbitrary processing in your builtin primitive can degrade performance of inferencing.
  2. Do not assume that you have control over when a rule will be triggered. Exactly when a rule will be triggered is dependent on when the Jena InfModel implementation decides to re-evaluate the rules, which is dependent on internal caching of the InfModel implementation and how it deals with modifications made to the model. Even though I believe InfModel implementations will avoid arbitrarily re-evaluating rules, I still think it is conceivable that under some circumstance the same rule may be triggered more than once for the same data. Furthermore, the Jena documentation of 3.6.0 states that the InfModel interface is still in flux, which could mean that even if a rule is only triggered once for given data currently, due to unforeseen changes it may be triggered more than once in future updates of Jena.

Conclusion

In this post I gave an example of how you can develop a custom rule primitive for Jena. The code for this example can be found at github.

Creating Custom Rules for Jena

In this post I will show you

  1. how to add your own custom rules to Jena,
  2. how to use rule primitives, and
  3. I will mention some things you may want to keep in mind when using rules.

Add and Activate Rules

We assume we start with the following simple triples:

:Peet :takesCourse :ComputerScience .
:Ruth :teachesCourse :ComputerScience .

for which we add the following rule to a student1.rules file:

[hasStudentRule: 
  (?student :takesCourse ?course) 
  (?lecturer :teachesCourse ?course) 
    -> (?lecturer :hasStudent ?student)] 

The hasStudentRule says that if a student takes a course and that course is presented by some lecturer, then the student is a student of that lecturer. The (?student :takesCourse ?course) (?lecturer :teachesCourse ?course) part of the rule is referred to as the premise or body of the rule and the (?lecturer :hasStudent ?student) part as the conclusion or head of the rule.

To activate the rules in Jena we have to create a model that is capable of doing inferences. In the context of Jena inferencing means that it will re-evaluate the rules against the data, which will cause additional statements to be potentially added to the model. This can be achieved by calling methods that will cause Jena to re-evaluate the model, i.e. calls like InfModel.rebind() and InfModel.validate().

Path path = Paths.get(".").toAbsolutePath().normalize();

// Load RDF data
String data = path.toFile().getAbsolutePath() + 
  "/src/main/resources/data.ttl";
Model model = ModelFactory.createDefaultModel();
model.read(data);
    
// Load rules
String rules = path.toFile().getAbsolutePath() + 
  "/src/main/resources/student.rules";
Reasoner reasoner = new GenericRuleReasoner(Rule.rulesFromURL(rules));

InfModel infModel = ModelFactory.createInfModel(reasoner, model);   
infModel.rebind();

When the hasStudentRule is activated a new statement will be added to the Jena model:

:Ruth :hasStudent :Peet .

Using Rule Primitives

Jena supports a number of builtin rule primitives that are intuitive to understand, i.e.

[pensionerRule: 
  (?person :hasAge ?age) greaterThan(?age, 60) 
    -> (?person a :Pensioner)]

which states that when ?person has an age greater than 60, ?person is considered to be a pensioner. Assuming we have the data

:Peet :hasAge 90 .

the following triple will be added to the Jena model:

:Peet a :Pensioner .

Things to Keep in Mind

There are two main things I think one needs to keep in mind with Jena rules:

  1. The purpose of rules are to manipulate the triples in the Jena model. For the most part it adds triples to the model, but it can also remove triples from the model if primitives like remove and drop are used.
  2. Adding and removing of triples can be achieved through SPARQL queries which can perform better or worse than rules. It is therefore best to check both approaches for your given use case.

Conclusion

In this post I gave a brief introduction in how to use custom rules and builtin rule primitives in Jena. This code is available at github.