Henriette's Notes

Home » Semantic technologies » RDF Stores » Ontotext GraphDB

Category Archives: Ontotext GraphDB

Using SHACL validation with Ontotext GraphDB

Today I have 1 of those moments where I am absolutely sure if I do not write this down, I will forget how to do this next time. For one of the projects I am working on, we need to do SHACL validation of RDF data that will be stored in Ontotext GraphDB. Here are the 10 things I needed to learn in doing this. Some of these are rather obvious, but some were less than obvious to me.

Number 1: To be able to do SHACL validation, your repository needs to be configured for SHACL when you create your repository. This cannot be done after the fact.

Number 2: It seems to be better to import your ontology (or ontologies) and data into different graphs. This is useful when you want to re-import your ontology (or ontologies) or your data, because then you can replace a specific named graph completely. This was very useful for me while prototyping. Screenshot below:

Number 3: SHACL shapes are imported into this named graph

http://rdf4j.org/schema/rdf4j#SHACLShapeGraph

by default. At configuration time you can provide a different named graph or graphs for your SHACL shapes.

Number 4: To find the named graphs in your repository, you can do the following SPARQL query:

select distinct ?g 
where {
  graph ?g {?s ?p ?o }
}

You can then query a specific named graph as follows:

select * 
from <myNamedGraph>
where { 
	?s ?p ?o .
}

Number 5: However, getting the named graphs does not return the SHACL named graph. On StackOverflow someone suggested SHACL shapes can be retrieved using:

http://address:7200/repositories/myRepo/rdf-graphs/service?graph=http://rdf4j.org/schema/rdf4j#SHACLShapeGraph

However, this did not work for me. Instead, the following code worked reliably:

import org.eclipse.rdf4j.model.Model;
import org.eclipse.rdf4j.model.impl.LinkedHashModel;
import org.eclipse.rdf4j.model.vocabulary.RDF4J;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;
import org.eclipse.rdf4j.rio.RDFFormat;
import org.eclipse.rdf4j.rio.Rio;
import org.eclipse.rdf4j.rio.WriterConfig;
import org.eclipse.rdf4j.rio.helpers.BasicWriterSettings;

import java.util.stream.Collectors;

public class RetrieveShaclShapes {
public static void main(String[] args) {
String address = args[0]; /* i.e. http://localhost/ */
String repositoryName = args[1]; /* i.e. myRepo */

HTTPRepository repository = new HTTPRepository(address, repositoryName);
try (RepositoryConnection connection = repository.getConnection()) {
Model statementsCollector = new LinkedHashModel(
connection.getStatements(null, null,null, RDF4J.SHACL_SHAPE_GRAPH)
.stream()
.collect(Collectors.toList()));
Rio.write(statementsCollector, System.out, RDFFormat.TURTLE, new WriterConfig().set(
BasicWriterSettings.INLINE_BLANK_NODES, true));
} catch (Throwable t) {
t.printStackTrace();
}
}
}

using the following dependencies in the pom.xml with

${rdf4j.version} = 4.2.3: 
    <dependency>
        <groupId>org.eclipse.rdf4j</groupId>
        <artifactId>rdf4j-client</artifactId>
        <version>${rdf4j.version}</version>
        <type>pom</type>
    </dependency>

Number 6: Getting the above code to run was not obvious since I opted to using a fat jar. I encountered an “org.eclipse.rdf4j.rio.UnsupportedRDFormatException: Did not recognise RDF format object” error. RFD4J uses the Java Service Provider Interface (SPI) which uses a file in the META-INF/services of the jar to register parser implementations. The maven-assembly-plugin I used, to generate the fat jar, causes different jars to overwrite META-INF/services thereby loosing registration information. The solution is to use the maven-shade-plugin which merge META-INF/services rather overwrite it. In your pom you need to add the following to your plugins configuration:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.4.1</version>
        <executions>
          <execution>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>

You can avoid this problem by using the separate jars rather than a single fat jar.

Number 7: Importing a new shape into the SHACL shape graph will cause new shape information to be appended. It will not replace the existing graph even when you have both the

  • “Enable replacement of existing data” and
  • “I understand that data in the replaced graphs will be cleared before importing new data.”

options enabled as seen in the next screenshot:

To replace the SHACL named graph you need to clear it explicitly by running the following SPARQL command:

clear graph <http://rdf4j.org/schema/rdf4j#SHACLShapeGraph>

For myself I found it easier to update the SHACL shapes programmatically. Note that I made use of the default SHACL named graph:

import org.eclipse.rdf4j.model.vocabulary.RDF4J;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;
import org.eclipse.rdf4j.rio.RDFFormat;

import java.io.File;

public class UpdateShacl {
    public static void main(String[] args)  {
        String address = args[0]; /* i.e. http://localhost/ */
        String repositoryName = args[1]; /* i.e. myRepo */
        String shacl = args[2];
        File shaclFile = new File(shacl);

        HTTPRepository repository = new HTTPRepository(address, repositoryName);
        try (RepositoryConnection connection = repository.getConnection()) {
            connection.begin();
            connection.clear(RDF4J.SHACL_SHAPE_GRAPH);
            connection.add(shaclFile, RDFFormat.TURTLE, RDF4J.SHACL_SHAPE_GRAPH);
            connection.commit();
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }
}

Number 8: Programmatically you can delete a named graph using this code and the same maven dependency as we used above:

import org.eclipse.rdf4j.model.IRI;
import org.eclipse.rdf4j.model.ValueFactory;
import org.eclipse.rdf4j.model.impl.SimpleValueFactory;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;

public class ClearGraph {
    public static void main(String[] args)  {
        String address = args[0]; /* i.e. http://localhost/ */
        String repositoryName = args[1]; /* i.e. myRepo */
        String graph = args[2]; /* i.e. http://rdf4j.org/schema/rdf4j#SHACLShapeGraph */

        ValueFactory valueFactory = SimpleValueFactory.getInstance();
        IRI graphIRI = valueFactory.createIRI(graph);
        
        HTTPRepository repository = new HTTPRepository(address, repositoryName);
        try (RepositoryConnection connection = repository.getConnection()) {
            connection.begin();
            connection.clear(graphIRI);
            connection.commit();
        }
    }
}

Number 9: If you update the shape graph with constraints that are violated by your existing data, you will need to first fix your data before you can upload your new shape definition.

Number 10: When uploading SHACL shapes, unsupported features fails silently. I had this idea to add human readable information to the shape definition to make it easier for users to understand validation errors. Unfortunately “sh:name” and “sh:description” are not supported by GraphDB version 10.0.2. and 10.2.0. Moreover, it fails silently. In the Workbench it will show that it loaded successfully as seen in the next screenshot:

However, in the logs I have noticed the following warnings:

As these are logged as warnings, I was expecting my shape to have loaded fine, except that triples pertaining to “sh:name” and “sh:description” are skipped. However, my shape did not load at all.

You find the list of supported SHACL features here.

Conclusion

This post may come across as being critical of GraphDB. However, this is not the intention. I think it is rather a case of growing pains that are still experienced around SHACL (and Shex, I suspect) adoption. Resources that have been helpful for me in resolving issues are:

Getting started with Ontotext GraphDB and Jena

In my previous post I explained how you can create an GraphDB repository and how you can update and query your repository using RDF4J. In this post I provide an example of how you can update and query a GraphDB repository using Jena. However, even though the code works, there are some pitfalls.

A Quick Example

First you need to add Jena as a Maven dependency:

<dependency>
 <groupId>org.apache.jena</groupId>
 <artifactId>apache-jena-libs</artifactId>
 <version>3.7.0</version>
 <type>pom</type>
</dependency>  

The Java code is straightforward:

package org.graphdb.jena.tutorial;

import org.apache.jena.query.QueryExecution;
import org.apache.jena.query.QueryExecutionFactory;
import org.apache.jena.query.QuerySolution;
import org.apache.jena.query.ResultSet;
import org.apache.jena.update.UpdateExecutionFactory;
import org.apache.jena.update.UpdateFactory;
import org.apache.jena.update.UpdateProcessor;
import org.apache.jena.update.UpdateRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;

public class SimpleInsertQueryExample {
  private static Logger logger = LoggerFactory.getLogger(SimpleInsertQueryExample.class);
  // Why This Failure marker
  private static final Marker WTF_MARKER = MarkerFactory.getMarker("WTF");
  
  // GraphDB 
  private static final String PERSONDATA_REPO_QUERY = 
      "http://localhost:7200/repositories/PersonData";
  private static final String PERSONDATA_REPO_UPDATE = 
      "http://localhost:7200/repositories/PersonData/statements";


  private static String strInsert;
  private static String strQuery;
  
  static {
    strInsert = 
        "INSERT DATA {"
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://dbpedia.org/ontology/birthDate>"
         + " \"1906-12-09\"^^<http://www.w3.org/2001/XMLSchema#date> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://dbpedia.org/ontology/birthPlace> "
         + "<http://dbpedia.org/resource/New_York_City> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://dbpedia.org/ontology/deathDate>"
         + " \"1992-01-01\"^^<http://www.w3.org/2001/XMLSchema#date> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://dbpedia.org/ontology/deathPlace> "
         + "<http://dbpedia.org/resource/Arlington_County,_Virginia> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://purl.org/dc/terms/description>"
         + " \"American computer scientist and United States Navy officer.\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> "
         + "<http://dbpedia.org/ontology/Person> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://xmlns.com/foaf/0.1/gender> \"female\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://xmlns.com/foaf/0.1/givenName> \"Grace\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> "
         + "<http://xmlns.com/foaf/0.1/name> \"Grace Hopper\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper>"
         + " <http://xmlns.com/foaf/0.1/surname> \"Hopper\" ."        
         + "}";
    
    strQuery = 
        "SELECT ?name WHERE {?s <http://xmlns.com/foaf/0.1/name> ?name .}";
  }   
  
  private static void insert() {
    UpdateRequest updateRequest = UpdateFactory.create(strInsert);
    UpdateProcessor updateProcessor = UpdateExecutionFactory
        .createRemote(updateRequest, 
        PERSONDATA_REPO_UPDATE);
    updateProcessor.execute();
  }
  
  private static void query() {
    QueryExecution queryExecution = QueryExecutionFactory
        .sparqlService(PERSONDATA_REPO_QUERY, strQuery);
    for (ResultSet results = queryExecution.execSelect(); results.hasNext();) {
      QuerySolution qs = results.next();
      String strName = qs.get("?name").toString();
      logger.trace("name = " + strName);
    }    
    queryExecution.close();
  }

  
  public static void main(String[] args) {
    try {
      insert(); 
      query();      
    } catch (Throwable t) {
      logger.error(WTF_MARKER, t.getMessage(), t);
    }   
  }  
}

Some Pitfalls

The example I provided will insert RDF data into GraphDB and query it successfully. However, the data is inserted into the repository in the absence of a transaction. The transaction API of Jena is based on a Dataset. Historically Ontotext provided a Jena adapter with which a Jena Dataset could be created. However, based on my question on Stack Overflow in this regard, the Jena adapter is no longer supported by Ontotext. Hence, currently it is not clear to me how to enable transactions when using Jena to access GraphDB. So, if you know how to address this, please be so kind as to leave a comment with your insight!

Conclusion

In this post I provided a quick example of how you can access GraphDB using Jena. However, this example does not support transactions, and therefore you may want to look at rather using RDF4J with GraphDB. You can find this code at github.

Getting started with Ontotext GraphDB and RDF4J

In this post I will explain how to quickly get started with the free version of Ontotext GraphDB and RDF4J. Ontotext GraphDB is an RDF datastore and RDF4J is a Java framework for accessing RDF datastores (not just GraphDB). I will explain

  1. how to install and start GraphDB, as well as how to use the workbench to add a repository, and
  2. how to do SPARQL queries against GraphDB using RDF4J.

Install and start GraphDB and create a Repository

To gain access to the free version of GraphDB you have to email Ontotext. They will respond with an email with links to a desktop and stand-alone server version of GraphDB. You want to download the stand-alone server version. This is a graphdb-free-VERSION-dist.zip file, that you can extract somewhere on your filesystem, which I will refer to here as $GRAPHDB_ROOT. To start GraphDB, go to $GRAPHDB_ROOT/bin and run ./graphdb.

To access the workbench you can go to http://localhost:7200. To create a new repository, in the left-hand side menu navigate to Setup>Repositories. Click the Create new repository button. For our simple example we will use PersonData as an Repository ID. The rest of the settings we leave as-is. At the bottom of the page you can press the Create button.

Accessing a GraphDB Repository using RDF4J

To access our PersonData repository we will use RDF4J. Since GraphDB is based on the RDF4J libraries, we only need to include the GraphDB dependencies since these already include RDF4J. Thus, in our pom.xml file we only need to add the following:

 <dependency>
   <groupId>com.ontotext.graphdb</groupId>
   <artifactId>graphdb-free-runtime</artifactId>
   <version>8.5.0</version>
 </dependency>

In our example Java code we first insert some RDF data and then do a query based on the added data. For inserting data we start a transaction and commit it, or, if it fails we do a rollback. For querying the data we iterate through the TupleQueryResult, retrieving values for the binding variables we are interested in (i.e. name in this case). In line with the TupleQueryResult documentation, we close the TupleQueryResult once we are done.

package org.graphdb.rdf4j.tutorial;
package org.graphdb.rdf4j.tutorial;

import org.eclipse.rdf4j.model.impl.SimpleLiteral;
import org.eclipse.rdf4j.query.BindingSet;
import org.eclipse.rdf4j.query.QueryEvaluationException;
import org.eclipse.rdf4j.query.QueryLanguage;
import org.eclipse.rdf4j.query.TupleQuery;
import org.eclipse.rdf4j.query.TupleQueryResult;
import org.eclipse.rdf4j.query.Update;
import org.eclipse.rdf4j.repository.Repository;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.http.HTTPRepository;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;

public class SimpleInsertQueryExample {
  private static Logger logger = 
    LoggerFactory.getLogger(SimpleInsertQueryExample.class);
  // Why This Failure marker
  private static final Marker WTF_MARKER = 
    MarkerFactory.getMarker("WTF");
  
  // GraphDB 
  private static final String GRAPHDB_SERVER = 
    "http://localhost:7200/";
  private static final String REPOSITORY_ID = "PersonData";

  private static String strInsert;
  private static String strQuery;
  
  static {
    strInsert = 
        "INSERT DATA {"
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://dbpedia.org/ontology/birthDate> \"1906-12-09\"^^<http://www.w3.org/2001/XMLSchema#date> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://dbpedia.org/ontology/birthPlace> <http://dbpedia.org/resource/New_York_City> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://dbpedia.org/ontology/deathDate> \"1992-01-01\"^^<http://www.w3.org/2001/XMLSchema#date> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://dbpedia.org/ontology/deathPlace> <http://dbpedia.org/resource/Arlington_County,_Virginia> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://purl.org/dc/terms/description> \"American computer scientist and United States Navy officer.\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/Person> ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://xmlns.com/foaf/0.1/gender> \"female\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://xmlns.com/foaf/0.1/givenName> \"Grace\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://xmlns.com/foaf/0.1/name> \"Grace Hopper\" ."
         + "<http://dbpedia.org/resource/Grace_Hopper> <http://xmlns.com/foaf/0.1/surname> \"Hopper\" ."        
         + "}";
    
    strQuery = 
        "SELECT ?name FROM DEFAULT WHERE {" +
        "?s <http://xmlns.com/foaf/0.1/name> ?name .}";
  }  
  
  private static RepositoryConnection getRepositoryConnection() {
    Repository repository = new HTTPRepository(
      GRAPHDB_SERVER, REPOSITORY_ID);
    repository.initialize();
    RepositoryConnection repositoryConnection = 
      repository.getConnection();
    return repositoryConnection;
  }
  
  private static void insert(
    RepositoryConnection repositoryConnection) {
    
    repositoryConnection.begin();    
    Update updateOperation = repositoryConnection
      .prepareUpdate(QueryLanguage.SPARQL, strInsert);
    updateOperation.execute();
    
    try {
      repositoryConnection.commit();
    } catch (Exception e) {
      if (repositoryConnection.isActive())
        repositoryConnection.rollback();
    }
  }

  private static void query(
    RepositoryConnection repositoryConnection) {
    
    TupleQuery tupleQuery = repositoryConnection
      .prepareTupleQuery(QueryLanguage.SPARQL, strQuery);
    TupleQueryResult result = null;
    try {
      result = tupleQuery.evaluate();
      while (result.hasNext()) {
        BindingSet bindingSet = result.next();

        SimpleLiteral name = 
          (SimpleLiteral)bindingSet.getValue("name");
        logger.trace("name = " + name.stringValue());
      }
    }
    catch (QueryEvaluationException qee) {
      logger.error(WTF_MARKER, 
        qee.getStackTrace().toString(), qee);
    } finally {
      result.close();
    }    
  }  
  
  public static void main(String[] args) {
    RepositoryConnection repositoryConnection = null;
    try {   
      repositoryConnection = getRepositoryConnection();
      
      insert(repositoryConnection);
      query(repositoryConnection);      
      
    } catch (Throwable t) {
      logger.error(WTF_MARKER, t.getMessage(), t);
    } finally {
      repositoryConnection.close();
    }
  }  
}

Conclusion

In this brief post I gave a quick example of how you can setup a simple GraphDB repository and query it using SPARQL. You can find sample code on github.