Archive | Semantic Web Tutorials RSS for this section

Virtuoso Servers installation, setup, data upload, and querying on Windows

Virtuoso is an innovative enterprise grade multi-model data server for agile enterprises & individuals. It delivers an unrivaled platform agnostic solution for data management, access, and integration.The unique hybrid server architecture of Virtuoso enables it to offer traditionally distinct server functionality within a single product offering that covers the following areas:
•    Relational Data Management
•    RDF Data Management
•    XML Data Management
•    Free Text Content Management & Full Text Indexing
•    Document Web Server
•    Linked Data Server
•    Web Application Server
•    Web Services Deployment (SOAP or REST)
*information taken from http://virtuoso.openlinksw.com/
The choice of Virtuoso SPARQL endpoints is because of its best performance for large data sets . Please see the below article for details. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.161.7773&rep=rep1&type=pdf
a.    Virtuoso  server download
Open Link virtuoso server can be downloaded from the following urlhttp://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSDownload. Please make sure that you have downloaded the right server according to your system operating system and specification (32 bit, 64 bit).
b.    Virtuoso  startup
Virtuoso is a portable server; therefore, it does not require any installation. Just you need to extract the zip file into some directory D:\ahmadchan in my case.  The zip extraction will create a folder with the name virtuoso-opensource containing all of the server files. Within the virtuoso-opensourcego to database directory and copy virtuoso.ini file into bin directory.
Now go to D:\ahmadchan\virtuoso-opensource\bin using command line and run the following command
Virtuoso-t  -f
A virtuoso server will be started as show in Figure 3. A SPARQL endpoint will be available at http://localhost:8890/sparql or http://your.system.ip.address/sparql as shown in Figure4.  You can directly run various SPARQL using this public interface and also you can get results using API calls. The details of the API calls can be found in the virtuoso web page. The http port for the server is 8890 which you can change in virtuoso.ini file if required.  You can also write the above command in a .bat file say start.bat within bin directory.  Then just go to bin directory and click on start.bat file. The virtuoso server will be automatically started without using command line.
c.    Loading data into virtuoso SPARQL endpoint
Various methods for inserting RDF data into virtuoso server is given at http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFInsert.  In this manual we will explain 2 of the methods.
i.    RDF upload using virtuoso conductor
This method is useful for uploading small RDF files (e.g. 100 or 200 MB files).  In this method only one file can be uploaded at one time.
Please follow the following steps .
1.    Go to the link http://localhost:8890/ and click on conductor on the left side.
2.    Type dbaas both login account and password.
3.    Click on the Linked Data tab and then Quad Store Upload
4.    Select your RDF file (only one at a time), give a proper graph name and click upload
5.    If there is no syntax error, the RDF file will be added to the virtuoso server which you can query using the public interface given at http://localhost:8890/sparql .
ii.    Using Bulk Load
Bulk load is useful for fast, large number of RDF files upload. Since our Linked TCGA contains a very large number of files, therefore this method is highly recommended for uploading tumors data.
Prerequisites
•    If your Virtuoso release is prior to the commercial 06.02.3129 or open source 6.1.3 releases, then the Virtuoso Bulk Loader functions need to be loaded manually.
•    The directory containing the data set files must be included in the DirsAllowed parameter defined in the virtuoso INI file (D:\ahmadchan\virtuoso-opensource\bin\virtuoso.ini), after which the Virtuoso server must be restarted. In our case, RDF files are stored the directory D:\ahmadchan\Clinical Data and the DirsAllowedparameter contains the default values given below.
DirsAllowed    = ., ../vad,
We need to append D:\ahmadchan\Clinical Data directory into the allowed list. This can be done by appending the directory location. The DirsAllowed parameter will become
DirsAllowed= ., ../vad,D:\ahmadchan\Clinical Data
*Please mind backward slash for windows systems.
•    The Virtuoso Server should be appropriately configured to use sufficient memory and other system resources as detailed in the Virtuoso RDF Performance Tuning Guide, or the load may take an unacceptably long time, approaching forever.  Specially, the NumberOfBuffer and MaxDirtyBuffersparameters of the virtuoso INI file should be set according to the system specification.  In our case we are using a system of 4GB RAM so we need to uncomment the corresponding two lines (highlighted in red in text given below) given in the D:\ahmadchan\virtuoso-opensource\bin\virtuoso.inifile.
;; When running with large data sets, one should configure the Virtuoso. process to use between 2/3 to 3/5 of free system memory and to stripe .storage on all available disks.
;; Uncomment next two lines if there is 2 GB system memory free
;       NumberOfBuffers          = 170000
;       MaxDirtyBuffers          = 130000
;; Uncomment next two lines if there is 4 GB system memory free
;       NumberOfBuffers          = 340000
;       MaxDirtyBuffers          = 250000
;; Uncomment next two lines if there is 8 GB system memory free
;       NumberOfBuffers          = 680000
;       MaxDirtyBuffers          = 500000
;; Uncomment next two lines if there is 16 GB system memory free
;       NumberOfBuffers          = 1360000
;       MaxDirtyBuffers          = 1000000
;; Uncomment next two lines if there is 32 GB system memory free
;       NumberOfBuffers          = 2720000
;       MaxDirtyBuffers          = 2000000
;; Uncomment next two lines if there is 48 GB system memory free
;       NumberOfBuffers          = 4000000
;       MaxDirtyBuffers          = 3000000
;; Uncomment next two lines if there is 64 GB system memory free
;       NumberOfBuffers          = 5450000
;       MaxDirtyBuffers          = 4000000
;; Note the default settings will take very little memory
;; but will not result in very good performance
Bulk loading process
Once a proper configuration has been done in the INI file,  follow the following steps for upload.
1.    Go to bin folder and click on isql (D:\ahmadchan\virtuoso-opensource\isql).
2.    Run the following command to clear any previous load list of files
SQL>delete from db.dba.load_list;
3.    Enter the following command by providing the appropriate input values
SQL>ld_dir(‘<sourcefilename-or-directory>’,'<file name pattern>’,’graph iri’);
Please mind forward slash. In our case the parameters are given below.
SQL>ld_dir (‘D:/ahmadchan/Clinical Data’, ‘*.nt’, ‘http://cbakerlab.unbsj.ca&#8217;);
4.    Next enter the command
SQL> select * from DB.DBA.load_list;
5.    Finally, enter the command to start the bulk load and wait for completion.
SQL>rdf_loader_run();
6.    After successful upload  don’t forget to run the shut down command otherwise the file will be not completely uploaded
SQL> shutdown;

Source: Virtuoso Manual

Installation and tuning of Open Link Virtuoso on Ubuntu

OpenLink Virtuoso is the first CROSS PLATFORM Universal Server to implement Web, File, and Database server functionality alongside Native XML Storage, and Universal Data Access Middleware, as a single server solution. OpenLink Virtuoso plays a vital role for semantic web programming, it can work as a solid SPARQL end point, which can be supported by almost all endpoint interfaces. There are following steps to install and start OpenLink Virtuoso universal server-

1. Install Dependencies

To install OpenLink Virtuoso first of all following dependencies are needed-

  • autoconf    2.57    http://www.gnu.org/software/autoconf/
  • automake    1.9    http://www.gnu.org/software/automake/
  • libtool    1.5.16    http://www.gnu.org/software/libtool/
  • flex    2.5.33    http://www.gnu.org/software/non-gnu/flex/
  • bison    2.3    http://www.gnu.org/software/bison/
  • gperf    2.7.2    http://www.gnu.org/software/gperf/
  • gawk    3.1.1    http://www.gnu.org/software/gawk/
  • m4    1.4.1    http://www.gnu.org/software/m4/
  • make    3.79.1    http://www.gnu.org/software/make/
  • OpenSSL    0.9.7i    http://www.openssl.org/
  • First of all check all the dependencies whether installed or not. (Version is important!)To check the version use following sequence.#autoconf –version
    # automake –version
    #libtoolize –version
    #flex –version
    #bison –version
    #gperf –version
    #gawk –version
    #m4 –version
    #make –version
    #openssl version
    During checking of the dependencies you will know whether they are installed or not and if not installed, for example,  you will get  something like-
    The program ‘autoconf’ is currently not installed.  You can install it by typing:apt-get install autoconf
    Then why should you wait!! install the packages using “apt-get” one by one in the sequence as checking the version.
    OR
    you can install these from synaptic package manager.
    2. run this command  apt-get install libssl-dev from your terminal
    3. Download OpenLink Virtuoso from source forge (Link)
    4. Extract the package in any directory.
    5. Go to the directory where you have extracted the package.
    6. It is better to run autogen.sh by typing ./autogen.sh, which checks for the presence and right version of some of the required components, and if it reports any missing package then install that package.
    7. Set following  environmental variable by typing (one by one) –
    CFLAGS=”-O3″
    export CFLAGS
    8. type  `./configure CFLAGS=”-O3″’  to configure the package for your system.
    Note: It should be “-O3”, not “-03” (big O, not zero), otherwise gcc won’t recognize the flag
    Sometimes, the system generates this error after running the ./ configure command. checking validity of the OpenSSL headers in /usr… configure: error: bad. Check config.log for details. To Resolve this, you have to install libssl-dev in your. Run the following command to install libssl-dev and again run the ./configure.
    sudo apt-get install libssl-dev
    9. Type `make’ to compile the package.
    10. Type `make install’ to install the programs and any data files and documentation.
    To start the server you have two steps-
    1. Go to
    ‘cd  /usr/local/virtuoso-opensource/var/lib/virtuoso/db’
    then type ‘/usr/local/virtuoso-opensource/bin/virtuoso-t -f &’
    Finally use the server from your browser by typing ‘http://localhost:8890/’ in the address bar.
    Use the powerful multipurpose opensource server !!
    Note:- All documentations are available at Wiki Page of  OpenLink Virtuoso.
    To grant the SPARQL to update and insert data into a graph-
    1. Go to “http://localhost:8890/conductor” and login as “dba” (Uname and Pass =” dba”)
    2. Click on “Interactive SQL (ISQL)”.
    3. In ISQL window type the command GRANT ALL PRIVILEGES TO “SPARQL”;
    To access to the virtuoso.ini file configuration and tuning:
    sudo gedit /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.ini
    To access to the isql through command line:
    Go to cd /usr/local/virtuoso-opensource/bin/
    / and then run ./isql 1111 <user> <pwd>
    By default, virtuoso sets the ‘dba’ as  user name and passwor

Remove a graph from virtuoso

To remove a graph from virtuoso, open isql through command line and then run the following command.


log_enable(3,1);
DELETE FROM rdf_quad WHERE g = iri_to_id ('http://mygraph.org');

Change Password of Virtuso dba and dev

One database user and 2 WebDAV users. These users have their passwords set to default values. It is therefore important to change them immediately after the installation.The one database user is the database administrator with username “dba” and password “dba”. This can be changed using the Interactive SQL utility. When started without parameters, the ISQL tries to log on as dba with the default password. The SQL statement to change a user’s password is:

 set password <old password> <new password>

The password is an identifier, so take care to use proper quotation.

You can also use the graphical Virtuoso Administration Interface to administer Virtuoso database users.The 2 WebDAV user accounts, dav and davuser also have their password set to their username. There are 2 easy ways to change them. Either use the GUI in Administration Interface under WebDAV Administration / WebDAV services / Users Administrator or use the SQL statement:

 update WS.WS.SYS_DAV_USER set U_PASSWORD='<new password>'
where U_NAME='<username>'

Note quotation around varchar values. Please remember to perform these operations for all Virtuoso server instances installed. By default these are the Virtuoso with an empty database and Virtuoso [demo] with the demo database.

Setup SNORQL with Sesame Triple Store

Download D2R Server v0.6 (alpha)

Direct link

TGZ file: http://downloads.sourceforge.net/d2rq-map/d2r-server-0.6.tar.gz?use_mirror=internode

ZIP file: http://downloads.sourceforge.net/d2rq-map/d2r-server-0.6.zip?use_mirror=transact

D2R Server page

http://www4.wiwiss.fu-berlin.de/bizer/d2r-server/

Extract the contents of the archive file. Edit the file d2r-server-0.6/webapp/snorql/snorql.js. Find the line this_endpoint = document.location.href.match(/^([^?]*)snorql\//)[1] + ‘sparql’;. Change sparql to openrdf-sesame/repositories/<repo name you want to use> (in this case I used new-nvcl for the repo name). Create the file namespaces.js in the same directory and add the following lines (namespaces taken from here for the NVCL repo):

var D2R_namespacePrefixes = {
owl2xml: ‘http://www.w3.org/2006/12/owl2-xml#’,
nvcl: ‘http://www.auscope.org/ontology/vocabs/nvcl/0.3#’,
xsd: ‘http://www.w3.org/2001/XMLSchema#’,
rdfs: ‘http://www.w3.org/2000/01/rdf-schema#’,
rdf: ‘http://www.w3.org/1999/02/22-rdf-syntax-ns#’,
skos: ‘http://www.w3.org/2004/02/skos/core#’,
owl: ‘http://www.w3.org/2002/07/owl#’,
};

Copy the directory d2r-server-0.6/webapp/snorql to the webapps directory on your server. Do chown -R tomcat55:nogroup on the webapps/snorql directory.

Fusuk with SEsame

https://wiki.duraspace.org/display/FF/Triplestore+Setup

 

Building SADI semantic web service

In order to make it easier to develop SADI services, we have provided a skeleton Java project. Instructions for using the skeleton are provided here, exemplified by the same Hello, World! service we used in What is a SADI service?.

Requirements

SADI requires Java 1.6.

SADI uses Maven 2 to handle dependency management, so Maven is the only base requirement. You can download Maven 2 (and obtain installation instructions) from http://maven.apache.org/download.html.

The SADI service skeleton project is optimised for the Eclipse IDE, but Eclipse is not required. If you are using Eclipse, you don’t have to download Maven; the latest versions of Eclipse include Maven integration. If you are using an older version of Eclipse that does not include Maven integration (i.e.: before version 3.7 Indigo), some of the instructions in this tutorial may not match exactly what you see and it is recommended that you either update Eclipse or follow the previous version of this tutorial.

The SADI service will be built as a Java WAR, so a servlet container is required to deploy the finished service. Apache Tomcat and Jetty are popular alternatives. The skeleton project uses the Jetty Maven plugin, so you will not need to setup a servlet container to test your service during development.

The SADI service skeleton

Download the SADI service skeleton.

If you are working from the command line, unzip the archive and change into the unzipped sadi-service-skeleton directory.

$ unzip sadi-service-skeleton-0.1.1-e3.7.zip
$ cd sadi-services

If you are working in Eclipse, import the sadi-services project into your workspace (choose File → Import → General → Existing Projects into Workspace, choose Select archive file and browse to sadi-service-skeleton-0.1.1-e3.7.zip, select the sadi-services project if it is not already selected, and click Finish).

Defining input and output OWL classes

Your service’s input and output OWL classes describe its interface to the world. The property restrictions on the input class define the properties your service needs to operate and the property restrictions on the output class define the properties your service will attach.

As described in What is a SADI service?, your input and output classes must each be identified by a URL that resolves to the class definition. Before you can proceed, you must have created the ontology describing your input and output and hosted it such that this is the case. Ontology design is beyond the scope of this document, but see DefiningInputAndOutputOWLClasses for some SADI-specific tips.

For our example, our input class will be http://sadiframework.org/examples/hello.owl#NamedIndividual. The class definition is reproduced below:

<owl:Class rdf:ID="NamedIndividual"> 
  <owl:equivalentClass> 
    <owl:Restriction> 
      <owl:onProperty rdf:resource="http://xmlns.com/foaf/0.1/name"/> 
      <owl:minCardinality rdf:datatype="http://www.w3.org/2001/XMLSchema#int">1</owl:minCardinality> 
    </owl:Restriction> 
  </owl:equivalentClass> 
</owl:Class>

This class specifies a single property restriction: that there is at least one value of the http://xmlns.com/foaf/0.1/name property. As suggested in DefiningInputAndOutputOWLClasses, this is a necessary and sufficient condition for class membership (indicated by the owl:equivalentClass construct), allowing any individual with a foaf:name to be dynamically identified as a NamedIndividual.

For our example, our output class will be http://sadiframework.org/examples/hello.owl#GreetedIndividual. The class definition is reproduced below:

<owl:Class rdf:ID="GreetedIndividual"> 
  <owl:equivalentClass> 
    <owl:Restriction> 
      <owl:onProperty rdf:resource="#greeting"/> 
      <owl:someValuesFrom rdf:resource="http://www.w3.org/2001/XMLSchema#string"/> 
    </owl:Restriction> 
  </owl:equivalentClass> 
</owl:Class>

This class also specifies a single property restriction, indicating that the service will attach a value of the http://sadiframework.org/examples/hello.owl#greeting property that is an xsd:string.

Note that, as mentioned in DefiningInputAndOutputOWLClasses, the [hello.owl http://sadiframework.org/examples/hello.owl] ontology completely specifies the properties it uses: it defines the greeting property and imports the foaf:name property.

Generating the SADI service code

Now you’ll use the SADI service generator Maven plugin to generate a Java class stub and other supporting files.

If you are working from the command line, execute the following Maven command:

$ mvn ca.wilkinsonlab.sadi:sadi-generator:generate-service 
  -DserviceName=hello 
  -DserviceClass=com.example.HelloWorldService 
  -DinputClass=http://sadiframework.org/examples/hello.owl#NamedIndividual 
  -DoutputClass=http://sadiframework.org/examples/hello.owl#GreetedIndividual 
  -DcontactEmail=your-email-address

Note: if you are copying-and-pasting the above command, be sure that it all appears on one line and substitute your own email address.

If you are working in Eclipse, there will be a preset run configuration for the service generator. Choose Run » Run configurations… and find generate sadi service under Maven Build in the left side panel of the dialog.

Once generate sadi service is selected, you can set the parameters in the main panel of the dialog by double-clicking a parameter under Parameter Name. Set the parameters as in the command-line example above and click Run.

Note: although the parameters above are the only ones required to use the SADI service generator, there are others that can be set. You can execute the help goal of the service-generator plugin (i.e.: mvn ca.wilkinsonlab.sadi:sadi-generator:help) to see all of the available parameters.

Adding business logic

The service generator plugin should now have created a Java stub under src/main/java. If you are using Eclipse, you may have to refresh the sadi-services project directory to see the generated file.

Your business logic should go in the processInput(Resource input, Resource output) method. This method will be called once for every instance of the input class received by the service. You should read the properties you need from the input Resource and attach the properties you provide to the output Resource. Resource is a class from the Jena Semantic Web Framework that represents any non-literal RDF node.) manipulate RDF. For more information, see the Jena Javadoc.

For our example, open src/main/java/com/example/HelloWorldService.java and edit it as shown below (new code is in the processInput(Resource input, Resource output) method):

package com.example;

import org.apache.log4j.Logger;

import ca.wilkinsonlab.sadi.service.annotations.Name; 
import ca.wilkinsonlab.sadi.service.annotations.Description; 
import ca.wilkinsonlab.sadi.service.annotations.ContactEmail; 
import ca.wilkinsonlab.sadi.service.annotations.InputClass; 
import ca.wilkinsonlab.sadi.service.annotations.OutputClass; 
import ca.wilkinsonlab.sadi.service.simple.SimpleSynchronousServiceServlet;

import com.hp.hpl.jena.rdf.model.Model; 
import com.hp.hpl.jena.rdf.model.ModelFactory; 
import com.hp.hpl.jena.rdf.model.Property; 
import com.hp.hpl.jena.rdf.model.Resource; 
//import com.hp.hpl.jena.rdf.model.Statement; 
//import com.hp.hpl.jena.rdf.model.StmtIterator;

@Name("hello") 
@ContactEmail("your-email-address") 
@InputClass("http://sadiframework.org/examples/hello.owl#NamedIndividual") 
@OutputClass("http://sadiframework.org/examples/hello.owl#GreetedIndividual") 
public class HelloWorld extends SimpleSynchronousServiceServlet 
{ 
    private static final Logger log = Logger.getLogger(HelloWorldService.class);

    public HelloWorldService() 
    { 
        super(); 
    } 

    public void processInput(Resource input, Resource output) 
    { 
        /* your code goes here 
         * (add properties to output node based on properties of input node…) 
         */ 
        String name = input.getProperty(Vocab.name).getString(); 
        output.addProperty(Vocab.greeting, String.format("Hello, %s!", name)); 
    }

    private static final class Vocab 
    { 
        private static Model m_model = ModelFactory.createDefaultModel(); 

        public static final Property name = m_model.createProperty("http://xmlns.com/foaf/0.1/name"); 
        public static final Property greeting = m_model.createProperty("http://sadiframework.org/examples/hello.owl#greeting"); 
        public static final Resource Literal = m_model.createResource("http://www.w3.org/2000/01/rdf-schema#Literal"); 
        public static final Resource NamedIndividual = m_model.createResource("http://sadiframework.org/examples/hello.owl#NamedIndividual"); 
        public static final Resource GreetedIndividual = m_model.createResource("http://sadiframework.org/examples/hello.owl#GreetedIndividual"); 
    } 
}

Note the inner class Vocab that’s been generated to provide easy access to the properties and classes you might need. This is the part of the generator that’s the most fragile and it may have compile errors (for example, it may attempt to define a variable with the name of a Java reserved word). These should, however, be easy to fix.

Running your service in Jetty

In order to test your service, it must first be running in a servlet container. Fortunately, the Maven Jetty plugin makes this easy.

If you are working from the command line, execute the following Maven command:

$ mvn org.mortbay.jetty:jetty-maven-plugin:run

Note: while Jetty is running, you will not be able to execute further commands in that terminal window.

If you are are working from Eclipse, there will be another preset run configuration to run your service. Choose Run » Run configurations… and find run sadi services in Jetty under Maven Build in the left side panel of the dialog. Once run sadi services in Jetty is selected click Run.

Testing your service

In order to test your service, you need an input RDF document and the corresponding expected output. For our example, we will use the following RDF input, available at http://sadiframework.org/test/hello-input.rdf.

<rdf:RDF 
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
  xmlns:foaf="http://xmlns.com/foaf/0.1/" 
  xmlns:hello="http://sadiframework.org/examples/hello.owl#"> 
    <hello:NamedIndividual rdf:about="http://sadiframework.org/examples/hello-input.rdf#1"> 
      <foaf:name>Guy Incognito</foaf:name> 
    </hello:NamedIndividual> 
</rdf:RDF>

And here is the corresponding expected output, available at http://sadiframework.org/test/hello-output.rdf.

<rdf:RDF 
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
  xmlns:hello="http://sadiframework.org/examples/hello.owl#"> 
    <hello:GreetedIndividual rdf:about="http://sadiframework.org/examples/hello-input.rdf#1"> 
      <hello:greeting>Hello, Guy Incognito!</hello:greeting> 
    </hello:GreetedIndividual> 
</rdf:RDF>

If you are working from the command line, execute the following Maven command to test the service (remember that Jetty is running in your original terminal, so you will have to open a new terminal and change to the sadi.service.skeleton directory):

$ mvn ca.wilkinsonlab.sadi:sadi-tester:test-service 
  -DserviceURL=http://localhost:8080/sadi-services/hello 
  -Dinput=http://sadiframework.org/test/hello-input.rdf 
  -Dexpected=http://sadiframework.org/test/hello-output.rdf

Note: if you are copying-and-pasting the above command, be sure that it all appears on one line.

If you are working in Eclipse, there will be another preset run configuration for the service tester. Choose Run » Run configurations… and find test sadi service under Maven Build in the left side panel of the dialog.

Once test sadi service is selected, you can set the parameters in the main panel of the dialog by double-clicking a parameter under Parameter Name. Set the parameters as in the command-line example above and click Run.

If you see BUILD SUCCESSFUL, the service produced the expected output.

Deploying and registering your service

The WAR file generated by mvn package (which will be found in the target directory) is suitable for immediate deployment to the servlet container of your choice. It will contain all of the required jars, so it might be quite large.

Once your service has been deployed, you can register it by visiting http://sadiframework.org/registry/ and submitting the URL of your service in the form on that page. If you wish to unregister your service later, simply undeploy it and resubmit the now invalid URL.