This repository contains the semantic technology software that was completed in 2019 for a project funded by the California Energy Commission, Electric Program Investment Charge Program - EPIC, and led by the University of California - Berkeley. Siemens Corporation and Carnegie Mellon University operated as subcontractors to UC Berkeley to provide these semantic technology deliverables.

Credit is due to the following team:

Team leads: Jack Hodges (Siemens), Steve Ray (CMU)

Team members (Siemens): Jason Koh, Kimberly Garcia, Brian Xia

Team members (CMU): Ankit Jain, Rajat Pandey, Dimitris Tzannetos, Prabhu Saitu

Note: This README file was generated from some excellent documentation written by Dimitris Tzannetos while he was a graduate student at CMU. However, the API calls have evolved since this document. The latest API calls are described in the Word document API-calls.docx in this repository.

• Project File Structure

• Technology

• Recommended Reading

• Technical Documentation

• Directory Structure

• SPARQLMotion Scripts

  • triplestoreOperations:XMLInsert

  • triplestoreOperations:SelectByEventID

• Postman Tool

• Appendix - OpenADR : Generate Ontology Schema

• Semantic XML

• OpenADR to EPIC Mapping

  • Namespace Functions

  • Post Mapping Fix

  • EPIC to OpenADR Mapping

  • PostReverseMappingFix

• References

• SemanticArchive, SemanticVocab: Archival and current (respectively) database directories, not intended for direct editing.

• SemanticModels

  • import: contains all External Ontology Schemata which are imported by and integrated with EPIC schema.
    • import/OpenADR: Automatically Converted from XSD Schemas using TBCME's Import Tool.
  • schemas: contains the definitions of the core class and properties of the EPIC Information Model and helper extensions as well as integrations with the External Ontologies : BRICK, QUDT and SSN.
  • sparqlmotion: contains all scripts which define web service endpoints and enable the interaction with Topbraid’s Triple Store.
    • sparqlmotion/debug: fully working parts of the production-ready triplestoreOperations.sms.ttl SPARQLMotion script - left to help future developers get accustomed and experiment with the basic flow of the underlying web services.
    • sparqlmotion/triplestoreOperations.sms.ttl: Definition of web-services : XMLInsert and SelectByEventID.
  • spin: contains RDF/SPIN files that define SPIN rules and SPINMap functions which are used during and after the mapping procedures
    • spin/namespace_functions.spin.ttl: a series of modified “buildURI”s spinmap:TargetFunctions for both mapping directions were constructed to facilitate and expedite the process of mapping.
    • spin/postMappingFix.spin.ttl: SPIN Rules that fire after the creation of EPIC Instances mapped from OpenADR.
    • spin/postReverseMappingFix.spin.ttl: SPIN Rules that fire after the creation of OpenADR Instances mapped from EPIC.
  • spinmap: contains SPINMap files which define rule-based mappings between EPIC and other Demand-Response Communication Protocols. Currently one OpenADR is supported.
    • mappings/oadr2epic.ttl: SPINMap definition for mapping OpenADR triples to EPIC triples.
    • mappings/epic2oadr.ttl: SPINMap definition for mapping EPIC triples to OpenADR triples.
  • vocab: contains all the instance vocabularies that refer to the EPIC schema.

We used an Eclipse based semantic development environment called TopBraid Composer Maestro Edition (TBC-ME) from TopQuadrant. Version: 6.1 - JAVA version 1.8.0_121. This IDE allowed us to easily create SPARQLMotion scripts to define web service endpoints and interact with the Triple Store.

To install this software, import each of the three folders (SemanticModels, SemanticVocab, SemanticArchive) as projects in the TopBraid development environment.

Before starting working on Semantic Modeling, it is strongly recommended to review the material in the order listed below :

1. Semantic Modeling and SPARQL
   • Semantic Web for the Working Ontologist: Effective Modeling in RDFS and OWL
   • Learning SPARQL: Querying and Updating with SPARQL 1.1
2. TopBraid Composer IDE
   • TopBraid Composer ™ Getting Started Guide Version 5.2 or later
   • TopBraid Composer : Features Overview
3. SPIN Rules and SPINMap
   • Getting Started with SPARQL Rules (SPIN)
   • Reshaping Relational Data using SPINMap
   • SPINMAP: SPARQL-BASED Ontology Mapping
   • SPINMap: SPARQL-based Ontology Mapping [Video]
4. SPARQLMotion
   • SPARQLMotionTM Getting Started Guide
   • SPARQLMotionTM Tutorial Version 4.1 or later
   • Creating a Semantic Web Service in 5 Easy Steps

This section provides a technical overview of the project structure, OpenADR Schema Ontology conversion, SPINMap files, SPARQLMotion script implementation, script interaction, and generated output. The purpose of this section is to help future developers get started on the project.

The high-level structure of the “Semantic Models” directory is presented below :

Figure 1: EPIC Semantic Models Project Structure

• The functions directory contains files which define “helper” functions specific to the EPIC project and assist in manipulating the properties of RDF Instances - For more details, refer to Post Mapping Fix.
• The import directory contains all External Ontology Schemata which are imported by and integrated with EPIC schema. For more details, refer to OpenADR : Generate Ontology Schema.
• The mappings directory contains SPINMap files which define rule-based mappings between EPIC and other Demand-Response Communication Protocols - Currently only OpenADR is supported. For more details, refer mainly to OpenADR to EPIC Mapping and Semantic XML.
• The sandbox directory contains all files which are used during a Proof-of-Concept development procedure and should be removed on the production-ready version - Left as a convenient place for future developers to experiment safely with new features and ideas.
• The schemas directory contains the definitions of the core class and properties of the EPIC Information Model and helper extensions as well as integrations with the following External Ontologies : BRICK, QUDT and SSN.
• The sparqlmotion directory contains all scripts which define web service endpoints and enable the interaction with Topbraid’s Triple Store. The subdirectory **debug **contains fully working parts of the production-ready triplestoreOperations.sms.ttl SPARQLMotion script and is left to help future developers get accustomed and experiment with the basic flow of the underlying web services. For more details, refer to SPARQLMotion Scripts.
• The spin directory contains RDF/SPIN files that define SPIN rules and SPINMap functions which are used during and after the mapping procedure. For more details, refer to Namespace Functions.
• The vocab directory contains all the instance vocabularies that refer to the EPIC schema.

This section describes the functionality and provides an overview of the Basic Flow of the webservice endpoints that were created to interact with the Triple Store. Again, the latest API calls are described in the Word document API-calls.docx in this repository.. The two API calls documented below are not the current ones, but the descriptions are included here as a tutorial on how the webservice calls are constructed.

Receiving incoming OpenADR messages from the grid involves several steps :

1. Parsing the incoming XML message
2. Converting the XML into RDF triples according to the OpenADR schema
3. Transforming the OpenADR triples into EPIC triples
4. Storing the EPIC triples in the triplestore

Figure 12. SPARQLMotion script for receiving OpenADR XML messages

As described in the Section Namespace Functions, in order to be able to distinguish between instances of the same class, we used the convention of having unique namespaces (base URIs) among RDF triples that correspond to different OpenADR XML messages. To guarantee the uniqueness of each namespace, we append the timestamp of the arrival to the baseURI.

Module “Create baseURI with Timestamp” creates the unique baseURI by appending the timestamp to the prefix URI specified by the user (baseURIString).

Figure 13. Create unique namespace by appending timestamps

The module labeled “ConvertOpenADRXMLToRDF” receives the XML message as a string (xmlStringInput) and interprets the string according to the OpenADR schema imported as oadr-2.ttl.

Figure 14. Convert XML to RDF via Semantic XML annotations

The RDF triples are then handed to the “Map OpenADR to EPIC instances” module that uses the oadr2epic.ttl SPINMap™ specification file to map the OpenADR triples to EPIC triples.(Fig. 15).

Figure 15. Configuration Panel of TopSPIN Module

The final step of the script involves inserting the newly defined instances into a named graph in our triplestore, which is accomplished using the “Update Named Graph” module, taking the name of the graph as an input argument (graphURIString). (Figure 16). \

Figure 16. Definition of Update Query used to insert EPIC triples to named graph

Generating messages in any desired format, including OpenADR encoded as XML, is more or less a matter of doing the previous steps in reverse. Specifically:

1. Retrieving the desired triples from the triplestore.
2. Transforming the EPIC triples into the triples consistent with a target schema, such as OpenADR.
3. Converting the triples into the desired encoding, such as XML.
4. Sending the resulting encoded message.

Figure 17. SPARQLMotion script for reconstructing OpenADR XML messages

Module “Get EPIC Instances associated with EventID” performs a SPARQL CONSTRUCT (equivalent to an SQL SELECT) call to retrieve the triples associated with an EventID specified as an input argument. Note that using this approach it will be just as easy to generate XML-encoded OpenADR as it is to generate data according to any other schema, provided that schema definition file is available in OWL.

Figure 18. Definition of CONSTRUCT statement to Filter Instances that are associated with a specific EventID - (filter by namespace convention)

The “filtered” EPIC RDF triples are then handed to the “Map EPIC to OpenADR ” module that uses the epic2oadr.ttl SPINMap™ specification file to map the EPIC triples to OpenADR triples, following exactly the same procedure/options are described previously in “Map OpenADR to EPIC instances” module.

The final step of the script involves converting the newly defined instances to XML which is accomplished using the “Convert RDF to OpenADR XML” module via Semantic XML (Figure 19).

Figure 19. Specify the sxml:Document to be used

Postman is a powerful HTTP client which provides a simple GUI to make requests to the RESTful APIs. It is strongly recommended to install Postman in order to test the web service endpoints from external tools rather than from within TBC-ME debug panel. For instructions on how to install the Chrome Extension, you can refer to this video.

As an example of how one integrates external standards, this section details how to import the OpenADR specification, map the contents to local classes, and translate messages from and to OpenADR-compliant XML.

As defined in EPIC’s Technical Specification Document [2], the Information Storage receives pricing and response events that are sent from the Utility using the OpenADR model.

Receiving incoming OpenADR messages from the grid involves several steps:

1. Parsing the incoming XML message
2. Converting the XML into RDF triples according to the OpenADR schema
3. Transforming the OpenADR triples into EPIC triples
4. Storing the EPIC triples in the triplestore

In order to be able to parse the incoming OpenADR XML message, then convert it to RDF triples compliant with the OpenADR Protocol, we must first define an Ontology Schema. We are able to automatically generate the OpenADR schema using the “Import XSD Schema Files” feature of TBC-ME. At this point it is strongly recommended to review the Import procedure described in TBC-ME Help > Import XSD and corresponding XML or or access this link while TBC-ME is running locally on your machine as well as this Blog Post. The required XSD files can be found here.

It is important to select the last option “Use only XSD datatypes for datatype properties and literals”.

Figure 2: Option Selection during “Import XSD Schema Files”

By enabling the last option, TBC-ME Import Tool will automatically convert any ObjectProperty which actually refers to a xsd:Datatype and change it to the appropriate literal.

After a successful conversion, you should see the following turtle (.ttl) files : \

Figure 3: OpenADR Ontology using “Import XSD Schema Files”

At the time of this writing, an Issue has been discovered and reported to TopBraid’s Google Group concerning the case when the enclosing namespace of an XML element is different than the datatype namespace of its child. This issue is best illustrated through a simple example. Consider the following OpenADR XML message :

<?xml version="1.0" encoding="UTF-8"?>
 <oadr:oadrDistributeEvent ei:schemaVersion="2.0b"   xsi:schemaLocation="http://openadr.org/oadr-2.0b/2012/07 oadr_20b.xsd"    xmlns:pyld="http://docs.oasis-open.org/ns/energyinterop/201110/payloads" xmlns:oadr="http://openadr.org/oadr-2.0b/2012/07"  >
	<pyld:requestID>REQ_12345</pyld:requestID>
</oadr:oadrDistributeEvent>

In this case, oadr:oadrDistributeEvent has a nested 'child' Datatype element (pyld:requestID) which os defined in a different namespace (pyld - alt. payloads). Instead of attaching a Datatype property to the parent class, the Import Tool creates an owl:ObjectProperty in the namespace of the 'parent' that points to a Datatype Class in the child's namespace, as shown below :

However, the payloads:RequestID is not defined anywhere and as a result the RDF instances are not populated correctly.

In the newly created instance of type oadr:oadrDistributeEvent, the Object property p2012-07:requestIDRef has not been populated. Instead, the nested pyld:requestID XML element has been mapped to an instance of class pyld:requestID which has been created on the fly by TBC-ME.

As it has already been stated the problem arises because the Import Tool creates an owl:ObjectProperty in the namespace of the 'parent' that points to a Datatype Class in the child's namespace as shown in the Figure below :

The solution to this issue is to manually hand-craft the appropriate Datatype properties in the child’s namespace - when they were originally defined by the XSD Schema File. In our example, we can solve the problem by creating the owl:DatatypeProperty payloads:requestID. It is vital to also transfer the sxml annotations to ensure that the round-tripping will be successful. The fix is presented in the following Figure.

NOTE : An owl:ObjectProperty is annotated with an sxml:tag whereas an owl:DatatypeProperty is annotated with an sxml:attribute, reflecting the original XML structure.

After we have defined the appropriate owl:DatatypeProperty in the child’s namespace and deleted the owl:ObjectProperty that has been incorrectly generated by TBC-ME, we have to also update the property definitions of the parent class as shown :

After applying the manual fix, we get the desired output as presented below :

For any updates and a possible solution by TopQuadrant, please refer to the GoogleGroup Thread.

At this point, it is also recommended to review the Help>Creating, Importing, Querying, Saving XML documents with Semantic XML or access this link while TBC-ME is running locally on your machine as well as watch this Video-Tutorial. Semantic XML is a technology from TopQuadrant for mapping arbitrary XML documents to RDF. Semantic XML can be used to convert XML files to OWL so that you can run SPARQL queries,etc on them. You can also manipulate the resulting OWL model and then save it back to XML. In order to control the Semantic XML mappings, you can use process ontologies, which can usually be generated by importing them from XML Schemas.

Semantic XML is based on a small ontology that defines various classes and attributes to enable round-tripping between XML and OWL. In particular it introduces two annotation properties sxml:element and sxml:attribute that are used to define the mapping. In our case, the process ontology is the converted OpenADR. The following mapping strategy will be applied (excerpt from TopBraid’s Documentation):

• Each element instance will be mapped into an anonymous instance of an OWL class having the element name as an sxml:element or a sxml:tag property value.
• Each attribute or child element with a simple type but no enumeration will be mapped into an OWL datatype property having the attribute or element name as an sxml:attribute property value. The attribute or child element instance value will be mapped into a typed literal, where the type is specified as the owl:allValuesFrom property value in a restriction on the OWL class of the mapped instance.
• Each child element with a complex type will be mapped into an OWL object property having the element name as a sxml:tag property value. The element instance value will be mapped into an anonymous instance of the OWL class, which is specified as the owl:allValuesFrom property value in a restriction on the OWL class of the mapped parent instance.
• Each attribute or child element with an enumeration type will be mapped into an OWL object property having the attribute or element name as a sxml:tag property value. The attribute or child element instance value will be mapped into an existing instance with the same dtype:value property value and has an OWL class, which is a subclass of an EnumeratedValue.
• The ordering of child element instances are preserved either in composite:index property values of the mapped instances or sxml:order property values in the restrictions on properties related to the child element.
• If a child element is defined in an XML Schema using a substitutionGroup, then it will be mapped into the OWL object property, which was derived from the substitutionGroup. However, the original type of the child element instance will still be used while mapping the instance itself.
• If an element instance uses xsi:type, then the mapping from that type will override the default type.

This section describes the process of mapping OpenADR RDF triples to EPIC triples using a SPINMap specification file with Graphical Notation. Mapping is hard to do programmatically because the names of compatible properties are often quite different. Examples of these mapping specifications . This mapping process not only creates new class instances according to the EPIC model, but also creates the relations between the newly created instances. A listing of all the OpenADR classes mapped to EPIC classes is shown in Figure 4 (20 mappings in total).

A new SPINMap file should be created for each generated JSON ontology using the following steps :

• Create a new SPINMap Mapping File.
• Once the file is created, add the two ontologies to map. (e.g., OpenADR and EPIC, A and B, etc.)
• Select a class from ontology “A” using the classes pane.
• Select the “Diagram” tab at the bottom of the “Class Form”.
• Find the equivalent class from ontology “B” using the classes pane and drag it into the diagram view.
• Drag an arrow from the class name of ontology “A” to the class name of ontology “B”.
  • Define a mapping between instance names in the resulting popup.
• Drag arrows from the properties of ontology “A” to the the properties of ontology “B”.
  • Define arbitrarily complex SPARQL mappings between properties in the resulting popups.

Figure 4. Listing of the OpenADR classes mapped to EPIC classes

Apart from the “Target Resource” SPINMap Target Function described in detail in SPINMAP: SPARQL-BASED Ontology Mapping, another useful feature of SPINMap is the “Property Path Traversal”. It’s powerful capabilities are best shown through a simple example scenario (found under “sandbox” directory).

Figure 5. Example Scenario to illustrate the Property Path Traversal Feature for SPINMap

As depicted in Figure 5, in the “source” ontology class classes:Thing_1 has an ObjectProperty hasThing_2 that points to classes:Thing_2 which has an ObjectProperty hasThing_3 that points to classes:Thing_3 which has an ObjectProperty hasThing_4 that points to classes:Thing_4. However, in the “destination” ontology the definition of NewClasses:SubThing_1 has 3 different ObjectProperties that each one point to one of NewClasses:Thing_{2, 3, 4} respectively. In this situation, we cannot simply use the “Target Resource” SPINMap Target Function to map classes:Thing_4 to NewClasses:Thing_4 and attach it directly to NewClasses:SubThing_1 as classes:Thing_1 does not directly point to classes:Thing_4. To achieve this, we have to define the mappings between the source and destination classes and then use the “Property Path” traversal feature as described below:

1. Click on the upper right arrow of the source class and drag the arrow to the corresponding ObjectProperty of the destination class. Selections are automatically colored in green as shown below.

2. Define the property path (e.g. classes:hasThing_2/ classes:hasThing_3/ classes: hasThing_4) in the resulting popup as shown :

3. In the resulting pop-up window “Create Mapping Function”, first select “spinmap:targetResource” found under “spl:URIFunctions” of the Navigation Panel.

Then, click on the “ ” sign of the context.

1. In the resulting pop-up, select the appropriate mapping and click “OK”. NOTE: The mapping relation between source and destination classes must be already defined prior to this procedure.

1. Finally, click “OK”. If successful, you should be able to see a Graphical notation similar to the image at Step 1.

In order to be able to distinguish between instances of the same class, we used the convention of having unique namespaces (base URIs) among RDF triples that correspond to different OpenADR XML messages. To guarantee the uniqueness of each namespace, we append the timestamp of the arrival to the baseURI. When mapping to/from OpenADR from/to EPIC, we want to preserve the uniqueness of each baseURI. We achieve this by appending “_EPIC” to the baseURI of OpenADR triple when mapping from OpenADR to EPIC and by replacing “_EPIC” with “_OpenADR” when mapping from EPIC back to OpenADR. As a result, a series of modified “buildURI”s spinmap:TargetFunctions for both mapping directions were constructed to facilitate and expedite the process of mapping. A listing of the spinmap:TargetFunctions along with a brief description of their functionality and usage are shown in Table 1. For additional information regarding the underlying implementation, please refer to /spin/namespace_functions.spin.ttl.

_Table 1. Listing of custom-made spinmap:TargetFunctions used for mapping
_

In the case of Event Signals, the OpenADR Specification assumes that the Intervals which are included in a Sequence are consecutive, with no gaps nor overlaps and their sum of which must equal the full duration of the Active Period (For more details, refer to page 25 of Profile Specification). As a result, the incoming XML message provides information only for the duration of each Interval along with its uid and does not explicitly state its dtStart. Therefore, based on the dtStart of the ActivePeriod and the pair <uid, duration> of each Interval we can calculate and populate the dtStart of the mapped epic:Intervals using SPIN rules. The file of interest is /spin/postMappingFix.spin.ttl.

                 <strm:intervals>
                    	<ei:interval>
                       	 <xcal:duration>
                            	<xcal:duration>PT15M</xcal:duration>
                        	</xcal:duration>
                        	<xcal:uid>
                            	<xcal:text>0</xcal:text>
                        	</xcal:uid>
                        	<ei:signalPayload>
                            	<ei:payloadFloat>
                                	<ei:value>1.20</ei:value>
                            	</ei:payloadFloat>
                        	</ei:signalPayload>
                    	</ei:interval>
                    	<ei:interval>
                        	<xcal:duration>
                            	<xcal:duration>PT15M</xcal:duration>
                        	</xcal:duration>
    	                    <xcal:uid>
                            	<xcal:text>1</xcal:text>
                        	</xcal:uid>
                        	<ei:signalPayload>
                            	<ei:payloadFloat>
                           	     <ei:value>1.30</ei:value>
                            	</ei:payloadFloat>
                        	</ei:signalPayload>
                    	</ei:interval>
                	</strm:intervals>

_Figure 6. Excerpt of an OpenADR message illustrating Intervals of an Pricing Event

A SPIN Rule is attached to the fsgim-sequence:IntervalDataContainer Class. The rule will fire iteratively for all Intervals in the Event Sequence. Using a “Property Path” traversal (in the reverse direction now, from child to parent), the rule first finds the dtStart time of the ActivePeriod which corresponds to the dtStart of the first Interval in the sequence (assuming it has uid = ‘0’ (zero)). For any consecutive Intervals which have not yet been assigned a dtStart, the rule calculates its dtStart via the function efunc:calculateDtStart.

Figure 7. SPIN rule attached to fsgim-sequence:IntervalDataContainer Class.

efunc:calculateDtStart: Calculates the hasStart value for an interval that has UID greater than 0. This function depends on the OpenADR requirement that intervals within an Event have consecutive uid's, beginning with 0 (zero) (Figure 8).

arg1: the intervalDataContainer with a missing fsgim-time:hasStart property

Procedure:

1. Find the intervalDataContainer with UID one less than that of ?arg1
2. Add the start time and duration to compute the hasStart property for ?arg1

Figure 8. Definition of Function efunc:calculateDtStart.

efunc:timeAdd : Increment ?arg1 (UTCDateTime) by a value given by ?arg2 (duration)

Procedure:

1. Convert existing UTCDateTime and duration to milliseconds
2. Add the start time and duration to compute the hasStart property and return result as UTCDateTime

Figure 9. Definition of Function efunc:timeAdd.

NOTE: The function time-functions:DurationStrToMS uses an external simple parser in JavaScript (/functions/durationToSeconds.js) which was created according to iCalendar (RFC 5545).

When mapping EPIC instances back to OpenADR, we followed the same approach as described in OpenADR to EPIC Mapping. A listing of all the EPIC classes mapped back to OpenADR classes is shown in Figure 10 (19 mappings in total). Both Semantic XML annotations and Namespace Functions TargetFunctions proved to be of great assistance.

Figure 10. Listing of the EPIC classes mapped to OpenADR classes

As described earlier, Semantic XML introduces two annotation properties sxml:element and sxml:attribute that are used to define the mapping and to enable round-tripping between XML and OWL. When an incoming OpenADR XML message is converted to RDF, the resultant triples are instances of classes that have already been annotated and linked to an instance of the class sxml:Document that basically enables the round-tripping between XML and OWL. However, when the OpenADR RDF triples are converted to EPIC triples, the Semantic XML annotations are not preserved neither the instance of the sxml:Document is mapped to any equivalent class as it is part of TopBraid’s ecosystem and not related to the EPIC schema. This creates a problem when mapping EPIC instances back to OpenADR aiming to reconstruct the incoming OpenADR XML message via Semantic XML. In order to overcome this issue, we have created a SPIN Rule that fires whenever an instance of a specific OpenADR Event class is created as a result of mapping from EPIC. This rule - currently attached only to DistributeEvent class - creates the required sxml:Document instance and the appropriate property connections (in this case with the DistributeEvent object - Figure 11).

Figure 11. SPIN rule to create the required sxml:Document instance

1. J. Koh, S. Ray and J. Hodges, "Information Mediator for Demand Response in Electrical Grids and Buildings," 2017 IEEE 11th International Conference on Semantic Computing (ICSC), San Diego, CA, 2017, pp. 73-76. doi: 10.1109/ICSC.2017.26 [link]
2. J.Hodges, S.Ray, and K.Garcia, “EPIC Project Technical Specification.” Siemens internal document. April 2017.