XQuery

XQuery Examples Collection

Welcome to the XQuery Examples Collection Wikibook! XQuery is a World Wide Web Consortium recommendation for selecting data from documents and databases.

Current Status

2019 refurbishment

There is a Github project to track issues with this book and collaborate amongst eXist db folk at least to bring this resource up-to-date.

Main tasks in the current refurbishment are to:

• get example code executable again
• remove complex case studies which use obsolete sources
• add XQuery 3.0 examples.

Recent Changes

Search

You may search the book here:

About this Project

This is a collaborative project and we encourage everyone who is using XQuery to contribute their XQuery examples. All example programs must conform to the creative-commons-2.5 share-alike with attribution license agreement [1].

Execution of examples use an eXist demo server.

1. Instructors: please sign our Guest Registry if you are using this book for learning or teaching XQuery
2. Contributors: please see our Naming Conventions to ensure your examples are consistent with the textbook
3. Learners: If you are looking for an example of a specific XQuery language construct, technique or problem but can't find an example, please add a suggestion to the Examples Wanted section.

Introduction

1. Background - A brief history and motivation for the XQuery standard.
2. Benefits - Why use XQuery?
3. Installing and Testing - How to install an XQuery server on your system.
4. Naming Conventions - Naming standards used throughout this book.

Using XQuery

Common tasks in using XQuery with XML

1. Executing a script - Executing an XQuery script
2. Loading data - A tour of several different ways to load data into a native XML database

Example Scripts

Beginning Examples

Examples that do not assume knowledge of functions and modules.

1. HelloWorld - A simple test to see if XQuery is installed correctly.
2. FLWOR Expression - A basic example of how XQuery FLWOR statements work.
3. Sequences - Working with sequences is central to XQuery.
4. XPath examples - Sample XPath samples for people new to XML and XPath
5. Regular Expressions - Regular expressions make it easy to parse text.
6. Searching multiple collections - How to search multiple collections in a database.
7. Displaying Lists - How to take a list of values in an XML structure and return a comma separated list.
8. Extracting data from XHTML files - How to use the doc() function to get data from XHTML pages.
9. Displaying data in HTML Tables - How to display XML data in an HTML table.
10. Limiting Result Sets - How to limit the number of records returned in an XQuery.
11. Filtering Words - How to test to see if a word is on a list.
12. Saving and Updating Data - How to have a single XQuery that saves new records or updates existing records.
13. Quantified Expressions - Testing all the items in a sequence.
14. Dates and Time - Sample expressions that work with date and time values

Intermediate Examples

Assumes knowledge of functions and modules.

1. Using XQuery Functions - How to read XQuery function documents and user XQuery functions
2. Creating XQuery Functions - How to create your own local XQuery functions
3. Returning the Longest String - A function to find the longest string from a list of strings
4. Net Working Days - How to calculate the number of working days between two dates
5. Tag Cloud - Counting and viewing the number of keywords
6. String Analysis - Regular expression string analysis
7. Manipulating URIs - How to get and manage URIs
8. Parsing Query Strings - Parsing query strings using alternate delimiters.
9. Splitting Files - Splitting a large XML files into many smaller files.
10. Filling Portlets - How to fill regions of a web page with XQuery
11. Filtering Nodes - How to use the identity transform to filter out nodes and add nodes
12. Limiting Child Trees - You have a tree of information and you want to "prune" only at a specific level
13. Higher Order Functions - Passing functions as arguments to functions
14. Timing Fibonacci algorithms - A couple of Fibonacci algorithms and timing display
15. Using Intermediate Documents - Analysis of a MusicXML file
16. Formatting Numbers - using picture formats to format numbers
17. Uploading Files - how to upload files using HTML forms
18. TEI Concordance - How to build a TEI-based concordance
19. Queries on Tables - How to extract data from tabular data
20. Namespace Constructors - how to dynamically add namespaces when constructing documents

Search

1. Introduction to XML Search - An overview of XML search terminology
2. Basic Search - A simple search page
3. Searching, Paging and Sorting - Searching and Viewing search results
4. Keyword Search - full text search with Google-style results
5. Employee Search - an Ajax example
6. Lucene Search - using eXist's Lucene-based fulltext search
7. Advanced Search - creating complex searches using multiple search fields
8. Open Search - creating an OpenSearch file to describe your search page
9. Auto-generation of Index Config Files - scripts to automatically generate the index configuration file

Transformations

Using XQuery to transform between XML and other representations

1. Parsing CSV - Using FLWOR on sequences to parse CSV text.

Interaction

1. Getting URL Parameters - How to get parameters from the URL.
2. Getting POST Data - How to get XML data posted to an XQuery.
3. Checking for Required Parameters - How to check for a required parameter using if/then/else.
4. Adder - Retaining sate in interactions :Creating a web service that adds two numbers.
5. Chaining Web Forms - Passing data from one web page to another using URL parameters, sessions or cookies
6. Simple XForms Examples
7. Incremental Search of the Chemical Elements - AJAX
8. Sending E-mail - How to send an e-mail message from within an XQuery

Visualization

1. Graph Visualization with Graphviz
2. Google Chart Sparkline - Creating a Tufte Sparkline using the Google Charts API
3. Google Chart Bullet Bar - Creating a bullet bar using the Google Charts API

Creating Custom Views

These examples use reflection on the structure of an XML document using name() to implement generic functions for XML transformations.

1. HTML Table View - A generic HTML table representation
2. Tree View - A generic HTML tree representation
3. Grouping Items - how to group items in a report

Transforming Complex XML Documents

XQuery has many features that allow you to transform XML and create full document-style transformation libraries. Unlike traditional "database" documents, complex XML documents have "complex content" that includes in-line elements in unpredictable order. This section provides a foundation for these transformations based on using the XQuery typeswitch functions. Typeswitch function transformations replace XSLT transforms but can also access indexes for very fast transforms of large collections.

1. Transformation Styles overview of the three styles of transformation
2. Typeswitch Transformations Using the typeswitch function for document-style transforms.
3. Transformation idioms Handling transformation tasks
4. Generating Skeleton Typeswitch Transformation Modules Using XQuery to generate a skeleton module for typeswitch-based document transformation
5. Web XML Viewer Using the typeswitch function to transform an XML document to HTML

Paginated Reports

Unlike HTML pages, paginated reports use the concept of text flows between pages. These examples show you how to convert raw XML into high-quality PDF files suitable for printing. The examples use a markup standard called XSL-FO for "Formatted Objects"

1. Installing the XSL-FO module - update your 1.4 configuration to get the current software from the Apache web site
2. Generating PDF from XSL-FO files - Converting XML-FO to PDF files
3. XSL-FO Tables - Generating XSL-FO tables from XML files
4. Converting HTML tables to XSL-FO tables - use a XQuery typeswitch transform to convert HTML tables to XSL-FO tables
5. XSL-FO Images - Embedding images in generated (PDF) files

Content Publishing

1. Publishing Overview - How to transfer a document from an internal intranet server to a public web site
2. Publishing to Subversion - How to transfer a document from an internal intranet to a public SVN server using SSL and digest authentication

XML Document Comparison and Merging

1. Compare two XML files - using the eXist compare() function to test to see if two XML files are exactly the same
2. XML Differences - displaying the difference between two XML files
3. Compare with XQuery - Using XQuery to Compare Lists
4. Time Comparison with XQuery - Using XQuery to Compare Dated Items
5. Synchronizing Remote Collections - Using lastModified time stamps to see what items have changed
6. Finding Duplicate Documents - Using a hash function to find duplicate documents

Time Based Queries

1. Time Based Queries - using dates and times to limit search results
2. Timing a Query - profiling how long a query takes

XML document kinds

TEI documents

Text Encoding Initiative.

1. TEI Concordance - How to build a TEI-based concordance
2. TEI Document Timeline - Using Simile Timeline to visualize a TEI document

DocBook Documents

1. DocBook to HTML
2. DocBook to PDF
3. DocBook to ePub
4. DocBook to Microsoft Word

OpenOffice

1. OpenOffice to HTML

Office Open XML

1. Office Open XML

XML Schemas

1. XML Schema to Instance
2. XML Schema to XForms
3. XML Schema to SVG

Processing Special Characters

1. Special Characters - dealing with newlines and other special characters.

XQuery and other languages

MusicXML

1. Using Intermediate Documents Analysing MusicXML documents
2. MusicXML to Arduino

Language Comparisons

Python

1. XQuery and Python

SQL

1. XQuery SQL Module - Calling SQL from within your XQuery
2. XQuery from SQL - Using XQuery to access a classic Relational database - Employee/Department/Salary

RDF/OWL

1. List OWL Classes - A simple XQuery script that will display all the OWL classes in an OWL file

Language combination

Excel

1. Excel and XML

JavaScript

1. Navigating Collections - basic AJAX
2. Employee Search - basic AJAX
3. DOJO data - basic JSON

SQL

1. XML to SQL
2. Microsoft-Access

XHTML Voice

1. Simple RSS reader
2. XHTML Voice Twitter Radio for Opera

XSLT

1. XQuery and XSLT Executing an XSLT transform from within XQuery

Data Mashups

Authentication

1. Basic Authentication - Logging in to a remote web server using HTTP Basic Authentication
2. Digest Authentication - Logging in to a remote web server using HTTP Digest Authentication
3. OAuth - A standard for protecting a set of user-owed data within a web service

Wikipedia interaction

1. Wikipedia Page scraping
2. Wikipedia Lookup
3. Wikipedia Events RSS
4. Wiki weapons page

Wikibook applications

1. Wikibook index page
2. Wikibook list of code links

Freebase

1. Freebase - use XQuery to get data via JSON from Freebase

Google Docs

1. Google Docs - use XQuery to get data from Google Docs

Visualization

1. Graph Visualization
2. Graphing from RDF
3. Dataflow diagrams
4. Sequence Diagrams
5. Example Sequencer - Step-by-step tutorial

Google Charts

Although the Google Charts functions only work when you are connected to the Internet, these examples show that XQuery is an ideal tool for converting XML data into charts.

1. Google Charts Using XML and XQuery to generate Google Charts using REST
2. Google Chart Sparkline - A demonstration of how to create a chart using the Google Charts API
3. Google Chart Bullet Bar - A demonstration of how to a dashboard bullet bar using the Google Charts API
4. Histogram of File Sizes - An XQuery report that generates a histogram of file sizes

There are also sample XForms that can be used to create front-ends in the XForms Tutorial and Cookbook

Digital Dashboards

Digital dashboards are single screens that compress a great deal of information into a single web page. This section will leverage many of the Google Charts examples from the prior section.

1. Dashboard Architecture - How to design dashboards that have fast response times

Page Scraping

Page scraping is the process of extracting well-formed XML data from any HTML web page. When creating mashup applications this is also known as the harvesting process.

1. Overview of Page Scraping Techniques
2. Page scraping and Yahoo Weather
3. UK shipping forecast
4. BBC Weather Forecast
5. Page scraping and Mashup
6. Simple RSS reader
7. Multiple page scraping and Voting behaviour
8. Link gathering
9. REST interface definition
10. Caching and indexes

Mapping

1. Google Geocoding
2. /String Analysis#Location Mapping/ Mapping Car Registrations
3. Flickr GoogleEarth
4. Nationalgrid and Google Maps
5. SMS tracker

Timelines

1. Creating a Timeline - Creating a simple timeline view of events
2. Timelines of Resource - Using creation and modification dates to create timelines
3. TEI Document Timeline - Creating a timeline of all dates within a single TEI document

The Semantic Web

1. DBpedia with SPARQL - Football teams
2. DBpedia with SPARQL and Simile Timeline - Album Chronology Creating a timeline of album releases using data from Wikipedia
3. DBpedia with SPARQL - Stadium locations
4. The Emp-Dept case study
   1. XML to RDF
   2. SPARQL Tutorial
   3. SPARQL interface
5. Graphing Triples
6. SPARQLing Country Calling Codes
7. Southampton Pubs
8. Alphabet Poster
9. Simile Exhibit Browser visualizations using the Simile JavaScript libraries
10. Latent Semantic Indexing Finding the semantic distance between documents

Development Tools

1. Sitemap for Content Management System XQuery functions can easily perform many common web site content management functions
2. Uptime monitor use XQuery to monitor a remote web service
3. XQuery IDE - XQuery Integrated development environment
4. Image Library - using an XQuery to preview your images
5. XML Schema to Instance - XQuery function to generate a sample XML instance from an XML Schema file (.xsd)
6. Lorum Ipsum text - generating sample text for inserting into test page layouts
7. XQuery and XML Schema - Generating an XML instance document
8. Generating XQDocs - Automating the generation of XQuery documentation for Modules and Functions
9. XqUSEme - Firefox extension to allow XQueries including against the loaded document (even against originally non-XML (poorly formed) HTML).
10. Call Graphs - Generating an call graph as an SVG file from module introspection
11. System Properties - getting a list of all standard and java system properties and their values
12. Environment Variables - getting a list of all the external environment variables and their values
13. Generating xqDoc-based XQuery Documentation - generating module documentation using the xqDoc commenting standard

Validation

1. Validating a document - Validate a document with an XML Schema
2. Validation using a Catalog - Using a Catalog file to validate documents
3. Validating a hierarchy -
4. Validation with Schematron - Apply specific rules to check a document

Path Analysis

1. All Paths - A report of all paths in a document or collection
2. All Leaf Paths - A report of all leaf paths in a document or collection

Security

1. Login and Logout - How to log users in and log them out
2. URL Driven Authorization How to use URL rewriting to check for valid users
3. Digital Signatures - How to use a custom module to use the XML Digital Signature standards
4. Changing Permissions on Collections and Resources - how to change permissions on collections and resources

Unit Testing Tools

1. XUnit Testing - what is XUnit testing and how to set them up
2. XUnit Annotations - how to add annotations to your XQuery functions to drive your XUnit tests

Case Studies

1. Fizzbuzz
2. Project Euler
3. Topological Sort
4. Slideshow
5. Sudoku
6. Pachube feed
7. World Temperature records - conversion of text data formats to XML, indexing and data presentation
8. UWE StudentsOnline

eXist db specific Functions and Configuration

Configuration

Installing modules

1. Installing the XSL-FO module

Setting HTTP Headers

1. Setting HTTP Headers

Modules

dbutil

1. Database Utilities - a set of database utility functions for database administration

compression

Function Reference

1. Get zipped XML file
2. Unzipping an Office Open XML docx file - Uncompressing and storing a docx file

eXgit

Module for interacting with any git revision control system

1. Installing eXgit

EXPath File Transfer XQuery Extension Module

This module does both FTP, SFTP (using the SSH protocol) using a standardized and consistent format.

1. File Transfer Client

ftp client

This module allows you to interact with a remote FTP server on a remote system. It includes functions for listing, getting and putting files.

1. FTP Client

httpclient

Function Reference

1. Digest Authentication
2. UK shipping forecast

JSON

1. Convert XML to JSON

Lucene

Function Reference Help

1. Lucene Search

mail

Function Reference

1. Sending E-mail
2. Basic Feedback Form

math

Function Reference

1. Using the Math Module

memcached

1. Using the Memcached Module

process

1. Execute External Process - execute a shell command from within XQuery

request

Function Reference

1. Getting URL Parameters
2. Getting POST Data
3. Checking for Required Parameters
4. Manipulating URIs
5. Parsing Query Strings
6. Adder simple client-server interaction

scheduler

Function Reference Help

1. XQuery Batch Jobs

sequences

Function Reference

1. Sequences Module - three additional functions (map, fold and filter)

session

Function Reference

1. Basic Session Management - the basics of session management including getting a setting session variables

subversion

Function Reference

1. Subversion - how to update a subversion repository from within XQuery using the subversion client

transform

Function Reference

1. String Analysis

util

Function Reference

1. Registered Modules : util:registered-modules()
2. Registered Functions : util:registered-functions()
3. Dynamic Module Loading : util:import-module(), util:eval()
4. Higher Order Functions : util:function(), util:call()
5. Timing Fibonacci algorithms : util:function(), util:call(), util:system-time()
6. XMP data : util:binary-doc(), util:binary-to-string(), util:parse()
7. Basic Authentication : util:string-to-binary(), httpclient:get()

validation

Function Reference Help

1. Validating a document

xmldb

Function Reference

1. Saving and Updating Data
2. Splitting Files

xqdoc

Function Reference

1. Generating xqDoc-based XQuery Documentation

xslfo

XSL-FO (Formatted Objects) is a way of converting XML into PDF. Function Reference

1. Installing the XSL-FO module - setting up your XSL-Module within eXist
2. Generating PDF from XSL-FO files - generating PDF from a FO file
3. XSL-FO Tables - adding tables to your PDF
4. XSL-FO Images - adding images to your PDF
5. XSL-FO SVG - adding SVG images to your PDF

Triggers

1. Using Triggers to Log Events - how to set up a trigger to log store, update and remove events on a collection
2. Using Triggers to assign identifiers - how to use triggers to assign identifiers to new documents or new nodes
3. Sending E-mail Email is one way to notify when a trigger has fired

XQuery Updates

1. Inserting and Updating Attributes
2. Updates and Namespaces - How updates can change serialization

URL Rewriting

1. URL Rewriting Basics How to make your URLs look nice

Apache Ant Tasks

1. Reindex a Collection

Replication

1. eXist Replication

General guidance

• eXist Crib sheet
• eXist-db maintenance - delete versions of binary files

Appendixes

Systems that Support XQuery

Using native and hybrid XML databases that implement XQuery

1. BaseX - Native open source XML Database with visual frontend
2. DataDirect XQuery - Java XQuery engine supporting relational, EDI, flat files and XML input/output
3. eXist - Open source native XML database
4. DB2 pureXML - DB2 9.1 includes the pureXML feature
5. MarkLogic Server - MarkLogic Server commercial XML Content Server
6. Microsoft SQL Server 2005 -
7. NetKernel - NetKernel
8. Oracle Berkeley DB XML - Open source embedded storage management
9. Oracle XML DB - Oracle Server 11g includes the XML DB (XDB) feature
10. Sedna - Open source native XML Database
11. Stylus Studio - XQuery mapping/editing/debugging, ships with Saxon (and SA) and DataDirect XQuery
12. EMC xDB - EMC Documentum xDB commercial native XML database
13. XQilla - Open source XQuery library and command line utility
14. Zorba - Open source XQuery engine C implementation with C, Java, Php, Python, Ruby library bindings and command line utility
15. Qizx - Open source and pro XQuery engine Java implementation

Debugging XQuery

1. Gotchas - some pitfalls
2. Ah-has - some ah-ha moments

Other sources

Function Libraries

1. FunctX XQuery Function Library by Priscilla Walmsley

Discussion Groups

1. XQuery General

Indexes

• Index of Application Areas - edited

Key to symbols:

indicates an XQuery/Best practice