CLI (command line interface) application to retrieve list of services and datasets in a simple JSON format from nationaalgeoregister.nl (NGR), leveraging the NGR CSW service.

Known users of this spider are:

• PDOK, used to provision several geo viewers
• PDOK services QGIS plugin, by rduivenvoorde

To install from source run (from the root this repo):

python3 -m pip install -e .

This should install the cli tool ngr-spider:

$ ngr-spider
usage: ngr-spider [-h] subcommand ...

Generate list of PDOK services and/or service layers

positional arguments:
  subcommand
    services  Generate list of all PDOK services
    layers    Generate list of all PDOK layers

options:
  -h, --help  show this help message and exit

1. To generate a list of layers per service run the following:

ngr-spider layers -p 'OGC:WMS,OGC:WMTS,OGC:API features' pdok-services.json
ngr-spider services -p 'OGC:WMS,OGC:WMTS,OGC:API features' pdok-services.json

Note: you may need to install pyproj manually (pip install pyproj) on some systems.

Install dev dependencies and package from source:

python3 -m pip install -e ".[dev]"  

To format code install black en run from the root of this repo:

black .

Delete unused imports with autoflake:

autoflake --remove-all-unused-imports -i -r .

Organise and order imports with isort:

isort  -m 3 .

Run mypy for type checking (from root):

mypy ngr_spider 

When running the layers command in flat mode (--mode flat), it is possible to sort the layers in the output by passing a path to a JSON file containing, sorting rules. See below for an example a sorting rules JSON file (use for example with: ngr-spider layers --sort sorting-rules.json -n 20 -m flat --pretty -p "OGC:WMS,OGC:WMTS" output.json):

[
  { "index": 0, "names": ["opentopo "], "types": ["OGC:WMTS"] },
  { "index": 10, "names": ["^actueel_orthohr$"], "types": ["OGC:WMTS"] },
  { "index": 11, "names": ["^actueel_ortho25$"], "types": ["OGC:WMTS"] },
  { "index": 12, "names": ["^actueel_ortho25ir$"], "types": ["OGC:WMTS"] },
  { "index": 12, "names": ["lufolabels"], "types": ["OGC:WMTS"] },
  {
    "index": 20,
    "names": ["landgebied", "provinciegebied", "gemeentegebied"],
    "types": ["OGC:WFS"]
  },
  { "index": 30, "names": ["top "], "types": ["OGC:WMTS"] },
  {
    "index": 32,
    "names": ["^standaard$", "^grijs$", "^pastel$", "^water$"],
    "types": ["OGC:WMTS"]
  },
  {
    "index": 34,
    "names": ["bgtstandaardv2", "bgtachtergrond"],
    "types": ["OGC:WMTS"]
  },
  { "index": 60, "names": ["ahn3 "], "types": ["OGC:WMTS"] }
]