The ingestion library assumes the document ingestion plugin from Elasticsearch has been installed.
This app is composed of two parts - the library and ingestion functions.
The core library located at /lib contains the api to interact with Elasticsearch.
The cleanIngest and alias functions take a settings object. It is in this format:
settings = {
index: 'myindex',
alias: 'alias',
host: 'http://myelasticsearchinstance:9200',
analyzer: {},
documentObject: {
"elastictype": [
{
"lastSaveDate": "2015-06-10",
"file": "Season four episode eight.pdf",
"title": "Season four episode eight"
},{
"lastSaveDate": "2013-05-10",
"file": "Season two episode two.pdf",
"title": "Season two episode two"
}
]
}
}
Note the documentObject is an array of the documents to be ingested. They will be into ingested into the index with the type specified in the settings object.
A clean ingest deletes the index (which you specify in the config), ingests documents, and optionally aliases an index.
let analyzer = JSON.parse(fs.readFileSync('./analyzer.json', 'utf8'));
let pipeline = JSON.parse(fs.readFileSync('./pipeline.json', 'utf8'));
let documentObject = JSON.parse(fs.readFileSync('./documentObject.json', 'utf8'));
let ingest = require('../');
ingest.cleanIngest({
index: 'myindex',
host: 'http://localhost:9200',
analyzer: analyzer,
pipeline: pipeline,
documentObject: documentObject
}).then(result => console.log).catch(error => console.log);
This creates an index, aliases it, and deletes previous index.
let analyzer = require('./analyzer.json');
let pipeline = require('./pipeline.json');
let documentObject = require('./documentObject.json');
let ingest = require('document-ingestion');
ingest.alias({
index: `myindex-${Date.now()}`,
host: 'http://localhost:9200',
alias: 'myalias',
analyzer: analyzer,
pipeline: pipeline,
documentObject: documentObject,
}).then(console.log).catch(console.error);
A short cut api is exposed too:
Returns an elasticsearch client bound to the host provided e.g 'http://localhost:9200'. If the same domain is passed, a cached client is returned.
Pipeline is an object, if pipeline.body exists, it is added to elasticsearch. If no body attribute it will assume that a pipeline with name pipeLine.name already exists in elasticsearch.
Example we need for an attachment ingestion:
{
"name": "attachment",
"body": {
"description" : "Attachment ingestion",
"processors" : [{
"attachment" : {
"field" : "data"
}
}]
}
}
When we want to switch aliases we need to know which index currently aliases. This function takes an elasticsearch client, and an alias, returns a promise which returns the name of the current index or null if the alias does not exist.