Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DB module #25

Merged
merged 10 commits into from
Jan 4, 2020
Merged

Add DB module #25

merged 10 commits into from
Jan 4, 2020

Conversation

zadean
Copy link
Owner

@zadean zadean commented Nov 11, 2019

Adds a statically bound "db" module.

Improves locking.

Adds new functions:
db:put, db:link, db:get, db:delete, and db:delete-collection
put saves any data type used in XQuery to the DB, including functions, arrays and maps, and sequences of any item type except for node(). Currently, only single document or element nodes can be saved under one URI.
link can save a reference to a filename that can be retrieved later by reading the file.
get works like fn:doc but will return any type of item saved.
delete will completely delete a document or item in the database.
delete-collection will delete a collection and all sub-collections.

Still needs a test suite:
The test suite will need to test a lot of things. Most importantly, a standard order of updates.

Will solve #22 once completed.

Improved locking

New functions:
db:put, db:link, db:get, db:delete, and db:delete-collection
(can save and retrieve any data type in XQuery)

Still needs a test suite
The test suite will need to test a lot of things. Most importantly, a
standard order of updates.
@grantmacken
Copy link
Contributor

@zadean
If it's not to late for this feature set, I would like to see a db:mv ,
which like the unix variant could be used for renaming or moving stored items.

I know, this would be possible with, get, delete, put (got resource) cycle,
but I think it would be nice convenance function

@zadean
Copy link
Owner Author

zadean commented Nov 11, 2019

Definitely not too late!
It's something I pondered on, but never quite decided how to do.
I'll give it some thought.
Inner-collection is simple enough, just rename.
Non-XML stuff is easy as well, but XML (with namespaces even worse) is a bit more difficult.
Each collection has its own set of known names and namespaces, and path/structure index. I'll just need to figure out a good way swap all that information in an efficient way without having to completely rebuild all the documents in the background.

@grantmacken
Copy link
Contributor

I really like the db:link idea, if I understand it correctly.
The 'link' would reference static resources location on the host file system or perhaps a remote cdn.
I think this will act something like the use case for media-link-entries in the atom api.

The user uploads a image, using a HTML multipart form.
Now you can't store the images in the db,
so you create a link to it and store the link in the db.
With these link resources, you could as an example, use xQuery to search for available images.

Is it possible the link can have can some cowboy useful meta data,
generate_etag(Path, Size, Mtime)
detect_mimetype(Path)
that is handy for cowboy static

@zadean
Copy link
Owner Author

zadean commented Nov 15, 2019

@grantmacken yes, the db:link is very similar to the atom media-link-entry.
The saving of the file would need to happen either with the file module, or could be done outside of xqerl in the OS.
The link is just a reference from URI to filename. Upon selecting the linked URI, it is deferenced and the file contents returned as a xs:base64Binary. It can be useful when the files shouldn't be saved in the DB, but rather updated from outside. Within an XQuery context, the file is read a maximum of once, and cached. Since files can change outside xqerl's locking system, this is the only way to keep it deterministic (which it must be).

As for ETags and other content types:
I think I'll add an optional $meta as map(xs:string, xs:string) parameter to db:put that allows saving custom metadata. Also I'll add a db:metadata function that will return said metadata about a given URI. This could then be used to set custom headers (or whatever information about the data).
There could be some values that are always set with binary and text types, such as etag, timestamp. For links, this data will be retrieved/calculated from the file itself.
The timestamp for XML is already in the DB and part of the document key.
This will also allow for saving text as csv, plain, json, or whatever. or XML as text/xml, application/xml, atom , etc.

I may think about adding the $meta parameter to functions meant to get collections. It could then be used as a filter. Or maybe a custom uri-collection function that can be filtered in this way instead of calling db:meta on every entry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants