pkoppstein/schema.jq

The above is a great first-order approximation inferencing algorithm. Thanks for sharing!

I'm curious if explored improving the "JSON" handling by using the "fusion" technique described in Mohamed-Amine Baazizi el al's Schema Inference for Massive JSON Datasets. An example Scala implementation may be found here from Ivan Veinhardt Lattak's "Schema Inference for NoSQL Databases" master thesis which is a great summary of the various approaches.

@btiernay - Thanks for your comments and the pointer to the article.
Would you be able to provide a small but concrete
example of a set of JSON texts and the inferred schema
you have in mind?

(My working hypothesis that it is not possible to add inference rules
to schema.jq without adding new types or breaking its commitment to
what I would call "universality". Of course specific applications can
be expected to require more constrained schemas, which is why I
created JESS (https://github.com/pkoppstein/JESS), which allows very
complex constraints to be specified. The main idea is that the schema
produced by schema.jq could be refined into a JESS schema.
Unfortunately, the JESS schema language is correspondingly complex as
well :-)

By the way, I was impressed by your "JSON like a boss"
presentation. The version that I saw is quite old but it's
still on the web, so if you're open to making revisions,
please let me know, perhaps by sending an email to
me as per the "author" tag above.

Thanks again!

Thanks for the feedback and complement! 🙇

Before giving some examples, can you please expand on “universality”? In the literature it is generally accepted that there is a trade off between conciseness and precision. It sounds like you are highly favouring conciseness in your approach?

@btiernay - Perhaps "universality" is subsumed here by the
constraint that there be no new types but
I used the term "univserality" to express
the idea that additional schema inference rules
would require some special assumptions.
For example, consider JSON objects of the form:

{"i": 1}
{"i": 2}

To infer the schema {"i": "integer"}
(meaning that .i must be an integer)
is to go well beyond the JSON type system,
and thus would best be handled by some
domain-specific or application-specific rule.

A similar case would be JSON arrays of the form:

["string", "number"]

i.e. where length ==2 and (.[0]|type=="string") and (.[1]|type=="number").

Hi @pkoppstein !

Following up on an observation I noticed in the significant time difference between the versions of schema.jq from 2017 and its current implementation ( jqlang/jq#2386 (comment))

Interested in trying to figure out if the additional conditional checks that have been implemented in the latest version are what's responsible, or something that rosetta 2 emulation on M1 was at fault for and was wondering if you might have any suggestions on valid data sets to perform a few different benchmarks.

Thanks!

@deric4 - Thanks for pointing out the mismatch. I've updated the Experimental Benchmarks page to reflect the updated version of schema.jq (https://gist.github.com/pkoppstein/a5abb4ebef3b0f72a6ed). In brief, schema.jq was updated to pay attention to the "nullable" flag, as documented in the gist. What was lost in the wash is that the default behavior changed. Omitting the nullable flag now produces slightly different results but takes considerably longer to execute.

Awesome thanks @pkoppstein !

@pkoppstein In trying to demonstrate some examples to show where I think the "JSON" extended-type can be improved, I stumbled on this case:

jq -n "$(curl -s https://gist.githubusercontent.com/pkoppstein/a5abb4ebef3b0f72a6ed/raw/b1315e7fb23a1c90df94d6425cfdba0907a6a284/schema.jq)
schema(inputs)" <<EOM
{"a":1}
{"b":2}
EOM

Which produces:

{
  "a": "JSON",
  "b": "number"
}

when I would have expected:

{
  "a": "number",
  "b": "number"
}

Curious if you think a case might be missing on L84:
i.e.:

  # [...]
  if $a == null or $a == $b then $b
  elif $b == null then $a # This?
  # [...]

Thoughts?

@btiernay - You're absolutely correct. Fixed. Thanks.

Thanks for the updates!

Here are a couple of cases I'm wondering if we can improve upon:

Case 1

Input

{"a":[]}
{"a":[1]}

Actual Output

{
  "a": "JSON"
}

Expected Output

{
  "a": [
    "number"
  ]
}

Case 2

Input

{"a":1}
{"a":{}}

Actual Output

{
  "a": "JSON"
}

Expected Output

{
  "a": [
    " ",
    "number",
    "object"
  ]
}

The above introduces the notion of type unions which are the general mechanism for loss-less schema inferencing. This is especially insightful for more complex array and object types.

@btiernay - Thanks for the two excellent examples.

I am inclined to change schema.jq in accordance with your first example:

-- the new extended-type for [] will be []
-- [] U [ T ] => [ T ]

I will have to see what changes are needed to JESS (the schema-checker) as
we obviously do not want the schema {"a": []} to imply that .a must be the empty array.

The second example, as you say, involves a significant extension of
schema.jq's extended-types. JESS should already handles these,
but I'm concerned about the performance implications for schema inference.
If these were supported as an option, what would you suggest as the option name?

UPDATE:

The update regarding [] is in V 0.0.3 at
https://gist.github.com/pkoppstein/70b7c053cf7757e1cf9665a0971d7383

Please let me know if it passes your tests; I'll then update the "gist" above.

The gist has been updated in accordance with @btiernay's "Case 1"
above, so that now (that is, with Version 0.0.3.1):

  schema(  [], [1] )         == ["number"]                 # formerly "JSON"
  schema(  {a:[]}, {a:[1]} ) == {a:["number"]}             # formerly {"a":"JSON"}

For reference, the prior version of this page (with Version 0.0.2.3 of schema.jq) can be found at
https://web.archive.org/web/20221017022540/https://gist.github.com/pkoppstein/a5abb4ebef3b0f72a6ed