Have you ever had to retrieve millions of items in a json array that multi gigabyte payload? This might help!

This library reads a json stream, picks out items from a json array and writes them to a channel.

Given the file test.json

{
  "list": [
    {"key":  "value1"},
    {"key":  "value2"},
    {"key":  "value3"},
    {"key":  "value4"}
  ]
}

Call the library

in := j2c.StartFileReader("test.json")
out := j2c.ReadObjects(in, "list")

for o := range out {
    fmt.Println(o)
}

in is a instance whose struct implements Reader defined as

type Reader interface {
	Next() (b byte, eof bool)
}

out := j2c.ReadObjects(in, "list") is the line that does the work. The output out is a channel that receives objects as strings as they are made available.

Output

{"key": "value1"}

{"key": "value2"}

{"key": "value3"}

{"key": "value4"}

Currently, in small tests the library produces over 500k items per second.