Introduce JSON_EXTRACT function #4743

wey-gu · 2022-10-19T03:49:03Z

close: #3513

Note, we don't support the path argument in this phase

What type of PR is this?

bug
feature
enhancement

What problem(s) does this PR solve?

Issue(s) number: #3513

Description:

It takes a string and returns a map:

YIELD JSON_EXTRACT('{"a": 1, "b": 2}') AS map_foo
 ------------- 
| map_foo     |
 ------------- 
| {a: 1, b: 2}|
 -------------

highlight ( should be included in docs, too)

"foo", "1", "true" are all valid JSON String though(RFC8259), they are treated as BAD_DATA here due to this function is trying to return either empty/null or Map() in its return value, thus "foo" will be considered to be dirty value.

  Scenario: Test Cases With Invalid JSON String
    When executing query:
      """
      YIELD JSON_EXTRACT('fuzz') AS result;
      """
    Then the result should be, in any order:
      | result   |
      | BAD_DATA |

limitation

Only Bool, Double, Int, String value and null are supported
Only depth-1 nested Map is supported now.

Here nested Map depth is 1, it's parsed perfectly:

> YIELD JSON_EXTRACT('{"a": 1, "b": 0.2, "c": {"d": true}}')
 ------------------------------------------------------ 
| JSON_EXTRACT("{"a": 1, "b": 0.2, "c": {"d": true}}") |
 ------------------------------------------------------ 
| {a: 1, b: 0.2, c: {d: true}}                         |
 ------------------------------------------------------

Here nested Map depth is 2, the nested item is left as an empty Map():

> YIELD JSON_EXTRACT('{"a": "foo", "b": 0.2, "c": {"d": {"e": 0.1}}}')
 ---------------------------------------------------------------- 
| JSON_EXTRACT("{"a": "foo", "b": 0.2, "c": {"d": {"e": 0.1}}}") |
 ---------------------------------------------------------------- 
| {a: "foo", b: 0.2, c: {}}                                      |
 ----------------------------------------------------------------

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

Unit test(positive and negative cases)
Function test
Performance test
N/A

Affects:

Documentation affected (Please add the label if documentation needs to be modified.)
Incompatibility (If it breaks the compatibility, please describe it and add the label.）
If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

Added function: JSON_EXTRACT to enable parsing JSON format string into Map.

close: #3513 Note, we don't support the path argument in this phase

src/common/datatypes/Map.h

removed the unecessary interface of construct Map from Value

jievince

LGTM

Only primitive types are supported

jievince

The ctest failed, please fix it.

src/common/datatypes/Map.h

jievince

Lgtm

jievince

Lgtm

Shylock-Hg · 2022-10-21T09:19:48Z

Plz add test in tck.

wey-gu · 2022-10-21T09:31:54Z

Plz add test in tck.

~~OK, will do so.~~
Done :)

wey-gu · 2022-10-21T12:59:04Z

@Shylock-Hg @jievince @Sophie-Xie it's now ready and rebased :)
Thanks!

src/common/function/FunctionManager.cpp

* Introduce JSON_EXTRACT function close: #3513 Note, we don't support the path argument in this phase * address jievince's review commit removed the unecessary interface of construct Map from Value * Type handling Only primitive types are supported * Support depth1 nested * lint: fmt * ut: ctest, fixed wrong expression of Map * fix ut errors * tck: case for json_extract added Co-authored-by: Sophie <84560950 [email protected]>

* Fix/find start error (#4771) * Fix find start error. * Fix test. Co-authored-by: Sophie <84560950 [email protected]> * Introduce JSON_EXTRACT function (#4743) * Introduce JSON_EXTRACT function close: #3513 Note, we don't support the path argument in this phase * address jievince's review commit removed the unecessary interface of construct Map from Value * Type handling Only primitive types are supported * Support depth1 nested * lint: fmt * ut: ctest, fixed wrong expression of Map * fix ut errors * tck: case for json_extract added Co-authored-by: Sophie <84560950 [email protected]> * Fix bug #1337 from ent (#4740) * Return an semantic error if no tag is found for a property while validating a match. * Return an semantic error if no tag is found for a property while validating a match. * Add a tck case for the fixed bug. * commented out unused codes. * add tag in tck cases * fixing tck * updated tck cases to add missing cases that are supposed to be there. * Return an semantic error if no tag is found for a property while validating a match. * Add a tck case for the fixed bug. * commented out unused codes. * add tag in tck cases * fixing tck * updated tck cases to add missing cases that are supposed to be there. * update * update tck case. Co-authored-by: Sophie <84560950 [email protected]> * Fix RollUpApplyExecutor (#4778) Co-authored-by: Sophie <84560950 [email protected]> * Fix mutil-match crash in optimization phase (#4780) fmt small fix small fix * fix subgraph step (#4776) * fix subgraph step * forbid function call in where clause * fix error Co-authored-by: shylock <33566796 [email protected]> Co-authored-by: Wey Gu <[email protected]> Co-authored-by: Cheng Xuntao <7731943 [email protected]> Co-authored-by: Yichen Wang <18348405 [email protected]> Co-authored-by: kyle.cao <[email protected]> Co-authored-by: jimingquan <[email protected]>

foesa-yang · 2022-10-31T08:16:21Z

vesoft-inc/nebula-docs-cn#2268

Introduce JSON_EXTRACT function

3c9b0b2

close: #3513 Note, we don't support the path argument in this phase

wey-gu requested a review from jievince October 19, 2022 03:49

jievince reviewed Oct 19, 2022

View reviewed changes

src/common/datatypes/Map.h Outdated Show resolved Hide resolved

address jievince's review commit

d832577

removed the unecessary interface of construct Map from Value

jievince previously approved these changes Oct 19, 2022

View reviewed changes

wey-gu added ready-for-testing PR: ready for the CI test doc affected PR: improvements or additions to documentation labels Oct 19, 2022

wey-gu dismissed jievince’s stale review via 754cf11 October 19, 2022 04:46

wey-gu added the wip Solution: work in progress label Oct 19, 2022

wey-gu force-pushed the fn_JSON_EXTRACT branch from d11d818 to ed26608 Compare October 19, 2022 06:45

Type handling

0111edd

Only primitive types are supported

wey-gu force-pushed the fn_JSON_EXTRACT branch from ed26608 to 0111edd Compare October 19, 2022 09:26

jievince reviewed Oct 20, 2022

View reviewed changes

src/common/datatypes/Map.h Outdated Show resolved Hide resolved

wey-gu added 3 commits October 20, 2022 11:54

Support depth1 nested

8636fe8

lint: fmt

afb1333

ut: ctest, fixed wrong expression of Map

bce36d7

wey-gu force-pushed the fn_JSON_EXTRACT branch 4 times, most recently from 2253175 to bf5fec9 Compare October 20, 2022 15:38

fix ut errors

772ce13

wey-gu force-pushed the fn_JSON_EXTRACT branch from bf5fec9 to 772ce13 Compare October 20, 2022 15:46

wey-gu added ready for review and removed wip Solution: work in progress labels Oct 20, 2022

wey-gu requested a review from jievince October 21, 2022 02:08

jievince approved these changes Oct 21, 2022

View reviewed changes

jievince previously approved these changes Oct 21, 2022

View reviewed changes

wey-gu requested review from Shylock-Hg and nevermore3 October 21, 2022 05:28

wey-gu added the cherry-pick-v3.3 PR: need cherry-pick to this version label Oct 21, 2022

wey-gu requested a review from Sophie-Xie October 21, 2022 07:21

tck: case for json_extract added

74fd01b

wey-gu dismissed jievince’s stale review via 74fd01b October 21, 2022 10:11

wey-gu requested a review from jievince October 21, 2022 11:00

Merge branch 'master' into fn_JSON_EXTRACT

d11989c

jievince approved these changes Oct 21, 2022

View reviewed changes

Merge branch 'master' into fn_JSON_EXTRACT

2c9c54f

Shylock-Hg reviewed Oct 24, 2022

View reviewed changes

src/common/function/FunctionManager.cpp Show resolved Hide resolved

wey-gu requested a review from Shylock-Hg October 24, 2022 03:17

Shylock-Hg approved these changes Oct 24, 2022

View reviewed changes

wey-gu requested review from Sophie-Xie and removed request for Sophie-Xie October 24, 2022 06:39

Sophie-Xie merged commit 47d6921 into master Oct 24, 2022

Sophie-Xie deleted the fn_JSON_EXTRACT branch October 24, 2022 09:07

wey-gu mentioned this pull request Oct 29, 2022

Weekly Report 2022-10-28 vesoft-inc/nebula-community#140

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce JSON_EXTRACT function #4743

Introduce JSON_EXTRACT function #4743

wey-gu commented Oct 19, 2022 •

edited

Loading

jievince left a comment

jievince left a comment

jievince left a comment

jievince left a comment

Shylock-Hg commented Oct 21, 2022

wey-gu commented Oct 21, 2022 •

edited

Loading

wey-gu commented Oct 21, 2022

foesa-yang commented Oct 31, 2022

Introduce JSON_EXTRACT function #4743

Introduce JSON_EXTRACT function #4743

Conversation

wey-gu commented Oct 19, 2022 • edited Loading

What type of PR is this?

What problem(s) does this PR solve?

Issue(s) number: #3513

Description:

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Release notes:

jievince left a comment

Choose a reason for hiding this comment

jievince left a comment

Choose a reason for hiding this comment

jievince left a comment

Choose a reason for hiding this comment

jievince left a comment

Choose a reason for hiding this comment

Shylock-Hg commented Oct 21, 2022

wey-gu commented Oct 21, 2022 • edited Loading

wey-gu commented Oct 21, 2022

foesa-yang commented Oct 31, 2022

wey-gu commented Oct 19, 2022 •

edited

Loading

wey-gu commented Oct 21, 2022 •

edited

Loading