Return error for string literal beginning with single quote #2964

mattmeyers · 2023-12-05T13:43:37Z

Summary

In order for a string literal to be valid in JSON, the value must be surrounded by double quotes. This PR updates the literal validation function to return an error message when the literal begins with a single quote.

I'm not sure if this is the best solution since the line/col that gets reported is at the end of the literal and might be confusing. Another possible solution would be to mark the single quote character as invalid in the classify function. This would give the exact line/col where the invalid string begins, but I'm not sure if there would be other consequences to doing this. It appears that reporting the position after an invalid literal is seen elsewhere e.g. echo '{"foo":flase}' | jq.

Testing

Run the command echo "{'foo': 1}" | jq and ensure the output mentions an invalid string literal. Compare this to the dev version of jq which mentions an invalid numeric literal.

Closes #501

In order for a string literal to be valid in JSON, the value must be surrounded by double quotes.

wader · 2023-12-05T13:54:35Z

src/jv_parse.c

@@ -512,6 +512,8 @@ static pfunc check_literal(struct jv_parser* p) {
  switch (p->tokenbuf[0]) {
  case 't': pattern = "true"; plen = 4; v = jv_true(); break;
  case 'f': pattern = "false"; plen = 5; v = jv_false(); break;
+  case '\'':
+    return "Invalid string literal";


Maybe should even say something like "Invalid string literal. Try use double quote instead of single quote."

@wader If you want that, add it to the lexer, or it will also be printed for fromjson/0 or tonumber/0 or JSON parsing errors

Maybe it could work if you rephrase it as something like

Invalid string literal: Got ', but was expecting "

wader · 2023-12-05T13:58:09Z

Probably good to add a test case for this to tests/jq.test:

try fromjson catch .
"{'a': 123}"
"Invalid numeric literal at line 1, column 5 (while parsing '{'a': 123}')"

(but with updated error message)

About other error cases: I wonder if the could somehow check the first character to know if this if something resembling a number so that we could distinguish between invalid number and other invalid things?

mattmeyers · 2023-12-05T14:20:48Z

Probably good to add a test case for this to tests/jq.test:

Of course! I got excited and committed before looking into how tests were handled 🤦

About other error cases: I wonder if the could somehow check the first character to know if this if something resembling a number so that we could distinguish between invalid number and other invalid things?

Not sure I follow. Isn't this what the parser already does around the code I added?

wader · 2023-12-05T14:43:08Z

Of course! I got excited and committed before looking into how tests were handled 🤦

No worries :)

About other error cases: I wonder if the could somehow check the first character to know if this if something resembling a number so that we could distinguish between invalid number and other invalid things?

Not sure I follow. Isn't this what the parser already does around the code I added?

Haven't look close att the current code but it seems like the clumps together a lots of different invalid jsons texts into "Invalid numeric literal" some i think we could probably split up into more user friendly errors and suggestions:

Did a script to try different strings:

$ for i in true false nan ident truea falsea nana aa1 1aa + - / "&" "*" '(' "'aa'"; do echo "$i:\t$(jq 2>&1 <<< "$i")"; done
true:	true
false:	false
nan:	null
ident:	jq: parse error: Invalid numeric literal at line 2, column 0
truea:	jq: parse error: Invalid literal at line 2, column 0
falsea:	jq: parse error: Invalid literal at line 2, column 0
nana:	jq: parse error: Invalid literal at line 2, column 0
aa1:	jq: parse error: Invalid numeric literal at line 2, column 0
1aa:	jq: parse error: Invalid numeric literal at line 2, column 0
+:	jq: parse error: Invalid numeric literal at line 2, column 0
-:	jq: parse error: Invalid numeric literal at line 2, column 0
/:	jq: parse error: Invalid numeric literal at line 2, column 0
&:	jq: parse error: Invalid numeric literal at line 2, column 0
*:	jq: parse error: Invalid numeric literal at line 2, column 0
(:	jq: parse error: Invalid numeric literal at line 2, column 0
'aa':	jq: parse error: Invalid numeric literal at line 2, column 0

Sidenote:
No idea what is going on with -:

$ echo - | jq
$

# weirdly gojq behaves the same
$ echo - | gojq
$

itchyny · 2023-12-05T14:47:31Z

Recognized as echo command option.

wader · 2023-12-05T14:50:21Z

Recognized as echo command option.

Thanks 🤦

emanuele6 · 2023-12-11T12:56:12Z

LGTM if you add the test

mattmeyers · 2023-12-12T14:16:15Z

Hey all, sorry for the delay. It's the busy part of the year. I went ahead and updated the error message, checked for a few more invalid literals, and added some tests. Let me know if there are any more changes you would like to see.

emanuele6 · 2023-12-12T14:16:34Z

src/jv_parse.c

+  case '/':
+  case '*':
+  case '&':
+    return "Invalid literal";


In @wader's comment above, he found other character's that were triggering the invalid numeric literal. This check catches those. Would this cause an issue?

Why should it say "Invalid literal" instead of "Invalid numeric literal" for those three characters? How is that better?

Does it make more sense for those to trigger Invalid numeric literal? Personally those don't seem like numeric characters (as compared to, say, - which if alone, could be an invalid negative number). I was just implementing the suggestions above. I'm happy to revert if this doesn't make sense.

"Invalid numeric literal" means that jq tried to parse a JSON number and it was not a number. so the error makes sense, yes.
"Invalid literal"? what does that even mean? :)

Oh I see. I guess if we got to this point, then we've tried parsing it as a string, boolean, and null. So all that's left is parsing it as a number, hence that message. Maybe the better error message would be something like Invalid character '&' encountered while parsing number? I'm happy to revert this change for now though if everything else looks good.

I guess that ppl search and find #501 (it's the most visited issue) because we mention numeric in many error cases and it's confusing

Had a go at compring jq to JSON.parse:

$ for i in ident truea falsea nana aa1 1aa + - / "&" "*" '(' "'aa'"; do echo "$i:\t$(jq 2>&1 <<< "$i")"; echo " $(node -p "JSON.parse(\"$i\")" 2>&1 | grep Error | head -n1)"; done ident: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token 'i', "ident" is not valid JSON truea: jq: parse error: Invalid literal at line 2, column 0 SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5) falsea: jq: parse error: Invalid literal at line 2, column 0 SyntaxError: Unexpected non-whitespace character after JSON at position 5 (line 1 column 6) nana: jq: parse error: Invalid literal at line 2, column 0 SyntaxError: Unexpected token 'a', "nana" is not valid JSON aa1: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token 'a', "aa1" is not valid JSON 1aa: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected non-whitespace character after JSON at position 1 (line 1 column 2) +: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token '+', "+" is not valid JSON -: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: No number after minus sign in JSON at position 1 (line 1 column 2) /: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token '/', "/" is not valid JSON &: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token '&', "&" is not valid JSON *: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token '*', "*" is not valid JSON (: jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token '(', "(" is not valid JSON 'aa': jq: parse error: Invalid numeric literal at line 2, column 0 SyntaxError: Unexpected token ''', "'aa'" is not valid JSON

For me it's ok changing the default error to Invalid character X instead of Invalid numeric literal. It is not ok however to make &, / and * randomly error with "Invalid literal"; while most other character would error e.g. boo, =, ! return Invalid literal.
Anyhow, I only just noticed that Invalid literal is an error that jq already uses instead of Invalid numeric literal for f, n, t because they are short circuited, so that should be changed to be more consistent I guess.
I am fine with the ' patch as a temporary fix for now, but speecial casing &, /, * feels too random.

Should we just change the default error to Invalid character instead, and not even special case ' since Invalid character "'" at is probably good enough?

I'd be happy to implement a general Invalid character "X" kind of error message, but I may need someone to point me in the right direction. My understanding is that the check_literal function returns a const char* meaning we wouldn't be able to get that dynamic error message here without changing a bunch of code. If this change is beyond the scope of the issue, I can just revert back to the simple ' special case.

foo.jq

This reverts commit d3295f1.

itchyny · 2024-09-10T10:51:22Z

This change looks good to merge. Any objections?

itchyny · 2024-09-11T11:12:38Z

tests/jq.test

@@ -2103,6 +2103,9 @@ try ["ok", setpath([1]; 1)] catch ["ko", .]
 {"hi":"hello"}
 ["ko","Cannot index object with number"]

+try fromjson catch .
+"{'a': 123}"
+"Invalid string literal; expected \", but got ' at line 1, column 5 (while parsing '{'a': 123}')"


Why column 5, not column 2?

itchyny

Thank you!

Return error for string literal beginning with single quote

90546b0

In order for a string literal to be valid in JSON, the value must be surrounded by double quotes.

wader reviewed Dec 5, 2023

View reviewed changes

emanuele6 added the feature request label Dec 11, 2023

emanuele6 added this to the 1.8 release milestone Dec 11, 2023

mattmeyers added 3 commits December 12, 2023 08:56

Update invalid string literal message and add test

e57701e

Add checks for additional invalid literals

d3295f1

Merge remote-tracking branch 'upstream/master' into fix/501

35ae8ba

emanuele6 reviewed Dec 12, 2023

View reviewed changes

foo.jq Outdated Show resolved Hide resolved

mattmeyers added 3 commits December 12, 2023 09:19

Remove test file

4b60de1

Revert "Add checks for additional invalid literals"

a8adc84

This reverts commit d3295f1.

Merge remote-tracking branch 'upstream/master' into fix/501

52ee3db

wader approved these changes Sep 10, 2024

View reviewed changes

itchyny reviewed Sep 11, 2024

View reviewed changes

itchyny approved these changes Oct 5, 2024

View reviewed changes

itchyny merged commit 562d5c5 into jqlang:master Oct 5, 2024
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return error for string literal beginning with single quote #2964

Return error for string literal beginning with single quote #2964

mattmeyers commented Dec 5, 2023

wader Dec 5, 2023

emanuele6 Dec 11, 2023

emanuele6 Dec 11, 2023 •

edited

Loading

wader commented Dec 5, 2023

mattmeyers commented Dec 5, 2023

wader commented Dec 5, 2023 •

edited

Loading

itchyny commented Dec 5, 2023

wader commented Dec 5, 2023

emanuele6 commented Dec 11, 2023

mattmeyers commented Dec 12, 2023

emanuele6 Dec 12, 2023

mattmeyers Dec 12, 2023

emanuele6 Dec 12, 2023

mattmeyers Dec 12, 2023

emanuele6 Dec 12, 2023

mattmeyers Dec 12, 2023

wader Dec 12, 2023

emanuele6 Dec 12, 2023

mattmeyers Dec 14, 2023

itchyny commented Sep 10, 2024

itchyny Sep 11, 2024

itchyny left a comment

Return error for string literal beginning with single quote #2964

Return error for string literal beginning with single quote #2964

Conversation

mattmeyers commented Dec 5, 2023

Summary

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emanuele6 Dec 11, 2023 • edited Loading

Choose a reason for hiding this comment

wader commented Dec 5, 2023

mattmeyers commented Dec 5, 2023

wader commented Dec 5, 2023 • edited Loading

itchyny commented Dec 5, 2023

wader commented Dec 5, 2023

emanuele6 commented Dec 11, 2023

mattmeyers commented Dec 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itchyny commented Sep 10, 2024

Choose a reason for hiding this comment

itchyny left a comment

Choose a reason for hiding this comment

emanuele6 Dec 11, 2023 •

edited

Loading

wader commented Dec 5, 2023 •

edited

Loading