Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

need command line options to print path and key values of JSON file #3097

Open
eostermueller opened this issue Apr 28, 2024 · 7 comments
Open

Comments

@eostermueller
Copy link

eostermueller commented Apr 28, 2024

Describe the Feature Request
Need a single command line parameter to display all jsonpaths and accompanying values for all values in a json structure.

Consider this sample json:
reservations.json:

{
  "Reservations": [
    {
      "Groups": [],
      "Instances": [
        {
          "ImageId": "ami-a",
          "InstanceId": "i-a",
          "InstanceType": "t2.micro",
          "KeyName": "ubuntu"
        }
      ]
    }
  ]
}

Existing approach:
Here is an existing approach (taken from here and not my own work) that displays the jsonpaths/values. Entering the 170-ish char expression is way too verbose/unwieldly, but it achieves the desired results.

cat reservations.json | jq -r 'def kv_to_str($k;$v): "\($k): \($v)"; paths(scalars|true) as $p | kv_to_str("." + ($p|map([.]|tojson)|join("")); getpath($p)|tojson)'
.["Reservations"][0]["Instances"][0]["ImageId"]: "ami-a"
.["Reservations"][0]["Instances"][0]["InstanceId"]: "i-a"
.["Reservations"][0]["Instances"][0]["InstanceType"]: "t2.micro"
.["Reservations"][0]["Instances"][0]["KeyName"]: "ubuntu"

Desired approach:
Instead, we need a single jq command line parameter to display the same results, like this if a -p parameter (for paths) were used:

cat reservations.json | jq -p
.["Reservations"][0]["Instances"][0]["ImageId"]: "ami-a"
.["Reservations"][0]["Instances"][0]["InstanceId"]: "i-a"
.["Reservations"][0]["Instances"][0]["InstanceType"]: "t2.micro"
.["Reservations"][0]["Instances"][0]["KeyName"]: "ubuntu"

Formatting the Output

The jsonpath format outputted must be presentable as input. The example above outputted a jsonpath which is reused here successfully as input.

jq '.["Reservations"][0]["Instances"][0]["InstanceType"]' reservations.json
"t2.micro"

With the above solution, the output for larger json or json with more complex object structure is admittedly not picturesque.... the pre-built jsonpaths are so valuable, the below-average formatting doesn't even matter.
Perhaps consider one parameter with this poor formatting and a different parameter for better formatting.

Additionally, IDE and other tools could use the pre-built json paths, and this jq feature could provide those paths.
jq could create a new IDE-friendly json structure that contains each pre-built json path along with the line/character number of it's value from the original json. Just to show that people would use this kind of thing, here is on existing IDE plugin that generates a jsonpath when the user selects a json value.

Rationale

Pre-built json paths expedite work -- look at the proliferation of online tools that pre-build jsonpaths: https://jsonpathfinder.com/ and https://www.site24x7.com/tools/jsonpath-finder-validator.html.

I would like this functionality available while taking a kubernetes exam. jq is available in the test environment, but I'll have to memorize the 170 characters and hand-key it into my .bashrc test environment....what a pain.

This would be helpful for everyone who does work with kubernetes and other json-heavy administration.

Environment (please complete the following information):

  • OS and Version: linux / RedHat 9.2
  • jq version 1.6

Additional context

Proper formatting for the jq expression to make it easier on the eyes:

def kv_to_str($k; $v): "\($k): \($v)";

paths(scalars | true) as $p |
    kv_to_str(
        "." + ($p | map([.] | tojson) | join(""));
        getpath($p) | tojson
    )

Just to show that there is interest in this kind of thing, here is a similar question with it's own answers.

@wader
Copy link
Member

wader commented Apr 28, 2024

Hi, interesting idea. I wonder if it could be modelled as another input and output format (#467)? thinking this basically is the gron "format"? example:

$ jq -n '{a:"a",b:[1]}' | gron
json = {};
json.a = "a";
json.b = [];
json.b[0] = 1;

$ jq -n '{a:"a",b:[1]}' | gron | gron -u
{
  "a": "a",
  "b": [
     1
  ]
}

I think there also has been some discussion, that i can't find now, about adding some helpers for dealing paths. That would at least simplify doing things like this a bit, for example fq has path_to_expr and expr_to_path:

$ fq -cn '".a[0]" | expr_to_path | ., path_to_expr'
["a",0]
".a[0]"

@eostermueller
Copy link
Author

@wader, didn't know about gron, thanks for sharing.
I was thinking this would be much easier. When -p is present, the -r would be implied and the expression mentioned above would be used. Also, I just updated the issue to state that the output jsonpath format should be presentable as input.

@wader
Copy link
Member

wader commented Apr 29, 2024

Agree on output should be in input format, maybe should even be <path> = <value> to be valid jq? not sure what i think about -p yet, have to think about it. It would also be great with input from other jq maintainers on this but it might take a while.

@pkoppstein
Copy link
Contributor

@wader wrote:

It would also be great with input from other jq maintainers

The jq authors and maintainers seem to be in pretty close agreement that new command-line options are strenuously to be avoided, especially when a straightforward jq program can fill the bill.

In this particular case, the existing --stream option does seem to meet the stated functional requirements. In fact, the JSON stream produced by --stream can be used programmatically even more easily that the proposed output, the point being that the output interfaces nicely with jq's setpath.

Please also note even if the output produced by --stream -c is itself deemed unsatisfactory, it can nevertheless
very easily and succinctly be tweaked to achieve variants such as the one that has been proposed.

@wader
Copy link
Member

wader commented Apr 30, 2024

@wader wrote:

It would also be great with input from other jq maintainers

The jq authors and maintainers seem to be in pretty close agreement that new command-line options are strenuously to be avoided, especially when a straightforward jq program can fill the bill.

Agreed

In this particular case, the existing --stream option does seem to meet the stated functional requirements. In fact, the JSON stream produced by --stream can be used programmatically even more easily that the proposed output, the point being that the output interfaces nicely with jq's setpath.

Please also note even if the output produced by --stream -c is itself deemed unsatisfactory, it can nevertheless very easily and succinctly be tweaked to achieve variants such as the one that has been proposed.

Good point, and it's way more readable then i expected 👍

$ pbpaste | jq --stream -c 'select(length==2)'
[["Reservations",0,"Groups"],[]]
[["Reservations",0,"Instances",0,"ImageId"],"ami-a"]
[["Reservations",0,"Instances",0,"InstanceId"],"i-a"]
[["Reservations",0,"Instances",0,"InstanceType"],"t2.micro"]
[["Reservations",0,"Instances",0,"KeyName"],"ubuntu"]

@eostermueller
Copy link
Author

...that's a step in the right direction, thanks for putting that together.

@mountaineerbr
Copy link

I have something that may help in my .bashrc:

#get all paths of a json
jqpath()
{
	 jq -r 'def path2text($value):
		  def tos: if type == "number" then . else tojson end;
		  reduce .[] as $segment ("";  .
		    + ($segment
		       | if type == "string" then "." + . else "[\(.)]" end))
		  + " = \($value | tos)";

		paths(scalars) as $p
		  | getpath($p) as $v
		  | $p | path2text($v)' "$@"
}
jqpath2()
{
	jq -rc 'path(..)|[.[]|tostring]|join("/")' "$@"
	#jq -r '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]' "$@"

}
#https://github.com/stedolan/jq/issues/243
#also see `gron' package to flatten json
#https://www.datafix.com.au/BASHing/2022-03-23.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants