I've been using jq for years. I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT. JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful
Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!
If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.
For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo
This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.
I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful. nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell. Definitely a good tool for data querying/manipulation though.
I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter). The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.
After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.
Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.
I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.
Prime pushed me get more knowledgable at my cli one of the 1st things after neovim was jq, surprise that you did not know this earlier. Im actually using to pipe from a curl and into a xargs to perform queries in a row
I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.
Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.
That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.
Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.
In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend
@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection. Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.
Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing
I've been building a JS library similar to this CLI for a while to add to my repo as first project. Supports most things SQL does: Fields, Expressions, Aggregates, DomainAggregates, filtering, sorting. JOINS not yet Not sure if its worth finishing now as people might see it as reinventing the wheel. Should I continue working on it ?
okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious there's jid, which allows interactive json filtering than there's jiq, which also allows interactive json filtering, but uses jq as its backend then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql
jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!
THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!
You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).
@theprimetime could you please describe what the hell were you doing before finding this out? were you like creating a new js app to filter things every time you needed to walk through data? or were you grepping and hopping?
I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too
I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is. I actually started using powershell on mac because its support for objects and converting from json is so much better.
jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...
This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!
been using nushell since early january for admin work k8s and its the best experience ive had since Im also a data engineer working with python, pandas/polars, sql
I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day
what is your setup on the terminal? i see at the bar below `1:nvim- 2:zsh*` some fancy stuff. can you scroll up your history with J, K buttons? ctrl + U ? what other fancy stuff you do with your terminal and how to set those up? make a tutorial please like you do for vim
its weird how his file is not an array of objects, just a line by line thingie. for use with actual json files, you have to use jq's map(), like jq 'map({name: .name})'
You can even pipe a curl to a rest api into jq that returns json.. awesome! Great for testing an api from the terminal.. thanks for sharing! If I could add jq to a bash script that’s part of a cron, you could do some interesting data extraction and save it to a new json file to be evaluated or transferred into a folder that an application actually can read or display through a web interface.
Every time I get overly excited about some tool, I look at my wife with this look on my face that says 'I really need to tell you about how cool this tool is I found' and she Immediately knows I'm about to nerd out... 😂
I've used jq occasionally for years. Unfortunately, the syntax is not intuitive and I always spend like an hour trying to remember it's idiosyncracies. I hope someone does a rewrite with better porcelain at some point.
I've been looking into jq at work, we have massive json files that we need to have search queries run on (35MB shite), problem is I need to wrap jq since I am using java :(
Hey Prime, how come you use the web interface of ChatGPT? It takes no time to write a CLI tool that uses the OpenAI API, and it's a huge boost for productivity. I wrote mine in python, and I use it all the time in Emacs. By specifying a custom system prompt, my GPT writes directly in org-mode format, allowing me to do things like executing code blocks in a conversation buffer. I can't overstate how powerful this is, but maybe NeoVim just can't handle such power?
Honestly very surprised to see that so many devs don't know jq. If you've tried to work with any sort of web API in a shell script, jq is like the tool they teach in 101.
And he said he doesn't like higher order functions... WELL LOOK WHERE WE ARE, its basically just a bunch of HOFs and HOF combinators. Reminds me a lot of optics from haskell as well.
Stamps:
1:35 - Reading a file
1:39 - Compact (-c)
1:45 - Specific field (.)
1:50 - Raw (-r)
2:21 - Showing keys (keys)
2:31 - New object
3:00 - Select not null fields
3:31 - has()
3:47 - Math
4:59 - Contains
I've been using jq for years.
I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT.
JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful
I thought only i knew about this XD
two other text manipulating commands that are useful are 'awk' and 'sed'
I use yq (the YAML equivalent) in my deploy jobs for a lightweight way of updating argocd applications in my IAC repos. Works great.
Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!
Look up one more character and search for jqp. Soooo nice. :)
If only I'd have known this a year ago. And it's been there on my computer for who knows how long! Very useful - thanks for sharing.
I used this to deal with a 9GB json file I was handed.
Split it into 90MB parts before my database ingest and it saved me so much effort.
Do you want the Earth to collapse into a black hole under the weight of 9 gigbyte JSON? Are you nut?
This is actually pretty sick, I wish I would have known about this literally 2 hours ago where I had a big json file I wanted to get some info from
for me was las week... fumbled like a mf...
If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.
For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo
This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.
I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful.
nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell.
Definitely a good tool for data querying/manipulation though.
This sounds like my next tool time moment. 👍
Update: Awesome it's in the Termux repos 😸
Was about to comment the same. I haven't looked into jq much since I can do the same and more with Nushell without being limited to just json
I am waiting for nushell to mature and stabilize, its awesome but changes too often and scripting just recently became a thing
Your wife's idea of a good tool, is different
nice nice nice nice ligma nice nice
what's ligma
ligma balls @@darukutsu
@@wowvain2989 GOTTEM
@@wowvain2989 lmao
I knew and I used jq for quite a long time, but instead I didn't knew about I can pipe a vim buffer to different program xD
My god. This left me speechless, I know NOTHING about CLI tools! Thank you Prime for showing us this!
Everything useful was first a CLI tool. Nothing you can’t do.
@@TheStickofWar It always comes back to command line and config files.
I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter).
The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.
It's great to see jq to be maintained again. Great tool that deserves to be maintained.
When I use jq, sometimes it's really nice to use the crate "fltn" to pipe json to fzf, and then maybe even pipe my interactive selection back into jq.
After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.
Still learning and only understand 15% of what you talk about but you’re still entertaining and motivate me to grind, that’s talent.
Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.
Wait what, this is the first time your hearing about this tool, just wait theres `yq` for yml
The new golang yq is the bomb diggety. Old python yq was whack
There's also xq, but I prefer xmllint --xpath as it's pre-installed on most systems
@@replikvltyoutube3727 It's called xslt, and it's been around for 25 years. xsltproc is the cli tool.
The yq I have installed includes xq and jq. It makes shell scripting suck just a little less.
YQ is a wrapper over JQ 😂
I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.
Prime pushed me get more knowledgable at my cli
one of the 1st things after neovim was jq, surprise that you did not know this earlier.
Im actually using to pipe from a curl and into a xargs to perform queries in a row
I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.
jq is awesome. Yet there's more power if you dive into the nushell. It's like a shell with jq and rust-like typed scripting built-in
Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.
I have to use jq so infrequently, I always forget the query syntax and have to look it up again.
That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.
“what could this guy possibly so hyped about-oh yeah jq is really cool and i should remember it more often”
jq is life, it makes the AWS cli tool actually usable
Hahah yeah been there before , the aws cli is a bit shit
Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.
In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend
I use powershell to do this. Can not only read/select/filter json but also xml and csv
That’s cool. How does this work using PowerShell?
@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection.
Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.
@@tom_marsdenpiping to convertfrom-json is the way. It just spits out a native powershell object/hashtable.
Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing
I've been building a JS library similar to this CLI for a while to add to my repo as first project.
Supports most things SQL does: Fields, Expressions, Aggregates, DomainAggregates, filtering, sorting. JOINS not yet
Not sure if its worth finishing now as people might see it as reinventing the wheel.
Should I continue working on it ?
nothing wrong with reinventing the wheel if you learn while doing it
0:55 that’s a Quagmire pause during flight announcement.
Actual tool is crazy.
okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious
there's jid, which allows interactive json filtering
than there's jiq, which also allows interactive json filtering, but uses jq as its backend
then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong
we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend
oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql
That's the kind of content I started to love you back in the days. Awesome
Im searching for that CLI thing all my life! Thank you!
i made something similar for web scraping at one point but other than that i have no idea what you need this for so i want to ask what you do with it
jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!
it goes very well with jless!
This is why I love the comments section. Now I have a new tool to pair with jq and jet.
9:20 "Tech humor: jq json jokes"
THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!
It always warms my heart seeing devs learn more about the myriad of cli tooling available. Just wait till you see parallel
It's been on some vid but just not featured as the main topic. He seemed to appreciate it.
You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).
Man finally learns a useful programming language and is blown away.
4:54, you dont have to attack me like that . That personal bruh...
4:43 - 4:58 Poor arch users. Prime went too harsh on them. 😅😅
Been using this bad boy for a week and oh my God I've been missing out badly. More quick tutorials like these and how they work for you!!!
I've been using nvim wrong too that !jq there opened my eyes
hmmm also noticed @5:40 a drinking jar...shouting out to Cewpins? :)
@theprimetime could you please describe what the hell were you doing before finding this out?
were you like creating a new js app to filter things every time you needed to walk through data?
or were you grepping and hopping?
I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too
Nice one. You could consider making short videos for such cli tools. It's great motivation for using them
I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is.
I actually started using powershell on mac because its support for objects and converting from json is so much better.
jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...
btw, there's also yq if you happen to have a whole bunch of yaml that you want to parse through with the same syntax.
I use jq too infrequently to remember the syntax despite using it for years.
i understand and agree with your excitement it's a God sent jewel to make my bash scripts less depressing
Yes, but have you tried visidata and nushell?
I just always use jq for parsing out data from AWS secrets manager in a bash script... Used it for years but nice to see it can do way more.
this video is going to change my life in more ways than prime can imagine.
This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!
There's nothing more satisfying than a well crafted jq
Well, congratulations Primeagen for being of one of the lucky 10.000s recently
Might have to check that out. Usually I just paste json into a browser console and use javascript. Handy to see it in a text editor though.
Mixing it with fzf makes even more awesome
Cool, using it often, btw did you know there's also a yq (same thing for yaml)
Did he try nushell? Basically does this to literally everything
been using nushell since early january for admin work k8s and its the best experience ive had since Im also a data engineer working with python, pandas/polars, sql
I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day
That opening was like poetry. jq is a vital tool.
For those who don't know, then there are substitutes for yaml and xml also: yq and xq
what is your setup on the terminal? i see at the bar below `1:nvim- 2:zsh*` some fancy stuff.
can you scroll up your history with J, K buttons? ctrl + U ?
what other fancy stuff you do with your terminal and how to set those up? make a tutorial please like you do for vim
using cli to visualise data in 2024... still... I didn't think we'll be here 10 years ago.
its weird how his file is not an array of objects, just a line by line thingie. for use with actual json files, you have to use jq's map(), like jq 'map({name: .name})'
have been using this for years. handy tool
What about NuShell with the nu though? What are your thoughts on that?
i use yq/jq to flatten out a file into a properties like format and pipe it through fzf for easy querys on the fly
You can even pipe a curl to a rest api into jq that returns json.. awesome! Great for testing an api from the terminal.. thanks for sharing! If I could add jq to a bash script that’s part of a cron, you could do some interesting data extraction and save it to a new json file to be evaluated or transferred into a folder that an application actually can read or display through a web interface.
Every time I get overly excited about some tool, I look at my wife with this look on my face that says 'I really need to tell you about how cool this tool is I found' and she Immediately knows I'm about to nerd out... 😂
I've used jq occasionally for years. Unfortunately, the syntax is not intuitive and I always spend like an hour trying to remember it's idiosyncracies. I hope someone does a rewrite with better porcelain at some point.
Still better than jsonpath /spit
Way better than xpath /spit
@@teejaded
If you're already working with XML, xpath really doesn't make it any worse.
My jq-fu is so weak - all these things that you could do with it makes me feel like a toddler trying to fit the square block through the round hole.
I use jq and yq every so often when I need it. However, because I use it infrequently, I need to go back and learn it again and again
I wish I had it yesterday when I had to remove some playlists from a massive Spotify backup json file.
Wait until Primeagen discovers "jc", a dark brother of "jq", then it completely changes the interaction with the output of any core linux/bsd utility.
I've been looking into jq at work, we have massive json files that we need to have search queries run on (35MB shite), problem is I need to wrap jq since I am using java :(
lol fucker just install it and ask chatgpt to teach u sbout it over the weekend
Hey Prime, how come you use the web interface of ChatGPT? It takes no time to write a CLI tool that uses the OpenAI API, and it's a huge boost for productivity. I wrote mine in python, and I use it all the time in Emacs. By specifying a custom system prompt, my GPT writes directly in org-mode format, allowing me to do things like executing code blocks in a conversation buffer. I can't overstate how powerful this is, but maybe NeoVim just can't handle such power?
The GPT-4 api is too expensive for me so I settled on the deepseek coder model with function execution
I'm about to blow your mind. There is a different tool called yq. It does the same thing, but YAML. And JSON is a subset of YAML. It does both.
There is also xq for XAML.
i thought it was just a formatrer 😶
so this tool is same as awk meaning it can do more than just print fields, will practise it more
I knew jq for a longtime just never played around with it this much
JQ? Whoa buddy, I thought this was a programming channel.
lmao
Lol
So this is what it actually feels to see Primeagen talk about something that you know what he's talking about?
This is a cool tool, I didn't know it existed. I usually just write some basic python code to parse json files. But this seems much faster and easier.
Never knew that Jquery had such an awesome command line interface!
jq+curl has supplanted python for exploring rest apis for me. Also nice to pipe certain fields to sort to get some basic frequency stats
knew about jq...but always struggled with the syntax. thx homie
You just pitched JQ to your wifey?! 😂 A legend ⭐
This video didnt just teach me more jq than I already have been using, you taught me more about (neo)vim as well
LEGEND
I love how you've got "jQuery jokes" on your chatgpt history haha
Honestly very surprised to see that so many devs don't know jq. If you've tried to work with any sort of web API in a shell script, jq is like the tool they teach in 101.
And he said he doesn't like higher order functions... WELL LOOK WHERE WE ARE, its basically just a bunch of HOFs and HOF combinators. Reminds me a lot of optics from haskell as well.
There's also zq, or "zed" as it is called.. faq is also apparently more "fllexible" but ive never tried it
what an easily googleable project name
This got me thinking about piping files in vim through nu-shell. Sounds nice.
That is handy, but right now I personally prioritize time into learning ksql to analyse data within kafka-topics.
i've been using also yq that's the equivalent for yaml, as a k8s heavy user is the best never happened to me in terms of file debuging