JSON Structs with bash

Welcome back curious reader, my name is Alex M. Schapelle, AKA Silent-Mobius. On this blessed day by Omnisaiah, we shell dedicate time focusing on JSON, and how to use it in context of shell programming and bash scripting.

Parsing JSON with Bash Script

When working with APIs or data from external sources, JSON (JavaScript Object Notation) is one of the most common formats for transmitting structured data. While languages like Python or JavaScript provide powerful libraries to work with JSON, Bash scripting can be used to parse JSON data as well. However, Bash doesn’t have native JSON support, so parsing JSON in Bash typically requires external tools like jq or grep combined with other utilities.

This article will guide you through the process of parsing JSON in Bash scripts using jq, a lightweight and flexible command-line JSON processor.

1. What is jq?

jq is a command-line tool that is used to process and manipulate JSON data. It allows you to extract and transform JSON content into a readable format for further use in scripts. Unlike other tools like grep or awk, jq is specifically designed for working with JSON data, making it the best choice for parsing JSON in Bash.

To get started with jq, you'll first need to install it. On most Linux distributions, jq can be installed easily with the following commands:

• On Ubuntu/Debian:

  sudo apt-get install jq

• On CentOS/RHEL:

  sudo yum install jq

2. Basic JSON Parsing with jq

Let’s start with a simple example. Suppose we have the following JSON object stored in a file named data.json:

{
  "name": "John",
  "age": 30,
  "city": "New York"
}

We can parse this JSON data using jq to retrieve individual fields.

Extracting a Single Field

To extract the name from the JSON file, you would use the following command:

jq '.name' data.json

Output:

"John"

Extracting Multiple Fields

If you want to extract multiple fields, you can combine them in a single command like so:

jq '.name, .age' data.json

Output:

"John"
30

You can also group the values into an object:

jq '{name, age}' data.json

Output:

{
  "name": "John",
  "age": 30
}

Parsing Nested JSON

JSON objects can be nested, and jq can handle this efficiently. For example, let’s consider the following JSON structure:

{
  "user": {
    "name": "John",
    "age": 30
  },
  "city": "New York"
}

To extract the name field inside the user object:

jq '.user.name' data.json

Output:

"John"

3. Filtering JSON Data

You can also filter the JSON data by applying conditions using jq. For instance, suppose you have an array of users like this:

[
  {
    "name": "John",
    "age": 30,
    "city": "New York"
  },
  {
    "name": "Jane",
    "age": 25,
    "city": "Los Angeles"
  },
  {
    "name": "Doe",
    "age": 35,
    "city": "Chicago"
  }
]

Filtering Based on a Condition

Let’s say you want to filter out users whose age is greater than 30. You can do that with jq as follows:

jq '.[] | select(.age > 30)' data.json

Output:

{
  "name": "Doe",
  "age": 35,
  "city": "Chicago"
}

This command filters the array and returns only the object where the age field is greater than 30.

4. Working with JSON from an API

If you’re working with a JSON response from a REST API, you can pipe the output of curl or wget directly into jq. For example, using curl to fetch JSON from an API and then parsing it:

curl -s https://api.example.com/users | jq '.data[] | .name'

This command fetches JSON from https://api.example.com/users, parses it with jq, and prints the name of each user in the response.

5. Modifying JSON Data with jq

jq allows not only for reading but also modifying JSON data. For example, if you want to modify the age of a user:

jq '.user.age = 31' data.json

This will change the age field to 31 and output the modified JSON. You can also use jq to update or add new fields.

Adding a New Field

jq '. + {"country": "USA"}' data.json

This adds a country field with the value "USA" to the JSON.

6. Handling JSON Arrays

Working with JSON arrays is straightforward with jq. Let’s say you have the following JSON array:

[1, 2, 3, 4, 5]

To extract all the values, you would simply use:

jq '.[]' data.json

Output:

1
2
3
4
5

You can also use the map function to modify or process each element of an array:

jq 'map(. * 2)' data.json

Output:

[2, 4, 6, 8, 10]

7. Combining Multiple Operations

You can combine multiple operations in jq to make your script more efficient. For example, you could filter the data, then transform it in one line:

jq '.[] | select(.age > 30) | .name' data.json

This command filters users over the age of 30 and then extracts their names.

Conclusion

While Bash isn't natively equipped to parse JSON, jq makes it extremely easy to handle JSON data in Bash scripts. Whether you're extracting values, filtering data, or modifying JSON, jq provides a rich set of tools to work with JSON in a way that's clean, efficient, and flexible.

By mastering jq, you can unlock a powerful tool for working with APIs, manipulating configuration files, or processing any JSON-formatted data directly within your Bash scripts.