Merge pull request #7 from seamile/dev

change the query language from JsonPath to JmesPath.

Author: seamile

README.md

@@ -1,4 +1,4 @@
-# JSON Formator
+# JSON Formatter
 
 [![Build Status](https://github.com/seamile/jsonfmt/actions/workflows/python-package.yml/badge.svg)](https://github.com/seamile/jsonfmt/actions)
 [![PyPI Version](https://img.shields.io/pypi/v/jsonfmt?color=blue&label=Version&logo=python&logoColor=white)](https://pypi.org/project/jsonfmt/)
@@ -8,7 +8,8 @@
 
 **jsonfmt** is a powerful tool for handling JSON document.
 
-It is similar to [jq](https://github.com/jqlang/jq), but simpler.
+It is as powerful as [jq](https://github.com/jqlang/jq), but simpler.
+
 
 ## Features
 
@@ -20,7 +21,7 @@ It is similar to [jq](https://github.com/jqlang/jq), but simpler.
     - [Show the overview of a large JSON.](#show-the-overview-of-a-large-json)
     - [Copy the result to clipboard.](#copy-the-result-to-clipboard)
 - [3. Minimize the JSON document.](#3-minimize-the-json-document)
-- [4. Pick out parts of a large JSON via JSONPath.](#4-pick-out-parts-of-a-large-json-via-jsonpath)
+- [4. Pick out parts of a large JSON via JmesPath.](#4-pick-out-parts-of-a-large-json-via-jmespath)
 - [5. Convert formats between JSON, TOML and YAML.](#5-convert-formats-between-json-toml-and-yaml)
     - [JSON to TOML and YAML](#json-to-toml-and-yaml)
     - [TOML to JSON and YAML](#toml-to-json-and-yaml)
@@ -58,7 +59,7 @@ $ jsonfmt [options] [files ...]
     - `-i {0-8,t}`: number of spaces for indentation (default: 2)
     - `-o`: show data structure overview
     - `-O`: overwrite the formated text to original file
-    - `-p JSONPATH`: output part of the object via jsonpath
+    - `-p JSONPATH`: output part of the object via jmespath
     - `-s`: sort keys of objects on output
     - `--set 'foo.k1=v1;k2[i]=v2'`: set the keys to values (seperated by `;`)
     - `--pop 'k1;foo.k2;k3[i]'`: pop the specified keys (seperated by `;`)
@@ -213,13 +214,15 @@ $ echo '{
 {"age":21,"items":["pen","phone"],"name":"alex"}
 ```
 
-### 4. Pick out parts of a large JSON via JSONPath.
+### 4. Pick out parts of a large JSON via JmesPath.
 
-**JSONPath** is a way to query the sub-elements of a JSON document.
+Unlike from jq's private solution, `jsonfmt` uses [JmesPath](https://jmespath.org/) as its query language.
 
-It likes the XPath for xml, which can extract part of the content of a given JSON document through a simple syntax.
+Among the many JSON query languages, `JmesPath` is the most popular one ([compared here](https://npmtrends.com/JSONPath-vs-jmespath-vs-jq-vs-json-path-vs-json-query-vs-jsonata-vs-jsonpath-vs-jsonpath-plus-vs-node-jq)).
+It is more general than `jq`, and more intuitive and powerful than `JsonPath`.
 
-JSONPath syntax reference: [goessner.net](https://goessner.net/articles/JsonPath/), [ietf.org](https://datatracker.ietf.org/doc/id/draft-goessner-dispatch-jsonpath-00.html).
+Like the XPath for xml, `JmesPath` can elegantly extract parts of a given JSON document with simple syntax.
+See the tutorial [here](https://jmespath.org/tutorial.html).
 
 Some examples:
 
@@ -231,6 +234,22 @@ Some examples:
 
     *Output:*
 
+    ```json
+    {
+        "calorie": 294.9,
+        "date": "2021-03-02",
+        "name": "eat"
+    }
+    ```
+
+- Filter all items in `actions` with `calorie` > 0.
+
+    ```shell
+    $ jsonfmt -p 'actions[?calorie>`0`]' test/example.json
+    ```
+
+    *Output:*
+
     ```json
     [
         {
@@ -241,22 +260,71 @@ Some examples:
     ]
     ```
 
-- Filters all occurrences of the `name` field in the JSON.
+- Show all the keys and actions' length.
+
+    ```shell
+    $ jsonfmt  -p '{all_keys:keys(@), actions_len:length(actions)}' test/example.json
+    ```
+
+    *Output:*
+
+    ```json
+    {
+        "all_keys": [
+            "actions",
+            "age",
+            "gender",
+            "money",
+            "name"
+    ],
+        "actions_len": 2
+    }
+    ```
+
+- Sort `actions` by `calorie` and redefine a dict.
 
     ```shell
-    $ jsonfmt -p '$..name' test/example.json
+    $ jsonfmt -p 'sort_by(actions, &calorie)[].{name: name, calorie:calorie}' test/example.json
     ```
 
     *Output:*
 
     ```json
     [
-        "Bob",
-        "eat",
-        "sport"
+        {
+            "name": "sport",
+            "calorie": -375
+        },
+        {
+            "name": "eat",
+            "calorie": 294.9
+        }
     ]
     ```
 
+- [More examples](https://jmespath.org/examples.html).
+
+
+**Amazingly**, you can do the same with YAML and TOML using JmesPath, and convert the result format arbitrarily.
+
+```shell
+# read the data from toml file, and convert the result to yaml
+$ jsonfmt  -p '{all_keys:keys(@), actions_len:length(actions)}' test/example.yaml -f toml
+```
+
+*Output:*
+
+```yaml
+all_keys:
+- age
+- gender
+- money
+- name
+- actions
+actions_len: 2
+```
+
+
 ### 5. Convert formats between JSON, TOML and YAML.
 
 The *jsonfmt* can recognize any format of JSON, TOML and YAML from files or `stdin`. Either formats can be converted to the other by specifying the "-f" option.
@@ -372,8 +440,8 @@ $ jsonfmt --set 'skills=["Django","Flask"];money=1000' test/example.json
 #### Pop some items.
 
 ```shell
-# remove the gender field and action[1]
-$ jsonfmt --pop 'gender;action[1]' test/example.json
+# remove the gender field and actions[1]
+$ jsonfmt --pop 'gender;actions[1]' test/example.json
 ```
 
 *Output:*
@@ -396,7 +464,7 @@ $ jsonfmt --pop 'gender;action[1]' test/example.json
 Of course you can use `--set` and `--pop` together.
 
 ```shell
-jsonfmt --set 'skills=["Django","Flask"];money=1000' --pop 'gender;action[1]' test/example.json
+jsonfmt --set 'skills=["Django","Flask"];money=1000' --pop 'gender;actions[1]' test/example.json
 ```
 
 **Note**, however, that the above command will not modify the original JSON file.

jsonfmt.py

@@ -1,25 +1,28 @@
 #!/usr/bin/env python
-'''JSON Format Tool'''
+'''JSON Formatter'''
 
 import json
-import pyperclip
 import re
 import toml
 import yaml
 from argparse import ArgumentParser
 from functools import partial
 from io import TextIOBase
-from jsonpath import jsonpath
 from pydoc import pager
-from pygments import highlight
-from pygments.formatters import TerminalFormatter
-from pygments.lexers import JsonLexer, TOMLLexer, YamlLexer
 from shutil import get_terminal_size
-from sys import stdin, stdout, stderr
+from sys import stdin, stdout, stderr, exit as sys_exit
 from typing import Any, List, IO, Optional, Sequence, Tuple, Union
 from unittest.mock import patch
 
-__version__ = '0.2.4'
+import jmespath
+import pyperclip
+from jmespath.exceptions import JMESPathError
+from jmespath.parser import ParsedResult
+from pygments import highlight
+from pygments.formatters import TerminalFormatter
+from pygments.lexers import JsonLexer, TOMLLexer, YamlLexer
+
+__version__ = '0.2.5'
 
 NUMERIC = re.compile(r'-?\d+$|-?\d+\.\d+$|^-?\d+\.?\d+e-?\d+$')
 DICT_OR_LIST = re.compile(r'^\{.*\}$|^\[.*\]$')
@@ -33,11 +36,7 @@ def print_err(msg: Any):
     print(f'\033[1;91mjsonfmt:\033[0m \033[0;91m{msg}\033[0m', file=stderr)
 
 
-class ParseError(Exception):
-    pass
-
-
-class JsonPathError(Exception):
+class FormatError(Exception):
     pass
 
 
@@ -48,8 +47,8 @@ def is_clipboard_available() -> bool:
         and paste_fn.__class__.__name__ != 'ClipboardUnavailable'
 
 
-def parse_to_pyobj(text: str, jpath: Optional[str]) -> Tuple[Any, str]:
-    '''read json, toml or yaml from IO and then match sub-element by jsonpath'''
+def parse_to_pyobj(text: str, jpath: Optional[ParsedResult]) -> Tuple[Any, str]:
+    '''read json, toml or yaml from IO and then match sub-element by jmespath'''
     # parse json, toml or yaml to python object
     loads_methods = {
         'json': json.loads,
@@ -65,17 +64,13 @@ def parse_to_pyobj(text: str, jpath: Optional[str]) -> Tuple[Any, str]:
         except Exception:
             continue
     else:
-        raise ParseError("no json, toml or yaml found in the text")
+        raise FormatError("no json, toml or yaml found in the text")
 
     if jpath is None:
         return py_obj, fmt
     else:
-        # match sub-elements via jsonpath
-        subelements = jsonpath(py_obj, jpath)
-        if subelements is False:
-            raise JsonPathError('invalid JSONPath or query result is empty')
-        else:
-            return subelements, fmt
+        # match sub-elements via jmespath
+        return jpath.search(py_obj), fmt
 
 
 def forward_by_keys(py_obj: Any, keys: str) -> Tuple[Any, Union[str, int]]:
@@ -120,20 +115,20 @@ def modify_pyobj(py_obj: Any, sets: List[str], pops: List[str]):
 
 
 def get_overview(py_obj: Any):
-    def clip_value(value: Any):
+    def clip(value: Any):
         if isinstance(value, str):
             return '...'
         elif isinstance(value, (list, tuple)):
             return []
         elif isinstance(value, dict):
-            return {k: clip_value(v) for k, v in value.items()}
+            return {k: clip(v) for k, v in value.items()}
         else:
             return value
 
-    if isinstance(py_obj, list):
-        return [clip_value(py_obj[0])]
+    if isinstance(py_obj, list) and len(py_obj) > 1:
+        return [clip(py_obj[0])]
     else:
-        return clip_value(py_obj)
+        return clip(py_obj)
 
 
 def format_to_text(py_obj: Any, fmt: str, *,
@@ -151,7 +146,7 @@ def format_to_text(py_obj: Any, fmt: str, *,
     elif fmt == 'toml':
         if not isinstance(py_obj, dict):
             msg = 'the pyobj must be a Mapping when format to toml'
-            raise ParseError(msg)
+            raise FormatError(msg)
         return toml.dumps(py_obj)
 
     elif fmt == 'yaml':
@@ -160,14 +155,14 @@ def format_to_text(py_obj: Any, fmt: str, *,
                               sort_keys=sort_keys)
 
     else:
-        raise ParseError('Unknow format')
+        raise FormatError('Unknow format')
 
 
 def output(output_fp: IO, text: str, fmt: str, cp2clip: bool):
     # copy the result to clipboard
     if cp2clip:
         pyperclip.copy(text)
-        print_inf('result copied to clipboard.')
+        print_inf('result copied to clipboard')
         return
     elif output_fp.isatty():
         # highlight the text when output to TTY divice
@@ -185,11 +180,11 @@ def output(output_fp: IO, text: str, fmt: str, cp2clip: bool):
         output_fp.truncate()
         output_fp.write(text)
         if output_fp.fileno() > 2:
-            print_inf(f'result written to {output_fp.name}.')
+            print_inf(f'result written to {output_fp.name}')
 
 
-def process(input_fp: IO, jpath: Optional[str], convert_fmt: Optional[str], *,
-            compact: bool, cp2clip: bool, escape: bool, indent: Union[int, str],
+def process(input_fp: IO, jpath: Optional[ParsedResult], convert_fmt: Optional[str],
+            *, compact: bool, cp2clip: bool, escape: bool, indent: Union[int, str],
             overview: bool, overwrite: bool, sort_keys: bool,
             sets: Optional[list], pops: Optional[list]):
     # parse and format
@@ -233,8 +228,8 @@ def parse_cmdline_args(args: Optional[Sequence[str]] = None):
                         help='show data structure overview')
     parser.add_argument('-O', dest='overwrite', action='store_true',
                         help='overwrite the formated text to original file')
-    parser.add_argument('-p', dest='jsonpath', type=str,
-                        help='output part of the object via jsonpath')
+    parser.add_argument('-p', dest='jmespath', type=str,
+                        help='output part of the object via jmespath')
     parser.add_argument('-s', dest='sort_keys', action='store_true',
                         help='sort keys of objects on output')
     parser.add_argument('--set', metavar="'foo.k1=v1;k2[i]=v2'",
@@ -251,6 +246,12 @@ def parse_cmdline_args(args: Optional[Sequence[str]] = None):
 def main():
     args = parse_cmdline_args()
 
+    try:
+        jpath = None if args.jmespath is None else jmespath.compile(args.jmespath)
+    except JMESPathError:
+        print_err(f'invalid JMESPath expression: {args.jmespath}')
+        sys_exit(1)
+
     # check if the clipboard is available
     cp2clip = args.cp2clip and is_clipboard_available()
     if args.cp2clip and not cp2clip:
@@ -273,7 +274,7 @@ def main():
             # read from file
             input_fp = open(file, 'r+') if isinstance(file, str) else file
             process(input_fp,
-                    args.jsonpath,
+                    jpath,
                     args.format,
                     compact=args.compact,
                     cp2clip=cp2clip,
@@ -284,10 +285,14 @@ def main():
                     sort_keys=args.sort_keys,
                     sets=sets,
                     pops=pops)
-        except ParseError as err:
+        except FormatError as err:
+            print_err(err)
+        except JMESPathError as err:
             print_err(err)
         except FileNotFoundError:
-            print_err(f'no such file `{file}`')
+            print_err(f'no such file: {file}')
+        except PermissionError:
+            print_err(f'permission denied: {file}')
         finally:
             input_fp = locals().get('input_fp')
             if isinstance(input_fp, TextIOBase):

pyproject.toml

@@ -9,10 +9,10 @@ requires-python = ">=3.8"
 license = {text = "MIT License"}
 description = "A simple tool for formatting JSON object."
 readme = "README.md"
-keywords = ["json", "formatter", "pretty-print", "highlight", "jsonpath"]
+keywords = ["json", "formatter", "pretty-print", "highlight", "jmespath"]
 authors = [{name = "Seamile", email = "[email protected]"}]
 dependencies = [
-    "jsonpath==0.82",
+    "jmespath >= 1.0.1",
     "Pygments >= 2.13.0",
     "pyperclip >= 1.8.2",
     "pyyaml >= 6.0",

requirements.txt

@@ -1,4 +1,4 @@
-jsonpath==0.82
+jmespath >= 1.0.1
 Pygments >= 2.13.0
 pyperclip >= 1.8.2
 pyyaml >= 6.0