Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add JSON-Format output parameter #30

Open
thorstenspille opened this issue Feb 19, 2021 · 3 comments
Open

Add JSON-Format output parameter #30

thorstenspille opened this issue Feb 19, 2021 · 3 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@thorstenspille
Copy link

Add a parameter to print output in json format.
@s-m-e s-m-e added enhancement New feature or request good first issue Good for newcomers labels Feb 20, 2021
@s-m-e
Copy link
Member

s-m-e commented Feb 22, 2021

Narrowing down the design - based on my demo screenshot:

One complete, closed JSON-string per line of output - and one line per item, i.e. transaction, I presume is best (?).

I can do both, JSON output for "future" transaction, i.e. before a user may (optionally) confirm the transactions with a "yes" - and JSON output for running transactions.

  1. Is an (optional) explicit confirmation by a user (or another bash script) a relevant feature when using JSON output?
  2. If yes, should the question for the "user" also be provided in some machine-readable form, i.e. JSON?
  3. Once the transactions are executed, I can emit one JSON string before the transaction (i.e. what is about to happen) and one after (i.e. what actually happened - some status). Does this make sense?

@thorstenspille
Copy link
Author

thorstenspille commented Feb 23, 2021 •
edited
Loading

Hi @s-m-e,
primarily JSON output is relevant for automation and monitoring, so I think the best solution would be printing one JSON dictionary including all relevant information after the execution has finished.

My idea is:

  • print the configuration, to be able to see default values undefined in config.yaml or maybe a separate subcommand config to dump the full configuration
  • merge query and execution into one dict per transaction
  • add performance metrics to to every transaction and to summary for overall process (added some random values in the example)
  • execution with --json parameter should be without any need for user interaction
  • for testing you could add a --dry-run parameter, which only queries the data

Here's an example output for snapshot creation:
abgleich snap config.yaml --json

{
    "config": {
        "source": {
            "zpool": "data_ssd",
            "prefix": "ernst",
            "host": "localhost",
            "user": ""
        },
        "target": {
            "zpool": "data",
            "prefix": "BACKUP_E9/ernst",
            "host": "anonymous",
            "user": "backup"
        },
        "include_root": true,
        "keep_snapshots": 2,
        "always_changed": false,
        "written_threshold": 1048576,
        "check_diff": true,
        "suffix": "_backup",
        "digits": 2,
        "ignore": [
            "some/ignored/subset",
            "some/ignored/folder"
        ],
        "ssh": {
            "compression": false,
            "cipher": "[email protected]"
        }
    },
    "transactions": [
        {
            "type": "snapshot",
            "zvol_type": "subvol",
            "dataset_subname": "",
            "snapshot_name": "2020071408_backup",
            "bytes_written": 2306867,
            "checked_diff": false,
            "create_snapshot": true,
            "status": "OK",
            "command": "zfs snapshot data_ssd/ernst@2020011408_backup",
            "timestamp_created": 1863713480,
            "query_duration_seconds": 0.203,
            "exec_duration_seconds": 0.020,
            "error_message": ""
        },
        {
            "type": "snapshot",
            "zvol_type": "dataset",
            "dataset_subname": "FIREFOX",
            "snapshot_name": "2020071410_backup",
            "bytes_written": 937.984,
            "checked_diff": true,
            "create_snapshot": true,
            "status": "OK",
            "command": "zfs snapshot data_ssd/ernst/FIREFOX@2020011410_backup",
            "timestamp_created": 1863678200,
            "query_duration_seconds": 0.203,
            "exec_duration_seconds": 0.009,
            "error_message": ""
        },
        {
            "type": "snapshot",
            "zvol_type": "dataset",
            "dataset_subname": "PROJEKTE/prj.ZFS",
            "snapshot_name": "2020071407_backup",
            "bytes_written": 4613734,
            "checked_diff": false,
            "create_snapshot": true,
            "status": "OK",
            "command": "zfs snapshot data_ssd/ernst/PROJEKTE/prj.ZFS@2020011407_backup",
            "timestamp_created": 1863678900,
            "query_duration_seconds": 0.203,
            "exec_duration_seconds": 0.036,
            "error_message": ""
        },
        {
            "type": "snapshot",
            "zvol_type": "dataset",
            "dataset_subname": "THUNDERBIRD",
            "snapshot_name": "2020071409_backup",
            "bytes_written": 5242880,
            "checked_diff": false,
            "create_snapshot": true,
            "status": "OK",
            "command": "zfs snapshot data_ssd/ernst/THUNDERBIRD@2020011409_backup",
            "timestamp_created": 1863677600,
            "query_duration_seconds": 0.203,
            "exec_duration_seconds": 0.039,
            "error_message": ""
        }
    ],
    "summary": {
        "num_zvols": 4,
        "snapshots_skipped": 0,
        "snapshots_created": 4,
        "snapshots_failed": [],
        "overall_runtime_seconds": 1.325,
        "overall_queries_seconds": 0.812,
        "overall_creation_seconds": 0.513,
        "success": true
    }
}

@s-m-e
Copy link
Member

s-m-e commented Mar 14, 2021

Ok, then there is just one "drawback" to your idea. Should the command fail or be interrupted while running, two scenarios might occur: The output is not valid JSON because part of it is missing or there is no output at all because the output is being serialized into JSON after the completion of all transactions. Either way, I'd prefer to output multiple independent blocks of JSON, e.g. one for the configuration and one per transaction - separated by new-line characters. The summary could be a another JSON block at the end. For easier parsing, I'd suggest that all JSON blocks use a common pattern / schema, indicating what type of message they contain (configuration, transaction or summary). Does this idea make sense?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@s-m-e @thorstenspille