persistent checkpoint storage #10

ccrssaa · 2021-05-03T10:18:28Z

Hi

After unsuccessful attempts to redefine checkpoints using names from qemu-img info --force-share I came up with the idea of "persistent checkpoins storage":

Separate directory with .xml checkpoint config files
Shared among all backups (full and incremental) for a given domain
After each checkpoint creation it's XMLDesc() saved in a virtnbdbackup..xml file
Before each new backup (full or incremental) all checkpoints could be redefined using saved .xml files (if necessary)

This approach allows to use virtnbdbackup in Pacemaker (or similar) environments where domains are transient
Could also be used when moving .qcow2 image (and existing backups) to another host

Not heavily tested yet, but looks working for basic scenarios: full and incremental backup with existing (with checkpoints) and new (without checkpoints yet) domains

changed checkpoint names to virtnbdbackup.0, .1 etc to simplify file sorting

abbbi · 2021-05-03T10:38:54Z

hi,

thanks, i will review at given time. Currently dumping and re-defining the checkpoint information is really the preferred solution if virtual machines are transient or move between hosts. I asked in #virt for that:

12:28 <abi`_> if an checkpoint is created for backup, where does libvirt save that information (besides the bitmap in the qcow file)?
12:29 <abi`_> so, if moving an virtual machine from host a to host b, i guess there needs something happen that host b knows about the checkpoints created during vm operation on host a
12:36 <pkrempa> abi`_: the official solution is to use 'virsh checkpoint-dumpxml' to gather the metadata
12:36 <abi`_> pkrempa: and then redefine the checkpoints on host b?
12:36 <pkrempa> abi`_: and then 'virsh checkpoint-create --redefine'
12:36 <abi`_> ok.
12:37 <pkrempa> abi`_: the non-official answer is that /var/lib/libvirt/qemu/checkpoint/
12:37 <pkrempa> abi`_: obviously that comes with caveats such as libvirtd not looking into the directory after starting up and such

As side note: /var/lib/libvirt/qemu/checkpoint/ is the place where the checkpoint xml is saved. But as the last statement suggests, it only read once upon libvirtd startup, so syncing this folder between hosts wont help :(

ccrssaa · 2021-05-03T11:14:36Z

i will review at given time

Nice, thanks

As side note: /var/lib/libvirt/qemu/checkpoint/ is the place where the checkpoint xml is saved. But as the last statement suggests, it only read once upon libvirtd startup, so syncing this folder between hosts wont help :(

With transient domains /var/lib/libvirt/qemu/checkpoint/domain directory disappears immediately after domain shutdown
And I'd prefer to explicitly define directory to store checkpoint configs (on DRBD storage for .qcow images and domain configs shared by all cluster nodes)

abbbi

And I'd prefer to explicitly define directory to store checkpoint configs (on DRBD storage for .qcow images and domain configs shared by all cluster nodes)

ok, makes sense. Currently its a quite specific setup and the checkpoint dir shouldnt be mandatory at the moment to allow for easier use of the utility.

If the functionality is merged, the save function needs to make sure the specified target directory for the checkpoints matches the virtual machine, otherwise one could overwrite existing checkpoint files. Also not sure what would happen if a user accidently specifies another checkpoint directory on the second backup run, or points to an already existing directory not unique...
Things can become messy then :)

I think it would make sense to put the checkpoint xml definition to the backup target directory, maybe in a subdirectory or something, because currently the tools saves information about existing checkpoints there too.

Thanks for your input!

abbbi · 2021-05-03T10:55:57Z

virtnbdbackup

@@ -59,6  59,9 @@ def main():
    parser.add_argument(
        "-o", "--output", required=True, type=str,
        help="Output target directory")
+    parser.add_argument(
+        '-C', '--checkpointdir', required=True, type=str,


If backing up multiple virtual machines you need one checkpoint directory per virtual machine, bcs the checkpoint name is quite hardcoded now. Wouldnt it make sense to store the checkpoint xml in the same directory as the backup files then? Otherwise, maybe it makes sense to have the virtual machines name included in the checkpoint name, but that would break existing backup chains.

If backing up multiple virtual machines you need one checkpoint directory per virtual machine,

Yes

bcs the checkpoint name is quite hardcoded now. Wouldnt it make sense to store the checkpoint xml in the same directory as the backup files then?

Let's consider scenario:

Have full and incremental backups in directory /backup/domain.20210502

Domain stopped and migrated to another host

Want to make full backup to /backup/domain.20210503

We'll need to specify "previous backup directory" to access saved checkpoint configs
It is possible but will add unnecessary complexity to backup scripts using virtnbdbackup IMHO

Otherwise, maybe it makes sense to have the virtual machines name included in the checkpoint name, but that would break existing backup chains.

Oops forgot to mention I've assumed that checkpointdir contains only one domain checkpoints, exactly "one checkpoint directory per virtual machine"

For example:

root@d02l:/backup/virtnbdbackup# virtnbdbackup -d vm-template -l inc -o /backup/vm-template.20210503 -C /var/lib/libvirt/images.drbd0/checkpoints/vm-template 2021-05-03 15:17:54 INFO common - printVersion: Version: 0.19 Arguments: /usr/local/bin/virtnbdbackup -d vm-template -l inc -o /backup/vm-template.20210503 -C /var/lib/libvirt/images.drbd0/checkpoints/vm-template ... 2021-05-03 15:17:55 INFO virtnbdbackup - main: Finished root@d02l:/backup/virtnbdbackup# ls -l /var/lib/libvirt/images.drbd0/checkpoints/vm-template total 36 -rw-r--r-- 1 root root 7843 May 3 12:42 virtnbdbackup.0.xml -rw-r--r-- 1 root root 7899 May 3 12:43 virtnbdbackup.1.xml -rw-r--r-- 1 root root 7899 May 3 15:17 virtnbdbackup.2.xml root@d02l:/backup/virtnbdbackup#

with VM name already present in storage path, add one more VM name to checkpoint woud be a tautology

Have full and incremental backups in directory /backup/domain.20210502

Domain stopped and migrated to another host

Want to make full backup to /backup/domain.20210503

We'll need to specify "previous backup directory" to access saved checkpoint configs

yes, thats true.. it needs to redefine the checkpoints if not existant so it can remove them correctly before the next full backup execution.. to have libvirt and qcow files correctly in sync.

ccrssaa · 2021-05-03T12:58:05Z

Currently its a quite specific setup and the checkpoint dir shouldnt be mandatory at the moment to allow for easier use of the utility.

Agree, going to make it optional

If the functionality is merged, the save function needs to make sure the specified target directory for the checkpoints matches the virtual machine, otherwise one could overwrite existing checkpoint files. Also not sure what would happen if a user accidently specifies another checkpoint directory on the second backup run, or points to an already existing directory not unique...
Things can become messy then :)

Hopefully libvirt will complain when redefining checkpoints, going to test it

I think it would make sense to put the checkpoint xml definition to the backup target directory, maybe in a subdirectory or something, because currently the tools saves information about existing checkpoints there too.

What if:

make checkpointdir optional
use output dir to store and load checkpoint configs if no checkpointdir provided
use specified directory if checkpointdir was explicitly provided
get rid of .cpt file ?

abbbi · 2021-05-03T19:06:09Z

make checkpointdir optional

use output dir to store and load checkpoint configs if no checkpointdir provided

use specified directory if checkpointdir was explicitly provided

sounds reasonable for me. Subfolder would be great in output dir.

get rid of .cpt file ?

i would leave the .cpt file in place at the moment, it prevents users beeing able to accidently write incremental backups to a folder where already backups of another virtual machine exist (see: #11)

ccrssaa · 2021-05-09T13:48:35Z

If the functionality is merged, the save function needs to make sure the specified target directory for the checkpoints matches the virtual machine, otherwise one could overwrite existing checkpoint files. Also not sure what would happen if a user accidently specifies another checkpoint directory on the second backup run, or points to an already existing directory not unique...
Things can become messy then :)

Hopefully libvirt will complain when redefining checkpoints, going to test it

accidentally tested, libvirtd refused to redefine checkpoint with incorrect uuid:
invalid argument: definition for checkpoint virtnbdbackup.0 must use uuid ...

abbbi · 2021-05-10T14:16:40Z

hi,

i merged this with some changes. Its not mandatory now anymore, but the rest should work as expected. Also i added some documentation for it here:

https://github.com/abbbi/virtnbdbackup#transient-virtual-machines-checkpoint-persistency

sovjet · 2023-01-21T19:28:23Z

Hello,
but how does one reset this thing : "Bitmap already exists:"
I moved domain to other kvm server and just want to start with full backup.

abbbi · 2023-01-21T19:37:10Z

Hello, but how does one reset this thing : "Bitmap already exists:" I moved domain to other kvm server and just want to start with full backup.

in this situation, you have to use the "qemu-img" tool to show existant bitmaps and remove them from the qcow image.

qemu-img info to list bitmaps in the qcow image and qemu-img bitmap --remove to remove them.

ccrssaa added 2 commits May 2, 2021 17:02

re-create checkpoints for incremental backup

2a0b22f

persistent checkpoint storage

24efc95

changed checkpoint names to virtnbdbackup.0, .1 etc to simplify file sorting

abbbi reviewed May 3, 2021

View reviewed changes

abbbi merged commit ebd201d into abbbi:master May 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

persistent checkpoint storage #10

persistent checkpoint storage #10

ccrssaa commented May 3, 2021

abbbi commented May 3, 2021 •

edited

Loading

ccrssaa commented May 3, 2021

abbbi left a comment •

edited

Loading

abbbi May 3, 2021

ccrssaa May 3, 2021 •

edited

Loading

abbbi May 3, 2021

ccrssaa commented May 3, 2021

abbbi commented May 3, 2021 •

edited

Loading

ccrssaa commented May 9, 2021

abbbi commented May 10, 2021 •

edited

Loading

sovjet commented Jan 21, 2023

abbbi commented Jan 21, 2023 •

edited

Loading

persistent checkpoint storage #10

persistent checkpoint storage #10

Conversation

ccrssaa commented May 3, 2021

abbbi commented May 3, 2021 • edited Loading

ccrssaa commented May 3, 2021

abbbi left a comment • edited Loading

Choose a reason for hiding this comment

abbbi May 3, 2021

Choose a reason for hiding this comment

ccrssaa May 3, 2021 • edited Loading

Choose a reason for hiding this comment

abbbi May 3, 2021

Choose a reason for hiding this comment

ccrssaa commented May 3, 2021

abbbi commented May 3, 2021 • edited Loading

ccrssaa commented May 9, 2021

abbbi commented May 10, 2021 • edited Loading

sovjet commented Jan 21, 2023

abbbi commented Jan 21, 2023 • edited Loading

abbbi commented May 3, 2021 •

edited

Loading

abbbi left a comment •

edited

Loading

ccrssaa May 3, 2021 •

edited

Loading

abbbi commented May 3, 2021 •

edited

Loading

abbbi commented May 10, 2021 •

edited

Loading

abbbi commented Jan 21, 2023 •

edited

Loading