Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default config: use virtual disk changer as FileStorage #1467

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

pstorz
Copy link
Member

@pstorz pstorz commented May 12, 2023

This PR changes the existing single storage device "FileStorage" into a multiple device disk autochanger.
This should make the default config better suitable to scale on customer installations compared to the old configuration.

Backport of PR #0000 to bareos-2x (remove this line, if it no backport)

Please check

  • Short description and the purpose of this PR is present above this paragraph
  • Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General
  • Is the PR title usable as CHANGELOG entry?
  • Purpose of the PR is understood
  • Commit descriptions are understandable and well formatted
  • Check backport line
  • Required backport PRs have been created
Source code quality
  • Source code changes are understandable
  • Variable and function names are meaningful
  • Code comments are correct (logically and spelling)
  • Required documentation changes are present and part of the PR
Tests
  • Decision taken that a test is required (if not, then remove this paragraph)
  • The choice of the type of test (unit test or systemtest) is reasonable
  • Testname matches exactly what is being tested
  • On a fail, output of the test leads quickly to the origin of the fault

pstorz added 3 commits April 26, 2023 09:09
This commit switches the FileStorage to be a multidevice
instead of one single device. As the name stays the same
updates should be as smooth as possible.
@pstorz pstorz self-assigned this May 12, 2023
@pstorz pstorz marked this pull request as draft May 12, 2023 08:37
@pstorz pstorz force-pushed the dev/pstorz/master/default-config-virt-diskchanger branch from 241714a to 3f45fc5 Compare June 21, 2023 12:20
@pstorz pstorz marked this pull request as ready for review June 21, 2023 12:20
@pstorz pstorz force-pushed the dev/pstorz/master/default-config-virt-diskchanger branch from 3f45fc5 to ed847e8 Compare June 21, 2023 13:38
@pstorz pstorz requested a review from bruno-at-bareos June 29, 2023 10:26
Copy link
Contributor

@bruno-at-bareos bruno-at-bareos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice start, I've some concerns, please see suggestions and remarks.

@@ -0,0 1,6 @@
Autochanger {
Name = "FileStorage"
Copy link
Contributor

@bruno-at-bareos bruno-at-bareos Jun 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 remarks: we shouldn't call it FileStorage as the name was used previously to describe the device and dir storage.
FileStorageChanger or FileChanger
Then if we keep the CamelCase format for the Name, then the filename should be using the same format

I understand that keeping FileStorage here allow the director config to be kept as it is. But what will happen to people having a working installation? Moving suddenly to a multiple devices will completely change the behavior, isn't it?

Can we test it in a systemtest?

Name = "FileStorage"
Changer Device = /dev/null # required for virtual disk changer
Changer Command = "" # required for virtual disk changer
Device = FileStorage # basename of the multidevice devices
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Align comments?

Suggested change
Device = FileStorage # basename of the multidevice devices
Device = FileStorage # basename of the multidevice devices

@@ -1,4 1,5 @@
Device {
Description = "Multiplied file device. A connecting Director must have the same Name and MediaType."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the name will be the autochanger name

Suggested change
Description = "Multiplied file device. A connecting Director must have the same Name and MediaType."
Description = "Multiplied file device. A connecting Director must have the same MediaType."

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should AutoChanger = yes also be set?

@@ -1,4 1,5 @@
Device {
Description = "Multiplied file device. A connecting Director must have the same Name and MediaType."
Name = FileStorage
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about placing the number just below the name, so people will better understand why they got FileStorage0001 etc. Isn't 10 a too high number?

More you have read/write device, more volumes you will use, and you will have to explain users, that they will need as much volume per day equal the number of devices if concurrency is high enough of course.

Suggested change
Name = FileStorage
Name = FileStorage
Count = 10 # Number of instances

Maybe going to 4 will be easier to explain and document like you have 2 writer / 2 reader for AI etc...

@@ -7,5 8,6 @@ Device {
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
Description = "File device. A connecting Director must have the same Name and MediaType."
Count = 10 # Number of instances
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Count = 10 # Number of instances

as placed just after the name

@@ -2,6 2,7 @@
@backenddir@/libbareossd-file.so*
@scriptdir@/disk-changer
@configtemplatedir@/bareos-sd.d/device/FileStorage.conf
@configtemplatedir@/bareos-sd.d/autochanger/filechanger.conf
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending of action (name formating) this need to be adjusted.

@@ -6,6 6,7 @@
@dir(bareos,bareos,0750) %%ETCDIR%%/bareos-sd.d/messages
@dir(bareos,bareos,0750) %%ETCDIR%%/bareos-sd.d/storage
@sample(bareos,bareos,0640) %%ETCDIR%%/bareos-sd.d/device/FileStorage.conf.sample
@sample(bareos,bareos,0640) %%ETCDIR%%/bareos-sd.d/autochanger/filechanger.conf.sample
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment below about filename/name

@@ -1260,6 1260,7 @@ mkdir -p %{?buildroot}/%{_libdir}/bareos/plugins/vmware_plugin
%attr(0750, %{storage_daemon_user}, %{daemon_group}) %dir %{_sysconfdir}/%{name}/bareos-sd.d/ndmp
%attr(0750, %{storage_daemon_user}, %{daemon_group}) %dir %{_sysconfdir}/%{name}/bareos-sd.d/messages
%attr(0750, %{storage_daemon_user}, %{daemon_group}) %dir %{_sysconfdir}/%{name}/bareos-sd.d/storage
%attr(0640, %{storage_daemon_user}, %{daemon_group}) %config(noreplace) %{_sysconfdir}/%{name}/bareos-sd.d/autochanger/filechanger.conf
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment below about filename/name

@@ -0,0 1,6 @@
Autochanger {
Name = "FileStorage"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Name = "FileStorage"
Name = "FileStorage"
Description = "A virtual autochanger for file device. A connecting Director must have the same Name and use the MediaType of related device.

@bruno-at-bareos
Copy link
Contributor

Also forget to mention, if this become the default configuration, I guess it would be a good idea, that it become also the default configuration for a bunch of the systemtest, what's your thoughts on this ?

@wizhippo
Copy link

Can you confirm the value of this? Using a virtual autochanger does not automatically allow concurrent/parallel jobs where the data is written to serrate volumes does it?

To do that you still would have to list our all the devices in the director sd config.

@arogge
Copy link
Member

arogge commented Sep 27, 2023

The whole point of this is to use an autochanger in the SD so that you just mention that autochanger in the director's configuration.
The individual devices will then all be handled inside the SD only.

@wizhippo
Copy link

The whole point of this is to use an autochanger in the SD so that you just mention that autochanger in the director's configuration. The individual devices will then all be handled inside the SD only.

I understand this part but what does using an autochanger on 'disk' do that a 'non-autochanger disk' storage provide? Does is allow parallel jobs to run saving each job to separate volume? or does it behave just like tape and only one volume is mounted at a time so you have no benefit of parallel jobs to separate volumes?

@arogge
Copy link
Member

arogge commented Sep 27, 2023

One volume at a time per device. But you can easily add 100 Devices to your file autochanger (which would be a real pain to configure without collecting them in an autochanger).

@wizhippo
Copy link

One volume at a time per device. But you can easily add 100 Devices to your file autochanger (which would be a real pain to configure without collecting them in an autochanger).

Sorry to drag the out, "One volume at a time per device"

If an autochanger has 10 devices defined by count 10, then '10 devices' can be mounted at once and 10 concurrent jobs will run writing to 10 volumes on 10 devices?

As I have never used tape, maybe I misunderstand that an autochanger can change multiple tape drives at once instead only one,

If this is the case then I see the value of the virtual auto changer.

@bruno-at-bareos
Copy link
Contributor

One volume at a time per device. But you can easily add 100 Devices to your file autochanger (which would be a real pain to configure without collecting them in an autochanger).

Sorry to drag the out, "One volume at a time per device"

If an autochanger has 10 devices defined by count 10, then '10 devices' can be mounted at once and 10 concurrent jobs will run writing to 10 volumes on 10 devices?

As I have never used tape, maybe I misunderstand that an autochanger can change multiple tape drives at once instead only one,

If this is the case then I see the value of the virtual auto changer.

Yes that's the case.

@bruno-at-bareos
Copy link
Contributor

In case we will certainly add the following tricks to keep the first drive always available for label and restore.

Autochanger {
 Name = "vchanger"
 Description = "Virtual changes for disk based devices"
 # We don't have a real changer
 Changer Device = /dev/null
 # We don't have yet a smart disk changer
 Changer Command = ""
 # the Autochanger will contain multiple drive devices
 # we use the autonumbering bareos function for that.
 # the following order is important
 Device = vdev-0000      # Reserved inactive drive autoselect=no for labeling and restores
 Device = vdev-          # the rest of the autonumbered drives
}

Device {
 Name = "vdev-0000"
 Description = "This device with autoselect=no is reserved for restores and labeling volumes"
 Auto Select = No
 Media Type = "File"
 Device Type = "File"
 Archive device = "/var/lib/bareos/storage"
 LabelMedia = Yes
 Random Access = Yes
 AutomaticMount = Yes
 RemovableMedia = No
 Maximum Concurrent Jobs = 1
 Maximum File Size = 100GB
}
Device {
 Name = "vdev-"
 Description = "8 devices for file backups"
 Count = 8                  # Will create multiple devices vdev-0001 to vdev-0008
 Media Type = "File"
 Device Type = "File"
 Archive device = "/var/lib/bareos/storage"
 LabelMedia = Yes
 Random Access = Yes
 AutomaticMount = Yes
 RemovableMedia = No
 Maximum Concurrent Jobs = 1
 Maximum File Size = 100GB
}

By default a label will can use the drive 0

label volume=test storage=vchanger drive 0 yes

@wizhippo
Copy link

In case we will certainly add the following tricks to keep the first drive always available for label and restore.

Autochanger {
 Name = "vchanger"
 Description = "Virtual changes for disk based devices"
 # We don't have a real changer
 Changer Device = /dev/null
 # We don't have yet a smart disk changer
 Changer Command = ""
 # the Autochanger will contain multiple drive devices
 # we use the autonumbering bareos function for that.
 # the following order is important
 Device = vdev-0000      # Reserved inactive drive autoselect=no for labeling and restores
 Device = vdev-          # the rest of the autonumbered drives
}

Device {
 Name = "vdev-0000"
 Description = "This device with autoselect=no is reserved for restores and labeling volumes"
 Auto Select = No
 Media Type = "File"
 Device Type = "File"
 Archive device = "/var/lib/bareos/storage"
 LabelMedia = Yes
 Random Access = Yes
 AutomaticMount = Yes
 RemovableMedia = No
 Maximum Concurrent Jobs = 1
 Maximum File Size = 100GB
}
Device {
 Name = "vdev-"
 Description = "8 devices for file backups"
 Count = 8                  # Will create multiple devices vdev-0001 to vdev-0008
 Media Type = "File"
 Device Type = "File"
 Archive device = "/var/lib/bareos/storage"
 LabelMedia = Yes
 Random Access = Yes
 AutomaticMount = Yes
 RemovableMedia = No
 Maximum Concurrent Jobs = 1
 Maximum File Size = 100GB
}

By default a label will can use the drive 0

label volume=test storage=vchanger drive 0 yes

Great tip thank you.

@bruno-at-bareos
Copy link
Contributor

@wizhippo

Great tip thank you.

The whole objective of this PR is to be the next default, but migrating the existent is tricky ;-)

@bruno-at-bareos
Copy link
Contributor

#2090 might be a candidate to superseed this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants