-
Notifications
You must be signed in to change notification settings - Fork 706
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very long starting time at every launch of rqlited #1798
Comments
450GB is a very large SQLite database obviously. So what happens is this -- on startup the SQLite database must be rebuilt by the Raft system. This involves making a copy of your 450GB database. It's possible this could be optimized, such that the copy is avoided, I have some ideas but have yet to implement. So the question (for now) is "it's taking rqlited over an hour to perform a 450GB copy on my system. Is that too long?". I have some questions that will help us answer the question.
I do think long-term for such large databases rqlite needs to avoid the copy. But it needs careful design and implementation. |
ok thank you |
In principle it could be solved like this (for my own records, mostly):
For this to be safe, the following would need to change in the code:
|
The key concern I need to address is the following: the copy of the SQLite database that is there at restart time (not the one in the snapshot store, but the one under management) is not correct and has been corrupted somehow. One way around this could be the following:
|
Is this check done only at first node startup? For example: let's say I have a running cluster of 3 nodes, one node goes offline, I add a new node. Of course this node has to resync, but the cluster remains operational, right? The snapshot is reloaded from scratch only when the first node of a cluster is started? |
and about this feature: Log Compaction and Truncation does the snapshot lock the database? and does this mean the disk usage is double the size of the database? |
Assuming I follow your question, yes, the cluster remains operational.
Yes, for a brief period of time, but usually too short to notice unless your write load is continuously very high.
Basically yes. This is core to the way Raft works, it is not something special that rqlite does. |
The PR linked to this issue fixes the start-up issue. It actually appears much easier to fix than I realised before. While the initial boot will take as long as it's going to take, restarts will skip the copy. One cost is that graceful shutdowns of a node may take a little longer, but shouldn't be more than a second or two. |
Wow! Great! I'm going to test it in the next few days. And about the periodic snapshot, just to be sure to understand, is it a full copy? Going back to my situation, the snapshot of a ~450Gb database would take the same time of copying the whole file? |
No, snapshotting takes an incremental approach. See: https://www.philipotoole.com/rqlite-8-0-large-dataset-support-and-core-feature-upgrades/ "A new approach to Raft snapshotting" |
Snapshotting works in an incremental manner. It copies just the changes that have taken place since the last snapshot. So once your initial Boot operation is done, Snapshotting doesn't copy the entire 450GB data every time. If your boot your system, and then write, say, 1MB of data to rqlite, then at most Snapshotting will copy 1MB of data. The issue you hit is just a restart thing. In that case the entire 450GB dataset is copied from the Raft subsystem back into rqlite on node restart. However I believe I have an approach that can avoid even that copy -- it looks easier than I initially thought (see the PR), which is why I never worked on it before. I may have an experimental version you could test in a day or two. Are you able to build from source? |
sure |
Thinking about this some more, it's given me some ideas for more fundamental changes that will solve this issue better -- and save disk space. So it may actually be sometime (weeks) before I decrease start-up times for such large databases. |
I built the branch reuse-sqlite-db, the startup time has decreased from more than 1 hour to about 10 minutes... It's very good, do you think it will be even faster? Or are you talking about better code? |
I might be able to make it faster. Can you show me the logs from your startup? I wonder where the 10 mins is being spent -- it must be in the sha256sum check. Background: to be sure the "main" SQLite database can be used without getting a copy from the Snapshot store, rqlite first calculates a SHA256 hash of the SQLite file. I guess it's taking 10 mins to calculate that sum? How long does it take to calculate such a sum of a 450GB file on the machine? Can you check? |
This is for a ~20GB file. So it takes ~40 seconds on a VM, which is running on modern hardware (yeah, I know this is kinda vague). So a file 20x large would take minutes. So a 10 minute start-up with a 450GB SQLite file seems about right. Simply skipping the checksum check would probably get your startup down to seconds. It could be an optional check. |
We could also consider a much faster integrity check, such as CRC32. Might be good for rqlite, where I am not concerned about malicious tampering. A CRC32 of the same data takes 13 seconds to run. |
Yes the time spent at startup is due to the sha256sum check. Could it be done somehow on chunks of data, to parallelize and make it faster? |
How do you know? Did you add logging?
Yes, use a different hash technique (see above), or just skip it. |
Rqlited startup told me this |
Could probably get a ~4x speed up by using this function: func computeCRC32(filePath string) (string, error) {
file, err := os.Open(filePath)
if err != nil {
return "", fmt.Errorf("failed to open file: %v", err)
}
defer file.Close()
hash := crc32.NewIEEE()
if _, err := io.Copy(hash, file); err != nil {
return "", fmt.Errorf("failed to compute checksum: %v", err)
}
checksum := hex.EncodeToString(hash.Sum(nil))
return checksum, nil
} instead of the existing function -- going forward that is. |
My plan is to fix this in 9.0. I've already started development of 9.0: #1810 |
Good morning.
I started a single node rqlited (planning to add nodes and make it a cluster of 3) and imported a big sqlite database (~450Gb) via .boot command in CLI.
Everything went as planned (it took some hours to import and validate the database), but when I stopped and restarted rqlited, it said there was a preexisting snapshot present and did a restore operation that lasted more than 1 hour.
Here's the log (some data is hidden by *****)
[rqlited] 2024/06/10 14:51:19 rqlited starting, version v8.25.0, SQLite 3.45.1, commit 312d6b0, branch master, compiler (toolchain) gc, compiler (command) gcc
[rqlited] 2024/06/10 14:51:19 go1.22.3, target architecture is amd64, operating system target is linux
[rqlited] 2024/06/10 14:51:19 launch command: rqlited -node-id 2 -http-addr 0.0.0.0:4001 -http-adv-addr *****:4001 -raft-addr 0.0.0.0:4002 -raft-adv-addr *****:4002 -auth config.json -http-cert ./server-cert.crt -http-key ./server-key.pem ./rqlite/file/data
[rqlited] 2024/06/10 14:51:19 preexisting node state detected in /media/NVMEdisk2/rqlite/file/data
[mux] 2024/06/10 14:51:19 mux serving on [::]:4002, advertising *****:4002
[cluster] 2024/06/10 14:51:19 service listening on *****:4002
[http] 2024/06/10 14:51:19 secure HTTPS server enabled with cert ./server-cert.crt, key ./server-key.pem, mutual TLS disabled
[http] 2024/06/10 14:51:19 execute queue processing started with capacity 1024, batch size 128, timeout 50ms
[http] 2024/06/10 14:51:19 service listening on [::]:4001
[store] 2024/06/10 14:51:19 opening store with node ID 2, listening on :4002
[store] 2024/06/10 14:51:19 ensuring data directory exists at /media/NVMEdisk2/rqlite/file/data
[store] 2024/06/10 14:51:19 old snapshot directory /media/NVMEdisk2/rqlite/file/data/snapshots does not exist, nothing to upgrade
[snapshot-store] 2024/06/10 14:51:19 store initialized using /media/NVMEdisk2/rqlite/file/data/rsnapshots
[snapshot-store] 2024/06/10 14:51:19 checking consistency of snapshot store at /media/NVMEdisk2/rqlite/file/data/rsnapshots
[snapshot-store] 2024/06/10 14:51:19 check complete
[store] 2024/06/10 14:51:19 1 preexisting snapshots present
[store] 2024/06/10 14:51:21 initiating node restore on node ID 2
[store] 2024/06/10 15:58:23 successfully opened database at /media/NVMEdisk2/rqlite/file/data/db.sqlite due to restore
[store] 2024/06/10 15:58:23 node restored in 1h7m1.313024855s
[rqlited] 2024/06/10 15:58:23 node HTTP API available at https://:4001
[rqlited] 2024/06/10 15:58:23 connect using the command-line tool via 'rqlite -H ***** -p 4001'
[raft] 2024/06/10 15:58:24 [WARN] heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
[store] 2024/06/10 15:58:24 this node (ID=2) is now Leader
^C[rqlited] 2024/06/10 16:14:55 received signal "interrupt"
[http] 2024/06/10 16:14:55 closing HTTP service on [::]:4001
[http] 2024/06/10 16:14:55 HTTP service on [::]:4001 stopped: http: Server closed
[rqlited] 2024/06/10 16:14:55 stepping down as Leader before shutdown
[store] 2024/06/10 16:14:55 store closed with node ID 2, listening on *****:4002
[rqlited] 2024/06/10 16:14:55 rqlite server stopped
The text was updated successfully, but these errors were encountered: