Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] Improve image import speed #386

Open
ghost opened this issue Oct 27, 2020 · 4 comments
Open

[Enhancement] Improve image import speed #386

ghost opened this issue Oct 27, 2020 · 4 comments
Assignees
@iwilltry42
Labels
enhancement New feature or request help wanted Extra attention is needed priority/medium
Milestone
Backlog

Comments

@ghost
Copy link

ghost commented Oct 27, 2020 •
edited by iwilltry42
Loading

Improve Image Import Speed

Example benchmark

  • Setup: kind and k3d with one node each (no special settings)
  • Skaffold: import 3 images (k3d, k3d-proxy, k3d-tools)
  • kind: ~6s
  • k3d: ~13s

Investigation

  • skaffold calls k3d image import 3 times (once per image) instead of once for all three images
    • this causes the tools node to be created/started/stopped/removed 3 times in a row, introducing some delay
      • even with an experimental dev setup that keeps the tools node running, we end up with around 10s for three images (command still called 3 times though)

Original Bug Report: [BUG] loading images into k3d with Skaffold takes ages (kind does it nearly instantly)

What did you do

I use Skaffold to start a Docker image in my k3d cluster.
This is the corresponding log output:

Loading images into kind cluster nodes...

  • eu.gcr.io/tools-org-veroo/party-service:75f7ed6465b22c5d8dd3d898e55e3a843cadde9ff051534b7442a31b4444c059 -> Loaded
  • eu.gcr.io/tools-org-veroo/party-service-flyway:7729370e127ad323294df235500d12326065e6c61e7a1a54713880a9393c8ef0 -> Found

Unfortunately, loading the new image into the cluster takes ages (30 seconds, 40 seconds) with k3d.
Loading it into a kind cluster is much faster. With kind it is loaded nearly instantly.
Is there a way to load an image with k3d as fast as with kind? How can I speed up this process?

  • How was the cluster created?

    • k3d cluster create --switch-context

  • What did you do afterwards?

    • skaffold dev

What did you expect to happen

Fast loading of image into the cluster.

Which OS & Architecture

  • macOS Catalina

Which version of k3d

k3d version v3.1.5
k3s version latest (default)

Which version of docker

Client: Docker Engine - Community
 Cloud integration  0.1.18
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 16:58:31 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
@ghost ghost added the bug Something isn't working label Oct 27, 2020
@iwilltry42
Copy link
Member

Hi @ErolPeel , thanks for opening this issue!
The import time is completely dependent on the images that you use.
In the end, we're only doing docker save and ctr image import, so there's not much we can optimize here.
For me it just took <20s to import 3 images in a fresh cluster (k3d-proxy, k3d-tools and k3d-dind images).
Can you paste some timings here for a fresh k3d and a fresh kind cluster?

@iwilltry42
Copy link
Member

iwilltry42 commented Oct 27, 2020 •
edited
Loading

Actually, I just had a look into the kind code and they're doing exactly the same thing (docker save -> ctr image import).
Did you do multiple runs? For k3d, the first run may take a little longer, as we're going a little different route using an extra k3d-tools container (first time, needs to pull the image and start the container):

  1. kind runs docker save directly on your host into some temp folder, while k3d starts a k3d-tools container and runs docker save in there, saving into a shared docker volume
  2. kind runs opens the saved tar for every node and "streams" the content to the node containers, exec'ing ctr image import in the node containers, while k3d runs ctr image import in the nodes, directly reading the file from the shared volume

UPDATE: I can confirm though, that it takes roughly double the time for k3d 🤔

UPDATE 2: Figured out, that skaffold calls k3d image import again for every image in the stack (3 times in my case), thus starting/stopping the k3d-tools container every time, which takes quite a while compared to just importing all images at once.

UPDATE 3: Managed to shave off some seconds by adding an extra flag to not delete the tools node.. will invest some more time into this soon

@ghost
Copy link
Author

ghost commented Oct 28, 2020

@iwilltry42 Thank you very much for looking into this issue.
We are using k3d in our GitHub Actions Workflow, so speeding up the image loading process would save us time.
Is there a way I can help? Would logs with timestamps still be helpful?

@iwilltry42 iwilltry42 added the enhancement New feature or request label Oct 28, 2020
@iwilltry42 iwilltry42 self-assigned this Oct 28, 2020
@iwilltry42 iwilltry42 added this to the 3.2.0 milestone Oct 28, 2020
@iwilltry42 iwilltry42 changed the title [BUG] loading images into k3d with Skaffold takes ages (kind does it nearly instantly) [Enhancement] Improve image import speed Oct 28, 2020
@iwilltry42 iwilltry42 added help wanted Extra attention is needed priority/medium and removed bug Something isn't working labels Oct 28, 2020
@iwilltry42
Copy link
Member

@ErolPeel , in the GH Actions workflow, you can probably improve already, by calling the k3d image import command once and pass it all the images instead of calling it per-image (in case you're currently doing it like skaffold does).
Timestamped logs are not needed anymore, thank you!

@iwilltry42 iwilltry42 modified the milestones: 3.2.0, v3.4.0 Nov 24, 2020
@iwilltry42 iwilltry42 modified the milestones: v3.4.0, v4.1.0 Dec 4, 2020
@iwilltry42 iwilltry42 modified the milestones: v4.1.0, v4.2.0 Feb 3, 2021
@iwilltry42 iwilltry42 modified the milestones: v4.3.0, v4.4.0 Mar 10, 2021
@iwilltry42 iwilltry42 modified the milestones: v4.4.5, Backlog Jun 11, 2021
@sbaier1 sbaier1 mentioned this issue Oct 26, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iwilltry42 iwilltry42

Labels
enhancement New feature or request help wanted Extra attention is needed priority/medium
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
1 participant
@iwilltry42