Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QUESTION/HELP] DNSError when switching to mobile network #1367

Open
schlichtanders opened this issue Oct 29, 2023 · 3 comments
Open

[QUESTION/HELP] DNSError when switching to mobile network #1367

schlichtanders opened this issue Oct 29, 2023 · 3 comments
Labels
question Further information is requested

Comments

@schlichtanders
Copy link

I would like to use k3d to work on the go, but when switching to mobile network, the cluster cannot resolve DNS. A similar http request from normal terminal does not run into this.

I've read about #209 but it didn't help in my case

What did you do

  • How was the cluster created?

    • k3d cluster create --registry-use k3d-registry.localhost:12345 --api-port 6550 -p "80:80@loadbalancer" --image "rancher/k3s:v1.26.9-k3s1" --servers 1 --agents 1

  • What did you do afterwards?

    • k3d commands? no further k3d commands
    • docker commands? I was deploying my application to the cluster
    • OS operations (e.g. shutdown/reboot)? I switched to mobile network

What did you expect to happen

K3d should be able to resolve DNS without error also after switching to mobile network. The browser and terminal do well, but applications running inside the k3d cluster fail to resolve DNS.

Screenshots or terminal output

DNSError: api.github.com, temporary failure (EAI_AGAIN)

My program tries to access the api.github.com server

Which OS & Architecture

  • output of k3d runtime-info
arch: x86_64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: gram17
name: docker
os: NixOS 23.05 (Stoat)
ostype: linux
version: 20.10.25

Which version of k3d

  • output of k3d version
k3d version v5.6.0
k3s version v1.21.7-k3s1 (default)

Which version of docker

  • output of docker version and docker info
Client:
 Version:           20.10.25
 API version:       1.41
 Go version:        go1.20.8
 Git commit:        v20.10.25
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.20.8
  Git commit:       v20.10.25
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.1
  GitCommit:        v1.7.1
 runc:
  Version:          1.1.5
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.4)
  compose: Docker Compose (Docker Inc., 2.18.1)

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 83
 Server Version: 20.10.25
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: v1.7.1
 runc version: 
 init version: 
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 6.5.3
 Operating System: NixOS 23.05 (Stoat)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.26GiB
 Name: gram17
 ID: (NOT SURE WHETHER THIS IS SENSIBLE INFORMATION - PLEASE DON'T LET PEOPLE SHARE SENSITIVE INFORMATION)
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: THIS IS PROBABLY SENSIBLE INFORMATION - PLEASE DON'T LET PEOPLE SHARE SENSITIVE INFORMATION
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true
@schlichtanders schlichtanders added the question Further information is requested label Oct 29, 2023
@schlichtanders schlichtanders mentioned this issue Oct 29, 2023
Closed
@schlichtanders
Copy link
Author

When enabling export K3D_FIX_DNS=1, the cert-manager (and probably others too) do not find rancher/mirrored-pause.

kubectl describe pod cert-manager-webhook -n cert-manager prints

...
 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp 18.215.138.58:443: i/o timeout
...

@schlichtanders
Copy link
Author

schlichtanders commented Oct 29, 2023 •
edited
Loading

A guess: my mobile network does not seem to find a couple of ip4 addresses. My host laptop resolves registry-1.docker.io not to the above mentioned ip4, but only to ip6 addresses.

EDIT: If I try to reach the ip addresses printed inside k3d from a laptop top-level terminal, I indeed cannot reach this IP.

Could this be the issues? Does k3d not support ip6 routing?

@schlichtanders
Copy link
Author

Further research brought me to this post which explains that my mobile provider (deutsche telekom) uses a new DNS resolution system which translates ip4 only addresses to some fancy ip6 wrapper which are then again unwrapped by the provider.

this DNS64 resolution does not seem to work when setting K3D_FIX_DNS=1 ... need to investigate further...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@schlichtanders