Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dqlite requires a large stack #679

Open
SimonRichardson opened this issue Aug 7, 2024 · 3 comments
Open

dqlite requires a large stack #679

SimonRichardson opened this issue Aug 7, 2024 · 3 comments

Comments

@SimonRichardson
Copy link
Member

SimonRichardson commented Aug 7, 2024 •
edited
Loading

Attempting to bootstrap Juju controller on arm64 with the latest version (1.16.7) of dqlite causes a segfault. This uses musl to statically compile the jujud-controller. The snapcraft.yaml contains all the dependencies for building the binary.

The CI run in question: https://github.com/juju/juju/actions/runs/10212413197/job/28255682924?pr=17836

Repro steps...

This has to be on aarch64 (arm64). I used multipass on a mac M1, but it could easily be done on an aws graviton machine.

Prerequisites:

  1. git
  2. build-essentials (make, etc).
  3. lxd (you might need to correctly run sudo lxd init --auto)
  4. snapcraft (grab from snap)
$ git checkout https://github.com/juju/juju/pull/17836
$ snapcraft --use-lxd
$ sudo snap install *.snap --dangerous
$ sudo snap connect juju:lxd lxd
$ sudo snap connect juju:config-lxd
$ sudo snap connect juju:dot-local-share-juju
$ sudo snap connect juju:ssh-keys
$ juju bootstrap lxd test --keep-broken

The segfault should happen.

$ lxc exec <juju container name> -- bash
$ apt install gdb
$ LIBDQLITE_TRACE=1 gdb /var/lib/juju/tools/3.6-beta2.1-ubuntu-arm64/jujud bootstrap-state --timeout 20m0s --data-dir '/var/lib/juju' --debug '/var/lib/juju/bootstrap-params'

The backtrace for the segfault: https://paste.ubuntu.com/p/FcY8WZ9GT3/
@hpidcock
Copy link
Member

hpidcock commented Aug 8, 2024

A follow-up to this can be found here.

TLDR: dqlite needs more than 128k of stack memory, musl's default.

This was referenced Aug 10, 2024
Merged
Merged
@cole-miller cole-miller mentioned this issue Sep 2, 2024
Draft
@SimonRichardson
Copy link
Member Author

I'd be happy to close this, with an advisory that if using musl you need to increase the default stack size in some docs or README.md.

@cole-miller
Copy link
Contributor

@SimonRichardson I've updated the README in #700. I was planning to keep this issue to open as a reminder to myself to investigate why dqlite needs a large stack---there might be another issue like the one with EXEC_SQL lurking.

@cole-miller cole-miller changed the title segfault on arm64, statically compiled with musl dqlite requires a large stack Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SimonRichardson @hpidcock @cole-miller