Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debian 9.6 AMD64: Segfault on start at os/user.Current (1.7.0 ) #6841

Closed
2 of 7 tasks
hendrikbl opened this issue May 3, 2019 · 19 comments · Fixed by #6862
Closed
2 of 7 tasks

Debian 9.6 AMD64: Segfault on start at os/user.Current (1.7.0 ) #6841

hendrikbl opened this issue May 3, 2019 · 19 comments · Fixed by #6862
Labels
topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile
Milestone
1.8.1

Comments

@hendrikbl
Copy link

  • Gitea version (or commit ref): 1.6.4
  • Git version: 2.11.0
  • Operating system: Debian 9.6
  • Database (use [x]):
    • PostgreSQL
    • MySQL (MariaDB)
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:

Description

After upgrading from gitea version 1.6.4 to 1.7.0 (replacing gitea binary with new one) I'm unable to start gitea. The gitea service fails with the following message (service gitea status):

● gitea.service - Gitea (Git with a cup of tea)
   Loaded: loaded (/etc/systemd/system/gitea.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2019-05-03 12:27:24 CEST; 1s ago
  Process: 31656 ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini (code=exited, status=2)
 Main PID: 31656 (code=exited, status=2)

If I try to run it directly via gitea web I get the following:
Gist

More logs:

Mai 03 12:18:43 gitea systemd[1]: Started Gitea (Git with a cup of tea).
Mai 03 12:18:43 gitea gitea[29491]: fatal error: unexpected signal during runtime execution
Mai 03 12:18:43 gitea gitea[29491]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
Mai 03 12:18:43 gitea gitea[29491]: runtime stack:
Mai 03 12:18:43 gitea gitea[29491]: runtime.throw(0x19c3d99, 0x2a)
Mai 03 12:18:43 gitea systemd[1]: gitea.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mai 03 12:18:43 gitea systemd[1]: gitea.service: Unit entered failed state.
Mai 03 12:18:43 gitea systemd[1]: gitea.service: Failed with result 'exit-code'.

I get this error with every version greater than 1.6.4. All permissions were set the same with every version I tried. I haven't tried building gitea myself since this is something I only want to do if there really is no other way.
@zeripath
Copy link
Contributor

zeripath commented May 3, 2019

Are you running Gitea on a raspberry pi? If so there is a known problem with the arm7 binary and you need to use arm6.

If not what kernel version do you have? If it's very old you're likely to need to build Gitea yourself as we build on Ubuntu 18.04.

Finally, please don't run 1.7.0 there are a number of security issues in that version and we recommend 1.8 if not at least 1.7.6 (there are fixes in 1.8 that cannot be ported back to 1.7.6)

@lunny lunny added the issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail label May 6, 2019
@hendrikbl
Copy link
Author

I'm on a virtual machine running Debian 9.6 x86_64 with kernel version 4.9.0-4-amd64.

I tried version 1.7.0, 1.7.6 and 1.8. They all give me the same error.

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019

Ok.

Under what user are you trying to run Gitea?
And where? Things don't make sense.

The segfault looks like it's happening when Gitea tries to get the current username. That implies that you're running Gitea as a weird user.

Although you state:

/usr/local/bin/gitea web -c /etc/gitea/app.ini

Gitea seems to be trying get its config:

/usr/local/bin/custom/conf/app.ini

Not in /etc...

What does your service file look like?

Does Gitea run when you just run it as a normal user from a normal directory?

@hendrikbl
Copy link
Author

hendrikbl commented May 6, 2019 •
edited
Loading

I followed the documentation when first installing gitea. So gitea service is running as git user:

root@gitea:~# cat /etc/passwd | grep git
git:x:109:116:Git Version Control,,,:/home/git:/bin/bash

The binary is located at /usr/local/bin. The working directory is /var/lib/gitea/. As seen in the documentation.

Here's my service file: gitea.service

I tried running gitea as root and git from the corresponding home directory but i still get a segfault.

Although you state:

/usr/local/bin/gitea web -c /etc/gitea/app.ini

Gitea seems to be trying get its config:

/usr/local/bin/custom/conf/app.ini

Not in /etc...

Thats my fault. The log I gave you was from running gitea without the -c argument. Here are the three things I tried:

  • User: git, Directory: /usr/local/bin - Gist
  • User: git, Directory: /home/git - Gist
  • User: root, Directory: home/root - Gist

Does Gitea run when you just run it as a normal user from a normal directory?

So the answer to this question is: No, I can't run gitea anywhere. The strange thing is that version 1.6.4 runs everytime on every circumstance. Everything thats newer than that fails.

Edit: Everything here was tested with version 1.8 of gitea.

@sapk
Copy link
Member

sapk commented May 6, 2019 •
edited
Loading

This thread on discourse seem related: https://discourse.gitea.io/t/solved-segfault-due-to-misconfigured-unix-user/1104

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019

Hmm. It looks like this is:

golang/go#24787

Which if I'm reading correctly causes panic on os/user.Current or Lookup for uid <1000.

Now it seems like it's suggested that we need to add more flags to our static build:

golang/go#24787 (comment)

But I bet a simple workaround is to have a uid>=1000. Now you may actually need that in any case because of the SSH requirements - in that if you want to clone over SSH you need to be able to login as that user and run authorized_keys or an authorized keys command which I'm not certain you can do as a uid <1000.

@sapk
Copy link
Member

sapk commented May 6, 2019

@zeripath It is a pity that we need an other tag (osusergo) like netgo but if it ensure consistency across env we should add it.

@hendrikbl
Copy link
Author

I tried running gitea with a new user with a uid>1000:

root@gitea:/usr/local/bin# grep gitea /etc/shadow
gitea:!:18022:0:99999:7:::
root@gitea:/usr/local/bin# grep gitea /etc/passwd
gitea:x:1003:1002:Gitea,,,:/home/gitea:/bin/bash

Still no luck. I get exactly the same. Current version 1.6.4 runs perfectly, everything above that fails.

Can I use the gitea dump command on my current version, create a fresh install of version 1.8 somewhere else and restore the dump? If that's the case I will try that.

@lunny
Copy link
Member

lunny commented May 6, 2019

@hendrikbl That's dump and restore are not guaranteed.

@sapk sapk mentioned this issue May 6, 2019
Merged
@zeripath
Copy link
Contributor

zeripath commented May 6, 2019

@hendrikbl I assume that by a uid > 1000 you haven't marked the user as a system user?

OK, so I wonder how we test this. I've just downloaded the docker debian:latest and that doesn't replicate the issue with 1.8.0

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019 •
edited
Loading

and I've just tried docker pull debian:9.6 and haven't got this issue either and 1.8 and 1.7.6

@hendrikbl
Copy link
Author

@zeripath correct, I just used adduser and followed the setup without setting a password.

Hmm, have you done everything according to the installation guide? Because that's what I did. I'll see if I can try this on a fresh debian install this week. I'm not sure when I have time for that though.

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019

OK, I mean I can start from the command line - I think getting systemd started on a debian docker is fairly difficult.

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019 •
edited
Loading

Oh I did add a password though... Nope that doesn't seem to matter

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019 

root@ef9a8c7df397:/home/gitsys# adduser git2
Adding user `git2' ...
Adding new group `git2' (1001) ...
Adding new user `git2' (1001) with group `git2' ...
Creating home directory `/home/git2' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
No password supplied
Enter new UNIX password: 
Retype new UNIX password: 
No password supplied
Enter new UNIX password: 
Retype new UNIX password: 
No password supplied
passwd: Authentication token manipulation error
passwd: password unchanged
Try again? [y/N] 
Changing the user information for git2
Enter the new value, or press ENTER for the default
	Full Name []: 
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] y
root@ef9a8c7df397:/home/gitsys# sudo -u git2 /gitea-1.7.6-linux-amd64 -c /home/git2/custom/conf/app.ini
2019/05/06 13:43:10 [W] Custom config '/home/git2/custom/conf/app.ini' not found, ignore this if you're running first time
2019/05/06 13:43:10 [T] AppPath: /gitea-1.7.6-linux-amd64
2019/05/06 13:43:10 [T] AppWorkPath: 
2019/05/06 13:43:10 [T] Custom path: custom
2019/05/06 13:43:10 [T] Log path: log
2019/05/06 13:43:10 [I] Gitea v1.7.6 built with go1.12.4 : bindata, sqlite, sqlite_unlock_notify
2019/05/06 13:43:10 [I] Log Mode: Console(Info)
2019/05/06 13:43:10 [I] XORM Log Mode: Console(Info)
2019/05/06 13:43:10 [I] Cache Service Enabled
2019/05/06 13:43:10 [I] Session Service Enabled
2019/05/06 13:43:10 [I] SQLite3 Supported
2019/05/06 13:43:10 [I] Run Mode: Development
2019/05/06 13:43:10 [I] Listen: http://0.0.0.0:3000
2019/05/06 13:43:10 Serving [::]:3000 with pid 5021
^C2019/05/06 13:43:12 Exiting pid 5021.

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019

Damn if we can't replicate this we can't be sure that osusergo will actually fix it.

@zeripath zeripath changed the title Update to 1.7.0 and above fails Debian 9.6 AMD64: Segfault on start with os/user.Current (1.7.0 ) May 6, 2019
@zeripath zeripath changed the title Debian 9.6 AMD64: Segfault on start with os/user.Current (1.7.0 ) Debian 9.6 AMD64: Segfault on start at os/user.Current (1.7.0 ) May 6, 2019
@zeripath
Copy link
Contributor

zeripath commented May 6, 2019 •
edited
Loading

Hmm what's the result of:

dpkg -l libc6

mine is: 2.24-11 deb9u3 and on my ubuntu box it's 2.27-3ubuntu, on unstable it's 2.28 or so.

@zeripath
Copy link
Contributor

zeripath commented May 6, 2019

So we're fairly certain that #6862 will fix this. We'll get it backported to the 1.8 line but that won't fix the other releases at present.

@lafriks lafriks added this to the 1.8.1 milestone May 6, 2019
@lafriks lafriks added topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile and removed issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail labels May 6, 2019
@hendrikbl
Copy link
Author

@zeripath @sapk You're all a bunch of geniuses! The new 1.8 build works perfectly. Thank you for solving this in only one day :)

@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile
Projects
None yet
Milestone
1.8.1
Development

Successfully merging a pull request may close this issue.

Enforce osusergo build tag for releases sapk-fork/gitea
5 participants
@lunny @lafriks @zeripath @sapk @hendrikbl