Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"too many open files" error #6

Closed
jgoodall opened this issue Jan 10, 2014 · 4 comments
Closed

"too many open files" error #6

jgoodall opened this issue Jan 10, 2014 · 4 comments
Labels

Comments

@jgoodall
Copy link

When I run a reflex command within my $GOPATH, I get a bunch of errors related to "too many open files". Here is an example:

[info] Error while watching new path src/code.google.com/p/go.net/websocket: open src/code.google.com/p/go.net/websocket: too many open files

I am on Mac OS X 10.9. I installed with go get (and then moved the binary into my ~/bin). I can run reflex in other directories that only have a few files. Here is the reflex command I am running:

reflex -g '*.go' -- go test github.com/jgoodall/test

It actually doesnt seem to matter what the command is, I get the same errors.

@cespare
Copy link
Owner

cespare commented Jan 10, 2014

Hi John! Thanks for your error report.

First I'll describe what the issue is, but skip to the end if you just want a workaround.

Reflex uses fsnotify, a cross-platform library for watching files. fsnotify does not provide a way to recursively watch directories (see howeyc/fsnotify#56), so in reflex I implemented a recursive watching mechanism. This has to traverse the whole tree underneath the level where you ask reflex to watch and recursively watch every directory in that tree for changes. If you're watching your whole $GOPATH, that's probably a lot of directories -- my machines all have several thousand.

You can check like this: find $GOPATH -type d | wc -l.

Mac OS X, by default, limits you tightly on the number of files you are allowed to open at once -- on my Mountain Lion MBP it's 2560. You can check with ulimit -a and look where it says "-n: file descriptors". For me, I was able to set it 10000 using ulimit -n 10000, and this made me able to watch my entire $GOPATH with reflex. (From reading about it here, here, and here, you might have to use launchctl. Google "mac os ulimit" and similar for more information.)

Doing this could possibly slow your system down and is basically a hacky workaround. There is a better way. On BSD-flavored systems, including Mac OS X, fsnotify uses kqueue. kqueue does not provide a way to efficiently watching directory trees. OS X does provide such a facility: it's FSEvents. There is an open issue at howeyc/fsnotify#54 to implement FSEvents support in fsnotify. Better, there is an initiative to include something like this package as os/notify right in the Go standard library for the next release, and some of the Go developers have shown interest in implementing recursive watches, possibly with FSEvents on Mac OS X, as part of that. Expect this to land within the next few months.

Finally -- is there a reason you need to watch your entire $GOPATH? Typically I run reflex in my project directory.

To summarize:

Solutions

  1. Run reflex in your project directory so it has fewer files to watch
  2. Use ulimit or launchctl to raise the number of open files you are allowed
  3. Wait several months for an FSEvents implementation to present itself

(Side note: you can watch the same issue for fsnotify: howeyc/fsnotify#35)

@jgoodall
Copy link
Author

Ok, that makes sense. #1 works for me and that is what I am doing, but wanted to run it from my 'root' dir.

It would be nice if I could run it from one directory (e.g. GOPATH) but only have it watch for files in another directory, using a --directory "path/to/dir" option.

@cespare
Copy link
Owner

cespare commented Sep 17, 2014

This issue has been significantly improved with the introduction of the --inverse-regex (-R) flag, which allows for ignoring files and directories.

Unfortunately, fsnotify is not progressing as quickly as anticipated and it doesn't look like an FSEvents implementation is likely to land soon. So for now, we're stuck with the current situation.

I've added a section to the README that lays out the options for dealing with this problem. I think that will have to do for now.

@cespare cespare closed this as completed Sep 17, 2014
@danawoodman
Copy link

-R ".git" was the fix for me 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants