Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Docker] [macOS] moby/vpnkit exhausts docker resources #43

Open
yiwangwuqian opened this issue Feb 18, 2019 · 2 comments
Open

[Docker] [macOS] moby/vpnkit exhausts docker resources #43

yiwangwuqian opened this issue Feb 18, 2019 · 2 comments
Labels
Doesn't Belong Here Issue doesn't relate to proxy server or is invalid macOS Issues found only on macOS

Comments

@yiwangwuqian
Copy link

yiwangwuqian commented Feb 18, 2019 •
edited by abhinavsingh
Loading

In macOS 10.14.2, run with docker

2019-02-18 10:16:35,884 - ERROR - run:604 - Exception while handling connection <socket.socket fd=1024, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.17.0.2', 8899), raddr=('172.17.0.1', 50078)> with reason ValueError('filedescriptor out of range in select()')
Traceback (most recent call last):
  File "./proxy.py", line 600, in run
    self._process()
  File "./proxy.py", line 575, in _process
    r, w, x = select.select(rlist, wlist, xlist, 1)
ValueError: filedescriptor out of range in select()
@abhinavsingh abhinavsingh added the Bug Bug report in proxy server label Feb 21, 2019
@abhinavsingh
Copy link
Owner

I did run into a similar issue during development. This seems to be a MacOS specific issue, that too when proxy.py is set up as a system-wide MacOS HTTP/HTTPS proxy.

Example:

  1. Start the docker container
  2. From command line stress test proxy.py, example:
    hey -x http://127.0.0.1:8899 -c 10 -n 100 http://httpbin.org/get
    See https://github.com/rakyll/hey to setup hey or
    one can also use apache benchmark ab
  3. Check open file descriptors by proxy.py in container:
    $ docker exec -it <container-id> /bin/sh
    /app # lsof -p 1 | wc -l
    53
  4. Now update MacOS system proxy settings to use proxy.py. Check open file descriptors again. There are thousands of socket connection open in CLOSE_WAIT state, within a few seconds.

My initial investigation hinted that client connection somehow gets dropped/closed right after proxy.py has received the request. Unsure if the connection is closed by the client or it has to do with docker network settings. Anyways, this results in clients making calls for the same URL repeatedly in a tight loop, overwhelming proxy.py and eventually running into file descriptor out of range error. You will notice access log for same host:port combination again and again.

2019-02-21 02:33:47,089 - INFO - 9:access_log:695 - 172.17.0.1:47872 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,089 - INFO - 8:access_log:695 - 172.17.0.1:47864 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,090 - INFO - 9:access_log:695 - 172.17.0.1:47856 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,091 - INFO - 8:access_log:695 - 172.17.0.1:47850 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,092 - INFO - 8:access_log:695 - 172.17.0.1:47834 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,093 - INFO - 9:access_log:695 - 172.17.0.1:47840 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,095 - INFO - 9:access_log:695 - 172.17.0.1:47826 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,095 - INFO - 8:access_log:695 - 172.17.0.1:47820 - b'CONNECT' b'17.249.9.246':443

In above case, IP 17.249.9.246 belongs to Apple. Sometimes destination IP may belong to Google or other service providers running on your system.

Client IP 172.17.0.1 is docker bridge IP address:

$ docker network inspect bridge | grep -i gateway
                    "Gateway": "172.17.0.1"

@abhinavsingh abhinavsingh mentioned this issue Feb 22, 2019
Open
@yiwangwuqian
Copy link
Author

I don't know how to solve this,but thanks for your reply!

@abhinavsingh abhinavsingh self-assigned this Apr 19, 2019
@abhinavsingh abhinavsingh changed the title filedescriptor out of range in select() filedescriptor out of range in select() while running in Docker Jul 30, 2019
@abhinavsingh abhinavsingh changed the title filedescriptor out of range in select() while running in Docker [Docker] filedescriptor out of range in select() Sep 17, 2019
@abhinavsingh abhinavsingh added the macOS Issues found only on macOS label Sep 23, 2019
@abhinavsingh abhinavsingh changed the title [Docker] filedescriptor out of range in select() [Docker] [macOS] filedescriptor out of range in select() Sep 23, 2019
@abhinavsingh abhinavsingh changed the title [Docker] [macOS] filedescriptor out of range in select() [Docker] [macOS] moby/vpnkit exhausts docker resources Sep 24, 2019
@abhinavsingh abhinavsingh added Doesn't Belong Here Issue doesn't relate to proxy server or is invalid Awaiting Response Waiting for more information / response from issue creator and removed Bug Bug report in proxy server labels Sep 25, 2019
@abhinavsingh abhinavsingh removed their assignment Nov 8, 2019
@abhinavsingh abhinavsingh removed the Awaiting Response Waiting for more information / response from issue creator label Nov 8, 2021
@abhinavsingh abhinavsingh self-assigned this Nov 10, 2021
@abhinavsingh abhinavsingh removed their assignment Jan 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Doesn't Belong Here Issue doesn't relate to proxy server or is invalid macOS Issues found only on macOS
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@abhinavsingh @yiwangwuqian