Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch fixes #959 and introduces API throttling in an alpha state for the purposes of code/workflow review. This feature depends on etcd v3 for persistent, external storage -- responsible for tracking active connections and associating them with sustained and burst TTL leases.
TODO
etcd
This section reviews how to install, configure, and use etcd.
Install etcd
To install
etcd
andetcdctl
type:Configure etcd
Please note that in a terminal in which
etcd
oretcdctl
may be executed, the following environment variable should be set:The above environment variable instructs both the etcd server and client to use v3 of its API.
Start etcd server
Once the server binary is built, start it by executing:
Use etcd client
Finally, the
etcdctl
program can be used to control etcd from the command line. Open a new terminal and type:Basic workflow
The basic workflow for how throttling is implemented in REX-Ray is as follows:
ETCD_ENDPOINTS
defined then proceed.Without ever trying to obtain a lock for the the throttling domain's mutex the code first checks to see if the connection count is already too high. If so, none of the heavy work is required. If it's passable then the mutex is retrieved and a lock attempt is made. This way it fails fast and adds a slight penalty only to proceed
Throttling Domains
Each libStorage service is associated with a throttling domain. By default the name of that domain is the name of the libStorage service. However, it's possible for libStorage services to define a custom throttling domain. For example:
In the above configuration the services
vfs1
andvfs2
both belong to thevfs
throttling domain.Testing this PR
Once etcd is installed and running, simply build REX-Ray using this PR and then execute then export the following environment variable:
export ETCD_ENDPOINTS=127.0.0.1:2379
Please note that the above value can be represented in a REX-Ray configuration file:
Now all libStorage requests will be subject to throttling. Currently the burst and sustained values are hardcoded as such:
Printing throttling stats
To list the active throttling information about REX-Ray, use
etcdctl
:The key structure is as follows:
/rexray/throttling
is the prefixrexray.throttling.domain
to specify a custom throttling domain.cnxns
"sub-dir". It's a "sub-dir" because/rexray/throttling/vfs/sustained/lease
also exists and is the ID of the lease associated with this throttle type.cnxns
are the transaction IDs associated with requests.