Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add benchmarks #38

Open
hjr3 opened this issue Aug 19, 2023 · 3 comments
Open

add benchmarks #38

hjr3 opened this issue Aug 19, 2023 · 3 comments

Comments

@hjr3
Copy link
Owner

hjr3 commented Aug 19, 2023 •
edited
Loading

need the following benchmarks:

  • intermittent failures - lots of completes and a few failures at random
  • outage - service is online, goes offline allowing a bunch of requests to build up and then comes back online
    • make sure queries, etc are tuned to lots of rows in requests and attempts tables
    • make sure we do not stampede the origin
@hjr3
Copy link
Owner Author

hjr3 commented Sep 15, 2023 •
edited
Loading

I used cargo bench --bench soldr-benchmark -- --profile-time 120 to generate a bunch of test data. Note: Make sure sqlite:soldr.db?mode=rwc is the database url

With ~100,000 requests and attempts (many of them failures)

$ sqlite3 soldr.db
SQLite version 3.37.0 2021-12-09 01:34:53
Enter ".help" for usage hints.
sqlite> SELECT COUNT(*) FROM attempts;
105683
sqlite> SELECT COUNT(*) FROM requests;
105683

Using sqlite:soldr.db?mode=rwc as the database url. These are the benchmark results:

proxy/empty post        time:   [56.520 µs 56.785 µs 57.136 µs]
                        change: [-4.3358% -2.1843% -0.0923%] (p = 0.05 < 0.05)
                        Change within noise threshold.
Found 16 outliers among 100 measurements (16.00%)
  7 (7.00%) high mild
  9 (9.00%) high severe
proxy/proxy empty post  time:   [2.5492 ms 2.5761 ms 2.6079 ms]
                        change: [-13.892% -9.5677% -5.4287%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe

proxy/success           time:   [57.422 µs 58.070 µs 58.929 µs]
                        change: [-46.078% -43.568% -40.931%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe
proxy/outage            time:   [2.5845 ms 2.6058 ms 2.6315 ms]
                        change: [-28.994% -26.053% -23.000%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe
proxy/success after outage
                        time:   [2.5512 ms 2.5818 ms 2.6209 ms]
                        change: [-8.7676% -6.8602% -4.9320%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe

Baseline performance for a request to axum is ~56 µs. The proxy, during at outage, was ~2.6 ms. The delay is mostly due to writing to the SQLite database.

Using sqlite::memory: as the database url. These are the benchmark results:

proxy/empty post        time:   [58.208 µs 59.038 µs 60.066 µs]
                        change: [ 2.1283%  4.2150%  6.1959%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  7 (7.00%) high mild
  2 (2.00%) high severe
proxy/proxy empty post  time:   [281.37 µs 283.89 µs 287.29 µs]
                        change: [-2.2399%  1.1427%  5.3630%] (p = 0.63 > 0.05)
                        No change in performance detected.
Found 16 outliers among 100 measurements (16.00%)
  5 (5.00%) high mild
  11 (11.00%) high severe

proxy/success           time:   [56.381 µs 56.983 µs 57.744 µs]
                        change: [-1.8240% -0.6719%  0.4316%] (p = 0.27 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe
proxy/outage            time:   [310.69 µs 312.85 µs 315.24 µs]
                        change: [-2.0567% -0.4123%  1.1969%] (p = 0.62 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  7 (7.00%) high mild
proxy/success after outage
                        time:   [285.31 µs 288.09 µs 291.91 µs]
                        change: [ 0.0352%  1.9166%  3.9046%] (p = 0.05 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

Baseline performance for a request to axum is ~59 µs (which is a 3 µs difference from the benchmark above). The proxy, during at outage, was ~313 µs. This suggests there is ~250 µs of overhead using the proxy outside of the SQLite overhead.

Note: this was on my laptop, so things may be different on a dedicated linux server. Also, proxy server and origin server were located on the same host.

@hjr3
Copy link
Owner Author

hjr3 commented Sep 16, 2023

A lot of this is fixed in #46

@hjr3
Copy link
Owner Author

hjr3 commented Sep 16, 2023

Regarding stampede, this client side throttling strategy is interesting

https://cs.stackexchange.com/questions/43149/algorithm-for-dynamic-client-side-throttling

https://www.cmcmarkets.com/en/trading-guides/exponential-moving-average

How to calculate the EMA

In order to learn how to calculate the exponential moving average, the simple moving average should be calculated first to get the initial EMA value. This will then lead you to finding the exponential moving average equation.

To find the simple moving average, you must find the average number of the past data points, which are often past closing prices. If you were seeking a security’s 50-day SMA, the closing prices of the past 50 days would be added together, then divided by 50.

Secondly, calculate the the weighting multiplier (or smoothing constant). This happens by dividing 2 by the number of time periods, plus 1.

Thirdly, calculate the EMA for each day between the initial EMA value and the current day, using the price, the multiplier and the EMA value of the previous time period.

Exponential moving average formula

  • SMA = (N – period sum) ÷ N
  • The weighting multiplier (or smoothing constant) = 2 ÷ (time period 1)
  • EMA = (closing price – previous day’s EMA) x weighting multiplier previous day’s EMA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hjr3