Add load test #3496
Add load test #3496
Conversation
There was a problem hiding this comment.
Thanks a lot @BabyChouSr ! this would be super useful for endpoint stress test
tests/load_test.py
Outdated
| for _ in batch: | ||
| if args.include_image: | ||
| # Generate a random dimension for the image | ||
| y_dimension = np.random.randint(100, 1025) |
There was a problem hiding this comment.
do you think we should use the same size image to measure latency more consistently?
There was a problem hiding this comment.
This is a good point but it also depends on the goal of our analysis. One thing to think about is the Phi-3.5 bug which was due to the model not being able to handle a certain resolution, which is why randomizing the input and feeding it in could be useful. But yes, for better determinism so we can compare across models, we should make this constant.
There was a problem hiding this comment.
great point.. maybe we can add an argument say --random-size then we randomize the input size
Load test to stress a certain server address with requests and test many different image configurations if necessary.