Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance benchmarks #18

Open
Nyholm opened this issue Apr 12, 2020 · 6 comments
Open

Performance benchmarks #18

Nyholm opened this issue Apr 12, 2020 · 6 comments

Comments

@Nyholm
Copy link
Collaborator

Nyholm commented Apr 12, 2020

So I did some testing on a small app. I tested with #17 included (that PR does not have any performance impact).

A normal request: 10ms
First request: 1805ms
First request with BrefKernel: 510ms

BrefKernel spent 88ms preparing cache directory. It spent 77 ms out of those 88ms to copy the "pools" directory.


So thanks so #17, I disabled copying "pools" and enabled opcache.preload.

A normal request: 7.5ms
First request: 1144ms
First request with BrefKernel: 160ms

Yes. The first (cold) request is still 20 times slower. (Or 51x slower without preloading) But it is better than 180x slower as we began with.


Note: These sample data may not be reliable. I only did 2 samples of each. To get better data I should take 5-10 samples and show the average. However, I did deploy over CI so I know the process of warmup etc is the same.

This was referenced Apr 12, 2020
@Nemo64
Copy link
Contributor

Nemo64 commented Apr 12, 2020

I'm confused as to why preloading improves the first request time.

Preload does load files on boot so they are available within the request right? So shouldn't the first request be slower because it loads stuff it does not need? Or is the preload time within the init time and therefore not counted?

@Nyholm
Copy link
Collaborator Author

Nyholm commented Apr 12, 2020

Actually no. Preload loads stuff before the first request. Ie, when PHP-Fpm is starting. That does technically happen before we get the "request data" from AWS.

So my "initialisation" should be a little bit slower when using preload. But that was nothing I noticed.

@jenschude
Copy link

Preload is btw also faster as the init process has more CPU power: https://medium.com/@hichaelmart/shave-99-93-off-your-lambda-bill-with-this-one-weird-trick-33c0acebb2ea

@markomitranic
Copy link

markomitranic commented Apr 13, 2020

First, awesome job man @Nyholm . Thank you.

Just some future prospects thoughts here.

With every day, and especially with bref, we are moving more and more towards pre-packaged PHP releases. Something that is quite unusual in PHP world - we are packaging compiled assets, compiled/installed vendors, soon compiled code, and one day i hope precompiled container itself. We might as well look into how other ecosystems do it and why. I briefly asked Nikita about the theorethical possibility of precompiling the whole thing into a binary release, but he said that apart from some little CPU benefit he did not see any real value.

Back to topic at hand...

There might be a future possibility of compiling and baking the preloaded opcodes as well as symfony cache into the distribution layer.

  • As far as opcodes go, at the moment i do not know of a way to forcibly specify the output location, which would allow us to distribute precompiled code along with the lambda code. Similarly to how we would treat a binary release. But if we look into it, there might be a way to compile on one lambda and bake the whole layer into a new lambda, or something similar. I think the exact same process is called compiler bootstrapping in C-world.

  • As for baking and including a precompiled Symfony container, caches and whatever else comes with it - i did do it on non-bref instances a year ago, and it kind of worked. But it was just a quick local experiment, and i never dwelled deeper into it, to prove that it actually worked and if it had any performance benefit. I hope @Nyholm could tell us more about the symfony side of things, as i beleive that the same principle as above could be employed. Just run the first request on a lambda, copy the whole container and caches, and make a new lambda. Unless there are some drawbacks or blockers i do not know about.

@mnapoli
Copy link
Member

mnapoli commented Apr 13, 2020

@markomitranic what I usually do is compile the Symfony cache in a Bref docker image -> the paths are all good then.

@Nemo64
Copy link
Contributor

Nemo64 commented Apr 13, 2020

Out of curiosity I benched my project too with different php versions and preload on/off.

I now used the configuration i mentioned in #21 (comment) even though that is irrelevant for my test request since it uses no validators or property-info. It is therefore similar to @Nyholm's test since his test had not copied the pools directory (what this library currently mainly does)

I always ran 2 requests to check for cold start and warm run.
I didn't repeat runs but the performance in lambda is really consistent so I don't worry too much about it.
The page I requested is a login page so there is a session to dynamodb in there and symfony form is loaded including lots of very small twig templates.

php 7.3

REPORT RequestId: 7cc167fb-5d41-4d59-b301-baf1358b63df	Duration: 1174.78 ms	Billed Duration: 1600 ms	Memory Size: 1024 MB	Max Memory Used: 120 MB	Init Duration: 337.25 ms	
REPORT RequestId: 320c114e-d88a-49b4-9f6c-237ca224eeab	Duration: 56.56 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 120 MB	

php 7.4

REPORT RequestId: b4c1a445-fc15-46d3-a73a-cdb93e5372cd	Duration: 1072.20 ms	Billed Duration: 1500 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	Init Duration: 332.41 ms	
REPORT RequestId: a8473aa9-0f1d-4800-872d-f9d22f55f9b7	Duration: 56.21 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	

php 7.4 with container.dumper.inline_class_loader set to true because it defaults to false in php 7.4 even tough it is needed for the preload feature. It is true by default in 7.3.

REPORT RequestId: 7d9e1292-dadd-4e01-a99f-2e9ce2424c0f	Duration: 1094.20 ms	Billed Duration: 1500 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	Init Duration: 336.00 ms	
REPORT RequestId: 3e55912c-40ea-4958-a4f3-995ed54963ab	Duration: 49.77 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	

php 7.4 with preload configured like opcache.preload=${LAMBDA_TASK_ROOT}/var/cache/prod/srcApp_KernelProdContainer.preload.php

REPORT RequestId: 363c7125-1255-4362-8239-99d5c2865e02	Duration: 609.20 ms	Billed Duration: 1200 ms	Memory Size: 1024 MB	Max Memory Used: 131 MB	Init Duration: 570.07 ms	
REPORT RequestId: cf862c09-ddd9-49f4-8057-a740566138de	Duration: 59.27 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 131 MB	

So php 7.4 is faster in any way. Although you should enable the inline_class_loader option which defaults to false starting with 7.4.
The preload feature does improve cold start time from 1430.20 ms to 1179.27 ms (duration init time) but it seams to slightly decrease performance for warm requests.

Not this is for symfony 4.4 which is the first version to include the preload feature. There are improvements in 5.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants