Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG][Compiler] Benchmark.Report does not calculate #3273

Closed
DWSimmons opened this issue Jul 19, 2024 · 1 comment
Closed

[BUG][Compiler] Benchmark.Report does not calculate #3273

DWSimmons opened this issue Jul 19, 2024 · 1 comment
Assignees
@parallelistix
Labels
bug Something isn't working mojo-repo Tag all issues with this label

Comments

@DWSimmons
Copy link

Bug description

Docs for Benchmark Report say that the Benchmark full report that
min : The batch of benchmarks that was the fastest to run.
max: The batch of benchmarks that was the slowest to run.
mean: The average duration of all benchmark runs.
iters: The total benchmark iterations.

but it only grabs the last batch and reports whatever last batch is

---------------------
Benchmark Report (s)
---------------------
Mean: 0.010431665711000002
Total: 10.431665711000001
Iters: 1000
Warmup Mean: 0.010488116
Warmup Total: 0.010488116
Warmup Iters: 1
Fastest Mean: 0.010431665711
Slowest Mean: 0.010431665711

Batch: 1
Iterations: 10
Mean: 0.010434411400000001
Duration: 0.104344114

Batch: 2
Iterations: 100
Mean: 0.0104227856
Duration: 1.04227856

Batch: 3
Iterations: 1000
Mean: 0.010431665711
Duration: 10.431665711000001
import benchmark
from time import sleep

fn sleeper():
    sleep(0.01)

fn main():
    var result = benchmark.run[sleeper](1,2000,10)
    result.print_full()

Steps to reproduce

  • Include anything else that might help us debug the issue.

System information

- What OS did you do install Mojo on ?
Ubuntu 22.04
- Provide version information for Mojo by pasting the output of `mojo -v`
Tried on nightly/mojo 24.04.1715 and on main 24.4.0 & 24.3.0: All had the above results
- Provide Modular CLI version by pasting the output of `modular -v`
modular 0.8
@DWSimmons DWSimmons added bug Something isn't working mojo-repo Tag all issues with this label labels Jul 19, 2024
@parallelistix parallelistix self-assigned this Jul 20, 2024
@JoeLoser
Copy link
Collaborator

JoeLoser commented Jul 24, 2024 •
edited
Loading

Hey @DWSimmons, thanks for opening the issue. @parallelistix responded internally, but here's a copy of his response since it wasn't originally part of the synced thread for you to see.

I don't think this is a bug, actually a feature. The current implementation of benchmark is such that it only considers the most significant measurements, that is:

  • if a max_batch_size is greater than 0, the number of iterations should have exceeded the batch_size AND the elapsed time should exceed min_runtime
  • the batch is within the last 10% of the total number of batches.

Currently, the benchmark module is not yet open sourced for you to check it out yourself.

Note that if the max_batch_size is not specified, then each batch will execute 10x the number of iterations of the previous batch until it reaches the max_time_ns or exceeds max_iters.

@JoeLoser JoeLoser closed this as not planned Won't fix, can't repro, duplicate, stale Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@parallelistix parallelistix

Labels
bug Something isn't working mojo-repo Tag all issues with this label
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JoeLoser @parallelistix @DWSimmons