Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add custom samplers, better collection, and better plotting to sinter #804

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

Strilanc
Copy link
Collaborator

@Strilanc Strilanc commented Jul 27, 2024

  • Add sinter.Sampler and sinter.CompiledSampler classes
    • They can go anywhere a Decoder would go, but they are responsible for all parts of the sampling instead of only prediction
  • Add a new default sampler perfectionist, which discards anything with detection events and predicts the observables are not flipped
  • Improved layout of the progress printouts when collect is running
  • Sinter decoders can now flag that they want to discard shots by adding an extra byte to the returned observable data, with 0 meaning keep and not-0 meaning discard
  • Change how sinter collect distributes work
    • Workers are now distributed as widely as possible, instead of all on one task
    • Workers are now never switched between tasks until their current task is done
  • Add sinter plot --point_label_func argument for drawing text next to data points
  • Augment sinter plot --group_func to support dictionaries with special keys controlling precise grouping behaviors
    • If group_func returns a dict with a "color" key, all items with the same "color" value are drawn with the same color
    • If group_func returns a dict with a "linestyle" key, all items with the same "linestyle" value are drawn with the same linestyle
    • If group_func returns a dict with a "marker" key, all items with the same "marker" value are drawn with the same marker
    • If group_func returns a dict with a "label" key, this forces the label shown in the legend
    • If group_func returns a dict with an "order" key, this takes priority for ordering the legend
  • sinter collect --processes is no longer required (defaults to "auto")
  • sinter plot --show is no longer required (defaults to showing, unless --out is specified, unless --show is specified)
  • Group some of sinter's code into private subpackages
  • Show traditional error bars instead of a filled region for high/low fit when only one data point is present
  • Add sinter plot --preprocess_stats_func
  • Add sinter.TaskStats.with_edits
  • Add safety error when adding stats that have equal strong ids but differing identifying information (json_metadata or decoder)

Some of the sampler design is adapted from @inmzhang's design in #735

Fixes #774

Fixes #682

Fixes #392

- Improve output printing to be in a table
- Change work distribution strategy to go wide instead of deep, and to avoid retargeting workers
- Added key property support to `sinter plot --group_func`
    - Activate this functionality by evaluating to a dict with certain keys
    - The "color" key controls color grouping
    - The "marker" key controls marker grouping
    - The "linestyle" key controls linestyle grouping
    - The "label" key controls what's shown in the legend
    - The "order" key can be used to force sorting order
@Strilanc Strilanc mentioned this pull request Jul 27, 2024
@Strilanc Strilanc marked this pull request as draft July 27, 2024 05:40
@Strilanc Strilanc marked this pull request as ready for review July 30, 2024 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants