Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT Reporting feature for Pyrit #384

Open
wadhwasahil opened this issue Sep 19, 2024 · 1 comment
Open

FEAT Reporting feature for Pyrit #384

wadhwasahil opened this issue Sep 19, 2024 · 1 comment

Comments

@wadhwasahil
Copy link

Is your feature request related to a problem? Please describe.

I am looking for a way to have a report generated for all the attacks I generate using different Orchestrators.

Describe the solution you'd like

A report either a csv etc. that gives me insights of what attacks were successful and what were not. Also, attacks breakdown would certainly help that would inform me about the different areas of LLM breakdowns.

Tldr:

I am looking for a module that could generate a single/multiple reports based on different attack strategies.
@wadhwasahil wadhwasahil changed the title Reporting feature for Pyrit FEAT Reporting feature for Pyrit Sep 19, 2024
@romanlutz
Copy link
Contributor

Yes, this is some time we think about a lot. The tricky part is making it generic with a CSV.

The raw conversations:
Multi-turn/multi-piece conversations need to be grouped together, in order, and with one row per piece, probably followed by all related scores (?) or alternatively scores on the same line as the corresponding piece.

Aggregates:
Then there would be a set of summary stats by orchestrator, by harm category, etc.
I could imagine a set of plots to be added as well.

Notably, all of this would be pulled from the DB, so some filters (by time period, by other metadata, by user, etc) would be useful.

If anyone is interested in contributing any of these parts please reach out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wadhwasahil @romanlutz