Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add every log warning emitted during reductions to header as COMMENT #133

Open
ndrory opened this issue Aug 8, 2024 · 2 comments
Open

Comments

@ndrory
Copy link
Contributor

ndrory commented Aug 8, 2024

Title says it all, add every log warning emitted during reductions to header as COMMENT so that we can filter by what happened. Additionally, we will add a keyword to the headers for each step in the reductions that at least reads 'SUCCESS', 'WARNING', 'FAIL'.

@kslong
Copy link
Contributor

kslong commented Aug 8, 2024 via email

@havok2063
Copy link
Collaborator

havok2063 commented Aug 9, 2024

I wouldn't recommend appending every log warning message into the headers. This could result in the headers being hundreds of fields long, filled with log comments. I would recommend using a single bitfield to capture which reduction steps a file has successfully made it through, maybe re-purposing these

class ReductionStage(BaseBitmask):
.

Alternatively, the DRP reduction log files are output in JSON format as well as text format. That makes the logs machine-readable and searchable. One could write a simple tool or script that could parse and extract information from the json logs.

In my opinion, the status and quality information should be captured with something like the following header keys, which would propagate into the drpall file. This has the advantage of keeping the headers light and readable, but still tracking the details.

  • QUALITY - a bitfield tracking the output of Dmitry's QA
  • DRPQUAL - a bitfield tracking errors and flags raised in the reductions
  • STAGE - a bitfield tracking which reduction stage of the pipeline was last successful
  • STATUS - a single, final, string of reduction status, indicating "SUCCESS" or "FAIL" (possibly redundant)
  • SCIENCEQUAL - a single, final, string quality of the overall quality, "GOOD" or "BAD" (possibly redundant)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants