This repository contains the official implementation of AutoPSV: Automated Process-Supervised Verifier, accepted at NeurIPS 2024 (poster).

We will release the code and corresponding finetuned process-enhanced verifier in the near future. Please note that certain portions of the codebase are currently withheld due to confidentiality requirements. We are working to ensure full compliance with open-access requirements before the complete release.

The AutoPSV framework consists of four key components:

1. Process-Outcome Verifier:

   • Automatically generates process annotations for each reasoning step
   • Monitors confidence variations to evaluate reasoning validity
   • Operates independently of ground truth annotations

2. Automated Annotation Generation:

   • Efficiently produces process-level supervision signals
   • Eliminates the need for costly manual annotations
   • Enables scalable training data creation

3. Large Language Model Training:

   • Utilizes generated annotations for targeted supervision
   • Enhances model comprehension of reasoning processes
   • Improves output quality through structured learning

4. Iterative Refinement:

   • Implements continuous improvement through feedback loops
   • Updates verifier capabilities based on LLM performance
   • Maintains dynamic adaptation to evolving model outputs

AutoPSV/
├── data_annotation.py
├── train.py
└── utils/
    ├── process_verifier_models.py
    ├── states.py
    └── verifier_datasets.py

1. data_annotation.py

   • Implements automated process labeling
   • Identifies process calculation hallucinations
   • Streamlines data annotation workflows

2. train.py

   • Manages verifier model training
   • Handles data ingestion and model initialization
   • Implements optimization for process- and outcome-supervised verifiers

The utils/ directory contains essential supporting modules:

• process_verifier_models.py: Implementation of core verifier architectures
• states.py: State management and related functionality
• verifier_datasets.py: Dataset handling for model training and validation

Install required dependencies using:

pip install -r requirements.txt

We welcome contributions that align with our project goals. Please submit issues or pull requests following our contribution guidelines.

This work is licensed under CC BY 4.0 (Creative Commons Attribution 4.0 International License).

If you find this work useful in your research, please cite our paper:

@inproceedings{lu2024autopsv,
  title={AutoPSV: Automated Process-Supervised Verifier},
  author={Lu, Jianqiao and Dou, Zhiyang and Wang, Hongru and Cao, Zeyu and Dai, Jianbo and Wan, Yingjia and Guo, Zhijiang},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}