-
Notifications
You must be signed in to change notification settings - Fork 32
Regression testing
Josh Soref edited this page Nov 1, 2024
·
4 revisions
Right now, most check-spelling testing is done through live use of the action.
- Create a repository check-spelling-sandbox/autotest-check-spelling
- Give it a secret so that it can push to its own repository
- Create a workflow which pushes the current commits from check-spelling and check-spelling-sandbox for
main
andprerelease
to newly https://github.com/check-spelling-sandbox/autotest-check-spelling/blob/main/.github/workflows/autotest.yml
This enables at least catching bugs involving dates changing (there was one fixed for v0.0.24).
- Create a repository
check-spelling/regression-tests
- Create a template of commits
- Create a workflow which on each push applies each of the templated commits filling in the targeted revision and pushing them to
check-spelling/regression-tests
- Ideally have the workflow retrieve the results of each of the commits' actions and compare them against expected results. -- It should be possible to do this using either additional steps in the workflow job or using another job in the workflow or by using workflow call with a final workflow.
- Create a
check-spelling-test
org. - Create a PAT for a bot user (possibly the existing @check-spelling-bot).
- Create a secret in the regression testing repository for that PAT.
- Use
gh
in workflows to create a brand new repository for each thing it tests.- Create the repository
- Insert the workflow(s)
- Create pull requests
- Add comments
- Check on how things are feeling.
- It could probably even have a second PAT which just has enough permission to poke the original repository to trigger data collection from the events in the test repository.
Date | Phase |
---|---|
2024-11-01 | Phase 0 check-spelling-sandbox/autotest-check-spelling |
Uncertain |
Plan 1 check-spelling/regression-tests
|
... |