Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error.name missing when errors are serialized when running tests in parallel - Causes test results to be incorrectly reported in allure-mocha #4994

Closed
4 tasks done
jamesmortensen opened this issue Jul 8, 2023 · 3 comments
Closed
4 tasks done

error.name missing when errors are serialized when running tests in parallel - Causes test results to be incorrectly reported in allure-mocha #4994

jamesmortensen opened this issue Jul 8, 2023 · 3 comments
Labels
stale this has been inactive for a while... status: waiting for author waiting on response from OP - more information needed

Comments

@jamesmortensen
Copy link

Prerequisites

  • Checked that your issue hasn't already been filed by cross-referencing issues with the faq label
  • Checked next-gen ES issues and syntax problems by using the same environment and/or transpiler configuration without Mocha to ensure it isn't just a feature that actually isn't supported in the environment in question or a bug in your code.
  • 'Smoke tested' the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, your usage of Mocha, or Mocha itself
  • Ensured that there is no discrepancy between the locally and globally installed versions of Mocha. You can find them with: node_modules/.bin/mocha --version(Local) and mocha --version(Global). We recommend that you not install Mocha globally.

Description

Some reporters use the type of error to determine how to treat a test that fails. Allure is one example. If a test fails due to an AssertionError, then the test is marked in red, as failed. All other errors are treated as broken tests. These results are marked in yellow, as broken.

When running mocha in parallel, the error objects are serialized and then later deserialized. During the serialization process, we are preserving the error.message but not the error.name. Because of this, it's impossible for Allure to determine the type of error by using the error.name property.

Steps to Reproduce

To Reproduce
Steps to reproduce the behavior:

  1. Clone this repo: https://github.com/jamesmortensen/allure-js-reporter-issues#when-run-in-parallel-mode-with-hooks-hookerror-is-not-a-function and follow the instructions to run the tests and generate the report.

The example has us run tests in sequential mode as well as parallel mode. The README also has screenshots which show the issue.

Expected behavior

  • For AssertionErrors, results should appear in the report in RED.

Actual behavior

  • In parallel mode, they're reported in YELLOW.

Screenshots

Reproduces how often:

Reproduces 100% of the time.

Versions

  • The output of mocha --version and node_modules/.bin/mocha --version: 10.2.0
  • The output of node --version: 18.3.0
  • Your operating system
    • name and version: macOS Ventura 13.3.1
    • architecture (32 or 64-bit): arm64
  • Your shell (e.g., bash, zsh, PowerShell, cmd): bash
  • Your browser and version (if running browser tests): N/A
  • Any third-party Mocha-related modules (and their versions): allure-mocha v2.4.0
  • Any code transpiler (e.g., TypeScript, CoffeeScript, Babel) being used (and its version): N/A

Additional information:

Issue originally reported here: allure-framework/allure-js#485 but I traced the root cause to Mocha.
jamesmortensen added a commit to jamesmortensen/mocha that referenced this issue Jul 8, 2023
@jamesmortensen
fixes mochajs#4994 - Preserves error.name during serialization proces…
1c59fac 
…s so type of error can be used by reporters who differentiate between failed tests and broken tests, such as allure-mocha
@JoshuaKGoldberg
Copy link
Member

Hey thanks for such a detailed report PR! I'm having a hard time reproducing this locally. Do you have a reproduction that doesn't involve installing the Java runtime?

~/repos/allure-js-reporter-issues $ npm run open-report 

> [email protected] open-report
> allure generate --clean && npx allure open

The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

~/repos/allure-js-reporter-issues $ npx allure-generate clean && npx allure open
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/allure-generate - Not found
npm ERR! 404 
npm ERR! 404  'allure-generate@*' is not in this registry.
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

#4995 looks like a good fix but it's hard to say without a more isolated reproduction.

@JoshuaKGoldberg JoshuaKGoldberg added status: waiting for author waiting on response from OP - more information needed and removed unconfirmed-bug labels Jan 21, 2024
@JoshuaKGoldberg JoshuaKGoldberg mentioned this issue Mar 4, 2024
Draft
@JoshuaKGoldberg
Copy link
Member

👋 ping @jamesmortensen, is this still on your radar?

@JoshuaKGoldberg JoshuaKGoldberg added the stale this has been inactive for a while... label Aug 14, 2024
@JoshuaKGoldberg
Copy link
Member

It's been over a month since last ping, so closing this out as stale. If you again have time for this and can provide a reproduction, feel free to reopen @jamesmortensen.

Cheers! 🤎

@JoshuaKGoldberg JoshuaKGoldberg closed this as not planned Won't fix, can't repro, duplicate, stale Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale this has been inactive for a while... status: waiting for author waiting on response from OP - more information needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jamesmortensen @JoshuaKGoldberg