Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add runtests subcommand #2952

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Add runtests subcommand #2952

wants to merge 4 commits into from

Conversation

shoo
Copy link
Contributor

@shoo shoo commented Jul 21, 2024

As with many official tools, projects often require integration testing for CI/CD. In such cases, it is useful to run integration tests in a unified format.
The runtests subcommand I added is similar to test/run-unittest.sh or test/run-unittest.d so integration tests of dub projects can be run dub itself.
To test dub, run the following command:

dub run -- runtests --skip-build

Copy link

github-actions bot commented Jul 21, 2024

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 8

Total warnings: 0

Build statistics:

 statistics (-before,  after)
-executable size=5293368 bin/dub
-rough build time=62s
 executable size=5364928 bin/dub
 rough build time=63s
Full build output
DUB version 1.38.0, built on Jul  4 2024
LDC - the LLVM D compiler (1.39.0):
  based on DMD v2.109.1 and LLVM 18.1.6
  built with LDC - the LLVM D compiler (1.39.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.39.0/x64/ldc2-1.39.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.39.0-beta.1 commit.9.gc8c3cbb0: building configuration [application]
source/dub/internal/dyaml/composer.d(210,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/composer.d(232,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/composer.d(336,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(196,5): Deprecation: cannot access overlapped field `Event.explicitDocument` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(214,5): Deprecation: cannot access overlapped field `Event.explicitDocument` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(241,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(148,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(148,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
     Linking dub
STAT:statistics (-before,  after)
STAT:executable size=5364928 bin/dub
STAT:rough build time=63s

@rikkimax
Copy link
Contributor

Ideally these two would be merged together by looking for the json/sdl comment at start of file.

- `*.d` : The D language source code is executed by rdmd.
- `*.script.d` : Runs the D language source code as a file as a single-file DUB project.

@shoo
Copy link
Contributor Author

shoo commented Jul 21, 2024

Ideally these two would be merged together by looking for the json/sdl comment at start of file.

- `*.d` : The D language source code is executed by rdmd.
- `*.script.d` : Runs the D language source code as a file as a single-file DUB project.

Although not impossible, only the minimum necessary functions have been implemented in this PR.
This change would be able to be enhanced later without breaking changes.

Here is a list of others that I can think of that would be nice to have but have not yet been implemented:

  • .fail_build / .no_build / .no_test / .no_run / .min_frontend / .testignore / --testdir These could be better configured, for example integrated into the project settings.
  • When running tests, it would be useful to have an environment variable that contains the path to the binary file under test.
  • A way to output the test results in CSV format, etc., for easier handling by other tools would be helpful.
  • It would be useful to have an option to specify the directory where the coverage results are stored.
  • An option to run each test in parallel may be useful to speed up the process.

If you can think of any very important enhancements, glitches, or future enhancements that might require breaking changes, please let me know.

@rikkimax
Copy link
Contributor

Support directory based dub projects.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

@shoo
Copy link
Contributor Author

shoo commented Jul 21, 2024

Support directory based dub projects.

If you are talking about test cases, they are already implemented.
The added test test/runtests/test/test01 is a directory based dub project.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

It is difficult to detect with certainty which directory contains test cases.
For example, dub includes three directories (test, examples, scripts) with similar file characteristics, but it is difficult to mechanically identify which directory is for integration testing. Furthermore, if this were to be automatically deduced, it would be confusing to the user, unless it were deduced with fairly simple logic.

@rikkimax
Copy link
Contributor

Support directory based dub projects.

If you are talking about test cases, they are already implemented. The added test test/runtests/test/test01 is a directory based dub project.

No, I'm talking about test/runtests/dub.sdl.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

It is difficult to detect with certainty which directory contains test cases. For example, dub includes three directories (test, examples, scripts) with similar file characteristics, but it is difficult to mechanically identify which directory is for integration testing. Furthermore, if this were to be automatically deduced, it would be confusing to the user, unless it were deduced with fairly simple logic.

I didn't suggest automatic detection. I suggested a directive in your dub file to specify the directory to use.

@shoo
Copy link
Contributor Author

shoo commented Jul 21, 2024

No, I'm talking about test/runtests/dub.sdl .

More information, please. Are you saying that you want to recognize the directory as being for the dub project without the dub.sdl?

I didn't suggest automatic detection. I suggested a directive in your dub file to specify the directory to use.

I see. However, I considered this but decided not to include it in this PR. I believe that adding dub project configuration directives with a higher priority than the default but lower than command line arguments can be added in a future update, not this time.

@rikkimax
Copy link
Contributor

No, I'm talking about test/runtests/dub.sdl .

More information, please. Are you saying that you want to recognize the directory as being for the dub project without the dub.sdl?

Below the tests directory, below that is either files that you're running. I would like it to also support dub projects at that level.

It could be a misunderstanding on my part, but that changelog doesn't imply it.

@shoo
Copy link
Contributor Author

shoo commented Jul 22, 2024

Below the tests directory, below that is either files that you're running. I would like it to also support dub projects at that level.

It could be a misunderstanding on my part, but that changelog doesn't imply it.

When you run dub runtests at the root of a dub project, it will execute directory based dub projects contained within the test directory. This behavior should align with your expectations. However, in the case of the test/runtests directory, it is excluded from the tests because it contains a .no_build file.
It might be confusing, but the test case I created runs test/runtests.script.d with test/runtests as the current directory. This test executes the test cases in test/runtests/test/* by running the dub runtests command.

@Geod24
Copy link
Member

Geod24 commented Jul 23, 2024

First thing that comes to mind: We already have a special configuration for unittest, why not do that for integration tests ? Why a special command ?

@shoo
Copy link
Contributor Author

shoo commented Jul 23, 2024

First thing that comes to mind: We already have a special configuration for unittest, why not do that for integration tests ? Why a special command ?

The idea of having a special configuration selected when running runtests subcommand might be a good one. By having a directive to replace --testdir, such as allowing --config to switch test directories, we could achieve good synergy.
On the other hand, I still think a special subcommand is necessary. Just as there is a dub test subcommand instead of dub run --config unittest -- --DRT-testmode=test-only .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants