For example_compiler_gym_service/demo_without_bazel.py, I wrote a simple "smoke test" that runs the main entry point:

I believes this should be run by the CI's example test jobs. Would this cover your needs for the other files?

Cheers,
Chris

Codecov Report

Merging #695 (75d0968) into development (9ad5fbb) will decrease coverage by 0.08%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           development     #695       /-   ##
===============================================
- Coverage        88.64%   88.55%   -0.09%     
===============================================
  Files              131      131              
  Lines             7926     7926              
===============================================
- Hits              7026     7019       -7     
- Misses             900      907        7     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ad5fbb...75d0968. Read the comment docs.

I believes this should be run by the CI's example test jobs. Would this cover your needs for the other files?

OK. Got it. I will update the CI accordingly...
How can we ensure that the new tests I create will only run in the cmake job?

The point is that the other 2 tests (loop_unroller and opt_loops) require building some executables. I have tried in this CI adding the other tests: #642 but I found myself opening a can of worms trying to get those test files working for different jobs that use bazel, CMake, or pip install.

If I can get those examples to work with CMake only, and then look into getting them to work in other jobs that would be great.

The point is that the other 2 tests (loop_unroller and opt_loops) require building some executables.

Ah, sorry, I see now.

If I can get those examples to work with CMake only, and then look into getting them to work in other jobs that would be great.

I think you need to use CMake's testing ctest program to run the tests:

https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing With CMake and CTest.html

TBH I'm not super familiar with it, but you could maybe write a test job that depends on the cmake build and runs ctest on the examples dir?

Cheers,
Chris

So far, this is working:

• running the example_unrolling_service and loop_optimizations_service python examples inside the build-linux-cmake job
• uploading the loop_unroller and opt_loops binaries inside the build-linux-cmake job
• downloading the loop_unroller and opt_loops binaries inside the test-linux-cmake job

However, this is not working:

• running the example_unrolling_service and loop_optimizations_service python examples inside the test-linux-cmake job
  It returns an error saying that it can't find the loop_unroller binary, although I made sure it's moved to its correct directory.

OK. I just noticed that about GitHub's upload/download artifcats: https://github.com/actions/download-artifact#permission-loss:

❗ File permissions are not maintained during artifact upload ❗ 

So now I have chmod on the binaries after downloading them. Hopefully this time it will work.

Good progress @mostafaelhoushi!

@mostafaelhoushi, these shouldn't be necessary:

Do you see the UI button to re-run jobs?

Do you see the UI button to re-run jobs?

For some reason that "re-run jobs" button doesn't always appear to me. Most of the time it is not there :(

Oh weird, but sometimes it is? So not a permission issue?

here is what the "details" screen for the current running CI job looks like:

If I were to hit Cancel, I'd have to wait a few seconds/minutes for the job to finish cancelling before the Re-try button shows up.

Cheers,
Chris

here is what the "details" screen for the current running CI job looks like:
If I were to hit Cancel, I'd have to wait a few seconds/minutes for the job to finish cancelling before the Re-try button shows up.

Actually, sometimes I would see the "re-run job" button in that "Details" screen and sometimes I don't.

To make things even more confusing, now when I click on the "Checks" tab of the PR, I do see the "re-run jobs" button, but earlier today I couldn't see it:

Only one job is remaining and it says "Waiting for a runner to pick up this job..." is this expected?

Hmm, guess we're stuck in a queue somewhere

Bit just be one of those "grab a cup of coffee and come back later" messages

Never mind... the test ran... the problem was solved... but now hit another error:

NotImplementedError: [Errno 2] No such file or directory: '/home/runner/work/CompilerGym/CompilerGym/examples/loop_optimizations_service/service_py/./../../../compiler_gym/third_party/programl/compute_programl'

I think that should be easy to solve: upload that compute_programl binary and download it.
However, I believe in the future that binary should be part of the *.whl file.

Is there a difference between compute_programl and running compute_observation Programl /path/to/bitcode.bc?

The compute_observation binary is in the wheel

Is there a difference between compute_programl and running compute_observation Programl /path/to/bitcode.bc?

The compute_observation binary is in the wheel

OK. I wasn't aware of compute_observation. I created an Issue to replace compute_programl with compute_observation : #702

build-linux-cmake and test-linux-cmake are now green. We can merge it now.

In upcoming PRs I will remove the calls to the examples from ci.yaml and ensure that pytest runs them instead.