@arturowczarek can we run "regular" integration tests on EMR, or only special tests otherwise located in the emr_test_jobs directory?

@mobuchowski The loading class locates the files in the specific directory for simplicity. We can extend it if necessary, but passing just the name of the file is convenient, because the name is used to create temporary directory where everything for this test is kept. Otherwise we could have multiple scripts with the same name and they would keep the events under the same location. Alternatively we could have a better strategy for determining the location for the events.

Makes sense, I just think we'd also potentially benefit with running a bunch of "regular" integration tests on EMR regularly.

Yes, it is one of the methods we could further develop these tests.
There is another challenge we have to find a good solution to - managing resources and determining which tests to run. In some tests I'd like to add, there is a need to have two AWS accounts (to have two separate Glue catalogs). It is not common that developers have access to multiple accounts. In such scenarios they have to skip these tests. I'd also like to create some (possibly) Terraform file to create the infrastructure. I'll do it when we have someone to sponsor it.

The parameters are listed in DynamicParameter enum. Most of them have some reasonable defaults (instance size, EMR version, roles which should be present in every account). Only one is required (openlineage.tests.bucketName). Apart from that you have to be authenticated (have ~/.aws/credentials file or use any method described in Default credentials provider chain)