Tests shouldn't depend on manual steps - if the file is required we should:

1. store it in resources
2. upload using operator, for example LocalFilesystemToGCSOperator
3. After the tests, remove any resources created during tests (usually it's enough to remove Bucket)

Hi @bhirsz I wanted to do upload the file programatically like the other chain. However, the sensor task GCSUploadSessionCompleteSensor waits for change in number of files in the bucket. If we have the upload task before GCSUploadSessionCompleteSensor task, it won't detect any changes. On the other hand, if we add upload task after GCSUploadSessionCompleteSensor task, it would be blocked until GCSUploadSessionCompleteSensor task completes and does not solve the need. I am unsure how to add dependency between such tasks. Any suggestions?

It's tricky in this case, indeed. How about starting sensors in parallel with upload_file?

Example:

    chain(
        # TEST SETUP
        create_bucket,
        upload_file,
        # TEST BODY
        [gcs_object_exists, gcs_object_with_prefix_exists],
        # TEST TEARDOWN
        delete_bucket,
    )
    chain(
        create_bucket,
        # TEST BODY
        [gcs_upload_session_complete, gcs_update_object_exists],
        delete_bucket
    )

We're starting the sensors and in meantime we're uploading the file - and sensors detect it:

I'm not sure though if it will not be flaky in some cases - running this in CI will show. I will update my PR with this change.

Right. Since, we are starting in parallel, it may happen that, the upload task is picked up before the sensor task begins and it may not detect the change as expected. Hence, added the comment to manually upload the file. :)

Also, the gcs_object_update_sensor_task needs to be activated after gcs_upload_session_complete_task (and not in parallel with it) as the object is expected to be detected by sensor and the object_update task is to confirm the manually upload has happened prior to it.

We can alleviate it a bit with sleep (yes, the old the ugly sleep ;)). Run sensors separably and in the meantime trigger sleep (5s will suffice) and then upload task:

chain(
    # TEST SETUP
    create_bucket,
    sleep,
    upload_file,
    # TEST BODY
    [gcs_object_exists, gcs_object_with_prefix_exists],
    # TEST TEARDOWN
    delete_bucket,
)
chain(
    create_bucket,
    # TEST BODY
    gcs_upload_session_complete,
    gcs_update_object_exists,
    delete_bucket
)

And of course put some explanation in comments why we're doing it.

Manual steps are unfortunately a big no for us since the plan is to run system tests in CI in community. We can't expect people performing manual task while running them so I'm looking for an automatic solution

Okay, I was not aware of the fact that we're migrating these example DAGs to tests. I was of the impression that the example DAGs are for references to our community on how to implement DAGs for certain operators. Understand and totally agree to your fair point that tests should not need manual interventions.
Okay, so we can try introducing sleep if we feel right about it. Also, I am new to contributing to Airflow :)

cc: @kaxil

Yeah, we had two goals with the migration - to make writing system tests easier and actually ensure that are example dags are runnable (what would be point in example that doesn't work, and it's actually often the case ;)). In the old design system tests actually "wrapped" and run examples - but not all example dags were used by system tests. In the new design we're also using examples as system tests, but without separating it to different files and now example dag is the system test itself.

Any change is highly welcome and it's good to see PRs such like yours - congrats! We only missed the notification on PR (sadly Github doesn't allow for an advanced notification system) and we were not able to have this discussion before merging the PR.