Page MenuHomePhabricator

[Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors
Closed, ResolvedPublic8 Estimated Story PointsSpike

Description

As a Data Engineer, I need to investigate and document the approach we plan to take to implement Iceberg sensors

In this ticket: https://phabricator.wikimedia.org/T340466 we had estimated the implementation of using a Postgres database to store the status of datasets, however, we should make sure we have considered other options before moving forward with implementation.

Another, potential solution is to use the Airflow database (as this contains a lot of the data we need already).

The ticket should be time boxed to ~2 weeks and is complete when:

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptMar 25 2024, 5:10 PM
Ahoelzl renamed this task from [Status Store] [SPIKE] Document Approach for Iceberg Sensors to [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors.Mar 26 2024, 3:58 PM

We have been working with @amastilovic on a Wikitech page to describe the reasoning behind choosing Airflow itself to store Dataset status.