As a Data Engineer, I need to investigate and document the approach we plan to take to implement Iceberg sensors
In this ticket: https://phabricator.wikimedia.org/T340466 we had estimated the implementation of using a Postgres database to store the status of datasets, however, we should make sure we have considered other options before moving forward with implementation.
Another, potential solution is to use the Airflow database (as this contains a lot of the data we need already).
The ticket should be time boxed to ~2 weeks and is complete when:
- Pros and cons of solutions are documented (here or in wikitech)
- Team agrees on approach
- Create follow up ticket for implementation: https://phabricator.wikimedia.org/T369900