Page MenuHomePhabricator

AQS content gap metrics ingestion job
Closed, DeclinedPublic

Description

Steps required to have a working pipeline to ingest content gap metrics from hive into AQS cassandra:

  • The content metrics input datasets have changed, the hql file to insert into cassandra needs to be updated
  • The airflow dag draft needs to be completed and merged.
  • Backfill dag using a start_date have airflow catch up.

Event Timeline

The updates for the hql scripts are ready, but I am lacking committer rights to the refinery repo - @Milimetric would you be able to add me? If this is complicated, and seen that gitlab is the future, I can also just send you a patch.

fkaelin moved this task from Backlog to In Progress on the Research board.

Change 958518 had a related patch set uploaded (by Fabian Kaelin; author: Fabian Kaelin):

[analytics/refinery@master] Update knowledge gap metrics into cassandra loading hql

https://gerrit.wikimedia.org/r/958518