I think this is now pretty close to actually happening. Could someone from {#DevRel} update the "Metrics requested" section to contain actually measurable things? We cannot measure the number of users (also what's "active"?) but we can measure unique IPs or useragents or whatever.

For now, the most precise description seems to be the one given by @Halfak which mentions API module(s), user agent, OAuth consumer ID, user ID, central user ID, request type as the things we log. Do we actually have a use case for user ID / central user ID? Seems somewhat privacy sensitive so it should only be collected if we really need it. Also, will OAuth consumer ID (or preferably name version if we can easily fetch that, I suppose) give any extra information not given by the user agent? Presumably OAuth tools would set a sane UA.

I tried, check the description. I have added the reasoning on each item, in order to help others help us asking for sensible data.

It would be useful to have a separation between data from Labs and data from third parties. The Labs and Strategic Partnerships teams probably agree?

It would be definitely useful to mix data from Action API, RESTBase, and Wikidata query service if that makes technical sense at all. What I mean is that we don't really care which technology is being used, we care about who is using our APIs and what for.

PS: it looks like #Reading-Infrastructure-Team is taking this task officially?

In T102079#1692344, @Qgil wrote:

I tried, check the description. I have added the reasoning on each item, in order to help others help us asking for sensible data.

It would be useful to have a separation between data from Labs and data from third parties. The Labs and Strategic Partnerships teams probably agree?

I think that the current metrics (UA by IP classification, requests by IP classification, actions) are things we can actually figure out how to count.

Using IP blocks we should be able to classify requests as "Internal" (coming from WMF production hosts like the Parsoid servers), "Labs" (coming from the grid engine servers or other Labs hosts), or "External" (anywhere else).

It would be definitely useful to mix data from Action API, RESTBase, and Wikidata query service if that makes technical sense at all. What I mean is that we don't really care which technology is being used, we care about who is using our APIs and what for.

PS: it looks like #Reading-Infrastructure-Team is taking this task officially?

#Reading-Infrastructure-Team will commit to helping get useful data from the Action API into Hadoop by publishing detailed per-request data to a Kafka topic. Actually building reports/dashboards from there will not be something we commit to. We would be glad to work with whoever builds those reports to tweak the data we send from MediaWiki into Hadoop.

@bd808, @EBernhardson I believe is producing binary Avro to Kafka. At this time, I advise that you to stick to JSON. Analytics is working on figuring out how to import the binary data into Hadoop properly, but it is not as easy as we thought it would be. EventBus will in the near term only support producing JSON.

Using an Avro schema is fine, as is using a JSON schema. If you do go with Avro schema (which you are!) you should be able to push the Avro-JSON representation of your data, instead of binary. I'm not sure how that works with the existing Avro Monolog implementation, but we should make sure that it works.

The avro encoder in php doesn't look to directly support json encoding, but i don't think it needs to either. We would just need to re-use the schema validation, then do a standard json encoding. It seems we might not need binary avro in the php side at all then?

Counts of errors (T113672) by action and user agent, in order to identify problem areas and proactively reach out to API clients getting errors (DevRel and documentation)

@Spage Do you envision needing this particular subset of the data over a long period of time or would something like the last 30 days or so suffice?

I haven't created a sub-task for it yet, but one of the things we are going to need to nail down for all of this is what the aggregate data tables look like that we roll information up into for historic reporting. We won't be able to keep all of the data around forever both because of storage space needed and because of our data retention policies. I'm thinking that this particular use case is more near-term operational than long-term trend related.

Counts of errors (T113672) by action and user agent, in order to identify problem areas and proactively reach out to API clients getting errors (DevRel and documentation)

If you are thinking http errors the data is present right now on webrequest table (for the last couple months) so you can get this information as of now, for application errors you will need to wait until those are published.

In T102079#1713841, @Nuria wrote:

If you are thinking http errors the data is present right now on webrequest table

The HTTP response to a failing API request like https://www.mediawiki.org/w/api.php?action=parse&page=Wrong is 200 OK. As T113672 says, the API response has a fantastic HTTP header:

MediaWiki-API-Error: missingtitle

Could we add this MediaWiki-API-Error header to the webrequest table? Should I file a separate bug?

Could we add this MediaWiki-API-Error header to the webrequest table? Should I file a separate bug?

We can add this value to x-Analytics header but it will be worth thinking whether we want a global value to signal "health-of-api-request" .Like, for example, does this header capture a 503?
I do not know much about api return codes so to capture the precise information I think it will be best if you open a new ticket and assign it to API developers (while tagging it with analytics-backlog so we also see it) and we can work together on it.

In T102079#1753045, @Nuria wrote:

Could we add this MediaWiki-API-Error header to the webrequest table? Should I file a separate bug?

We can add this value to x-Analytics header but it will be worth thinking whether we want a global value to signal "health-of-api-request" .Like, for example, does this header capture a 503?

A 503 from the API means a PHP fatal or something so badly screwed up that exceptions aren't being caught. I have no idea what "it will be worth thinking whether we want a global value to signal 'health-of-api-request'" might mean.

I do not know much about api return codes so to capture the precise information I think it will be best if you open a new ticket and assign it to API developers (while tagging it with analytics-backlog so we also see it) and we can work together on it.

I see nothing relevant to the API developers (i.e. me) here that would require a new ticket.

I see nothing relevant to the API developers (i.e. me) here that would require a new ticket.

I think a ticket that is concise regarding the header will be more clear that this wide scope ticket.

Now, If you want this done in the near term you can ping us on #wikimedia-analytics and we can help you do the needed changes on varnish so the header is part of x-analytics map. Otherwise the work can go into our backlog but we will not get to it for a bit.

In T102079#1754485, @Nuria wrote:

I see nothing relevant to the API developers (i.e. me) here that would require a new ticket.

I think a ticket that is concise regarding the header will be more clear that this wide scope ticket.

That doesn't say why you suggested it be assigned to the API developers, or why you implied that the API developers would necessarily have anything to do to make it happen with analytics only needing to "see" it.

@Spage: added a (sub) task : https://phabricator.wikimedia.org/T116658 to keep things organized.