Hi @Anonycoders 👋🏻 ,

Thank you for reporting the issue, and for all the info! Will take a look and get back to you.

Hi @InesNi 👋
Do we have any update on this issue?
Please let me know if more information is needed from my end.

Hello, we have a similar issue

When running a simple

  phases:
    - duration: 60
      arrivalRate: 1
      name: One per sec

I expect 60 aggregated user and this is the case when running from my machine and with fargate --count 1, but when I bump up the --count I start having weird numbers, is this expected ?

After investigating a bit, if I use --output to look at the json report, the metrics are wrong also in this file, so my guess is that its not coming from datadog.

@jCOTINEAU can you share more on the numbers you're seeing? --count should act as a multiplier, e.g. with the phases config that you shared and --count 2 you should see a total of 120 VUs created. The period may be slightly longer than 60s, but by no more than a few seconds (this is to be expected in a distributed test as Fargate tasks are not guaranteed to start at the exact same time).

and for everyone else seeing mismatched numbers between Artillery and Datadog - can you share the exact config of the chart you're using in Datadog? Sometimes if the correct aggregation is not set in Datadog the numbers can be different to what Artillery is reporting.

@hassy hooo that is interesting, i was not understanding that the --count would multiply the phase, I was thinking the phase load would get splited between the count members.

I was expecting the aggregation thingy, will take a deeper look and come back here, thanks a lot for the clarifications.

@jCOTINEAU that's a fair expectation, we need to make it clearer in the docs. We definitely want to add support for distributing the load as you described as well, but no ETA on that yet.

Side note, the product is insanely cool, integration with fargate and datadog is super smooth, huge kudos!

@jCOTINEAU thank you, very happy to hear that!

ANother question @hassy, you were mentioning datadog aggregation, what value should be setup over there?

It should always be SUM for counter metrics (like vusers.created or vusers.failed) and depending on the time window you might need to add an explicit rollup with a sum as well: https://docs.datadoghq.com/dashboards/functions/rollup/

that is interesting, because indeed when i run some more big tests with fargate count=10, the numbers on datadog does not match the one from the json report.

I will dig a bit into the rollup thing, thanks for the clarification.

Edit: damn even when adding a rollup sum from 1 second to X minutes, it does not change anything I am still 2/3 times below the number of vuser created.