So further to this, it seems like the size of the step up is connected to how long the sensors has been 'off' or detecting zero, because this is the ridiculous jump it did today, a day of continuous solar production, and then when the Glow kicked in at the end of the day, it instantaneously added nearly 18kWh to the total.

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

This issue continues, and renders the Glow useless for daily power consumption, and only useful for instantaneous power monitoring. This issue shouldn't be closed off, even if there is no solution, because people need to be able to find that it as a serious limitation of the platform.

I think the best solution at this point is to put a note somewhere in the documentation that it is best not to use glow if you also have solar panels. What you also can try is using a different method: left or trapezoid.

./Klaas

What you also can try is using a different method: left or trapezoid.

Happy to try this out - is there a way I can make this change onboard the Glow itself, rather than making a Template sensor in HA - would be good to still get the advantage of the 'always recording' glow, even during a system reboot, etc.

I have noticed this in testing as the test pulses I've been using are not well timed and I will wait long periods before testing it again. I think Klaas means you can change the method of the total_daily_energy sensor by adding this to the adopted devices ESPHome YAML:

sensor:
  - id: !extend sensor_total_daily_energy
    method: left

Petro on the HA Community Forum has a great explanation of this here, where he also suggests that trapezoid is not good if you have large periods of the same value (or 0, in the case of solar):
https://community.home-assistant.io/t/calculating-watts-kw-and-kwh/173250/3

There is also this post which says that for square waveforms, the left method should be used.
https://community.home-assistant.io/t/integral-trapezoidal-rule-incorrect-data-for-power-energy/771985/4

This is totally untested and only based on some research and assumptions on what Klaas meant because I have also noticed this issue. I will report back if/when I get a chance to test.

Can confirm that changing to left does indeed dramatically reduce the problem, though you can still see the drift against a clamp meter.

I think that is a limitation of the pulse meter method in general. The pulse meter only knows the data that is there (the pulses), it makes assumptions about what the value was in between polling samples because the Riemann sum is an approximation. The CT clamp has a much higher polling rate in comparison. Have a look at the Wikipedia page for graphs that visualise the methods: https://en.wikipedia.org/wiki/Riemann_sum

Despite that, I think maybe you'd got some false signals in your data, or noise? I think that even if the approximations of energy consumption are wrong, the total at the end of the day should be correct because for every x kWh consumed, there must have been x * pulse_rate pulses sent. The fact it doesn't add up with your CT Clamp at the end of the day, I think, means it must have counted some pulses that didn't happen.

Disclaimer, it's a Friday morning before I finish work for Christmas so take the above with a grain of salt.

I think you're right, though I imagine the "drift" that is still happening between the glow and the CT clamp is the same "stepping up" when the glow goes to 0, however it is now just happening at the 'left' moment, so it isn't as dramatic a result.

There's also the chance that the clamp is slightly under-measuring the actual current, but this is a notable improvement, as in the past it was out by orders of magnitude - multiple kWh a day.

The graph does suggest the "stepping up" is the cause, based on the CT clamp being flat. I've just checked the total daily energy component page and it says it uses the integral. I'll admit complex math is not one of my strengths but I can't understand why it uses an integral. Counting total daily energy with a pulse meter is as simple as dividing the number of total pulses received in a 24 hour period by the pulse rate.

@klaasnicolaas am I missing something critical here? It seems to be that doing the above and simply counting every pulse as 1/pulse_rate kWh would give more accurate numbers than relying on integrals and assumptions. The total of the pulse meter, for instance, reset every 24 hours, would be more accurate at counting daily energy usage than the total_daily_energy sensor which is intended to take a power input in watts and estimate the daily usage.

A template sensor that calculates the difference between the value of total_sensor_energy at 00:00 and now would be pretty much perfectly accurate, assuming a perfect input signal, no? If this number was updated frequently enough you'd have a pretty nice smooth graph similar to Kinetic's CT clamp graph, rather than one with big steps and jumps.

I hope this won't get seen for a few days as you should all be celebrating with your families but I've been thinking about this for days and it's bothering me tonight.

@kineticscreen after the Christmas period, when you get a chance, try dropping this Home Assistant automation in and allow your total energy graph to build up over a 24 hour period. Compare the zeroed total energy graph against the CT clamp and the Total Daily Energy (integral based) graphs.

alias: Reset Total Energy at Midnight
description: ""
triggers:
  - trigger: time
    at: "00:00:00"
conditions: []
actions:
  - action: button.press
    metadata: {}
    data: {}
    target:
      entity_id: button.reset_total_energy
mode: single

Barring any filters you've applied to the total pulse meter output, I think you will find that the total at the end of the day is much closer to your CT clamp (assuming it's accurate) and you'll also have a smoother graph with less steps.

If my logic is right, this should fix all of the numerous issues of this nature reported with Glow, especially from this with renewables. Of course it will need a properly integrated solution but I'd like to prove that graphing the total is more accurate than using the Riemann sum.

Hope you all have a great holiday period and please don't prioritise this meaningless stuff over your families. I'm only posting this now because it's stopping me from sleeping tonight! Merry Christmas.

I'm confused what this would solve, as it just triggers a reset to zero at midnight which happens anyway. The issue isn't the midnight reset to zero, it's the 'coming out of zero' moments that happen throughout the day, when grid use changes from export to import. Is there something else apart from this automation that I'm missing?

You might have missed the extra stuff I put in the comment prior.

I'm confused what this would solve, as it just triggers a reset to zero at midnight which happens anyway.

The total doesn't reset, the daily energy graph resets but the total is persistent until manually reset. They use different counting methods.

The daily energy component is intended to be used with hardware that outputs energy use in watts, measured directly. In the case of the pulse meter, we're not really measuring watts, we're measuring the time between pulses and estimating the watts. This means the "daily energy" is based on an estimation and is often wrong when pulses are irregular. It also uses the Riemann sum which is also an estimation and imperfect, doubling down on the errors.

The total amount of energy we use in any period is much easier to calculate than that. Every pulse is always 1/pulse_rate kWh. You just count the number of pulses received and divide by the pulse rate. The pulse meter total does this for us, but it's not reset daily.

The HA automation I posted is a quick fix to reset the pulse meter total daily. This was the quickest way I could think of to use the pulse meter total. This should be temporary to prove that this method is more accurate than the total_daily_energy component. If it is, then we can make a proper ESP template/other sensor.

If you add the automation to reset the sensor at midnight, we can allow it to graph for a 24 hour period and you can compare it to your CT clamp and the total_daily_energy component and hopefully prove this.

Hope I explained myself clearly enough. Merry Christmas :)

Ah I think I get your point now. So the current consumption is a derived Watts value calculated from two primary sources (flashes / time) and then the daily consumption is then derived from this derived secondary Watts value, when it could just come from the primary flash value.
And this more direct method is already used in the total value, but for whatever reason wasn't implemented for the total daily.
And with a quick glance at the last 2 days, I think you've solved this one. Here are all 3 plotted:

And when I take the 2 glow values, and correct the Total value back to 0 each day in Photooshop, it appears to track exactly to the clamp meter's graph above.

So even just using a utility meter or whatever sensor to generate a daily from the total would further confirm this.

Glad you caught on to my iffy explanation! I don't always explain things very well. It does look like the yellow line follows the shape of the red one better than blue does, especially around 6:00pm when the blue line has a bunch of steps.

I'm still with in-laws at the moment but I'll see if I can get a better ESPHome component to track the yellow line daily. I fear the other utility meter components may also use the Riemann sum but I'll have to check. Thanks for your data

So this is looking promising - so for the first full day of data utilising a Utility Meter derived from the Total Energy (resetting to 0 every night) we have.. a total that is actually slightly less than the clamp meter.

It will be more revealing on a cloudy day, as frequent changes between import and export to grid (and thus the glow continuously coming in and out of a zero state) makes the discrepancies go wild.

That's reassuring. A bit odd that it strayed at the end of the day. I wonder if your photodiode was missing pulses? As you say, trialing for a few days will see how accurate it is long term against the CT clamp and also the existing daily Riemann sum sensor.

Data update between all 3 sources. On certain days the new Glow sensor is the lowest of the 3, haven't quite worked out what the relationship is, as it only happens on certain days.

Haven't yet been able to check this against the values reported by the power company.

How strange. I'm not sure why that would be. There doesn't appear to be a pattern. My possibly slightly biased theory would be that the CT clamp is overmeasuring because of estimations it makes itself in between data points, although it's clear it has a much higher sampling rate than the meters usage LED provides.

Of course Glow is also relying on capturing all pulses with 100% accuracy. Are you confident it's not missing any, maybe due to internal_filter being too high or the photodiode not being sensitive enough? If it is capturing with 100% accuracy I can only imagine that the CT clamp is overestimating, if my previous assumptions are correct.