Home Assistant MQTT Discovery Support

‼️ Make sure to read the instructions below carfully!.

1. Backup your configuration (use the System -> Backup/Restore page)!

2. You should update to 12.0.1 before you update to this release. All other migrations are untested.

3. Upload and update the update-*.zip file from this release.

4. Let it restart and check on the System -> Info page that the Firmware as well as the Web UI got updated. If only one got updated, redo the update. If it fails several times, you also can update the Firmware and the Web UI separately.

5. Safe way:

   1. Update first the firmware.bin (extract from zip file) and do the Reboot
   2. Update with the full zip file (update-*.zip, ignore the version warning after the reboot)

6. Please go to Settings -> Configuration and address the changed parameters:

   • DataLogging (storing the values for data graph)
   • Debug (extended by different debug reporting levels)

If anything breaks you can try to enforce manual update as following:

OTA: 1. Make sure the last run of the update completed the Uploading step. 1. Call http://<IP>/ota?task=update&file=<UPLOAD_FILENAME> to enforce the extraction/flashing.

Initial Setup: 1. Use the initial_esp32_setup.zip ( https://github.com/jomjol/AI-on-the-edge-device/wiki/Installation ) as alternative to have a clean install.

• Implementation of Home Assistant MQTT Discovery

• Improved ROIs configuration: locked ROI geometry, equidistant delta x

• Improved OTA Update mechanism (only working after installation for next update)

• Added data logging in /log/data - One day per file and each measurement is on one line

  • Format: csv - comma separated
  • Content: time, name-of-number, raw-value, return-value, pre-value, change-rate, change-absolute, error-text, cnn-digital, cnn-analog

• Show graph of values direct in the user interface (thanks to @rdmueller)

  • Using new data logging (see above)
  • Possibility to choose different values and switch between different numbers (if present)

  Note: You need to activate data logging for this feature to work, see above!

• PreValue is now contained in /json (#1154)

• SD card info into the System>Info menu (thanks to @Slider007)

• Version check (Firmware vs. Web UI)

• Various minor new features

• Updated tflite (dig-cont_0600_s3.tflite)
• Updated OTA functionality (more robust, but not fully bullet prove yet)
• Updated Espressif library to [email protected]
• #1176 accept minor negative values (-0.2) if extended resolution is enabled
• #1143 added config parameter AnalogDigitalTransitionStart. It can setup very early and very late digit transition starts.
• New version of dig-class100 (v1.4.0): added images of heliowatt powermeter
• NEW v13.0.2: Update Tool "Logfile downloader and combiner" to handle the new csv file format.
• NEW v13.0.2: MQTT: Added MQTT topic status (Digitalization Status), Timezone to MQTT topic timestamp.#
• NEW v13.0.2: Logging: Disable heap logs by default, cleanup

• #1116 precision problem at setting prevalue
• #1119 renamed firmware.bin not working in OTA
• #1143 changed postprocess for analog->digit (lowest digit processing)
• #1280 check ROIs name for unsupported characters
• #983 old log files did not get deleted
• Failed NTP time sync during startup gets now retried every round if needed
• Whitespaces and = in MQTT and InfluxDB passwords
• Various minor fixes and improvements
• NEW v13.0.2: Corrected Version comparison between firmware and Web UI.
• NEW v13.0.3: Re-updated build environment to v5.2.0 (from accidental downgrad to v4.4.0)
• NEW v13.0.4: Fix for reboot in case of MQTT not used
• NEW v13.0.5: No reboot in case of missing NTP-connection

• n.a.

Improve user experience

‼️ The release breaks a few things in ota update ‼️

Make sure to read the instructions below carfully!.

1. Backup your configuration (use the System > Backup/Restore page)!
2. You should update to 11.3.1 before you update to this release. All other migrations are not tested. Rolling newer than 11.3.1 can also be used, but no guaranty.
3. Upload and update the firmware.bin file from this release. but do not reboot
4. Upload the html-from-11.3.1.zip in html upload and update the web interface.
5. Now you can reboot.

If anything breaks you can try to 1. Call http://<IP>/ota?task=update&file=firmware.bin resp. http://<IP>/ota?task=update&file=html.zip if the upload successed but the extraction failed. 1. Use the initial_esp32_setup.zip ( https://github.com/jomjol/AI-on-the-edge-device/wiki/Installation ) as alternative.

• Automatic release creation
• Newest firmware of rolling branch now automatically build and provided in Github Actions Output (developers only)
• #1068 New update mechanism:
  • Handling of all files (zip, tfl, tflite, bin) within in one common update interface
  • Using the update.zip from the Release page
  • Status (upload, processing, ...) displayed on Web Interface
  • Automatical detection and suggestion for reboot where needed (Web Interface uupdates only need a page refresh)
  • ‼️ Best for OTA use Firefox. Chrome works with warnings. Safari stuck in upload.

• Integrated version info better shown on the Info page and in the log
• Updated menu
• Update used libraries (tflite, esp32-cam, esp-nn, as of 20220924)

• #1092 censor passwords in log outputs
• #1029 wrong change of checkDigitConsistency now working like releases before 11.3.1
• Spelling corrections (cristianmitran)

• Remove the folder /firmware from GitHub repository. If you want to get the latest firmware.bin and html.zip files, please download from the automated build action or release page

Intermediate Digits

• ATTENTION:

  • first update the firmware.bin and ensure that the new version is running

  • Only afterwards update the html.zip

  • Otherwise the downwards compatibility of the new counter clockwise feature is not given and you end in a reboot loop, that needs manual flashing!

• NEW v11.3.1: corrected corrupted asset firmware.bin

• Increased precision (more than 6-7 digits)

• Implements Counter Clockwise Analog Pointers

• Improved post processing algorithm

• Debugging: intensive use of testcases

• MQTT: improved handling, extended logging, automated reconnect

• HTML: Backup Option for Configuration

• HTML: Improved Reboot

• HTML: Update WebUI (Reboot, Infos, CPU Temp, RSSI)

• This version is largely also based on the work of caco3, adellafave, haverland, stefanbode, PLCHome

Intermediate Digits

• Updated Tensorflow / TFlite to newest tflite (version as of 2022-07-27)

• Updated analog neural network file (ana-cont_11.3.0_s2.tflite - default, ana-class100_0120_s1_q.tflite)

• Updated digital neural network file (dig-cont_0570_s3.tflite - default, dig-class100_0120_s2_q.tflite)

• Added automated filtering of tflite-file in the graphical configuration (thanks to @caco3)

• Updated consistency algorithm & test cases

• HTML: added favicon and system name, Improved reboot dialog (thanks to @caco3)

Intermediate Digits

• New and improved consistency check (especially with analog and digital counters mixed)
• Bug Fix: digital counter algorithm

Intermediate Digits

• NEW v11.0.1: Bug Fix InfluxDB configuration (only update of html.zip necessary)

• Implementation of new CNN types to detect intermediate values of digits with rolling numbers

  • By default the old algo (0, 1, ..., 9, "N") is active (due to the limited types of digits trained so far)
  • Activation can be done by selection a tflite file with the new trained model in the 'config.ini'
  • Details can be found in the wiki (different types, trained image types, naming convention)

• Updated neural network files (and adaption to new naming convention)

• Published a tool to download and combine log files - **Thanks to **

  • Files see '/tools/logfile-tool', How-to see wiki

• Bug Fix: InfluxDB enabling in grahic configuration

Stability Increase

• NEW 10.6.2: ignore hidden files in model selection (configuration page)

• NEW 10.6.1: Revoke esp32cam & tflite update

• NEW 10.6.1: Bug Fix: tflite-filename with ".", HTML spelling error

• IndluxDB: direct injection into InfluxDB - thanks to wetneb

• MQTT: implemented "Retain Flag" and extend with absolute Change (in addition to rate)

• config.ini: removal of modelsize (readout from tflite)

• Updated analog neural network file (ana1000s2.tflite) & digital neural network file (dig1400s2q.tflite)

• TFMicro/Lite: Update (espressif Version 20220716)

• Updated esp32cam (v20220716)

• ESP-IDF: Update to 4.4

• Internal update (CNN algorithm optimizations, reparation for new neural network type)

• Bug Fix: no time with fixed IP, Postprocessing, MQTT

Stability Increase

• NEW 10.5.2: Bug Fix: wrong firmware.bin (no rate update)
• NEW 10.5.1: Bug Fix: wrong return value, rate value & PreValue status, HTML: SSID & IP were not displayed
• MQTT: changed wifi naming to "wifiRSSI"
• HTML: check selectable values for consistency
• Refactoring of check postprocessing consistency (e.g. max rate, negative rate, ...)
• Bug Fix: corrected error in "Check Consistency Increase"

Stability Increase

• Graphical configuration: select available neural network files (.tfl,.tflite) from drop down menu
• OTA-update: add option to upload tfl / tflite files to the correct location (/config/)
  • In the future the new files will also be copied to the firmware directory of the repository
• Added Wifi RSSI to MQTT information
• Updated analog neural network file (ana-s3-q-20220105.tflite)
• Updated digital neural network file (dig-s1-q-20220102.tflite)
• Updated build environment to Espressif 3.5.0

Stability Increase

• Implemented LED flash dimming (LEDIntensity). Remark: as auto illumination in the camera is used, this is rather for energy saving. It will not help reducing reflections
• Additional camera parameters: saturation, contrast (although not too much impact yet)
• Some readings will have removable "N"s that can not be removed automatically and are handled with an "error" --> no return value in the field "value" anymore (still reported back via field "raw value")
• Updated esp32 camera hardware driver
• Bug fix: MQTT, HTML improvements

ATTENTION: The new ESP32 camera hardware driver is much more stable on newer OV2640 versions (no or much less reboots) but seems to be not fully compatible with older versions.

If you have problem with stalled systems you can try the following

• Update the parameter ImageQuality to 12 instead of current value 5 (manually in the config.ini)

• If this is not helping, you might need to update your hardware or stay with version 9.2

Stability Increase

• Due to the updated camera driver, the image looks different and a new setup might be needed

  • Update reference image
  • Update Alignment marks

• Reduce reboot due to camera problems

• Update esp32-camera to new version (master as of 2022-01-09)

Stability Increase

• Bug Fix MQTT problem
• Issue:
  • Changing from v9.x to 10.x the MQTT-parameter "Topic" was renamed into "MainTopic" to address multiple number meters. This renaming should have been done automatically in the background within the graphical configuration, but was not working. Instead the parameter "Topic" was deleted and "MainTopic" was set to disabled and "undefined".
• ToDo
  • Update the html.zip
  • If old config.ini available: copy it to /config, open the graphical configuration and save it again.
  • If old config.ini not available: reset the parameter "MainTopic" within the config.ini manually
  • Reboot

Stability Increase

• Reduce ESP32 frequency to 160MHz

• Update tflite (new source: https://github.com/espressif/tflite-micro-esp-examples)

• Update analog neural network (ana-s3-q-20220105.tflite)

• Update digital neural network (dig-s1-q-20220102.tflite)

• Increased web-server buffers

• bug fix: compiler compatibility

Stability Increase

• NEW v10.0.2: Corrected JSON error

• Updated compiler toolchain to ESP-IDF 4.3

• Removal of memory leak

• Improved error handling during startup (check PSRAM and camera with remark in logfile)

• MQTT: implemented raw value additionally, removal of regex contrain

• Normalized Parameter MaxRateValue to "change per minute"

• HTML: improved input handling

• Corrected error handling: in case of error the old value, rate, timestamp are not transmitted any more

External Illumination

• Direct JSON access: http://IP-ADRESS/json
• Error message in log file in case camera error during startup
• Upgrade analog CNN to v9.1.0
• Upgrade digital CNN to v13.3.0 (added new images)
• html: support of different ports

• NEW 9.1.1 bug fix: LED implemenetation
• External LEDs: change control mode (resolve bug with more than 2 LEDs)
• Additional info into log file
• Bug fix: decimal shift, html, log file

• Implementation of external illumination to adjust positioning, brightness and color of the illumination now set individually
  • Technical details can be found in the wiki: https://github.com/jomjol/AI-on-the-edge-device/wiki/External-LED
• New housing published for external LEDs and small clearing: https://www.thingiverse.com/thing:5028229

• Upgrade digital CNN to v13.1.0 (added new images)
• bug fix: wlan password with space, double digit output

• License change (remove MIT license, remark see below)

• html: show hostname in title and main page

• configuration:

  • moved setting ExtendedResolution to individual number settings
  • New parameter IgnoreLeadingNaN (delete leading NaN's specifically)
  • ATTENTION: update of the config.ini needed (open, adjust ExtendedResolution, save)

• Bug fixing (html, images of recognized numbers)

  ATTENTION: LICENSE CHANGE - removal of MIT License.

• Currently no licence published - copyright belongs to author

• If you are interested in a commercial usage or dedicated versions please contact the developer

  • no limits to private usage

• Upgrade digital CNN to v12.1.0 (added new images)
• Dedicated NaN handling, internal refactoring (CNN-Handling)
• HTML: confirmation after config.ini update
• Bug fixing

• Improve server responsiveness

• Flow status and prevalue status in overview

• Improved prevalue handling

• GPIO: using the general mqtt main topic for GPIO

• Upgrade digital CNN to v12.0.0 (added new images)

• Update tfmicro to new master (2021-08-07)

• Bug fix: remove text in mqtt value, remove connect limit in wlan reconnet

• NEW 8.0.5: bug fix: saving prevalue

• NEW 8.0.4: bug fix: load config.ini after upgrade

• NEW 8.0.3: bug fix: reboot during config.ini handling, html error

• NEW 8.0.2: saving roundes prevalue, bug fix html server

• NEW 8.0.1: bug fix: html handling of parameter FixedExposure and ImageSize

• Dual / multi meter support (more than 1 number to be recognized) This is implemented with the feature "number" on the ROI definition as well as selected options

• MQTT: standardization of the naming - including new topics (json, freeMem, uptime)c

• Preparation for extended GPIO support (thanks to Zwerk2k) - not tested and fully functional yet

• Bug fixing: html server, memory leak, MQTT connect, hostname, turn of flash LED

ATTENTION: the configuration and prevalue files are modified automatically and will not be backward compatible!

• NEW: 7.1.2: bug fix setting hostname, Flash-LED not off during reboot

• NEW: 7.1.1: bug fix wlan password with "=" (again)

• MQTT error message: changes "no error", send retain flag

• Update wlan handling to esp-idf 4.1

• Upgrade digital CNN to v8.7.0 (added new images)

• Bug fix: MQTT, WLAN, LED-Controll, GPIO usage, fixed IP, calculation flow rate

• NEW: 7.0.1: bug fix wlan password with "="

• Upgrade digital CNN to v8.5.0 (added new images)

• New MQTT topics: flow rate (units/minute), time stamp (last correct read readout)

• Update MQTT/Error topic to " " in case no error (instead of empty string)

• Portrait or landscape image orientation in rotated image (avoid cropping)

• NEW 6.7.2: Updated html for setup modus - remove reboot on edit configuration)

• NEW 6.7.1: Improved stability of camera (back to v6.6.1) - remove black strips and areas

• Upgrade digital CNN to v8.3.0 (added new type of digits)

• Internal update: TFlite (v2.5), esp32cam, startup sequence

• Rollback to espressif v2.1.0, as v3.2.0 shows unstable reboot

• Bugfix: WLan-passwords, reset of hostname

• NEW 6.6.1: failed SD card initialization indicated by fast blinking LED at startup

• Improved SD-card handling (increase compatibility with more type of cards)

• Upgrade digital CNN to v8.2.0 (added new type of digits)

• Supporting alignment structures in ROI definition

• Bug fixing: definition of hostname in config.ini

• Additional alignment marks for settings the ROIs (analog and digit)

• Upgrade analog CNN to v7.0.0 (added new type of pointer)

• NEW: 6.3.1: bug fixing in initial edit reference image and config.ini (Spelling error in InitialRotate)

• Initial setup mode: bug fixing, error correction

• Bug-fixing

• NEW 6.2.2: bug fixing

• NEW 6.2.1: Changed brightness and contrast to default if not enabled (resolves to bright images)

• Determination of fixed illumination settings during startup - speed up of 5s in each run

• Update digital CNN to v8.1.1 (additional digital images trained)

• Extended error message in MQTT error message

• Image brightness is now adjustable

• Bug fixing: minor topics

• Disabling of analog / digital counters in configuration

• Improved Alignment Algorithm (AlignmentAlgo = Default, Accurate , Fast)

• Analog counters: ExtendedResolution (last digit is extended by sub comma value of CNN)

• config.ini: additional parameter hostname (additional to wlan.ini)

• Switching of GPIO12/13 via http-interface: /GPIO?GPIO=12&Status=high/low

• Bug fixing: html configuration page, wlan password ("=" now possible)

• Major change: image processing fully in memory - no need of SD card buffer anymore

  • Need to limit camera resolution to VGA (due to memory limits)

• MQTT: Last Will Testament (LWT) implemented: "connection lost" in case of connection lost to TopicError

• Disabled CheckDigitIncreaseConsistency in default configuration - must now be explicit enabled if needed

• Update digital CNN to v7.2.1 (additional digital images trained)

• Setting of arbitrary time server in config.ini

• Option for fixed IP-, DNS-Settings in wlan.ini

• Increased stability (internal image and camera handling)

• Bug fixing: edit digits, handling PreValue, html-bugs

• Implementation of initial setup modus for fresh installation

• Code restructuring (full compatibility between pure ESP-IDF and Platformio w/ espressif)

• Bug fixing: internal improvement of file handling (reduce not responding)

• Implementation of configuration editor (including basic and expert mode)

• Adjustable time zone to adjust to local time setting (incl. daylight saving time)

• MQTT: additional topic for error reporting

• standardized access to current logfile via http://IP-ADRESS/logfileact

• Update digital CNN to v7.2.0, analog CNN to 6.3.0

• Bug fixing: truncation error, CheckDigitConsistency & PreValue implementation

• Implementation of rolling log-files

• Update Tflite-Core to master@20201108 (v2.4)

• Bug-fixing for reducing reboots

• Update digital CNN to v6.5.0 and HTML (Info to hostname, IP, ssid)

• New implementation of "checkDigitConsistency" also for digits

• MQTT-Adapter: user and password for sign in MQTT-Broker

• Implementation of MQTT Client

• Improved Version Control

• bug-fixing

• Bug-Fixing (hostname in wlan.ini and error handling inside flow)

• Integrated automated versioning system (menu: SYSTEM --> INFO)

• Update Build-System to PlatformIO - Espressif 32 v2.0.0 (ESP-IDF 4.1)

• Implementation of Decimal Shift

• Update default CNN for digits to v6.4.0

• Improvement HTML

• Support for Chrome and Edge

• Reduce logging to minimum - extended logging on demand

• Implementation of hostname in wlan.ini (hostname = "HOSTNAME")

• Bug fixing, code corrections

• Update to new and modern layout
• Support for Chrome improved
• Improved robustness: improved error handling in auto flow reduces spontaneous reboots
• File server: Option for "DELETE ALL"
• WLan: support of spaces in SSID and password
• Reference Image: Option for mirror image, option for image update on the fly
• additional parameter in wasserzaehler.html?noerror=true to suppress an potential error message
• bug fixing

• Bug in configuration of analog ROIs corrected - correction in v.1.0.2 did not work properly

• Improved update page for the web server (/html can be updated via a zip-file, which is provided in /firmware/html.zip)

• Improved Chrome support

• Implementation of "delete complete directory" Attention: beside the firmware.bin, also the content of /html needs to be updated!

• Bug in configuration of analog ROIs corrected

• minor bug correction

• preValue.ini Bug corrected

• minor bug correction

• First usable version - compatible to previous project (https://github.com/jomjol/water-meter-system-complete)

• NEW:

  • no docker container for CNN calculation necessary
  • web based configuration editor on board

• Initial Version