full memory tracking and profiling of Netdata Agent #13789
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ktsaou
requested review from
thiagoftsm,
vkalintiris,
stelfrag and
underhood
as code owners
|
This pull request has been mentioned on Netdata Community Forums. There might be relevant details there: https://community.netdata.cloud/t/insane-netdata-memory-usage/3342/9 |
stelfrag
approved these changes
Oct 9, 2022
Dim-P
added a commit
to Dim-P/netdata
that referenced
this pull request
Oct 25, 2022
commit 64386fa3839d87a252d24513cd1bdae7dd1c4896 Merge: 96246c9ed 4b9f02da2 Author: Dim-P <[email protected]> Date: Tue Oct 25 12:44:32 2022 0100 Merge branch 'master' into logs-manag-master commit 4b9f02da2603c528eef0c7de77a20369556bc66f Author: Costa Tsaousis <[email protected]> Date: Tue Oct 25 12:43:07 2022 0300 apps.plugin function processes cosmetic changes (#13880) commit d7346bcfe453beba3345fe55462992e3b3a41b12 Author: Emmanuel Vasilakis <[email protected]> Date: Tue Oct 25 11:33:03 2022 0300 Don't create a REMOVED alert event after a REMOVED. (#13871) dont create REMOVED after a REMOVED commit 52263275d79f4188df9b8a5a416e072473db03f4 Author: Costa Tsaousis <[email protected]> Date: Tue Oct 25 10:08:02 2022 0300 allow single chart to be filtered in context queries (#13879) commit ca09852a60826684fc175af410facd8c24745829 Author: netdatabot <[email protected]> Date: Tue Oct 25 00:25:26 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-264-nightly. commit 8958da110e5aad5e518770fceb6e56f2d6ade450 Author: Stelios Fragkakis <52996999 [email protected]> Date: Tue Oct 25 01:17:34 2022 0300 Store hidden status when creating / updating dimension metadata (#13869) commit 2d8b5d104b689b6faf28b0e6fc6663b37dc3605f Author: thiagoftsm <[email protected]> Date: Mon Oct 24 19:02:25 2022 0000 Cassandra dashboard description (#13835) commit 103daac879c100e35ae8070aade3a4c3f90583fe Author: Ilya Mashchenko <[email protected]> Date: Mon Oct 24 21:28:09 2022 0300 bump go.d.plugin v0.41.1 (#13874) commit de96dd73f2878346edaf1aa4e7c6cf45ad11173e Author: Timotej S <6674623 [email protected]> Date: Mon Oct 24 15:17:17 2022 0200 app to api netdata cloud (#13856) commit 29271f575a65ed5cd13265413bff3b32398bd538 Author: Stelios Fragkakis <52996999 [email protected]> Date: Mon Oct 24 10:20:39 2022 0300 Use mmap to read an extent from a datafile (#13834) * Use MMAP to read an extent from the datafile. Fallback to uv_fs_read if that fails * Schedule the fallback using uv_fs_read in the MMAP callback * Check the return value of mmap correctly * Add unlikely commit a9135f47bbb36e9cb437b18a7109607569580db7 Author: Emmanuel Vasilakis <[email protected]> Date: Mon Oct 24 09:56:10 2022 0300 Use llvm's ar and ranlib when compiling with clang (#13854) * use llvm ar and ranlib for clang * equality commit 19f023c114d352fc515111a1f53268a48bf02f42 Author: netdatabot <[email protected]> Date: Mon Oct 24 00:23:06 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-257-nightly. commit 00712b351b3c83a54a147ca23365458acbef3105 Author: Costa Tsaousis <[email protected]> Date: Sun Oct 23 23:46:43 2022 0300 QUERY_TARGET: new query engine for Netdata Agent (#13697) * initial implementation of QUERY_TARGET * rrd2rrdr() interface * rrddim_find_best_tier_for_timeframe() ported * added dimension filtering * added db object in query target * rrd2rrdr() ported * working on formatters * working on jsonwrapper * finally, it compiles... * 1st run without crashes * query planer working * cleanup old code * review changes * fix also changing data collection frequency * fix signess * fix rrdlabels and dimension ordering * fixes * remove unused variable * ml should accept NULL response from rrd2rrdr() * number formatting fixes * more number formatting fixes * more number formatting fixes * support mc parallel queries * formatting and cleanup * added rrd2rrdr_legacy() as a simplified interface to run a query * make sure rrdset_find_natural_update_every_for_timeframe() returns a value * make signed comparisons * weights endpoint using rrdcontexts * fix for legacy db modes and cleanup * fix for chart_ids and remove AR chart from weights endpoint * Ignore command if not initialized yet * remove unused members * properly initialize window * code cleanup - rrddim linked list is gone; rrdset rwlock is gone too * reviewed RRDR.internal members * eliminate unnecessary members of QUERY_TARGET * more complete query ids; more detailed information on aborted queries * properly terminate option strings * query id contains group_options which is controlled by users, so escaping is necessary * tense in query id * tense in query id - again * added the remaining query options to the query id * Expose hidden option to the dimension * use the hidden flag when loading context dimensions * Specify table alias for option * dont update chart last access time, unless at least a dimension of the chart will be queried Co-authored-by: Stelios Fragkakis <52996999 [email protected]> commit 9798a2b71e880a73b5b95d62d2e0c63dbc649a0e Author: Ilya Mashchenko <[email protected]> Date: Sun Oct 23 19:36:36 2022 0300 fix(cgroup.plugin): handle qemu-1- prefix when extracting virsh domain (#13866) commit 98b1fdefcbd57667b78b45e87f0a0836c2f58b65 Author: netdatabot <[email protected]> Date: Sun Oct 23 00:23:35 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-254-nightly. commit 8c7fd75515daee2954edc44ce71e5dc61ad2e6fc Author: DShreve2 <[email protected]> Date: Sat Oct 22 05:56:39 2022 -0400 Update step-09 for dbmode update.md (#13864) commit 774fec53cfdf82ab9a4aab263e4f9c21defa058a Author: netdatabot <[email protected]> Date: Sat Oct 22 00:23:35 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-252-nightly. commit 33638cf191996d9aee88007d4af60f0c70acc388 Author: Timotej S <6674623 [email protected]> Date: Fri Oct 21 18:43:19 2022 0200 add ACLK access to ml_info (fix anomalies tab in cloud) (#13863) add ACLK access to ml_info commit 9286530bcd8c4d5029bb4a58fa3aaf965bf37147 Author: Ilya Mashchenko <[email protected]> Date: Fri Oct 21 19:12:25 2022 0300 bump go.d.plugin to v0.41.0 (#13861) commit d02695ea6915d8d7a4ec7a5dde73f6dd18ae3bc1 Author: Vladimir Kobal <[email protected]> Date: Fri Oct 21 14:38:36 2022 0200 Retry reading carrier, duplex, and speed files periodically (#13850) * Retry reading carrier file periodically * Fix formatting type * Don't log error messages on every iteration for duplex and speed commit 12671ae84859cb886e4875d15ecf4daa5d3bf1bc Author: netdatabot <[email protected]> Date: Fri Oct 21 00:22:41 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-248-nightly. commit 1197f0d067ee0e0b570f8d79c8e7809e12fba912 Author: Andrew Maguire <[email protected]> Date: Thu Oct 20 11:34:31 2022 0100 fix typo (#13853) commit 6b7f327e204cae14c5e2374cf0608a008d784276 Author: netdatabot <[email protected]> Date: Thu Oct 20 00:24:24 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-246-nightly. commit 42e85b5a092b18881501b0fe76f91e4969dca088 Author: Emmanuel Vasilakis <[email protected]> Date: Wed Oct 19 18:30:12 2022 0300 Health thread per host (#13712) * Rebased * rebased * health_execute_pending_updates -> health_execute_delayed_initializations * fix labels for current host only * missing bracket * misc fixes, reload health for disconnected hosts * remove volatile, add comment commit b5e1e3690c9a8e59abe3a3a247ce7906f3259053 Author: Austin S. Hemmelgarn <[email protected]> Date: Wed Oct 19 07:11:09 2022 -0400 Properly guard commands when installing services for offline service managers. (#13848) commit 05ef3a0c55747ab41abe5e2459d787d9a43a4416 Author: Austin S. Hemmelgarn <[email protected]> Date: Wed Oct 19 07:10:28 2022 -0400 Properly propagate errors from installer/updater to kickstart script. (#13802) This should result in errors in these scripts being properly deaggregated, instead of being always reported as ‘generic’ failures, thus simplifying diagnosis of end-user problems with the installation or update process. commit d41cded5def1b1046062be9e0288c214e2e088b9 Author: Andrew Maguire <[email protected]> Date: Tue Oct 18 20:30:27 2022 0100 Pandas collector (#13773) Add pandas collector. commit 218ca02b8d26dfba6493eefb7dc88b6e1ca2c0e9 Author: Austin S. Hemmelgarn <[email protected]> Date: Tue Oct 18 13:14:39 2022 -0400 Fix runtime directory ownership when installed as non-root user. (#13797) * Fix runtime directory ownership when installed as non-root user. Followup to #13870. * Fix typo. commit 48e7a6613054dafcc3a1910ef85ef7e2a03be93d Author: Austin S. Hemmelgarn <[email protected]> Date: Tue Oct 18 10:50:25 2022 -0400 Change CI trigger label prefix from `ci/` to `run-ci/`. (#13849) * Change CI trigger label prefix from `ci/` to `run-ci/`. This way it better describes what it’s doing * Update .github/workflows/packaging.yml Co-authored-by: Ilya Mashchenko <[email protected]> Co-authored-by: Ilya Mashchenko <[email protected]> commit bef28e9dd6f2087a02444848e6e6dea1a8fe5046 Author: Austin S. Hemmelgarn <[email protected]> Date: Tue Oct 18 08:30:32 2022 -0400 Use PR label to trigger full packaging CI instead of PR body contents. (#13839) commit 172e049862b1eb8391bb86eb62fe483c3ef8a203 Author: netdatabot <[email protected]> Date: Tue Oct 18 00:24:29 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-238-nightly. commit 66d914d00d5790c7e57d59a5331fabaabdd2b73a Author: Costa Tsaousis <[email protected]> Date: Mon Oct 17 22:10:21 2022 0300 fix tiers update frequency (#13844) * fix tiers update frequency * dont change mqtt_websockets commit d4e22b8d0e749334b91a23b0d947422c00a8caa5 Author: Austin S. Hemmelgarn <[email protected]> Date: Mon Oct 17 13:31:01 2022 -0400 Fix service installation on FreeBSD. (#13842) Instead of using the correct path for installing the system service file, we were always hard-coding `/etc/init.d/netdata`. This happened to work correctly on Linux, but obviously breaks horribly on FreeBSD. commit 2de4e77cc9388fcbbe6e0bda40edefb655d86e96 Author: Timotej S <6674623 [email protected]> Date: Mon Oct 17 17:12:56 2022 0200 Inject costallocz to mqtt_websockets library and its children (#13813) * use mallocz, freez & family also from within the mqtt libs commit bfd9b3445019c238bcd6edcd6c99d299376f4670 Author: Austin S. Hemmelgarn <[email protected]> Date: Mon Oct 17 08:50:20 2022 -0400 Add PR labels to force running Reviewdog checks. (#13818) commit f4a4918cc9aef526cb02eec1e3629c148c2af06b Author: Austin S. Hemmelgarn <[email protected]> Date: Mon Oct 17 08:25:25 2022 -0400 Further improvements to the new service installation code. (#13774) * Improve OpenRC detection. - Check for rc-update command, without that we don’t work so it’s ‘not OpenRC’. - Sort checks so that less expensive ones run first. - If we see `/etc/init.d/local` and it’s got a shebang line with `openrc-run`, then assume we’re dealing with OpenRC. This file gets installed by default as part of OpenRC itself. * Fix runit detection. - `/lib/rc/sv.d` is only used on some runit systems (notably Artix), not all, so don’t bail if we can’t find it. - Check not just for a `runit` command, but also `runsvdir`. * Add support for systemd in WSL. THis just involves running the normal systemd check if we detected WSL, and updating the service manager type appropriately. * Fix tmpdir handling for service installation. * Fix invocation of service install script. * Add fallback to detected but not running service managers. Currently supported for systemd and OpenRC. If no other service manager is detected, but we detected that either systemd or OpenRC is present but not running, fall back to them instead of outright failing. This does not allow starting/stopping the agent using the service manager, but _does_ happen to work for installing the service properly on an offline system. * Fix service-type fallback handling. If we see systemd, even if it not booted into it, we should not install init.d or LSB init scripts. * Fix recursion bug in initd install type checking. * Skip initd if OpenRC is found. Just like systemd, any OpenRC on the system automatically means that it’s _not_ a classic init.d setup. * Fall back to init.d in WSL in some cases. If there is no _usable_ systemd, and the service manager otherwise looks like an init.d setup, fall back to an init.d setup. * Support LSB style init scripts in WSL as well. Also consolidate and update the error message that gets printed when running under WSL. * Fix typo in LSB init script handling code. * Fix command handling for LSB and initd service management. Instead of using `service`, just run the init scripts directly. This way we avoid any bugs in `service` that would cause it to pretend that systemd is in use even though it isn’t. * Fix botched partial commit. * Restructure service type detection to not rely on return codes. * Added option to display service type detection results. This should simplify debugging. commit df38c0c2c4336d2d6fdac471fc0f2f6c3db6531c Author: Austin S. Hemmelgarn <[email protected]> Date: Mon Oct 17 07:19:49 2022 -0400 Improve error and warning messages in the kickstart script. (#13825) commit fcc6971415f7c585d876b8d4d63afdd3409709d0 Author: Ilya Mashchenko <[email protected]> Date: Mon Oct 17 11:00:12 2022 0300 chore(health): rm pihole_blocklist_gravity_file_existence_state (#13826) commit 75f23321a00c6b5e424c7577886cc461390bdff6 Author: dependabot[bot] <49699333 dependabot[bot]@users.noreply.github.com> Date: Mon Oct 17 10:42:07 2022 0300 Bump google-github-actions/upload-cloud-storage from 0.10.2 to 0.10.4 (#13837) Co-authored-by: dependabot[bot] <49699333 dependabot[bot]@users.noreply.github.com> commit 08e1d0b9d7191549a4bffcf6751734d4b47c3d30 Author: dependabot[bot] <49699333 dependabot[bot]@users.noreply.github.com> Date: Mon Oct 17 10:41:49 2022 0300 Bump google-github-actions/setup-gcloud from 0.6.0 to 0.6.2 (#13836) Co-authored-by: dependabot[bot] <49699333 dependabot[bot]@users.noreply.github.com> commit 2d64f4c7e167c58b8daa15602d60303e6a0b0f40 Author: netdatabot <[email protected]> Date: Mon Oct 17 00:23:49 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-228-nightly. commit 08cab7222478d06481c070b7f5027ead1c5d2b01 Author: Stelios Fragkakis <52996999 [email protected]> Date: Sun Oct 16 23:15:14 2022 0300 Add a thread to asynchronously process metadata updates (#13783) * Remove old metalog text fle processing * Add metadata event loop * Move functions from sqlite_functions.c to sqlite_metadata.c Queue updates to the metadata event loop Migration to remove unused tables Cleanup unused functions * Queue chart labels to metadata * Store chart labels to metadata * During shutdown, run full speed * Add shutdown prepare Handle SHUTDOWN in the cmd queue function Add worker thread to handle host/chart/dimension metadata doing dictionary traversals * Remove unused RRDIM_FLAG_ACLK Add flags to trigger host/chart/dimension metadata processing * Incremental processing of chart metadata writes * Store host labels * Remove redundant return statements * Change unit tests / cleanup * Fix rescheduling * Schedule chart labels update by setting the RRDSET_FLAG_METADATA_UPDATE flag * Queue commands to update metadata for dimension and host labels * Make sure we do a final scan to store metadata during shutdown (if needed) * Remove unused structures Adjust queue size since we do batch processing of updates without queueing individual messages Remove pragma mmap for now Fix memory leak during sqlite unittest (minor) * Dont update if we are in archive mode * Cleanup * Build entire message payload and store * Initialize worker completion properly * Properly skip host check for pending metadata updates * Report bind param failures Add worker request inside the data payload Initialize variables to silence warnings Rebase on master * Report the chart id (not the dimension) and the dimension id when storing a dimension * Compilation warnings in 32bit * Add DEFINE for the queries * Remove commented out code * * Remove items parameter from unitest * Remove commented out code * sqlite_metadata.h contains only public items * Use sleep_usec instead of usleep * Rename metadata_database_init_cmd_queue to metadata_init_cmd_queue * Rename metadata_database_enq_cmd_noblock to metadata_enq_cmd_noblock commit 70dad7e1263d066319ad13f13ea0c5b39cf664f5 Author: netdatabot <[email protected]> Date: Sat Oct 15 00:21:11 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-226-nightly. commit 559c4ccdb88421f077f471ebbc3222c948b47c13 Author: Costa Tsaousis <[email protected]> Date: Fri Oct 14 21:16:11 2022 0300 extended processes function info from apps.plugin (#13822) extended processes function info commit 3eeebf7bace136a565a9c8d7040ebfc69283eacd Author: Austin S. Hemmelgarn <[email protected]> Date: Fri Oct 14 14:14:52 2022 -0400 Add a CodeQL analysis workflow. (#13812) * Add a CodeQL analysis workflow. This currently is limited to checking C/C code and Python code. Analysis is run on PRs, pushes to the master branch, and as a scheduled run every Monday morning. The PR checks auto-skip analysis for languages that have no code changes. * Use a label to force running CodeQL checks on PRs. * Add config to skip things we don’t need to scan for Python. * Linting fixes. * Add notice in workflow for finding label to run all checks. * Fix CodeQL warnings on PR. * Skip vendored Python modules in Python scanning. commit f5abfad0e0af633737e42bfab66923956a3edcf5 Author: vkalintiris <[email protected]> Date: Fri Oct 14 15:53:17 2022 0300 Add variants of functions allowing callers to specify the time to use. (#13791) Add variants of functions allowing callers to specify the timeval to use. These functions will be prove helfupl in writting tests that are both (a) memory mode agnostic, and (b) oblivious of rrd internals. commit 8182d439d6c6fc8691613d26e6a7054b1254635c Author: Vladimir Kobal <[email protected]> Date: Fri Oct 14 10:21:25 2022 0300 Remove NFS readahead histogram (#13819) commit fc52535a16cc25be155747f644ff5268492f459d Author: Stelios Fragkakis <52996999 [email protected]> Date: Fri Oct 14 07:52:07 2022 0300 Fix crash on child reconnect and lost metrics (#13821) Increase page_index reference count on metric_get Don't free the db_metric_handle commit 87cb78191fd95ca52d3d6b0a4212f6c9f7591ec5 Author: netdatabot <[email protected]> Date: Fri Oct 14 00:23:10 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-220-nightly. commit 47cb7811b93db2100766b17b627537525fdbb8e2 Author: Vladimir Kobal <[email protected]> Date: Thu Oct 13 18:40:53 2022 0300 Fix exporting unit tests (#13816) commit c6f563da745f7d5d97b9408700fd87d484835e55 Author: Timotej S <6674623 [email protected]> Date: Thu Oct 13 16:20:51 2022 0200 minor - add trace alloc to buildinfo (#13817) add trace alloc to buildinfo commit 4521b76927488a8af02b83b7dce742a8f250e5e3 Author: Austin S. Hemmelgarn <[email protected]> Date: Thu Oct 13 09:36:55 2022 -0400 Update fping plugin documentation with better details about the required version. (#13765) Update fping plugin documentation with better details about required version. commit d766183f6c1574f9998400fc8b1fc89ba99550c5 Author: Austin S. Hemmelgarn <[email protected]> Date: Thu Oct 13 07:25:43 2022 -0400 Fix existing install detection for FreeBSD and macOS. (#13243) commit fb86ddd70d14f7943e0314b02235ce3cbca5cb2d Author: Emmanuel Vasilakis <[email protected]> Date: Thu Oct 13 11:21:23 2022 0300 Count currently streaming senders on the localhost (#13755) rebased commit 61767a8a08d089962ff82e45b3e459c8bd05b0c4 Author: Costa Tsaousis <[email protected]> Date: Thu Oct 13 08:12:52 2022 0300 allow disabling netdata monitoring section of the dashboard (#13788) * allow disabling netdata monitoring section of the dashboard * disable-netdata-stats: Modify eBPF.plugin to disable statistic charts according user selection, by default it is enabled * Don't send internal statistics for exporting engine if it's disabled. * Fix global statistics flag initialization * Don't send internal statistics for checks plugin if it's disabled. Co-authored-by: Thiago Marques <[email protected]> Co-authored-by: Vladimir Kobal <[email protected]> commit afe1b704857a7307547341a1027c019bbe68e910 Author: Costa Tsaousis <[email protected]> Date: Thu Oct 13 08:05:15 2022 0300 dbengine free from RRDSET and RRDDIM (#13772) * dbengine free from RRDSET and RRDDIM * fix for excess parameters to query ops * add comment about ML * update_every from int to uint32_t * rrddim_mem storage engine working * fixes for update_every_s * working dbengine * a lot of changes in dbengine regarding timestamps * better logging of not sequential points * rrdset_done() now gives aligned timestamps for higher tiers * dont change the end_time of descriptors, because they cant be loaded back * fixes for cmake * fixes for db mode ram * Global counters for dbengine loading errors. Ensure dbengine store metrics always has aligned metrics or breaks the page when storing new data. * update lgtm config * fixes for 32-bit systems * update unittests * Don't try to find and create a host on the fly if not already in memory * Remove unused functions * print backtrace in case of fatal * always set ctx to page_index * detect ctx and metric uuid discrepancies * use legacy uuid if multihost is not available * fix for last commit * prevent repeating log * Do not try to access archived charts when executing a data query * Remove unused function * log inconsistent collections once every 10 mins Co-authored-by: Stelios Fragkakis <52996999 [email protected]> commit 2974f525ec703329ef6ad079d8f6c685cfab11ad Author: Costa Tsaousis <[email protected]> Date: Thu Oct 13 08:04:14 2022 0300 overload libc memory allocators with custom ones to trace all allocations (#13810) * overload libc memory allocators with custom ones to trace all allocations * grab libc pointers for external c plugins * use -ldl when necessary; fallback to work without dlsym when it is not available * initialize global variable * add optional dl libs * dynamically link every library function when needed for the first time * prevent crashes on musl libc * another attempt * dont dereference function * attempt no 3 * attempt no 4 * cleanup - all attempts failed * dont enable tracing of allocations * missing parenthesis commit c805a9afad71ac96e703d599cbd6f54c29142ca7 Author: netdatabot <[email protected]> Date: Thu Oct 13 00:22:29 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-211-nightly. commit 414bc49d7efc188bb0217c646a8969114dc28fc4 Author: Ilya Mashchenko <[email protected]> Date: Wed Oct 12 10:14:59 2022 0300 bump go.d.plugin v0.40.4 (#13808) commit 62e79d9b939d0ce7a4edd3e3a86b43c45989090b Author: netdatabot <[email protected]> Date: Wed Oct 12 00:25:20 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-209-nightly. commit 4a389b6ff35b8cbb7157ceaf4a43abf3df4e684f Author: Vladimir Kobal <[email protected]> Date: Tue Oct 11 18:05:23 2022 0300 Merge netstat, snmp, and snmp6 modules (#13806) commit ec95f306f35cff802a7f08bad43411945c0ba977 Author: Costa Tsaousis <[email protected]> Date: Tue Oct 11 18:01:46 2022 0300 fix post-processing of contexts (#13807) commit 849e34209208af2ad7939f221872a49e20321883 Author: Timotej S <6674623 [email protected]> Date: Tue Oct 11 14:01:04 2022 0200 fix warning when -Wfree-nonheap-object is used (#13805) commit b796f3abb78905e6c9e8b2e71b8d0b2cd61cafe3 Author: Austin S. Hemmelgarn <[email protected]> Date: Tue Oct 11 07:38:50 2022 -0400 Add Fedora 37 to CI and package builds. (#13489) Tentative release date is 2022-08-09. commit d91c510f0f4c0d80296939872de018c691933345 Author: Austin S. Hemmelgarn <[email protected]> Date: Tue Oct 11 07:26:46 2022 -0400 Add Ubuntu 22.10 to supported distros, CI, and package builds. (#13785) * Add Ubuntu 22.10 to supported distros, CI, and package builds. Expected release date is 2022-10-20. * Fix handling of netcat in newer Debian and Ubuntu distros. commit 4a935930abe0cbed5aa5f9b583e91fdaf4278223 Author: Timotej S <6674623 [email protected]> Date: Tue Oct 11 12:27:42 2022 0200 minor - add host labels for ephemerality and nodes with unstable connections (#13784) * add host labels for ephemerality and nodes with unstable connections commit 8ae9a6eb93410cb3bb620013701318f83351c128 Author: netdatabot <[email protected]> Date: Tue Oct 11 00:22:38 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-202-nightly. commit bd38daa40cd21f2f357b2930f529b68d96523102 Author: Costa Tsaousis <[email protected]> Date: Mon Oct 10 21:21:39 2022 0300 ARAL optimal alloc size (#13804) aral optimal alloc size commit 51e6ce2863daa4ab430771ff38983bf8396dcc03 Author: Costa Tsaousis <[email protected]> Date: Mon Oct 10 20:32:18 2022 0300 internal log error, when passing NULL dictionary (#13803) * fixes internal log error, when passing NULL dictionary * item is needed - fatal provides enough info commit ebf433e4ab52f989dc75e3ec10041581206ed671 Author: Tasos Katsoulas <[email protected]> Date: Mon Oct 10 16:59:44 2022 0300 Fix minor typo in systemdunits.conf alert (#13796) commit 8f2337dbfc605b6f271931c9864a447c2b37e3df Author: Emmanuel Vasilakis <[email protected]> Date: Mon Oct 10 16:43:59 2022 0300 Use string_freez instead of freez in rrdhost_init_timezone (#13798) use string_freez instead of freez commit f11eb8e074ade091107de2992619d1ce51fefeca Author: Stelios Fragkakis <52996999 [email protected]> Date: Mon Oct 10 16:42:53 2022 0300 Return memory freed properly (#13799) commit 8bffd8fb8e4a584ce69e0fed0bc7733da0d39625 Author: Emmanuel Vasilakis <[email protected]> Date: Mon Oct 10 16:30:08 2022 0300 Also enable rrdvars from health (#13795) also enable rrdvars from health commit b565fd6df456e1d3d9282645e8bccbe9c8ed882e Author: Austin S. Hemmelgarn <[email protected]> Date: Mon Oct 10 08:53:06 2022 -0400 Stop pulling in netcat as a mandatory dependency. (#13787) It’s only used by a single exporter which is itself not in widespread usage. Requiring users to install netcat themselves if they want to use that exporter is not a significant burden, and not pulling it in ourselves reduces our install footprint for ‘regular’ users without really impacting usability for them. commit 360e57d47bc29d0f0362c74493e4e3d932ea147e Author: Stelios Fragkakis <52996999 [email protected]> Date: Mon Oct 10 15:27:35 2022 0300 Parser cleanup (#13782) * Remove end, set disable and flush actions * Remove begin action * Remove label action * Remove overwrite action * Remove chart action * Remove dimension action * Remove label action (part 2) * Remove variable action Co-authored-by: vkalintiris <[email protected]> commit e14b42359048b8a6eb39ae6b2d3434e73ff947b9 Author: thiagoftsm <[email protected]> Date: Mon Oct 10 12:08:24 2022 0000 Update libbpf 1.0.1 (#13778) commit 49363f50c8d9f8d450cd25b57ccb84f52e8925dd Author: Ilya Mashchenko <[email protected]> Date: Mon Oct 10 15:01:21 2022 0300 feat(python.d): respect NETDATA_INTERNALS_MONITORING (#13793) commit 44c0246c6ba4ab0a3bd63342687166dbfdd1b626 Author: Costa Tsaousis <[email protected]> Date: Mon Oct 10 14:20:51 2022 0300 Array Allocator Memory Leak Fix (#13792) * aral cleans up previous mmap files; aral uses double linked list macros; aral got more internal checks * fixed aral memory leak * updated macro commit bff1a5c8eabcca1bb4e910704b4b904cceb93a78 Author: Emmanuel Vasilakis <[email protected]> Date: Mon Oct 10 12:44:01 2022 0300 Initialize st->rrdvars from rrdset insert callback (#13786) * init st->rrdvars from rrdset callback * also init rrdfamily and rrdimvar from rrdset * remove empty line commit bfe86dcb3c0b9af57c5e0d9c7d45eaa6c2d882eb Author: netdatabot <[email protected]> Date: Mon Oct 10 00:21:13 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-189-nightly. commit 758d9c405d2d768a3c125052a02c7a1503b01bd8 Author: Costa Tsaousis <[email protected]> Date: Sun Oct 9 21:58:21 2022 0300 full memory tracking and profiling of Netdata Agent (#13789) * full memory tracking and profiling of Netdata Agent * initialize dbengine only when it is needed * handling of dbengine compiled but not available * restore unittest * restore unittest again * more improvements about ifdef dbengine * fix compilation when dbengine is not enabled * check if dbengine is enabled on exit * call freez() not free() * aral unittest * internal checks activate trace allocations; dev mode activates internal checks commit 067305602f373d12286e492143bf6cb2a32ffe31 Author: Costa Tsaousis <[email protected]> Date: Sun Oct 9 17:02:30 2022 0300 allow netdata installer to install and run netdata as any user (#13780) * allow netdata installer to install and run netdata as any user * Update netdata-installer.sh Co-authored-by: Austin S. Hemmelgarn <[email protected]> * Update netdata-installer.sh Co-authored-by: Austin S. Hemmelgarn <[email protected]> Co-authored-by: Austin S. Hemmelgarn <[email protected]> commit ccfbdb5c3d253a391cab0561dfc8a524b93d2e7c Author: vkalintiris <[email protected]> Date: Sun Oct 9 16:38:49 2022 0300 Remove extern from function declared in headers. (#13790) By default functions are declared as extern in C/C headers. The goal of this PR is to reduce the wall of text that many headers have and, more importantly, to make the declaration of extern'd variables - of which we have many dispersed in various places - easily and quickly identifiable. Automatically generated with: $ git grep -l '^extern.*(' '**.h' | \ grep -v libjudy | \ grep -v 'sqlite3.h' | \ xargs sed -i -e 's/extern \(.*(.*$\)/\1/' This is a NFC. commit 284d5450ec938b667db9985aca6d3cd02b96487f Author: netdatabot <[email protected]> Date: Sat Oct 8 00:23:06 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-185-nightly. commit 294390f3b7514e8e23f3eb840be7eb25de442d6f Author: Ilya Mashchenko <[email protected]> Date: Fri Oct 7 12:33:26 2022 0300 feat: disable cloud if `NETDATA_DISABLE_CLOUD` is set to 1 (#13106) commit 45ff154cbd657f72b333ffb7cb15ac4d0a1b1ec5 Author: netdatabot <[email protected]> Date: Fri Oct 7 00:24:54 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-183-nightly. commit 0dfa82ec8e2072a3be07b67e7f3f234d045b54bf Author: Timotej S <6674623 [email protected]> Date: Thu Oct 6 16:56:11 2022 0200 Bump websockets submodule (#13776) * update mqtt websockets submodule and reflect API changes commit 1c73562dda318bb99f888ba5a0ea5dab74c81d4f Author: thiagoftsm <[email protected]> Date: Thu Oct 6 14:43:47 2022 0000 Rename variable for old CentOS version (#13775) commit a42c1e125341e341788b4ab76bcab74ad2d3ef79 Author: Ilya Mashchenko <[email protected]> Date: Thu Oct 6 15:28:58 2022 0300 bump go.d.plugin v0.40.3 (#13771) commit 981acd25e01c0b18ae046bc6864550ecb95dc7ed Author: Austin S. Hemmelgarn <[email protected]> Date: Thu Oct 6 07:34:56 2022 -0400 Overhaul handling of installation of Netdata as a system service. (#13451) * Install templated files from system directory on target system. This will allow more robust handling of auto-updates and installation of service files. * Add a script to handle installing Netdata as a system service. This uses the files installed as a result of the previous commit, and provides more robust detection and system handling than the existing code used for this purpose. A subsequent commit will convert the various installation mechanisms to use this script instead of their own internal code for this purpose. * Assorted cleanup and fixes for install-service.sh script. * Use new service install script on installs when present. * Fix missing .gitignore line. * Fix install command. * Integrate with warning handling in kickstart script. * Fix systemd version check. * Explicitly exit successfully when done. * Further fixes. * Fix handling of start commands on service install. * Fix handling of passing service commands in installer. * Fix handling of inability to detect service manager type. * Centralize install-service.sh invocation and improve error reporting. * Fix typos in Linux service handling. commit 1bf4c77f8ec16a6c473edb11ba3f11d7386dd584 Author: netdatabot <[email protected]> Date: Thu Oct 6 00:22:14 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-178-nightly. commit ecbf10c87ca0d4296ecb0cf8e485b68a15ba9613 Author: Costa Tsaousis <[email protected]> Date: Wed Oct 5 14:59:25 2022 0300 fix bad merge (#13764) * fix bad merge * fix for ssl_security_cert commit 8fc3b351a2e7fc96eced8f924de2e9cec9842128 Author: Costa Tsaousis <[email protected]> Date: Wed Oct 5 14:13:46 2022 0300 Allow netdata plugins to expose functions for querying more information about specific charts (#13720) * function renames and code cleanup in popen.c; no actual code changes * netdata popen() now opens both child process stdin and stdout and returns FILE * for both * pass both input and output to parser structures * updated rrdset to call custom functions * RRDSET FUNCTION leading calls for both sync and async operation * put RRDSET functions to a separate file * added format and timeout at function definition * support for synchronous (internal plugins) and asynchronous (external plugins and children) functions * /api/v1/function endpoint * functions are now attached to the host and there is a dictionary view per chart * functions implemented at plugins.d * remove the defer until keyword hook from plugins.d when it is done * stream sender implementation of functions * sanitization of all functions so that certain characters are only allowed * strictier sanitization * common max size * 1st working plugins.d example * always init inflight dictionary * properly destroy dictionaries to avoid parallel insertion of items * add more debugging on disconnection reasons * add more debugging on disconnection reasons again * streaming receiver respects newlines * dont use the same fp for both streaming receive and send * dont free dbengine memory with internal checks * make sender proceed in the buffer * added timing info and garbage collection at plugins.d * added info about routing nodes * added info about routing nodes with delay * added more info about delays * added more info about delays again * signal sending thread to wake up * streaming version labeling and commented code to support capabilities * added functions to /api/v1/data, /api/v1/charts, /api/v1/chart, /api/v1/info * redirect top output to stdout * address coverity findings * fix resource leaks of popen * log attempts to connect to individual destinations * better messages * properly parse destinations * try to find a function from the most matching to the least matching * log added streaming destinations * rotate destinations bypassing a node in the middle that does not accept our connection * break the loops properly * use typedef to define callbacks * capabilities negotiation during streaming * functions exposed upstream based on capabilities; compression disabled per node persisting reconnects; always try to connect with all capabilities * restore functionality to lookup functions * better logging of capabilities * remove old versions from capabilities when a newer version is there * fix formatting * optimization for plugins.d rrdlabels to avoid creating and destructing dictionaries all the time * delayed health initialization for rrddim and rrdset * cleanup health initialization * fix for popen() not returning the right value * add health worker jobs for initializing rrdset and rrddim * added content type support for functions; apps.plugin permanent function to display all the processes * fixes for functions parameters parsing in apps.plugin * fix for process matching in apps.plugiin * first working function for apps.plugin * Dashboard ACL is disabled for functions; Function errors are all in JSON format * apps.plugin function processes returns json table * use json_escape_string() to escape message * fix formatting * apps.plugin exposes all its metrics to function processes * fix json formatting when filtering out some rows * reopen the internal pipe of rrdpush in case of errors * misplaced statement * do not use buffer->len * support for GLOBAL functions (functions that are not linked to a chart * added /api/v1/functions endpoint; removed format from the FUNCTIONS api; * swagger documentation about the new api end points * added plugins.d documentation about functions * never re-close a file * remove uncessesary ifdef * fixed issues identified by codacy * fix for null label value * make edit-config copy-and-paste friendly * Revert "make edit-config copy-and-paste friendly" This reverts commit 54500c0e0a97f65a0c66c4d34e966f6a9056698e. * reworked sender handshake to fix coverity findings * timeout is zero, for both send_timeout() and recv_timeout() * properly detect that parent closed the socket * support caching of function responses; limit function response to 10MB; added protection from malformed function responses * disabled excessive logging * added units to apps.plugin function processes and normalized all values to be human readable * shorter field names * fixed issues reported * fixed apps.plugin error response; tested that pluginsd can properly handle faulty responses * use double linked list macros for double linked list management * faster apps.plugin function printing by minimizing file operations * added memory percentage * fix compatibility issues with older compilers and FreeBSD * rrdpush sender code cleanup; rrhost structure cleanup from sender flags and variables; * fix letftover variable in ifdef * apps.plugin: do not call detach from the thread; exit immediately when input is broken * exclude AR charts from health * flush cleaner; prefer sender output * clarity * do not fill the cbuffer if not connected * fix * dont enabled host->sender if streaming is not enabled; send host label updates to parent; * functions are only available through ACLK * Prepared statement reports only in dev mode * fix AR chart detection * fix for streaming not being enabling itself * more cleanup of sender and receiver structures * moved read-only flags and configuration options to rrdhost->options * fixed merge with master * fix for incomplete rename * prevent service thread from working on charts that are being collected Co-authored-by: Stelios Fragkakis <52996999 [email protected]> commit 6850878e697d66dc90b9af1e750b22238c63c292 Author: vkalintiris <[email protected]> Date: Wed Oct 5 10:11:12 2022 0300 Remove anomaly detector (#13657) * Move all dims under one class. * Dimension owns anomaly rate RD. * Remove Dimension::isAnomalous() * Remove Dimension::trainEvery() * Rm ml/kmeans * Remove anomaly detector The same logic can be implemented by using the host anomaly rate dim. * Profile plugin. * Revert "Profile plugin." This reverts commit e3db37cb49c514502c5216cfe7bca2a003fb90f1. * Add separate source files for anomaly detection charts. * Handle training/prediction sync at the dimension level. * Keep multiple KMeans models in mem. * Move feature extraction outside KMeans class. * Use multiple models. * Add /api/v1/ml_models endpoint. * Remove Dimension::getID() * Use just 1 model and fix tests. * Add detection logic based on rrdr. * Remove config options related to anomaly detection. * Make anomaly detection queries configurable. * Fix ad query duration option. * Finalize queries in all code paths. * Check if query was initialized before finalizing it * Do not leak OWA * Profile plugin. * Revert "Profile plugin." This reverts commit 5c77145d0df7e091d030476c480ab8d9cbceb89e. * Change context from anomaly_detection to detector_events. commit 2b7a964d49df6deda32bffbe6141ec53429d68fd Author: netdatabot <[email protected]> Date: Wed Oct 5 00:25:00 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-174-nightly. commit 70c40ef0f133b6fe13d1b699fd615edeaa800f44 Author: Ilya Mashchenko <[email protected]> Date: Tue Oct 4 18:50:57 2022 0300 add 1m delay for tcp reset alarms (#13761) commit 78327ae80f5e7020b74d3b845fd10963fbb5374a Author: Emmanuel Vasilakis <[email protected]> Date: Tue Oct 4 18:25:12 2022 0300 Use /bin/sh instead of ls to detect glibc (#13758) * use ln instead of ls to detect glibc * use /bin/sh * remove echo commit 08c3d24f79a14722e5a60998a2b487218abd54b9 Author: Vladimir Kobal <[email protected]> Date: Tue Oct 4 15:08:24 2022 0300 Add ZFS rate charts (#13757) commit 50f8339073dfeac3e6d59eec351e23a4e897bb4e Author: Pulseeey <[email protected]> Date: Tue Oct 4 09:12:05 2022 0100 Add CloudLinux OS detection to the updater script (#13752) commit b37cdf9cef83e5212d1715eef9e8127ddd1c6d17 Author: netdatabot <[email protected]> Date: Tue Oct 4 00:24:23 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-169-nightly. commit f05707fa17369d51689ec8fa76df1e2778f9d9c2 Author: Pulseeey <[email protected]> Date: Mon Oct 3 17:19:28 2022 0100 Add CloudLinux OS detection to kickstart (#13750) commit 31a46c5412df7d093267364fdf32cef368b1b939 Author: Andrew Maguire <[email protected]> Date: Mon Oct 3 16:47:36 2022 0200 add node level AR based example (#13684) add node level anomaly rate based example commit 6c5418016b419ef3ae96b8af81835b2f81a93cf4 Author: Stelios Fragkakis <52996999 [email protected]> Date: Mon Oct 3 11:43:53 2022 0300 Fix streaming crash when child reconnects and is archived on the parent (#13754) Remove unused RRDIM_FLAG_ACLK Create dictionaries that may have been deleted commit 8115a7055503b620b4c94250e17a04d43e23aef6 Author: netdatabot <[email protected]> Date: Sat Oct 1 00:26:28 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-165-nightly. commit 74fc4831bfcee8b87f194b54adc2a9245e207a86 Author: DShreve2 <[email protected]> Date: Fri Sep 30 10:51:00 2022 -0400 Provide Details on Label Filtering/Custom Labels (#13745) * Provide Details on Label Filtering/Custom Labels Adding documentation for the new Chart Label Filtering feature and custom labels for collectors. * Add screenshots to label filtering * Update docs/dashboard/interact-charts.mdx Co-authored-by: Hugo Valente <82235632 [email protected]> Co-authored-by: Hugo Valente <82235632 [email protected]> commit 0e2033c743163f9b1193666a8d7e04e28ea2fc2d Author: Austin S. Hemmelgarn <[email protected]> Date: Fri Sep 30 09:17:10 2022 -0400 Add info for Docker containers about using hostname from host. (#13685) * Add info for Docker containers about using hostname from host. By directly mounting `/etc/hostname` into the container read-only. * Update packaging/docker/README.md Co-authored-by: DShreve2 <[email protected]> * Update packaging/docker/README.md Co-authored-by: Ilya Mashchenko <[email protected]> Co-authored-by: DShreve2 <[email protected]> Co-authored-by: Ilya Mashchenko <[email protected]> commit a8b4b1ec81a17cdbdf3062b5e2265dc88f60ba5f Author: Ilya Mashchenko <[email protected]> Date: Fri Sep 30 15:53:58 2022 0300 bump go.d v0.40.2 (#13747) commit 209e2c0caffa415cbd2ba18e2490491a0eeeb1e7 Author: Ilya Mashchenko <[email protected]> Date: Fri Sep 30 10:26:26 2022 0300 fix(python.d): set correct label source for _collect_job label (#13746) commit 95b9336c46fc62352dd8c58451bd608a779338c2 Author: netdatabot <[email protected]> Date: Fri Sep 30 00:27:45 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-160-nightly. commit cc2586de697702f86a3c34e60e23652dd4ddcb42 Author: Austin S. Hemmelgarn <[email protected]> Date: Thu Sep 29 10:14:54 2022 -0400 Fix handling of temporary directories in kickstart code. (#13744) Instead of checking whether the `$tmpdir` variable is set and short-circuiting if it is, we should instead check if the directory it references exists. This ensures that code that needs to use temporary directories can do so even if earlier code calls `cleanup()` commit 22cc4d60d1af3a5e06e600307d50fc99d5409056 Author: netdatabot <[email protected]> Date: Thu Sep 29 00:26:22 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-158-nightly. commit 95cf9a870213c3561e88578c46528511ab691dbc Author: Emmanuel Vasilakis <[email protected]> Date: Wed Sep 28 20:35:01 2022 0300 Dont send NodeInfo during first database cleanup (#13740) commit 0190a1b556a78e50c5507bbfdbe34c63ebb56931 Author: thiagoftsm <[email protected]> Date: Wed Sep 28 11:42:21 2022 0000 Change cast to remove coverity warnings (#13735) commit 8e886651f1f39e5281f8e19f3cd9cb65772d2e27 Author: Timotej S <6674623 [email protected]> Date: Wed Sep 28 11:12:44 2022 0200 CMake - add possibility to build without ACLK (#13736) make aclk optional in CMake commit 373c97d3b5cf488a95ed4b5ec99bfba326219df4 Author: netdatabot <[email protected]> Date: Wed Sep 28 00:25:34 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-154-nightly. commit f89f8845252370c70403c1f3badb26518aa63182 Author: Timotej S <6674623 [email protected]> Date: Tue Sep 27 18:31:24 2022 0200 Remove Chart/Dim based communication (#13650) Co-authored-by: Stelios Fragkakis <52996999 [email protected]> commit 3ea35e2dcd32e8fa7958298798608ff63798575f Author: netdatabot <[email protected]> Date: Tue Sep 27 00:25:30 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-152-nightly. commit 71cb1ad68707718671ef57c901dfa2041f15bbe6 Author: thiagoftsm <[email protected]> Date: Mon Sep 26 13:49:56 2022 0000 Fix warnings during compilation time on ARM (32 bits) (#13681) commit 570a716100f313026c127e9dbf3b9c65e423e3a3 Author: Stelios Fragkakis <52996999 [email protected]> Date: Mon Sep 26 15:56:35 2022 0300 Do not try to start an archived host in dbengine if dbengine is not compiled (#13724) commit c4c02e19cbbb7a28eeffeb0998e57b071aead12d Author: Timotej S <6674623 [email protected]> Date: Mon Sep 26 13:54:45 2022 0200 Use CMake generated config.h also in out of tree CMake build (#13692) Use generated config.h also in out of tree build commit a1758307f9924a2c8a942aae258ff26a77329cae Author: Vladimir Kobal <[email protected]> Date: Mon Sep 26 10:24:36 2022 0200 Update exporting unit tests (#13706) commit 8ad75bbdfd6ec9a69206133c3f7e138236acb463 Author: netdatabot <[email protected]> Date: Sun Sep 25 00:21:31 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-147-nightly. commit 12cd3988949b604af74c28c6424b7753cf3a5069 Author: Costa Tsaousis <[email protected]> Date: Sat Sep 24 14:10:46 2022 0300 Faster streaming by 25% on the child (#13708) * faster printing of BEGIN, SET, END * fewer conditions * faster buffer_fast_strcat() * faster buffer_fast_strcat() fix * eliminate atomic operations and conditions in the BEGIN, SET, END flow * removed unecessary condition commit 5e0d532e19ad338dd499bad49773d51d797b45d6 Author: Ilya Mashchenko <[email protected]> Date: Sat Sep 24 12:50:39 2022 0300 feat(health): add new Redis alarms (#13715) commit bbd980845c33e50822650fb9f7d812ec3d430aa0 Author: netdatabot <[email protected]> Date: Sat Sep 24 00:20:24 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-144-nightly. commit 1625c0718fb2fa5c8adccc90c1f330f565e823f3 Author: Tomáš Kopal <[email protected]> Date: Fri Sep 23 20:03:50 2022 0200 Specify paths to source files for out-of-tree build. (#11475) * Specify paths to source files for out-of-tree build. * Update also dashboard makefile generation script. * Sync submodules with master commit d2a3975d0e60aec5f8316debfff11500d9480ef8 Author: vkalintiris <[email protected]> Date: Fri Sep 23 13:02:36 2022 0300 Do not create train/predict dimensions meant for tracking anomaly rates. (#13707) commit c82893f2a3230e0b6d8deaccd29119ea50cf36d7 Author: netdatabot <[email protected]> Date: Fri Sep 23 00:19:53 2022 0000 [ci skip] Update changelog and version for nightly build: v1.36.0-141-nightly. commit 0b21bddae065ea4187484eddf408fa6e7c3d7346 Author: Ilya Mashchenko <[email protected]> Date: Fri Sep 23 00:47:44 2022 0300 bump go.d.plugin to v0.40.1 (#13704) commit 06368384c10e81d15b2c5408cfeff0a29f65b47f Author: Timotej S <6674623 [email protected]> Date: Thu Sep 22 17:31:55 2022 0200 Build judy even without dbengine (#13703) always build judy commit 4453613cba6d939f74e982c1d0e8d85cef8bd6be Author: Andrew Maguire <[email protected]> Date: Thu Sep 22 14:50:55 2022 0100 alarms collector: ability to exclude certain alarms via config (#13701) alarms collector: ability to exclude certain alarms via config
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR tracks all
mallocz(),callocz(),reallocz(),strdupz()andfreez()calls, per function calling them, and generates 4 charts in the Netdata Monitoring section:To enable it, compile netdata with
-DNETDATA_TRACE_ALLOCATIONS=1. It is slower that running without it, so it is only available as a compile time option.It works like this:
While implementing this, I found that a few modules of netdata do not use these functions, or mix these functions with the libc ones:
freez()withmalloc()(instead ofmallocz()) - @underhood please fix this - You will also using thefreez()as a function pointer. This is not possible sincefreez()is now a macro. So, I created 2 wrapper functions for you. I receive lines like these at error.log:pointer 0x55556e84f440 is not our pointer (called freez_int() from 19@aclk/aclk_tx_msgs.c, freez_aclk_publish5a()).What they mean is that you are attempting tofreez()a pointer that has not been allocated with any of thezfunctions.Example charts generated:
Why is this important?
It all started this this forum thread: https://community.netdata.cloud/t/insane-netdata-memory-usage/3342
The user believes our memory footprint is insane.
He is right. We do need a lot of memory with every single module enabled.
As I see it, the main problem is that are failing to explain to our users how they should use Netdata. Netdata has really a lot of functions and modules and of course it needs memory when all of them are enabled.
So I plan to use this PR to create the necessary documentation, with facts, of what exactly is happening and what is the suggested deployment option for every case.