QED Release Candidate 2

QED Release Candidate 1

Changelog

4e99e75 Fix goreleaser environment variables
9ecea95 Feat: Split clean and task features.
5bfbf6c Warning envsubst is required for some scripts
ae81013 Add license headers
c42895d Fix typo
55513df Add periodic health checks
39c3f12 Set max retries to 1 in agent's client
96994f1 Set SilenceUsage at Root cmd level
941a913 Remove unused dependency
144ed31 Update licenses in NOTICE file
bca61c9 Remove all code references to Badger
f5d71e5 Move up client configuration
4c5ae44 Fix: delete old condition from deployment process
67bf1c4 Change auditor's client config.
af25d17 Add more comments
57d474b Merge pull request #104 from gdiazlo/agents_platform
04ce872 Fix RocksDB cache level size
85b4389 Fix: use metrics counters
8fb9d65 Fix: prometheus agent metrics port
d08fad8 register taskFactory metrics in agent metrics server if it present
8f967d0 Fix: server and agents deployment on AWS without config file
ec66aac AWS deploy: add panel with disk throughput in grafana dashboard
3348034 increase timeout in endpoint test
8e1e874 Update tests to reflect latest changes
a089e95 Handle cancel function in metrics server shutdown
87a9b0f remove old e2d code
c8a5544 AWS deploy: add panel with disk throughput in grafana dashboard
f973299 Add var to allow cflags usage in go runs inside the script
b004e3c Remove unneeded debug messages
e3ca384 Configure QED client for monitor agent needs
5a400d2 Fix default endpoint in client config
e3c511b Fix version checking in balloon consistency query
db80337 remove old agents code
d63a6c1 Increase batch size to 500, as the qed log performance has increased.
4f3ad48 REmove old test_service e2e implementation
cf1b2a0 Check for tasks errors in batchProcessor
397d95d Add error to message bus publish api
9161b1f Update comments
42a152f Add default configurations for simple implementations of notifier, store and taskmanager
d36dc3d Update start scripts with new options
e86fec6 MOve test_service to notifierstore in testutils
2a5a133 Update startup scripts to support new cli options
dfc0417 Move test service to tests folder
d6a8b68 Add publisher agent
2a256fd Mark event as required option for qed client add subcommand
8c11b0a Remove old files
59df917 Add documentation to config options for the cli
940a34b Comment old testing code until it is refactored with the new gossip api
542d9f0 Remove metadata from BatchSnapshots as it is now in gossip.Message
4810039 Change internal server error by bad request errors.
066b569 Server start() does not block. Use stop() to shutitdown
8729a04 Include sender into the server package.
58844f9 Objects with start / stop should not block
6ec665d Update scripts to launch qed components
40d10bb Redesign cli flag parsing. Use new library to generate options from config objects
8a610d9 Redesign gossip package to refine the API
eb4da9b Refactor of agents: new snapshot sore client, task manager, and agent api
ac07447 Fix version metric.
4d95a59 Remove DiscoveryTimeout config option.
feb5da1 Add discovery tests
da04138 Add protocol for /info/shards endpoint
8dee396 Enable AttemptToReviveEndpoints option in agent and riot clients
372ec41 Add log_stream_name sufix terrraform workspace
1e16a79 Profiler now creates a result dir per IP addr
032a44e AWS deploy: organize RocksDB dashboard a bit better
be92e87 AWS deploy: update grafana QED dashboard
b777d3c AWS deploy: make a couple of dashboards dynamic
eb249d6 AWS deploy: add dashboard with RocksDB tables
8552870 AWS deploy: fix metric names in grafana panels
9b0d567 tests/start_profiler: enable custom IP profiling
4dda40c Ability to enable/disable profiling in AWS installation
f59e064 Profiling now listen at *:6060
5a0b48e Ease requirements for profiling
fd426ab Merge pull request #102 from aalda/metrics
3d571ac Use String() for default table
3d2625d Fix metrics registering
82f9617 Remove unused metric
5bf1c05 Register all pending metrics
4adc086 Fix previous definitions
1d61f47 Declare metrics at raft balloon level
79ae579 Move balloon metrics to corresponding package
1652a0c Remove unused metric definitions
5b85e8a Move history tree metrics to the corresponding package
3c3a77a Fix metrics
0aca63e Move test metrics server to testutil package
8a53278 Expose per storage table metrics
9178532 Export DB properties
f72e430 AWS deploy: add storage metrics to graphana dashboards
b85daa7 AWS deploy: set graphTooltip to 2 in all occurrences for consistency
d94a90b Improve deploy to enable custom resource targets
6c972e0 Add parallel test for raft node
e711d8c Enable profiling
101eb15 Feat: Change QED instance types to z1d.xlarge
eec548e Merge pull request #101 from suizman/master
a9d8564 Add Riot support for list of QED IP addr
7669721 Expose block cache usage
e843593 Unfix max open files
9042614 Activate partitioned index filters.
66df51e Fix help message in metric
b502bd0 Add benchmark for FSM apply method
f7a4b63 Reduce write buffer size and number for hyper column family
fb625ff Add decisions comments
78526b3 Increase block cache size to 8GB
9de65e2 Enable caching filter and indexes in block cache
d44935d Update use_cases.md
c1e7e6b Feat: update QED instance sizes
6163dfe Merge pull request #98 from aalda/agents_queue
33035b4 Move agents queue from FSM to RaftBalloon
f8a8bae Remove max_total_wal_size option
126eead Fix: qed_server_instances metric didn't show up. Dashboards improved.
9ac5a54 Set max WAL size for both rocksdb instances.
a941df5 AWS deploy: add more block Cache metrics to graphana dashboard
21a7e39 Merge pull request #97 from aalda/remove_index
e563f63 Feat: create QED isolated AWS deployments using terraform workspaces
0593929 Remove index table.
218289f Add riot openapi description
2bfcb14 test:start_server empty ip addr in params cause failures
b07a7d6 Refactor: place images into docs folder.
f6721a1 Restart QED service when config changes
bc3d1c2 Change read preference in agents.
1e2e68b Improve Ansible provision
fea7722 Fix agents config to use list of QEDUrls
fd1c0ad Set deploy riot to 1000M requests by default
e044788 Ignore modules dir changes
ee53275 Separate server metrics from default definitions
09efd47 Fix metrics registry in agents
9a6a9d9 Add more rocksdb metrics
06a1490 Feat: New metric in grafana dashboard - snapshot stored
cbe4d45 Fix: Dashboards wrong time range and refresh rate
28916f3 Feat: Add tags to keep VMs running.
c39064e Feat: Enable grafana systemd.
30240fe Fix metrics test
73cb1a0 Reset RocksDB stats after getting ticker
43ba855 Tune rocksdb options for every column family
4bd229f Feat: New storage dashboard
a48dea7 Fix deploy template pub_urls
f0fdbdb Fix test scripts for QED & QED Agents
d2b4e30 Add auditor specific config in deploy on AWS
90d56e5 Feat: Add storage metrics
cbc8d4b Refactor: naming in prometheus config and QED dashboard
55a1cfa Doc: refactor documentation
a23acfe add license headers
8d33e7a refactor: deploy process to use systemd
f682fee Add atomic flushes to rocksdb bindings
968442b Merge pull request #87 from aalda/column_families
d4a05f8 Remove repeated method
6e31394 Tune some rocksdb options per column family
248a2d7 Replace storage prefixes with tables
956cd55 Support for cross-compiling in provisioning files.
c80a4a5 Include clean script for c-deps
def4f8a Add Dockerfile to enable cross-compiling in MacOS.
85fca47 Add introductory documentation
7bf60f6 Remove tampering code
ebd1127 Integrate prometheus metrics with rocksdb
05b4135 Use pointers instead values to avoid locks copies
3d908d9 Merge pull request #85 from aalda/rocks_stats
6cae8c3 Fix rocksdb include path for CXXFLAGS
1da0db6 Configure azure pipelines to fail tests task on test failures
12c6586 Add missing godocs
ce186ff Add cgo bindings for RocksDB statistics
d3a6963 Fix typo in provisions
1ea7ad3 Configure metrics param for provision and config.example.yml
daa0759 Merge pull request #83 from gdiazlo/agents
5bee53e Remove sleeps and use client retries to handle the e2e async tests
840006a Register metrics in server start and move sender metrics to sender file
84d7a7f Refactor: generalize metrics, integrating server and agents metrics API. Sender and qed server share the same metrics server.
984e60d Fix: typo on log message
bf96bcd Fix: merge with client changes
11422e2 Add some scripts to use qed on development
57b4e88 Remove unneeded tests, while the agents functions get redefined
06b3e0d Add a retrier function to reduce the numhber of sleeps in the code
7ba2a45 Fix: change debug message without the use of deleted function
52b1900 Simplify sender, removing uneeded code. Make it send snaposhots as soon as they are enqueued
a6d896a Add metrics address configuration option to agents
a50f92f Add metrics server to be included on each agent, including methods to stop, start and register metrics
6842d8f Use freecache as storage to evict old items on long tests
233b329 Add metrics to each processor
c328890 Delete metrics and include each metric on the corresponding agent
17b699c Change processor interface to include register metrics method, so processors get their cusrtom metrics registered into the agent server
494ac97 Add alerts and metrics urls as separate options to the different agents
cdb526f Add metric server and remove queue code, refactor sendQueue to send a single batch
a59d02e Add metrics url parameter to agents, because now the metrics server resides in the agent code and not in the processor
4ee600e Change function call so it does not look like an struct
6cbe080 wip: change alertsUrl endPoint in commands and tests
baa047d Add alertsUrl as an standalone endpoint
a3b276f Update configuration options, make agent and config non public atributes, clean up design, implement out queue as a timedout channel, simplify stop code
e861b74 Add new metric to count total snapshots put in test_service
ed8434f Add deduplication of batches in gossip agents, to avoid excessive gossiping.
8677dbf Pass batch by reference to processors
b833486 Clean processor messages
5929023 Add timeout queue and proce...

Changelog

a787b2e Update archive files
c4818b6 Cleanup goreleaser options
b2e1de6 tests/riot.go include reqNumber to chart name
b33b7c3 Enable detailed profiling for uor stress test
3fb2646 Update goreleaser
76858c6 Update gomod
8f9d653 Add goreleaser
a7e16d3 tests/riot.go new can create basic useful graphs
b394903 tests/riot.go: new offload parameter
b89abff Change MULTINODE -> CLUSTER_SIZE in test scripts
c6d2371 tests/riot.go: dynamic read throughput control
6a36298 Add new stats to balloon
a04747c Use struct variables for cache
78117d8 Unlock history tree to enable parallel reads and writes
fd25b89 Add test to backup funcion, modify tools to use temp dirs for test badger stores.
f83cf0e Benchmark: Fix numRequests being defined in two places
b6e42ea Add test to check snapshot consistency.
67fa64c add balloon version to the debug message when generating snapshots
6e2f1c6 Add three bytes more to GetLast to use all the key length in the search.
d530619 Use BigEndian encoding as its whats badger use, so store.GetLastVersion works.
302e377 Fix comment in GetLastVersion
ded51cb Use ValidForPrefix() instead of Valid() because if Seek() does not found key , it will return the next key > searched_key. That makes GetLast to return values with other prefixes.
e605b5e Fix typo in log and return errors from refresh version call
473cc46 Update tests/start_server usage info
e4af609 Update var definition tests/riot.go
1247d94 Additional flags tests/riot.go, ie: read throughput
d4af239 Benchmark: Add -delta flag to tune the range of the IncrementalProof
4debf44 Pass config to tests/riot.go getVersion
0d7e300 Add badger version to fsm snapshots
ffc66cb Benchmark: Add IncrementalProof benchmark
8b3ec47 Benchmark: Drop stale comment
52deb5c Benchmark: coalesce singleNode() and multiNode() into membershipCluster()
f019f18 Add benchmark to apihttp package
1c94ec1 Add RaftBalloon unit benchmark
e12e87d Reduce the size of raft snapshots by only saving the last value of each key
ee53bb9 Add profiling options to test scripts
7220ae2 Cleanup reinitialization of variables
72199f9 Fix mispells
5a3569d Benchmark: Fix missing err var
a6c2c5a Benchmark: Support run with 4 followers
d004b0b Cleanup lock in trees
e401298 Fix usage of variable before check for errors
cfbb376 Fix sync.Mutex passed by value
0d4e215 add query membership unit benchmark
9bf3500 Add some metrics to Balloon and Trees
814c696 Rename traverseWithoutCaching method to traverseWithoutCollecting
a027523 Update QueryMembership to create tree proofs in parallel
20c25dc Update Get() to use recommendations from badger documentation
b7cd9a1 Benchmark: Enable single|multi node membership benchmark with a single script
8470ee6 Benchmark: Connect numRequests with balloonVersion