This is WIP framework to load test redis cluster(s). The idea is to have an end to end setup that allows configurable load going to either a single or multiple Redis Clusters.
Performance of applications and services are highly dependent on the data being used. One will get different set of performance numbers if KBs of data is being used versus MBs of data.
There are hardcoded paths and strings
in this code base atm - it will be removed in later cleanup cycles.
This framework uses Dataflow
Streaming Data Generator Template
to generate fake data for performance tests. Once we have fake data,
in the examples included, we convert this JSON data to CSV - this step is completely optional and one could work with JSON files directly.
We control Writing to and Reading from Redis via Pub/Sub. We publish below messages for starting the write and read flows.
Begin Write:
start-mme:run-0504-1040:10.40.208.21:10000:60:mme_csv:mme_minute_dropzone:60
This start write message to pubsub
is then parsed by the start-write
cloud function as below:
tokens = message.split(":")
if(tokens[0] != "start-mme"):
return
run_id = tokens[1]
redis_ip = tokens[2]
chunk_size = tokens[3]
max_threads = tokens[4]
source = tokens[5]
destination = tokens[6]
files_to_copy = int(tokens[7])
where,
redis_ip = not in use
chunk_size= size in which to divide a file into
max_threads = number of parallel threads readind file chunks and uploading data to Redis
source = source gcs bucket to read data files from
destination = destination gcs bucket to write files to at a given frequency
files_to_copy = number of files to copy every min (hard coded for now)
Begin Read:
start-ipfr:run-0504-0803:mme_csv:fake_ipfr:75
This start read message to pubsub
is then parsed by the start-read
cloud function as below:
tokens = message.split(":")
if(tokens[0] != "start-ipfr"):
return
run_id = tokens[1]
source = tokens[2]
destination = tokens[3]
files_to_copy = int(tokens[4])
where,
source = source gcs bucket to read data files from
destination = destination gcs bucket to write files to at a given frequency
files_to_copy = number of files to copy every 2 mins (hard coded for now)
Important to note that the chunk_size
and max_threads
is not currently passed to read flow and are hardcoded in the read-mme-redis
cloud fucntion.
We use a unique run-id
for each performance test to identify metrics for that run and also to copy the same set of files with different names to avoid
creating new files for each run.
The strat-write
function writes part of the config it received from pubsub
message to a secret manager
key. This is key is then read by subsequent function that actually reads N files per min and writes to Redis.
This framework uses ZADD
i.e. Sorted Set
writes and ZREVRANGE
for reads. In future, the framework would make that configurable as well. In the example data used for performance tests - a total of max 100 bytes
of data is sent/read from redis for each record in a file.
A few things to note
- processing all files in a process does not scale
- processing a file line by line does not scale
- processing a file in a single thread does not scale
- writing or reading records/keys one at a time from Redis does not scale.
- data size dictates the max throughput we can get out of Redis (And network as well)
- several hundread thousand per second load may not work on a single cluster. Multuple clusters are needed.
In this framework, several optimisation are used:
- Each file invokes a cloud function to process it. (via triggers on gcs buckets)
- Each cloud function devides the incoming files into chunks of certain size to process it.
- Each cloud function uses multiple threads to process file chunks in parallel.
- Redis connection pooling is used to avoid creating connections on the fly.
- Redis pipeline is used to batch writes and reads together.
- Consistent hashing is used to distribute load on multiple Redis Clusters.
- Memoerystore Redis' write and read endpoint are used.
- Writes are only sent to the write endpoint
- Reads are sent to read endpoint which in turn takes care of balacing it across available read replicas.