We should create an integration test that splits up a big file or it performs an expected mathematical operations on generating records, in a way that the output can be inferred and checked.

e.g.:

• splits up a 100MB video file into several 1MB files, sends to the commit log, re-assemble the file and checks md5 of the original and the re-assembled.
• create a series of events that have random numbers, sum them from the source, then send the numbers as records through the stream and sum then on the output of the stream. Check if they are the same.

Acceptance criteria

Have a test that ensures consistency of the commit-log running on every build.