HUGE 📄file processor

by Ricardo Fernández Serrata

Version 9 (March 7, 2022)

Download (724 downloads)

This shows how to read and process large amounts of data without causing an overflow. Thanks to buffering, loading data from storage won't blow up the memory heap.

The example data processor here is a bit flipper or byte inverter (I call it "NOTter"), it inverts all bits of each and all bytes. Of course you can use buffering for something else, like encryption, regex find-&-replace, encoding, decoding, parsing, serializating, etc...

A[1] (Buffer Size) is specified in Bytes, not kiloBytes. It can be any positive multiple of 4 (because of the unrolled loop), but it's recommended to be a power of 2 larger than 256. A[1] defines DD's block size and this flow's max `inp` string size. Bigger is faster, but it can have diminishing returns.

A[2] is the output filename. A[3] is the temporary file used to load the buffer. Don't invert an inverted file while the original (non-inverted) file is still in the same directory with the same name. This flow always appends data instead of overwriting. If you stop the flow while it's processing, the temporary file won't be deleted

Please understand that AM is slow even for 2MB files, especially when iterating over individual bytes instead of DWords or QWords. This flow iterates over QWords to increase performance like loop unrolling.

If the file size is larger than 2^53B and A[1] = 1, then A[0] (the chunk index) will overflow causing an infinite loop that repeatedly inverts and appends the same chunk to the file. But because A[1] = 2^18, this overflow will only happen if the file size is larger than 2^71 (because 53 + 18 = 71). Most versions of Android have a file system that supports up-to 4GB (2^32B) files, so all these files are safe to use.

Piping dd output to od, xxd, or base64 cmds, is bad for retrocompatibility and memory allocation (even though storage R/W is improved), so I avoided their use. Also AM's B64 decoder doesn't support custom charsets like ISO-8859-1, so trying to decode B64 corrupts data. To avoid this, xxd and hexDecode must be used, but memory allocation gets even worse.

This update makes use of branchless programming, loop unrolling, in-place mutation of array (instead of immutable string concatenation), and other optimization techniques to increase performance even more

4.8 average rating from 9 reviews

5 stars
4 stars
3 stars
2 stars
1 star

Rate and review within the app in the Community section.