HUGE 📄file processor

1. Automate Community
2. Tutorials & Examples

This shows how to read and process large amounts of data without causing an overflow. Thanks to buffering, loading data from storage won't blow up the memory heap.

The example data processor here is a bit-flipper/byte-inverter (I call it "NOTter"), it inverts all bits of each and all bytes. Of course you can use buffering for something else, like encryption, regex find-&-replace, encoding, decoding, parsing, serializating, etc...

A[1] (buffer size) is specified in Bytes. It can be any positive multiple of 4 (because of the unrolled loop), but it's recommended to be a power of 2 larger than 256. A[1] defines DD's block size, and this flow's max `inp` string size. Bigger is faster, but it can have diminishing returns.

A[2] is the output filename. A[3] is the temporary file used to load the buffer. Don't invert an inverted file while the original (non-inverted) file is still in the same directory with the same name. This flow always appends data instead of overwriting. If you stop the flow while it's processing, the temporary file won't be deleted.

Please understand that AM is VERY SLOW even for 2MB files, especially when iterating over bytes instead of DWords (32bit) or QWords (64bit). This flow iterates over QW to increase performance like loop unrolling.

If the file size is larger than 2^53B and A[1] = 1, then A[0] (the chunk index) will overflow causing an infinite loop that repeatedly inverts and appends the same chunk to the file. But because A[1] = 2^18, this overflow will only happen if the file is larger than 2^71B (because 53 + 18 = 71). Most Android versions have a file system that supports 4GB (2^32B) files, so all these files are safe to use.

Piping `dd` output to `od`, `xxd`, or `base64` cmds, is bad for backwards compatibility and memory allocation (even though storage R/W is improved), so I avoided their use. Also AM's B64 decoder doesn't support custom charsets like ISO-8859-1, so trying to decode B64 corrupts data. To avoid this, `xxd` and hexDecode() must be used, but memory allocation gets even worse.

This update makes use of branchless programming, loop unrolling, in-place mutation of array (instead of immutable string concatenation), and other optimization techniques to increase performance even more.

For some reason, repeatedly creating empty arrays (while deleting the old one) AND appending data to it, is SOMEHOW FASTER than reusing the same array. I suspect this happens for 2 reasons: 1. The tracking index adds more blocks to be executed in the loop. 2. Using too much memory for an extended period may increase the garbage-collection frequency, thus stealing CPU cycles

Rudxain profile: llamalab.com/automate/community/flows/42921