How to create the content hash for a Dropbox file upload in Elixir

Donald Feury -

Dropbox has a specific way they want you to calculate the content hash for a file you are uploading. You can find the instructions here but to summarize:

  • Split the file into blocks of 4 MB (4,194,304 or 4 * 1024 * 1024 bytes). The last block (if any) may be smaller than 4 MB.
  • Compute the hash of each block using SHA-256.
  • Concatenate the hash of all blocks in the binary format to form a single binary string.
  • Compute the hash of the concatenated string using SHA-256. Output the resulting hash in hexadecimal format.

We can convert each of these steps into parts of a pipeline!

First we need to start reading in a file, split into chunks of the specified size. File.stream! is a good way of achieving this.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)

Next, we need to create a new hash from each chunk. Iterating over the stream chunks with Enum.map and passing those chunks into the Erlang module :crypto.hash will get us our chunk hashes.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))

After that, we'll combine all individual hashes into one big chunky string of hashes. Enum.join will let us that list of strings and slap them together

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()

Getting warmer, time to take that big boy and get one final hash using :crypto.hash again. We'll have to wrap it inside an anonymous function since we need to pass the string in as the second argument.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
|> (&:crypto.hash(:sha256, &1)).()

Home stretch! Lastily, base 16 encode the final hash using Base.encode16. Don't forget to lowercase it by passing in case: :lower.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
|> (&:crypto.hash(:sha256, &1)).()
|> Base.encode16(case: :lower)

There ya go! You should now be able to create the content hash for a file that you intend to upload to Dropbox.