blob.getMd5() doesn't work as expected

[niclarcipretti]

When I get any blob MD5 checksum using java API, the result is a Base64 string, that, when decoded, is not a MD5 string.

Expected result: the file MD5 checksum string.

Can anyone explain me why getMd5() returns a base64 of a MD5 instead of the MD5 itself?

Environment details

• OS: Linux
• Java version: 1.8.0_172
• google-cloud-java version(s): google-cloud-storage:1.53.0

Steps to reproduce

1. Save a blob using bucket.create(completeFilePath, fileByteArray)
2. Get the file metadata using storage.get(blobId)
3. Get the file MD5 using blob.getMd5()

[andrey-qlogic]

An MD5 value is always 22 (useful) characters long in Base64 notation.
The explanation is available here.
https://stackoverflow.com/questions/4278170/md5-hash-and-base64-encoding

[sduskis]

@frankyn do you have any comments on this?

[frankyn]

The GCS API returns the MD5 checksum as a Base64 encoded string and the Java library returns the value untouched. Following @andrey-qlogic's comment you can decode the MD5 checksum into a byte[].

As a follow-up we could add a helper to help users decode the MD5 checksum into a human readable hex string. As another reference gsutil and the Python library also return the Base64 encoded MD5 checksum. I think this is WAI but we may need to update docs to state the output format better.

Some rationale for the Base64 representation:

• It's easier to work with programmatically instead of having to convert a human readable representation to machine representation.
• The underlying response is using a JSON structure and binary data is usually Base64 encoded. There's more discussion at https://stackoverflow.com/questions/1443158/binary-data-in-json-string-something-better-than-base64

[niclarcipretti]

Hi @frankyn, thanks for the quick response! You guys are great!

I understand now, however, I've never seen an use case where you use the MD5's bytes to do your comparisons, and I`ve been out there for a while now. I guess your idea to add a helper (or simply a .getMd5String()?) would be much appreciated by the community.

I couldn't find in @andrey-qlogic`s comment a proper example on how to decode the value decoded from the base64 string, could you provide some example just for the record here, so other people could benefit?

Thanks again

[frankyn]

@niclarcipretti I've responded below:

I understand now, however, I've never seen an use case where you use the MD5's bytes to do your comparisons, and I`ve been out there for a while now. I guess your idea to add a helper (or simply a .getMd5String()?) would be much appreciated by the community.

Thanks for the feedback! I'm curious how are you using the MD5 checksum?

I couldn't find in @andrey-qlogic`s comment a proper example on how to decode the value decoded from the base64 string, could you provide some example just for the record here, so other people could benefit?

I found https://stackoverflow.com/questions/14413169/which-java-library-provides-base64-encoding-decoding to be a good place to start depending on your Java version.

To get a human readable format of the MD5 checksum using Java 8:

// Get a storage client
Storage storage = StorageOptions.getDefaultInstance().getService();

// Get Blob metadata including the MD5 Hash
// GCS client library requires that you select it explicitly
String bucketName = "bucket-name";
String blobName = "blob-name";
Blob blob = storage.get(bucketName, blobName, BlobGetOption.fields(Storage.BlobField.MD5HASH));

// Get MD5 Hash
byte[] decodedMd5 = Base64.getDecoder().decode(blob.getMd5());

// Convert into human readable string
for(byte b : decodedMd5){
   stringBuilder.append(String.format("%02x", b&0xff));
}
String humanReadableMd5 = stringBuilder.toString();

[niclarcipretti]

@frankyn my app uploads files to gcs. Sometimes, users try to upload files that are arealdy stored as a blob. So, I generate a MD5 checksum string after uploading a file and store it in my database, then, before a new upload of a file, I check it's MD5 with my database to prevent unecessary waste of money, once gcs charges for data transfering. Another use case I have is a file cache for mobile app, if the MD5 string that is stored on my app's cache diffs from the gcs one, then I download it again.

Thanks for your time and the example above.

[frankyn]

Thanks @niclarcipretti!

• For the first case, an alternative would be to have the GCS checksum stored in your database and compare against it instead of decoding it.
• For the second case, I think you're checksumming locally beforehand. How do you get the human readable string of the checksum?
  Possibly you could Base64 encode instead to match GCS's checksum value then compare.

I agree it isn't the easiest to compare against given the different formats. If anything it could help to reduce the number of times you Base64 encode and decode.

Alternatively, the GCS client could have the Md5ToString helper method. I suspect that will need to be addressed across libraries.

[niclarcipretti]

@frankyn personaly, I'm not a big fan of storing raw binary data in databases, it's just messy for querying and handling. But I see your point.

I guess I could help with a PR for the Java library, not so sure about the others though. Let me know if you need a hand with that.

Thanks ;)

[frankyn]

Thanks @niclarcipretti! Apologies, I meant to store the Base64 encoded string in the database instead of the raw binary.

Will relabel this as a FR.