PubSub JSON Double base64 Encode

[HakShak]

We recently upgraded from 0.18.1 to 0.21.0 and we do not use gRPC/GAX/Protobuf fun times, but we use the JSONPublisherApi.

Maybe we are using the topic.Batch().Commit() method wrong, but we have been using it in our retry strategy to just simply call Commit() again.

With the change in #2577 it seems that Commit() will double encode if you just call Commit() again on a network failure.

Is this the intended design? Are we supposed to keep another copy of the message batch or create another message batch?

[dhermes]

@HakShak You reference "the change in #2577" but that was actually a bug report of this same issue, which presumably was fixed in #2578. Can you give a snippet of code showing the issue?

[HakShak]

topic_batch = topic.batch()
topic_batch.publish(somedata, someattributes)

try:
    topic_batch.commit()
except NetworkErrorOfSomeKind:
    topic_batch.commit()
# Imagine a proper retry loop here.

Ultimately this: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0.18.1/gcloud/pubsub/topic.py#L442

Changed to this: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/pubsub/google/cloud/pubsub/_http.py#L206

The base64 encoding was done on Batch.Publish(), but now is done in Batch.Commit()

[daspecster]

I can test this more later on today.

We could return a copy of the messages from _transform_messages_base64(), instead of editing them in place, or we might have to flag the messages that are encoded already.

[HakShak]

We ended up copying the Batch.messages before trying commit. The flag seems more "efficient".