Cannot json.Unmarshall dynamodbstream.StreamRecord because of unix time set in field

[xoraes]

The dynamodbstream.StreamRecord struct has a time.Time field:
//ApproximateCreationDateTime *time.Time type:"timestamp" timestampFormat:"unix".

Its not convenient to json.Unmarshal unix Time in Go without writing a custom type for time. I am trying to use this struct to unmarshal stream records from dynamo, but that fails because json.Unmarshal cannot parse unix time by default. I get this error:

{"error":"parsing time "1470788940" as ""2006-01-02T15:04:05Z07:00"": cannot parse "1470788940" as """"}

Can we wrap the time fields into a struct and provide a marshaller and unmarshaller for parsing unix time ? something like this perhaps:

type UnixTime struct {
time.Time
}

func (t *UnixTime) UnmarshalJSON(buf []byte) error
func (t *UnixTime) MarshalJSON(buf []byte) error

Without this I have to maintain a copy of this struct in my code, which is bad in many ways.

[xibz]

Hello @xoraes, thank you for reaching out to us. Can you provide some example code of what it is that you are trying to do? From the looks of it, couldn't you just use a regular time.Time object then call time.Time.Unix() on it to get unix time, instead of storing it?

[xoraes]

Simply trying to unmarshal the dynamodbstream event in the Go json library.

type Event struct {
Records []*Record json:"Records"
}
type Record struct {
Dynamodb *dynamodbstreams.StreamRecord json:"dynamodb"
}

// this call is where the problem happens because StreamRecord has time.Time field
// which has unix time and cannot be unmarshaled by go native json unmarshaller.
if err := json.Unmarshal(jsonStr, &event); err != nil {
return nil, err

// this is not relevant to the problem but its what i want to do with the event
for _, record := range event.Records {
if record.EventName == "INSERT" {
// Now Unmarshal the data for NewImage
var rec interface{}
dynamodbattribute.UnmarshalMap(record.Dynamodb.NewImage, &rec)
}
}

[xibz]

Where is the jsonStr coming from?

[xoraes]

Comes dynamodb stream event. When our dynamo table is upserted, it sends a event back lambda in the following form. Thats what jsonStr represents.

{
  "Records": [
    {
      "eventID": "1",
      "eventVersion": "1.0",
      "dynamodb": {
       "ApproximateCreationDateTime": 1470788940,
        "Keys": {
          "Id": {
            "N": "101"
          }
        },
        "NewImage": {
          "Message": {
            "S": "New item!"
          },
          "Id": {
            "N": "101"
          }
        },
        "StreamViewType": "NEW_AND_OLD_IMAGES",
        "SequenceNumber": "111",
        "SizeBytes": 26
      },
      "awsRegion": "us-west-2",
      "eventName": "INSERT",
      "eventSourceARN": "arn:aws:dynamodb:us-west-2:account-id:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899",
      "eventSource": "aws:dynamodb"
    }
  ]
}

[xibz]

I see, thanks for the information, @xoraes. Yes, I believe this is a bug. I'll mark it as service API for now, until I fully test it for verification.

[xoraes]

Here is the code you can add. I can do a pull req but it looks like you guys are using some sort of code generator?

type Timestamp struct {
    time.Time
}
func (t *Timestamp) MarshalJSON() ([]byte, error) {
    ts := t.Time.Unix()
    stamp := fmt.Sprint(ts)
    return []byte(stamp), nil
}

func (t *Timestamp) UnmarshalJSON(b []byte) error {
    ts, err := strconv.Atoi(string(b))
    if err != nil {
        return err
    }
    t.Time = time.Unix(int64(ts), 0)
    return nil
}

[xibz]

@xoraes - the issue there is that would be a breaking change if it were added to the StreamRecord structure. We are always willing to take a look at PRs and really encourage them, but we need to make sure we aren't breaking existing customers using the feature. Since no one can use this feature, that change may be fine, except we autogenerate most of the services code.

[xibz]

@xoraes, it looks like the reason why this is failing there is a tag that defines it as unix and we handle that internally. So, if you use json marshallers, it will throw an error. So your wrapper for now is the best option. However, if we have autogenerated marshallers, I definitely can see that solving this issue. I'll mark this as a feature request for autogenerated marshallers.

[fsenart]

@xoraes @xibz we use similar solutions for the ApproximateArrivalTimestamp of Kinesis records. Notice that the timestamp has millisecond precision in these cases with a particular format:

// "sec"."msec"
1479269477.816

You have to take care of it in the UnmarshalJSON method:

func (t *Timestamp) UnmarshalJSON(data []byte) error {
    v, err := strconv.ParseFloat(string(data), 64)
    if err != nil {
        return err
    }

    sec := int64(v)
    nsec := int64((v - float64(sec)) * float64(time.Second))

    t.Time = time.Unix(sec, nsec)
    return nil
}