Problem opening server-sent events from firebase public dataset

[davoclavo]

Hi, awesome library!

I'm trying to use shotgun to read from firebase bitcoin event-source. If I try to read it using curl it works, but somehow can't get it to work with shotgun, it gets the first response with the content-type set as "text/event-stream" but the subsequent events never show up. Any ideas of what am I doing wrong, or could it be a bug?

$ curl "https://s-usc1c-nss-128.firebaseio.com/bitcoin.json?ns=publicdata-cryptocurrency&sse=true"
event: put
data: {"path":"/","data":{"_credits":"Powered By Coinbase.com","_updated":"Mon Nov 30 2015 01:13:45 GMT+0000 (UTC)","ask":"377.65","bid":"377.64","last":"377.64"}}

event: put
data: {"path":"/_updated","data":"Mon Nov 30 2015 01:14:01 GMT+0000 (UTC)"}
...
event: put
data: {"path":"/last","data":"377.65"}

The following example is in Elixir:

iex(37)> {:ok, conn_pid} = :shotgun.open('s-usc1c-nss-128.firebaseio.com', 443, :https)
{:ok, #PID<0.236.0>}
iex(38)> {:ok, response} = :shotgun.get(conn_pid, '/bitcoin.json?ns=publicdata-cryptocurrency&sse=true', %{}, %{async: true, async_mode: :sse})
{:ok,
 %{body: "event: put\ndata: {\"path\":\"/\",\"data\":{\"_credits\":\"Powered By Coinbase.com\",\"_updated\":\"Mon Nov 30 2015 01:06:17 GMT+0000 (UTC)\",\"ask\":\"377.23\",\"bid\":\"377.22\",\"last\":\"377.23\"}}\n\n",
   headers: [{"content-type", "text/event-stream"},
    {"cache-control", "no-cache"}, {"access-control-allow-origin", "*"}],
   status_code: 200}}
...
iex(39)> flush
:ok
iex(40)> :shotgun.events(conn_pid)
[]

and hopefully properly translated to Erlang, which I basically copied from the example on PR #115

F=fun() ->
  l_all(shotgun),
  ensure_all_started(shotgun),
  {ok, Pid}=shotgun:open("s-usc1c-nss-128.firebaseio.com", 443, https),
  io:format("~p~n", [shotgun:get(Pid,  "/bitcoin.json?ns=publicdata-cryptocurrency&sse=true", #{}, #{async => true, async_mode => sse})]),
  io:format("~p~n", [shotgun:events(Pid)]),
  shotgun:close(Pid)
end.

F().

[jfacorro]

@davoclavo Hi, David. Thanks for reporting this. Could you please share what version or commit are you using? Thanks!

[jfacorro]

@davoclavo After checking all the headers the firebase endpoint is returning, I can see the problem. shotgun currently only considers as a SSE response the ones that include the headers Transfer-Encoding: chunked. The RFC specifically mentions the text/event-stream as the content-type expected for SSE so I'm not sure how we have missed that 😖.

I will add this as a bug and fix it as soon as possible. Thanks!

[jfacorro]

This is what I found in cURL's source code regarding how to handle the special case where there is no Content-Length, Transfer-Encoding is not chunked and the connection should not be closed:

      ///...
      else {
        k->header = FALSE; /* no more header to parse! */

        if((k->size == -1) && !k->chunk && !conn->bits.close &&
           (conn->httpversion == 11) &&
           !(conn->handler->protocol & CURLPROTO_RTSP) &&
           data->set.httpreq != HTTPREQ_HEAD) {
          /* On HTTP 1.1, when connection is not to get closed, but no
             Content-Length nor Content-Encoding chunked have been
             received, according to RFC2616 section 4.4 point 5, we
             assume that the server will close the connection to
             signal the end of the document. */
          infof(data, "no chunk, no close, no size. Assume close to "
                "signal end\n");
          connclose(conn, "HTTP: No end-of-message indicator");
        }
      }
      ///...

[davoclavo]

Whoa, nice forensic work 😄

I tested the changes on your branch. I now successfully get a reference instead of the response body, and I am able to query for events. However, the connection seems to get closed right away, (or at least that's what I understand fin atom represents on the last event)

iex> :shotgun.events(conn_pid)
[{:nofin, #Reference<0.0.2.7188>,
  "event: put\ndata: {\"path\":\"/\",\"data\":{\"_credits\":\"Powered By Coinbase.com\",\"_updated\":\"Wed Dec 02 2015 01:33:55 GMT+0000 (UTC)\",\"ask\":\"352.53\",\"bid\":\"352.51\",\"last\":\"352.59\"}}"},
 {:fin, #Reference<0.0.2.7188>, ""}]

[jfacorro]

@davoclavo Indeed, I'm seeing the same behavior, I'm not sure if it is expected behavior (i.e. the server expects the SSE client to reconnect) or if it is something related to gun's implementation. I haven't had the time to look further into this. I started looking into cURL's source code so that maybe I could find the way it was handling this case, but the only thing I could find related to this was the snippet I pasted in the previous comment. I'm not sure when I will be able to work on this problem, I'm hoping before the year is over 😞.

[davoclavo]

@jfacorro Thanks for the update! On my end I'll try to understand how gun inners work.

[joustava]

Any update on this one?

I'm also trying to use this lib in a elixir/phoenix project, first issue I had is with mix not being able to resolve deps due to version mismatch of gun (if I remember correctly) so I just created a standalone mix app but there I seem not to get the gist of using this library (also quite new to elixir 😊 )

The sse endoint on the server seems to be ok as I can get events with curl. But running shotgun from iex -S mix results in the following. (I checked if the tokens and urls are correct)

iex(1)> {:ok, conn_pid} = :shotgun.open('someurl', 443, :https)
{:ok, #PID<0.169.0>}
iex(2)> {:ok, response} = :shotgun.get(conn_pid, 'url/api/endpoint', %{Authorization: "Bearer sometoken", Accept: "text/event-stream"}, %{timeout: 5000, async: true, async_mode: :sse})
** (MatchError) no match of right hand side value: {:error, {:timeout, {:gen_fsm, :sync_send_event, [#PID<0.169.0>, {:get_async, {:undefined, :sse}, {'url/api/endpoint', [Accept: "text/event-stream", Authorization: "Bearer sometoken"], []}}, 5000]}}}

iex(2)>

(removed original urls and token)
For the example I am using the branch containing the commit mentioned in this ticket.

I might be making some basic errors here though due to my elixir/erlang inexperience. Any feedback appreciated.

Thanks for the effort creating this btw!

[elbrujohalcon]

Hi @joustava. We haven't had the chance to dig deeper into this yet, sorry. Maybe it would be better for you to try using gun directly and/or contact @essen to see if he can figure this out. If you find a solution, please share it.

[essen]

The issue is in Gun. It's very possible that it does not behave properly when chunked is not used. Can you open a ticket there?

[essen]

I do not seem to see this issue with Gun master. I only get "keep-alive" events though.