[gatsby-remark-images-contentful] intermittent build errors with/to base64

[floriangaechter]

Description

We're having an issue during the build phase of our project: sometimes the run graphql queries... step finishes without any issues and the site builds fine. But since recently the build process gets stuck at the ⡀ run graphql queries — 123/125 46.15 queries/second step and keeps either spinning forever or, if it finishes, takes a very long time (it doesn't always stop at 123. Sometimes it stops at 7/125 or other numbers).

Steps to reproduce

1. delete .cache folder
2. run gatsby build

Expected result

The build process should finish without issues.

Actual result

The build process gets stuck at the "run graphql queries..." phase.

Environment

  System:
    OS: macOS 10.14.3
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 11.10.0 - ~/.nvm/versions/node/v11.10.0/bin/node
    npm: 6.8.0 - ~/.nvm/versions/node/v11.10.0/bin/npm
  Languages:
    Python: 2.7.15 - /usr/local/bin/python
  Browsers:
    Chrome: 72.0.3626.109
    Firefox: 65.0.1
    Safari: 12.0.3
  npmPackages:
    gatsby: ^2.1.4 => 2.1.4 
    gatsby-image: ^2.0.29 => 2.0.29 
    gatsby-plugin-manifest: ^2.0.17 => 2.0.17 
    gatsby-plugin-offline: ^2.0.23 => 2.0.23 
    gatsby-plugin-postcss: ^2.0.5 => 2.0.5 
    gatsby-plugin-react-helmet: ^3.0.6 => 3.0.6 
    gatsby-plugin-remove-trailing-slashes: ^2.0.7 => 2.0.7 
    gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0 
    gatsby-plugin-sharp: ^2.0.20 => 2.0.20 
    gatsby-plugin-sitemap: ^2.0.5 => 2.0.5 
    gatsby-plugin-svgr: ^2.0.1 => 2.0.1 
    gatsby-remark-embed-video: ^1.7.0 => 1.7.0 
    gatsby-remark-images-contentful: ^2.0.8 => 2.0.8 
    gatsby-remark-responsive-iframe: ^2.0.9 => 2.0.9 
    gatsby-source-contentful: ^2.0.29 => 2.0.29 
    gatsby-source-filesystem: ^2.0.20 => 2.0.20 
    gatsby-source-hire-with-google: ^1.3.0 => 1.3.0 
    gatsby-transformer-remark: ^2.2.5 => 2.2.5 
    gatsby-transformer-sharp: ^2.1.13 => 2.1.13 
  npmGlobalPackages:
    gatsby-cli: 2.4.10

[wardpeet]

@floriangaechter it seems to me that this could be a case like #11747. I'll be closing it in favour of that one please tell us as much as you can about this issue (maybe share your repo).

If it does not, feel free to re-open 😄

[floriangaechter]

@wardpeet thanks for your response. I’m not sure if those two are related as in our case, deleting the cache folder actually makes the issue worse instead of better. I’ll leave the issue closed for now and follow the other topic but might reopen it later if it turns out that the issues are not related.

[floriangaechter]

Turns out the issue is the _tracedSVG part of the images we're using. If we just query for "normal" images it works just fine.

[wardpeet]

lets re-open. Could you maybe try to create a small reproduction? ^^

[floriangaechter]

Sure, check out https://github.com/floriangaechter/tracedsvgtest

I hard-coded the api keys for contentful, since it seems to only happen when the images are stored there.

If you have a look at the query in https://github.com/floriangaechter/tracedsvgtest/blob/master/src/pages/index.js#L37 and remove the _tracedSVG part, the build process runs without any issues. But with the _tracedSVG part, it gets stuck during the run graphql queries step.

[floriangaechter]

Some new information – sometimes I'm getting the following error:

error UNHANDLED REJECTION


  Error: read ETIMEDOUT

  - stream_base_commons.js:162 TLSWrap.onStreamRead
    internal/stream_base_commons.js:162:27

Here's the stack trace:

{ Error: read ETIMEDOUT
      at TLSWrap.onStreamRead (internal/stream_base_commons.js:162:27)
    errno: 'ETIMEDOUT',
    code: 'ETIMEDOUT',
    syscall: 'read',
    config:
     { adapter: [Function: httpAdapter],
       transformRequest: [Object],
       transformResponse: [Object],
       timeout: 0,
       xsrfCookieName: 'XSRF-TOKEN',
       xsrfHeaderName: 'X-XSRF-TOKEN',
       maxContentLength: -1,
       validateStatus: [Function: validateStatus],
       headers: [Object],
       method: 'get',
       responseType: 'arraybuffer',
       url:
        'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
       data: undefined },
    request:
     Writable {
       _writableState: [WritableState],
       writable: true,
       _events: [Object],
       _eventsCount: 2,
       _maxListeners: undefined,
       _options: [Object],
       _redirectCount: 0,
       _redirects: [],
       _requestBodyLength: 0,
       _requestBodyBuffers: [],
       _onNativeResponse: [Function],
       _currentRequest: [ClientRequest],
       _currentUrl:
        'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40' },
    response: undefined } }

and:

{ Error: read ETIMEDOUT
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:162:27)
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  syscall: 'read',
  config:
   { adapter: [Function: httpAdapter],
     transformRequest: { '0': [Function: transformRequest] },
     transformResponse: { '0': [Function: transformResponse] },
     timeout: 0,
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: -1,
     validateStatus: [Function: validateStatus],
     headers:
      { Accept: 'application/json, text/plain, */*',
        'User-Agent': 'axios/0.18.0' },
     method: 'get',
     responseType: 'arraybuffer',
     url:
      'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
     data: undefined },
  request:
   Writable {
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: false,
        ended: false,
        finished: false,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: false,
        errorEmitted: false,
        emitClose: true,
        autoDestroy: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: true,
     _events:
      [Object: null prototype] {
        response: [Function: handleResponse],
        error: [Function: handleRequestError] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _options:
      { protocol: 'https:',
        maxRedirects: 21,
        maxBodyLength: 10485760,
        path:
         '/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
        method: 'get',
        headers: [Object],
        agent: undefined,
        auth: undefined,
        hostname: 'images.ctfassets.net',
        port: null,
        nativeProtocols: [Object],
        pathname:
         '/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif',
        search: '?w=40' },
     _redirectCount: 0,
     _redirects: [],
     _requestBodyLength: 0,
     _requestBodyBuffers: [],
     _onNativeResponse: [Function],
     _currentRequest:
      ClientRequest {
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        outputData: [],
        outputSize: 0,
        writable: true,
        _last: true,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: false,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 0,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [TLSSocket],
        connection: [TLSSocket],
        _header:
         'GET /i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.18.0\r\nHost: images.ctfassets.net\r\nConnection: close\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'GET',
        path:
         '/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
        _ended: false,
        res: null,
        aborted: false,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Circular],
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     _currentUrl:
      'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40' },
  response: undefined }

I think it has to do with the way the images are pulled from Contentful…

[ColeTownsend]

@floriangaechter We've been running into similar issues using Contentful with @robinpowered

[dirtyredz]

I am having a similar issue, which was confusing to me since my latest build on Netlify was working perfectly fine.
I tried multiple troubleshooting steps but one in particular worked. I dont remember why I thought to try disabling my antivirus but after I did everything worked again without issue.
Digging deeper it was my HIPS (Host Intrusion Protection System, Whenever this protection is activated I get this error:

> gatsby develop

error UNHANDLED EXCEPTION


  Error: read EINVAL

  - stream_base_commons.js:162 TTY.onStreamRead
    internal/stream_base_commons.js:162:27

But with HIPS disabled I am able to perform gatsby develep without any issues. I made sure to delete both the public and .cache dir after each test.

I dont know exactly what this means for Gatsby or why, but I hope this helps.

Edit:
It seems this was being blocked:

[ChurnM4c]

We have the same problem. A fix would be appreciated.

[floriangaechter]

It really seems to be an issue with the images coming from Contentful. If I add the _noBase64 suffix, the build seems to succeed every time.

Maybe @Khaledgarbaya has an idea what the issue could be?

[Khaledgarbaya]

Hey Folks,
I think when the plugin tries to load the image and generate a base64 out of it times out could be this part

But I am not sure where exactly this is failing. Is it happening only with GIF images?

[floriangaechter]

Yes, I think it has something to do with the generation of either the base64 or tracedSVG variant of the image. Because if I remove both of them and tell Gatsby to generate neither base64 or tracedSVG the build succeeds each time. And it happens to all images, not just GIF files.