Long string being truncated

[honestbonsai]

I was using ngrok to proxy my local dev server before, but I would like to move to smee. I've noticed that for a really long string (a signature) that I'm not getting the same results from ngrok and smee. The lengths:

ngrok: 684
smee: 678

smee's version will fail my webhook signature validation. Is there some kind of max length to a field before it gets truncated?

Update: I can see my webhook on smee.io and it's not being truncated there, and I added logs to to inside of smee-client here https://github.com/probot/smee-client/blob/master/index.js#L19 and it seems to receive the data fine.

Also getting the following errors:

{ Error: Bad Request
    at Request.callback (/Users/hl/development/teams/node_modules/superagent/lib/node/index.js:706:15)
    at IncomingMessage.parser (/Users/hl/development/teams/node_modules/superagent/lib/node/index.js:916:18)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  response:
   Response {
     _events: [Object: null prototype] {},
     _eventsCount: 0,
     _maxListeners: undefined,
     res:
      IncomingMessage {
        _readableState: [ReadableState],
        readable: false,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        socket: [Socket],
        connection: [Socket],
        httpVersionMajor: 1,
        httpVersionMinor: 1,
        httpVersion: '1.1',
        complete: true,
        headers: {},
        rawHeaders: [],
        trailers: {},
        rawTrailers: [],
        aborted: false,
        upgrade: false,
        url: '',
        method: null,
        statusCode: 400,
        statusMessage: 'Bad Request',
        client: [Socket],
        _consuming: true,
        _dumped: false,
        req: [ClientRequest],
        text: '' },
     request:
      Request {
        _events: [Object: null prototype] {},
        _eventsCount: 0,
        _maxListeners: undefined,
        _agent: false,
        _formData: null,
        method: 'POST',
        url: 'http://localhost:8001/api',
        _header: [Object],
        header: [Object],
        writable: true,
        _redirects: 0,
        _maxRedirects: 5,
        cookies: '',
        qs: {},
        _query: [],
        qsRaw: [],
        _redirectList: [],
        _streamRequest: false,
        _data: [Object],
        req: [ClientRequest],
        protocol: 'http:',
        host: 'localhost:8001',
        _endCalled: true,
        _callback: [Function],
        res: [IncomingMessage],
        response: [Circular],
        called: true },
     req:
      ClientRequest {
        _events: [Object],
        _eventsCount: 3,
        _maxListeners: undefined,
        output: [],
        outputEncodings: [],
        outputCallbacks: [],
        outputSize: 0,
        writable: true,
        _last: true,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: true,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 1386,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [Socket],
        connection: [Socket],
        _header:
         'POST /api/paddle HTTP/1.1\r\nhost: smee.io\r\nAccept-Encoding: gzip, deflate\r\nuser-agent: Paddle\r\ncontent-type: application/x-www-form-urlencoded\r\nconnection: close\r\naccept: */*\r\nx-request-id: 2093910a-46b9-4454-8d06-38484cece245\r\nx-forwarded-for: 34.194.127.46\r\nx-forwarded-proto: https\r\nx-forwarded-port: 443\r\nvia: 1.1 vegur\r\nconnect-time: 1\r\nx-request-start: 1590699759565\r\ntotal-route-time: 0\r\ncontent-length: 1378\r\ntimestamp: 1590699759569\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'POST',
        path: '/api/paddle',
        _ended: true,
        res: [IncomingMessage],
        aborted: undefined,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     text: '',
     body: {},
     files: undefined,
     buffered: true,
     headers: {},
     header: {},
     statusCode: 400,
     status: 400,
     statusType: 4,
     info: false,
     ok: false,
     redirect: false,
     clientError: true,
     serverError: false,
     error:
      { Error: cannot POST /api/paddle (400)
          at Response.toError (/Users/hl/development/teams/node_modules/superagent/lib/node/response.js:94:15)
          at ResponseBase._setStatusProperties (/Users/hl/development/teams/node_modules/superagent/lib/response-base.js:123:16)
          at new Response (/Users/hl/development/teams/node_modules/superagent/lib/node/response.js:41:8)
          at Request._emitResponse (/Users/hl/development/teams/node_modules/superagent/lib/node/index.js:752:20)
          at IncomingMessage.parser (/Users/hl/development/teams/node_modules/superagent/lib/node/index.js:916:38)
          at IncomingMessage.emit (events.js:194:15)
          at endReadableNT (_stream_readable.js:1103:12)
          at process._tickCallback (internal/process/next_tick.js:63:19) status: 400, text: '', method: 'POST', path: '/api/paddle' },
     created: false,
     accepted: false,
     noContent: false,
     badRequest: true,
     unauthorized: false,
     notAcceptable: false,
     forbidden: false,
     notFound: false,
     unprocessableEntity: false,
     type: '',
     links: {},
     setEncoding: [Function: bound ],
     redirects: [] } }

[honestbonsai]

If I add the following to https://github.com/probot/smee-client/blob/master/index.js#L34

const contentLength = req.get("content-length");
const newContentLength = parseInt(contentLength) + 8;
req.set('content-length', `${newContentLength}`)
req.end(...)

It works fine. Not sure why it's +8 though.

I tried this PR's code locally and it didn't work https://github.com/probot/smee-client/pull/148/files

EDIT: It's not always +8, so some of my webhooks work and some don't.

[honestbonsai]

I tried this #122 locally and it worked.

[zalexki]

#148 , #152 fixes it in a nicer way than #122 IMO, i hope it will be merged one day

[zalexki]

@hiimbex @JasonEtco @lee-dohm @zeke 🙏 can one of you accept one of the above listed PR for this issue (that will also close #142 i think)

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?