BUG | fast-crc32c does not work when running on a flex GAE instance

[louisgv]

Thanks for stopping by to let us know something could be better!

1. Is this a client library issue or a product issue?

• client library

2. Did someone already solve this?

• no

3. Do you have a support contract?

• no

Environment details

• OS: GAE Flex (app.yaml included below)
• Node.js version: ">=12"
• npm version: GAE's included version
• @google-cloud/storage version: "^5.8.1"

runtime: nodejs

env: flex
network:
  session_affinity: true

automatic_scaling:
  min_num_instances: 1
  max_num_instances: 2

Steps to reproduce

1. Implement the stream download example
2. Install fast-crc32c
3. Deploy to GAE

Error

Error: no CRC-32C implementation is available

Proposed solution

• @stephenplusplus can you please review Replace sse4_crc32 with @node-rs/crc32 ashi009/node-fast-crc32c#21 . Even firebase-tools also resorted to remove it recently: Remove fast-crc32c dependency firebase/firebase-tools#3249
• Replace hash-stream-validation with node-rs/crc32

[shaffeeullah]

Steps to reproduce

1. Implement the stream download example
2. Install fast-crc32c
3. Deploy to GAE

@louisgv Can you please link me to what you're referring to as the 'stream download example`?

[louisgv]

@shaffeeullah Here it is:

https://googleapis.dev/nodejs/storage/latest/File.html#createReadStream
https://googleapis.dev/nodejs/storage/latest/File.html#createWriteStream

[shaffeeullah]

Hi @louisgv, I am unable to reproduce this issue. I skipped step 2, of your repro steps expecting the request to fail with an error telling me to install fast-crc32c, but the request succeeded. This was the script I deployed to GAE:

'use strict';

const express = require('express');

const app = express();

async function createReadStream() {

  const {Storage} = require('@google-cloud/storage');
  const fs = require('fs');

  // Your Google Cloud Platform project ID
  const projectId = 'my-project-id';
  // Creates a client
  const storage = new Storage({
    projectId: projectId,
  });

  const bucket = await storage.bucket('generic-test-bucket');
  const file = bucket.file('my-existing-file.png');
  const localFile = 'testFile.txt';
  fs.createReadStream(localFile)
  .pipe(file.createWriteStream())
  .on('error', function(err) {})
  .on('finish', function() {
    // The file upload is complete.
  });
}

app.get('/', async function(req, res) {
  const result = await createReadStream();
  return res.status(200).send("hello world")
});

// Start the server
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
  console.log(`App listening on port ${PORT}`);
  console.log('Press Ctrl+C to quit.');
});

module.exports = app;

My yaml file matched the one you included above (thanks for pasting it!). After running my script above, I can go to my bucket and see that the file was updated with the correct contents. The script includes the sample from here (thanks for linking me to it above). Any additional information you can include to help me reproduce this issue would be much appreciated.

[louisgv]

@shaffeeullah Thanks for picking up this issue, I really appreciate your help! Can you please try to install fast-crc32c, redeploy to GAE, and see if audio stream download/upload works? My hope was that by including fast-crc32c, I can have faster file validation. However, after I installed the library, GAE would complain regarding missing fast-crc32c library/module

[shaffeeullah]

Hi @louisgv ,

I've installed fast-crc32c and changed my file type to .mp3 and I am still unable to reproduce this issue. Are there any other reproduction steps you can provide me with?

[louisgv]

@shaffeeullah Can you try out the following:

Install pump;

npm i pump

Add a download route:

'use strict';

const express = require('express');

const app = express();

const {Storage} = require('@google-cloud/storage');

const fs = require('fs');

async function uploadAudio() {
  // Your Google Cloud Platform project ID
  const projectId = 'my-project-id';
  // Creates a client
  const storage = new Storage({
    projectId: projectId,
  });

  const bucket = await storage.bucket('generic-test-bucket');
  const file = bucket.file('test.mp3');
  const localFile = 'test.mp3';

  fs.createReadStream(localFile)
  .pipe(file.createWriteStream())
  .on('error', function(err) {})
  .on('finish', function() {
    // The file upload is complete.
  });
}

app.get('/upload', async function(req, res) {
  const result = await uploadAudio();
  return res.status(200).send("hello world")
});

app.get('/download', async function(req, res) {

     const bucket = await storage.bucket('generic-test-bucket');
     const file = bucket.file('test.mp3');
    res.setHeader("Content-Type", "audio/mp3")

    pump(file.createReadStream(), res, async (err) => {
      if (err) {
        console.error(err)
       }
    })
});

// Start the server
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
  console.log(`App listening on port ${PORT}`);
  console.log('Press Ctrl+C to quit.');
});

module.exports = app;

Then try to open /download in a browser and see if the audio can be downloaded

Here're some extra notes:

1. It works in my local development (with fastcrc32c), however when I deploy to GAE environment, and whenever my app trigger one of the stream process, the app would hang and this error would be shown on stack driver: Error: no CRC-32C implementation is available
2. I'm using "express": "^5.0.0-alpha.8", not sure if that is the root of the problem

[shaffeeullah]

@louisgv Given that this works locally and I am not able to reproduce it with the barebones code, I do not believe this is an issue with this library. Please file an issue with GAE support.
In the meantime, you can workaround the issue by not installing fast-crc32c.