Problem in retrieving message data from large message from s3

[abghosh82]

Hi,

I am using this library to work with large messages but I have observed there is an issue while identifying whether the message in the queue is s3 pointer or not and returns it as a normal message. Also the while retrieving the file from s3 it is just returning the GuzzleHttp\Psr7\Stream object instead of content.

So correction at both places is needed for identifying a s3 pointer and retrieval of s3 file content. I have done some overrides in my class to handle this. Like below:

static private function isGuzzleStreamObject($object) {
    return (is_object($object) && get_class($object) == 'GuzzleHttp\Psr7\Stream') ? TRUE : FALSE;
  }

private function getQueueItemData($message) {
    $data = $this->unserialize($message['Body']);
    // Check if $data is empty, this could be the
    // case when message a pointer to s3 file and
    // the AwsExtended\SqsClient could not detect it and
    // returned it as a normal message.
    if (empty($item->data) && variable_get('aws_sqs_extended_s3_file_retrieval_policy', 0) == 1) {
      $s3_file_data = drupal_json_decode($message['Body']);
      // Check if it is a valid s3 pointer.
      if (is_array($s3_file_data) && count($s3_file_data) == 2) {
        $s3_file_access_data = end($s3_file_data);
        // Check for valid s3 file access details.
        if (!empty($s3_file_access_data['s3BucketName']) && !empty($s3_file_access_data['s3Key'])) {
          $result = $this->client->getS3Client()->getObject([
            'Bucket' => $s3_file_access_data['s3BucketName'],
            'Key'    => $s3_file_access_data['s3Key']
          ]);
          $stream = $result->get('Body');
          if (self::isGuzzleStreamObject($stream)) {
            $data = $this->unserialize($stream->getContents());
          }
        }
      }
    }
    return $data;
  }

[hirenzelican]

Hi @abghosh82 I am also facing the same issue,
$this->client->receiveMessage() returns the Messages, not a single large message from S3,
S3Pointer::isS3Pointer($result) is not working at all
@e0ipso any update on this?

Hi,

I am using this library to work with large messages but I have observed there is an issue while identifying whether the message in the queue is s3 pointer or not and returns it as a normal message. Also the while retrieving the file from s3 it is just returning the GuzzleHttp\Psr7\Stream object instead of content.

So correction at both places is needed for identifying a s3 pointer and retrieval of s3 file content. I have done some overrides in my class to handle this. Like below:

static private function isGuzzleStreamObject($object) {
    return (is_object($object) && get_class($object) == 'GuzzleHttp\Psr7\Stream') ? TRUE : FALSE;
  }

private function getQueueItemData($message) {
    $data = $this->unserialize($message['Body']);
    // Check if $data is empty, this could be the
    // case when message a pointer to s3 file and
    // the AwsExtended\SqsClient could not detect it and
    // returned it as a normal message.
    if (empty($item->data) && variable_get('aws_sqs_extended_s3_file_retrieval_policy', 0) == 1) {
      $s3_file_data = drupal_json_decode($message['Body']);
      // Check if it is a valid s3 pointer.
      if (is_array($s3_file_data) && count($s3_file_data) == 2) {
        $s3_file_access_data = end($s3_file_data);
        // Check for valid s3 file access details.
        if (!empty($s3_file_access_data['s3BucketName']) && !empty($s3_file_access_data['s3Key'])) {
          $result = $this->client->getS3Client()->getObject([
            'Bucket' => $s3_file_access_data['s3BucketName'],
            'Key'    => $s3_file_access_data['s3Key']
          ]);
          $stream = $result->get('Body');
          if (self::isGuzzleStreamObject($stream)) {
            $data = $this->unserialize($stream->getContents());
          }
        }
      }
    }
    return $data;
  }

[e0ipso]

No updates on this. I am not currently using this project in any of my clients, so I may not be able to work on these or review PRs.