Skip to content

Basics of Verifying Pravega Security

Jump to bottom
SaiCharan edited this page Apr 26, 2022 · 5 revisions

Table of Contents:

Say, there is a new Pravega release candidate (RC) build and you want to quickly verify that Pravega security is working. This document lists some of the tests you should run.

Note:

  • We are assuming the branch/tag being tested is v0.10.1-rc0. Replace it with the appropriate version when running the commands mentioned in these tests when you run them against another branch.
  • These tests don't verify all the Security features of Pravega, but rather verify Pravega securiy at a "sanity" level only.

Test #1: Verify security in Pravega standalone mode cluster

The goal of this test is to verify TLS and auth in Pravega standalone mode cluster. In this example, we run Pravega from the source code, but you can also use the release distribution.

  1. Clone the repo. $ git clone -b v0.10.1-rc0 https://github.com/pravega/pravega.git

  2. Change directory to pravega. $ cd pravega

  3. Enable and configure TLS and auth by modifying /path/to/pravega/config/standalone-config.properties file.

    New Config (since Pravega 0.8.x):

    # Enable and configure Auth: 
singlenode.security.auth.enable=true
singlenode.security.auth.credentials.username=admin
singlenode.security.auth.credentials.pwd=1111_aaaa
singlenode.security.auth.pwdAuthHandler.accountsDb.location=../config/passwd

# Enable and configure TLS: 
singlenode.security.tls.enable=true
singlenode.security.tls.protocolVersion=TLSv1.2,TLSv1.3
singlenode.security.tls.privateKey.location=../config/server-key.key
singlenode.security.tls.certificate.location=../config/server-cert.crt
singlenode.security.tls.keyStore.location=../config/server.keystore.jks
singlenode.security.tls.keyStore.pwd.location=../config/server.keystore.jks.passwd
singlenode.security.tls.trustStore.location=../config/client.truststore.jks

    Old Config:

    # Enable and configure Auth: 
singlenode.enableAuth=true
singlenode.userName=admin
singlenode.passwd=1111_aaaa
singlenode.passwdFile=../config/passwd

# Enable and configure TLS: 
singlenode.enableTls=true
singlenode.keyFile=../config/server-key.key
singlenode.certFile=../config/server-cert.crt
singlenode.keyStoreJKS=../config/server.keystore.jks
singlenode.keyStoreJKSPasswordFile=../config/server.keystore.jks.passwd
singlenode.trustStoreJKS=../config/client.truststore.jks

  4. Start the Standalone mode cluster. ./gradlew startStandalone

  5. Now, run client side tasks for verifying TLS and Auth.

Test #2: Verify security in distributed mode cluster using Docker Compose

  1. Build pravega/pravega and pravega/bookkeeper Docker images locally from source code.

    $ git clone -b v0.10.1-rc0 https://github.com/pravega/pravega.git

$ cd pravega

$ sudo ./gradlew docker

  2. Ensure that the pravega/pravega:latest and pravega/bookkeeper:latest images point to the newly created images, by inspecting the images manually using: $ sudo docker images.

  3. Create TLS material for Controller and Segment Store. See this document for how to go about it.

  4. Prepare a Docker Compose application deployment file. In this manifest, we enable TLS and Auth for both Controller and Segment Store.

version: "3"
services:
zookeeper:
  image: zookeeper:3.6.3
  ports:
    - "2181:2181"

hdfs:
  image: pravega/hdfs:2.7.7
  ports:
    - "2222:2222"
    - "8020:8020"
    - "50090:50090"
    - "50010:50010"
    - "50020:50020"
    - "50075:50075"
    - "50070:50070"
  environment:
    SSH_PORT: 2222
    HDFS_HOST: ${HOST_IP}

bookie1:
  image: pravega/bookkeeper:latest
  ports:
    - "3181:3181"
  restart: always
  environment:
    ZK_URL: zookeeper:2181
    bookiePort: 3181
  links:
    - zookeeper

bookie2:
  image: pravega/bookkeeper:latest
  ports:
    - "3182:3182"
  restart: always
  environment:
    ZK_URL: zookeeper:2181
    bookiePort: 3182
  links:
    - zookeeper

bookie3:
  image: pravega/bookkeeper:latest
  ports:
    - "3183:3183"
  restart: always
  environment:
    ZK_URL: zookeeper:2181
    bookiePort: 3183
  links:
    - zookeeper

controller:
  image: pravega/pravega:latest
  ports:
    - "9090:9090"
    - "10080:10080"
  command: controller
  environment:
    WAIT_FOR: zookeeper:2181
    ZK_URL: zookeeper:2181
    REST_SERVER_PORT: 10080
    JAVA_OPTS: |
      -Dlog.level="DEBUG"
      -Dcontroller.service.port=9090
      -Dcontroller.auth.tlsEnabled=true
      -controller.security.tls.protocolVersion=TLSv1.2,TLSv1.3
      -Dcontroller.auth.tlsCertFile="/pki/server-cert.crt"
      -Dcontroller.auth.tlsTrustStore="/pki/ca-cert.crt"
      -Dcontroller.auth.tlsKeyFile="/pki/server-key.key"
      -Dcontroller.zk.secureConnection=false
      -Dcontroller.zk.tlsTrustStoreFile="/pki/zk.truststore.jks"
      -Dcontroller.zk.tlsTrustStorePasswordFile="/pki/zk.truststore.jks.password"
      -Dcontroller.rest.tlsKeyStoreFile="/pki/server.keystore.jks"
      -Dcontroller.rest.tlsKeyStorePasswordFile="/pki/server.keystore.jks.passwd"
      -Dcontroller.auth.enabled=true
      -Dcontroller.auth.userPasswordFile="/opt/pravega/conf/passwd"
      -Dconfig.controller.metricenableCSVReporter=false
      -Dcontroller.auth.tokenSigningKey=secret
      -Xmx512m
      -XX:OnError="kill -9 p%"
      -XX:+ExitOnOutOfMemoryError
      -XX:+CrashOnOutOfMemoryError
      -XX:+HeapDumpOnOutOfMemoryError
    SERVICE_HOST_IP: segmentstore
  volumes:
    - /pravega/docker-compose/pki:/pki  
  links:
    - zookeeper

segmentstore:
  image: pravega/pravega:latest
  ports:
    - "12345:12345"
  command: segmentstore
  depends_on: 
    - hdfs
  environment:
    WAIT_FOR: bookie1:3181,bookie2:3182,bookie3:3183,hdfs:8020
    TIER2_STORAGE: "HDFS"
    HDFS_REPLICATION: 1
    HDFS_URL: ${HOST_IP}:8020
    ZK_URL: zookeeper:2181
    CONTROLLER_URL: tcp://${HOST_IP}:9090
    JAVA_OPTS: |
      -Dlog.level="DEBUG"
      -Dpravegaservice.enableTls=true
      -Dpravegaservice.security.tls.protocolVersion=TLSv1.2,TLSv1.3
      -Dpravegaservice.enableTlsReload=false
      -Dpravegaservice.certFile="/pki/server-cert.crt"
      -Dpravegaservice.keyFile="/pki/server-key.key"
      -Dpravegaservice.secureZK=false
      -Dpravegaservice.zkTrustStore="/pki/zk.truststore.jks"
      -Dpravegaservice.zkTrustStorePasswordPath="/pki/zk.truststore.jks.password"
      -Dpravegaservice.security.tls.server.keyStore.location="/pki/server.keystore.jks"
      -Dpravegaservice.security.tls.server.keyStore.pwd.location="/pki/server.keystore.jks.passwd"
      -DautoScale.tlsEnabled=true
      -DautoScale.tlsCertFile="/pki/server-cert.crt"
      -DautoScale.validateHostName=false
      -DautoScale.authEnabled=true
      -DautoScale.tokenSigningKey=secret
      -Dbookkeeper.tlsEnabled=false
      -Dbookkeeper.tlsTrustStorePath="/pki/bk.truststore.jks"
      -Dmetrics.enableCSVReporter=false
      -Dpravegaservice.publishedIPAddress=${HOST_IP}
      -Dbookkeeper.bkEnsembleSize=2
      -Dbookkeeper.bkAckQuorumSize=2
      -Dbookkeeper.bkWriteQuorumSize=2
      -Dpravega.client.auth.token="YWRtaW46MTExMV9hYWFh"
      -Dpravega.client.auth.method="Basic"
      -Xmx900m
      -XX:OnError="kill -9 p%"
      -XX:+ExitOnOutOfMemoryError
      -XX:+CrashOnOutOfMemoryError
      -XX:+HeapDumpOnOutOfMemoryError
  volumes:
    - /pravega/docker-compose/pki:/pki    
  links:
    - zookeeper
    - hdfs
    - bookie1
    - bookie2
    - bookie3

  1. Launch a cluster using steps described here.

  2. Now, run client side tasks for verifying TLS and Auth.

Client-side tasks for verifying TLS and Auth

Note:

  • Make sure to replace <host-ip> with localhost when running these tests against Standalone mode cluster. If you are running it against a distributed mode cluster, replace it with the host's IP address.

Verify that REST API is responding

  1. Verify that the server is responding by invoking the Controller REST API. The endpoint /v1/scopes should list the _system scope.

    # The default REST interface port ois `9091` for Standalone mode cluster and manually 
# deployed distributed mode cluster. 
#
# NOTE: If you are running this test against Docker-based
# or Kubernetes-based deployments, you might need to replace the port `9091` with the 
# the default port on those environments: `10080`. 
$ curl -v -k -u admin:1111_aaaa https://<host-ip>:9091/v1/scopes
Expected response: 
...
...
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 36
<
* Connection #0 to host localhost left intact
{"scopes":[{"scopeName":"_system"}]}

Now, we'll run a Pravega event writer and reader application to verify that the system is working properly.

Verify that event writes and reads work

  1. Install the Pravega client and other libraries into the local Maven repository.

    /path/to/pravega $./gradlew install

  2. Note the Pravega client version. $ ls -lrt ~/.m2/repository/io/pravega/pravega-client. Say, the most recent version built from the last step is 0.9.0.

  3. Clone the Pravega samples repo.

    Note: We are using the branch r0.9 here as the server is of a similar branch. Replace it with an appropriate version, depending on the version you are trying to verify.

    $ git clone -b r0.9 https://github.com/pravega/pravega-samples.git

  4. Change directory: $ cd pravega-samples

  5. Modify the pravegaVersion in gradle.properties. (We noted version 0.10.1 in the current version, earlier in this document.)

  6. Build Pravega samples.

    $ ./gradlew clean installDist

  7. Change directory.

    $ cd pravega-samples/pravega-client-examples/build/install/pravega-client-examples/bin`

  8. Run the writer:

    $ bin/secureWriter -scope "myScope" -stream "myStream" -uri "tls://<host-ip>:9090" -routingkey "myKey" -message "hello world!"  -truststore "/path/to/pravega/forked/batchclient-tokenprov/pravega/config/server-cert.crt" -username "admin" -password "1111_aaaa"

Expected output:

Done creating client config.
Done creating a stream manager.
Done creating a scope with the specified name: [myScope].
Done creating a stream configuration.
Done creating a stream with the specified name: [myStream] and stream configuration.
Done creating a client factory with the specified scope and client config.
Done creating a writer.
Done writing an event: [hello world!].
All done with writing! Exiting...

  9. Run the reader:

    $ bin/secureReader -scope "myScope" -stream "myStream" -uri "tls://<host-ip>:9090" -truststore "/path/to/pravega/forked/batchclient-tokenprov/pravega/config/server-cert.crt" -username "admin" -password "1111_aaaa"	

Expected output:

Done creating a client config.
Done creating a reader group config with specified scope: [myScope] and stream name: [myStream].
Done creating a reader group with specified name  and config.
Done creating a client factory with the specified scope and client config.
Done creating a reader.
Done reading an event: [hello world!].
All done with reading! Exiting...
Clone this wiki locally