Pack installation

[pavanfhw]

I was able to install the slack package in my stackstorm-ha.

This contradicts the documentation that says: In distributed environment of the Kubernetes cluster st2 pack install won’t work.
It is not detailed why it should no work. Is it because the paths /opt/stackstorm/{packs,virtualenvs} are not a shared volume between all deployments? That was my thinking so I created volumes for /opt/stackstorm/packs and /opt/stackstorm/virtualenvs.

But I'm not able to install the kubernetes pack. Is it the expected behavior? Install one pack but not another?

The logs from the packs installation:

slack:

id: 601af25d977d9e0c3a92f8ed
action.ref: packs.install
parameters: 
  packs:
  - slack
status: succeeded (76s elapsed)
start_timestamp: Wed, 03 Feb 2021 18:58:37 UTC
end_timestamp: Wed, 03 Feb 2021 18:59:53 UTC
result: 
  output:
    conflict_list: []
    message: Successfully installed packs
    packs_list:
    - slack
    warning_list: []
+--------------------------+-------------------------+------------------------------------+-----------------------------+-------------------------------+
| id                       | status                  | task                               | action                      | start_timestamp               |
+--------------------------+-------------------------+------------------------------------+-----------------------------+-------------------------------+
| 601af25ed960e506c6a15f26 | succeeded (1s elapsed)  | init_task                          | core.noop                   | Wed, 03 Feb 2021 18:58:38 UTC |
| 601af260d960e506c6a15f35 | succeeded (21s elapsed) | download_pack                      | packs.download              | Wed, 03 Feb 2021 18:58:40 UTC |
| 601af276d57f8e4e94ea2e97 | succeeded (0s elapsed)  | make_a_prerun                      | packs.virtualenv_prerun     | Wed, 03 Feb 2021 18:59:02 UTC |
| 601af281d57f8e4e94ea2ea6 | succeeded (1s elapsed)  | get_pack_dependencies              | packs.get_pack_dependencies | Wed, 03 Feb 2021 18:59:13 UTC |
| 601af283d960e506c6a15f46 | succeeded (0s elapsed)  | check_dependency_and_conflict_list | core.noop                   | Wed, 03 Feb 2021 18:59:15 UTC |
| 601af284d960e506c6a15f55 | succeeded (28s elapsed) | install_pack_requirements          | packs.setup_virtualenv      | Wed, 03 Feb 2021 18:59:16 UTC |
| 601af2a0d960e506c6a15f7d | succeeded (0s elapsed)  | get_pack_warnings                  | packs.get_pack_warnings     | Wed, 03 Feb 2021 18:59:44 UTC |
| 601af2a3d57f8e4e94ea2eb8 | succeeded (6s elapsed)  | register_pack                      | packs.load                  | Wed, 03 Feb 2021 18:59:47 UTC |
+--------------------------+-------------------------+------------------------------------+-----------------------------+-------------------------------+

kubernetes:

action.ref: packs.install
parameters: 
  packs:
  - kubernetes
status: failed (19s elapsed)
start_timestamp: Wed, 03 Feb 2021 19:59:57 UTC
end_timestamp: Wed, 03 Feb 2021 20:00:16 UTC
result: 
  errors:
  - message: Execution failed. See result for details.
    result:
      exit_code: 1
      result: None
      stderr: "st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "kubernetes" (/opt/stackstorm/packs/kubernetes)
st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/kubernetes" doesn't exist
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "kubernetes" in "/opt/stackstorm/virtualenvs/kubernetes"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/kubernetes" using Python binary "/opt/stackstorm/st2/bin/python"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --no-download /opt/stackstorm/virtualenvs/kubernetes" to create virtualenv.
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/kubernetes/bin/pip install six>=1.9.0,<2.0.
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/kubernetes/requirements.txt"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/kubernetes/requirements.txt with command /opt/stackstorm/virtualenvs/kubernetes/bin/pip install -U -r /opt/stackstorm/packs/kubernetes/requirements.txt.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 193, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 89, in run
    no_download=no_download)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/virtualenvs.py", line 112, in setup_pack_virtualenv
    logger=logger)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/virtualenvs.py", line 251, in install_requirements
    (requirements_file_path, stdout, stderr))
Exception: Failed to install requirements from "/opt/stackstorm/packs/kubernetes/requirements.txt": Collecting pyswagger (from -r /opt/stackstorm/packs/kubernetes/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/ec/6e/af4a2531e379ad64388615327b6c2e0f3df4b22013718553189cff501905/pyswagger-0.8.39-py3-none-any.whl (78kB)
Collecting jinja2 (from -r /opt/stackstorm/packs/kubernetes/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/7e/c2/1eece8c95ddbc9b1aeb64f5783a9e07a286de42191b7204d67b7496ddf35/Jinja2-2.11.3-py2.py3-none-any.whl (125kB)
Collecting http_parser (from -r /opt/stackstorm/packs/kubernetes/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/04/b2/c98d0a2b759fcf91ae0a3e7691e840e4c861dac2c7f5a4e36220129d0cc8/http-parser-0.9.0.tar.gz (167kB)
Collecting backports.ssl==0.0.9 (from -r /opt/stackstorm/packs/kubernetes/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/ca/39/e968b8823a469dd7135ff960057aa27e8a226d2b1ac0a38701e26ce2e19f/backports.ssl-0.0.9.tar.gz
Collecting pyopenssl==17.5.0 (from -r /opt/stackstorm/packs/kubernetes/requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/79/db/7c0cfe4aa8341a5fab4638952520d8db6ab85ff84505e12c00ea311c3516/pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)
Collecting validate-email (from pyswagger->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/84/a0/cb53fb64b52123513d04f9b913b905f3eb6fda7264e639b4573cc715c29f/validate_email-1.3.tar.gz
Requirement already satisfied, skipping upgrade: six>=1.7.2 in ./virtualenvs/kubernetes/lib/python3.6/site-packages (from pyswagger->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 1)) (1.15.0)
Collecting pyaml>=15.03.1 (from pyswagger->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/15/c4/1310a054d33abc318426a956e7d6df0df76a6ddfa9c66f6310274fb75d42/pyaml-20.4.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/9d/d3/75cddfad6ca1d1bb3a017cece499a65e54ceb4583800f1256b8ad07bb57f/MarkupSafe-1.1.1-cp36-cp36m-manylinux2010_x86_64.whl
Collecting cryptography>=2.1.4 (from pyopenssl==17.5.0->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/c9/de/7054df0620b5411ba45480f0261e1fb66a53f3db31b28e3aa52c026e72d9/cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6MB)
Collecting PyYAML (from pyaml>=15.03.1->pyswagger->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640kB)
Collecting cffi>=1.12 (from cryptography>=2.1.4->pyopenssl==17.5.0->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/1c/1a/90fa7e7ee05d91d0339ef264bd8c008f57292aba4a91ec512a0bbb379d1d/cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401kB)
Collecting pycparser (from cffi>=1.12->cryptography>=2.1.4->pyopenssl==17.5.0->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
Building wheels for collected packages: http-parser, backports.ssl, validate-email
  Building wheel for http-parser (setup.py): started
  Building wheel for http-parser (setup.py): finished with status 'error'
  Running setup.py clean for http-parser
  Building wheel for backports.ssl (setup.py): started
  Building wheel for backports.ssl (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/ba/76/4b/c4d89c6e7710b1517162b0846c13b53e7ec015e1e882d943fd
  Building wheel for validate-email (setup.py): started
  Building wheel for validate-email (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/14/46/e7/4a2edb0d03a802cb59a75668734c04cceacdd6c0c585448773
Successfully built backports.ssl validate-email
Failed to build http-parser
Installing collected packages: validate-email, PyYAML, pyaml, pyswagger, MarkupSafe, jinja2, http-parser, backports.ssl, pycparser, cffi, cryptography, pyopenssl
  Running setup.py install for http-parser: started
    Running setup.py install for http-parser: finished with status 'error'
\(stderr:   ERROR: Complete output from command /opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-ixx7k2ze/http-parser/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gi6q70jy --python-tag cp36:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/util.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/http.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/reader.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/_socketio.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/__init__.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/pyparser.py -> build/lib.linux-x86_64-3.6/http_parser
  running build_ext
  building 'http_parser.parser' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/http_parser
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iparser -I/usr/include/python3.6m -I/opt/stackstorm/virtualenvs/kubernetes/include/python3.6m -c http_parser/http_parser.c -o build/temp.linux-x86_64-3.6/http_parser/http_parser.o
  unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for http-parser
\   ERROR: Complete output from command /opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-ixx7k2ze/http-parser/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-crrmxaug/install-record.txt --single-version-externally-managed --compile --install-headers /opt/stackstorm/virtualenvs/kubernetes/include/site/python3.6/http-parser:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/util.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/http.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/reader.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/_socketio.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/__init__.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/pyparser.py -> build/lib.linux-x86_64-3.6/http_parser
    running build_ext
    building 'http_parser.parser' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/http_parser
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iparser -I/usr/include/python3.6m -I/opt/stackstorm/virtualenvs/kubernetes/include/python3.6m -c http_parser/http_parser.c -o build/temp.linux-x86_64-3.6/http_parser/http_parser.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
\RROR: Command "/opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-ixx7k2ze/http-parser/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-crrmxaug/install-record.txt --single-version-externally-managed --compile --install-headers /opt/stackstorm/virtualenvs/kubernetes/include/site/python3.6/http-parser" failed with error code 1 in /tmp/pip-install-ixx7k2ze/http-parser/
WARNING: You are using pip version 19.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
)
"
      stdout: ''
    task_id: install_pack_requirements
    type: error
  output:
    conflict_list: []
    message: ''
    packs_list:
    - kubernetes
    warning_list: null
+--------------------------+------------------------+------------------------------------+-----------------------------+-------------------------------+
| id                       | status                 | task                               | action                      | start_timestamp               |
+--------------------------+------------------------+------------------------------------+-----------------------------+-------------------------------+
| 601b00bed960e506c6a15f9a | succeeded (1s elapsed) | init_task                          | core.noop                   | Wed, 03 Feb 2021 19:59:58 UTC |
| 601b00bfd960e506c6a15fa9 | succeeded (6s elapsed) | download_pack                      | packs.download              | Wed, 03 Feb 2021 19:59:59 UTC |
| 601b00c5d960e506c6a15fb8 | succeeded (1s elapsed) | make_a_prerun                      | packs.virtualenv_prerun     | Wed, 03 Feb 2021 20:00:05 UTC |
| 601b00c7d960e506c6a15fc7 | succeeded (0s elapsed) | get_pack_dependencies              | packs.get_pack_dependencies | Wed, 03 Feb 2021 20:00:07 UTC |
| 601b00c8d960e506c6a15fd6 | succeeded (0s elapsed) | check_dependency_and_conflict_list | core.noop                   | Wed, 03 Feb 2021 20:00:08 UTC |
| 601b00c8d960e506c6a15fe5 | failed (8s elapsed)    | install_pack_requirements          | packs.setup_virtualenv      | Wed, 03 Feb 2021 20:00:08 UTC |
+--------------------------+------------------------+------------------------------------+-----------------------------+-------------------------------+

The error unable to execute 'x86_64-linux-gnu-gcc': No such file or directory just seems like there is a dependency necessary to install the pack missing.

Version:

# st2 --version
st2 3.4dev (c422f0029), on Python 3.6.9

[arms11]

Just curious...do you have NFS share where the pack is being attempted to be installed? because the only other way is to be done via docker image to my knowledge.

[pavanfhw]

I have a Ceph Shared Filesystem in the paths /opt/stackstorm/packs and /opt/stackstorm/virtualenvs on all containers. Should this be enough to install packs using the Kubernetes HA mode?

[arm4b]

Yeah, ReadWriteMany kind of shared storage that you have makes the virtualenvs and pack content shared between all the st2actionrunner pods which should make the pack install work.

In your specific case, the error is different:

  unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Some packs might need additional OS-level dependencies for building the virtualenv/requirements. Looks like you need gcc package to be installed in the st2actionrunner containers.

Hope that helps.

[pavanfhw]

@armab That was on point. I installed the gcc package on the action runner containers and it got me past that point on the pack installation.

But it could not finish, the packs.load action failed:

id: 601c18a5595ca69c52894996
action.ref: packs.load
context.user: st2admin
parameters: 
  packs:
  - kubernetes
  register: all
status: failed (13s elapsed)
start_timestamp: Thu, 04 Feb 2021 15:54:13 UTC
end_timestamp: Thu, 04 Feb 2021 15:54:26 UTC
result: 
  exit_code: 1
  result: None
  stderr: "st2.actions.python.St2RegisterAction: DEBUG    Calling client method "register" with kwargs "{'types': ['all'], 'packs': ['kubernetes']}"
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 193, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py", line 82, in run
    format_func=format_result)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py", line 150, in _run_client_method
    result = method(**method_kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2client/models/core.py", line 42, in decorate
    return func(*args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2client/models/core.py", line 563, in register
    self.handle_error(response)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2client/models/core.py", line 168, in handle_error
    response.raise_for_status()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request
MESSAGE: Failed to register action "/opt/stackstorm/packs/kubernetes/actions/register_tpr.meta.yaml" from pack "kubernetes": RunnerType mistral-v2 is not found. If you are using old and deprecated runner name, you need to switch to a new one. For more information, please see https://docs.stackstorm.com/upgrade_notes.html#st2-v0-9 for url: http://stackstorm-st2api:9101/v1/packs/register
"
  stdout: ''

This seems more a problem with the pack being out of date with my stackstorm installation. Is it possible for me to six it mannualy? Editing the action file causing the error:

# cat packs/kubernetes/actions/register_tpr.meta.yaml 
---
  name: "register_tpr"
  pack: "kubernetes"
  description: "migrate services"
  runner_type: "mistral-v2"
  enabled: true
  entry_point: "workflows/register_tpr.yaml"
  parameters:
    payload:
      type: "object"
      description: "payload object from kubernetes sensor"

Changing the runner type would fix it? If so, how do I get my correct runner type?

[arm4b]

Looks like you're using some very old K8s pack version.
Check the upstream version: https://github.com/StackStorm-Exchange/stackstorm-kubernetes/blob/master/actions/workflows/register_tpr.yaml Mistral was deprecated since that an replaced with the Orquestra workflow engine.

[pavanfhw]

@armab Using the github repo url instead of the pack name the installation worked. Thank you for your help!

[arm4b]

Thanks for reporting.

What you're saying signalizes that StackStorm Exchange index is unsync with the latest release for the K8s pack.
Created an issue for that: StackStorm-Exchange/stackstorm-kubernetes#46 so we can fix that pack later.