Pipeline and scripts for custom node 18 build (with glibc 2.17) #4
Conversation
…pport exporting manifest lists'
…ss up the state for others
There was a problem hiding this comment.
Have we looked into how transparent we need to be that we aren't shipping an official Node.js distribution? Are there any licensing concerns? Should we open an issue at https://github.com/elastic/open-source?
cc @mistic
Co-authored-by: Jon <[email protected]>
|
@jbudz valid concern! I haven't considered it, and I don't really know how to approach it. I'll open an issue with open-source, I think they're happy to discuss the details. |
|
Dear @elastic/product-security -
|
|
@delanni for anything related to license you should contact https://github.com/elastic/open-source folks.
Can we give us some context or reference some docs / wiki with regards to why we need custom builds and what are the changes we are making to it, how we're building it, where we are hosting it and where the Kibana CI/CD build is piking it up from etc etc ? This is the first time I hear about a custom node.js build 🙏 |
There was a problem hiding this comment.
Just finished a first pass and left a couple of comments. Overall this looks in the right direction to me 🎉
In regards to what @jbudz mention I do think we should think about a plan to advertise to our users why we had the need to do this custom builds (for a couple of specific platforms) and how this will allow us to support node 18 on those problematic platforms; how we achieve those custom builds and why they are secure. Let's also wait to see what open source folks have to say about that 😃
.buildkite/pipeline.yml
Outdated
| # command: | | ||
| # buildah --version | ||
| # drivah build --changed-since=main ./.buildkite/containers/amd64 | ||
| - command: echo "OK" |
There was a problem hiding this comment.
Good question -
This is part of the elastic's ci-systems build pipeline from the research where I tried to pre-build docker images, and use those as agents in buildkite. It seems that wasn't the ultimate direction, so if we're going with this kibana-buildkite pipeline, then I can clean up this other pipeline.
There was a problem hiding this comment.
@brianseeders just to double check here, can we move on for now without using the ci-systems pipeline ?
There was a problem hiding this comment.
It's up to you, but you'll have to migrate eventually. What are you not able to do?
Also, just FYI, I'm not on the CI systems team anymore
There was a problem hiding this comment.
I'll wait on @delanni to let you know about the challenges he found on using the ci-systems but in case we are allowed to continue with the kibana pipeline, I think we can proceed like that until we are forced to migrate eventually.
Thanks for letting me know Brian, I didn't realised before you have changed teams 🙌
There was a problem hiding this comment.
The track using this pipeline was a bit different than what we ultimately took. I think the ci-systems way would have been a bit better modularised in the sense that we build the agent images separately, and the build job only builds the binaries, and not the images.
I got stuck because with drivah, I couldn't really build multiarch images, and building separate images for separate platforms was also difficult, because docker didn't find the right base images for the centos:7/arm64 combination (whereas it worked locally). This is when I back-tracked, and went on another route.
The two approaches are probably mostly orthogonal, so if we're eventually moving over, then maybe I'll keep the pipeline in place, and try to work out the docker/centos issue later.
There was a problem hiding this comment.
I believe what you're trying to do with docker should work fine with the ci-systems agents, as long as you are using a vm-based agent and not a k8s one.
You may need to create a custom agent image with the dependencies installed to do multiarch (if you didn't already), and you might also need enableNestedVirtualization: true on your agents: config in your pipeline. I'm guessing the multiarch virtualization stuff in Docker would need it.
You can also just install the dependencies at runtime to test to see if it works before committing to it.
There was a problem hiding this comment.
For the record, we agreed with @mistic that it would be probably a waste as of now to try to fit this build on the ci-systems agents. It's probably not going to live for more than 1 year, we'll probably still have the kibana CI available to create new patch versions if needed, and hopefully the unofficial node releases will take over. (nodejs/unofficial-builds#83)
|
@mistic - check the PR now, I've removed all the extra bits. |
Co-authored-by: Tiago Costa <[email protected]>
Co-authored-by: Tiago Costa <[email protected]>
## Summary Bumps node.js to 18.17.0 (replacement for PR #144012 which was later reverted) As a result, these categorical additions were needed: - `node` evocations will need the `--openssl-legacy-provider` flag, wherever it would use certain crypto functionalities - tests required updating of the expected HTTPS Agent call arguments, `noDelay` seems to be a default - `window.[NAME]` fields cannot be written directly - some stricter typechecks This is using our in-house built node.js 18 versions through the URLs the proxy-cache. (built with elastic/kibana-custom-nodejs-builds#4) These urls are served from a bucket, where the RHEL7/Centos7 compatible node distributables are. (see: elastic/kibana-ci-proxy-cache#7) Further todos: - [x] check docs wording and consistency - [ ] update the dependency report - [x] explain custom builds in documentation - [x] node_sass prebuilts --------- Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: Thomas Watson <[email protected]>
## Summary Bumps node.js to 18.17.0 (replacement for PR elastic#144012 which was later reverted) As a result, these categorical additions were needed: - `node` evocations will need the `--openssl-legacy-provider` flag, wherever it would use certain crypto functionalities - tests required updating of the expected HTTPS Agent call arguments, `noDelay` seems to be a default - `window.[NAME]` fields cannot be written directly - some stricter typechecks This is using our in-house built node.js 18 versions through the URLs the proxy-cache. (built with elastic/kibana-custom-nodejs-builds#4) These urls are served from a bucket, where the RHEL7/Centos7 compatible node distributables are. (see: elastic/kibana-ci-proxy-cache#7) Further todos: - [x] check docs wording and consistency - [ ] update the dependency report - [x] explain custom builds in documentation - [x] node_sass prebuilts --------- Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: Thomas Watson <[email protected]>
## Summary Bumps node.js to 18.17.0 (replacement for PR elastic#144012 which was later reverted) As a result, these categorical additions were needed: - `node` evocations will need the `--openssl-legacy-provider` flag, wherever it would use certain crypto functionalities - tests required updating of the expected HTTPS Agent call arguments, `noDelay` seems to be a default - `window.[NAME]` fields cannot be written directly - some stricter typechecks This is using our in-house built node.js 18 versions through the URLs the proxy-cache. (built with elastic/kibana-custom-nodejs-builds#4) These urls are served from a bucket, where the RHEL7/Centos7 compatible node distributables are. (see: elastic/kibana-ci-proxy-cache#7) Further todos: - [x] check docs wording and consistency - [ ] update the dependency report - [x] explain custom builds in documentation - [x] node_sass prebuilts --------- Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: Thomas Watson <[email protected]>
Closes #162695 # Backport This will backport the following commits from `main` to `7.17`: - [[Ops] Bump Node.js to version 18 (#160289)](#160289) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Alex Szabo","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-07-27T12:12:48Z","message":"[Ops] Bump Node.js to version 18 (#160289)\n\n## Summary\r\n\r\nBumps node.js to 18.17.0 (replacement for PR #144012 which was later\r\nreverted)\r\n\r\nAs a result, these categorical additions were needed: \r\n- `node` evocations will need the `--openssl-legacy-provider` flag,\r\nwherever it would use certain crypto functionalities\r\n- tests required updating of the expected HTTPS Agent call arguments,\r\n`noDelay` seems to be a default\r\n - `window.[NAME]` fields cannot be written directly\r\n - some stricter typechecks\r\n\r\nThis is using our in-house built node.js 18 versions through the URLs\r\nthe proxy-cache. (built with\r\nhttps://github.com/elastic/kibana-custom-nodejs-builds/pull/4)\r\n\r\nThese urls are served from a bucket, where the RHEL7/Centos7 compatible\r\nnode distributables are. (see:\r\nhttps://github.com/elastic/kibana-ci-proxy-cache/pull/7)\r\n\r\nFurther todos: \r\n - [x] check docs wording and consistency\r\n - [ ] update the dependency report\r\n - [x] explain custom builds in documentation\r\n - [x] node_sass prebuilts\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>\r\nCo-authored-by: Tiago Costa <[email protected]>\r\nCo-authored-by: Thomas Watson <[email protected]>","sha":"8cf68dc6ba8f010e36538c1e7c92601a341efcf4","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","Team:uptime","backport:skip","release_note:feature","ci:all-cypress-suites","v8.10.0"],"number":160289,"url":"https://github.com/elastic/kibana/pull/160289","mergeCommit":{"message":"[Ops] Bump Node.js to version 18 (#160289)\n\n## Summary\r\n\r\nBumps node.js to 18.17.0 (replacement for PR #144012 which was later\r\nreverted)\r\n\r\nAs a result, these categorical additions were needed: \r\n- `node` evocations will need the `--openssl-legacy-provider` flag,\r\nwherever it would use certain crypto functionalities\r\n- tests required updating of the expected HTTPS Agent call arguments,\r\n`noDelay` seems to be a default\r\n - `window.[NAME]` fields cannot be written directly\r\n - some stricter typechecks\r\n\r\nThis is using our in-house built node.js 18 versions through the URLs\r\nthe proxy-cache. (built with\r\nhttps://github.com/elastic/kibana-custom-nodejs-builds/pull/4)\r\n\r\nThese urls are served from a bucket, where the RHEL7/Centos7 compatible\r\nnode distributables are. (see:\r\nhttps://github.com/elastic/kibana-ci-proxy-cache/pull/7)\r\n\r\nFurther todos: \r\n - [x] check docs wording and consistency\r\n - [ ] update the dependency report\r\n - [x] explain custom builds in documentation\r\n - [x] node_sass prebuilts\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>\r\nCo-authored-by: Tiago Costa <[email protected]>\r\nCo-authored-by: Thomas Watson <[email protected]>","sha":"8cf68dc6ba8f010e36538c1e7c92601a341efcf4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/160289","number":160289,"mergeCommit":{"message":"[Ops] Bump Node.js to version 18 (#160289)\n\n## Summary\r\n\r\nBumps node.js to 18.17.0 (replacement for PR #144012 which was later\r\nreverted)\r\n\r\nAs a result, these categorical additions were needed: \r\n- `node` evocations will need the `--openssl-legacy-provider` flag,\r\nwherever it would use certain crypto functionalities\r\n- tests required updating of the expected HTTPS Agent call arguments,\r\n`noDelay` seems to be a default\r\n - `window.[NAME]` fields cannot be written directly\r\n - some stricter typechecks\r\n\r\nThis is using our in-house built node.js 18 versions through the URLs\r\nthe proxy-cache. (built with\r\nhttps://github.com/elastic/kibana-custom-nodejs-builds/pull/4)\r\n\r\nThese urls are served from a bucket, where the RHEL7/Centos7 compatible\r\nnode distributables are. (see:\r\nhttps://github.com/elastic/kibana-ci-proxy-cache/pull/7)\r\n\r\nFurther todos: \r\n - [x] check docs wording and consistency\r\n - [ ] update the dependency report\r\n - [x] explain custom builds in documentation\r\n - [x] node_sass prebuilts\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>\r\nCo-authored-by: Tiago Costa <[email protected]>\r\nCo-authored-by: Thomas Watson <[email protected]>","sha":"8cf68dc6ba8f010e36538c1e7c92601a341efcf4"}}]}] BACKPORT--> --------- Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: Thomas Watson <[email protected]> Co-authored-by: Jonathan Budzenski <[email protected]>
Adds pipeline scripts and build utilities for building node.js with
glibc 2.17.Currently, the build can be ran through the kibana-buildkite job (https://github.com/elastic/kibana-buildkite/pull/86)
The latest builds' results are here: https://console.cloud.google.com/storage/browser/kibana-custom-node-artifacts/node-glibc-217/dist/v18.16.1
See: README.md for more details