Download source code archive directly for ubuntu and macos setup #9198

liujiayi771 · 2024-03-21T12:41:45Z

It is better to use wget_and_untar to download the source code archive
directly, instead of using git to clone the source code. Because direct download
is much faster than using git clone. The setup-centos8.sh script has been
consistently downloading source code archive directly.

netlify · 2024-03-21T12:42:03Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`6346875`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/66a1e5e59a527e0008dbffe1

liujiayi771 · 2024-03-21T12:42:59Z

In our test environment, for boost, it took about 11 min before (from 2024-03-21T06:20:47 to 2024-03-21T06:31:13):

2024-03-21T06:20:47.3262778Z + run_and_time install_boost
2024-03-21T06:20:47.3263537Z + install_boost
2024-03-21T06:20:47.3264743Z + github_checkout boostorg/boost boost-1.84.0 --recursive
2024-03-21T06:20:47.3271539Z + local REPO=boostorg/boost
2024-03-21T06:20:47.3272490Z + shift
2024-03-21T06:20:47.3273275Z + local VERSION=boost-1.84.0
2024-03-21T06:20:47.3274005Z + shift
2024-03-21T06:20:47.3274716Z + local GIT_CLONE_PARAMS=--recursive
2024-03-21T06:20:47.3275628Z ++ basename boostorg/boost
2024-03-21T06:20:47.3276434Z + Finished running install_fmt
2024-03-21T06:20:47.3277281Z + local DIRNAME=boost
2024-03-21T06:20:47.3278123Z + SUDO='sudo --preserve-env'
2024-03-21T06:20:47.3279167Z + cd /opt/gluten/ep/build-velox/build/velox_ep
2024-03-21T06:20:47.3280153Z + '[' -z boost ']'
2024-03-21T06:20:47.3281462Z + '[' -d boost ']'
2024-03-21T06:20:47.3282170Z + '[' '!' -d boost ']'
2024-03-21T06:20:47.3283545Z + git clone -q -b boost-1.84.0 --recursive https://github.com/boostorg/boost.git
2024-03-21T06:21:00.5554970Z Note: switching to 'ad09f667e61e18f5c31590941e748ac38e5a81bf'.
2024-03-21T06:21:00.5556728Z 
2024-03-21T06:21:00.5557598Z You are in 'detached HEAD' state. You can look around, make experimental
2024-03-21T06:21:00.5559211Z changes and commit them, and you can discard any commits you make in this
2024-03-21T06:21:00.5560889Z state without impacting any branches by switching back to a branch.
2024-03-21T06:21:00.5561865Z 
2024-03-21T06:21:00.5562552Z If you want to create a new branch to retain commits you create, you may
2024-03-21T06:21:00.5564281Z do so (now or later) by using -c with the switch command. Example:
2024-03-21T06:21:00.5565210Z 
2024-03-21T06:21:00.5565585Z   git switch -c <new-branch-name>
2024-03-21T06:21:00.5667647Z 
2024-03-21T06:21:00.5668525Z Or undo this operation with:
2024-03-21T06:21:00.5669179Z 
2024-03-21T06:21:00.5669430Z   git switch -
2024-03-21T06:21:00.5788570Z 
2024-03-21T06:21:00.5869925Z Turn off this advice by setting config variable advice.detachedHead to false
2024-03-21T06:21:00.5948424Z 
2024-03-21T06:31:13.8519110Z + cd boost

It took about 20s now (from 2024-03-21T08:25:15 to 2024-03-21T08:25:32)

2024-03-21T08:25:15.1901014Z + run_and_time install_boost
2024-03-21T08:25:15.1902104Z + install_boost
2024-03-21T08:25:15.1904364Z + wget_and_untar https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.gz boost
2024-03-21T08:25:15.1907965Z + local URL=https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.gz
2024-03-21T08:25:15.1909631Z + local DIR=boost
2024-03-21T08:25:15.1910348Z + mkdir -p boost
2024-03-21T08:25:15.1922276Z + pushd boost
2024-03-21T08:25:15.1924758Z + curl -L https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.gz
2024-03-21T08:25:15.1927531Z /opt/gluten/ep/build-velox/build/velox_ep/fmt/boost /opt/gluten/ep/build-velox/build/velox_ep/fmt
2024-03-21T08:25:15.2047924Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2024-03-21T08:25:15.2049463Z                                  Dload  Upload   Total   Spent    Left  Speed
2024-03-21T08:25:15.2050292Z 
2024-03-21T08:25:16.2292144Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2024-03-21T08:25:16.2293827Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2024-03-21T08:25:16.2295320Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2024-03-21T08:25:17.3928559Z 
2024-03-21T08:25:18.1827510Z   0  118M    0  7421    0     0   3391      0 10:08:26  0:00:02 10:08:24  3391
2024-03-21T08:25:19.1764844Z   0  118M    0  191k    0     0  65753      0  0:31:22  0:00:02  0:31:20  232k
2024-03-21T08:25:20.1637281Z   4  118M    4 5913k    0     0  1489k      0  0:01:21  0:00:03  0:01:18 3310k
2024-03-21T08:25:21.1679483Z  15  118M   15 18.4M    0     0  3807k      0  0:00:31  0:00:04  0:00:27 6811k
2024-03-21T08:25:22.1508048Z  26  118M   26 31.2M    0     0  5364k      0  0:00:22  0:00:05  0:00:17 8469k
2024-03-21T08:25:23.1499963Z  37  118M   37 44.0M    0     0  6490k      0  0:00:18  0:00:06  0:00:12 9473k
2024-03-21T08:25:24.1504551Z  48  118M   48 56.9M    0     0  7343k      0  0:00:16  0:00:07  0:00:09 11.4M
2024-03-21T08:25:25.1513762Z  59  118M   59 69.8M    0     0  8000k      0  0:00:15  0:00:08  0:00:07 12.8M
2024-03-21T08:25:26.1492539Z  70  118M   70 82.8M    0     0  8525k      0  0:00:14  0:00:09  0:00:05 12.9M
2024-03-21T08:25:27.1490347Z  81  118M   81 95.8M    0     0  8966k      0  0:00:13  0:00:10  0:00:03 12.9M
2024-03-21T08:25:27.7938709Z  92  118M   92  108M    0     0  9332k      0  0:00:12  0:00:11  0:00:01 12.9M
2024-03-21T08:25:27.7941828Z 100  118M  100  118M    0     0  9603k      0  0:00:12  0:00:12 --:--:-- 13.1M
2024-03-21T08:25:27.8029221Z + tar -xz --strip-components=1 -f boost.tar.gz
2024-03-21T08:25:32.2022371Z + popd
2024-03-21T08:25:32.2023384Z + cd boost

liujiayi771 · 2024-03-21T15:51:58Z

@kgpai Could you help to review?

liujiayi771 · 2024-03-22T01:32:48Z

cc @assignUser. Could you help to review?

assignUser

I don't have any objections to this.

The number of submodules is the issue with boost, as a workaround there is a git config that allows you to increase the number of parallel submodule checkouts.

scripts/setup-ubuntu.sh

liujiayi771 · 2024-04-02T03:04:25Z

@assignUser @kgpai Could you help review again? I have added the functionality to change directory to a specific path in the cmake_install method.

assignUser · 2024-04-12T23:44:40Z

Sorry for the delay, could you rebase again?

liujiayi771 · 2024-04-13T01:19:00Z

@assignUser Rebased to the latest main branch.

assignUser

LGTM @majetideepak you have also worked on the setup script extensively recently, any objections?

kgpai · 2024-04-13T20:50:20Z

@liujiayi771 I havent noticed any particular slowness with git and it seems roughly equivalent to download. The advantage of using git is that its probably a little more secure than plain old downloads of tar.gz's (FYI the checksums we calculate arent really secure anyway).
Do other folks see significant differences b/w git and downloading tar.gz's ?
@majetideepak ?

liujiayi771 · 2024-04-14T01:47:51Z

@kgpai In my test, the time difference for Boost is quite significant, and the script for CentOS 8 does not use git currently, so both methods should work.

stale · 2024-07-13T06:25:03Z

This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!

majetideepak · 2024-07-19T16:06:06Z

Sorry, I missed this PR. I think we should uniformly use tar.gz for all because 1) We can then use a common shared file across the platforms to download and install all the dependencies. 2) Downloading a single file should be better than git (involves extra metadata + processing)?
We can also include the checksums in a single file to validate.
@liujiayi771 can you please rebase this PR?

liujiayi771 · 2024-07-20T03:23:05Z

@majetideepak Rebased to the latest main branch.

majetideepak

Thanks @liujiayi771

majetideepak

@liujiayi771 one change needed.

scripts/setup-helper-functions.sh

majetideepak · 2024-07-26T02:43:52Z

@Yuhta, can you please import this PR? Thanks!

facebook-github-bot · 2024-07-26T14:18:53Z

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-07-26T16:22:15Z

@Yuhta merged this pull request in b5713ef.

conbench-facebook · 2024-07-26T16:49:42Z

Conbench analyzed the 1 benchmark run on commit b5713efc.

There was 1 benchmark result indicating a performance regression:

Commit Run on GitHub-runner-8-core at 2024-07-26 16:49:12Z
- eq (C++) with source=cpp-micro, suite=velox_benchmark_basic_comparison_conjunct

The full Conbench report has more details.

…ebookincubator#9198) Summary: It is better to use wget_and_untar to download the source code archive directly, instead of using git to clone the source code. Because direct download is much faster than using git clone. The setup-centos8.sh script has been consistently downloading source code archive directly. Pull Request resolved: facebookincubator#9198 Reviewed By: kgpai Differential Revision: D60288820 Pulled By: Yuhta fbshipit-source-id: acf111d61673e8648adf0e82319375de92a2e942

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 21, 2024

liujiayi771 force-pushed the ubuntu-download branch 4 times, most recently from c694651 to 99cca16 Compare March 21, 2024 13:52

liujiayi771 force-pushed the ubuntu-download branch from 661c570 to 957fbcb Compare March 22, 2024 00:29

liujiayi771 force-pushed the ubuntu-download branch 2 times, most recently from b545d28 to fdbcd20 Compare March 26, 2024 01:38

assignUser reviewed Mar 28, 2024

View reviewed changes

scripts/setup-ubuntu.sh Outdated Show resolved Hide resolved

liujiayi771 force-pushed the ubuntu-download branch 2 times, most recently from 7565004 to 2540184 Compare April 2, 2024 01:53

liujiayi771 force-pushed the ubuntu-download branch from 2540184 to 1c4cda9 Compare April 3, 2024 00:22

liujiayi771 force-pushed the ubuntu-download branch from 1c4cda9 to 8950ec5 Compare April 13, 2024 01:09

assignUser approved these changes Apr 13, 2024

View reviewed changes

assignUser requested a review from majetideepak April 13, 2024 02:51

stale bot added the stale label Jul 13, 2024

stale bot removed the stale label Jul 19, 2024

majetideepak self-assigned this Jul 19, 2024

liujiayi771 force-pushed the ubuntu-download branch from 8950ec5 to bac7d61 Compare July 19, 2024 23:40

majetideepak approved these changes Jul 20, 2024

View reviewed changes

majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jul 20, 2024

majetideepak requested changes Jul 20, 2024

View reviewed changes

scripts/setup-helper-functions.sh Outdated Show resolved Hide resolved

majetideepak removed the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jul 20, 2024

liujiayi771 force-pushed the ubuntu-download branch from 9076816 to 163a164 Compare July 20, 2024 15:22

majetideepak approved these changes Jul 21, 2024

View reviewed changes

majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jul 21, 2024

liujiayi771 added 7 commits July 25, 2024 13:42

Download source code archive directly for ubuntu and macos setup

1623b5e

Remove -S

378e953

Add cd in cmake_install

e0af9f2

Use pushd and popd in cmake_install

3f07122

fix duckdb build

a30dcbf

Use -d to check cmake install dir

fa05a12

Fix conda cd

6346875

liujiayi771 force-pushed the ubuntu-download branch from 947b45c to 6346875 Compare July 25, 2024 05:43

kgpai approved these changes Jul 25, 2024

View reviewed changes

facebook-github-bot closed this in b5713ef Jul 26, 2024

facebook-github-bot added the Merged label Jul 26, 2024

liujiayi771 mentioned this pull request Jul 28, 2024

[VL] Adapt setup scripts for the changes in upstream velox apache/incubator-gluten#6614

Closed

zuyu mentioned this pull request Sep 1, 2024

Fix showing repo name in cmake_install #10908

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download source code archive directly for ubuntu and macos setup #9198

Download source code archive directly for ubuntu and macos setup #9198

liujiayi771 commented Mar 21, 2024 •

edited

Loading

netlify bot commented Mar 21, 2024 •

edited

Loading

liujiayi771 commented Mar 21, 2024

liujiayi771 commented Mar 21, 2024

liujiayi771 commented Mar 22, 2024

assignUser left a comment

liujiayi771 commented Apr 2, 2024 •

edited

Loading

assignUser commented Apr 12, 2024

liujiayi771 commented Apr 13, 2024

assignUser left a comment

kgpai commented Apr 13, 2024

liujiayi771 commented Apr 14, 2024 •

edited

Loading

stale bot commented Jul 13, 2024

majetideepak commented Jul 19, 2024 •

edited

Loading

liujiayi771 commented Jul 20, 2024

majetideepak left a comment

majetideepak left a comment

majetideepak commented Jul 26, 2024

facebook-github-bot commented Jul 26, 2024

facebook-github-bot commented Jul 26, 2024

conbench-facebook bot commented Jul 26, 2024

Download source code archive directly for ubuntu and macos setup #9198

Download source code archive directly for ubuntu and macos setup #9198

Conversation

liujiayi771 commented Mar 21, 2024 • edited Loading

netlify bot commented Mar 21, 2024 • edited Loading

✅ Deploy Preview for meta-velox canceled.

liujiayi771 commented Mar 21, 2024

liujiayi771 commented Mar 21, 2024

liujiayi771 commented Mar 22, 2024

assignUser left a comment

Choose a reason for hiding this comment

liujiayi771 commented Apr 2, 2024 • edited Loading

assignUser commented Apr 12, 2024

liujiayi771 commented Apr 13, 2024

assignUser left a comment

Choose a reason for hiding this comment

kgpai commented Apr 13, 2024

liujiayi771 commented Apr 14, 2024 • edited Loading

stale bot commented Jul 13, 2024

majetideepak commented Jul 19, 2024 • edited Loading

liujiayi771 commented Jul 20, 2024

majetideepak left a comment

Choose a reason for hiding this comment

majetideepak left a comment

Choose a reason for hiding this comment

majetideepak commented Jul 26, 2024

facebook-github-bot commented Jul 26, 2024

facebook-github-bot commented Jul 26, 2024

conbench-facebook bot commented Jul 26, 2024

liujiayi771 commented Mar 21, 2024 •

edited

Loading

netlify bot commented Mar 21, 2024 •

edited

Loading

liujiayi771 commented Apr 2, 2024 •

edited

Loading

liujiayi771 commented Apr 14, 2024 •

edited

Loading

majetideepak commented Jul 19, 2024 •

edited

Loading