new feature: ch-test #447

j-ogas · 2019-05-15T19:47:04Z

Addresses #445.

shanegoff

I think the permissions test should be moved to an option by itself. When I frequently am testing/rebuilding I have to redo the setup for the permissions dirs. Taking those tests out of the run segment would allow the permissions tests to be run separately, so users without root who can't run them anyway don't have to take action to skip them.

heasterday

Needs a tidy but sounds good to me

reidpr

Nice work. I proposed a revision.

Note you'll need to put it in conf.py as well to create the man page.

As for Shane's comment, I agree that we don't want to require the user to mess around with this. I propose that ch-test create the permissions fixtures if appropriate. I think he's right that it could go in a separate test phase, but I'd prefer to keep the three existing phases for this PR.

doc-src/ch-test.rst

j-ogas · 2019-07-10T20:40:06Z

Aside from a minor travis fix in run/ch-test.bats to address the the installation case testing, this is ready to be reviewed.

How this will be integrated with travis is not clear. I don't imagine doubling the test time to run the tests via test/Makefile and again via ch-test is an option. Do we then alter travis to run the test suite exclusively through ch-test?

reidpr

Thanks. Various suggestions in-line.

My overall impression was that it seems kind of complicated for what basically is a cd followed by make. However, I became less sure of that as I read on. I do think some of the suggestions could make things simpler.

This will need to be updated to include the builder, which for some has additional configuration (in the case of ch-grow).

Do we then alter travis to run the test suite exclusively through ch-test?

Yes. I'd say this should become the sole supported way to run the tests. That may require updating the docs.

bin/ch-test

doc-src/ch-test_desc.rst

doc-src/conf.py

bin/ch-test

j-ogas · 2019-09-11T16:33:10Z

Also related to #329.

reidpr

I think overall this looks solid, but I'd like to discuss and walk through it in person. Can we do that next week?

One thing I thought to add: There are currently a few setup/teardown tests, I think mostly to manage the test directories. I think these maybe should be moved to ch-test?

j-ogas · 2019-09-18T15:24:28Z

I think overall this looks solid, but I'd like to discuss and walk through it in person. Can we do that next week?

Sounds good.

One thing I thought to add: There are currently a few setup/teardown tests, I think mostly to manage the test directories. I think these maybe should be moved to ch-test?

Agreed.

reidpr · 2019-09-23T16:50:13Z

Add check/workaround for #278?

j-ogas · 2019-10-25T16:02:39Z

doc-src/ch-test_desc.rst

+  :code:`clean`
+    Delete test artifacts on disk and all images in builder storage (whether
+    or not Charliecloud-related), except for the permissions test fixtures.


Spicy. Not sure I'd like clean to wipe my docker/buildah image cache. Am I misunderstanding this?

Your understanding is correct. Why don't we leave it alone for now (i.e., don't wipe builder storage). I would like an easy way to do that though; our current test/docker-clean.sh isn't very user-facing.

I took this out.

j-ogas · 2019-10-25T16:07:28Z

doc-src/ch-test_desc.rst

+  :code:`--pack-dir DIR`
+    Set packed images directory to :code:`DIR`. Default:
+    :code:`$CH_TEST_TARDIR` if set, or :code:`/var/tmp/pack`.


Perhaps we should go ahead and update CH_TEST_TARDIR to CH_TEST_PACKDIR as well? I know it is long, but I'm also wondering if CH_TEST_IMGDIR should be CH_TEST_UNPACKDIR.

I guess the reasoning to leave the variables along is that it's a pretty pervasive change that alters an existing API, so I wanted to leave it alone for now, in case the new vocabulary is not satisfactory.

reidpr

Great work. Some tidy comments. Doesn't pass Travis after my changed; haven't looked into that. Perhaps we can wrap it up today and get it merged?

bin/ch-test

test/run/ch-run_misc.bats

reidpr · 2019-10-24T23:08:38Z

test/travis.bash

-    cd "$PREFIX/libexec/charliecloud-$version"
+    ch_test="${PREFIX}/bin/ch-test"
+else
+    ch_test=$(readlink -f bin/ch-test)


What is the problem readlink solves here?

Removing the absolute path here will cause travis to error (https://travis-ci.org/hpc/charliecloud/jobs/604006996?utm_medium=notification&utm_source=email).

147 top_dir=$(readlink -f "${ch_bin%/*}") [...] 173 # Determine source vs prefix installation. 174 if [[ -d "${top_dir}/test" ]]; then 175 # source install 176 testdir=${top_dir}/test 177 else 178 # prefix installation 179 testdir=${top_dir%/charliecloud*}/libexec/charliecloud-$(version 2>&1)/test 180 fi

The check for ${top_dir}/test fails (it shouldn't, it is a source install). I feel like this is because base.sh uses dirname "$0" instead of dirname "${BASH_SOURCE[0]}. When we do ch_test=$(readlink -f bin/ch-test in the travis.bash, the issue is resolved (https://travis-ci.org/hpc/charliecloud/jobs/604009862?utm_medium=notification&utm_source=github_status).

test/travis.bash

bin/ch-test

reidpr

Very nice work. If you are OK with my tidying and it passes tests, then I think this is ready to merge.

bin/ch-test

reidpr · 2019-10-30T22:47:36Z

doc-src/ch-test_desc.rst

+  :code:`clean`
+    Delete test artifacts on disk and all images in builder storage (whether
+    or not Charliecloud-related), except for the permissions test fixtures.


I took this out.

j-ogas requested review from shanegoff, heasterday and rstyd May 15, 2019 19:47

shanegoff approved these changes May 15, 2019

View reviewed changes

rstyd approved these changes May 15, 2019

View reviewed changes

j-ogas requested a review from reidpr May 16, 2019 15:12

heasterday approved these changes May 16, 2019

View reviewed changes

reidpr approved these changes Jun 6, 2019

View reviewed changes

doc-src/ch-test.rst Outdated Show resolved Hide resolved

j-ogas requested a review from reidpr July 10, 2019 20:36

reidpr suggested changes Jul 10, 2019

View reviewed changes

j-ogas requested a review from reidpr July 16, 2019 18:31

j-ogas force-pushed the ch-test branch from 71929d3 to 11aee3a Compare July 16, 2019 20:15

j-ogas mentioned this pull request Sep 5, 2019

unhelpful test suite error when CH_TEST_*DIR does not exist #347

Closed

j-ogas force-pushed the ch-test branch 2 times, most recently from c9ec1cd to b698a4c Compare September 13, 2019 21:46

reidpr reviewed Sep 17, 2019

View reviewed changes

This was referenced Sep 26, 2019

Tidy ch-checkns #558

Closed

ch-grow should list all missing dependencies #559

Closed

j-ogas force-pushed the ch-test branch from c91904c to f9c826c Compare October 4, 2019 17:40

reidpr mentioned this pull request Oct 4, 2019

Vagrantfile: tests do not pass on second run #566

Closed

j-ogas force-pushed the ch-test branch 3 times, most recently from 5fdba3d to 1ab9dbd Compare October 10, 2019 17:45

This was referenced Oct 14, 2019

add ch-grow dependencies to Vagrantfile #517

Merged

buildah 1.11.2 errors with ch-run-oci #568

Closed

j-ogas force-pushed the ch-test branch from 8c10dbf to 43402fa Compare October 15, 2019 14:21

j-ogas added 2 commits October 16, 2019 08:51

add ch-test prototype

62f23a4

update ch-test

8a1c601

j-ogas force-pushed the ch-test branch from deb8284 to 8a1c601 Compare October 16, 2019 14:57

fix typo

3dd6751

j-ogas mentioned this pull request Oct 17, 2019

ch-run-oci: check multiple paths for inferred bundles #570

Merged

reidpr added 2 commits October 24, 2019 15:33

revise and simplify docs

1fe5b77

tidy

c04be55

j-ogas commented Oct 25, 2019

View reviewed changes

reidpr suggested changes Oct 25, 2019

View reviewed changes

reidpr and others added 9 commits October 25, 2019 10:31

silence shellcheck

5ee1708

update with feedback

f8e582c

merge master

36b2b55

fix typos

208bff2

fix typo

c65ff4c

debug travis: revert readlink removal

d52eb9f

make single assignment function generic

779e362

remove SUDO_AVOID_AFTER_BUILD

6cfaa8e

merge from master

6e76396

reidpr mentioned this pull request Oct 30, 2019

delete test/{Makefile,make-auto} #576

Closed

tidy

576b1a0

reidpr approved these changes Oct 30, 2019

View reviewed changes

reidpr added the ready to merge? label Oct 30, 2019

reidpr mentioned this pull request Oct 31, 2019

Remove existing test directories in test make clean #520

Closed

reidpr merged commit 82eee98 into master Oct 31, 2019

reidpr deleted the ch-test branch October 31, 2019 20:39

j-ogas added a commit that referenced this pull request Nov 1, 2019

PR #447 from @j-ogas: new test driver ch-test (closes #445)

55af7a5

reidpr removed the ready to merge? label Nov 4, 2019

This was referenced Nov 4, 2019

test suite: missing error message if Charliecloud not built #329

Closed

user-specific permissions fixture directories #68

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new feature: ch-test #447

new feature: ch-test #447

j-ogas commented May 15, 2019

shanegoff left a comment

heasterday left a comment

reidpr left a comment

j-ogas commented Jul 10, 2019

reidpr left a comment

j-ogas commented Sep 11, 2019

reidpr left a comment

j-ogas commented Sep 18, 2019

reidpr commented Sep 23, 2019

j-ogas Oct 25, 2019

reidpr Oct 25, 2019

reidpr Oct 30, 2019

j-ogas Oct 25, 2019

reidpr Oct 25, 2019

reidpr left a comment

reidpr Oct 24, 2019

j-ogas Oct 28, 2019 •

edited

Loading

reidpr left a comment •

edited

Loading

reidpr Oct 30, 2019

new feature: ch-test #447

new feature: ch-test #447

Conversation

j-ogas commented May 15, 2019

shanegoff left a comment

Choose a reason for hiding this comment

heasterday left a comment

Choose a reason for hiding this comment

reidpr left a comment

Choose a reason for hiding this comment

j-ogas commented Jul 10, 2019

reidpr left a comment

Choose a reason for hiding this comment

j-ogas commented Sep 11, 2019

reidpr left a comment

Choose a reason for hiding this comment

j-ogas commented Sep 18, 2019

reidpr commented Sep 23, 2019

j-ogas Oct 25, 2019

Choose a reason for hiding this comment

reidpr Oct 25, 2019

Choose a reason for hiding this comment

reidpr Oct 30, 2019

Choose a reason for hiding this comment

j-ogas Oct 25, 2019

Choose a reason for hiding this comment

reidpr Oct 25, 2019

Choose a reason for hiding this comment

reidpr left a comment

Choose a reason for hiding this comment

reidpr Oct 24, 2019

Choose a reason for hiding this comment

j-ogas Oct 28, 2019 • edited Loading

Choose a reason for hiding this comment

reidpr left a comment • edited Loading

Choose a reason for hiding this comment

reidpr Oct 30, 2019

Choose a reason for hiding this comment

j-ogas Oct 28, 2019 •

edited

Loading

reidpr left a comment •

edited

Loading