rak does not search in docker container

[melezhik]

Hi! I am not sure if this is a Rak or Rakudo or SparkyCI + Docker issue, my feeling is that it all started from #5 , anyway now when SparkyCI logs improved (imho) , it's easier to make a sense of it. Let's consider a simple example of creating file containing foo and doing a search with rak foo in cwd:

[rak-count-only]
[repository] - index updated from http://sparrowhub.io/repo//api/v1/index
[task run: task.bash - .tomty/tasks/rak-count-only]
[dump code: task.bash]
[1] set -e
[2] 
[3] cd $cache_dir
[4] 
[5] rak --version
[6] 
[7] echo foo > file.txt
[8] echo foo > file2.txt
[9] echo foo >> file2.txt
[10] echo foo > file3.txt
[11] 
[12] rak foo --count-only
[13] 
[14] rak foo --files-with-matches --count-only
[15] 
[16] 
[task stdout]
rak - provided by App::Rak 0.0.98, running Raku 6.d with Rakudo 2022.07.
No files with matches
No files with matches
[task check]
stdout match (s) <4 matches in 3 files> False
stdout match (s) <3 files with matches> False
=================
TASK CHECK FAIL

So for some reasons rak just says "No files with matches", even though file is here, interesting? What makes thing more harder that the issue is only reproduced by on SparkyCI - see full log as en example - https://ci.sparrowhub.io/report/1224 , most of failure (excluding git related) are due to rak just does not see files in cwd, or it looks up at different location . I have not digged into rak code yet, but like a guess, maybe it something where rak relies on some ENV vars, which values are not guaranteed for some (docker based) none standard environments ?

Just a guess. And HTH

Alexey

[melezhik]

Or even that more clear example, as we can see a file is here and has content with "foo" sting, however rak does not see anything:

[rak-search]
[repository] - index updated from http://sparrowhub.io/repo//api/v1/index
[task run: task.bash - .tomty/tasks/rak-search/]
[dump code: task.bash]
[1] set -e
[2] 
[3] echo "foo" > $cache_root_dir/file.txt
[4] 
[5] cd $cache_root_dir
[6] 
[7] rak --version
[8] 
[9] echo "====="
[10] cat file.txt
[11] echo "====="
[12] 
[13] echo "<<<"
[14] rak foo --/highlight
[15] echo ">>>"
[16] 
[17] 
[task stdout]
rak - provided by App::Rak 0.0.98, running Raku 6.d with Rakudo 2022.07.
=====
foo
=====
<<<
>>>

[lizmat]

If rak is run with its output redirected (aka $*OUT.t is False), then the output should be:

file.txt:1:foo

Is that still not the case, now with version 1.1 ?

[melezhik]

Unfortunately, the issue still presents on the latest commit - https://ci.sparrowhub.io/report/1261

[melezhik]

And this looks weird to me:

[task run: task.bash - .tomty/tasks/rak-count-only]
[dump code: task.bash]
[1] set -e
[2] 
[3] cd $cache_dir
[4] 
[5] rak --version
[6] 
[7] echo foo > file.txt
[8] echo foo > file2.txt
[9] echo foo >> file2.txt
[10] echo foo > file3.txt
[11] 
[12] echo
[13] cat file.txt
[14] echo
[15] cat file2.txt
[16] echo
[17] cat file3.txt
[18] 
[19] rak foo --count-only
[20] 
[21] rak foo --files-with-matches --count-only
[22] 
[23] 
[task stdout]
rak - provided by App::Rak 0.1.1, running Raku 6.d with Rakudo 2022.07.

foo

foo
foo

foo
No files with matches
No files with matches
[task stderr]
Unexpected leftovers: {:show-item-number(Bool::False)}
[task check]
stdout match (s) <4 matches in 3 files> False
stdout match (s) <3 files with matches> False
=================
TASK CHECK FAIL

[lizmat]

Running that bash script gives me:

$ bash doit
rak - provided by App::Rak 0.1.1, running Raku 6.d with Rakudo 2022.07.50.

foo

foo
foo

foo
4 matches in 3 files
3 files with matches

So I have no idea what is going on in your case. Also:

Unexpected leftovers: {:show-item-number(Bool::False)}

is very strange. I don't see that either. It almost feels like it's actually executing an older version of App::Rak / rak somehow.

[melezhik]

It should not, as a test runs every time on new docker alpine container , from the scratch, you can see all the versions used on the top of the report …

[lizmat]

Overnight I thought that maybe App::Rak was using some post 2022.07+ feature, so I did this on a Debian box with 2022.07 release version installed. App::Rak tested and installed ok there, and also ran this test to completion as expected.

So, I have really no clue what's going on here.

[melezhik]

I think I have a guess what going on here, I did some tests yesterday with some docker containers , running rak from there, an interesting issue I have found , rak search , for example “rak foo” hangs there , when run docker exec command without -t flag ( terminal support ), I guess this somehow relates to this issue , where rak also runs on docker container , long story short I would dig into how rak handles none tty run cases , I briefly looked at the code , however I don’t have enough expertises to say anything concrete … Or maybe this could be even down the rabbit hole further , something Raku related … 🤔

[melezhik]

Another interesting fact when I run “echo foo>file.txt; foo --output-file=out.txt &” to overcome hang issue on docker , the command no longer hangs and returns, however out.txt file is empty , which might represents sprarkyci tests cases on azure docker instances, where we see that rak search does find anything even though we know that some search results should be there … 🤔 HTH

So I would recommend you to spin up a docker container and do all this experiments with docker none terminal exec ( e.g. “docker exec $container rak foo”, without -t flag) …

I could continue this debugging once I get back from my short vacation on Monday …

[melezhik]

See “-t” flag on https://docs.docker.com/engine/reference/commandline/exec/

[melezhik]

Please compare these two almost identical commands with different results ( the second command has -t in docker exec):

Julias-MacBook-Pro:raku-alpin-repo alex$ docker exec  alpine /bin/ash -c 'cd /tmp/foo && ls -l && cat foo.txt && /usr/share/rakudo/site/bin/rak foo'
total 4
-rw-r--r--    1 root     root             4 Oct 11 16:40 foo.txt
foo

Julias-MacBook-Pro:raku-alpin-repo alex$ docker exec alpine /bin/ash -c 'cd /tmp/foo && ls -l && cat foo.txt && /usr/share/rakudo/site/bin/rak foo'
total 4
-rw-r--r--    1 root     root             4 Oct 11 16:40 foo.txt
foo
Julias-MacBook-Pro:raku-alpin-repo alex$ docker exec -t alpine /bin/ash -c 'cd /tmp/foo && ls -l && cat foo.txt && /usr/share/rakudo/site/bin/rak foo'
total 4
-rw-r--r--    1 root     root             4 Oct 11 16:40 foo.txt
foo
foo.txt
1:foo

This reproduces the bug.

Env:

/tmp/foo # rak --version
rak - provided by App::Rak 0.1.3, running Raku 6.d with Rakudo 2022.06.
/tmp/foo # raku -v 
Welcome to Rakudo™ v2022.06.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.06.

[lizmat]

What does dd $*IN.t say?

[melezhik]

Julias-MacBook-Pro:raku-alpin-repo alex$ docker exec  -t alpine raku -e 'dd $*IN.t'
Bool::True
Julias-MacBook-Pro:raku-alpin-repo alex$ docker exec   alpine raku -e 'dd $*IN.t'
Bool::False

[lizmat]

Is there a way to recognize running in a Docket container without -t option from Raku?

[melezhik]

I am not sure if Raku has a good way to handle that, this SO could be relevant:

https://stackoverflow.com/a/63513815

BTW, running with unbuffer does the trick:

docker exec   debian  bash -c 'PATH=~/.raku/bin/:/opt/rakudo-pkg/bin:$PATH && mkdir -p /tmp/foo && cd /tmp/foo && echo foo > foo.txt && unbuffer rak foo'
foo.txt
1:foo

For the reference - https://linux.die.net/man/1/unbuffer

"unbuffer disables the output buffering that occurs when program output is redirected from non-interactive programs. For example, suppose you are watching the output from a fifo by running it through od and then more."

[melezhik]

so the question is does rak or any other dependencies/modules enable buffering on STDOUT?

[lizmat]

Buffering