Automated docker build #9

holiman · 2018-02-14T09:25:52Z

There should be an automated docker build at hub.docker.com/r/ethereum .

Define a docker file
Set up an automated build

Preferrably using a multistage build to get the size down as much as possible.
Examples:

Multistage cpp-ethereum (though, seems to be failing): https://hub.docker.com/r/ethereum/eth/~/dockerfile/
Old cpp-ethereum: https://hub.docker.com/r/ethereum/client-go/~/dockerfile/

holiman · 2018-02-14T10:25:11Z

When trying to build with a multistage docker file:

RUN apk add --no-cache \
        git \
        cmake \
        g++ \
        make \
        linux-headers \
        leveldb-dev --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
#RUN git clone --recursive https://github.com/ethereum/cpp-ethereum --branch develop --single-branch --depth 1
RUN git clone --recursive https://github.com/ethereum/testeth --branch develop --single-branch --depth 1
RUN mkdir /build
#todo -DSTATIC_LINKING=1  ??
RUN cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0
RUN cd /build && make -j8 && make install
# Install stage 
FROM alpine:latest 
RUN apk add --no-cache \
        libstdc++ \
        leveldb --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
COPY --from=builder /usr/bin/testeth /usr/bin/testeth
ENTRYPOINT ["/usr/bin/testeth"]

I ran into the following error:


common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/portability.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o
In file included from /usr/include/pthread.h:30:0,
                 from libs/fiber/src/numa/linux/pin_thread.cpp:10:
libs/fiber/src/numa/linux/pin_thread.cpp: In function 'void boost::fibers::numa::pin_thread(uint32_t)':
libs/fiber/src/numa/linux/pin_thread.cpp:27:5: error: 'memset' was not declared in this scope
     CPU_ZERO( & set);
     ^

    "/usr/bin/c++" "-std=c++11"   -O3 -finline-functions -Wno-inline -Wall -pthread -m64  -DBOOST_ALL_NO_LIB=1 -DBOOST_DISABLE_ASSERTS -DBOOST_FIBERS_SOURCE -DBOOST_FILESYSTEM_STATIC_LINK=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o" "libs/fiber/src/numa/linux/pin_thread.cpp"

...failed gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o...
common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/utf8_codecvt_facet.o
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/operations.o
gcc.archive bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/libboost_filesystem-mt.a
common.copy stage/lib/libboost_filesystem-mt.a
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/algorithm.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/round_robin.o
common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/numa
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/shared_work.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/work_stealing.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/barrier.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/numa/work_stealing.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/condition_variable.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/future.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/fiber.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/context.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/properties.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/recursive_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/recursive_timed_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/timed_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/scheduler.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/topology.o
...skipped <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a(clean) for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>numa/linux/pin_thread.o...
...skipped <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>numa/linux/pin_thread.o...
...skipped <pstage/lib>libboost_fiber-mt.a for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a...
...failed updating 2 targets...
...skipped 6 targets...
...updated 124 targets...
make[2]: *** [CMakeFiles/Boost-fiber.dir/build.make:114: Boost-fiber-prefix/src/Boost-fiber-stamp/Boost-fiber-build] Error 1
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/Boost-fiber.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[hunter ** FATAL ERROR **] Build step failed (dir: /root/.hunter/_Base/a412c45/e35271b/2450d4d/Build/Boost/__fiber
[hunter ** FATAL ERROR **] [Directory:/root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/projects/Boost/fiber]

------------------------------ WIKI -------------------------------
    https://github.com/ruslo/hunter/wiki/error.external.build.failed
-------------------------------------------------------------------

CMake Error at /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_wiki.cmake:12 (message):
Call Stack (most recent call first):
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_fatal_error.cmake:20 (hunter_wiki)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_download.cmake:648 (hunter_fatal_error)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/projects/Boost/fiber/hunter.cmake:16 (hunter_download)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_add_package.cmake:53 (include)
  CMakeLists.txt:106 (hunter_add_package)


-- Configuring incomplete, errors occurred!
See also "/build/CMakeFiles/CMakeOutput.log".
See also "/build/CMakeFiles/CMakeError.log".
The command '/bin/sh -c cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0' returned a non-zero code: 1

holiman · 2018-02-14T11:09:50Z

This works:

cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTOOLS=OFF -DTESTS=OFF

This does not :

cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0

I'm guessing either TOOLS or TESTS cause the failure above. cc @chfast

holiman · 2018-02-14T13:35:33Z

Seems to fail when I leave out -DTOOLS=OFF

holiman · 2018-02-16T11:36:40Z

When I use TESTS (but TOOLS disabled), the cmake works, but make fails with the following error :

[ 97%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libtesteth/blockchainTest.cpp.o
[ 98%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libtesteth/testHelperTest.cpp.o
[ 98%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3core/memorydb.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3core/overlaydb.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/AccountHolder.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/Client.cpp.o
make[2]: *** No rule to make target 'deps/lib/libbinaryen.a', needed by 'test/testeth'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[100%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/jsonrpc.cpp.o
make[1]: *** [CMakeFiles/Makefile2:963: test/CMakeFiles/testeth.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
The command '/bin/sh -c cd /build && make -j2 && make install' returned a non-zero code: 2

(note^ I turned down paralellism to 2, in case it was a race issue)

holiman · 2018-02-16T12:12:47Z

(tried with 1 aswell, same problem)

holiman · 2018-02-19T09:45:50Z

It builds it correctly, but seems to install it at another place than we search for it


[100%] Built target s2wasm
[ 10%] Performing install step for 'binaryen'
[  7%] Built target wasm
[ 11%] Built target ir
[ 14%] Built target asmjs
[ 16%] Built target cfg
[ 59%] Built target passes
[ 67%] Built target support
[ 70%] Built target wasm-reduce
[ 74%] Built target emscripten-optimizer
[ 77%] Built target wasm-opt
[ 80%] Built target wasm-shell
[ 82%] Built target wasm-as
[ 84%] Built target binaryen
[ 86%] Built target asm2wasm
[ 88%] Built target wasm-merge
[ 90%] Built target wasm-metadce
[ 92%] Built target wasm-dis
[ 94%] Built target wasm-ctor-eval
[ 96%] Built target wasm2asm
[100%] Built target s2wasm
Install the project...
-- Install configuration: "Release"
-- Installing: /build/deps/lib64/libbinaryen.a
-- Installing: /build/deps/include/binaryen-c.h
-- Installing: /build/deps/share/binaryen/wasm.js
-- Installing: /build/deps/share/binaryen/binaryen.js
-- Installing: /build/deps/bin/wasm-shell
-- Installing: /build/deps/bin/wasm-opt
-- Installing: /build/deps/bin/wasm-merge
-- Installing: /build/deps/bin/wasm-metadce
-- Installing: /build/deps/bin/asm2wasm
-- Installing: /build/deps/bin/wasm2asm
-- Installing: /build/deps/bin/s2wasm
-- Installing: /build/deps/bin/wasm-as
-- Installing: /build/deps/bin/wasm-dis
-- Installing: /build/deps/bin/wasm-ctor-eval
-- Installing: /build/deps/bin/wasm-reduce
[ 10%] Completed 'binaryen'
[ 10%] Built target binaryen

Notably: Installing: /build/deps/lib64/libbinaryen.a, err msg: No rule to make target 'deps/lib/libbinaryen.a'. lib64 vs lib.

pirapira · 2018-02-19T14:44:48Z

try git submodule init and git submodule update.

holiman · 2018-02-22T13:33:05Z

Ok, finally got it working. Here's my docker image

# Build stage 
FROM alpine:latest as builder
RUN apk add --no-cache \
        git \
        cmake \
        g++ \
        make \
        linux-headers \
        leveldb-dev --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
RUN git clone --recursive https://github.com/ethereum/testeth --branch develop --single-branch --depth 1
RUN mkdir /build
# See https://github.com/ethereum/cpp-ethereum/issues/4834
RUN cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTOOLS=OFF
RUN cmake --version
RUN cd /build && make -j1 && make install; exit 0
RUN cd /build && cp -r deps/lib64/libbinaryen.a deps/lib/; cp -r deps/lib64/libmpir.a deps/lib/
RUN cd /build && make -j1 && make install

# Install stage 
FROM alpine:latest
RUN apk add --no-cache \
        libstdc++ \
        leveldb --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
COPY --from=builder /build/test/testeth /usr/bin/testeth

#ENTRYPOINT ["/usr/bin/testeth"]

I disabled the entrypoint, but you should probably enable it if you use this as a 'regular' docker image. And you can probably set -j8 back aswell, I lowered it to ensure that the build failure wasn't due to paralellism.

pirapira · 2018-02-22T13:46:56Z

Thank you. I'll try the old cpp image to see what should happen.

(I've never used docker for testeth because my machine somewhere can compile testeth reasonably fast.)

pirapira · 2018-02-27T16:57:28Z

@holiman the links

https://hub.docker.com/r/ethereum/eth/~/dockerfile/
https://hub.docker.com/r/ethereum/client-go/~/dockerfile/

go 404.

olk · 2018-03-06T07:35:04Z

for error 'CPU_ZERO() ... missing memset' see boostorg/fiber#142
-> was a bug in musl

axic · 2018-03-21T15:08:27Z

Need to update the Hera submodule (point it to a recent commit) to fix this. See ethereum/aleth#4906 as a reference.

chfast · 2018-03-21T15:11:31Z

Maintaining both testeth in cpp-ethereum and the one here looks like a waste of time.

pirapira · 2018-03-22T09:19:03Z

@axic @gumb0 please talk with @chfast and decide if you guys still want to oust testeth from cpp-ethereum.

axic · 2018-03-22T09:42:48Z

Well it was supposed to be changed to use RPC only and then moved out. It seems that instead of testeth being moved to RPC, retesteth is being written in such a way.

pirapira · 2018-03-22T09:51:28Z

@axic that's a new piece of information. That sounds like I can keep using testeth in cpp-ethereum repo for filling tests. Do you still mind if I keep maintaining testeth in cpp-ethereum repository with PRs like #13 and #14?

pirapira · 2018-03-22T09:54:20Z

I thought @axic and @gumb0 didn't want to see test-maintaining PRs like #13 and #14 in cpp-ethereum repository.

pirapira · 2018-03-22T11:00:57Z

There are worse PRs like #22 for testing new tests. (The current process uses the Travis on testeth repository for PRs in tests repository.) I don't think I should bring these back to cpp-ethereum repo. Issue: ethereum/tests#436

pirapira · 2018-03-22T11:05:16Z

The duplicate of testeth cpp-ethereum repositories will be discussed in #23.

holiman added enhancement New feature or request help wanted Extra attention is needed good first issue Good for newcomers labels Feb 14, 2018

holiman mentioned this issue Feb 21, 2018

fast-sync branch build libbinaryen.a error ethereum/aleth#4834

Closed

holiman removed good first issue Good for newcomers help wanted Extra attention is needed labels Feb 22, 2018

pirapira mentioned this issue Mar 22, 2018

Can we move testeth back into cpp-ethereum? #23

Open

pirapira mentioned this issue Mar 26, 2018

Remove Hera #24

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automated docker build #9

Automated docker build #9

holiman commented Feb 14, 2018

holiman commented Feb 14, 2018

holiman commented Feb 14, 2018

holiman commented Feb 14, 2018

holiman commented Feb 16, 2018

holiman commented Feb 16, 2018

holiman commented Feb 19, 2018

pirapira commented Feb 19, 2018

holiman commented Feb 22, 2018

pirapira commented Feb 22, 2018

pirapira commented Feb 27, 2018 •

edited

Loading

olk commented Mar 6, 2018

axic commented Mar 21, 2018

chfast commented Mar 21, 2018

pirapira commented Mar 22, 2018

axic commented Mar 22, 2018

pirapira commented Mar 22, 2018

pirapira commented Mar 22, 2018

pirapira commented Mar 22, 2018 •

edited

Loading

pirapira commented Mar 22, 2018

Automated docker build #9

Automated docker build #9

Comments

holiman commented Feb 14, 2018

holiman commented Feb 14, 2018

holiman commented Feb 14, 2018

holiman commented Feb 14, 2018

holiman commented Feb 16, 2018

holiman commented Feb 16, 2018

holiman commented Feb 19, 2018

pirapira commented Feb 19, 2018

holiman commented Feb 22, 2018

pirapira commented Feb 22, 2018

pirapira commented Feb 27, 2018 • edited Loading

olk commented Mar 6, 2018

axic commented Mar 21, 2018

chfast commented Mar 21, 2018

pirapira commented Mar 22, 2018

axic commented Mar 22, 2018

pirapira commented Mar 22, 2018

pirapira commented Mar 22, 2018

pirapira commented Mar 22, 2018 • edited Loading

pirapira commented Mar 22, 2018

pirapira commented Feb 27, 2018 •

edited

Loading

pirapira commented Mar 22, 2018 •

edited

Loading