tests: add a runc based test

Submitted by Adrian Reber on July 18, 2018, 7:49 p.m.

Details

Message ID 1531943369-10751-1-git-send-email-adrian@lisas.de
State New
Series "tests: add a runc based test"
Headers show

Commit Message

Adrian Reber July 18, 2018, 7:49 p.m.
From: Adrian Reber <areber@redhat.com>

Using runc with CRIU does currently not work if /tmp is read-only.

To detect runc breakage earlier this downloads the runc repository to
run the checkpoint part of runc's test suite.

Related: #520

Signed-off-by: Adrian Reber <areber@redhat.com>
---
 .travis.yml                    |  1 +
 scripts/travis/Dockerfile.runc |  9 +++++++++
 scripts/travis/Makefile        | 21 +++++++++++++++++++++
 3 files changed, 31 insertions(+)
 create mode 100644 scripts/travis/Dockerfile.runc

Patch hide | download patch | download mbox

diff --git a/.travis.yml b/.travis.yml
index d8a50a0..3718d70 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -25,6 +25,7 @@  env:
   - TR_ARCH=local       GCOV=1
   - TR_ARCH=centos
   - TR_ARCH=phaul-test
+  - TR_ARCH=runc-test
 matrix:
   allow_failures:
     - env: TR_ARCH=docker-test
diff --git a/scripts/travis/Dockerfile.runc b/scripts/travis/Dockerfile.runc
new file mode 100644
index 0000000..f482f5b
--- /dev/null
+++ b/scripts/travis/Dockerfile.runc
@@ -0,0 +1,9 @@ 
+FROM golang:1.10-stretch
+
+RUN apt-get update && apt-get install -y build-essential
+
+WORKDIR /go/src/github.com/opencontainers/runc
+
+ADD runc /go/src/github.com/opencontainers/runc
+
+RUN make recvtty
diff --git a/scripts/travis/Makefile b/scripts/travis/Makefile
index f7181ac..59343a9 100644
--- a/scripts/travis/Makefile
+++ b/scripts/travis/Makefile
@@ -37,5 +37,26 @@  phaul-test:
 docker-test:
 	./docker-test.sh
 
+runc-test:
+	SKIP_TRAVIS_TEST=1 ./travis-tests
+	# clone runc we actually just want the test suite
+	git clone https://github.com/opencontainers/runc
+	# download the released binary
+	curl -o runc/runc -sSL https://github.com/opencontainers/runc/releases/download/v1.0.0-rc5/runc.amd64
+	chmod 755 runc/runc
+	# for the test suite we need to build recvtty
+	docker build -t runc -f Dockerfile.runc .
+	docker run runc tar c -C contrib/cmd/recvtty recvtty | tar x -C runc/contrib/cmd/recvtty
+	# install current criu
+	install -D -m0755 ../../criu/criu /usr/local/sbin/criu
+	# get bats to run tests
+	git clone https://github.com/sstephenson/bats.git
+	cd bats && git reset --hard 03608115df2071fff4eaaff1605768c275e5f81f
+	cd bats && ./install.sh /usr/local
+	# the runc test suite does 'go env GOARCH' to detect the arch, let's fake this
+	echo '#!/bin/bash\necho travis\n' > /usr/local/bin/go
+	chmod 755 /usr/local/bin/go
+	cd runc && bats -t tests/integration/checkpoint.bats
+
 %:
 	$(MAKE) -C ../build $@$(target-suffix)

Comments

Adrian Reber July 18, 2018, 8:23 p.m.
Is this a better runc test setup? It is a bit hacky to get the runc
test suite running standalone as it can be seen in my patch.

So should we rather use the this approach or my older approach with our
own runc test setup?

I liked the previous a bit better, but this also works for me.

		Adrian

On Wed, Jul 18, 2018 at 07:49:29PM +0000, Adrian Reber wrote:
> From: Adrian Reber <areber@redhat.com>
> 
> Using runc with CRIU does currently not work if /tmp is read-only.
> 
> To detect runc breakage earlier this downloads the runc repository to
> run the checkpoint part of runc's test suite.
> 
> Related: #520
> 
> Signed-off-by: Adrian Reber <areber@redhat.com>
> ---
>  .travis.yml                    |  1 +
>  scripts/travis/Dockerfile.runc |  9 +++++++++
>  scripts/travis/Makefile        | 21 +++++++++++++++++++++
>  3 files changed, 31 insertions(+)
>  create mode 100644 scripts/travis/Dockerfile.runc
> 
> diff --git a/.travis.yml b/.travis.yml
> index d8a50a0..3718d70 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -25,6 +25,7 @@ env:
>    - TR_ARCH=local       GCOV=1
>    - TR_ARCH=centos
>    - TR_ARCH=phaul-test
> +  - TR_ARCH=runc-test
>  matrix:
>    allow_failures:
>      - env: TR_ARCH=docker-test
> diff --git a/scripts/travis/Dockerfile.runc b/scripts/travis/Dockerfile.runc
> new file mode 100644
> index 0000000..f482f5b
> --- /dev/null
> +++ b/scripts/travis/Dockerfile.runc
> @@ -0,0 +1,9 @@
> +FROM golang:1.10-stretch
> +
> +RUN apt-get update && apt-get install -y build-essential
> +
> +WORKDIR /go/src/github.com/opencontainers/runc
> +
> +ADD runc /go/src/github.com/opencontainers/runc
> +
> +RUN make recvtty
> diff --git a/scripts/travis/Makefile b/scripts/travis/Makefile
> index f7181ac..59343a9 100644
> --- a/scripts/travis/Makefile
> +++ b/scripts/travis/Makefile
> @@ -37,5 +37,26 @@ phaul-test:
>  docker-test:
>  	./docker-test.sh
>  
> +runc-test:
> +	SKIP_TRAVIS_TEST=1 ./travis-tests
> +	# clone runc we actually just want the test suite
> +	git clone https://github.com/opencontainers/runc
> +	# download the released binary
> +	curl -o runc/runc -sSL https://github.com/opencontainers/runc/releases/download/v1.0.0-rc5/runc.amd64
> +	chmod 755 runc/runc
> +	# for the test suite we need to build recvtty
> +	docker build -t runc -f Dockerfile.runc .
> +	docker run runc tar c -C contrib/cmd/recvtty recvtty | tar x -C runc/contrib/cmd/recvtty
> +	# install current criu
> +	install -D -m0755 ../../criu/criu /usr/local/sbin/criu
> +	# get bats to run tests
> +	git clone https://github.com/sstephenson/bats.git
> +	cd bats && git reset --hard 03608115df2071fff4eaaff1605768c275e5f81f
> +	cd bats && ./install.sh /usr/local
> +	# the runc test suite does 'go env GOARCH' to detect the arch, let's fake this
> +	echo '#!/bin/bash\necho travis\n' > /usr/local/bin/go
> +	chmod 755 /usr/local/bin/go
> +	cd runc && bats -t tests/integration/checkpoint.bats
> +
>  %:
>  	$(MAKE) -C ../build $@$(target-suffix)
> -- 
> 1.8.3.1
> 
> _______________________________________________
> CRIU mailing list
> CRIU@openvz.org
> https://lists.openvz.org/mailman/listinfo/criu
Andrei Vagin July 25, 2018, 5:26 a.m.
On Wed, Jul 18, 2018 at 10:23:55PM +0200, Adrian Reber wrote:
> Is this a better runc test setup? It is a bit hacky to get the runc
> test suite running standalone as it can be seen in my patch.
> 
> So should we rather use the this approach or my older approach with our
> own runc test setup?
> 
> I liked the previous a bit better, but this also works for me.
> 
> 		Adrian
>

Hi Adrian,


Look at this
https://github.com/avagin/runc/commit/fbe69e8b5c68bb52b48dfaa43b80ce693fe3da2f

I'm thinking about the idea to have a separate repo to run additional
tests in travis-ci.

I spent a few minutes and create the "ideal" runc test. Mr Travis allows
tuning a job to execute it once a day. Yes, it will not be executed for
each patch, but I think our current travis test suite is too big
already. I am not sure whether it is the good way to run runc tests.
What do you think?

Thanks,
Andrei


> On Wed, Jul 18, 2018 at 07:49:29PM +0000, Adrian Reber wrote:
> > From: Adrian Reber <areber@redhat.com>
> > 
> > Using runc with CRIU does currently not work if /tmp is read-only.
> > 
> > To detect runc breakage earlier this downloads the runc repository to
> > run the checkpoint part of runc's test suite.
> > 
> > Related: #520
> > 
> > Signed-off-by: Adrian Reber <areber@redhat.com>
> > ---
> >  .travis.yml                    |  1 +
> >  scripts/travis/Dockerfile.runc |  9 +++++++++
> >  scripts/travis/Makefile        | 21 +++++++++++++++++++++
> >  3 files changed, 31 insertions(+)
> >  create mode 100644 scripts/travis/Dockerfile.runc
> > 
> > diff --git a/.travis.yml b/.travis.yml
> > index d8a50a0..3718d70 100644
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > @@ -25,6 +25,7 @@ env:
> >    - TR_ARCH=local       GCOV=1
> >    - TR_ARCH=centos
> >    - TR_ARCH=phaul-test
> > +  - TR_ARCH=runc-test
> >  matrix:
> >    allow_failures:
> >      - env: TR_ARCH=docker-test
> > diff --git a/scripts/travis/Dockerfile.runc b/scripts/travis/Dockerfile.runc
> > new file mode 100644
> > index 0000000..f482f5b
> > --- /dev/null
> > +++ b/scripts/travis/Dockerfile.runc
> > @@ -0,0 +1,9 @@
> > +FROM golang:1.10-stretch
> > +
> > +RUN apt-get update && apt-get install -y build-essential
> > +
> > +WORKDIR /go/src/github.com/opencontainers/runc
> > +
> > +ADD runc /go/src/github.com/opencontainers/runc
> > +
> > +RUN make recvtty
> > diff --git a/scripts/travis/Makefile b/scripts/travis/Makefile
> > index f7181ac..59343a9 100644
> > --- a/scripts/travis/Makefile
> > +++ b/scripts/travis/Makefile
> > @@ -37,5 +37,26 @@ phaul-test:
> >  docker-test:
> >  	./docker-test.sh
> >  
> > +runc-test:
> > +	SKIP_TRAVIS_TEST=1 ./travis-tests
> > +	# clone runc we actually just want the test suite
> > +	git clone https://github.com/opencontainers/runc
> > +	# download the released binary
> > +	curl -o runc/runc -sSL https://github.com/opencontainers/runc/releases/download/v1.0.0-rc5/runc.amd64
> > +	chmod 755 runc/runc
> > +	# for the test suite we need to build recvtty
> > +	docker build -t runc -f Dockerfile.runc .
> > +	docker run runc tar c -C contrib/cmd/recvtty recvtty | tar x -C runc/contrib/cmd/recvtty
> > +	# install current criu
> > +	install -D -m0755 ../../criu/criu /usr/local/sbin/criu
> > +	# get bats to run tests
> > +	git clone https://github.com/sstephenson/bats.git
> > +	cd bats && git reset --hard 03608115df2071fff4eaaff1605768c275e5f81f
> > +	cd bats && ./install.sh /usr/local
> > +	# the runc test suite does 'go env GOARCH' to detect the arch, let's fake this
> > +	echo '#!/bin/bash\necho travis\n' > /usr/local/bin/go
> > +	chmod 755 /usr/local/bin/go
> > +	cd runc && bats -t tests/integration/checkpoint.bats
> > +
> >  %:
> >  	$(MAKE) -C ../build $@$(target-suffix)
> > -- 
> > 1.8.3.1
> > 
> > _______________________________________________
> > CRIU mailing list
> > CRIU@openvz.org
> > https://lists.openvz.org/mailman/listinfo/criu
Adrian Reber July 25, 2018, 6:03 a.m.
On Tue, Jul 24, 2018 at 10:26:03PM -0700, Andrei Vagin wrote:
> On Wed, Jul 18, 2018 at 10:23:55PM +0200, Adrian Reber wrote:
> > Is this a better runc test setup? It is a bit hacky to get the runc
> > test suite running standalone as it can be seen in my patch.
> > 
> > So should we rather use the this approach or my older approach with our
> > own runc test setup?
> > 
> > I liked the previous a bit better, but this also works for me.
> > 
> >
> 
> Look at this
> https://github.com/avagin/runc/commit/fbe69e8b5c68bb52b48dfaa43b80ce693fe3da2f
> 
> I'm thinking about the idea to have a separate repo to run additional
> tests in travis-ci.
> 
> I spent a few minutes and create the "ideal" runc test. Mr Travis allows
> tuning a job to execute it once a day. Yes, it will not be executed for
> each patch, but I think our current travis test suite is too big
> already.

Right, it takes sometimes over 4 hours which really seems too long. So
something like a minimal test on each supported architecture after each
push or something like that. And once a day a full test run.

Would patchwork still do the full test suite run?

> I am not sure whether it is the good way to run runc tests.
> What do you think?

For me this are two questions. Splitting the travis testing and how we
integrate the runc test.

About your changes to use the runc test suite: This runs the complete
runc test suite which might not be what we are interested in. At least
we are not interested if there are other failures in the runc test suite
which are CRIU unrelated.

I kind of like the idea of a faster result from travis. Before
submitting a patch I usually do a push to my github CRIU clone for a
travis run on my repository. If I could get the result faster, I think I
would like that.

		Adrian