Uso de testcontainers en un Jenkins Docker Agent: los contenedores no se inician, NoRouteToHostException

Estoy usando una canalización declarativa de Jenkins con Docker Agents para crear y probar mi software, incluida la ejecución de pruebas de integración con testcontainers. Puedo ejecutar mis pruebas de contenedores de prueba correctamente en mi entorno de desarrollo (sin usar Jenkins), pero fallan en Jenkins.

El demonio de recolección de recursos de testcontainers Ryuk no funciona

16:29:20.255 [testcontainers-ryuk] WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32769
java.net.NoRouteToHostException: No route to host (Host unreachable)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.base/java.net.Socket.connect(Socket.java:591)
    at java.base/java.net.Socket.connect(Socket.java:540)
    at java.base/java.net.Socket.<init>(Socket.java:436)
    at java.base/java.net.Socket.<init>(Socket.java:213)
    at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:112)
    at java.base/java.lang.Thread.run(Thread.java:834)

Pude solucionar ese problema desactivando el demonio configurando la TESTCONTAINERS_RYUK_DISABLEDvariable de entorno en true. Sin embargo, algunas de las pruebas de integración siguen fallando repetidamente.

Una prueba de integración que usa un ElasticsearchContainerrepetidamente no se inicia: se agota el tiempo de espera a que responda el puerto HTTP.

17:04:57.595 [main] INFO  d.e.c.7.1] - Starting container with ID: f5c653442103b9073c76f6ed91fc9117f7cb388d576606be8bd85bd9f3b2051d
17:04:58.465 [main] INFO  d.e.c.7.1] - Container docker.elastic.co/elasticsearch/elasticsearch:6.7.1 is starting: f5c653442103b9073c76f6ed91fc9117f7cb388d576606be8bd85bd9f3b2051d
17:04:58.479 [main] INFO  o.t.c.wait.strategy.HttpWaitStrategy - /loving_swartz: Waiting for 240 seconds for URL: http://172.17.0.1:32833/
17:08:58.480 [main] ERROR d.e.c.7.1] - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:32833/ should return HTTP 200)
    at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:197)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:582)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:259)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:212)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:210)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:199)
    at
...
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
17:08:58.513 [main] ERROR d.e.c.7.1] - Log output from the failed container:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0

[2019-04-11T17:05:02,527][INFO ][o.e.e.NodeEnvironment    ] [1a_XhBT] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1.2tb], net total_space [1.2tb], types [rootfs]

[2019-04-11T17:05:02,532][INFO ][o.e.e.NodeEnvironment    ] [1a_XhBT] heap size [989.8mb], compressed ordinary object pointers [true]

[2019-04-11T17:05:02,536][INFO ][o.e.n.Node               ] [1a_XhBT] node name derived from node ID [1a_XhBTfQZWw1XLZMXrp4A]; set [node.name] to override

[2019-04-11T17:05:02,536][INFO ][o.e.n.Node               ] [1a_XhBT] version[6.7.1], pid[1], build[default/docker/2f32220/2019-04-02T15:59:27.961366Z], OS[Linux/3.10.0-957.10.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]

[2019-04-11T17:05:02,536][INFO ][o.e.n.Node               ] [1a_XhBT] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14081126934203442674, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]

...

[2019-04-11T17:05:16,338][INFO ][o.e.d.DiscoveryModule    ] [1a_XhBT] using discovery type [single-node] and host providers [settings]

[2019-04-11T17:05:17,795][INFO ][o.e.n.Node               ] [1a_XhBT] initialized

[2019-04-11T17:05:17,795][INFO ][o.e.n.Node               ] [1a_XhBT] starting ...

[2019-04-11T17:05:18,086][INFO ][o.e.t.TransportService   ] [1a_XhBT] publish_address {172.28.0.3:9300}, bound_addresses {0.0.0.0:9300}

[2019-04-11T17:05:18,128][WARN ][o.e.b.BootstrapChecks    ] [1a_XhBT] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[2019-04-11T17:05:18,299][INFO ][o.e.h.n.Netty4HttpServerTransport] [1a_XhBT] publish_address {172.28.0.3:9200}, bound_addresses {0.0.0.0:9200}

[2019-04-11T17:05:18,299][INFO ][o.e.n.Node               ] [1a_XhBT] started

[2019-04-11T17:05:18,461][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [1a_XhBT] Failed to clear cache for realms [[]]

[2019-04-11T17:05:18,542][INFO ][o.e.g.GatewayService     ] [1a_XhBT] recovered [0] indices into cluster_state

[2019-04-11T17:05:18,822][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.watch-history-9] for index patterns [.watcher-history-9*]

[2019-04-11T17:05:18,871][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.watches] for index patterns [.watches*]

[2019-04-11T17:05:18,906][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.triggered_watches] for index patterns [.triggered_watches*]

[2019-04-11T17:05:18,955][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]

[2019-04-11T17:05:19,017][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]

[2019-04-11T17:05:19,054][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]

[2019-04-11T17:05:19,100][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]

[2019-04-11T17:05:19,148][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]

[2019-04-11T17:05:19,480][INFO ][o.e.l.LicenseService     ] [1a_XhBT] license [17853035-5cf6-49c8-96ca-4d14b26325f6] mode [basic] - valid

Sin embargo, el archivo de registro de Elasticsearch parece correcto e incluye el último mensaje de registro que Elasticsearch escribe durante el inicio (sobre la licencia).

Cambiar manualmente ese contenedor para usar un en HostPortWaitStrategylugar del predeterminado HttpWaitStrategyno ayudó.

Mientras intentaba investigar o solucionar este problema, cambié mi código de prueba para iniciar explícitamente la red Docker, llamando network.getId()al Networkobjeto testcontainers . Eso luego falló con un NoRouteToHostException.

¿Cómo puedo solucionar esto?

Después de un poco de experimentación, descubrí la causa del problema. La acción fundamental está tratando de crear una red acoplable puente (utilizando docker network create, o una testcontainers Networkobjeto) dentro de un contenedor acoplable que es en sí se ejecuta en una red puente del estibador. Si hace esto, no recibirá un mensaje de error de Docker, ni el archivo de registro del demonio de Docker incluirá mensajes útiles. Pero los intentos de utilizar la red darán como resultado que "no haya una ruta al host".

Solucioné el problema dando a mis contenedores Docker más externos (los Agentes de Jenkins) acceso a la red de host, haciendo que Jenkins proporcionara una --network="host"opción a su docker runcomando:

pipeline {
    agent {
        dockerfile {
            filename 'Dockerfile.jenkinsAgent'
            additionalBuildArgs  ...
            args '-v /var/run/docker.sock:/var/run/docker.sock ... --network="host" -u jenkins:docker'
       }
    }
    stages {
...

Eso está bien porque los agentes de Jenkins no necesitan el nivel de aislamiento proporcionado por una red puente.

En mi caso fue suficiente agregar dos argumentos a las opciones del agente de Docker:

• Toma de Docker como volumen
• agregar --group-addparámetro con ID del grupo de docker

pipeline {
    agent any
    stages {
        stage('Gradle build') {
            agent {
                docker {
                    reuseNode true
                    image 'openjdk:11.0-jdk-slim'
                    args  '-v /var/run/docker.sock:/var/run/docker.sock --group-add 992'
                }
            }

            steps {
                sh 'env | sort'
                sh './gradlew build --no-daemon --stacktrace'
            }
        }
    } // stages
} // pipeline