Need to monitor and troubleshoot problems to launch build agents

[cyrille-leclerc]

Problem Description

The launch of build agents can have problems, administrators need to be alerted on, to monitor, and to troubleshoot these problems.

Example of problems

• Problem with the configuration of dynamic agents (invalid credentials...)
• Outage of underlying cloud provisioning infrastructure (no more VMs available, quota reached...)

Solutions

Probably related to hudson.slaves.ComputerListener#onLaunchFailure

but probably not enough.

hudson.slaves.CloudProvisioningListener#onFailure

Issues

Cloud Providers might not report errors with the provisioner:

• throw an error when the provisioning a node failed google-compute-engine-plugin#261

[v1v]

I've been testing it out the CloudProvisioningListener interface with the Google Compute Engine plugin.

What have I seen so far?

• Google Compute Engine with oneShot configuration produces some stacktrace warnings

2021-05-21 12:02:51.759+0000 [id=100]	WARNING	hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent obs11-ubuntu-18-linux-370cgd
java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details.
	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:320)
	at hudson.slaves.SlaveComputer$$Lambda$264/0x0000000000000000.call(Unknown Source)
Caused: java.util.concurrent.ExecutionException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
	at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.lambda$getPlannedNodeFuture$0(ComputeEngineCloud.java:315)
	at com.google.jenkins.plugins.computeengine.ComputeEngineCloud$$Lambda$507/0x0000000000000000.call(Unknown Source)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:823)

Even though the agent was created when the build requested an agent and removed afterwards.

Pipeline:

pipeline {
    agent {
      label 'linux'
    }

    stages {
        stage('Build') {
            steps {
                echo 'hi'
            }
        }
    }
}

A bit of context,

• CloudProvisioningListener#onStarted is correctly executed.
• CloudProvisioningListener#onFailure is executed but I think it should not be since the worker was an ephemeral one and therefore the stacktrace should not happen after the build has finished correctly. See https://github.com/jenkinsci/jenkins/blob/862acf3e2d3fd48330a7326bd8901d82085244d6/core/src/main/java/hudson/slaves/NodeProvisioner.java#L235-L241

As far as I see the Google Compute Engine uses the OnceRetentionStrategy when the task has finished and therefore the taskCompleted is executed for the AbstractCloudSlave.

I'm not sure if we can use the CloudProvisioningListener without fixing the root cause error with the exceptions when using one shot workers.

What are your thoughts?

[v1v]

I've just asked in the jenkins-dev-mailing-list

[v1v]

The issue with the oneShot provisioning seems to be fixed with the Jenkins core version in our test instance: 2.189.1.

[v1v]

Metrics are now available in the plugin, so I'd like now to move forward with the comment in #101 (comment) and tracking the provisioning of the cloud agents using a distributed trace.

Should both traces the one we are tracking so far, linked to the ci build, linked with the one for the cloud agents? I'm not sure whether it's possible though