Why is Jenkins suddenly unable to delete a workspace

Question

We have a Jenkins server which has successfully built our code over 200 times - until a couple of days ago.

We are now getting an error to indicate that Jenkins was unable to delete the workspace (full message to follow with identifying elements redacted.)

I have checked through the recent code changes, and can see nothing which may have contributed to this issue, and nothing on that server has changed for weeks.

The stack trace suggests that "the context class hudson.FilePath is missing", but the config has not been changed from the config which worked over 200 times.

Can anybody suggest steps which could be taken to fix this ?

Started by user <REDACTED>
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://<REDACTED>@bitbucket.org/<REDACTED>/<REDACTED>.git
 > git config remote.origin.url             
https://<REDACTED>@bitbucket.org/<REDACTED>/<REDACTED>.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials <REDACTED>@bitbucket
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen branch in repository origin/temp
Seen 2 remote branches
Obtained code/Jenkinsfile from <REDACTED>
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/<REDACTED>
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://<REDACTED>@bitbucket.org/<REDACTED>/<REDACTED>.git
ERROR: Failed to clean the workspace
java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/<REDACTED>. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
    at hudson.Util.deleteContentsRecursive(Util.java:252)
    at     org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/<REDACTED>/code/<REDACTED>-query/target/classes/application.properties: Operation not permitted
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238)
at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260)
at java.nio.file.Files.setPosixFilePermissions(Files.java:2045)
at hudson.Util.makeWritable(Util.java:332)
at hudson.Util.tryOnceDeleteFile(Util.java:292)
at hudson.Util.tryOnceDeleteRecursive(Util.java:383)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.deleteContentsRecursive(Util.java:247)
... 14 more
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Failed to delete workspace
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:558)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/<REDACTED>'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.Util.deleteContentsRecursive(Util.java:252)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555)
... 13 more
Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/<REDACTED>/code/<REDACTED>-query/target/classes/application.properties: Operation not permitted
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238)
at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260)
at java.nio.file.Files.setPosixFilePermissions(Files.java:2045)
at hudson.Util.makeWritable(Util.java:332)
at hudson.Util.tryOnceDeleteFile(Util.java:292)
at hudson.Util.tryOnceDeleteRecursive(Util.java:383)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.deleteContentsRecursive(Util.java:247)
... 14 more
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] step
Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: Error cloning remote repo 'origin'
Finished: FAILURE

Usually it's either a `permission` issue or your workspace is being used by another job - i.e. if your job can be ran multiple times simultaneously and you haven't added `disableConcurrentBuilds()`. — tftd, Jun 10 '18 at 16:14
We see this often on windows where a process left over from a previous build is still running. Because this process has a file open in the workspace, the workspace can not be cleaned. — pcjr, Jun 10 '18 at 20:27
Is that because of anti-virus software? We have similar problem on windows and suspect anti-virus software is inspecting the executive files. — AIMIN PAN, Oct 22 '18 at 06:54
does your filename contain any special characters? I have the same issue seems to be caused by german umlaut in filename. — Metaphox, Nov 07 '18 at 10:56
@pcjr how do you terminate that leftover process on Windows? — Amedee Van Gasse, Feb 02 '21 at 09:17
@AmedeeVanGasse I used Task Manager to manually delete the process. If I were more knowledgeable with Windows I would have tried to automate the deletion. — pcjr, Feb 04 '21 at 16:11
Yeah I was afraid it would be something like that. We stop/restart the agent service, which automatically terminates the leftover process, but unfortunately there is no way to do that from _within_ a running job. So my idea is now: avoid Windows agents as much as possible, and if you do need them, use one-shot agents that you can quickly spin up and terminate when the job is done. — Amedee Van Gasse, Feb 04 '21 at 16:23

score 40 · Answer 1 · answered Jan 11 '20 at 06:21

40

I finally found the solution that explains everything.

Cause: I was using docker within Jenkins and mounted Jenkins work directory in Docker (-v pwd:/code:rw). During runtime my program generate few files that also goes in to Jenkins work directory as it is mounted. but the user is docker root not Jenkins user, because of that Jenkins user is unable to delete those files and cause this error.

Solution: you can also share Jenkins user with docker, So all file created is same as file created by Jenkins itself.

docker run --user "$(id -u):$(id -g)" -i --rm -v /etc/passwd:/etc/passwd:ro  -v `pwd`:/code:rw docker_image:tag

So Adding This basically solve the issue by Using Jenkins user inside docker.

--user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro

answered Jan 11 '20 at 06:21

Sudhanshu Patel

759
1
5
12

2

I never have seen the need to use `--user`. Now I understand why it's so useful haha. Thank you. – Frank Escobar Mar 26 '20 at 17:47
1

Thanks! I had to also add `-v /home/jenkins/.m2:/home/jenkins/.m2`, otherwise Maven couldn't create a local repository. – msa Jan 20 '21 at 21:04
1

Addition to @msa's comment: More generally: `-v "$HOME":"$HOME"`, or if the actual Jenkins home directory shoul not be affected, mount the current directory as home directory: `-v "$(pwd)":"$HOME"` – Fabian Schmengler Mar 03 '21 at 16:30

score 25 · Accepted Answer · answered Oct 07 '20 at 14:32

This page still gets a lot of attention, and this is probably the correct answer to my original question.

I had the same problem, found some files with permission root:root in the workspace directory. Running "sudo chown -R jenkins:jenkins /var/lib/jenkins/workspace" solved the problem.

Thanks to Olivier Boudry

score 20 · Answer 3 · edited Jan 14 '20 at 06:57

20

There is something locking the file or directory in the workspace.

Someone has logged into Jenkins and is accessing the files from the workspace directly using the file system but not via Jenkins.
The build process left a zombie program.

Some other similar cases.

If you have the shell access to FS use something like lsof to find the locking process.

EDIT: After several votes I implemented several items on some linux and windows slaves. In my case I know the names of the possible zombies.
I added a special job/build that kills processes that might be left in linux.
In windows I wrote a CS program that kills possible processes and/or runs the 'UnLock IT' with the Jenkins job that also frees up resources.

edited Jan 14 '20 at 06:57

Draken

3,134
13
34
54

answered Jun 10 '18 at 21:15

KSI

353
2
9

Thank you for your help. It was most odd. I rebooted the server, and ensured that I had sole access, but still had the problem. In the end, I deleted the directory manually using sudo, and the Jenkins build was back on track ! – MarkA Jun 11 '18 at 20:14
1

@MarkA, if the problem persist after reboot I'd continue to search to find out the cause. This might repeat again. I believe you know this is not good to reboot the working server. – KSI Jun 14 '18 at 07:51
13

I had the same problem, found some files with permission root:root in the workspace directory. Running "sudo chown -R jenkins:jenkins /var/lib/jenkins/workspace" solved the problem. – Olivier Boudry May 08 '19 at 11:52
1

In my case the issue was that one of the files in the jobs' catalog had `root` as owner, therefore, `jenkins` was not allowed to remove it. I sudoed to the catalog and removed it by hand. – Remigiusz Sep 05 '19 at 10:54

score 10 · Answer 4 · answered Jul 11 '19 at 05:55

10

As a temporary solution, try renaming your project.

answered Jul 11 '19 at 05:55

Vince Twano

117
1
2

score 3 · Answer 5 · answered Apr 04 '22 at 12:56

3

I just tried renaming the job and it worked fine

answered Apr 04 '22 at 12:56

Aditya

340
2
10

score 2 · Answer 6 · answered Sep 05 '19 at 07:46

2

I am getting similar issue. The cause of issue is I am mounting Jenkins workspace to docker and execute the python code, It creates some Pycache executables with different permission and Jenkins is not able to delete that.

Temp fix for this is to remove git clone command then use sudo rm -rf code_directory then build. It will delete all code. Then restore git clone and your all other commands

answered Sep 05 '19 at 07:46

Sudhanshu Patel

759
1
5
12

1

sudo permissions aren't available to most jenkins jobs or users – hrbdg Apr 01 '20 at 20:25

score 1 · Answer 7 · answered Feb 12 '21 at 21:40

1

I deleted the directory within the workspace directory, re-ran the job, and it fixed the issue.

answered Feb 12 '21 at 21:40

bobbyt

111
2

score 1 · Answer 8 · edited Mar 17 '22 at 16:50

I had the same problem, found some files with permission root:root in the workspace directory. Tried running find . -user root -exec chmod 777 {} ; to change the permission of files but it failed because of some dangling symbolic links. Tried deleting the dangling symbolic link by find <folder name> -xtype l -delete but it also failed for some symbolic link (Since those symbolic giving Too many levels of symbolic links)

Running

sudo chown -R jenkins:jenkins /var/lib/jenkins/workspace

solved the problem.

score 1 · Answer 9 · edited Mar 17 '22 at 16:50

I am using docker within Jenkins and mounted Jenkins work directory in Docker container.

I saw the solution above

Solution: you can also share Jenkins user with docker, So all file created is same as file created by Jenkins itself.

docker run --user "$(id -u):$(id -g)" -i --rm -v /etc/passwd:/etc/passwd:ro  -v `pwd`:/code:rw docker_image:tag

So Adding This basically solve the issue by Using Jenkins user inside docker.

--user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro"

But it is not clear for me where should I run this commands in the Jenkinsfile or on the machine/node where I run my job.

score 1 · Answer 10 · answered Apr 13 '23 at 09:01

1

In my case, the job title had spaces in the name. Jenkins created the folders with quotes and apparently could not read or delete them:

I replaced the spaces with dashes and the job ran again.

answered Apr 13 '23 at 09:01

ThreeCheeseHigh

1,429
5
22
40

score 0 · Answer 11 · answered Jan 17 '21 at 15:28

0

I deleted the directory and tried again. Worked for me.

answered Jan 17 '21 at 15:28

Jaseem Abbas

5,028
6
48
73

score 0 · Answer 12 · answered May 03 '21 at 15:20

In my case, I had an error

Caused by: jenkins.util.io.CompositeIOException: Unable to delete '/home/jenkins/workspace/my_project'. Tried 3 times

But in fact, when I ssh'ed to that folder, it was empty.

Real location of the problematic folder was

/media/jenkins_home/workspace/my_project

When I deleted there and restarted the build, all went successfully. /home/jenkins/workspace/ is definitely not a symlink and I still don't understand why the folder in error log is different but guessing it is related to Docker volumes or smth..

score 0 · Answer 13 · answered Jan 04 '22 at 18:16

I ran into this problem too within my Jenkins pipeline, specifically when integrating with Flywaydb before I had any migration files so the directory I was mounting didn't exist. What fixed it for me was to create the empty directory with .gitkeep and then the volume was able to be removed afterwards.

score 0 · Answer 14 · answered Sep 21 '22 at 04:46

0

You can use this command for giving access to jenkins user on your server that is "chown -R jenkins:jenkins job_name".

answered Sep 21 '22 at 04:46

Ashutosh Kaushik

1

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Sep 24 '22 at 18:08

Why is Jenkins suddenly unable to delete a workspace

14 Answers14

Linked