Today my Bitbucket instance stopped working for some reason. Stop, start and reboot only shows message
Migrating home directory
Without any progress. It just sits there with the progress bar without doing anything. I don't understand why it is "migrating" the home directory. I'm not upgrading, changing or anything my instance.
When I stop and then start the log file shows a few lines of messages and sits at the line that states
c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /opt/bitbucket-5.15.0/home
Is it possible to recover from this error in any way, shape or form? Why is Bitbucket migrating my home directory? What is it waiting for?
==> /opt/bitbucket/home/log/atlassian-bitbucket.log <==
2019-02-22 13:40:56,195 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.15.RELEASE)
2019-02-22 13:40:56,396 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on jupiter with PID 12057 (/opt/bitbucket-5.15.0/app/app/WEB-INF/classes started by bitbucket in /opt/bitbucket-5.15.0/home)
2019-02-22 13:40:56,397 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2019-02-22 13:40:57,590 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.15.0 (bd5e9b4 built on Wed Oct 17 03:20:25 CEST 2018)
2019-02-22 13:40:57,590 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12
2019-02-22 13:40:59,132 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 3.971 seconds (JVM running for 5.519)
2019-02-22 13:41:01,964 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /opt/bitbucket-5.15.0/home
When I stop the service Bitbucket barfs up some error messages. Which suggest that Bitbucket is no longer able to successfully manage its various threads.
==> /opt/bitbucket/home/log/launcher.log <==
2019-02-22 13:54:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode):
"localhost-startStop-2" #44 daemon prio=5 os_prio=0 tid=0x00007f6570001000 nid=0x485d in Object.wait() [0x00007f65b1759000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c19661c0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c19661c0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at com.atlassian.johnson.spring.lifecycle.LifecycleDispatcherServlet.destroy(LifecycleDispatcherServlet.java:55)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1301)
- locked <0x00000000c18bf930> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1635)
- locked <0x00000000c18bf930> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c18bf930> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5466)
- locked <0x00000000c0872398> (a org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c0872398> (a org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1436)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1425)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Tomcat-startStop-2" #43 daemon prio=5 os_prio=0 tid=0x00007f6520424000 nid=0x485c waiting on condition [0x00007f65b3041000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ccc91900> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:998)
- locked <0x00000000c0872748> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c0872748> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1436)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1425)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Thread-4" #34 prio=5 os_prio=0 tid=0x00007f6578348800 nid=0x485b waiting on condition [0x00007f6536b98000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cca8cbb0> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:998)
- locked <0x00000000c0872eb0> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c0872eb0> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:486)
- locked <0x00000000c0872eb0> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c08758d8> (a org.apache.catalina.core.StandardService)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:814)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
- locked <0x00000000c07b26c8> (a org.apache.catalina.core.StandardServer)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:376)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stopTomcat(TomcatEmbeddedServletContainer.java:256)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:310)
- locked <0x00000000c088ae18> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:306)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onClose(EmbeddedWebApplicationContext.java:155)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1014)
at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)
- locked <0x00000000c073b9a8> (a java.lang.Object)
"SIGTERM handler" #42 daemon prio=9 os_prio=0 tid=0x00007f659c001000 nid=0x4859 in Object.wait() [0x00007f65b1c5a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c1965eb0> (a org.springframework.context.support.AbstractApplicationContext$2)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c1965eb0> (a org.springframework.context.support.AbstractApplicationContext$2)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000c02068b8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
"PostgreSQL-JDBC-SharedTimer-2" #41 daemon prio=5 os_prio=0 tid=0x00007f6528f78000 nid=0x2fe9 in Object.wait() [0x00007f6536596000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ccb8d168> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000ccb8d168> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
