We have implemented Quartz Scheduler 2.2.1 using Spring 4.0.5 . The application starts up fine in a single node , however whenever we try to deploy it to a 2 node cluster , it just hangs while trying to connect to the DB and then the server rolls back the deployment to both of the nodes.
It is interesting to note that the first node to connect Quartz database starts up fine and the 2nd one just hangs , and due to the 2nd deployment timing out, both are rolled back.
Node which works trace snippet:
Working Instance:
16:13:47,116 INFO [org.quartz.core.QuartzScheduler] (ServerService Thread Pool -- 74) Scheduler meta-data: Quartz Scheduler (v2.2.1) 'WHSE12Scheduler' with instanceId 'XTHINK1424639626086'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is clustered.
16:13:47,116 INFO [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 74) Quartz scheduler 'WHSE12Scheduler' initialized from an externally provided properties instance.
16:13:47,116 INFO [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 74) Quartz scheduler version: 2.2.1
16:13:47,116 INFO [org.quartz.core.QuartzScheduler] (ServerService Thread Pool -- 74) JobFactory set to: com.innotrac.wmsbridge.utils.AutowiringSpringBeanJobFactory@18d058f6
16:13:47,116 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 74) Fetching JDBC Connection from DataSource
16:13:47,116 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 74) dds.Quartz: getConnection(null, WrappedConnectionRequestInfo@1c69b554[userName=Wmsbridge]) [0/50]
16:13:47,147 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 74) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.SimpleTriggerPersistenceDelegate
16:13:47,147 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 74) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.CronTriggerPersistenceDelegate
16:13:47,147 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 74) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.CalendarIntervalTriggerPersistenceDelegate
16:13:47,147 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 74) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.DailyTimeIntervalTriggerPersistenceDelegate
16:13:47,444 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 74) Returning JDBC Connection to DataSource
16:13:47,444 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 74) dds.Quartz: returnConnection(4149d6d, false) [0/50]
16:13:47,444 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 74) Fetching JDBC Connection from DataSource
16:13:47,444 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 74) dds.Quartz: getConnection(null, WrappedConnectionRequestInfo@1c69b554[userName=Wmsbridge]) [0/50]
16:13:47,444 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 74) Lock 'TRIGGER_ACCESS' is desired by: ServerService Thread Pool -- 74
16:13:47,444 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 74) Lock 'TRIGGER_ACCESS' is being obtained: ServerService Thread Pool -- 74
16:13:47,475 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 74) Lock 'TRIGGER_ACCESS' given to: ServerService Thread Pool -- 74
16:13:47,553 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 74) Lock 'TRIGGER_ACCESS' returned by: ServerService Thread Pool -- 74
16:13:47,553 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 74) Returning JDBC Connection to DataSource
......
18:52:18,335 INFO [org.springframework.web.servlet.DispatcherServlet] (ServerService Thread Pool -- 65) FrameworkServlet 'mvc-dispatcher': initialization completed in 1335 ms
18:52:18,335 DEBUG [org.springframework.web.servlet.DispatcherServlet] (ServerService Thread Pool -- 65) Servlet 'mvc-dispatcher' configured successfully
18:52:18,356 DEBUG [org.apache.jasper.compiler] (ServerService Thread Pool -- 65) JBWEB005025: Parent class loader is: ModuleClassLoader for Module "deployment.WmsBridgeScheduler.1.0.0.dev.war:main" from Service Module Loader
18:52:18,364 DEBUG [org.jboss.as.security] (MSC service thread 1-2) Qualified url patterns: {/=PatternInfo[pattern=/,type=3,isOverridden=false,qualifiers=[]]}
18:52:18,543 INFO [org.jboss.as.server] (ServerService Thread Pool -- 64) JBAS018559: Deployed "WmsBridgeScheduler.1.0.0.dev.war" (runtime-name : "WmsBridgeScheduler.1.0.0.dev.war")
2nd Node trace snippet just stops at:
16:13:47,163 INFO [org.quartz.core.QuartzScheduler] (ServerService Thread Pool -- 92) Scheduler meta-data: Quartz Scheduler (v2.2.1) 'WHSE12Scheduler' with instanceId 'XTHINK1424639626210'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is clustered.
16:13:47,163 INFO [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 92) Quartz scheduler 'WHSE12Scheduler' initialized from an externally provided properties instance.
16:13:47,163 INFO [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 92) Quartz scheduler version: 2.2.1
16:13:47,163 INFO [org.quartz.core.QuartzScheduler] (ServerService Thread Pool -- 92) JobFactory set to: com.innotrac.wmsbridge.utils.AutowiringSpringBeanJobFactory@3e433bf3
16:13:47,163 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 92) Fetching JDBC Connection from DataSource
16:13:47,163 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 92) dds.Quartz: getConnection(null, WrappedConnectionRequestInfo@1640f765[userName=Wmsbridge]) [0/50]
16:13:47,194 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 92) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.SimpleTriggerPersistenceDelegate
16:13:47,194 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 92) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.CronTriggerPersistenceDelegate
16:13:47,194 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 92) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.CalendarIntervalTriggerPersistenceDelegate
16:13:47,194 DEBUG [org.springframework.scheduling.quartz.LocalDataSourceJobStore] (ServerService Thread Pool -- 92) Adding TriggerPersistenceDelegate of type: org.quartz.impl.jdbcjobstore.DailyTimeIntervalTriggerPersistenceDelegate
16:13:47,459 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 92) Returning JDBC Connection to DataSource
16:13:47,459 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 92) dds.Quartz: returnConnection(72b7b5e5, false) [0/50]
16:13:47,475 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] (ServerService Thread Pool -- 92) Fetching JDBC Connection from DataSource
16:13:47,475 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 92) dds.Quartz: getConnection(null, WrappedConnectionRequestInfo@1640f765[userName=Wmsbridge]) [0/50]
16:13:47,475 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 92) Lock 'TRIGGER_ACCESS' is desired by: ServerService Thread Pool -- 92
16:13:47,475 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 92) Lock 'TRIGGER_ACCESS' is being obtained: ServerService Thread Pool -- 92
16:13:55,477 DEBUG [org.apache.catalina.session] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Start expire sessions StandardManager at 1424639635477 sessioncount 0
Then timing out:
16:18:37,395 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 62) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "WmsBridgeScheduler.1.0.0.dev.war")]'
16:18:37,395 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 62) JBAS015870: Deploy of deployment "WmsBridgeScheduler.1.0.0.dev.war" was rolled back with the following failure message: "JBAS013487: Operation timed out awaiting service container stability"
16:18:47,411 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 62) JBAS013413: Timeout after [5000] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'deploy' at address '[("deployment" => "WmsBridgeScheduler.1.0.0.dev.war")]'
16:18:55,753 DEBUG [org.apache.catalina.session] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Start expire sessions StandardManager at 1424639935753 sessioncount 0
16:18:55,753 DEBUG [org.apache.catalina.session] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) End expire sessions StandardManager processingTime 0 expired sessions: 0
If anyone has deployed Quartz Scheduler to a 2 node cluster without any locking issues, I would appreciate any insight into this issue:
Quartz Properties:
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX org.quartz.jobStore.isClustered=true org.quartz.jobStore.clusterCheckinInterval=10000 org.quartz.jobStore.tablePrefix=WHSE12_ org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount=10 org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true org.quartz.scheduler.userTransactionURL=java:jboss/UserTransaction org.quartz.scheduler.wrapJobExecutionInUserTransaction = false org.quartz.jobStore.dontSetAutoCommitFalse=false org.quartz.scheduler.instanceId=AUTO org.quartz.jobStore.misfireThreshold=10000 org.quartz.jobStore.useProperties=true org.quartz.scheduler.interruptJobsOnShutdown=true org.quartz.scheduler.skipUpdateCheck=true org.quartz.jobStore.txIsolationLevelSerializable=false org.quartz.jobStore.acquireTriggersWithinLock=false org.quartz.plugin.triggHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin org.quartz.plugin.triggHistory.triggerFiredMessage=Trigger {1}.{0} fired job {6}.{5} at {4, date,yyyy-MM-dd HH:mm:ss} org.quartz.plugin.triggHistory.triggerCompleteMessage=Trigger {1}.{0} completed firing job {6}.{5} at {4,date, yyyy-MM-dd HH:mm:ss} with resulting trigger instruction code
Quartz DataSource is JNDI Based which is for Local Transactions and use JTA =false, CCM=false on JBOSS EAP 6.3.0:
<bean id="schedulerFactoryBean" class="org.springframework.scheduling.quartz.SchedulerFactoryBean" destroy-method="destroy" lazy-init="false">
<property name="triggers">
<list>
<ref bean="someTrigger"/>
</list>
</property>
<property name="applicationContextSchedulerContextKey"><value>applicationContext</value></property>
Quartz Properties for the Scheduler
<property name="configLocation" value="classpath:properties/quartz.properties" />
<property name="dataSource" ref="quartzDataSource" />
<property name="schedulerName" value="${jdbc.schedulerName.quartz}" />
<property name="overwriteExistingJobs" value="${jdbc.overwriteExistingJobs.quartz}" />
<property name="autoStartup" value="${jdbc.autoStartup.quartz}" />
<property name="waitForJobsToCompleteOnShutdown" value="${jdbc.waitForJobsToCompleteOnShutdown.quartz}" />
<property name="jobFactory" ref="autowiringSpringBeanJobFactory" />
</bean>
On timing out, I get this:
Caused by: org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a [See nested exception: java.sql.SQLException: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a]'
23:14:49,996 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS013413: Timeout after [5000] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'add' at address '[("interface" => "management")]'
23:14:50,113 FATAL [org.jboss.as.server] (Controller Boot Thread) JBAS015957: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
23:25:08,221 ERROR [org.springframework.web.context.ContextLoader] (ServerService Thread Pool -- 70) Context initialization failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'schedulerFactoryBean' defined in ServletContext resource [/WEB-INF/spring-infrastructure.xml]: Invocation of init method failed; nested exception is org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a [See nested exception: java.sql.SQLException: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:684) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760) [spring-context-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) [spring-context-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) [spring-web-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) [spring-web-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) [spring-web-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:3339) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.catalina.core.StandardContext.start(StandardContext.java:3777) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.jboss.as.web.deployment.WebDeploymentService.doStart(WebDeploymentService.java:161) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.deployment.WebDeploymentService.access$000(WebDeploymentService.java:59) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.deployment.WebDeploymentService$1.run(WebDeploymentService.java:94) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a [See nested exception: java.sql.SQLException: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a]
at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:157) [quartz-2.2.1.jar:]
at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:113) [quartz-2.2.1.jar:]
at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:238) [quartz-2.2.1.jar:]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1086) [quartz-2.2.1.jar:]
at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:969) [quartz-2.2.1.jar:]
at org.quartz.core.QuartzScheduler.addJob(QuartzScheduler.java:958) [quartz-2.2.1.jar:]
at org.quartz.impl.StdScheduler.addJob(StdScheduler.java:268) [quartz-2.2.1.jar:]
at org.springframework.scheduling.quartz.SchedulerAccessor.addJobToScheduler(SchedulerAccessor.java:337) [spring-context-support-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.scheduling.quartz.SchedulerAccessor.addTriggerToScheduler(SchedulerAccessor.java:360) [spring-context-support-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:298) [spring-context-support-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:514) [spring-context-support-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1612) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549) [spring-beans-4.0.5.RELEASE.jar:4.0.5.RELEASE]
... 23 more
Caused by: java.sql.SQLException: Connection is not associated with a managed connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@37427b2a
at org.jboss.jca.adapters.jdbc.WrappedConnection.lock(WrappedConnection.java:155)
at org.jboss.jca.adapters.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:395)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_71]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73) [quartz-2.2.1.jar:]
at com.sun.proxy.$Proxy36.prepareStatement(Unknown Source)
at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:88) [quartz-2.2.1.jar:]
... 35 more
