1

Background: We are in a proxy based environment. The white-listing of Maven central by our security team takes time. Hence we have manually downloaded the project's dependencies to Jenkins agent's ~.m2/repository folder to start of build automation journey.

Nexus is our artifact publishing platform.

The problem is Maven is not referring to dependencies stored in m2 folder. It is always trying to connect with internet to download deps.

Here is our pom.xml:

<?xml version="1.0" encoding="UTF-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<!--
 | This is the configuration file for Maven. It can be specified at two levels:
 |
 |  1. User Level. This settings.xml file provides configuration for a single user,
 |                 and is normally provided in ${user.home}/.m2/settings.xml.
 |
 |                 NOTE: This location can be overridden with the CLI option:
 |
 |                 -s /path/to/user/settings.xml
 |
 |  2. Global Level. This settings.xml file provides configuration for all Maven
 |                 users on a machine (assuming they're all using the same Maven
 |                 installation). It's normally provided in
 |                 ${maven.conf}/settings.xml.
 |
 |                 NOTE: This location can be overridden with the CLI option:
 |
 |                 -gs /path/to/global/settings.xml
 |
 | The sections in this sample file are intended to give you a running start at
 | getting the most out of your Maven installation. Where appropriate, the default
 | values (values used when the setting is not specified) are provided.
 |
 |-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <!-- localRepository
   | The path to the local repository maven will use to store artifacts.
   |
   | Default: ${user.home}/.m2/repository
  <localRepository>/path/to/local/repo</localRepository>
  -->
  <localRepository>C:\Users\Netadmin\.m2\repository</localRepository>

  <!-- interactiveMode
   | This will determine whether maven prompts you when it needs input. If set to false,
   | maven will use a sensible default value, perhaps based on some other setting, for
   | the parameter in question.
   |
   | Default: true
  <interactiveMode>true</interactiveMode>
  -->

  <!-- offline
   | Determines whether maven should attempt to connect to the network when executing a build.
   | This will have an effect on artifact downloads, artifact deployment, and others.
   |
   | Default: false
  <offline>false</offline>
  -->

  <!-- pluginGroups
   | This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.
   | when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers
   | "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.
   |-->
  <pluginGroups>
    <!-- pluginGroup
     | Specifies a further group identifier to use for plugin lookup.
    <pluginGroup>com.your.plugins</pluginGroup>
    -->
  </pluginGroups>

  <!-- proxies
   | This is a list of proxies which can be used on this machine to connect to the network.
   | Unless otherwise specified (by system property or command-line switch), the first proxy
   | specification in this list marked as active will be used.
   |-->
  <proxies>
    <proxy>
      <id>nexus-abcd</id>
      <active>true</active>
      <protocol>http</protocol>
      <username>abcd</username>
      <password>abcd@1213</password>
      <host>proxylb.abcd.com</host>
      <port>80</port>
    </proxy>
  </proxies>

  <!-- servers
   | This is a list of authentication profiles, keyed by the server-id used within the system.
   | Authentication profiles can be used whenever maven must make a connection to a remote server.
   |-->
  <servers>
    <!-- server
     | Specifies the authentication information to use when connecting to a particular server, identified by
     | a unique name within the system (referred to by the 'id' attribute below).
     |
     | NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are
     |       used together.
     |
    <server>
      <id>deploymentRepo</id>
      <username>repouser</username>
      <password>repopwd</password>
    </server>
    -->
    <server>
      <id>sdg-snapshots</id>
      <username>admin</username>
      <password>admin@123</password>
    </server>

     <server>
      <id>sdg-releases</id>
      <username>admin</username>
      <password>admin@123</password>
    </server>

    <!-- Another sample, using keys to authenticate.
    <server>
      <id>siteServer</id>
      <privateKey>/path/to/private/key</privateKey>
      <passphrase>optional; leave empty if not used.</passphrase>
    </server>
    -->
  </servers>

  <!-- mirrors
   | This is a list of mirrors to be used in downloading artifacts from remote repositories.
   |
   | It works like this: a POM may declare a repository to use in resolving certain artifacts.
   | However, this repository may have problems with heavy traffic at times, so people have mirrored
   | it to several places.
   |
   | That repository definition will have a unique id, so we can create a mirror reference for that
   | repository, to be used as an alternate download site. The mirror site will be the preferred
   | server for that repository.
   |-->
  <mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    <mirror>
      <id>mirrorId</id>
      <mirrorOf>repositoryId</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://my.repository.com/repo/path</url>
    </mirror>
     -->
    <mirror>
      <id>sdg-group</id>
      <name>sdg-group</name>
      <url>http://xx.xx.xx.xxx:9043/repository/sdg-group/</url>
      <mirrorOf>*</mirrorOf>
    </mirror>
  </mirrors>

  <!-- profiles
   | This is a list of profiles which can be activated in a variety of ways, and which can modify
   | the build process. Profiles provided in the settings.xml are intended to provide local machine-
   | specific paths and repository locations which allow the build to work in the local environment.
   |
   | For example, if you have an integration testing plugin - like cactus - that needs to know where
   | your Tomcat instance is installed, you can provide a variable here such that the variable is
   | dereferenced during the build process to configure the cactus plugin.
   |
   | As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles
   | section of this document (settings.xml) - will be discussed later. Another way essentially
   | relies on the detection of a system property, either matching a particular value for the property,
   | or merely testing its existence. Profiles can also be activated by JDK version prefix, where a
   | value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.
   | Finally, the list of active profiles can be specified directly from the command line.
   |
   | NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact
   |       repositories, plugin repositories, and free-form properties to be used as configuration
   |       variables for plugins in the POM.
   |
   |-->
  <profiles>
    <!-- profile
     | Specifies a set of introductions to the build process, to be activated using one or more of the
     | mechanisms described above. For inheritance purposes, and to activate profiles via <activatedProfiles/>
     | or the command line, profiles have to have an ID that is unique.
     |
     | An encouraged best practice for profile identification is to use a consistent naming convention
     | for profiles, such as 'env-dev', 'env-test', 'env-production', 'user-jdcasey', 'user-brett', etc.
     | This will make it more intuitive to understand what the set of introduced profiles is attempting
     | to accomplish, particularly when you only have a list of profile id's for debug.
     |
     | This profile example uses the JDK version to trigger activation, and provides a JDK-specific repo.
    <profile>
      <id>jdk-1.4</id>

      <activation>
        <jdk>1.4</jdk>
      </activation>

      <repositories>
        <repository>
          <id>jdk14</id>
          <name>Repository for JDK 1.4 builds</name>
          <url>http://www.myhost.com/maven/jdk14</url>
          <layout>default</layout>
          <snapshotPolicy>always</snapshotPolicy>
        </repository>
      </repositories>
    </profile>
    -->
    <profile>
      <id>sdg</id>
      <repositories>
        <repository>
          <id>releases</id>
          <name>SDGReleaseRepository</name>
          <url>http://xx.xx.xx.xxx:9043/repository/sdg-releases/</url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
        </repository>
      </repositories>
    </profile>

    <!--
     | Here is another profile, activated by the system property 'target-env' with a value of 'dev',
     | which provides a specific path to the Tomcat instance. To use this, your plugin configuration
     | might hypothetically look like:
     |
     | ...
     | <plugin>
     |   <groupId>org.myco.myplugins</groupId>
     |   <artifactId>myplugin</artifactId>
     |
     |   <configuration>
     |     <tomcatLocation>${tomcatPath}</tomcatLocation>
     |   </configuration>
     | </plugin>
     | ...
     |
     | NOTE: If you just wanted to inject this configuration whenever someone set 'target-env' to
     |       anything, you could just leave off the <value/> inside the activation-property.
     |
    <profile>
      <id>env-dev</id>

      <activation>
        <property>
          <name>target-env</name>
          <value>dev</value>
        </property>
      </activation>

      <properties>
        <tomcatPath>/path/to/tomcat/instance</tomcatPath>
      </properties>
    </profile>
    -->
  </profiles>

  <activeProfiles>
    <activeProfile>sdg</activeProfile>
  </activeProfiles>

</settings>

We had also commented out nexus settings and yet faced same build time error (unable to download dependencies).

How to make maven refer the local repository?

Error Logs:

12:02:33  C:\Users\Netadmin\.jenkins\workspace\MART\MART_Project>D:\apache-maven-3.6.1-bin\apache-maven-3.6.1\bin\mvn -o -f parent/pom.xml clean compile install -e 
12:02:35  [INFO] Error stacktraces are turned on.
12:02:35  [INFO] Scanning for projects...
12:02:35  [WARNING] 
12:02:35  [WARNING] Some problems were encountered while building the effective model for com.icici:cmart-schedular:jar:1.0
12:02:35  [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.commons:commons-csv:jar -> duplicate declaration of version 1.0 @ line 120, column 15
12:02:35  [WARNING] 
12:02:35  [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
12:02:35  [WARNING] 
12:02:35  [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
12:02:35  [WARNING] 
12:02:35  [INFO] ------------------------------------------------------------------------
12:02:35  [INFO] Reactor Build Order:
12:02:35  [INFO] 
12:02:35  [INFO] parent                                                             [pom]
12:02:35  [INFO] mart-framework                                                    [jar]
12:02:35  [INFO] mart-domain                                                       [jar]
12:02:35  [INFO] mart-transfer-object                                              [jar]
12:02:35  [INFO] mart-fi-integration                                               [jar]
12:02:35  [INFO] mart-web                                                          [war]
12:02:35  [INFO] mart-report                                                       [war]
12:02:35  [INFO] mart-schedular                                                    [jar]
12:02:35  [INFO] mart-web-scheduler                                                [war]
12:02:35  [INFO] 
12:02:35  [INFO] --------------------------< com.icici:parent >--------------------------
12:02:35  [INFO] Building parent 1.0                                                [1/9]
12:02:35  [INFO] --------------------------------[ pom ]---------------------------------
12:02:35  [INFO] 
12:02:35  [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ parent ---
12:02:35  [INFO] ------------------------------------------------------------------------
12:02:35  [INFO] Reactor Summary:
12:02:35  [INFO] 
12:02:35  [INFO] parent 1.0 ......................................... FAILURE [  0.147 s]
12:02:35  [INFO] mart-framework 1.0 ................................ SKIPPED
12:02:35  [INFO] mart-domain 1.0 ................................... SKIPPED
12:02:35  [INFO] mart-transfer-object 1.1 .......................... SKIPPED
12:02:35  [INFO] mart-fi-integration 1.0 ........................... SKIPPED
12:02:35  [INFO] mart-web 1.0 ...................................... SKIPPED
12:02:35  [INFO] mart-report 1.0 ................................... SKIPPED
12:02:35  [INFO] mart-schedular 1.0 ................................ SKIPPED
12:02:35  [INFO] mart-web-scheduler 1.0 ............................ SKIPPED
12:02:35  [INFO] ------------------------------------------------------------------------
12:02:35  [INFO] BUILD FAILURE
12:02:35  [INFO] ------------------------------------------------------------------------
12:02:35  [INFO] Total time:  0.399 s
12:02:35  [INFO] Finished at: 2019-07-31T12:02:35+05:30
12:02:35  [INFO] ------------------------------------------------------------------------
12:02:35  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project parent: Execution default-clean of goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean failed: Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Cannot access central (https://repo.maven.apache.org/maven2) in offline mode and the artifact org.apache.maven:maven-plugin-api:jar:2.0.6 has not been downloaded from it before. -> [Help 1]
12:02:35  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project parent: Execution default-clean of goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean failed: Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Cannot access central (https://repo.maven.apache.org/maven2) in offline mode and the artifact org.apache.maven:maven-plugin-api:jar:2.0.6 has not been downloaded from it before.

POM File:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.abcd</groupId>
    <artifactId>parent</artifactId>
    <version>1.0</version>
    <packaging>pom</packaging>

    <scm>
        <url>scm:svn:https://abcd/svn/mart-workspace/branches/mart-maven</url>
        <connection>scm:svn:https://abcd/svn/repos/mart_code_workspace/branches/mart-maven</connection>
        <developerConnection>scm:svn:https://abcd/svn/repos/mart_code_workspace/branches/mart-maven</developerConnection>
        <tag>HEAD</tag>
    </scm>

    <modules>
        <module>../mart-domain</module>
        <module>../mart-framework</module>
        <module>../mart-transfer-object</module>
        <module>../mart-fi-integration</module>
        <module>../mart-web</module>
        <module>../mart-report</module>
        <module>../mart-schedular</module>
        <module>../mart-web-scheduler</module>


    </modules>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>

        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0.2</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-scm-plugin</artifactId>
                <version>1.9.2</version>
                 <configuration>
                         <connectionType>connection</connectionType>
                     <goals>install</goals>
                 </configuration>                   
            </plugin>

        </plugins>

    </build>

<!-- <distributionManagement>
    <snapshotRepository>
      <id>sdg-snapshots</id>
      <url>http://xx.xx.xx.xxx:9043/repository/sdg-snapshots/</url>
    </snapshotRepository>
    <repository>
      <id>sdg-releases</id>
      <url>http://xx.xx.xx.xxx:9043/repository/sdg-releases/</url>
    </repository>
</distributionManagement>

 -->
</project>

Note1: The build is happening successfully in eclipse (where local repository is referenced). We pasted our desktop’s m2 repo folder in our server’s m2 folder. Ideally maven in jenkins should work.

Note2: We have commented out Nexus settings in Pom and settings.xml file with this new update. Still it is behaving the same way where maven is trying to contact maven central repo.

Regards

CK5
  • 1,055
  • 3
  • 16
  • 29
  • Sounds like you have missed some dependencies in your local cache? – khmarbaise Jul 29 '19 at 07:35
  • First check on plain command line please also post your pom file etc. and add also the log output to see what is tried to be downloaded... – khmarbaise Jul 29 '19 at 08:05
  • @khmarbaise I have posted my pom file. Regarding download...it is failing right at the start with maven clean plugin download failure. But this plugin and other ones are there in my m2 folder – CK5 Jul 29 '19 at 08:10
  • First this is not a pom file it's the `settings.xml` furthermore If a plugin download fails you have not the correct version in your local cache? How are you calling Maven? – khmarbaise Jul 29 '19 at 10:15
  • @khmarbaise oops...i will paste the pom file. We are calling maven from command line (as a build step) – CK5 Jul 29 '19 at 10:35
  • I would like to know the full parameters etc. how you maven called? Furthermore the logging output would help... – khmarbaise Jul 29 '19 at 13:06
  • @khmarbaise I have updated the question with pom and error logs. – CK5 Jul 31 '19 at 06:52
  • As I expected: `e and the artifact org.apache.maven:maven-plugin-api:jar:2.0.6 has not been downloaded from it before. -> [Help 1]` ...so you missed things..furthermore I would suggest to change the structure of the project (which is indicated by your parent) apart from that `mvn -o -f parent/pom.xml clean compile install` this will run several things twice (compile for example) which is triggered also by `install`...simply use `mvn clean install` or usually you need `mvn clean verify` only. – khmarbaise Jul 31 '19 at 07:03
  • @khmarbaise I didn’t get *change the structure of your project* part...please elaborate. – CK5 Jul 31 '19 at 07:05
  • I also strongly recommend to upgrade your plugins cause you are using a maven-compiler-plugin 2.0.2 which is 12 years old..please check https://maven.apache.org/plugins/ one more thing defining a dependency in a parent is usually wrong better to use dependencyManagement instead. – khmarbaise Jul 31 '19 at 07:05
  • See for example https://github.com/khmarbaise/javaee/ parent on the root level childs in sub directories...no need to call maven like `mvn -f parent/pom.xml ..` you can simply do from the root location via `mvn clean package`...easier to handle...also check the warnings about duplicate dependencies...also best practice is to define all plugins in your pom to get reproducible builds... – khmarbaise Jul 31 '19 at 07:06
  • @khmarbaise Got it. I will update and note the observations. PS: our version control is SVN. – CK5 Jul 31 '19 at 07:09
  • It's unrelated to the used version control... – khmarbaise Jul 31 '19 at 11:14
  • @khmarbaise you were right. We talked with developers and found they forgot to add lot of maven based dependent jar files in m2 repository folder. Hence it was contacting maven central to download missing dependencies. Thanks for imparting new suggestions on project structuring, etc. I have ported those to our developer teams. – CK5 Jul 31 '19 at 11:17

1 Answers1

0

From the Maven CLI options, you can simply force it to run in offline mode:

mvn -o install or mvn --offline install

Why is it looking remotely? It could be as @KrisT indicates,, you are missing some dependencies locally.

Or you could be using a -SNAPSHOT.

The -U,--update-snapshots Forces a check for updated releases and snapshots on remote repositories. This is normally governed by the updatePolicy:

updatePolicy: This element specifies how often updates should attempt to occur. Maven will compare the local POM’s timestamp (stored in a repository’s maven-metadata file) to the remote. The choices are: always, daily (default), interval:X (where X is an integer in minutes) or never.

Essentially, if using SNAPSHOTS, every day it looks remotely for either a new SNAPSHOT or a release.

Assuming you have all dependencies locally, use the --offline option. Just remember you are in fact using that option as it is not normal and you'll be chasing ghosts if you are missing updated dependencies, even internal ones, especially if each Dev is responsible for keeping their own local updated!

Two other thoughts:

  • Consider setting up a separate account, not "admin" with the privileges to deploy artifacts and not "admin" privileges, a service account for Jenkins or set up individual users in Nexus; much better for traceability and more secure. In fact, create a separate named user as admin and disable the admin account all together!
  • Eclipse uses a different mechanism via one of its plugins and it's well known the behavior difference between letting Eclipse build natively and running mvn install is different.

ps: the opposite issue can trouble others. A SNAPSHOT, updated remotely multiple times in a day will not get pulled down again locally unless forced -U,--update-snapshot or the maven-dependency-plugin is invoked to

dependency:purge-local-repository tells Maven to clear dependency artifact files out of the local repository, and optionally re-resolve them.

thereby forcing a remote fetch.


Your build.log shows an issue, though may be unrelated:

12:02:35  [WARNING] Some problems were encountered while building the effective model for com.icici:cmart-schedular:jar:1.0
12:02:35  [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.commons:commons-csv:jar -> duplicate declaration of version 1.0 @ line 120, column 15

Investigative options

Following your post update, it would suggest you are actually still missing some dependencies locally, either direct, transitive or plugin, forcing a searc externally. I'd suggest a few thoughts, but I'm not sure how you can work through them if you can't update the local repository (how did you get the jars there the first time?).

Maven help offers some configuration insight:
mvn help:effective-settings : goal is used to view the settings that Maven actually uses to run the build. These settings are a result of merging the global file with the user's file, with the user's file taking precedence.

mvn help:effective-pom : goal is used to make visible the POM that results from the application of interpolation, inheritance and active profiles. It provides a useful way of removing the guesswork about just what ends up in the POM that Maven uses to build your project.

Maven Dependency plugin offers some application dependency insight:

mvn dependency:go-offline:Goal that resolves all project dependencies, including plugins and reports and their dependencies.

mvn dependency:list and mvn dependency:resolve-plugins provide specific information regarding dependencies or plugins

Of course, the problem may be that in order execute these, you must have the plugins and their dependencies available locally. In order to effectively process list, you must also have the dependencies.

You can perform a find ~/.m2/repository -name "*.jar" -a ! -name "*.sources.jar" -a ! -name "*.javadoc.jar" to get the list of all the jars locally

Ian W
  • 4,559
  • 2
  • 18
  • 37
  • [Updated the question with offline logs] I have added offline mode in my maven command. yet it is not seeing ~.m2/repository folder – CK5 Jul 31 '19 at 06:53