Maven import avro definition from dependant jar

Question

I have two maven projects that use the avro-maven-plugin for code generation. The issue I am facing is that one of these project has an avro object that uses a reference to another avro object from the second maven project. Though I include the jar produced from the second project as a maven dependency I am unable to find a way to generate the code for the second project.

In essence consider the case that my second project has an avro definition like the one below

{
  "namespace": "my.second.project.avro",
  "name": "listNode",
  "type": "record",
  "fields":[
        {
            "name":"eventList",
            "type":{
                "type": "array",  
                "items": "my.first.project.AvroDefinition.avsc"
            }
         }
    ] 
}

In this definition it refers to AvroDefinition.avsc which is included in one of the dependencies jar, but I cannot find any way to extract AvroDefinition.avsc and use it for code generation in the second project. Btw avro-maven-plugin has been setup as below:

   <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.9.1</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>schema</goal>
                    <goal>protocol</goal>
                    <goal>idl-protocol</goal>
                </goals>
                <configuration>
                    <imports>
                        <import>${project.basedir}/src/main/avro/include/</import>
                    </imports>
                    <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>                        
                    <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                    <stringType>String</stringType>
                </configuration>
            </execution>
        </executions>
    </plugin>

and the error I get is the following:

Execution default of goal org.apache.avro:avro-maven-plugin:1.9.1:schema failed: 
Undefined name: "my.first.project.AvroDefinition.avsc"

Anyway, I hoped that I could resort using maven remote resources plugin, but I couldn't make it work, so my question is how is this possible? How I can I import an avro definition from a dependant jar to generate my code ?

Answer 1

I did finally found a solution, more of a workaround though, using two maven plugins (maven-resources-plugin) and (maven-remote-resources-plugin).

Essentially the first plugin (maven-resources-plugin) was simply used to copy avsc files to the resources directory of the project so if you already have you files in the specified directory you can omit this step

The actual job is done through (maven-remote-resources-plugin) which needs to find the files to include in the jar in the src/main/resources directory (by the way for some reason I was unable to use the option on plugin) (hence the step above)

<plugin>
   <artifactId>maven-remote-resources-plugin</artifactId>
   <version>1.6.0</version>
   <executions>
     <execution>
       <goals>
         <goal>bundle</goal>
       </goals>
     </execution>
   </executions>
   <configuration>
     <includes>
        <include>**/*.avsc</include>
     </includes>
   </configuration>
</plugin>

Doing that assures that the avsc files are included in the jar, and then in your second project/module you need to execute this goal (see below) from the plugin in order to extract avsc files and generate the code

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-remote-resources-plugin</artifactId>
    <version>1.6.0</version>
    <configuration>
      <resourceBundles>
        <resourceBundle>{groupId}:{artifactId}:{version}</resourceBundle>
      </resourceBundles>
    </configuration>
    <executions>
      <execution>
        <phase>generate-sources</phase>
        <goals>
          <goal>process</goal>
        </goals>
      </execution>
    </executions>
</plugin>

Note two things, first to include the dependency jar on the resourceBundles option and secondly set the execution phase to generate-sources instead of the default generate-resources otherwise it won't work because the avro-maven-plugin does execute on the generate-sources lifecycle and it should have by then the avsc files.

As a final note you should also include on the import section of the avro-maven-plugin the following line

<import>${project.build.directory}/maven-shared-archive-resources/</import>

which is basically the directory where the avsc files will be exported by default from the maven-remote-resources-plugin

Answer 2

Building off of the accepted answer, I was able to get this to work using only the maven-remote-resources-plugin. With Maven debugging logs turned on, I noticed that no remote resource bundles were being imported, even though they were being created. This was due to an empty remote-resources.xml file in each of the dependent JARs.

This other post pointed me in the right direction, Maven remote resource plugin issue

I needed to add

<includes>
    <include>**/*.avsc</include>
</includes>

to the maven-remote-resource-plugin configuration in each of the dependent pom.xml.

In the pom.xml of the main project, I needed to import each of the remote schemas explicitly, to avoid a NullPointerException. I added this to the configuration of the Avro plugin in the pom.xml

<imports>
    <import>${project.build.directory}/maven-shared-archive-resources/MyAvroSchema.avsc</import>
</imports>

Answer 3

Here is another way to do it - pull schemas from jar file and use schemas to generate java classes

Add maven dependency plugin to pull avro jar and unpack schema files into desired location

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <configuration>
                <artifactItems>
                    <artifactItem>
                        <groupId>com.company.events.schema</groupId>
                        <artifactId>avro</artifactId>
                        <version>${avro.version}</version>
                        <outputDirectory>${project.build.outputDirectory}/avro</outputDirectory>
                        <type>jar</type>
                        <includes>**/common-header.avsc,**/label*.avsc</includes>
                    </artifactItem>
                </artifactItems>
            </configuration>
            <executions>
                <execution>
                    <id>unpack</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>unpack</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

Then add maven-avro-plugin to generate code

         <plugin>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <version>1.8.2</version>
            <executions>
                <execution>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>schema</goal>
                    </goals>
                    <configuration>
                        <imports>${project.build.outputDirectory}/avro/common-header.avsc</imports>
                        <sourceDirectory>${project.build.outputDirectory}/avro</sourceDirectory>
                        <outputDirectory>${project.build.directory}/generated-sources/avro</outputDirectory>
                        <fieldVisibility>PRIVATE</fieldVisibility>
                        <includes>
                            <include>**/*.avsc</include>
                        </includes>
                    </configuration>
                </execution>
            </executions>
        </plugin>