0
shapeless.DefaultSymbolicLabelling  
shapeless.DefaultSymbolicLabelling$.instance(shapeless.HList)

getting this error while using both pureconfig and circe. I'm using spark 3.1.2 with spark k8s operator.

Sergey Bushmanov
  • 23,310
  • 7
  • 53
  • 72
Pradyumna
  • 195
  • 3
  • 11

1 Answers1

1

This error is because of conflicting shapeless library versions. Spark 3.1.2 ships with shapeless 2.3.3 whereas both these packages need shapeless 2.3.7. To solve this I followed the steps mentioned here which involve shading ie renaming the dependency.

For SBT

If you are using the sbt-assembly plugin to create your JARs you can shade shapeless by adding to your assembly.sbt file the following setting:

assembly / assemblyShadeRules := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)

Maven

The maven-shade-plugin can shade shapeless by adding to your pom.xml file the following block:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.0.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <createDependencyReducedPom>false</createDependencyReducedPom>
        <relocations>
            <relocation>
                <pattern>shapeless</pattern>
                <shadedPattern>shapelesspureconfig</shadedPattern>
            </relocation>
        </relocations>
    </configuration>
</plugin>
Pradyumna
  • 195
  • 3
  • 11
  • I have the same issue but I couldnt find my assembly.sbt file. I am using Intellij IDEA – CodingStark Apr 21 '22 at 14:56
  • Do you happen to know where should I add the shade rules? I am using sbt but I only got plugin and build sbt files. – CodingStark Apr 21 '22 at 15:24
  • 1
    @CodingStark Just add them to build.sbt – Pradyumna Apr 25 '22 at 06:59
  • I tried that but it didnt work – CodingStark Apr 26 '22 at 03:05
  • I use assembly.sbt is used to make fat jars. Follow [this](https://www.baeldung.com/scala/sbt-fat-jar) to set it up. I have named it assembly.sbt under the project folder, whereas the tutorial has named it plugins.sbt. If you are using Injellij, you can go to the sbt shell and type assembly (ctrl + shift + s) to make the fat jar – Pradyumna Apr 26 '22 at 06:34