3

I am getting the following error while trying to convert a PipeLine Model in spark to Pmml using JPmml.

java.lang.NoSuchMethodError: org.dmg.pmml.MiningField.setUsageType(Lorg/dmg/pmml/MiningField$UsageType;)Lorg/dmg/pmml/MiningField;

I have added all the dependencies regarding JPmml.

// https://mvnrepository.com/artifact/org.jpmml/jpmml-sparkml libraryDependencies += "org.jpmml" % "jpmml-sparkml" % "1.1.6"

// https://mvnrepository.com/artifact/org.jpmml/pmml-model libraryDependencies += "org.jpmml" % "pmml-model" % "1.3.6"

// https://mvnrepository.com/artifact/org.jpmml/pmml-evaluator libraryDependencies += "org.jpmml" % "pmml-evaluator" % "1.3.5"

Tejasvi Sharma
  • 73
  • 6

1 Answers1

1

I have added all the dependencies regarding JPmml.

You have added all dependencies, but your application can't see them, because in your application classpath, there is an Apache Spark ML provided org.jpmml:pmml-model:1.2.X (has method MiningField#setFieldUsage(MiningField$FieldUsage)), which shadows your org.jpmml:pmml-model:1.3.X (has method MiningField#setUsageType(MiningField$UsageType)): https://issues.apache.org/jira/browse/SPARK-15526

You should stop reinventing the wheel, and use the JPMML-SparkML-Package library. This application classpath/packaging issue is specifically covered in its documentation.

user1808924
  • 4,563
  • 2
  • 17
  • 20
  • Thanks for helping now code works fine on continuous features,but still i am not able to use Categorical variables. It gives error Model not created:java.lang.ClassCastException: org.jpmml.converter.BinaryFeature cannot be cast to org.jpmml.converter.ContinuousFeature – Tejasvi Sharma Mar 16 '17 at 08:59
  • Must be the same thing: https://groups.google.com/forum/#!topic/jpmml/R6t8L0pAkoc In brief, you should replace all casts with `Feature#toContinuousFeature()` method invocations – user1808924 Mar 16 '17 at 09:11
  • I have used String Indexer and One Hot Encoder to convert them to continuous features and then assembled categorical and continuous using vector assembler. – Tejasvi Sharma Mar 16 '17 at 09:14
  • The use of OneHotEncoder means that a feature has "categorical" operational type. Perhaps you shouldn't be using this feature in contexts that require "continuous" operational type? If you don't like the idea of modifying JPMML-SparkML for the time being, then it may be possible to avoid this ClassCastException by simply refactoring your pipeline - change the order of steps, assemble partial vectors not complete vectors, etc. – user1808924 Mar 16 '17 at 09:59
  • what can I use in place of one hot encoder or can u tell what can I use to use categorical features in regression? – Tejasvi Sharma Mar 16 '17 at 11:53
  • Where exactly does this `ClassCastException` occur? There are many "vulnerabilities" in that part of JPMML-SparkML codebase that deals with transformation conversions, but not a single one that deals with regression model conversions. – user1808924 Mar 16 '17 at 13:27
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/138287/discussion-between-tejasvi-sharma-and-user1808924). – Tejasvi Sharma Mar 17 '17 at 04:48