1

I've been able to import the package:

import pixiedust
pixiedust.installPackage("databricks:tensorframes:0")

But when I try a simple example:

import tensorflow as tf
import tensorframes as tfs
from pyspark.sql import Row
data = [Row(x=[float(x), float(2 * x)],
            key=str(x % 2),
            z = float(x+1)) for x in range(1, 6)]
df = spark.createDataFrame(data)
tfs.print_schema(df)

I get the following error:

...

Py4JJavaError: An error occurred while calling o97.loadClass.
: java.lang.NoClassDefFoundError:com.typesafe.scalalogging.slf4j.LazyLogging

...

I've looked up the issue and it seems like there is an older scala-logging-slf4j artifact in the dependency tree. How do I remove this artifact? Once it's removed, I assume I can add a newer version with PixieDust:

pixiedust.installPackage("https://mvnrepository.com/artifact/com.typesafe.scala-logging/scala-logging-slf4j_2.10/2.1.2") pixiedust.installPackage("https://mvnrepository.com/artifact/com.typesafe.scala-logging/scala-logging-api_2.10/2.1.2")

Ross Lewis
  • 755
  • 2
  • 7
  • 17
  • [Here](https://stackoverflow.com/questions/44140581/tensorframes-not-working-with-tensorflow-on-data-science-experience) is the follow up question. I think it's more precise. – Ross Lewis May 23 '17 at 16:51

1 Answers1

1

Charles from IBM support helped me find which jars to include:

pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-slf4j_2.10/2.1.2/scala-logging-slf4j_2.10-2.1.2.jar") pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-api_2.10/2.1.2/scala-logging-api_2.10-2.1.2.jar")

This technically solves the first error, however TesnsorFrames still aren't working. I'll post another question that is more specific.

Ross Lewis
  • 755
  • 2
  • 7
  • 17
  • I've posted the follow up question [here](https://stackoverflow.com/questions/44140581/tensorframes-not-working-with-tensorflow-on-data-science-experience). – Ross Lewis May 23 '17 at 16:52