0

I'm trying to capture the string representation generated by the show() function as suggested here, but it seems like the showString method is no longer public.

logger.info('\n{}'.format(raw_data._jdf.showString(20, False, False)))

is returning the following error trace

Traceback (most recent call last):
  File "xxxxxxxx/SparkTest/main.py", line 63, in <module>
    logger.info('\n{}\n'.format(raw_data._jdf.showString(20, False, False)))
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in __call__
    return_value = get_return_value(
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/pyspark/sql/utils.py", line 128, in deco
    return f(*a, **kw)
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/protocol.py", line 330, in get_return_value
    raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling o45.showString. Trace:
py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Boolean, class java.lang.Boolean]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:832)

is there another alternative to this approach?

blackbishop
  • 30,945
  • 11
  • 55
  • 76
Juan David
  • 2,676
  • 4
  • 32
  • 42

1 Answers1

2

The error says that the method showString(Integer, Boolean, Boolean) does not exist. If you look at showString source code, it takes Integer as second argument but you're passing a boolean.

You want to set truncate=False so simply pass 0 in the second argument:

logger.info('\n{}'.format(raw_data._jdf.showString(20, 0, False)))
blackbishop
  • 30,945
  • 11
  • 55
  • 76