0

I'm running a Spark job that does some Http calls along the line. I decided to use a library Sttp which requires an implicit Http backend, in my case HttpURLConnectionBackend(): an object apply which creates a new class instance, which is not serializable. 

trait ApiUtils extends MediaTypes {

  implicit val backend: SttpBackend[Identity, Nothing, NothingT] = HttpURLConnectionBackend()

  def post = { ...

As a consequence, because my Spark job extends this trait, all methods from this trait inside the Spark job are not available on the executors.

As a fairly brainless workaround, I made the val a def, but what's the recommended way for sorting these things out?

The Spark job itself is a class, not an object. Reason for this is to mock/stub the mentioned Http backend inside the tests.

Raf
  • 842
  • 1
  • 9
  • 25
  • why not create a new sttp connection from scratch inside the post method? And therefore placing the implicit inside the post method too. – hellmean Nov 05 '19 at 15:53
  • sure, making the `val` a `def` is kind of the same I suppose. I feel it's rather backwards to always recreate something which is just a value. I'm just wondering what the options are. – Raf Nov 05 '19 at 18:09
  • it is only logical, imho. Since you will be passing the object around to various executors/driverS. Each with a separate TCP/IP stack. You will definitely need to "recreate" it several times. You can try `implicit lazy val`, may be it'll work:) – hellmean Nov 05 '19 at 20:19

0 Answers0