3

I have a spark project using scala and sbt. At one point it references a text file which I want to be packaged. This is how it is referenced in the application source:

getClass.getResource("/myFile.txt")

This works fine running the source code with sbt run. But I want it to be packaged and deployed to a server.

In build.sbt, after some googling I have got this to work

import NativePackagerHelper._
mappings in Universal ++= directory("src/main/resources")

adding this meant that the myFile.txt appears in the resources folder in the package. created using

sbt universal:packageBin

resulting folder structure:

target - universal - bin
                   - lib
                   - resources

however when I run my packaged application from bin/my-application.bat , I get the following error

Exception in thread "main" org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:/Software/my-application-0.0.1/lib/my-application-0.0.1.jar!/myFile.txt;

Bear in mind I have zero experience of deploying scala or jvm based things so you may have to spoonfeed me the explanation

EDIT I later realised that the text file was in fact included in the .jar file. the issue then was that getResource does not work in this case and I had to adapt my code to use getResourceAsStream

Jonny Cundall
  • 2,552
  • 1
  • 21
  • 33
  • 1
    Are you getting whatever is present in resources in your jar file(jar created after packing it under target directory)? In your system after building jar(SBT jar in target) try to unzip it using winzip and try to find whatever is present in your resource folder without compiling. Your file should be present in your jar file(SBT jar which you are running) classpath. – khushbu kanojia Mar 07 '18 at 19:19

1 Answers1

4

This can have multiple reasons.

Include files in your resulting jar

You added this line, which is not correct

import NativePackagerHelper._
mappings in Universal ++= directory("src/main/resources")

The src/main/resources directory is the resourceDirectory in Compile and the contents are always present in the package jar file (not the zip!). So I would highly recommend removing this snippet as you will have your files twice in your classpath.

The mappings in Universal (documentation link) define the content of the created package (with universal:packageBin the zip file). I assume that you are using the JavaAppPackaging plugin, which configures your entire build. By default all dependencies and your actual build artifact end up in the libs folder. Start scripts are being place in bin.

The start scripts also create a valid classpath, which includes all libraries in lib and nothing else by default.

TL;DR You simply put your files in src/main/resources and they will be available on the classpath.

How to find files on the classpath

You posted this snippet

 getClass.getResource("/myFile.txt")

This will lookup a file called myFile.txt in the roots of your classpath. As in the comment suggested you should open your application jar file and find a text file myFile.txt at the root, otherwise it won't be found.

hope that helps, Muki

Muki
  • 3,513
  • 3
  • 27
  • 50