0

I am attempting to extract emmbedded connection credentials in reference to this question. But I get the following error.

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._  //import everything from the package    
import org.apache.spark.sql.expressions.{Window => W} 
import org.apache.spark.sql.{functions => F}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser 

import com.amazonaws.services.glue.DynamicFrame

import com.amazonaws.regions.Regions
import com.amazonaws.services.glue.model._

import com.amazonaws.services.glue.{AWSGlue, AWSGlueClient}
import scala.collection.JavaConverters.{mapAsJavaMapConverter, seqAsJavaListConverter}
import com.amazonaws.services.sagemaker.sparksdk.IAMRole

val sc = spark.sparkContext
val glueContext: GlueContext = new GlueContext(sc)

val region = Regions.fromName("us-east-1")

// Function to create AWS glue client
def glueClient(region: Regions):
AWSGlue = AWSGlueClient.builder().withRegion(region).build()

val glue = glueClient(region =region)

glue.getConnection("{Name: name-of-embedded-connection,HidePassword: False}")

An error was encountered: :71: error: type mismatch; found : String("{Name: preprod-samtec-redw,HidePassword: False}") required: com.amazonaws.services.glue.model.GetConnectionRequest glue.getConnection("{Name: name-of-embedded-connection,HidePassword: False}")

jayrythium
  • 679
  • 4
  • 11

1 Answers1

1

As the error indicates, you need to provide a GetConnectionRequest object to the getConnectionmethod, something like:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._  //import everything from the package    
import org.apache.spark.sql.expressions.{Window => W} 
import org.apache.spark.sql.{functions => F}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser 

import com.amazonaws.services.glue.DynamicFrame

import com.amazonaws.regions.Regions
import com.amazonaws.services.glue.model._

import com.amazonaws.services.glue.{AWSGlue, AWSGlueClient}
import scala.collection.JavaConverters.{mapAsJavaMapConverter, seqAsJavaListConverter}
import com.amazonaws.services.sagemaker.sparksdk.IAMRole

val sc = spark.sparkContext
val glueContext: GlueContext = new GlueContext(sc)

val region = Regions.fromName("us-east-1")

// Function to create AWS glue client
def glueClient(region: Regions):
AWSGlue = AWSGlueClient.builder().withRegion(region).build()

val glue = glueClient(region =region)

val getConnectionRequest: GetConnectionRequest = new GetConnectionRequest()
  .withName("name-of-embedded-connection")
  .withHidePassword(false)

val getConnectionResult: GetConnectionResult =  glue.getConnection(getConnectionRequest)

val connection: Connection = getConnectionResult.getConnection()

val connectionProperties = connection.getConnectionProperties()

// Now you can extract as many properties as you need (see https://github.com/aws/aws-sdk-java/blob/2d73c9847a327eea2f673a4cd2b3449e433649e5/aws-java-sdk-glue/src/main/java/com/amazonaws/services/glue/model/Connection.java#L687)
// For instance, the username or password
val username = connectionProperties.get("USERNAME")
val password = connectionProperties.get("PASSWORD")
jccampanero
  • 50,989
  • 3
  • 20
  • 49
  • I'm getting the following error: `:1: error: identifier expected but 'new' found. val connectionRequest: new GetConnectionRequest() ^ ` – jayrythium Sep 22 '20 at 15:55
  • Sorry Jay, I forgot to include the type of the variable. Can you try now? You can also try just ```val connectionRequest = new GetConnectionRequest() .withName("name-of-embedded-connection") .withHidePassword(false)``` – jccampanero Sep 22 '20 at 16:00
  • Nice Jay!! I am very happy to hear that the answer was helpful. – jccampanero Sep 22 '20 at 16:06
  • I updated my answer with, I think, the code needed to replicate your Python code (please note that I am editing it manually, hope I haven't made any mistakes) – jccampanero Sep 22 '20 at 17:46
  • Thank you, and do not worry Jay, I am fluent in Python. Yes, I think that you only need to obtain the value like this ```val username = connectionProperties.get("USERNAME")```. I do not understand what you mean with JSON: is it related with the way you process the result in python? In Java/Scala, you have a ```Map```, analogous to a Python dictionary, and you can get the value that correspond to a certain key passing it as an argument of the ```get``` method. In fact, in your code, you are processing a Python dictionary, aren't you? – jccampanero Sep 22 '20 at 18:06
  • I basically just need it to extract the connection so that I dont have to explicitly pass credentials. I reran your code and get this error `:2: error: ';' expected but ',' found. val connectionProperties: java.util.Map = connection.getConnectionProperties() ^` any ideas? – jayrythium Sep 22 '20 at 18:28
  • Please, just remove the type: ```val connectionProperties = connection.getConnectionProperties()``` - sorry, Java bad habits!! Please, can you see if it works? – jccampanero Sep 22 '20 at 18:32
  • This did the trick. Thanks so much for all your help and prompt responses. – jayrythium Sep 22 '20 at 18:33
  • You are welcome Jay. I am really glad to know that it helps. – jccampanero Sep 22 '20 at 18:34