convert spark dataframe to aws glue dynamic frame

Question

I tried converting my spark dataframes to dynamic to output as glueparquet files but I'm getting the error

'DataFrame' object has no attribute 'fromDF'"

My code uses heavily spark dataframes. Is there a way to convert from spark dataframe to dynamic frame so I can write out as glueparquet? If so could you please provide an example, and point out what I'm doing wrong below?

code:

# importing libraries

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

glueContext = GlueContext(SparkContext.getOrCreate())

# updated 11/19/19 for error caused in error logging function

spark = glueContext.spark_session

from pyspark.sql import Window
from pyspark.sql.functions import col
from pyspark.sql.functions import first
from pyspark.sql.functions  import date_format
from pyspark.sql.functions import lit,StringType
from pyspark.sql.types import *
from pyspark.sql.functions import substring, length, min,when,format_number,dayofmonth,hour,dayofyear,month,year,weekofyear,date_format,unix_timestamp


base_pth='s3://test/'

bckt_pth1=base_pth+'test_write/glueparquet/'


test_df=glueContext.create_dynamic_frame.from_catalog(
                 database='test_inventory',
                 table_name='inventory_tz_inventory').toDF()

test_df.fromDF(test_df, glueContext, "test_nest")


glueContext.write_dynamic_frame.from_options(frame = test_nest,
                                             connection_type = "s3",
                                             connection_options = {"path": bckt_pth1+'inventory'},
                                             format = "glueparquet")

error:

'DataFrame' object has no attribute 'fromDF'
Traceback (most recent call last):
  File "/mnt/yarn/usercache/livy/appcache/application_1574556353910_0001/container_1574556353910_0001_01_000001/pyspark.zip/pyspark/sql/dataframe.py", line 1300, in __getattr__
    "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
AttributeError: 'DataFrame' object has no attribute 'fromDF'

It looks like you are trying to create dynamic frame from dynamic frame. Can you confirm test_df is a data frame, from the script I see that you are creating it as dynamic frame and not data frame. — Prabhakar Reddy, Nov 25 '19 at 02:14

score 32 · Answer 1 · edited Mar 24 '23 at 04:35

32

fromDF is a class function. Here is how you can convert Dataframe to DynamicFrame

from awsglue.dynamicframe import DynamicFrame

DynamicFrame.fromDF(test_df, glueContext, "test_nest")

AWS Docs

edited Mar 24 '23 at 04:35

John Mee

50,179
34
152
186

answered Jan 09 '20 at 16:36

Ali Sarchami

1,127
10
8

score 13 · Answer 2 · answered Feb 13 '20 at 11:58

13

Just to consolidate the answers for Scala users too, here's how to transform a Spark Dataframe to a DynamicFrame (the method fromDF doesn't exist in the scala API of the DynamicFrame) :

import com.amazonaws.services.glue.DynamicFrame  
val dynamicFrame = DynamicFrame(df, glueContext)

I hope it helps !

answered Feb 13 '20 at 11:58

mahmoud mehdi

1,493
1
19
28

probably not the place for this question, but what are the benefit to scala in glue vs pyspark for df transformations and loads? – pdangelo4 Jan 22 '22 at 01:18

convert spark dataframe to aws glue dynamic frame

2 Answers2