AWS Glue Crawler creating empty tables in Lake Formation

Question

I have successfully ingested data from a MySQL RDS database to S3 buckets with a Lake Formation blueprint.

Upon inspecting the data, approximately 41/60 tables have been correctly ingested.

Bug searching has revealed two things:

My blueprint workflow is not ingesting all tables because of this error in the blueprint/workflow:

An error occurred while calling o319.pyWriteDynamicFrame. Unknown type '245 in column 9 of 14 in binary-encoded result set.

Missing tables are being created but with no data in it. From inspecting the JSON Table properties this is being performed by the initial crawl.

I have understood that this error from point 1 is glue recognising JSON as a column type with MySQL databases.

Has anyone had an issue like this before? I have no experience with editing AWS the JDBC driver on Glue as the documentation, as always, is poor.

Is there an obvious workaround I am missing?

Here is a screenshot of the workflow which is labelled as "importing failed"

Here is the JSON table properties of a table (successful_table) that has been successfully ingested:

{
     "Name": "rds_DB_successful_table",
     "DatabaseName": "rds-ingestion",
     "CreateTime": "2020-06-23T14:07:04.000Z",
     "UpdateTime": "2020-06-23T14:07:20.000Z",
     "Retention": 0,
     "StorageDescriptor": {
          "Columns": [
               {
                    "Name": "updated_at",
                    "Type": "timestamp"
               },
               {
                    "Name": "name",
                    "Type": "string"
               },
               {
                    "Name": "created_at",
                    "Type": "timestamp"
               },
               {
                    "Name": "id",
                    "Type": "int"
               }
          ],
          "Location": "s3://XXX-data-lake/DB/rds_DB_successful_tableversion_0/",
          "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
          "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
          "Compressed": false,
          "NumberOfBuckets": 0,
          "SerdeInfo": {
               "SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
               "Parameters": {
                    "serialization.format": "1"
               }
          },
          "SortColumns": [],
          "StoredAsSubDirectories": false
     },
     "TableType": "EXTERNAL_TABLE",
     "Parameters": {
          "CreatedByJob": "RDSCONNECTOR_etl_4_b968999a",
          "CreatedByJobRun": "jr_37cc04c6fd928b9ff7a77fd50d6a98397a30c08ce3d56fae3fd618594585daea",
          "LastTransformCompletedOn": "2020-06-23 14:07:20.508091",
          "LastUpdatedByJob": "RDSCONNECTOR_etl_4_b968999a",
          "LastUpdatedByJobRun": "jr_37cc04c6fd928b9ff7a77fd50d6a98397a30c08ce3d56fae3fd618594585daea",
          "SourceConnection": "RDS Connection Type",
          "SourceTableName": "DB_successful_table",
          "SourceType": "JDBC",
          "TableVersion": "0",
          "TransformTime": "0:00:15.347357",
          "classification": "PARQUET"
     },
     "IsRegisteredWithLakeFormation": true
}

Here is the JSON table properties of a table (bad_table) that has been unsuccessfully ingested but created:

{
     "Name": "_rds_DB_bad_table",
     "DatabaseName": "rds-ingestion",
     "Owner": "owner",
     "CreateTime": "2020-06-23T13:44:19.000Z",
     "UpdateTime": "2020-06-23T13:44:19.000Z",
     "LastAccessTime": "2020-06-23T13:44:19.000Z",
     "Retention": 0,
     "StorageDescriptor": {
          "Columns": [
               {
                    "Name": "office_id",
                    "Type": "int"
               },
               {
                    "Name": "updated_at",
                    "Type": "timestamp"
               },
               {
                    "Name": "created_at",
                    "Type": "timestamp"
               },
               {
                    "Name": "id",
                    "Type": "int"
               },
               {
                    "Name": "position",
                    "Type": "int"
               },
               {
                    "Name": "id",
                    "Type": "int"
               },
               {
                    "Name": "deadline",
                    "Type": "date"
               }
          ],
          "Location": "DB.bad_table",
          "Compressed": false,
          "NumberOfBuckets": -1,
          "SerdeInfo": {
               "Parameters": {}
          },
          "BucketColumns": [],
          "SortColumns": [],
          "Parameters": {
               "CrawlerSchemaDeserializerVersion": "1.0",
               "CrawlerSchemaSerializerVersion": "1.0",
               "UPDATED_BY_CRAWLER": "RDSCONNECTOR_discoverer_57904714",
               "classification": "mysql",
               "compressionType": "none",
               "connectionName": "RDS Connection Type",
               "typeOfData": "table"
          },
          "StoredAsSubDirectories": false
     },
     "PartitionKeys": [],
     "TableType": "EXTERNAL_TABLE",
     "Parameters": {
          "CrawlerSchemaDeserializerVersion": "1.0",
          "CrawlerSchemaSerializerVersion": "1.0",
          "UPDATED_BY_CRAWLER": "RDSCONNECTOR_discoverer_57904714",
          "classification": "mysql",
          "compressionType": "none",
          "connectionName": "RDS Connection Type",
          "typeOfData": "table"
     },
     "CreatedBy": "arn:aws:sts::724135113484:assumed-role/LakeFormationWorkflowRole/AWS-Crawler",
     "IsRegisteredWithLakeFormation": false
}

Perhaps the comparison of these successful and unsuccessful JSON table properties hold the key.

Any help would be kindly appreciated!

AWS Glue Crawler creating empty tables in Lake Formation

0 Answers0