Synopsis
I can't successfully write the metadata.json to either the local file system or minio in the environment. Using the latest AWS SDK I get an error asking for AWS Region the URI despite using MINIO. I've tried.
- Defining the AWS_REGION, AWS_DEFAULT_REGION on my flink containers
- Defining MINIO_REGION on my minio container
- Prefixing my minio url with a region, http://us-east-1.storage:9000
- passing credentials and region in flink-conf.yaml
No matter what I do I get the following error:
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
- To see the initial details, docker-compose, code, and more I documented it on this issue
I've documented my journey to this point here including stack traces: https://github.com/developer-advocacy-dremio/flink-iceberg-nessie-environment/issues/1
Here is a Link to the current docker-compose.yml (all custom containers available on docker hub): https://github.com/developer-advocacy-dremio/flink-iceberg-nessie-environment/blob/main/docker-compose.yml
Dockerfile for the alexmerced/flink-iceberg image, it's where I set the flink-conf: https://github.com/AlexMercedCoder/apache-iceberg-docker-starter-image/blob/main/FLINK-ICEBERG.Dockerfile
For the jars:
- The pom.xml: https://github.com/developer-advocacy-dremio/flink-iceberg-nessie-environment/blob/main/flink-job/iceberg-flink-example/pom.xml
- The Main class: https://github.com/developer-advocacy-dremio/flink-iceberg-nessie-environment/blob/main/flink-job/iceberg-flink-example/src/main/java/com/dremio_developer_advocacy/Main.java
