1

I am trying to write to an Azure Storage using Spring.

I am configuring the resource inside a Bean instead of Autowiring it from the Class.

BlobServiceClient client = new BlobServiceClientBuilder().connectionString(connectionString).endpoint(endpoint).buildClient();

Here is the blob name inside the Azure Storage Container:

String resourceString = "azure-blob://foo/bar.csv"

Set up the Azure Storage Pattern Resolver:

AzureStorageResourcePatternResolver storageResourcePatternResolver = new AzureStorageResourcePatternResolver(client);

Then get the Resouce:

Resource resource = storageResourcePatternResolver.getResource(resourceString);

This is the Error I am getting:

java.lang.UnsupportedOperationException: Azure storage account blob resource [container='foo', blob='bar.csv'] cannot be resolved to absolute file path
at com.azure.spring.autoconfigure.storage.resource.BlobStorageResource.getFile(BlobStorageResource.java:83) ~[azure-spring-boot-3.4.0.jar:na]
at org.springframework.batch.item.support.AbstractFileItemWriter.getOutputState(AbstractFileItemWriter.java:367) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.item.support.AbstractFileItemWriter.open(AbstractFileItemWriter.java:308) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.item.support.AbstractFileItemWriter$$FastClassBySpringCGLIB$$f2d35c3.invoke(<generated>) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.5.jar:5.3.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) ~[spring-aop-5.3.5.jar:5.3.5]
at org.springframework.batch.item.file.FlatFileItemWriter$$EnhancerBySpringCGLIB$$5ce6a900.open(<generated>) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:104) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:311) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:205) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:138) [spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:135) [spring-batch-core-4.3.2.jar:4.3.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_221]2021-05-25 09:24:36.168  INFO 4916 --- [ taskExecutor-3] o.s.batch.core.step.AbstractStep         : Step: [slaveStep:partition155] executed in 347ms2021-05-25 09:24:36.240 ERROR 4916 --- [ taskExecutor-1] o.s.batch.core.step.AbstractStep         : Encountered an error executing step slaveStep in job job1

Any help is greatly appreciated.

msuzuki
  • 105
  • 2
  • 15
  • Ok. I changed the resourceString from azure-blob to blob. And now I am getting java.lang.IllegalArgumentException Resource not found at: blob://foo/bar.csv . – msuzuki May 25 '21 at 14:38

2 Answers2

1

The searchLocation should start with azure-blob:// or azure-file://. The "blob" in your comment is incorrect.

azure-blob://foo/bar.csv means the "bar.csv" blob in "foo" container. Please check your storage, make sure the blob exists.

For example, my blob URL is https://pamelastorage123.blob.core.windows.net/pamelac/test.txt, so azure-blob://pamelac/test.txt is right.

StorageExampleApplication.java:

@SpringBootApplication
public class StorageExampleApplication {

    public static void main(String[] args) {
        SpringApplication.run(StorageExampleApplication.class, args);
        testRescource();
    }

    public static void testRescource(){
        String searchLocation = "azure-blob://<container-name>/<blob-name>";
        String connectionString = "<Connection string>";
        String endpoint = "https://<account-name>.blob.core.windows.net";

        BlobServiceClient client = new BlobServiceClientBuilder().connectionString(connectionString).endpoint(endpoint).buildClient();
        AzureStorageResourcePatternResolver storageResourcePatternResolver = new AzureStorageResourcePatternResolver(client);

        Resource resource = storageResourcePatternResolver.getResource(searchLocation);
        System.out.println(resource.getFilename());
    }

}

application.properties:

azure.storage.account-name=[storage-account-name]
azure.storage.account-key=[storage-account-access-key]
azure.storage.blob-endpoint=[storage-blob-endpoint-URL]

Result:

enter image description here

unknown
  • 6,778
  • 1
  • 5
  • 14
  • That was helpful, so it cleared the right way of wiring to the container/file. I guess my title was too open leaving to just Spring. I edited to Spring Batch. – msuzuki May 26 '21 at 15:48
  • Hi, @msuzuki. Are there any updates? Has the issue solved? – unknown May 28 '21 at 07:09
  • Hi Pamela, based on @Mahmoud Ben Hassine response, I set up the implementation for FileWriterItem for Azure Blob, and the error got error. Now I am getting another error complaining about the Output Stream being closed. I opened another question in https://stackoverflow.com/questions/67793915/spring-batch-flatfileitemwriter-error-14416-stream-is-already-closed. – msuzuki Jun 01 '21 at 19:48
0

The FlatFileItemWriter is designed to work against any resource that represents a writable file. Here is an excerpt from the Javadoc:

The location of the output file is defined by a Resource and must represent
a writable file.

The initialization process of this writer checks if the passed resource is a writable file by calling Resource#getFile. In your case, the Resource implementation that you are using, ie BlobStorageResource throws a java.lang.UnsupportedOperationException for this operation, hence your error. You need to provide a Resource implementation that represents a file as expected by the FlatFileItemWriter.

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • Thx Mahmoud! Let me give a shot on implementing the FlatFileItem for an Azure Blob. I find something similar on GCP... so I will use it as a guide https://stackoverflow.com/questions/66200606/how-to-configure-flatfileitemwriter-to-output-the-file-to-a-bytearrayrecource. I will report how this goes. – msuzuki May 26 '21 at 15:51
  • Hi Mahmoud, thx for the answer! I did the implementation of FlatFileItemWriter, now I am getting another error. Seems like the Output Stream closes when writing in a multi-threaded, https://stackoverflow.com/questions/67793915/spring-batch-flatfileitemwriter-error-14416-stream-is-already-closed – msuzuki Jun 01 '21 at 18:02