Unable to access the azure data lake contents through network proxy using Azure SDK for Java

Question

I am trying to list out the files in azure data lake Gen 2 container using azure sdk for java API through network proxy, but getting connection timed out errors every time. Followed the sample code from azure website itself.

Sample Code Link:: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-java

Code

public void connectToAzureDataLake() {
    String endpoint = "endPoint_URL";

    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
            .clientId("clientId")
            .clientSecret("clientSecret")
            .tenantId("tenantId")
            .build();

    int proxyPort = 1111;
    HttpClient httpClient = new NettyAsyncHttpClientBuilder()
            .proxy(new ProxyOptions(ProxyOptions.Type.HTTP,
                    new InetSocketAddress("proxyURL", proxyPort))
                    .setCredentials("proxyUsername", "proxyPassword")).build();


    DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    DataLakeServiceClient dataLakeServiceClient = builder.credential(clientSecretCredential)
                                                    .endpoint(endpoint).httpClient(httpClient).buildClient();


    DataLakeFileSystemClient dataLakeFileSystemClient = new DataLakeFileSystemClientBuilder()
            .endpoint(endpoint + "/file_system_name")
            .credential(clientSecretCredential)
            .buildClient();
    listFilesInDirectory(dataLakeFileSystemClient);
}

private void listFilesInDirectory(DataLakeFileSystemClient fileSystemClient) {
    ListPathsOptions options = new ListPathsOptions();
    options.setPath("folder_to_list_contents");
    PagedIterable<PathItem> pagedIterable =
            fileSystemClient.listPaths(options, null);
    java.util.Iterator<PathItem> iterator = pagedIterable.iterator();
    PathItem item = iterator.next();

    while (item != null) {
        System.out.println(item.getName());
        if (!iterator.hasNext()) {
            break;
        }
        item = iterator.next();
    }
}

Even tried providing the proxy details through system.properties and -D options as well, but not succeeded.

-D options given:

-Dhttp.proxyHost
-Dhttp.proxyPort
-Dhttp.proxyUser
-Dhttp.proxyPassword
-Dhttps.proxyHost
-Dhttps.proxyPort
-Dhttps.proxyUser
-Dhttps.proxyPassword

Any pointers to resolve the issue would be highly appreciated. Thanks in advance.

krishg · Answer 1 · 2020-11-12T11:26:01.760

[I would otherwise put this as comment since I do not have any proxy handy to test at this moment, but due to char limit, posting as answer].

It looks like you also need to set proxy option to ClientSecretCredentialBuilder

public void connectToAzureDataLake() {
    String endpoint = "endPoint_URL";
    int proxyPort = 1111;
    ProxyOptions proxyOptions = new new ProxyOptions(ProxyOptions.Type.HTTP,
                    new InetSocketAddress("proxyURL", proxyPort))
                    .setCredentials("proxyUsername", "proxyPassword");

    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
            .clientId("clientId")
            .clientSecret("clientSecret")
            .tenantId("tenantId")
            .proxyOptions(proxyOptions)
            .build();

    HttpClient httpClient = new NettyAsyncHttpClientBuilder()
            .proxy(proxyOptions).build();


    DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    DataLakeServiceClient dataLakeServiceClient = builder.credential(clientSecretCredential)
                                                    .endpoint(endpoint).httpClient(httpClient).buildClient();


    DataLakeFileSystemClient dataLakeFileSystemClient = new DataLakeFileSystemClientBuilder()
            .endpoint(endpoint + "/file_system_name")
            .credential(clientSecretCredential)
            .buildClient();
    listFilesInDirectory(dataLakeFileSystemClient);
}

Passed httpPipeline object to ClientSecretCredential as proxyOptions got deprecated and after that I am able to get the acquire the microsoft access token but still getting connection timed out error while trying to hit the datalake end point URL. Probably I might need to pass httpPipeline object to DataLakeFileSystemClient I think. — Udaya Shankara Gandhi Thalabat, Nov 12 '20 at 13:33
Thank you krishg. Tried passing httpClient for DataLakeFileSystemClient and everything is working now. So the lesson I have learnt is pass httpClient for every client instance. — Udaya Shankara Gandhi Thalabat, Nov 12 '20 at 13:38

score 1 · Answer 2 · answered Nov 12 '20 at 13:41

Solution

public void connectToAzureDataLake() {
    String endpoint = "endPoint_URL";
    int proxyPort = 1111;
    ProxyOptions proxyOptions = new new ProxyOptions(ProxyOptions.Type.HTTP,
                    new InetSocketAddress("proxyURL", proxyPort))
                    .setCredentials("proxyUsername", "proxyPassword");
    HttpClient httpClient = new NettyAsyncHttpClientBuilder()
            .proxy(proxyOptions).build();

    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
            .clientId("clientId")
            .clientSecret("clientSecret")
            .tenantId("tenantId")
            .httpClient(httpClient)
            .build();


    DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    DataLakeServiceClient dataLakeServiceClient = builder.credential(clientSecretCredential)
                                                    .endpoint(endpoint).httpClient(httpClient).buildClient();


    DataLakeFileSystemClient dataLakeFileSystemClient = new DataLakeFileSystemClientBuilder()
            .endpoint(endpoint + "/file_system_name")
            .credential(clientSecretCredential)
            .httpClient(httpClient)
            .buildClient();
    listFilesInDirectory(dataLakeFileSystemClient);
}

Unable to access the azure data lake contents through network proxy using Azure SDK for Java

2 Answers2