0

Scenario:

Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure cannot be accessible directly, even not from Azure VNETs. In other Words, HLIs cannot access Servers hosted on other VNET or on Customer On-Prem Datacentres and vice-versa. So, customer has provisioned express route between HLIs and one of the Prod VNET. So, any Servers or VM's can access these HLIs. But other VNETs (like DEV, TEST) or from Customer Locations, these HLIs are not accessible.   So, to overcome the above problem, we have provisioned SLES 12 VM on Prod VNET and made it as IPTABLE Server for NAT Translations to solve two type of issues   a) When HLI Servers needs to access APPs/DBs (here, Hadoop Server).

b) When On-Prem Datacentre Servers needs to access HLIs.   So, in this case, we have assigned Natted IPs for On-Prem Servers and also HLIs as follows. Refer the Diagram attached  

Original Subnet – 10.10.x.x

HADNA001 - 192.168.1.2

HADNA002 - 192.168.1.3

HADNA003 - 192.168.1.4  

Original Subnet – 172.168.x.x

HANANA001 – 192.168.1.5

HANANA002 – 192.168.1.6

HANANA003 – 192.168.1.7  

Problem:

Case 1: When SAP Support team of customer, started pulling data from Hadoop spark controller, they are seeing time out requests. As per them, they are seeing this only in case of High loads. For normal loads this is not happening. Stages of Pulling Data:

1) Using HANA Studio which is installed on users Laptop.

2) Connect to any one of the HLI Server (In this case, HANANA001)

3) Configure loads and trigger loads from Hadoop Controller (HADNA001) via HLI (HANANA001)

And the request timeout is dynamic, it may happen 1 – 10 times in 4 hours of loads.  

Case 2: It is observed that when we try to copy data using rsync with multiple parallel sessions from on-prem servers to HLI via iptable, after sometime sessions are getting stalled. This is not happening in case of rsync with 5 Sessions. If it is more than 5 Sessions, sessions are getting stalled. The data we are trying to copy is 2 TB having 5 GB files each. This is the reason we are trying with multiple parallel sessions, assumption is with one go more 5 GB files will be copied.

We have tried capturing tcpdump and analysed the logs, we have seens only [F] and [R] flags which is not giving much information. And also there are other networking devices in between on-prem and HLI which are not shown in attached diagram and also we have no clue on those.

This is issue is not there with other iptable servers on different environment or VNET. Till, now customer has reported us few errors they can see in HANA Studio

1) SQL Error, 403 Request Time out

2) Unable to Create Virtual Table, Request Time out.

enter image description here

enter image description here

Ram Too
  • 11
  • 2
  • Paste the output of `iptables-save -c -t nat`. Seems like the pool for SNAT is full in the some moments. – Anton Danilov Jul 29 '19 at 15:09
  • Find the output in the below link https://drive.google.com/open?id=129qhf_zNmbCS1ZCX7p4sIqg-fHkg9YAd – Ram Too Jul 30 '19 at 13:32
  • Check also output of the `conntrack -S` command. – Anton Danilov Jul 30 '19 at 13:42
  • I am unable to run this command. We are on SLES 12 SP4 When I see installed packages, following are there libnetfilter_conntrack3 libnetfilter_cttimeout1 libnetfilter_log1 libnetfilter_queue1 which are already installed – Ram Too Jul 30 '19 at 14:28
  • So investigate the kernel logs. Also you can try to increase the `nf_conntrack_max` value. – Anton Danilov Jul 30 '19 at 14:31

0 Answers0