0

I am using the ML Azure Enterprise v11.0.2 to set up a 3-node cluster.

3-node-cluster

I would like to confirm the behaviour on how to restore a db archive via mlcp. It seems it is more reliable to use the individual ml VM.

1

I will get timeout error if I restore the db via the gateway (loadbalancer)

mlHost=mlcluster-prod-appgateway.australiaeast.cloudapp.azure.com

However I do see that some data have been successfully restored in 3-node-cluster.

2

I could successfully restore the db via mclp to vm1.

mlHost=mlcluster-prod-gtg45gmyxbkbw-0.australiaeast.cloudapp.azure.com

Question:

  • When should I use the loadbalancer for MLCP?
  • Is there anything I could configure in Azure gateway to resolve that timeout issue?

Follow up information

Based on David's direction, I tried with fastload and restrict_hosts. (Before I did not use those.) The mlcp import behavior is a little bit different from before. But I am still getting gateway timeout issue.

I tried to include additional settings. I think the key the thread_count. It will work if I set it to 1. If I set it to 2, I could get the fastest loading time. In conjunction with the fastload, the loading time was reduced from 600s to 180s.

I guess I will have to test to find the optimum setting. The setting might be different for different cluster configurations (eg different number or cores per node etc). But indeed, LBS increase the throughput for ML cluster.

MLCP setting

XCELERENT - I want to dance
  • 523
  • 2
  • 7
  • 2
    It would be helpful if you shared the overall MLCP configuration used. Examples of interesting points from the config: Are you using attempting to use fast-load? Have you tried to use the -restrict_hosts flag and point to the load balancer? These are the types of things that make a difference when using a load balancer in front of MarkLogic – David Ennis -CleverLlamas.com May 29 '23 at 09:02
  • @DavidEnnis-ACleverLlama, great thanks for your feedback. I left more information on the question. Could you pls help to see whether my understanding is correct? – XCELERENT - I want to dance May 29 '23 at 19:45
  • 1
    Load balancers can be tricky (max connections per client, sticky session etc can all play against you). MLCP connects and then finds out the topology of the servers - then tries to connect to each node automatically. behind a load balancer, this makes little sense since the individual IP adddresses often cannot be reached(hence fastload has little value). MLCP will automatically use -restrict-host if it thinks there is a load-balancer. I suggest restricting the host, larger batch sizes and larger transaction size to try to get more through the LB per connection. . Also threads=throughput.. – David Ennis -CleverLlamas.com May 30 '23 at 07:11
  • 1
    Can't you increase the timeout setting for the load balancer? https://stackoverflow.com/a/73288127/14419 – Mads Hansen May 31 '23 at 15:25
  • In Azure AppGateway (load balancer), the default request time out is 30s. After changing it to 300s. It works. It seems there are setting about load balancer about capacity (default basic plan is 2 nodes) and time out etc to consider when setting up ML Cluster. – XCELERENT - I want to dance May 31 '23 at 18:12

0 Answers0