High latency due to non-presence of a transit provider in my country

Question

My ISP, a state owned incumbent, buys bandwidth from different transit providers. Whenever it buys transits it announces only a specific prefix (in most cases a hitherto unused) through the new transit AS. For e.g. if it runs out of bandwidth, it buys bandwidth from a new transit and announces a new prefix through it, while the same prefix is not announced (or announced with lowest metrics, so that the routes are very rarely used) via the old transits which continue to provide bandwidth to it. I am a business customer, so I have a fiber based link to the ISP and a tiny subnet is given to me.

The subnet which is provide to me is part of a prefix which is announced by the AS of a transit who, it seems, do not have a presence in my country. So when I do a trace the packets, when they leave my ISP's AS, they take about 275ms to reach the transit providers core router, which is located in USA (half the world away). Also for upstream traffic my ISP uses a transit provider (tier 1) who has a presence in my country. But the return path is always through the transit which is in USA.

So, average latency is 400ms. All the users of other ISPs in my country discover my subnet via USA. Even the traffic from neighboring countries, from Europe (which is much nearer) follows the path via USA. Sites using CDNs also resolve to ips in USA.

I have informed the ISP NOC about the issue and I have asked them to provide an ip subnet belonging to a prefix announced by a local transit (preferably a tier 1 transit provider) and I am waiting for a reply. My question: Is it a serious issue that I must follow up to get it resolved? When I compared the latency on other providers in my country, it is, in most cases, less than half of my ISPs latency. Why my ISP doesn't announce all its prefixes to all of its transit providers, so that the packets can take efficient and nearest routes to reach prefixes that originate within its network?

Can you be more clear about whether your issue is with inbound traffic (from other networks to your machines), outbound traffic (from your machines to destinations on other networks), or both? I read it as an inbound traffic issue, but others seem to read it differently. — David Schwartz, Jun 06 '12 at 08:14

Mike Pennington · Accepted Answer · 2012-06-06T09:17:51.417

Is it a serious issue that I must follow up to get it resolved?

Yes, essentially your provider is giving you bad connectivity. All customers expect their traffic to be routed within their own continent first. So if you're in India and typical Indian / European customers must cross the ocean twice to hit your servers, this is a major problem.

When I compared the latency on other providers in my country, it is, in most cases, less than half of my ISPs latency. Why my ISP doesn't announce all its prefixes to all of its transit providers, so that the packets can take efficient and nearest routes to reach prefixes that originate within its network?

This usually comes down to (choose one or more):

They failed to check whether their upstream announces your prefixes to other NAPs on your continent; perhaps they have a junior engineer handling their BGP policies who has no idea what he needs to check after making routing policy changes
They are trying to load-balance their traffic to minimize costs (because they pay per-Mbps for transit)
The upstream (in this case C&W) decided to change their own outbound routing policies and VSNL did not know

EDIT:

Actually your provider seems to be announcing the supernet to at least two different upstreams: AS 1273 and AS 4755 (VSNL)

route-views>sh ip bgp 117.240.120.0
BGP routing table entry for 117.240.112.0/20, version 1257325932
Paths: (35 available, best #26, table Default-IP-Routing-Table)
  Not advertised to any peer
  3277 3216 1273 9829  <------------------------------------------- C&W
    194.85.102.33 from 194.85.102.33 (194.85.4.4)
      Origin IGP, localpref 100, valid, external
      Community: 1273:11840 3216:3000 3216:3001 3277:3216
  6539 1273 9829
    66.59.190.221 from 66.59.190.221 (66.59.190.221)
      Origin IGP, localpref 100, valid, external
  4436 1273 9829
    69.31.111.244 from 69.31.111.244 (69.31.111.244)
      Origin IGP, metric 186, localpref 100, valid, external
      Community: 1273:11840 4436:31611
  101 101 11164 22822 1273 9829
    209.124.176.223 from 209.124.176.223 (209.124.176.223)
      Origin IGP, localpref 100, valid, external
      Community: 101:20100 101:20120 101:22100 11164:1170 11164:7790
      Extended Community: RT:101:22100
  3549 1299 1273 9829
    208.51.134.254 from 208.51.134.254 (67.17.81.150)
      Origin IGP, metric 1, localpref 100, valid, external
  2828 6453 4755 9829  <------------------------------------------- VSNL
    65.106.7.139 from 65.106.7.139 (66.239.189.139)
      Origin IGP, metric 3, localpref 100, valid, external
  16150 1299 1273 9829
    217.75.96.60 from 217.75.96.60 (217.75.96.60)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 1299:20000 16150:63392 16150:65326
  2914 1273 9829
    129.250.0.11 from 129.250.0.11 (129.250.0.12)
      Origin IGP, metric 37, localpref 100, valid, external
      Community: 2914:420 2914:1005 2914:2000 2914:3000 65504:1273
  1239 1273 9829
    144.228.241.130 from 144.228.241.130 (144.228.241.130)
      Origin IGP, localpref 100, valid, external
  3333 1273 9829
    193.0.0.56 from 193.0.0.56 (193.0.0.56)
      Origin IGP, localpref 100, valid, external

route-views>

Tracing from one of my servers in the US (Texas) takes me across the Pacific and through AS 4755 (VSNL):

[mpenning@Bucksnort ~]$ sudo lft -A 117.240.120.67

Tracing _________________________________________________________________________

TTL  LFT trace to 117.240.120.67:80/tcp
 1   [ASxxxxx] REDACTED 0.4ms
 2   [ASxxxxx] REDACTED 0.4ms
**   [neglected] no reply packets received from TTL 3
 4   [AS174] te0-1-0-7.ccr22.dfw01.atlas.cogentco.com (154.54.0.121) 0.7ms
**   [neglected] no reply packets received from TTLs 5 through 6
 7   [AS174] teleglobe.dfw03.atlas.cogentco.com (154.54.13.134) 0.9ms
 8   [AS6453] if-2-2.tcore2.DT8-Dallas.as6453.net (66.110.56.6) 261.0ms
 9   [AS6453] if-3-2.tcore1.LVW-LosAngeles.as6453.net (66.110.57.62) 262.0ms
10   [AS6453] if-2-2.tcore2.LVW-LosAngeles.as6453.net (66.110.59.2) 263.6ms
11   [ASN?] if-7-2.tcore2.SVW-Singapore.as6453.net (180.87.15.25) 267.2ms
12   [ASN?] if-5-2.tcore2.CXR-Chennai.as6453.net (180.87.15.70) 270.7ms
13   [ASN?] 180.87.37.46 272.8ms
**   [neglected] no reply packets received from TTL 14
15   [AS4755] 59.163.206.158.static.chennai.vsnl.net.in (59.163.206.158) 295.7ms
16   [AS9829] 218.248.237.161 296.7ms
**   [80/tcp failed]  Try alternate options or use -V to see packets.

[mpenning@Bucksnort ~]$

@nixnotwin, I'm curious... what country are you in? Won't matter for solving this problem, but I think economic issues like this are interesting — Mike Pennington, Jun 06 '12 at 08:08
@nixnotwin, technically it looks like they are also announcing to VSNL... please see my edit... — Mike Pennington, Jun 06 '12 at 08:50
That is because HE.net is sending your traffic to C&W first, and C&W prefers to send traffic through BSNL's NYC pop (based on your `traceroute`). What specific problem is this causing you? — Mike Pennington, Jun 06 '12 at 09:07
Here are traces to cloudflare.com who use anycasting: http://pastebin.com/DH5iLE36 There is huge latency difference, even if the path and distance are almost same. The upstream path is same, but with my ip the last two hops show heavy latency, which means the return packets come via ny cw. The bgp routes to my prefix that you posted above show only one route via VSNL. — nixnotwin, Jun 07 '12 at 01:48
Could you please anonymize/redact my IP in the traceroutes in your answer. Also if possible please delete your comments where my ISP name is mentioned. — nixnotwin, Jul 10 '12 at 04:31

High latency due to non-presence of a transit provider in my country

1 Answers1