Why does packet loss seriously hurt application performance over the WAN? How do we address it?

Network-Switches-234-x-575

TCP was never optimized for high-bandwidth WANs or interactive applications over the WAN. Packet loss has the greatest impact on the performance of most applications over the WAN, by design.

Why is packet loss such a killer? There are many reasons, most having to do with the nature of how TCP was designed especially how TCP does congestion control/congestion avoidance. The key issue revolves around dealing with contention for limited bandwidth.

TCP is designed to use all available bandwidth, and to use it “fairly” across flows on average. To do this, given that each end station and TCP flow doesn’t know how much bandwidth is available – neither if the single flow was the only one using bandwidth end-to-end at the moment, nor in the more typical case when given multiple flows, the amount available changes moment to moment.  So the sender of the TCP data needs a way to know when “enough is enough.” Packet loss is the basic signal of this.

TCP and routers together are designed to control data flow to prevent over-utilization of the network and the potential of congestion. The goals of TCP’s design are to minimize the amount of time that the data flow grinds to a halt (congestion avoidance), and to react appropriately to reduce traffic at those times that it does (congestion control).

TCP packets received by the receiving station are acknowledged back to the sending station. TCP is a window-based protocol, meaning that it can have a certain amount of traffic “in flight” between sending station and receiving station. It is designed to back off and substantially reduce the amount of bandwidth offered (by half) when packet loss is observed. Further, until the lost packet is received, and acknowledged by the receiver, only limited amounts of additional packets will be offered. Even for those applications that use multiple TCP flows, the similar principle applies that only so many new flows opened/packets sent until a lost packet is received at the other end and its receipt acknowledged.

Packet loss is detected in one of two ways. For a longer transfer where just a packet or two is lost, the sender notices and reacts to the loss when subsequent packets are acknowledged by the receiver, but not the missing one. Alternatively – and more typically for new or short TCP flows – packet loss is detected by the occurrence of a “timeout”: the absence of receipt of an acknowledgement of the packet. The amount of time until a “timeout” is deemed to have occurred varies typically between a couple hundred milliseconds and three seconds.

TCP is an elegant protocol designed over 40 years ago when CPU and memory was extremely expensive. This worked – and continues to work – fantastically well on high-bandwidth, low-latency LANs and on low-bandwidth, high-latency WANs. But TCP wasn’t designed to work optimally in the medium-to-high bandwidth, high-latency environment that characterizes most WAN use today. TCP also wasn’t designed optimally for running interactive applications (web browsing, remote desktop) across very long-distance WANs.

TCP particularly was designed so that each end station could make its decisions completely independently of every end station. This conservative approach contributes to network stability and minimization of congestion.

Because the amount of data offered into the network is reduced by half – and only increased slowly thereafter as packets received successfully are acknowledged – when a single packet loss is detected by the sending station, WAN packet loss can have a huge impact on large transfer performance.  This is why private networks, such as MPLS, VPLS or IEPL improve application performance so significantly: they nearly eliminate packet loss.

What else can be done about  packet loss? Well, at a standards-compliant end station, pretty much nothing. But for an intelligent device in the middle of the network, and especially one at a key WAN edge location, there are many possibilities. There are at least six different approaches to minimizing the impact of WAN packet loss on application performance:

- Drastically reduce the number of WAN packets transmitted.

- React differently to loss (if good knowledge of the network in between).

- Mitigate the effects of the loss and hide it from the end station.

- Enable the end stations to react more quickly to loss.

- Avoid much of the loss in the first place (think MPLS, VPLS, IEPL)

- Avoid the additional loss that often follows after a burst of loss.

Application layer solutions are the first, most obvious approach here.  Doing replicated file service avoids WAN packet loss in accessing files, delivering full LAN-speed performance, because all client access to the data is in fact done locally.

Similarly, “static” caching of objects via a local web (HTTP) object cache completely avoids WAN access for those objects, and thus any impact from packet loss.

Beyond these, drastically reducing the number of packets transmitted is an area where WAN Optimization offerings do a great job.  Now, since we’re talking about reducing the number of packets transmitted, you might think first of memory-based compression, which is one of the techniques almost every WAN Optimization solution offers. Memory-based compression can reduce the time it takes to do the first-time transmission of data – a factor of two for compressible data is typical – but in fact it doesn’t do proportionately better in the face of packet loss than when there is little or no loss. Reducing the amount of data sent by 50% doesn’t really help that much when it comes to packet loss and its impact on a window-based protocol like TCP. So while memory-based compression certainly doesn’t hurt here, it’s not really the answer when the problem is WAN packet loss.

There are two other technologies in most WAN Optimization products that do have a large performance impact in the face of packet loss: data deduplication, and CIFS-specific application proxy.

Data deduplication essentially does “dynamic” caching of data locally, and while this requires at least one round-trip across the WAN, it will always involve far fewer such round-trip transactions than when the data is not stored locally. Besides saving bandwidth and speeding up data transfers in the more typical case of little to no packet loss, the application speed-up is proportionately greater still in the face of any meaningful amount of packet loss. And data deduplication is usually applicable for any application, not just for file access.

For the very chatty Microsoft CIFS protocol, data deduplication is usually combined with an application-specific proxy that will reduce round-trip requests still further. By essentially doing local CIFS termination, a CIFS proxy provides much faster access to files on a remotely located file server even for the first access. The impact on application performance of the combination of data deduplication and CIFS proxy can be 10 to 40 times even when there is no packet loss; in the face of packet loss, the additional benefit can be another 2x to 10x, meaning a combined performance impact of anywhere from 20x to 400x or more. For files that have been previously accessed across the WAN, this is essentially full LAN-speed performance, versus the very slow, often unusable WAN performance under packet loss if accessing large files across a WAN completely unaided.

Andy Gottleib is a twenty-five year data networking veteran, who founded Talari Networks, a pioneer in WAN Virtualization technology, and served as its first CEO, and is now leading product management at Aryaka Networks. Andy is the author of an upcoming book on Next-generation Enterprise WANs.  His bog is located at http://www.networkworld.com/community/blog/26142

 

Bursting on an MPLS or VPLS Network

Global Ethernet VPLSA handful of carriers support bursting on their IP-VPN networks.

Bursting can provide a real cost savings benefit to customers. First, it’s a solution for customers who don’t how much bandwidth they need at a given location. Bursting lets these customers add locations to their network at the lowest level of cost commitment.

Second, it is an ideal solution for customers who know that their bandwidth needs may spike much higher than normal due to, for example, seasonal traffic peaks. Bursting lets these customers commit to the least amount of bandwidth they need for continued use and pay for only what they use in excess of that amount.

Here’s an example of how it works:

A customer commits to 100 Mbps of bandwidth at a location. The carrier sets up their IP VPN port to handle bursting. That location can now burst traffic all the way up to 1 Gbps, or whatever their local loop capacity is. For instance, you might pay for a 100M Ethernet local loop, but pay for a committed port of 20M.  This will allow you to burst to the full 100M when the need arises.

The carrier then samples traffic leaving the port throughout the month. At the end of the month, billing is calculated for the:

  • 100M local loop
  • Committed 20 Mbps port rate; plus the,
  • Sustained traffic rate in excess of the committed rate.

The carrier typically discards the top 5% of the traffic samples taken during the month. This eliminates any spurious or unusual traffic from the billing measurement.

If customers find that their sustained traffic is significantly higher than their committed data rate (in this case, 20 Mbps), they can increase their commitment and take advantage of lower prices at higher committed sustained data rates.

If you are considering changes to your Wide Area Network and would like some specialized assistance with the process, please contact us!

VPLS Prices Drop Dramatically

Prices in the marketplace are dropping as more carriers offer Ethernet Virtual Private LAN Service (VPLS).  But due to limited availability in some regions, VPLS can still be an expensive WAN option to implement.

According to a recent report by research firm TeleGeography, Western Europe and the United States saw the largest price drops for VPLS services year over year.

What helped drive down VPLS costs in Europe were two factors: availability and growing competition. Europe was the first region to deploy carrier Ethernet, meaning that VPLS is now “more widely available in Europe than in any other region.” Throughout Europe there is a growing base of incumbent and competitive players offering VPLS service.

London and Frankfurt saw a strong drop in VPLS service prices. A FastE VPLS port in London was $2,918 per month in H2 2012, while the price of the same service in Frankfurt was $1,752 per month, 39 percent less than what enterprise customers had to pay last year.  So for corporate networks that include Europe and the USA, prices are exceedingly attractive.

In markets like the key cities of South America,  there aren’t as many service options, so VPLS continues to be expensive. The median price for a FastE connection in Sao Paulo was $10,973 in Q2 2012, which is twice what a business customer would pay in New York or Los Angeles, two markets where there are multiple service provider options. Again, the issue with the prices is simply service availability. In New York, VPLS availability was 28 percent versus 6 percent in Sao Paulo.

Besides Latin America, VPLS prices in Asian cities continue to be high. The median price of a FastE VPLS port in Mumbai was $22,111 per month in Q2 2012. Although Mumbai’s VPLS prices dropped 13 percent over 2011, they are still eight times what a customer would pay for the same service in London.

There does appear to be hope with VPLS pricing.

Brianna Boudreau, a TeleGeography analyst, said that as domestic U.S. and international service providers expanded their Ethernet footprints to meet the needs of multinational clients that were moving into areas such as Latin America and India, overall “VPLS service availability increased 9 percent worldwide between 2011 and 2012.”

Boudreau added that as more players “enter the market and the service continues to mature, prices will continue to decline, and regional disparities will narrow.”

VPLS is, of course, not the only service where geography and availability dictate prices. Similar trends have been seen in local access loops and international bandwidth.

Why consider VPLS for your WAN

1) More flexibility and manageability with with VPLS

When it comes to rapid change and advancement, companies which can respond quickly to market shifts will  benefit from VPLS, a Virtual Private LAN Service (VPLS) solution. VPLS uses MAC addresses with Layer 2 switching as opposed to Layer 3 MPLS solutions which use IP addresses and Layer 3 routing.

The main advantage of this is that with VPLS you are in control of your own IP routing. Therefore, your IT department can be much more agile in responding to varying levels of customer demand. VPLS networks allow you to conduct rapid reconfigurations yourselves without having to contact your service provider and wait for the provider to act upon the request. Even if you do require a service provider change, the typical time to make network changes to Layer 2 VPLS networks is only a fraction of that for Layer 3 MPLS networks because the network planning process is much simpler, which could be crucial for some businesses. Another feature which aids agility is the ease of adding new sites. With a VPLS-enabled network, a new site can be added by simply changing the network router that connects the site to the VPLS network. With Layer 3 MPLS solutions, however, it is a much more complex process as all of the service provider’s routers need to be changed which typically takes 10 times as long.

2) More efficiency with VPLS

Companies with a VPLS-enabled wide area networks will be more smooth-running and thus should be able to provide a better level of service to their customers. This is down to the fact that with VPLS the company has access to its own network information so faults in a VPLS network can be isolated much faster and the IT department can trouble-shoot to fix an urgent crisis rather than having to go through a number of support engineers to get the information required from a carrier. Less network down-time means higher corporate efficiency and productivity. Another aspect of our VPLS solutions is that they offer 5 levels of Quality of Service (QoS) and allow you to define your own priority levels either through labeling your traffic or using the service aware QoS feature on the core network. This is how VPLS maximizes efficient network usage according to your business needs, so you can rest assured that mission-critical data such as CRM, ERP and SCM are allocated enough bandwidth, alongside key services such as video conferencing and telephony, even during peak usage and without costly over provisioning of network capacity.

3) Lower costs with VPLS

Companies that use VPLS solutions will find they have lower costs for a number of reasons. Firstly, VPLS enables convergence of services such as VoIP, video etc. so that all traffic can be delivered over a single Ethernet interface, eliminating multiple leased lines and resulting in economies of scale. Secondly, working with VPLS uses the same skills sets that LAN specialists have, so you would not need to provide additional training on WAN skills or hire WAN specialists. In addition, VPLS requires a lower cost CPE as it requires smaller and fewer routers than MPLS solutions.

4) Lower latencies with VPLS

As a switched, Layer 2 solution VPLS is zero-hop in the core of the network, so extremely low round-trip latencies and jitter can be achieved. For example sub 1millisecond within a metropolitan area and 67 milliseconds round-trip from London to New York. This improves the productivity of the workforce as information is available faster. It also saves retail customers using Point-of-Sale systems time dialling up to make credit/debit card payments, improving their customers’ sales experience.

Thanks to Exponential-e

 

 

 

Ethernet E-LAN, WAN and VPLS Adoption Trends

Global Ethernet VPLS An enterprise end-user survey indicates that enterprises have a good understanding of different types of Ethernet service configurations available in the market. Key findings from the survey include:

  •  While metro E-Line and E-LAN services continue to see widespread adoption, long haul E-Line circuits are seeing a faster rate of adoption in the market. This indicates a greater penetration of (and future demand for) intercity Ethernet services in the market today.
  • In terms of bandwidth adoption trends for E-Line services, small businesses indicated a preference for 2-10M and 11-100M, with medium and large businesses choosing 101M-1G speeds. However, the results for E-LAN services are different with all sizes of businesses showing a greater preference for 101M-1G—for both current and planned usage.
  • Ethernet access delivered over fiber is the most preferred type of Ethernet access; however, Ethernet over Copper (EoC) is not far behind, with 40 percent of current users indicating they use EoC access services.

Conclusions of this survey indicated the following:

  • Survey results indicate growing penetration of Carrier Ethernet services in enterprise WAN networks. More than 55 percent of the survey respondents indicated they currently use Ethernet services. More than 61 percent indicated they plan to use Ethernet services in the next 12-24 months.
  • The overall demand for Ethernet is expected to grow in the near future.
  • E-Line adoption trends, as indicated by this survey, confirm the growing demand for long haul E-Line circuits.
  • Leading Ethernet service providers have been making great progress in terms of their long haul Ethernet network expansion, and are well positioned to take advantage of the growth trends indicated by these survey results.
  • E-LAN adoption trends indicate that 23 percent of current Ethernet service users are using metro E-LAN service versus 15 percent using VPLS. However, future adoption trends indicate that 40 percent of respondents planning to use Ethernet will use metro E-LAN service versus just 17 percent choosing VPLS.
  • Enterprises prefer MPLS VPNs for connecting highly distributed locations.
  • However, for enterprises wanting to connect locations nationally or globally, MPLS VPN is a better option. MPLS VPNs offer better reach and the flexibility of using multiple access technologies (DSL, T1/T3, SONET, Waves, Wireless, and Satellite), which is critical for enterprises wanting to connect thousands of locations where Ethernet is not ubiquitous.
  • Ethernet VPLS, by its very nature, requires Ethernet access at all locations.
  • Ethernet access delivered over fiber networks is the most widely used service, currently. However, Ethernet overCopper is gaining significant traction. This should make Ethernet providers that have invested in expanding their EoC networks in the past few years happy as not only is the demand for the service increasing, but some respondents indicated they would choose a higher bandwidth profile (51-100M) on EoC. This is a deviation from the typical 10M or lower speeds adopted on EoC in the past.

 

IP Transit Prices Drop Further

Prices for wholesale IP transit service continue to decline throughout the world. According to new data from TeleGeography’s IP Transit Pricing Service, price declines in most locations accelerated between Q2 2011 and Q2 2012, compared with the longer-term trend.  These trends will reduce the cost of local loop or tail-circuits for wide area networks and internet access.

The median monthly lease price for a full GigE port in London dropped 57 percent between Q2 2011 and Q2 2012 to $3.13 per Mbps, compared with a 31 percent decline compounded annually from Q2 2007 to Q2 2012. In New York, the comparable price dropped 50 percent to $3.50 per Mbps over the past year, and 26 percent compounded annually over the five-year period. Pricing for short term promotions and high capacities has dropped below $1.00 per Mbps per month.

While prices have declined globally, significant geographic disparities persist. For example, the median price of a GigE port in Hong Kong has remained 2.7 to 5.1 times the price of a GigE port in London over the past five years. The price of a GigE port in São Paulo  has remained between 5.2 and 8.2 times the price of a comparable port in New York.

IP transit prices have reached extremely low levels in developed markets, but remain high in many developing markets and in countries that are remote from major IP transit hubs. Nevertheless, few places remain where transit prices exceed $100 per Mbps. As carriers expand into emerging markets and establish new price floors in developed markets, global IP transit prices will continue to fall.

The above information provided by TeleGeography, the leading source of wholesale bandwidth pricing from nearly fifty carriers in seventy cities around the world.

Tap the expertise of MPLS-Experts in procuring and implementing your global wide area network.  Whether you need design and engineering, or management of the RFQ and bidding process, we will save you time and money.

Bandwidth Demand for International Circuits Grows by 45%

Data from TeleGeography’s Global Bandwidth Research Service reveal that demand for international bandwidth grew 45 percent in 2011, and at a compounded rate of 57 percent annually between 2007 and 2011. Although growth has slowed since 2008, when network capacity increased nearly 70 percent, the pace remains brisk, with aggregate capacity requirements more than doubling every two years.

The rate of growth varies widely by region, and has been fastest on links to less-developed regions. Between 2007 and 2011, international bandwidth usage in the Middle East grew at a compounded rate of 98 percent annually, from 148 Gbps to 2.3 Tbps. Over the same time period, Africa’s international bandwidth usage increased 85 percent annually, to 677 Gbps, and Latin America’s international bandwidth usage grew 71 percent, to 5.6 Tbps.

International Bandwidth Growth Used by Region, 2007-2011

International bandwidth requirements in Asia and Europe grew at a compounded rate of more than 55 percent between 2007 and 2011, while international bandwidth demand in North America and Oceania grew 47 percent. Although international bandwidth usage growth is slower in these mature markets, their capacity requirements are far larger than those of emerging markets. North America’s international bandwidth usage is nearly 10 times greater than that of the entire Middle East, while used capacity connected to Norway is greater than that connected to all of the countries in Africa.

Broadband subscriber growth is the primary driver of bandwidth demand in the Middle East and Africa, where the number of subscribers grew from 9.4 million to 19.4 million between 2007 and 2011, and in Asia, where broadband subscriptions doubled to 250 million over the same period. While broadband subscriber growth has slowed in Latin America, Europe, and North America, bandwidth demand in these regions has been fueled by increases in average broadband access speeds, enabling more frequent use of high-bandwidth applications such as video.

Tata Communications has launched a low latency Global Ethernet network

Submarine cable landing in Asia

Today, Tata Communications has launched a low latency network to connect financial trading capitals in Asia, the US and the UK.

The company claims the development is the industry’s first global low latency network, offering a multipoint Ethernet platform for the financial services sector.

John Hoffman, head of Ethernet product management at the company said it is likely other global companies will make similar developments, offering multipoint solutions, if the network proves successful.

“You may now start to see more purpose-built networks created because it gives financial trading companies an option on which cable they would like to purchase capacity on,” he said. “We recognize that while we have the largest global cable network in the world, there is also a big requirement for the fastest network – and that is the basis behind such investment.”

Hoffman heralded the development as part of a new strategy from Tata, and claims it allows customers to work with a single global supplier instead of multiple country specific point-to-point network providers.

He said he believes this could serve as a better approach to low latency trading because investments often get dated with new and emerging technologies. “It is very difficult in this scenario to make large long-term investments into a single cable, because you are never sure when a faster cable is going to be built,” he said. “We didn’t want to make that single cable investment because that investment is ultimately wasted in the end – we are aiming to provide flexibility and resiliency.”

Tata confirmed to Capacity this was only phase one of its low latency project. The company aims to bring low latency connectivity in its home market India and in South Africa, where it owns a majority share in operator Neotel. This is part of a long-running strategy to tap in to other emerging markets. “We look to get into locations where there is less competition and we have a better chance of winning business,” added Hoffman.

Readers should note that this technology is available to all companies interested in low latency communication.  The key is the utilization of the most direct cable path across the globe.

Gig-E Prices Continue to Drop as Supply Rises

As enterprises around the world replace traditional SDH/SONET private line networks with newer generation point-to-point EoMPLS and port based VPLS services, carriers have rapidly expanded their Ethernet network footprints to keep up with the demand. As a consequence of growing EoMPLS service availability, competition is increasing and prices are declining around the world. However, data from TeleGeography’s Ethernet Pricing Service show that great price disparities persist, both among regions and service providers.

Ethernet services are most readily available, and prices are lowest, in Europe and North America. In Q1 2012, the median monthly price of a GigE EoMPLS pseudowire circuit between Frankfurt and London was $3,448, down 29 percent from Q1 2011, while the median price from New York to Los Angeles was $8,000 per month, 12 percent lower than in the previous year. By contrast, Ethernet circuits connecting to Asian cities remain far more expensive due to more limited service availability, less robust competition, and higher underlying transport costs. The median price of a GigE EoMPLS connection from Hong Kong to London in Q1 2012 was $40,138, down 17 percent from a year earlier, while a circuit from Hong Kong to Tokyo cost $31,096 per month, more than five times the price of a comparable link on the longer London-New York route.

Price differences among service providers can be just as dramatic as price differences among regional markets, with the variance between the high and low prices on some lower-cost routes just as great, or greater, than on more expensive routes. For example, in Q1 2012, the high price of a GigE circuit from Hong Kong to London was $52,500—approximately three times greater than the low price of $16,704. The range of prices on the much less costly London-New York route is far wider: the highest price quoted to TeleGeography, $17,250 per month, was 10 times more than the low price of $1,729.

“Prices of enterprise WAN services often vary far more widely than prices of wholesale carrier services,” said TeleGeography analyst Brianna Boudreau. “While prices will continue to decline as wide area Ethernet services become more standardized and availability grows, large price disparities are likely to persist for years to come.”

TeleGeography’s Ethernet Pricing Service benchmarks the price of international Ethernet service by provider, capacity, service, and route.

 

Perseus Telecom and Reliance Globalcom Launch World’s Fastest Trans-Atlantic Trading Network

NEW YORK, April 23, 2012 /PRNewswire/ – Perseus Telecom, a global provider of connectivity, along with Reliance Globalcom, the Global telecommunications services arm of India’s largest integrated telecom operator Reliance Communications, today announced the launch of the world’s fastest available trans-Atlantic network connection, QuanTA, between major global financial exchanges. The launch of QuanTA represents a landmark development in the latency race to zero and creates new opportunities for trading firms on both sides of the Atlantic. (obviously latency will never be zero or close to zero.  But when it comes to trading, milliseconds matter)

The partnership between Perseus Telecom and Reliance Globalcom has resulted in the creation of an innovative high-speed, ultra low-latency network connection across the Atlantic by leveraging an existing system on the FLAG Atlantic-1 (FA-1) North cable, a trans-Atlantic six-fiber pair system between Long Island, New York and Lands End, United Kingdom. Designed with the latest advancements in optical technology, better dispersion compensation methodology, faster processing equipment and shorter cable paths, this ultra low-latency link represents a capital assured, cost-efficient solution for the sub-60ms RTD latency financial market participants require across the Atlantic. 

“Fast-paced trading environments demand even faster connectivity, particularly across the Atlantic where, traditionally, the patchwork grid of cable systems across the ocean had not allowed for a truly low-latency network,” comments Dr. Jock Percy, CEO of Perseus Telecom. “Our ability to create this network in partnership with Reliance Globalcom at relatively low cost and minimal time to market is a testament to our expertise in developing network solutions for the financial industry. We’re incredibly excited at the launch of QuanTA which we believe marks a game-changing development for the global trading community. As high-frequency trading strategies proliferate, ultra low-latency network connectivity becomes even more of a competitive differentiator.”

Commenting on this offering, Mr. Rory Cole, President and COO – Carrier Business, Reliance Globalcom, said, “Being one of the world’s leading carriers with connectivity across the globe, we are happy to announce that Reliance Globalcom now offers the fastest ultra low-latency route on FA-1. With this service, we are now connecting the U.S. and U.K. on the fastest link, addressing the business needs of our customers, especially in the financial sector, to help them gain a significant market advantage.”

The launch of QuanTA provides trading firms with a more secure, low-latency and efficient network connection across the Atlantic. Now, firms tapping into liquidity at exchanges in the U.S. and Europe can do so with guaranteed minimal latency on a highly reliable connection, minimizing potential risk in network outages or latency.