WAN using IP VPN over Internet versus MPLS – Pros and Cons

There’s a price for everything in this world, and  Internet based IP VPNs are no exception. While  IP VPNs are a cheaper alternative to any MPLS network, it doesn’t necessarily mean they’re for everyone, as customer requirements always vary. In this posting, I will explain both the Internet IP VPN advantages and disadvantages.

Let’s take a look at a few IP VPN advantages over most MPLS circuits:

  • Cheaper rates. Internet service providers provide a simple NxT1, Ethernet or Cable connection to the Internet, using the highest possible speed with. The price for internet connectivityis considerably cheaper than almost any WAN MPLS service, making it extremely attractive for companies seeking to cut telecom costs.
  • Fully configurable. WAN engineers have total control over the VPN tunnel created between sites. They are able to perform on-the-fly configuration changes to compensate for any network problems or help rectify any problem that might arise. With full access to the VPN, terminating equipment like routers and firewalls, engineers have the ability to see the condition of the internet circuit and take any action(s) deemed necessary…provided they have the staff resources and skills.
  • VPN backup included. For mission-critical sites, backup via another internet circuit is possible if your primary connection fails.  Time response for the backup line to come online is configurable by the network engineer, and there is no need to wait for the ISP to fix a line so your company can continue working.
  • Two-in-one. When configuring the site-to-site VPN, engineers can also configure remote VPN access for users traveling around the country or world, a feature most companies would have to pay additional money for to receive from their service providers.
  • Upgradable features. Perhaps one of the strongest advantages is the fact that your site-to-site VPN characteristics are strictly dependant on those that your VPN routers/firewall support. This means that as new features are introduced with the newer router operating systems (i.e., Cisco IOS), they will be available to your engineers to implement. For example, QoS pre-classification was a feature Cisco introduced in its IOS that fixed a number of QoS features for different services running over VPN tunnels. Dynamic Multiple VPN (DMVPN) was another great feature allowing scalable IPsec VPN tunnels between multiple sites. DMVPN allows every endpoint to dynamically build a VPN tunnel with any of its other peers, providing a low-cost mesh VPN solution.

If the brief list of the above  of Internet IP VPN advantages seems overwhelming , you have read a few of its disadvantages.

Here is a list of a few disadvantages of Internet IP VPNs over almost all WAN MPLS circuits:

  • Limited QoS. In order to have a fully functional QoS model, you need to have control of all equipment and paths that your VPN packets run through. In the Internet IP VPN model, QoS is effective in each site’s LAN, up until the L interface of the routers. From there on, packets enter the ISP’s network, and your ISP will clearly state that there is no QoS for such connections. Everything is based on a “best effort” delivery mechanism and you can’t argue about that. Any QoS parameters inserted in your WAN packets are, in most cases, ignored by the ISP.
  • No Class of Service Prioritization. It’s the internet, sorry.
  • Higher Packet Loss and Latency. If you use interactive applications, video, voice domestically or are connecting to locations more than 3,000 miles away, the MPLS network will outperform the IP VPN hands down.
  • Undependable voice and video. If you use voice or video over your network, the MPLS network will outperform the IP VPN, hands down with dependable and consistent performace.
  • Possible bottlenecks and low speeds. In an Internet IP VPN scenario, your company connects to the Internet, which has quite a variation of performance.  If there is heavy traffic on the Internet, chances are you might experience lower speeds during peak-hour times. Again, there is no guarantee of the performance.
  • VPN and router/firewall security. You are exposed directly to the Internet. This means that the security of your VPN and terminating equipment (routers and/or firewalls) are your responsibility. If your engineers do not take the necessary measures to secure the equipment correctly, this can lead to the exposure of your company to the Internet. This is not a topic to be taken lightly, as the damage can be devastating. It is extremely important to understand the risk involved and to have the required technical expertise to ensure the job is performed correctly. Under ideal circumstances, where the equipment is correctly configured, there is no need to worry—you’re safe.
  • Denial of service attacks. With a direct Internet connection, you are exposed to any denial of service (DoS) attack. All attempts can be successfully repelled; however, keep in mind that the traffic will have to reach your router/firewall first. This means that the heaviest damage that can be produced by a DoS attack—for a correctly configured endpoint—is to create a bottleneck on your connection and greatly reduce speeds for the duration of the attack.

If you want a rock-solid WAN with almost no packet loss and the lowest possible latency and quality, consider an MPLS network.

The importance of a Wayleave Agreement prior to installation

This posting is procedural, not technical.  It was motivated by an installation delay that was unexpected for a customer in London.

The local loop was to be installed by BT, in an office complex owned by a large national commercial property company.  For some reason, the HQ of the property owner would not agree to the Wayleave Agreement that BT required. This meant that BT would not install the local loop to the client office.  What is more surprising is that BT would install regular telephone lines to their office.  This made no sense to anyone.  But it caused an unexpected delay and needless aggravation.  So I want to make all my readers aware of this, so they can confirm with their landlords that a Wayleave Agreement is in place with the local phone company.  This is of particular importance in the UK and India.  In China, payment of bribes to the building management is not an unusual way to manage this situation.

For clarity, a Wayleave Agreement is an agreement under which a property owner gives a service provider (for example, an electricity, telephone or cable TV services provider) a right to install pipe or cable passing through or over the owner’s property.

Check with your landlord that such an agreement is in place to avoid any delays in your network installation.

MPLS, AToM and VPLS

There is often a great deal of confusion in understanding the different “flavours” of MPLS networks.  I recently read a post online by a gentleman named Mbong Ekwoge who wrote a rather clear online posting:

MPLS is the enabler of all these fancy services and applications we hear about today, such as MPLS VPNs, AToM (Any Transport over MPLS), MPLS TE (Traffic Engineering), etc.

In order to clearly understand what VPLS is, you need to understand what led to the “birth” of VPLS (Virtual Private LAN Service).  It all began with MPLS VPNs. The client had to form a peer-to-peer relationship with the Provider’s PE routers. What this means is that the provider is intricately involved with routing and forwarding the customer’s traffic and some customers did not like this idea. Also, providers had invested heavily into Layer 2 VPN techniques such as ATM, Frame Relay, etc and completely eliminating these overlay VPN techniques didn’t feel right with their financial people. Some engineers did not like the idea of having to let go of their beloved ATMs, Frame Relay PVCs for some new chap coming in.

This led Cisco and IETF to develop a solution which would let you run MPLS in the core but users will still maintain their private Layer-2 VPN service across the MPLS core of the service provider. What this means is, the provider will provide a VPN service, across MPLS, but it will be kind of a pseudowire experience. The customer still retains their highly valued privacy, the Service Provider maintains her MPLS core and should the customer be convinced, transitioning to MPLS VPNs will be like “bread and butter”.

Now this led to the introduction of AToM. AToM is the Cisco name for the Layer 2 transport service over an MPLS backbone. The customer routers interconnect with the service provider routers at Layer 2 (Ethernet, High-Level Data Link Control [HDLC], PPP, ATM, or Frame Relay). This eliminates the need for the legacy network from the service provider carrying these kinds of traffic and integrates this service into the MPLS network that already transports the MPLS VPN traffic.

AToM is an open standards-based architecture that uses the label switching architecture of MPLS and can be integrated into any network that is running MPLS. The advantage to the customer is that they do not need to change anything. Their routers that are connecting to the service provider routers can still use the same Layer 2 encapsulation type as before and do not need to run an IP routing protocol to the provider edge routers as in the MPLS VPN solution. As such, the move from the legacy network that is running ATM or Frame Relay to the network that is running AToM is completely transparent to the customer. The service provider does not need to change anything on the provider (P) routers in the core of the MPLS network. The intelligence to support AToM sits entirely on the PE routers. As such, the core and edge technologies (MPLS and AToM, respectively) are decoupled. The core label switching routers (LSRs) only switch labeled packets, whereas the edge LSRs impose and dispose of labels on the Layer 2 frames. This is similar to the MPLS VPN solution, in which the P routers switch only labeled packets and the PE routers need the intelligence to impose and dispose of labels on the IP VPN traffic from the customers.

Now how does VPLS come into the equation????

AToM is a point-to-point service and hence cannot broadcast frames.

Now some technologies such as Ethernet are broadcast in nature and take for example, the Spanning Tree Protocol (STP). These protocols operate in a broadcast nature. VPLS is the point-to-multipoint cousin of AToM.

VPLS Networks for Disaster Recovery & Business Continuity

By Douglas Lantigua, Principal at MUSA Technology Partners

Effectively solving the problem of corporate Disaster Recovery and Business Continuity (DR/BC) starts with proper planning and networking. A company that owns only a few servers or a complete datacenter will need a failover location and a plan known as the Run Book. The other location can be a collocation facility, another business location, other service providers or some sort of hybrid. The business needs to address key questions such as:

  • How much data can we afford to lose in the event of a system failure? The amount of possible data loss, measured by either data or time, will help direct the DR/BC solution.
  • The Recovery Point Objective (RPO). The RPO is the acceptable level of data loss measured in time (i.e. 5 minutes or 4 hours). The RPO is married to the Recovery Time Objective (RTO) which is the amount of time it takes to get critical systems back into a functional state.

How does networking assist in the RTO and RPO objectives? Larger companies can take advantage of Virtual Private LAN Service (VPLS) to extend the datacenter network to another physical location. Unlike its cousin Multiprotocol Label Switching (MPLS), VPLS acts on a lower level of network activity. With VPLS, users and computers connect to systems by name (a friendly translation of an IP address). MPLS works at the IP level. Geographically dispersed locations connected by MPLS need to have different networks (or IP address ranges). With MPLS, even if you fail a server over to another location using the latest application technology, you will still need to change the IP address of the server, and you will probably need to change dozens of other attributes in the network modified to bring the system back into an operational state. The failover procedure and plan are compiled and updated in the Run Book.

VPLS’s key advantage is that it works at a lower network level than the IP address; it works at the machine address (MAC address). This makes the IP address transferrable anywhere in the network. So a failover of a system can move geographical locations and still maintain its’ IP address and remain reachable by users and computers alike. VPLS can be expensive, but there are alternatives for companies on a budget and those who do not need the large bandwidth requirements most VPLS providers mandate. Companies with sub-VPLS requirements can use IP tunneling and/or channeling to achieve the same goals. By extending the network across a geographically dispersed location(s) at the machine address level (layer 2), you allow the IP addresses of the servers to move freely. The latest in virtualization technology and storage replication makes an aggressive RTO and RPO very inexpensive.

The Run Book is the instruction and procedure plan on how to handle DR/BC scenarios. Given in previous scenarios of failovers where the IP needs to change of the system, the dependency on the IP address can be far reaching. Not only would the server need to change its IP address, but the name to IP (DNS) relationship, connections to data sources, internal application settings and finally, the end user network path to the server/service– which could include dozens of pieces of network gear, will all need to be updated. These types of systems are set up over days/weeks or months when originally deployed, an emergency change under tight deadline for a single system could be difficult even under perfect preparation. Then assume people get busy and the Run Book doesn’t get updated when changes occur. The Run Book then becomes a massive paperweight and budget nightmare to maintain effectively.

Leveraging a geographically dispersed layer 2 network either by VPLS or IP tunneling/channeling shrinks the DR/BC run book, allows the staff to fix the original problem and frees engineers to solving unforeseen issues. Any failover involving IP address changes is fraught with time consuming issues in order to bring missing critical systems back online. Those industries with heavy compliance requirements are in need of simple solutions to meet regulation standards. The networking base does include an upfront investment for setup and enough bandwidth for failover. Managers must maintain routine checks that enough bandwidth is available for a catastrophic failover event of critical systems. Secondary access points should be considered to the failover location if key users will need to perform their job function from outside your network for a prolonged period of time. Routine testing of failovers should be part of the standard operating procedure (SOP) of the IT/IS department. The network is only one part of the overall picture. With a flexible, geographically dispersed network the ground is fertile for system and application failover tools to work their magic with the least complications to achieve success.

SIP Trunking with your MPLS network

A google search (June 2009)  on a ‘SIP Trunking’ returns ~500k references. 

From Wikipedia (replace ‘connection’ with Trunk)
 A SIP (Session Initiation Protocol) connection is a service offered by many ITSP (Internet Telephony Service Providers) that connects a company’s PBX to the existing telephone system infrastructure (PSTN) via Internet using the SIP VoIP standard. 

This probably is the most common understanding of SIP Trunking – that being, a SIP-based interconnection from a SIP-based IP-PBX to a SIP Service Provider  which facilitate communications to and from the PSTN.  Within this context, there’s ample evidence to the value of SIP Trunking.  The first, and probably most obvious advantage is a reduction of costs by leveraging IP as the mode of communications between the Enterprise and the Service Provider. Historically, businesses had two distinct and separate network infrastructures – one for data, and another for voice.  Maintaining one is easier (but a little bit more complicated at times) than two.  If done correctly, it’s definitely cheaper.  This point pertains to the LAN as well as the WAN.  

Now, if the PSTN were the sole interconnection of interest for SIP Trunking, owning and managing SIP Trunking from the Service Provider POV would be easier. (not easy, but easier) BUT, here things are changing – as more and more businesses (Enterprises and/or Carriers) Transition to SIP, the burden of ensuring that all the SIP Trunks interact successfully with one another falls on the Service Provider. All SIP networks are not built the same.  There are numerous differences between Enterprises and Services Providers alike.  

Possibly this analogy will help with the understanding.  Say each SIP-based network, represents a different language, French, Chinese, Spanish, etc.  In order for everyone to communicate with one another, everyone would need to be able to speak and understand everyone else’s native language.  There must be a common denominator.  Without a common language, then translation services are necessary to contract out to facilitate the communications.  The PSTN has historically served as the “common denominator” in the world of IP Telephony integration.  The problem stems from the cost of using the PSTN in this manner, both tangible and intangible.  First the tangible costs.  It’s more expensive to use the PSTN for this reason.  Secondly, things get lost in translation.  If the PSTN is used as the common denominator, then features other than standard voice (video, HD voice) are not possible, and there’s oftentimes service degradation from multiple encoding/decoding of the voice.

The SIP Trunking Service Providers (at least the good ones) ensure that the necessary translation services required to enable communications do NOT limit the capabilities of SIP, and still enable the end-users to realize the savings of SIP Trunking.  This is the true value of the SIP Trunking Service Provider.  The Enterprise only needs to ensure that the communications interconnection (SIP Trunk) between them and the SIP Trunking Service Provider are compatible, and the rest can be assumed.  This is something that is too often overlooked in the world of SIP Trunking.

Another little know fact is that with international calls, many of the carriers are already using SIP, even if the caller is not.  If you start with SIP from your network , you eliminate one of the encoding/decoding steps and end up with better voice quality.

Remember, I am talking about SIP trunks using an MPLS network…NOT the internet.  This eliminates the lack of quality control that come to mind for most people when they think of SIP (think Vonage).

Have you ever thought about wide area networks for Subway systems?

This post does not really relate to most of the others on this blog.  But I just finished a meeting with some network architects for a subway (Metro) system located outside of the United States.  What I learned was remarkable.  I have changed some of the numbers to protect the privacy and security of the city.  Look at these numbers:

  • 45 Metro stations
  • 160 CCTV color cameras in every station
  • Each camera uses 2Mbps of bandwidth or 320Mbps per station
  • Archiving of all CCTV cameras for thirty days
  • Wireless access for passenger use in every train car and the stations
  • Electronic ticketing and transactions
  • Train management via network

When you own the tunnels, you have your own fiber for a network of this nature.  None-the-less, the bandwidth being managed on this single network is remarkable for anyone who appreciates networking!

Global Wide Area Networks – Single or Multi-Carrier Solutions

Everyone can appreciate the convenience of a single carrier solution when managing a global WAN, which is why many multinational companies do so.  But most MNCs use multiple carriers to diversify their risk and control costs.  Using a single carrier for a global network can cost up to 25% more than a multiple carrier solution.

If your company has  a global network, unless you require the fully meshed functionality of a single carrier MPLS network, you should consider a review of your options.  MPLS-Experts can perform that review for you at a very nominal fee.  Then when your contracts approach the termination date, you can have all the information you need to make a knowledgeable decision.

What are the biggest challenges installing an MPLS network?

While the implementation of an MPLS can be complex, it really doesn’t have to be.  If you keep the following factors in mind, you will have smooth sailing:

1. The first thing to realize is that you will need to extend the demark from your building telephone closet/room to your computer room.  Some carriers will extend this for you, but realize this:  even though they tell you so, it may not happen.  This is simply the reality.  So have a cabling contractor lined up to do this work.  To avoid the frustration, plan on extending the line instead of having your carrier do this.  The only exception is when your MPLS carrier is also the local phone company, such as Verizon in their territory, AT&T in their region and Qwest in their states of coverage.  But remember, these three carriers are not the “local” phone company throughout the country.  Plan accordingly.

2. Put thought into your network diagram by discussing it with your carrier sales engineer.

3. DO NOT change your network configuration from what you submit to the carrier until everything is installed and working properly.  What you might view as a simple change might not be communicated to all the parties on the implementation team.  This can result in a new network that doesn’t work at your turn-on.

4. Know what you don’t know! The carrier’s job is to provide the network for connectivity between your routers.  If you are not 100% confident of your ability to configure the routers to work with your new network, arrange for third party help in advance.  You have no idea how many (mostly smaller) companies learn at the turn-on that they don’t know how to make the new network work with their internal network.  MPLS-Experts has engineers that can perform this service for you.  But it can be very embarassing when your management expects your network to work by a particular date and it doesn’t.

5. When your network is live, test it’s capacity.  If you are paying for a 3xT1 with 4.5Mbps of bandwidth, ftp files accross the network to determine that you really are getting the bandwidth that you are paying for.  We have seen networks work perfectly for months, until the customer decided to ftp some files for the first time, when they learned that the thru-put for ftp was 33% of what they were paying for.  You want to discover this BEFORE you sign off on the acceptance of the network.

These are some key factors to keep in mind.  I’ll make another post specific to companies that will be using VoIP on their wide area network, as well as the class of service configurations to confirm.

What will the latency of my network be?

One very interesting part of being an independent MPLS consultant is having the opportunity to view the ping time of identical networks at the time of migration from one carrier to another.  It is the only time to obtain empirical measures of latency between different locations on a network.  This data is invaluable to our work.

When you investigate different carriers, they will be able to provide you with their service level agreements for different paths on a network, but this number is invariably a worst case scenario.  If a carrier has Looking Glass, they can provide latency numbers for  a limited number of POP to POP paths, but not your distinct network.  That is why MPLS-Experts records this information for the benefit of our clients.

WAN Accelerators and MPLS – Important Facts

WAN Accelerators are wonderful tools in improving your network performance, provided your traffic can benefit from this technology.

If you obtain an MPLS network, your network performance will be better than a VPN over the internet.  But you need to select your Classes of Service appropriately.  Different CoS levels have different packet loss SLAs.  On a simple level, the SLAs might be:

  • Basic CoS: 99.9% packet delivery
  • Middle CoS: 99.99% packet delivery
  • Best CoS: 99.999% packet delivery

If you decide to subscribe to all Basic CoS, the SLA is 99.9% packet delivery.  That is typically the same as an uncongested internet access circuit, so you might not see any performance improvement.  But if you use your WAN Accelerator with the Middle CoS with 99.99% packet delivery, you will experience a more noticeable improvement.  Obviously, the Basic CoS will work better than the internet when the internet is congested, since the MPLS network avoids those bottlenecks.

When using a WAN Accelerator, since you are using compression, if your compression ratio is 20:1, if you lose 1 packet, you are really losing 20 or more packets.  So you maximize performance with a network that has less packet loss/better packet delivery.

To reduce or eliminate the number of undelivered packets, select a higher CoS.

One thing you should be aware of, that is not widely publicized is that the lower level Class of Service levels will not provide the expected performance improvements when you use a WAN Accelerator.  But if you design your network accordingly, you will be very pleased with the performance boost.