Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
As an ISP, handling multiple uplinks and ensuring seamless connectivity is critical, and that’s where BGP (Border Gateway Protocol) comes into play. In my experience as a Mikrotik consultant managing ISP infrastructures, particularly using RouterOS v7.x, BGP plays a significant role in maintaining reliable, cost-effective, and scalable networks. Multihoming with both Transit and IXP connections is one of the key techniques that can truly optimize ISP operations, providing a balanced approach to redundancy, traffic optimization, and cost control.
In this post, I’ll share insights from my experience with Mikrotik’s RouterOS v7.x and explore how ISPs can leverage BGP multihoming to build a robust network by connecting to both IP Transit providers and Internet Exchange Points (IXPs). We’ll also cover the advancements in RouterOS v7.x, how to configure BGP for multiple uplinks, and solve real-world challenges that ISPs face with this setup.
What is Multihoming?
Multihoming refers to connecting your network to multiple upstream providers. In the ISP space, this usually means linking to IP Transit providers like Airtel or Tata Communications, while also connecting to IXPs like Extreme IX or DE-CIX to peer with other networks. This setup ensures that you can route traffic through multiple paths, enabling better resilience, cost optimization, and latency reduction.
Why Multihoming is Essential for ISPs:
- Resilience: If one of your uplinks goes down, traffic can automatically fail over to another, maintaining service continuity.
- Latency Optimization: By connecting to IXPs, you can route local traffic more efficiently, reducing latency for customers.
- Cost Optimization: ISPs can route international traffic through high-cost Transit links while sending local traffic over peering connections at IXPs, significantly cutting costs.
Example:
In one of the ISP networks I manage, connecting to Extreme IX drastically reduced our local traffic latency while cutting down transit costs by offloading heavy traffic (such as YouTube and Netflix) through the IX instead of expensive transit bandwidth.
BGP in Mikrotik RouterOS v7.x: Advancements and Capabilities
RouterOS v7.x brought a lot of new features and improvements for BGP, which are crucial for handling ISP-scale multihoming. Some of the major advancements include:
- Improved BGP Performance: v7.x can handle full BGP routing tables more efficiently, which is important when managing multiple uplinks.
- Route Filtering and Communities: RouterOS v7.x improved the way you can filter inbound and outbound routes, ensuring optimal traffic handling.
- Enhanced Route Policies: The new version allows for more complex routing policies, enabling ISPs to implement advanced traffic management strategies.
These features make Mikrotik a viable option even for larger ISPs, a topic that I often find misunderstood in discussions about using Mikrotik for heavy routing tasks.
Configuring BGP for Multihoming with IP Transit and IXPs
Setting up BGP in Mikrotik RouterOS v7.x for multihoming requires configuring BGP sessions with both Transit providers and IXPs. Here’s a simplified walkthrough of the configuration steps:
- BGP with Transit Providers:
Establishing BGP with a Transit provider is relatively straightforward. You’ll want to configure BGP peering with the Transit provider’s router, then apply route filters to control which prefixes you accept and advertise. This ensures that your network only receives necessary routes and prevents overloading the routing table. - BGP Peering with IXP:
Peering at an IXP like Extreme IX or DE-CIX involves connecting to other networks that are part of the exchange. You can use BGP peering to exchange routes with these networks, reducing the load on your transit providers. In my experience, this reduces the cost significantly by routing local traffic directly through the IXP instead of going over expensive transit paths. - Best Path Selection:
By adjusting BGP attributes like Local Preference and AS Path Prepending, you can control how traffic flows between your different uplinks. For example, you can prioritize routes learned from the IXP for local traffic while preferring the Transit route for international destinations.
Traffic Engineering with BGP in RouterOS v7.x
One of the most powerful features of BGP is its ability to support Traffic Engineering, allowing you to control the flow of traffic based on various attributes. Here are a few strategies I use regularly in ISP networks:
- Local Preference: This attribute is crucial for prioritizing certain routes over others. For example, I set a higher local preference for routes learned from the IXP, so all local traffic is routed through the IXP while international traffic takes the transit route.
- BGP Communities and Route Filtering: Using BGP communities is an excellent way to tag and manage routes efficiently. This helps in controlling how traffic is routed through the network and beyond, ensuring you adhere to traffic policies and optimize costs.
Pro Tips:
- Pro Tip 1: Always monitor your AS Path and Local Preference values to avoid routing loops or unwanted traffic flows between Transit and IX peers.
- Pro Tip 2: Use BGP communities to simplify route management and apply policies effectively.
- Pro Tip 3: When connecting to an IXP, pay attention to the capacity of the exchange. You don’t want to overload your IXP port and cause performance issues.
Real-Life Use Cases of BGP Multihoming
From my experience managing networks for ISPs, here are some real-world use cases of how BGP multihoming helps solve day-to-day challenges:
- Reducing Transit Costs:
By routing domestic traffic through an IXP (like Extreme IX) and only using IP Transit for international routes, one of my ISP clients reduced their overall bandwidth costs by nearly 30%. - Improving Latency for End Customers:
One of my clients was experiencing high latency for local traffic because they were sending it via their Transit provider. By adding an IXP connection and using BGP to prefer routes through the IXP, we improved their local traffic latency by over 50%. - Failover for Redundancy:
In the event of a transit link failure, BGP automatically reroutes traffic through the available IXP or another transit provider. This keeps the network up and running without significant downtime—something critical for ISPs providing services to enterprise customers.
Comparing IP Transit vs. IXP Peering for Multihoming
Multihoming is about leveraging both IP Transit and IXP Peering to get the best of both worlds. Here’s a comparison based on my experience:
| Parameter | IP Transit | IXP Peering |
|---|---|---|
| Global Reach | Full access to all global destinations | Limited to networks peered at the IXP |
| Cost | Higher (charged per Mbps) | Typically lower (one-time connection fees) |
| Latency | Can be high for local traffic | Lower for local traffic |
| Reliability | High (based on SLA) | High, but depends on peering relationships |
| Bandwidth | Guaranteed, often symmetrical | Shared across peers at the IXP |
| Scalability | Scales with provider contracts | Limited by the IXP’s port capacity |
RouterOS v7.x: Current Limitations and Enhancements
Mikrotik’s RouterOS v7.x introduces several enhancements that improve BGP performance and make it more suitable for multihoming in ISP environments:
- Better BGP Handling: RouterOS v7.x can process large BGP routing tables with reduced CPU usage.
- Advanced Filtering and Communities: More granular control over traffic policies.
However, it's important to note that BGP Multipath is not currently supported in RouterOS v7.x. This means that while ISPs can benefit from improved performance and route management, load balancing across multiple paths for Outgoing Traffic remains a limitation.
Conclusion
In conclusion, BGP multihoming with both IP Transit and IXP peering offers ISPs the flexibility, redundancy, and cost savings needed to run a robust network. Mikrotik RouterOS v7.x makes this setup more accessible, providing powerful tools for traffic engineering and route management. As a Mikrotik consultant, I’ve seen firsthand how effectively multihoming can transform an ISP’s network—improving resilience, optimizing costs, and ensuring better performance for customers.
Whether you’re just getting started with multihoming or looking to optimize an existing setup, Mikrotik’s RouterOS v7.x is more than capable of handling the demands of modern ISP networks.
