RENATER Basic network design considerations Rome, June 22nd–25th 2009
Frederic LOUI Network Backbone Operation & Engineering
[email protected] 1
Housekeeping • We value your feedback - don't forget to complete your training session evaluations • Please switch off the bell of your mobile phones • All the slides will be made available • Don’t hesitate to ask questions ☺
2
Round table • Who are you ? • About me ☺ • Do you have any specific expectations regarding this training course ?
3
Training course objective • This training course aims: • To expose an approach proposal when designing a network • This approach is based on lessons learned when designing the French NREN network • Network design is a permanent activity that is not restricted to infrastructure/equipment selection
• MAIN GOAL is to provide you a network design overall approach based on our experience, so that YOU can start/adjust your own 4
Training course objective • This course is not: • A universal and extensive network design approach • Some statements/rules are valid now but might be obsolete in the future
• We value your experience ! So if you have any suggestion, share your knowledge
5
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 6
My customers ? • Identifying his customer • This is essential to know his customer. Depending on their field area their requirement might be totally different • Students in universities: • Just need basic services • Can represent a high volume of traffic
• Army: • Mission critical services with high availability • Might need real-time
• Specific projects: • Large Hadron Collider project • JIVE project: radio telescope project • GRID users
• Elaborate customer baseline and traffic pattern 7
My customers ? • Impact on the network design • • • • •
Bandwidth requirement Port density High availability and redundancy Real-time speed Security requirement
8
My customers ? • Bandwidth requirement • • • •
This criteria has a direct impact on link selection Force you to setup an end to end high capacity path Force you to build a parallel network Basically project related to massive data transfer can force you to upgrade your entire backbone and equipment • LHC project • JIVE projet • GRID computing
• Possible alternative for RENATER POP • • • •
Dark fibre 10GE Lambda 1 GE Lambda 2,5G OC48 SONET circuit 9
My customers ? • Port density • This criteria has a direct impact on L2/L3 equipment selection • Router port is far more expensive than switch port • For each pop, keep up to date and accurate the network inventory • Possible alternative for RENATER POP • • • •
PE Router + switch Collapse core design P,PE Switch/Router P router, PE router/switch 3 tier architecture: P router, PE router+switch 10
My customers ? • High availability & redundancy • This criteria has a direct impact on L2/L3 equipment AND links selection • Router port is far more expensive than switch port • For each pop, keep up to date and accurate the network inventory • Equipment impact on RENATER POP • Additional backbone interface • Additional card on different slot
• Link impact on RENATER POP • • • •
Duplicate POP meshing Make sure that each link has different path It might be better to have 10GE secured lambda Or have 2 lambda from different provider
• Deploy an additional TWIN POP ?
11
My customers ? • Real-time speed • This criteria has a direct impact on L2/L3 equipment selection • Equipment impact on RENATER POP • Edge traffic/packet shaper (if you are managing also the CPE) • Dedicated and specialized card on PE router • Expensive but can be cost effective for low bandwidth links
• QoS policy
• Qos in core ? • QoS at the edge ?
• Dedicated infrastructure • End to end Lightpath
12
My customers ? • Security requirement • This criteria has an impact on L2/L3 equipment selection • Firewall at the edge and transit point • Additional firewall card • Deep Packet Inspection at high rate
• Elaborate a strict security policy • Disable all features not used • Get inspiration from best current practices • IETF BCP38 BCP 39 13
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 14
Services • • • • •
Layer 1 services Layer 2 services Layer 3 services Other added value services And so on …
15
Services • Layer 1 services • Point to point lambdas • End to end lightpath • Layer 1 VPNs
16
Services • Layer 2 services • • • • •
Pure L2 VLAN services (dot1q, ISL etc.) VLL services VPLS services Full mesh layer 2 vpns Partial mesh layer 2 vpns
17
Services • Layer 3 services • Basic connectivity to the INTERNET • Basic L3 connectivity, L3VPN, MPLS VPN, IPSEC, DMVPN • IPv6 unicast • Multicast IPv4 / IPv6
18
Services • Other added value services • AAA services • Remote access services • Banking services
19
Services • And so on … • IP telephony • IP TV
• Keep in mind that each service must be managed and secured • Deploying a service without being able to supervise it, is a nonsense • This can influence some choices during the network design specification 20
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 21
Geographical topology • Take into account geographic criteria • Identify “key places” • Depending on the market, connectivity might be cheaper because of the competition in big cities • But … It can be also more expensive because of hosting and civil engineering cost • Don’t implement a POP in the mountains
• Take into account frontier connectivity • Connectivity with GEANT2 from PARIS • GEANT Back-up will been made available with DFN 22
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 23
Technology availability • Be aware of technology available • Fiber availability • For redundant path make sure to have 2 independent path
• PSTN/DSL availability • For out of band management console administration
• Submarine cable availability • If your network must span several territories • Overseas territory • Identify technology convergence point: • PARIS / LYON /MARSEILLE
• Satellite • Should be the last resort (induce DELAY, Jitter and is subject to interference) 24
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 25
Network topology • Physical topology • Full-mesh / Partial mesh topology
26
Network topology • Physical topology • Star topology
27
Network topology • Physical topology • Interconnected ring topology
28
Network topology • Network topology • Map as much as possible network topology to physical topology • Take into account physical topology when deploying network protocol (OSPF DR/BDR, IS-DIS, BGP route-reflector)
29
Network topology • Physical / Network topology mapping
30
Network topology
31
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 32
Routing protocol • Interior Gateway Protocol • Routing protocol bound to the internal network • This network is managed by a common administrative organization (sometimes referred as AS) • Question: which IGP to choose ? • Classful / Classless routing • Link state or distance vector ? • Hierarchical or not ?
•
Most answers rely on the previous topics
33
Routing protocol • Classful routing • Or has the default mask associated to the major network (the classful network) • Class A: /8 • Class B: /16 • Class C: /24
• Assume the subnet mask is the same attached to the interface • Only the prefix is advertized
• Old routing protocol • RIP version 1 • IGRP (Cisco proprietary) 34
Routing protocol • Classless routing • Include the subnet mask in their advertisement • “Variable Length Subnet Masks (VLSM) refers to the property of a network that allows different subnet masks to be mixed throughout the network” • “Classless Interdomain Routing (CIDR) is a property of a network that allows classful networks to be aggregated”
• Modern protocol • • • •
RIP version 2 EIGRP OSPFv2 Integrated IS-IS
• Watch out when troubleshooting !!! 35
Routing protocol • Distance vector • Broadcast copies of their routing table to their neighbors • Aka “Routing by rumor” • because neighbors talk to neighbors • And not the source of the route
• Less CPU and memory greedy than link state protocol • But in certain circumstances DVP can present loop
• Incomplete routing table transmitted • Some mechanism are used to prevent such situation but slows down the convergence time • Split-horizon • Poison-reverse
• Converge time
• Slow convergence (~can take minutes depending on your architecture the number of nodes)
• Example of distance vector protocol • RIPv1, v2 hop count • IGRP, EIGRP
36
Routing protocol • Link state • Advertise a list of their neighbors and the networks attached to their neighbors • Once all systems in the routing domain have a copy of all participants
• Each system build a tree with itself as the root tree • Run upon this tree the SPF (Aka. Dijkstra) algorithm to determine the best path to each destination
• More CPU and memory greedy that Distance vector routing protocol • Loop free protocol • Everyone has the same image of the network • Upon topology change a topology change notification is triggered and propagated
• Converge time
• Faster that distance vector protocol (~10s depending on your architecture) • Convergence time can be improved by fine tuning the protocol parameter
• Example of link state protocol • OSPF, ISIS • EIGRP
37
Routing protocol • Hierarchical or not ? • The problem
• High number of nodes in the same routing domain • Lots of route is injected per node • The routing protocol convergence depends on its algorithm complexity which is O(n) • Merge of different networks
• Routing instability
• A small instability can trigger a never ending SPF route recalculation • The network never reaches a stable state.
• Answers • • • •
Aggregate Aggregate ☺ Divide the routing domain in several area instead of one Perform route summarization on border routers
• Protocol available • OSPF • IS-IS
38
Routing protocol • Hierarchical or not ?
39
Routing protocol • Hierarchical or not ?
40
Routing protocol Routing protocol alternatives • RIP / RIPv2 / RIPng • IGRP, EIGRP • OSPF, OSPFv3 • Integrated ISIS 41
Routing protocol •
Observation • ISIS • • • • • •
Standard based (RFC 1142, RFC 1195) Historically, most widely used in service provider environment because of its scalability Clean and robust design Not that but extension as it is less popular than OSPF Start with level 1-2 routers High learning curve and more difficult to find IS-IS GURU
• OSPF
• Standard based (RFC 2328 for IPv4 and RFC 5340) • More popular and widely used in “small” service provider environment and/or enterprise environment • Flexible protocol featuring a lot of advanced functionalities and extensions • High learning curve and more difficult to maintain than EIGRP
• EIGRP
• Cisco proprietary • Most seen in enterprise environment and national service provider • Very fast convergence and simple to configure
• The final choice also depends on your staff knowledge and the service proposed 42
Routing protocol • Additional protocol extensions • Fast convergence • IP fast-reroute • MPLS fast-reroute
• LDP support for MPLS services • IPv6 support
43
Routing protocol • OK but what should I advertize within the IGP ? • Point to point backbone links • Specific point to point links • Network management system • Active measurement beacons
• Management loopback • Loopback dedicated to SNMP polling • Loopback dedicated to Syslog • BGP loopback
44
Routing protocol • Exterior Gateway Protocol • BGP (Border Gateway Protocol) • • • •
Designed to scale to the largest networks (~300k routes) Designed with stability in mind Interdomain routing BGP Interdomain routing between Autonomous System
• BGP in essence • • • •
Path vector based protocol Supports variable-length subnet mask (VLSM), Classless InterDomain Routing (CIDR), and summarization BGP connections between peers, using the destination TCP port 179 and keepalives • BGP path attributes allow granularity in path selection • BGP has its own table. BGP routes appearing in the routing table are selected from the BGP table
45
Routing protocol • BGP deployment constraint • Synchronization rule
• Do not use or internally advertise a route until the route is learned from a source other than BGP • It prevents traffic from being forwarded to unreachable destinations • It reduces unnecessary traffic • It ensures consistency within the autonomous system
• This rule is most of the time disabled but…
• Remember the problem that synchronization want to solve and why it exist ! • This rule can be disabled in the following situation: • All routers in the domain speaks BGP • All the routers in the AS are running BGP • All the BGP routers inside the AS are meshed 46
Routing protocol • BGP deployment constraint •
“When a BGP speaker receives an update from an iBGP speakers, the receiving BGP speaker will not redistribute that information to other iBGP speakers”
• iBGP peer within the domain must be fully-meshed • Or … Confederation (Not that much used) • Route reflection is the way to go
• Route-reflection
• Additional roles: Route-reflector client / non-client
• When a route is received by a RR, it will act as follow depending on the peer type: • 1- Route from a non-client peer: reflect to all the clients within the cluster.
• 2- Route from a client peer: reflect to all the non-client peers and also to the client peers. • 3- Route from an EBGP peer: send the update to all client and non-client peers.
47
Routing protocol • BGP common best practice: Next-hop self • Without Next-Hop self AS 1 193.1.1.0/24
R1
R2
eBGP 193.49.1.0/30
193.49.1.1
193.49.1.2
iBGP
R3
195.98.10.0/30
195.98.10.1
195.98.10.2
193.2.1.0/24
AS 2
• From R1 next-hop to get 193.2.1.0/24 is 193.49.1.2 • From R2 next-hop to get 193.1.1.0/24 is 193.49.1.1 • From R3 next-hop to get 193.1.1.0/24 is 193.49.1.1 48
Routing protocol • BGP common best practice: Next-hop self • With Next-Hop self on R2 AS 1 193.1.1.0/24
R1
R2
eBGP 193.49.1.0/30
193.49.1.1
193.49.1.2
iBGP
R3
195.98.10.0/30
195.98.10.1
195.98.10.2
193.2.1.0/24
AS 2
• From R1 next-hop to get 193.2.1.0/24 is 193.49.1.2 • From R2 next-hop to get 193.1.1.0/24 is 193.49.1.1 • From R3 next-hop to get 193.1.1.0/24 is 195.98.10.1 49
Routing protocol • BGP common best practice: Next-hop self
• Service provider example 50
Routing protocol • BGP common best practice: Next-hop self • Service provider environment example • 1) Redistribute connected interface or advertize CE-PE network • Not recommended as the IGP will flap each time a CE-PE link is brought down
• 2) Advertize PE-CE link into BGP • Valid option but, there will be an additional route in the BGP table for each CE-PE link. BGP table size will grow and recursive lookup might take longer time
• 3) Next-hop self toward the RR • This is the common best practice • The IGP encompasses the internal segment and the BGP loopback interface 51
Routing protocol • BGP common best practice
• Dual homing to 2 upstream ISP • Some organization need a constant connectivity to the INTERNET • Redundant connections is a way to re-inforce the availability fo the Internet service. • Redundant Internet connectivity is referred to as multihoming. The idea of multihoming is that failures at one link or ISP can cause a failover to a second link and ISP • Watch out ! When multi-homed to 2 transit service providers there is a potential risk that your AS can become a transit AS !
52
Routing protocol •
BGP common best practice • When both links are up, this configuration also increases capacity and improves performance. • BGP load sharing between multiple links to distinct autonomous systems is possible via policy configurations. • Outgoing traffic • •
Method 1) Tune your BGP attribute Method 2) Tune your IGP metric
• Incoming traffic • •
MED: valid but has a very low priority. This is not likely to work. AS prepend valid but pragmatically …
• Ask your Upstream SP if he has COMMUNITIES that can be used to influence the traffic returning to your AS. • •
This is the preferred way to engineer the traffic going back to your AS Could be a key criteria to select your transit provider
53
Routing protocol
54
Routing protocol • BGP common best practice • Identify all the route types
• Assign BGP community for well identified route/entity • • • •
Customer Primary/Secondary (or Backup)/Ternary/ etc… Identify route coming from GEANT2/EUMEDCONNECT2 Identify route coming from IX peering Identify your upstream transit provider
• Customer service community
• Dedicate community for your customer
• Give the customer the possibility to perform route selection • Special case when the customer is multi-homed and you’re not the preferred ISP
• Elaborate your BGP policy
• Prioritize BGP learned route origin • Allocate LOCAL-PREF according to your policy • Privilege some route originators among others
• Be consistent across various protocol (IPv4 and IPv6)
55
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 56
Additional network services design • MPLS based services • Additional services available compared to IPv4 service • Flexible added value services • Well-suited for ISP • Cost effective
• MPLS is necessary • Activate LDP on all backbone interface • MPBGP: VPNv4 address type • RT mpls label at the PE (edge) level 57
Additional network services design • MPLS based services
58
Additional network services design • Layer 3 VPN • Any-to-any VPN (Full mesh VPN)
59
Additional network services design • Layer 3 VPN • Any-to-one VPN (Hub and spoke VPN)
60
Additional network services design • Layer 3 VPN • Many-to-one VPN (Central service topology)
61
Additional network services design • Layer 3 VPN • Many-to-one Variant(Central service topology)
62
Additional network services design • MPLS Route Target policy • Use case 1#: Company re-organization • Network merge • Network split • Migration from frame-relay/ATM PVC based VPN
• Use case 2#: Network service deployment • Possibility to deploy a central network service quickly and easily (Hosting, AAA, IP telephony, Internet gateway etc.) • Extranet 63
Additional network services design • Layer 2 VLL(s) • Point to point ethernet link • Do not participate in customer STP site site
RENATER
site
64
Additional network services design • Layer 2 VLL(s) full-mesh • Full mesh point to point ethernet link site site
RENATER
site
65
Additional network services design • Layer 2 VLL(s) partial-mesh site spoke
site spoke
site spoke
site spoke
RENATER
RENATER
site hub
site hub 1
site hub 2
66
Additional network services design • Layer 2 VPLS site
site
RENATER
site
site
LAN 1 LAN 2
L2VPN p2p
site
site
NR standard NR VPLS
Routeur client
67
Additional network services design • Layer 2 VPLS distributed customer sites
68
Additional network services design • Classic VLAN service • IEEE 802.1d STP • IEEE 802.1w RSTP • IEEE 802.1s MST or MISTP
• Watch out ! • L2 Broadcast Storm control • As a SP, transport customer BPDU, do not take them into account in your own STP…
• Not easy to troubleshoot: • Ethernet OAM is the answer ? 69
Agenda • • • • • • • • •
Identify your customer Which services do you want to propose ? Geographic topology Technology availability Network topology Routing protocol: IGP and EGP Additional network services design Strict policy: KISS (+T) NMS & security Straw-man proposal 70
Strict policy: KISS (+T) NMS & security • KISS
• Keep • Avoid as much as possible “special cases” • Normalize configuration after each operation according to the template
• It
• All equipment should have the same global configuration • And should follow the same “rules”
• Stupid
• Get inspiration from the best practices • If 2 solutions is possible to get the same thing “stupid” one
choose the
• Simple
• Even if it is interesting avoid PBR, Traffic engineering • Avoid as much as possible NAT, Tunneling
• And …
71
Strict policy: KISS (+T) NMS & security • And … Tidy ! • Elaborate a strict engineering process deployment • Document is highly critical because it is the memory/history of the network • Write specification document • From that, an implementation document should follow
• The process outcome: configuration template • Each functionality should have a template • Versioning is not an option: Use RANCID
• Constantly perform configuration consistency check • Correlate equipment configuration against the template on a day to day basis • Use “Network Config Checker” 72
Strict policy: KISS (+T) NMS & security • Document, document document ! • Addressing scheme • KISS method • Aggregate, Aggregate, aggregate • Not that easy task …
• Naming rules • • • •
Equipment name Configuration element naming convention Not that easy DNS name
• Put everything in a versioned document !!! 73
Straw-man proposal • During the engineering process • Compile all the previous information in a document. • • • • •
Customer types inventory Network services proposed … Geographic topology Technology availability Etc.
• First iteration of this document is called “straw-man” • Very good way to start a document specification and kick-out the discussion • The document should evolve and pass through different milestones so as to reach its final stage. • This document is a base for all specification document 74
Straw-man proposal •
Taken from Wikipedia: • “A "straw-man proposal", also known as an Aunt Sally, is a brainstormed simple business proposal intended to generate discussion of its disadvantages and to provoke the generation of new and better proposals. Often, a straw man document will be prepared by one or two people prior to kicking off a larger project. In this way, the team can jump start their discussions with a document that is likely to contain many, but not all the key aspects to be discussed. As the document is revised, it may be given other edition names such as the more solid-sounding "stone-man", "iron-man", and so on, etc.[citation needed] • The succession of names comes from the requirements document for the programming language Ada. In this document, the various stages were Strawman, Woodman, Tinman and Ironman. Later, another Ada document added the following sequence of men: Sandman, Pebbleman and Stoneman.[citation needed] • In software development, a crude plan or document may serve as the strawman or starting point in the evolution of a project. The strawman is not expected to be the last word; it is refined until a final model or document is obtained that resolves all issues concerning the scope and nature of the project. In this context, a strawman can take the form of an outline [1], a set of charts, a presentation, or a paper.”
75
That’s all folks !
76