vQFX10000 で VXLAN+EVPN (L2 over L3 編) (original : 2017/01/02)
この記事は某所で 2017/01/02
に書いた記事のコピーです。
そのため 2017/05/11
時点ではやや古い情報も含まれています。(以下一例)
- Juniper さんの vQFX ダウンロード規約が変わっていそう (詳細は未確認)
- 201705 に
GNS3 ver2.0 stable
が出たので KVM 間のパケットキャプチャ用に GNS3 のハブを挟む必要がなくなった
.
- 最初に
- 構築~動作確認
- おしまい
最初に
前回(vQFX10000 を KVM+GNS3 で動かす)、Juniper vQFX10000(以降 vQFX) の DL 権限を個人で得て GNS3 で軽く動作確認がとれました。 今回は「仮想版って L2全般/L2VPN 系機能が動かなかったりするけど、vQFX はどうなんだ?...お、EVPN もちゃんと動くやんけ!」ってところまでを見ていきます。
本項でやること
以下をやります。
- vQFX で VXLAN の Control Plane として EVPN を動かす
- 仮想版だと L2VPN 系機能が動かないアプライアンスもあるので
- EVPN の適用範囲は L2 シングル構成のみで、DataPlane としては VXLAN を使う
- L2VPN への L3 統合周り(inter-subnet-forwarding 周り)やマルチホーミングなどの機能は、今回の範囲外
- Dataplane としては VXLAN を使い、MPLS は取り扱わない
- Juniper公式 | ジュニパーネットワークス EVPN 次世代データセンター アーキテクチャの実装 で MX と QFX5100 でやっていることの一部を vQFX10000 で試す
- 各要素技術の詳細な解説は行わない
個人的には、2014/12 時点で VXLAN のマルチキャスト実装を確認した時 (VyOS と Arista で VXLAN 相互接続)に微妙だと思った点にも着目しておきたいです。今回、データプレーン周りはほぼ同じ構成を組んでますので。
- 上の記事の〆が「マルチキャストルーティングを勉強しよう。」だったのですが、この2年間以下のようにウダウダやっていて、特に進捗はありませんでした。
- 上の記事では VTEP 間の到達性をもたせるのに「VyOS が認識しないといけない Arista 側の VTEP IP アドレスは、Arista の Loopback アドレスになるので、StaticRoute を追加しておきます。#これだと拡張が面倒なので、実環境では DefaultRoute か動的ルーティングを使うことが多い気がします。」とか書いてますが、今回は実環境をある程度想定した例示ができそうです。
概要構成図 / 環境
以下のような構成を組んでいきます。node11 と node21 が L2overL3 で通信するやつです。
環境は、いずれも前回の通りです。
参考資料
前述の通り、各要素技術の詳細な解説は放棄していますが...こんなニッチな記事を読む人向けなので、まあね?
- EVPN 標準
- RFC7432 (BGP MPLS-Based Ethernet VPN)
- Alcatel-Lucent 発表資料 | ETHERNET VPN Standardization and Status
- 2015/11 時点のもの
- スライド9あたりが標準化動向の最新に近いかな...
- VXLAN 標準
- JUNOS での VXLAN+EVPN 実装・設定周り
- Juniper公式 | ジュニパーネットワークス EVPN 次世代データセンター アーキテクチャの実装
- MX と QFX5100 で EVPN の設定サンプルを step by step で解説
- 日本語版
- Juniper公式 | Junos OS for the QFX Series, Release 15.1X53 (for QFX10000 Switches) / EVPN Control Plane and VXLAN Data Plane Feature Guide for QFX Series Switches
- vQFX を動かす
15.1X53
での EVPN + VxLAN 全般
- vQFX を動かす
- Juniper公式 github | JNPRAutomate/ansible-junos-evpn-vxlan
- MX, QFX5100, QFX10000 の EVPN/VXLAN サンプルコンフィグを含む
- Juniper公式 | ジュニパーネットワークス EVPN 次世代データセンター アーキテクチャの実装
- O'Reilly Juniper QFX10000 Series
- Juniper QFX10000 の独自プロセッサ開発経緯や特徴・Architecture(HW/SW) の結構詳しい説明・MPLS/VXLAN+EVPNの設計ポイント解説など、非常に良い本
Juniper QFX10000 Series: A Comprehensive Guide to Building Next-Generation Data Centers
- 作者: Douglas Richard, Jr. Hanks
- 出版社/メーカー: O'Reilly Media
- 発売日: 2016/08/11
- メディア: ペーパーバック
- この商品を含むブログを見る
- Juniper QFX10000 の独自プロセッサ開発経緯や特徴・Architecture(HW/SW) の結構詳しい説明・MPLS/VXLAN+EVPNの設計ポイント解説など、非常に良い本
構築~動作確認
GNS3 で仮想マシンのデプロイ ~ 結線 ~ 起動
前回の通りに、3ペアの vQFX をデプロイして接続していきます。 以下の感じで。
torSW101a
と torSW201a
というのは、GNS3
の Ethernet Switch
を使っていますが、各環境に合わせて適当な dot1Q 食えるスイッチ置けば良いです。
設定はそれぞれ以下の感じです。(今回は Port 3-4
は使いませんが)
node11
と node21
は、インターフェースに IP アドレス付与するだけなので、適当な疎通確認用ノードなので好きなのをどうぞ。(node12
と node22
は今回使わないです)
あと captureSW
というのは GNS3 1.5.2
では github gns3-gui issues | QEMU link Packet Captures の通り、qemu 同士の結線をパケットキャプチャできないので、「bb01
でポートミラーリング」する手法をとっているために置いているものです。
未試行ですが、RE 同士の結線をしている箇所(bb01
と spine[12]1
)に全て Ethernet Switch
を挟めば、ポートミラーリング不要でパケットキャプチャできると思います。(そちらの方がやりやすい筈)
できたら起動して待ちます。
基本設定
各環境に合わせて syslog なり NTP なり ssh key 登録なりしておいて下さい。
あ、今回は BGP のログを /var/log/bgp.log
に残すために、以下をやっておくと良いですよ。
set protocols bgp traceoptions file bgp.log set protocols bgp traceoptions file size 10k set protocols bgp traceoptions file files 30 set protocols bgp traceoptions flag normal
Underlay 設定~確認 (物理IF と eBGP)
まずは物理 Interface と eBGP 周りを設定して、各 lo0
の IP アドレスを経路交換できたことを確認します。
まあ、長々となってしまいますが、絵にすると単純で以下のような感じです。
物理IF設定
bb01
set interfaces xe-0/0/0 description "DEV=spine11 IF=xe-0/0/0" set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.1/30 delete interfaces xe-0/0/0 unit 0 family inet dhcp set interfaces xe-0/0/1 description "DEV=spine21 IF=xe-0/0/0" set interfaces xe-0/0/1 unit 0 family inet address 192.0.2.5/30 delete interfaces xe-0/0/1 unit 0 family inet dhcp set protocols lldp port-id-subtype interface-name set protocols lldp interface xe-0/0/0 set protocols lldp interface xe-0/0/1
spine11
set interfaces xe-0/0/0 description "DEV=bb01 IF=xe-0/0/0" set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.2/30 delete interfaces xe-0/0/0 unit 0 family inet dhcp set protocols lldp port-id-subtype interface-name set protocols lldp interface xe-0/0/0
spine21
set interfaces xe-0/0/0 description "DEV=bb01 IF=xe-0/0/1" set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.6/30 delete interfaces xe-0/0/0 unit 0 family inet dhcp set protocols lldp port-id-subtype interface-name set protocols lldp interface xe-0/0/0
物理IF疎通確認
リソースをケチったからか RTT
長すぎですね...。
bb01
kotetsu@bb01> show lldp neighbors Local Interface Parent Interface Chassis Id Port info System Name xe-0/0/0 - 02:05:86:71:84:00 DEV=bb01 IF=xe-0/0/0 spine11 xe-0/0/1 - 02:05:86:71:ff:00 DEV=bb01 IF=xe-0/0/1 spine21 kotetsu@bb01> show route inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.0.0.0/24 *[Direct/0] 19:47:39 > via em0.0 10.0.0.191/32 *[Local/0] 19:47:39 Local via em0.0 169.254.0.0/24 *[Direct/0] 19:57:21 > via em1.0 169.254.0.2/32 *[Local/0] 19:57:21 Local via em1.0 192.0.2.0/30 *[Direct/0] 00:17:12 > via xe-0/0/0.0 192.0.2.1/32 *[Local/0] 00:17:12 Local via xe-0/0/0.0 192.0.2.4/30 *[Direct/0] 00:07:04 > via xe-0/0/1.0 192.0.2.5/32 *[Local/0] 00:07:04 Local via xe-0/0/1.0 inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both fe80::286:2b0f:fc44:ab00/128 *[Direct/0] 19:56:54 > via lo0.0 {master:0} kotetsu@bb01> ping 192.0.2.2 PING 192.0.2.2 (192.0.2.2): 56 data bytes 64 bytes from 192.0.2.2: icmp_seq=1 ttl=64 time=2420.112 ms 64 bytes from 192.0.2.2: icmp_seq=2 ttl=64 time=1027.245 ms 64 bytes from 192.0.2.2: icmp_seq=3 ttl=64 time=1525.667 ms ^C --- 192.0.2.2 ping statistics --- 5 packets transmitted, 3 packets received, 40% packet loss round-trip min/avg/max/stddev = 1027.245/1657.675/2420.112/576.246 ms {master:0} kotetsu@bb01> ping 192.0.2.6 PING 192.0.2.6 (192.0.2.6): 56 data bytes 64 bytes from 192.0.2.6: icmp_seq=0 ttl=64 time=3378.582 ms 64 bytes from 192.0.2.6: icmp_seq=1 ttl=64 time=1374.159 ms 64 bytes from 192.0.2.6: icmp_seq=2 ttl=64 time=1474.743 ms ^C --- 192.0.2.6 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max/stddev = 1374.159/2075.828/3378.582/922.101 ms
spine11
{master:0} kotetsu@spine11> show lldp neighbors Local Interface Parent Interface Chassis Id Port info System Name xe-0/0/0 - 02:05:86:71:55:00 DEV=spine11 IF=xe-0/0/0 bb01 {master:0} kotetsu@spine11> show route inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.0.0.0/24 *[Direct/0] 19:23:06 > via em0.0 10.0.0.201/32 *[Local/0] 19:23:06 Local via em0.0 169.254.0.0/24 *[Direct/0] 19:22:26 > via em1.0 169.254.0.2/32 *[Local/0] 19:22:26 Local via em1.0 192.0.2.0/30 *[Direct/0] 00:16:05 > via xe-0/0/0.0 192.0.2.2/32 *[Local/0] 00:16:05 Local via xe-0/0/0.0 inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both fe80::286:2b0f:fca1:f500/128 *[Direct/0] 19:22:25 > via lo0.0 {master:0} kotetsu@spine11> ping 192.0.2.1 PING 192.0.2.1 (192.0.2.1): 56 data bytes 64 bytes from 192.0.2.1: icmp_seq=0 ttl=64 time=2839.663 ms 64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=2463.433 ms 64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=2269.077 ms 64 bytes from 192.0.2.1: icmp_seq=3 ttl=64 time=1548.765 ms 64 bytes from 192.0.2.1: icmp_seq=4 ttl=64 time=844.779 ms 64 bytes from 192.0.2.1: icmp_seq=5 ttl=64 time=505.582 ms 64 bytes from 192.0.2.1: icmp_seq=6 ttl=64 time=852.323 ms 64 bytes from 192.0.2.1: icmp_seq=7 ttl=64 time=1724.594 ms ^C --- 192.0.2.1 ping statistics --- 9 packets transmitted, 8 packets received, 11% packet loss round-trip min/avg/max/stddev = 505.582/1631.027/2839.663/795.890 ms
spine21
{master:0} kotetsu@spine21> show lldp neighbors Local Interface Parent Interface Chassis Id Port info System Name xe-0/0/0 - 02:05:86:71:55:00 DEV=spine21 IF=xe-0/0/0 bb01 {master:0} kotetsu@spine21> show route inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.0.0.0/24 *[Direct/0] 08:41:29 > via em0.0 10.0.0.202/32 *[Local/0] 08:41:30 Local via em0.0 169.254.0.0/24 *[Direct/0] 08:46:47 > via em1.0 169.254.0.2/32 *[Local/0] 08:46:47 Local via em1.0 192.0.2.4/30 *[Direct/0] 00:03:53 > via xe-0/0/0.0 192.0.2.6/32 *[Local/0] 00:03:53 Local via xe-0/0/0.0 inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both fe80::286:2b0f:fcd0:9b00/128 *[Direct/0] 08:46:47 > via lo0.0 {master:0} kotetsu@spine21> ping 192.0.2.5 PING 192.0.2.5 (192.0.2.5): 56 data bytes 64 bytes from 192.0.2.5: icmp_seq=0 ttl=64 time=427.225 ms 64 bytes from 192.0.2.5: icmp_seq=1 ttl=64 time=709.372 ms 64 bytes from 192.0.2.5: icmp_seq=2 ttl=64 time=1231.701 ms 64 bytes from 192.0.2.5: icmp_seq=3 ttl=64 time=732.131 ms ^C --- 192.0.2.5 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max/stddev = 427.225/775.107/1231.701/289.684 ms
lo0 + eBGP 設定
今回、いずれもシングル構成なのでマルチパス関係の設定は入れていませんよ。
bb01
set interfaces lo0 unit 0 family inet address 172.31.0.1/32 set policy-options policy-statement POLICY_EXPORT_LO0 from family inet set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement POLICY_EXPORT_LO0 then accept set routing-options router-id 172.31.0.1 set routing-options autonomous-system 65000 set protocols bgp group BGP_UNDERLAY type external set protocols bgp group BGP_UNDERLAY advertise-peer-as set protocols bgp group BGP_UNDERLAY family inet unicast loops 2 set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.2 description spine11 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.2 peer-as 65001 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.6 description spine21 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.6 peer-as 65002
spine11
set interfaces lo0 unit 0 family inet address 172.16.1.1/32 set policy-options policy-statement POLICY_EXPORT_LO0 from family inet set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement POLICY_EXPORT_LO0 then accept set routing-options router-id 172.16.1.1 set routing-options autonomous-system 65001 set protocols bgp group BGP_UNDERLAY type external set protocols bgp group BGP_UNDERLAY advertise-peer-as set protocols bgp group BGP_UNDERLAY family inet unicast loops 2 set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.1 description bb00 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.1 peer-as 65000
spine21
set interfaces lo0 unit 0 family inet address 172.16.2.1/32 set policy-options policy-statement POLICY_EXPORT_LO0 from family inet set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement POLICY_EXPORT_LO0 then accept set routing-options router-id 172.16.2.1 set routing-options autonomous-system 65002 set protocols bgp group BGP_UNDERLAY type external set protocols bgp group BGP_UNDERLAY advertise-peer-as set protocols bgp group BGP_UNDERLAY family inet unicast loops 2 set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.5 description bb00 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.5 peer-as 65000
eBGP確認
それぞれ lo0
の IPアドレスを相互学習したことを確認します。
この後やる Overlay 用の MP-BGP (iBGP) で bb01
が Route Reflector になるので、実際には bb01
と spine[12]1
で交換できていれば十分な筈ですが。
bb01
{master:0} kotetsu@bb01> show bgp summary Groups: 1 Peers: 2 Down peers: 1 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 1 1 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.0.2.2 65001 4 4 0 0 20 1/1/1/0 0/0/0/0 192.0.2.6 65002 1 2 0 0 5:57 OpenConfirm {master:0} kotetsu@bb01> show bgp summary Groups: 1 Peers: 2 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.0.2.2 65001 4 5 0 0 25 1/1/1/0 0/0/0/0 192.0.2.6 65002 4 4 0 0 4 1/1/1/0 0/0/0/0 {master:0} kotetsu@bb01> {master:0} kotetsu@bb01> show bgp summary Groups: 1 Peers: 2 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.0.2.2 65001 4 5 0 0 26 1/1/1/0 0/0/0/0 192.0.2.6 65002 4 4 0 0 5 1/1/1/0 0/0/0/0 {master:0} kotetsu@bb01> show bgp group BGP_UNDERLAY brief Group Type: External Local AS: 65000 Name: BGP_UNDERLAY Index: 0 Flags: <Export Eval> Export: [ POLICY_EXPORT_LO0 ] Options: <AdvertisePeerAs> Holdtime: 0 Total peers: 2 Established: 2 192.0.2.2+56114 192.0.2.6+49934 inet.0: 2/2/2/0 {master:0} kotetsu@bb01> show bgp neighbor Peer: 192.0.2.2+56114 AS 65001 Local: 192.0.2.1+179 AS 65000 Description: spine11 Group: BGP_UNDERLAY Routing-Instance: master Forwarding routing-instance: master Type: External State: Established Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: None Export: [ POLICY_EXPORT_LO0 ] Options: <Preference AddressFamily PeerAS Refresh> Options: <AdvertisePeerAs PeerSpecficLoopsAllowed> Address families configured: inet-unicast Holdtime: 90 Preference: 170 Number of flaps: 0 Peer ID: 172.16.1.1 Local ID: 172.31.0.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 0 Peer index: 0 BFD: disabled, down Local Interface: xe-0/0/0.0 NLRI for restart configured on peer: inet-unicast NLRI advertised by peer: inet-unicast NLRI for this session: inet-unicast Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: inet-unicast NLRI of received end-of-rib markers: inet-unicast NLRI of all end-of-rib markers sent: inet-unicast Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 65001) Peer does not support Addpath Table inet.0 Bit: 10000 RIB State: BGP restart is complete Send state: in sync Active prefixes: 1 Received prefixes: 1 Accepted prefixes: 1 Suppressed due to damping: 0 Advertised prefixes: 2 Last traffic (seconds): Received 23 Sent 5 Checked 56 Input messages: Total 179 Updates 2 Refreshes 0 Octets 3478 Output messages: Total 186 Updates 2 Refreshes 0 Octets 3663 Output Queue[0]: 0 (inet.0, inet-unicast) Peer: 192.0.2.6+49934 AS 65002 Local: 192.0.2.5+179 AS 65000 Description: spine21 Group: BGP_UNDERLAY Routing-Instance: master Forwarding routing-instance: master Type: External State: Established Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: None Export: [ POLICY_EXPORT_LO0 ] Options: <Preference AddressFamily PeerAS Refresh> Options: <AdvertisePeerAs PeerSpecficLoopsAllowed> Address families configured: inet-unicast Holdtime: 90 Preference: 170 Number of flaps: 0 Peer ID: 172.16.2.1 Local ID: 172.31.0.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 0 Peer index: 1 BFD: disabled, down Local Interface: xe-0/0/1.0 NLRI for restart configured on peer: inet-unicast NLRI advertised by peer: inet-unicast NLRI for this session: inet-unicast Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: inet-unicast NLRI of received end-of-rib markers: inet-unicast NLRI of all end-of-rib markers sent: inet-unicast Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 65002) Peer does not support Addpath Table inet.0 Bit: 10000 RIB State: BGP restart is complete Send state: in sync Active prefixes: 1 Received prefixes: 1 Accepted prefixes: 1 Suppressed due to damping: 0 Advertised prefixes: 2 Last traffic (seconds): Received 54 Sent 19 Checked 15 Input messages: Total 92 Updates 2 Refreshes 0 Octets 1825 Output messages: Total 184 Updates 2 Refreshes 0 Octets 3625 Output Queue[0]: 0 (inet.0, inet-unicast)
{master:0} kotetsu@bb01> show route protocol bgp inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.16.1.1/32 *[BGP/170] 00:02:11, localpref 100 AS path: 65001 I, validation-state: unverified > to 192.0.2.2 via xe-0/0/0.0 172.16.2.1/32 *[BGP/170] 00:01:49, localpref 100 AS path: 65002 I, validation-state: unverified > to 192.0.2.6 via xe-0/0/1.0 inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) {master:0} kotetsu@bb01> ping 172.16.2.1 PING 172.16.2.1 (172.16.2.1): 56 data bytes 64 bytes from 172.16.2.1: icmp_seq=0 ttl=64 time=908.929 ms 64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=1249.259 ms ^C --- 172.16.2.1 ping statistics --- 3 packets transmitted, 2 packets received, 33% packet loss round-trip min/avg/max/stddev = 908.929/1079.094/1249.259/170.165 ms {master:0} kotetsu@bb01> ping 172.16.1.1 PING 172.16.1.1 (172.16.1.1): 56 data bytes 64 bytes from 172.16.1.1: icmp_seq=0 ttl=64 time=2595.413 ms 64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=1406.501 ms ^C --- 172.16.1.1 ping statistics --- 3 packets transmitted, 2 packets received, 33% packet loss round-trip min/avg/max/stddev = 1406.501/2000.957/2595.413/594.456 ms
spine11
{master:0} kotetsu@spine11> show bgp summary Groups: 1 Peers: 1 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.0.2.1 65000 174 169 0 0 1:16:34 2/2/2/0 0/0/0/0 {master:0} kotetsu@spine11> show bgp group BGP_UNDERLAY brief Group Type: External Local AS: 65001 Name: BGP_UNDERLAY Index: 0 Flags: <Export Eval> Export: [ POLICY_EXPORT_LO0 ] Options: <AdvertisePeerAs> Holdtime: 0 Total peers: 1 Established: 1 192.0.2.1+179 inet.0: 2/2/2/0 {master:0} kotetsu@spine11> show bgp neighbor Peer: 192.0.2.1+179 AS 65000 Local: 192.0.2.2+56114 AS 65001 Description: bb00 Group: BGP_UNDERLAY Routing-Instance: master Forwarding routing-instance: master Type: External State: Established Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: None Export: [ POLICY_EXPORT_LO0 ] Options: <Preference AddressFamily PeerAS Refresh> Options: <AdvertisePeerAs PeerSpecficLoopsAllowed> Address families configured: inet-unicast Holdtime: 90 Preference: 170 Number of flaps: 0 Peer ID: 172.31.0.1 Local ID: 172.16.1.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 0 Peer index: 0 BFD: disabled, down Local Interface: xe-0/0/0.0 NLRI for restart configured on peer: inet-unicast NLRI advertised by peer: inet-unicast NLRI for this session: inet-unicast Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: inet-unicast NLRI of received end-of-rib markers: inet-unicast NLRI of all end-of-rib markers sent: inet-unicast Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 65000) Peer does not support Addpath Table inet.0 Bit: 10000 RIB State: BGP restart is complete Send state: in sync Active prefixes: 2 Received prefixes: 2 Accepted prefixes: 2 Suppressed due to damping: 0 Advertised prefixes: 1 Last traffic (seconds): Received 9 Sent 7 Checked 50 Input messages: Total 182 Updates 3 Refreshes 0 Octets 3524 Output messages: Total 177 Updates 1 Refreshes 0 Octets 3459 Output Queue[0]: 0 (inet.0, inet-unicast)
{master:0} kotetsu@spine11> show route protocol bgp inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.16.2.1/32 *[BGP/170] 01:22:49, localpref 100 AS path: 65000 65002 I, validation-state: unverified > to 192.0.2.1 via xe-0/0/0.0 172.31.0.1/32 *[BGP/170] 01:23:10, localpref 100 AS path: 65000 I, validation-state: unverified > to 192.0.2.1 via xe-0/0/0.0 inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) {master:0} kotetsu@spine11> ping 172.31.0.1 PING 172.31.0.1 (172.31.0.1): 56 data bytes 64 bytes from 172.31.0.1: icmp_seq=0 ttl=64 time=1597.291 ms 64 bytes from 172.31.0.1: icmp_seq=1 ttl=64 time=1215.306 ms ^C --- 172.31.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 1215.306/1406.299/1597.291/190.993 ms
spine21
{master:0} kotetsu@spine21> show bgp summary Groups: 1 Peers: 1 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.0.2.5 65000 173 88 0 0 38:05 2/2/2/0 0/0/0/0 {master:0} kotetsu@spine21> show bgp group BGP_UNDERLAY brief Group Type: External Local AS: 65002 Name: BGP_UNDERLAY Index: 0 Flags: <Export Eval> Export: [ POLICY_EXPORT_LO0 ] Options: <AdvertisePeerAs> Holdtime: 0 Total peers: 1 Established: 1 192.0.2.5+179 inet.0: 2/2/2/0 kotetsu@spine21> show bgp neighbor Peer: 192.0.2.5+179 AS 65000 Local: 192.0.2.6+49934 AS 65002 Description: bb00 Group: BGP_UNDERLAY Routing-Instance: master Forwarding routing-instance: master Type: External State: Established Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: None Export: [ POLICY_EXPORT_LO0 ] Options: <Preference AddressFamily PeerAS Refresh> Options: <AdvertisePeerAs PeerSpecficLoopsAllowed> Address families configured: inet-unicast Holdtime: 90 Preference: 170 Number of flaps: 0 Peer ID: 172.31.0.1 Local ID: 172.16.2.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 0 Peer index: 0 BFD: disabled, down Local Interface: xe-0/0/0.0 NLRI for restart configured on peer: inet-unicast NLRI advertised by peer: inet-unicast NLRI for this session: inet-unicast Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: inet-unicast NLRI of received end-of-rib markers: inet-unicast NLRI of all end-of-rib markers sent: inet-unicast Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 65000) Peer does not support Addpath Table inet.0 Bit: 10000 RIB State: BGP restart is complete Send state: in sync Active prefixes: 2 Received prefixes: 2 Accepted prefixes: 2 Suppressed due to damping: 0 Advertised prefixes: 1 Last traffic (seconds): Received 1 Sent 24 Checked 3 Input messages: Total 175 Updates 3 Refreshes 0 Octets 3391 Output messages: Total 88 Updates 1 Refreshes 0 Octets 1768 Output Queue[0]: 0 (inet.0, inet-unicast)
{master:0} kotetsu@spine21> show route protocol bgp inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.16.1.1/32 *[BGP/170] 00:41:43, localpref 100 AS path: 65000 65001 I, validation-state: unverified > to 192.0.2.5 via xe-0/0/0.0 172.31.0.1/32 *[BGP/170] 00:41:43, localpref 100 AS path: 65000 I, validation-state: unverified > to 192.0.2.5 via xe-0/0/0.0 inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) {master:0} kotetsu@spine21> ping 172.31.0.1 PING 172.31.0.1 (172.31.0.1): 56 data bytes 64 bytes from 172.31.0.1: icmp_seq=0 ttl=64 time=691.327 ms 64 bytes from 172.31.0.1: icmp_seq=1 ttl=64 time=1227.117 ms 64 bytes from 172.31.0.1: icmp_seq=2 ttl=64 time=238.494 ms ^C --- 172.31.0.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max/stddev = 238.494/718.979/1227.117/404.077 ms
Overlay 設定~確認 (MP-BGP)
Underlay eBGP で経路交換した各 lo0
同士で iBGP を組みます。
まあ、長々となってしまいますが、絵にすると単純で以下のような感じです。
iBGP (MP-BGP )設定
ポイントは1つだけで family evpn signaling
を設定して EVPN の NLRI を扱うってとこですかね。
bb01
set protocols bgp group BGP_OVERLAY type internal set protocols bgp group BGP_OVERLAY local-address 172.31.0.1 set protocols bgp group BGP_OVERLAY family evpn signaling set protocols bgp group BGP_OVERLAY cluster 172.31.0.1 set protocols bgp group BGP_OVERLAY local-as 64512 set protocols bgp group BGP_OVERLAY neighbor 172.16.1.1 description spine11 set protocols bgp group BGP_OVERLAY neighbor 172.16.2.1 description spine21
spine11
set protocols bgp group BGP_OVERLAY type internal set protocols bgp group BGP_OVERLAY local-address 172.16.1.1 set protocols bgp group BGP_OVERLAY family evpn signaling set protocols bgp group BGP_OVERLAY local-as 64512 set protocols bgp group BGP_OVERLAY neighbor 172.31.0.1 description bb01
spine21
set protocols bgp group BGP_OVERLAY type internal set protocols bgp group BGP_OVERLAY local-address 172.16.2.1 set protocols bgp group BGP_OVERLAY family evpn signaling set protocols bgp group BGP_OVERLAY local-as 64512 set protocols bgp group BGP_OVERLAY neighbor 172.31.0.1 description bb01
iBGP (MP-BGP) 確認
最初 neighbor IP アドレス設定をタイポしたせいで、Last Error: Open Message Error
とか残っているのはご愛敬ということでひとつ...。
bb01
{master:0} kotetsu@bb01> show bgp summary Groups: 2 Peers: 4 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 bgp.evpn.0 0 0 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 172.16.1.1 64512 3 3 0 0 25 Establ bgp.evpn.0: 0/0/0/0 172.16.2.1 64512 3 2 0 0 12 Establ bgp.evpn.0: 0/0/0/0 192.0.2.2 65001 502 518 0 0 3:58:25 1/1/1/0 0/0/0/0 192.0.2.6 65002 257 517 0 0 3:58:04 1/1/1/0 0/0/0/0 {master:0} kotetsu@bb01> show bgp group BGP_OVERLAY Group Type: Internal AS: 64512 Local AS: 64512 Name: BGP_OVERLAY Index: 1 Flags: <Export Eval> Options: <Cluster LocalAS> Holdtime: 0 Local AS: 64512 Local System AS: 65000 Total peers: 2 Established: 2 172.16.1.1+52050 172.16.2.1+62794 Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 bgp.evpn.0: 0/0/0/0 {master:0} kotetsu@bb01> show bgp neighbor Peer: 172.16.1.1+52050 AS 64512 Local: 172.31.0.1+179 AS 64512 Description: spine11 Group: BGP_OVERLAY Routing-Instance: master Forwarding routing-instance: master Type: Internal State: Established (route reflector client)Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: Open Message Error Options: <Preference LocalAddress Cluster AddressFamily LocalAS Rib-group Refresh> Address families configured: evpn Local Address: 172.31.0.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65000 Number of flaps: 0 Error: 'Open Message Error' Sent: 20 Recv: 0 Peer ID: 172.16.1.1 Local ID: 172.31.0.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 1 Peer index: 0 BFD: disabled, down NLRI for restart configured on peer: evpn NLRI advertised by peer: evpn NLRI for this session: evpn Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: evpn NLRI of received end-of-rib markers: evpn NLRI of all end-of-rib markers sent: evpn Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 64512) Peer does not support Addpath Table bgp.evpn.0 Bit: 20000 RIB State: BGP restart is complete RIB State: VPN restart is complete Send state: in sync Active prefixes: 0 Received prefixes: 0 Accepted prefixes: 0 Suppressed due to damping: 0 Advertised prefixes: 0 Last traffic (seconds): Received 1 Sent 9 Checked 0 Input messages: Total 6 Updates 1 Refreshes 0 Octets 169 Output messages: Total 6 Updates 0 Refreshes 0 Octets 188 Output Queue[1]: 0 (bgp.evpn.0, evpn) Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 Peer: 172.16.2.1+62794 AS 64512 Local: 172.31.0.1+179 AS 64512 Description: spine21 Group: BGP_OVERLAY Routing-Instance: master Forwarding routing-instance: master Type: Internal State: Established (route reflector client)Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: Open Message Error Options: <Preference LocalAddress Cluster AddressFamily LocalAS Rib-group Refresh> Address families configured: evpn Local Address: 172.31.0.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65000 Number of flaps: 0 Error: 'Open Message Error' Sent: 20 Recv: 0 Peer ID: 172.16.2.1 Local ID: 172.31.0.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 1 Peer index: 1 BFD: disabled, down NLRI for restart configured on peer: evpn NLRI advertised by peer: evpn NLRI for this session: evpn Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: evpn NLRI of received end-of-rib markers: evpn NLRI of all end-of-rib markers sent: evpn Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 64512) Peer does not support Addpath Table bgp.evpn.0 Bit: 20000 RIB State: BGP restart is complete RIB State: VPN restart is complete Send state: in sync Active prefixes: 0 Received prefixes: 0 Accepted prefixes: 0 Suppressed due to damping: 0 Advertised prefixes: 0 Last traffic (seconds): Received 46 Sent 0 Checked 9 Input messages: Total 10 Updates 1 Refreshes 0 Octets 245 Output messages: Total 18 Updates 0 Refreshes 0 Octets 416 Output Queue[1]: 0 (bgp.evpn.0, evpn) Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 ...
spine11
{master:0} kotetsu@spine11> show bgp summary Groups: 2 Peers: 2 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 bgp.evpn.0 0 0 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 172.31.0.1 64512 21 22 0 0 9:07 Establ bgp.evpn.0: 0/0/0/0 192.0.2.1 65000 536 521 0 0 3:59:55 2/2/2/0 0/0/0/ {master:0} kotetsu@spine11> show bgp group BGP_OVERLAY Group Type: Internal AS: 64512 Local AS: 64512 Name: BGP_OVERLAY Index: 1 Flags: <Export Eval> Options: <LocalAS> Holdtime: 0 Local AS: 64512 Local System AS: 65001 Total peers: 1 Established: 1 172.31.0.1+179 Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 bgp.evpn.0: 0/0/0/0 {master:0} kotetsu@spine11> show bgp neighbor Peer: 172.31.0.1+179 AS 64512 Local: 172.16.1.1+52050 AS 64512 Description: bb01 Group: BGP_OVERLAY Routing-Instance: master Forwarding routing-instance: master Type: Internal State: Established Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: None Options: <Preference LocalAddress AddressFamily LocalAS Rib-group Refresh> Address families configured: evpn Local Address: 172.16.1.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65001 Number of flaps: 0 Peer ID: 172.31.0.1 Local ID: 172.16.1.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 1 Peer index: 1 BFD: disabled, down NLRI for restart configured on peer: evpn NLRI advertised by peer: evpn NLRI for this session: evpn Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: evpn NLRI of received end-of-rib markers: evpn NLRI of all end-of-rib markers sent: evpn Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 64512) Peer does not support Addpath Table bgp.evpn.0 RIB State: BGP restart is complete RIB State: VPN restart is complete Send state: not advertising Active prefixes: 0 Received prefixes: 0 Accepted prefixes: 0 Suppressed due to damping: 0 Last traffic (seconds): Received 1 Sent 8 Checked 46 Input messages: Total 34 Updates 1 Refreshes 0 Octets 657 Output messages: Total 34 Updates 0 Refreshes 0 Octets 720 Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 ...
spine21
{master:0} kotetsu@spine21> show bgp summary Groups: 2 Peers: 2 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 2 0 0 0 0 bgp.evpn.0 0 0 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 172.31.0.1 64512 34 19 0 0 7:30 Establ bgp.evpn.0: 0/0/0/0 192.0.2.5 65000 549 273 0 0 2:02:08 2/2/2/0 0/0/0/0 {master:0} kotetsu@spine21> show bgp group BGP_OVERLAY Group Type: Internal AS: 64512 Local AS: 64512 Name: BGP_OVERLAY Index: 1 Flags: <Export Eval> Options: <LocalAS> Holdtime: 0 Local AS: 64512 Local System AS: 65002 Total peers: 1 Established: 1 172.31.0.1+179 Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 bgp.evpn.0: 0/0/0/0 {master:0} kotetsu@spine21> show bgp neighbor Peer: 172.31.0.1+179 AS 64512 Local: 172.16.2.1+62794 AS 64512 Description: bb01 Group: BGP_OVERLAY Routing-Instance: master Forwarding routing-instance: master Type: Internal State: Established Flags: <Sync> Last State: OpenConfirm Last Event: RecvKeepAlive Last Error: None Options: <Preference LocalAddress AddressFamily LocalAS Rib-group Refresh> Address families configured: evpn Local Address: 172.16.2.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65002 Number of flaps: 0 Peer ID: 172.31.0.1 Local ID: 172.16.2.1 Active Holdtime: 90 Keepalive Interval: 30 Group index: 1 Peer index: 1 BFD: disabled, down NLRI for restart configured on peer: evpn NLRI advertised by peer: evpn NLRI for this session: evpn Peer supports Refresh capability (2) Stale routes from peer are kept for: 300 Peer does not support Restarter functionality Restart flag received from the peer: Notification NLRI that restart is negotiated for: evpn NLRI of received end-of-rib markers: evpn NLRI of all end-of-rib markers sent: evpn Peer does not support LLGR Restarter functionality Peer supports 4 byte AS extension (peer-as 64512) Peer does not support Addpath Table bgp.evpn.0 RIB State: BGP restart is complete RIB State: VPN restart is complete Send state: not advertising Active prefixes: 0 Received prefixes: 0 Accepted prefixes: 0 Suppressed due to damping: 0 Last traffic (seconds): Received 14 Sent 17 Checked 58 Input messages: Total 36 Updates 1 Refreshes 0 Octets 695 Output messages: Total 20 Updates 0 Refreshes 0 Octets 454 Trace options: normal Trace file: /var/log/bgp.log size 10240 files 30 ...
Overlay設定~確認(EVPN+VXLAN)
EVPN の設定と、その Dataplane として使う VXLAN 周りの設定をしていきます。(息切れ)
EVPN+VXLAN 設定
冒頭の全体構成に書いたのですが bb01
はこの辺の挙動に関しては土管に徹しているので登場しません。
VNI は 1..16777214
まで使えるので、単純に全 spine で VLAN ID と 1:1 で同じ対応付けをさせて、一律で VLAN ID + 10000
の値を振ってます。
あ、VNI 10100 以外は本項ではまだ使いません。
spine11
set vlans VLAN0100 vlan-id 100 set vlans VLAN0100 vxlan vni 10100 set vlans VLAN0100 vxlan ingress-node-replication set vlans VLAN0300 vlan-id 300 set vlans VLAN0300 vxlan vni 10300 set vlans VLAN0300 vxlan ingress-node-replication set protocols evpn encapsulation vxlan set protocols evpn extended-vni-list all set protocols evpn multicast-mode ingress-replication set protocols evpn vni-options vni 10100 vrf-target export target:1:10100 set protocols evpn vni-options vni 10300 vrf-target export target:1:10300 set policy-options community COM_10100 members target:1:10100 set policy-options community COM_10300 members target:1:10300 set policy-options community COM_LEAF_ESI members target:9999:9999 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 from community COM_10100 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_10300 from community COM_10300 set policy-options policy-statement POLICY_VRF_IMPORT term T_10300 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 from community COM_LEAF_ESI set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99999 then reject set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 64512:11 set switch-options vrf-import POLICY_VRF_IMPORT set switch-options vrf-target target:9999:9999 set switch-options vrf-target auto set interfaces xe-0/0/1 description "DEV=torSW11 IF=1" set interfaces xe-0/0/1 unit 0 family ethernet-switching interface-mode trunk set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members all
spine21
set vlans VLAN0100 vlan-id 100 set vlans VLAN0100 vxlan vni 10100 set vlans VLAN0100 vxlan ingress-node-replication set vlans VLAN0200 vlan-id 200 set vlans VLAN0200 vxlan vni 10200 set vlans VLAN0200 vxlan ingress-node-replication set protocols evpn encapsulation vxlan set protocols evpn extended-vni-list all set protocols evpn multicast-mode ingress-replication set protocols evpn vni-options vni 10100 vrf-target export target:1:10100 set protocols evpn vni-options vni 10200 vrf-target export target:1:10200 set policy-options community COM_10100 members target:1:10100 set policy-options community COM_10200 members target:1:10200 set policy-options community COM_LEAF_ESI members target:9999:9999 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 from community COM_10100 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_10200 from community COM_10200 set policy-options policy-statement POLICY_VRF_IMPORT term T_10200 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 from community COM_LEAF_ESI set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99999 then reject set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 64512:21 set switch-options vrf-import POLICY_VRF_IMPORT set switch-options vrf-target target:9999:9999 set switch-options vrf-target auto set interfaces xe-0/0/1 description "DEV=torSW21 IF=1" set interfaces xe-0/0/1 unit 0 family ethernet-switching interface-mode trunk set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members all
動作確認
node 同士の疎通確認
- node11
$ ip a show dev ens4 3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:86:2b:8d:10:01 brd ff:ff:ff:ff:ff:ff inet 192.168.1.1/24 brd 192.168.1.255 scope global ens4 valid_lft forever preferred_lft forever inet6 fe80::286:2bff:fe8d:1001/64 scope link valid_lft forever preferred_lft forever $ ping 192.168.1.2 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. 64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=975 ms 64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=949 ms $ ip n show dev ens4 192.168.1.2 lladdr 00:86:2b:5c:0d:01 STALE
- node21
$ ip a show dev ens4 3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:86:2b:5c:0d:01 brd ff:ff:ff:ff:ff:ff inet 192.168.1.2/24 brd 192.168.1.255 scope global ens4 valid_lft forever preferred_lft forever inet6 fe80::286:2bff:fe5c:d01/64 scope link valid_lft forever preferred_lft forever $ ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=639 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=926 ms $ ip n show dev ens4 192.168.1.1 lladdr 00:86:2b:8d:10:01 STALE
vQFX のテーブル確認
前述の Juniper公式日本語資料 のページ 36 によると
bgp.evpn.0 = Junos OS ルーティング プロトコル プロセス(RPD)内のグローバル EVPN ルーティング テーブル default.switch.evpn.0(QFX5100 の場合) = Junos OS RPD 内のスイッチ レベル EVPN 転送テーブル \<virtual-switch-name>.evpn.0(MX の場合) = Junos OS RPD 内の仮想スイッチ レベル EVPN 転送テーブル
らしいですよ。
spine11
こいつにとっての vtep.32769
てのは spine21
(の VTEP)
{master:0} kotetsu@spine11> show ethernet-switching table MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 2 entries, 2 learned Routing instance : default-switch Vlan MAC MAC Logical Active name address flags interface source VLAN0100 00:86:2b:5c:0d:01 D vtep.32769 172.16.2.1 VLAN0100 00:86:2b:8d:10:01 D xe-0/0/1.0 {master:0} kotetsu@spine11> show interfaces vtep.32769 Logical interface vtep.32769 (Index 566) (SNMP ifIndex 537) Flags: Up SNMP-Traps Encapsulation: ENET2 VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 172.16.2.1, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 4 Output packets: 4 Protocol eth-switch, MTU: Unlimited Flags: Trunk-Mode
EVPN の Type2(MAC/IP Advertisement route
) と Type3(Inclusive Multicast Ethernet Tag route
) を spine21
(RD 64512:21
)から学習して bgp.evpn.0
から default-switch.evpn.0
にロードされている様子
{master:0} kotetsu@spine11> show route ... :vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.0.0.0/24 *[Direct/0] 00:23:24 > via em0.0 10.0.0.201/32 *[Local/0] 00:06:15 Local via em0.0 169.254.0.0/24 *[Direct/0] 00:23:24 > via em1.0 169.254.0.2/32 *[Local/0] 00:23:24 Local via em1.0 172.16.1.1/32 *[Direct/0] 00:23:24 > via lo0.0 172.16.2.1/32 *[Static/1] 00:13:22, metric2 0 > to 192.0.2.1 via xe-0/0/0.0 192.0.2.0/30 *[Direct/0] 00:23:24 > via xe-0/0/0.0 192.0.2.2/32 *[Local/0] 00:23:24 Local via xe-0/0/0.0 bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:64512:21::10100::00:86:2b:5c:0d:01/304 *[BGP/170] 00:08:29, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.1 via xe-0/0/0.0 3:64512:21::10100::172.16.2.1/304 *[BGP/170] 00:13:23, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.1 via xe-0/0/0.0 default-switch.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:64512:11::10100::00:86:2b:8d:10:01/304 *[EVPN/170] 00:08:33 Indirect 2:64512:21::10100::00:86:2b:5c:0d:01/304 *[BGP/170] 00:08:29, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.1 via xe-0/0/0.0 3:64512:11::10100::172.16.1.1/304 *[EVPN/170] 00:23:27 Indirect 3:64512:11::10300::172.16.1.1/304 *[EVPN/170] 00:15:11 Indirect 3:64512:21::10100::172.16.2.1/304 *[BGP/170] 00:13:23, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.1 via xe-0/0/0.0
{master:0} kotetsu@spine11> show route forwarding-table ... Routing table: :vxlan.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 1724 1 0.0.0.0/32 perm 0 dscd 1722 1 169.254.0.0/24 user 0 rtbl 1 5 169.254.0.2/32 user 0 169.254.0.2 locl 338 3 172.16.1.1/32 user 0 rtbl 1 5 172.16.2.1/32 user 0 indr 131070 3 192.0.2.1 ucst 1719 7 xe-0/0/0.0 192.0.2.0/30 user 0 rtbl 1 5 192.0.2.2/32 user 0 192.0.2.2 locl 1717 3 224.0.0.0/4 perm 0 mdsc 1723 1 224.0.0.1/32 perm 0 224.0.0.1 mcst 1726 1 255.255.255.255/32 perm 0 bcst 1727 1 ... Routing table: default-switch.bridge Bridging domain: VLAN0100.bridge VPLS: Destination Type RtRef Next hop Type Index NhRef Netif 00:86:2b:5c:0d:01/48 user 0 comp 1707 6 00:86:2b:8d:10:01/48 user 0 ucst 1712 6 xe-0/0/1.0 0x30001/51 user 0 comp 1739 2 0x30006/51 user 0 comp 1732 2 0x30002/51 user 0 comp 1733 2 Routing table: default-switch.bridge Bridging domain: VLAN0300.bridge VPLS: Destination Type RtRef Next hop Type Index NhRef Netif 0x30007/51 user 0 comp 1741 2 0x30005/51 user 0 comp 1714 2 0x30003/51 user 0 comp 1734 2 ...
spine21
こいつにとっての vtep.32769
てのは spine11
(の VTEP)
{master:0} kotetsu@spine21> show ethernet-switching table MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 2 entries, 2 learned Routing instance : default-switch Vlan MAC MAC Logical Active name address flags interface source VLAN0100 00:86:2b:5c:0d:01 D xe-0/0/1.0 VLAN0100 00:86:2b:8d:10:01 D vtep.32769 172.16.1.1
EVPN の Type2(MAC/IP Advertisement route
) と Type3(Inclusive Multicast Ethernet Tag route
) を spine11
(RD 64512:11
)から学習して bgp.evpn.0
から default-switch.evpn.0
にロードされている様子
{master:0} kotetsu@spine21> show route ... :vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.0.0.0/24 *[Direct/0] 00:18:41 > via em0.0 10.0.0.202/32 *[Local/0] 00:11:20 Local via em0.0 169.254.0.0/24 *[Direct/0] 00:18:41 > via em1.0 169.254.0.2/32 *[Local/0] 00:18:41 Local via em1.0 172.16.1.1/32 *[Static/1] 00:18:39, metric2 0 > to 192.0.2.5 via xe-0/0/0.0 172.16.2.1/32 *[Direct/0] 00:18:41 > via lo0.0 192.0.2.4/30 *[Direct/0] 00:18:41 > via xe-0/0/0.0 192.0.2.6/32 *[Local/0] 00:18:41 Local via xe-0/0/0.0 bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:64512:11::10100::00:86:2b:8d:10:01/304 *[BGP/170] 00:12:17, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.5 via xe-0/0/0.0 3:64512:11::10100::172.16.1.1/304 *[BGP/170] 00:18:40, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.5 via xe-0/0/0.0 default-switch.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:64512:11::10100::00:86:2b:8d:10:01/304 *[BGP/170] 00:12:17, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.5 via xe-0/0/0.0 2:64512:21::10100::00:86:2b:5c:0d:01/304 *[EVPN/170] 00:12:16 Indirect 3:64512:11::10100::172.16.1.1/304 *[BGP/170] 00:18:40, localpref 100, from 172.31.0.1 AS path: I, validation-state: unverified > to 192.0.2.5 via xe-0/0/0.0 3:64512:21::10100::172.16.2.1/304 *[EVPN/170] 00:14:43 Indirect 3:64512:21::10200::172.16.2.1/304 *[EVPN/170] 00:14:43 Indirect
{master:0} kotetsu@spine21> show route forwarding-table ... Routing table: :vxlan.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 1716 1 0.0.0.0/32 perm 0 dscd 1714 1 169.254.0.0/24 user 0 rtbl 1 5 169.254.0.2/32 user 0 169.254.0.2 locl 334 3 172.16.1.1/32 user 0 indr 131070 3 192.0.2.5 ucst 1711 7 xe-0/0/0.0 172.16.2.1/32 user 0 rtbl 1 5 192.0.2.4/30 user 0 rtbl 1 5 192.0.2.6/32 user 0 192.0.2.6 locl 1709 3 224.0.0.0/4 perm 0 mdsc 1715 1 224.0.0.1/32 perm 0 224.0.0.1 mcst 1718 1 255.255.255.255/32 perm 0 bcst 1719 1 ... Routing table: default-switch.bridge Bridging domain: VLAN0100.bridge VPLS: Destination Type RtRef Next hop Type Index NhRef Netif 00:86:2b:5c:0d:01/48 user 0 ucst 1730 6 xe-0/0/1.0 00:86:2b:8d:10:01/48 user 0 comp 1723 6 0x30006/51 user 0 comp 1731 2 0x30004/51 user 0 comp 1725 2 0x30001/51 user 0 comp 1726 2 Routing table: default-switch.bridge Bridging domain: VLAN0200.bridge VPLS: Destination Type RtRef Next hop Type Index NhRef Netif 0x30007/51 user 0 comp 1734 2 0x30005/51 user 0 comp 1729 2 0x30002/51 user 0 comp 1728 2
bb01
{master:0} kotetsu@bb01> show ethernet-switching table
単純に EVPN 経路を reflect する土管なので bgp.evpn.0
にしか載っていない様子
{master:0} kotetsu@bb01> show route ... bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:64512:11::10100::00:86:2b:8d:10:01/304 *[BGP/170] 00:31:03, localpref 100, from 172.16.1.1 AS path: I, validation-state: unverified > to 192.0.2.2 via xe-0/0/0.0 2:64512:21::10100::00:86:2b:5c:0d:01/304 *[BGP/170] 00:31:01, localpref 100, from 172.16.2.1 AS path: I, validation-state: unverified > to 192.0.2.6 via xe-0/0/1.0 3:64512:11::10100::172.16.1.1/304 *[BGP/170] 00:46:21, localpref 100, from 172.16.1.1 AS path: I, validation-state: unverified > to 192.0.2.2 via xe-0/0/0.0 3:64512:11::10300::172.16.1.1/304 *[BGP/170] 00:37:52, localpref 100, from 172.16.1.1 AS path: I, validation-state: unverified > to 192.0.2.2 via xe-0/0/0.0 3:64512:21::10100::172.16.2.1/304 *[BGP/170] 00:36:05, localpref 100, from 172.16.2.1 AS path: I, validation-state: unverified > to 192.0.2.6 via xe-0/0/1.0 3:64512:21::10200::172.16.2.1/304 *[BGP/170] 00:35:45, localpref 100, from 172.16.2.1 AS path: I, validation-state: unverified > to 192.0.2.6 via xe-0/0/1.0 {master:0} kotetsu@bb01> show route forwarding-table ...
パケットキャプチャ
いくつか特徴的なパケットを拾って眺めていきます。
前述の通り、今回は RE 同士を接続するところで直結構成にしてしまい bb01
が xe-0/0/2
から captureSW
方面にポートミラーリングして、GNS3 の bb01(RE)
と captureSW
間のリンクを右クリックして start capture
とかやっていきます。
なので bb01
に以下のような設定をして、xe-0/0/0
(spine11
側)の出入りを xe-0/0/2
にミラーリングしときます。
set interfaces xe-0/0/2 description "DEV=captureSW IF=1" set interfaces xe-0/0/2 unit 0 family ethernet-switching set forwarding-options analyzer ANAL_PORT input ingress interface xe-0/0/0.0 set forwarding-options analyzer ANAL_PORT input egress interface xe-0/0/0.0 set forwarding-options analyzer ANAL_PORT output interface xe-0/0/2.0
見返していて思ったんすが、アナルポートって...ちょっと...
あと、vQFX でも start shell
して tcpdump
とか使えますよ。Control Plane 宛のトラフィックだけとりたい時とかには。
DataPlane
bb01
で拾った VXLAN のパケットを見ていきます。
node11
から node21
に ping うった時、最初に出る ARP Request と ARP reply
VXLAN カプセルの IP ヘッダ的にはマルチキャストではなく、spine11 VTEP
と spine21 VTEP
間のユニキャストになってます。
node11
と node21
間の ICMP echo Request と ICMP echo Reply
これは EVPN 使おうが関係なく、ただの VXLAN ヘッダついたパケットです。
ControlPlane
bb01
で拾った EVPN のパケットを見ていきます。
EVPN NLRI Type2(MAC/IP Advertisement route) Update
まずは node11
からの ARP Request を受信した spine11
が advertise する様子
次に node21
からの ARP Reply を受信した spine21
が advertise したのを bb01
が spine11
方面に reflect する様子
EVPN NLRI Type2(MAC/IP Advertisement route) Withdrawn
spine11
で clear ethernet-switching table
して node11
の MAC を消す
EVPN NLRI Type3(Inclusive Multicast Ethernet Tag route) Withdrawn
spine21
で以下のような感じで、雑に VLAN 100 = VNI10100 at RD 64512:21 が消えたことを advertise させた時の様子
{master:0}[edit] kotetsu@spine21# deactivate vlans VLAN0100 {master:0}[edit] kotetsu@spine21# show | compare [edit vlans] ! inactive: VLAN0100 { ... } {master:0}[edit] kotetsu@spine21# commit
次に spine21
で VLAN 100 = VNI10100 at RD 64512:21 ができたことを advertise させた時の様子
{master:0}[edit] kotetsu@spine21# activate vlans VLAN0100 {master:0}[edit] kotetsu@spine21# show | compare [edit vlans] ! active: VLAN0100 { ... } {master:0}[edit] kotetsu@spine21# commit