vQFX10000 で VXLAN+EVPN (L2 over L3 編) (original : 2017/01/02)
この記事は某所で 2017/01/02 に書いた記事のコピーです。
そのため 2017/05/11 時点ではやや古い情報も含まれています。(以下一例)
- Juniper さんの vQFX ダウンロード規約が変わっていそう (詳細は未確認)
- 201705 に
GNS3 ver2.0 stableが出たので KVM 間のパケットキャプチャ用に GNS3 のハブを挟む必要がなくなった
.
- 最初に
- 構築～動作確認
- おしまい
最初に
前回(vQFX10000 を KVM+GNS3 で動かす)、Juniper vQFX10000(以降 vQFX) の DL 権限を個人で得て GNS3 で軽く動作確認がとれました。 今回は「仮想版って L2全般/L2VPN 系機能が動かなかったりするけど、vQFX はどうなんだ？...お、EVPN もちゃんと動くやんけ！」ってところまでを見ていきます。
本項でやること
以下をやります。
- vQFX で VXLAN の Control Plane として EVPN を動かす
- 仮想版だと L2VPN 系機能が動かないアプライアンスもあるので
- EVPN の適用範囲は L2 シングル構成のみで、DataPlane としては VXLAN を使う
- L2VPN への L3 統合周り(inter-subnet-forwarding 周り)やマルチホーミングなどの機能は、今回の範囲外
- Dataplane としては VXLAN を使い、MPLS は取り扱わない
- Juniper公式 | ジュニパーネットワークス EVPN 次世代データセンター アーキテクチャの実装 で MX と QFX5100 でやっていることの一部を vQFX10000 で試す
- 各要素技術の詳細な解説は行わない
個人的には、2014/12 時点で VXLAN のマルチキャスト実装を確認した時 (VyOS と Arista で VXLAN 相互接続)に微妙だと思った点にも着目しておきたいです。今回、データプレーン周りはほぼ同じ構成を組んでますので。
- 上の記事の〆が「マルチキャストルーティングを勉強しよう。」だったのですが、この2年間以下のようにウダウダやっていて、特に進捗はありませんでした。
- 上の記事では VTEP 間の到達性をもたせるのに「VyOS が認識しないといけない Arista 側の VTEP IP アドレスは、Arista の Loopback アドレスになるので、StaticRoute を追加しておきます。#これだと拡張が面倒なので、実環境では DefaultRoute か動的ルーティングを使うことが多い気がします。」とか書いてますが、今回は実環境をある程度想定した例示ができそうです。
概要構成図 / 環境
以下のような構成を組んでいきます。node11 と node21 が L2overL3 で通信するやつです。
環境は、いずれも前回の通りです。
参考資料
前述の通り、各要素技術の詳細な解説は放棄していますが...こんなニッチな記事を読む人向けなので、まあね？
- EVPN 標準
- RFC7432 (BGP MPLS-Based Ethernet VPN)
- Alcatel-Lucent 発表資料 | ETHERNET VPN Standardization and Status
- 2015/11 時点のもの
- スライド9あたりが標準化動向の最新に近いかな...
- VXLAN 標準
- JUNOS での VXLAN+EVPN 実装・設定周り
- Juniper公式 | ジュニパーネットワークス EVPN 次世代データセンター アーキテクチャの実装
- MX と QFX5100 で EVPN の設定サンプルを step by step で解説
- 日本語版
- Juniper公式 | Junos OS for the QFX Series, Release 15.1X53 (for QFX10000 Switches) / EVPN Control Plane and VXLAN Data Plane Feature Guide for QFX Series Switches
- vQFX を動かす
15.1X53での EVPN + VxLAN 全般
- vQFX を動かす
- Juniper公式 github | JNPRAutomate/ansible-junos-evpn-vxlan
- MX, QFX5100, QFX10000 の EVPN/VXLAN サンプルコンフィグを含む
- Juniper公式 | ジュニパーネットワークス EVPN 次世代データセンター アーキテクチャの実装
- O'Reilly Juniper QFX10000 Series
- Juniper QFX10000 の独自プロセッサ開発経緯や特徴・Architecture(HW/SW) の結構詳しい説明・MPLS/VXLAN+EVPNの設計ポイント解説など、非常に良い本
Juniper QFX10000 Series: A Comprehensive Guide to Building Next-Generation Data Centers
- 作者: Douglas Richard, Jr. Hanks
- 出版社/メーカー: O'Reilly Media
- 発売日: 2016/08/11
- メディア: ペーパーバック
- この商品を含むブログを見る
- Juniper QFX10000 の独自プロセッサ開発経緯や特徴・Architecture(HW/SW) の結構詳しい説明・MPLS/VXLAN+EVPNの設計ポイント解説など、非常に良い本
構築～動作確認
GNS3 で仮想マシンのデプロイ ～ 結線 ～ 起動
前回の通りに、3ペアの vQFX をデプロイして接続していきます。 以下の感じで。
torSW101a と torSW201a というのは、GNS3 の Ethernet Switch を使っていますが、各環境に合わせて適当な dot1Q 食えるスイッチ置けば良いです。
設定はそれぞれ以下の感じです。(今回は Port 3-4 は使いませんが)
node11 と node21 は、インターフェースに IP アドレス付与するだけなので、適当な疎通確認用ノードなので好きなのをどうぞ。(node12 と node22 は今回使わないです)
あと captureSW というのは GNS3 1.5.2 では github gns3-gui issues | QEMU link Packet Captures の通り、qemu 同士の結線をパケットキャプチャできないので、「bb01 でポートミラーリング」する手法をとっているために置いているものです。
未試行ですが、RE 同士の結線をしている箇所(bb01 と spine[12]1)に全て Ethernet Switch を挟めば、ポートミラーリング不要でパケットキャプチャできると思います。(そちらの方がやりやすい筈)
できたら起動して待ちます。
基本設定
各環境に合わせて syslog なり NTP なり ssh key 登録なりしておいて下さい。
あ、今回は BGP のログを /var/log/bgp.log に残すために、以下をやっておくと良いですよ。
set protocols bgp traceoptions file bgp.log set protocols bgp traceoptions file size 10k set protocols bgp traceoptions file files 30 set protocols bgp traceoptions flag normal
Underlay 設定～確認 (物理IF と eBGP)
まずは物理 Interface と eBGP 周りを設定して、各 lo0 の IP アドレスを経路交換できたことを確認します。
まあ、長々となってしまいますが、絵にすると単純で以下のような感じです。
物理IF設定
bb01
set interfaces xe-0/0/0 description "DEV=spine11 IF=xe-0/0/0" set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.1/30 delete interfaces xe-0/0/0 unit 0 family inet dhcp set interfaces xe-0/0/1 description "DEV=spine21 IF=xe-0/0/0" set interfaces xe-0/0/1 unit 0 family inet address 192.0.2.5/30 delete interfaces xe-0/0/1 unit 0 family inet dhcp set protocols lldp port-id-subtype interface-name set protocols lldp interface xe-0/0/0 set protocols lldp interface xe-0/0/1
spine11
set interfaces xe-0/0/0 description "DEV=bb01 IF=xe-0/0/0" set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.2/30 delete interfaces xe-0/0/0 unit 0 family inet dhcp set protocols lldp port-id-subtype interface-name set protocols lldp interface xe-0/0/0
spine21
set interfaces xe-0/0/0 description "DEV=bb01 IF=xe-0/0/1" set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.6/30 delete interfaces xe-0/0/0 unit 0 family inet dhcp set protocols lldp port-id-subtype interface-name set protocols lldp interface xe-0/0/0
物理IF疎通確認
リソースをケチったからか RTT 長すぎですね...。
bb01
kotetsu@bb01> show lldp neighbors
Local Interface Parent Interface Chassis Id Port info System Name
xe-0/0/0 - 02:05:86:71:84:00 DEV=bb01 IF=xe-0/0/0 spine11
xe-0/0/1 - 02:05:86:71:ff:00 DEV=bb01 IF=xe-0/0/1 spine21
kotetsu@bb01> show route
inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.0.0.0/24 *[Direct/0] 19:47:39
> via em0.0
10.0.0.191/32 *[Local/0] 19:47:39
Local via em0.0
169.254.0.0/24 *[Direct/0] 19:57:21
> via em1.0
169.254.0.2/32 *[Local/0] 19:57:21
Local via em1.0
192.0.2.0/30 *[Direct/0] 00:17:12
> via xe-0/0/0.0
192.0.2.1/32 *[Local/0] 00:17:12
Local via xe-0/0/0.0
192.0.2.4/30 *[Direct/0] 00:07:04
> via xe-0/0/1.0
192.0.2.5/32 *[Local/0] 00:07:04
Local via xe-0/0/1.0
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
fe80::286:2b0f:fc44:ab00/128
*[Direct/0] 19:56:54
> via lo0.0
{master:0}
kotetsu@bb01> ping 192.0.2.2
PING 192.0.2.2 (192.0.2.2): 56 data bytes
64 bytes from 192.0.2.2: icmp_seq=1 ttl=64 time=2420.112 ms
64 bytes from 192.0.2.2: icmp_seq=2 ttl=64 time=1027.245 ms
64 bytes from 192.0.2.2: icmp_seq=3 ttl=64 time=1525.667 ms
^C
--- 192.0.2.2 ping statistics ---
5 packets transmitted, 3 packets received, 40% packet loss
round-trip min/avg/max/stddev = 1027.245/1657.675/2420.112/576.246 ms
{master:0}
kotetsu@bb01> ping 192.0.2.6
PING 192.0.2.6 (192.0.2.6): 56 data bytes
64 bytes from 192.0.2.6: icmp_seq=0 ttl=64 time=3378.582 ms
64 bytes from 192.0.2.6: icmp_seq=1 ttl=64 time=1374.159 ms
64 bytes from 192.0.2.6: icmp_seq=2 ttl=64 time=1474.743 ms
^C
--- 192.0.2.6 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1374.159/2075.828/3378.582/922.101 ms
spine11
{master:0}
kotetsu@spine11> show lldp neighbors
Local Interface Parent Interface Chassis Id Port info System Name
xe-0/0/0 - 02:05:86:71:55:00 DEV=spine11 IF=xe-0/0/0 bb01
{master:0}
kotetsu@spine11> show route
inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.0.0.0/24 *[Direct/0] 19:23:06
> via em0.0
10.0.0.201/32 *[Local/0] 19:23:06
Local via em0.0
169.254.0.0/24 *[Direct/0] 19:22:26
> via em1.0
169.254.0.2/32 *[Local/0] 19:22:26
Local via em1.0
192.0.2.0/30 *[Direct/0] 00:16:05
> via xe-0/0/0.0
192.0.2.2/32 *[Local/0] 00:16:05
Local via xe-0/0/0.0
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
fe80::286:2b0f:fca1:f500/128
*[Direct/0] 19:22:25
> via lo0.0
{master:0}
kotetsu@spine11> ping 192.0.2.1
PING 192.0.2.1 (192.0.2.1): 56 data bytes
64 bytes from 192.0.2.1: icmp_seq=0 ttl=64 time=2839.663 ms
64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=2463.433 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=2269.077 ms
64 bytes from 192.0.2.1: icmp_seq=3 ttl=64 time=1548.765 ms
64 bytes from 192.0.2.1: icmp_seq=4 ttl=64 time=844.779 ms
64 bytes from 192.0.2.1: icmp_seq=5 ttl=64 time=505.582 ms
64 bytes from 192.0.2.1: icmp_seq=6 ttl=64 time=852.323 ms
64 bytes from 192.0.2.1: icmp_seq=7 ttl=64 time=1724.594 ms
^C
--- 192.0.2.1 ping statistics ---
9 packets transmitted, 8 packets received, 11% packet loss
round-trip min/avg/max/stddev = 505.582/1631.027/2839.663/795.890 ms
spine21
{master:0}
kotetsu@spine21> show lldp neighbors
Local Interface Parent Interface Chassis Id Port info System Name
xe-0/0/0 - 02:05:86:71:55:00 DEV=spine21 IF=xe-0/0/0 bb01
{master:0}
kotetsu@spine21> show route
inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.0.0.0/24 *[Direct/0] 08:41:29
> via em0.0
10.0.0.202/32 *[Local/0] 08:41:30
Local via em0.0
169.254.0.0/24 *[Direct/0] 08:46:47
> via em1.0
169.254.0.2/32 *[Local/0] 08:46:47
Local via em1.0
192.0.2.4/30 *[Direct/0] 00:03:53
> via xe-0/0/0.0
192.0.2.6/32 *[Local/0] 00:03:53
Local via xe-0/0/0.0
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
fe80::286:2b0f:fcd0:9b00/128
*[Direct/0] 08:46:47
> via lo0.0
{master:0}
kotetsu@spine21> ping 192.0.2.5
PING 192.0.2.5 (192.0.2.5): 56 data bytes
64 bytes from 192.0.2.5: icmp_seq=0 ttl=64 time=427.225 ms
64 bytes from 192.0.2.5: icmp_seq=1 ttl=64 time=709.372 ms
64 bytes from 192.0.2.5: icmp_seq=2 ttl=64 time=1231.701 ms
64 bytes from 192.0.2.5: icmp_seq=3 ttl=64 time=732.131 ms
^C
--- 192.0.2.5 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 427.225/775.107/1231.701/289.684 ms
lo0 + eBGP 設定
今回、いずれもシングル構成なのでマルチパス関係の設定は入れていませんよ。
bb01
set interfaces lo0 unit 0 family inet address 172.31.0.1/32 set policy-options policy-statement POLICY_EXPORT_LO0 from family inet set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement POLICY_EXPORT_LO0 then accept set routing-options router-id 172.31.0.1 set routing-options autonomous-system 65000 set protocols bgp group BGP_UNDERLAY type external set protocols bgp group BGP_UNDERLAY advertise-peer-as set protocols bgp group BGP_UNDERLAY family inet unicast loops 2 set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.2 description spine11 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.2 peer-as 65001 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.6 description spine21 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.6 peer-as 65002
spine11
set interfaces lo0 unit 0 family inet address 172.16.1.1/32 set policy-options policy-statement POLICY_EXPORT_LO0 from family inet set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement POLICY_EXPORT_LO0 then accept set routing-options router-id 172.16.1.1 set routing-options autonomous-system 65001 set protocols bgp group BGP_UNDERLAY type external set protocols bgp group BGP_UNDERLAY advertise-peer-as set protocols bgp group BGP_UNDERLAY family inet unicast loops 2 set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.1 description bb00 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.1 peer-as 65000
spine21
set interfaces lo0 unit 0 family inet address 172.16.2.1/32 set policy-options policy-statement POLICY_EXPORT_LO0 from family inet set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement POLICY_EXPORT_LO0 then accept set routing-options router-id 172.16.2.1 set routing-options autonomous-system 65002 set protocols bgp group BGP_UNDERLAY type external set protocols bgp group BGP_UNDERLAY advertise-peer-as set protocols bgp group BGP_UNDERLAY family inet unicast loops 2 set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.5 description bb00 set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.5 peer-as 65000
eBGP確認
それぞれ lo0 の IPアドレスを相互学習したことを確認します。
この後やる Overlay 用の MP-BGP (iBGP) で bb01 が Route Reflector になるので、実際には bb01 と spine[12]1 で交換できていれば十分な筈ですが。
bb01
{master:0}
kotetsu@bb01> show bgp summary
Groups: 1 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.2 65001 4 4 0 0 20 1/1/1/0 0/0/0/0
192.0.2.6 65002 1 2 0 0 5:57 OpenConfirm
{master:0}
kotetsu@bb01> show bgp summary
Groups: 1 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.2 65001 4 5 0 0 25 1/1/1/0 0/0/0/0
192.0.2.6 65002 4 4 0 0 4 1/1/1/0 0/0/0/0
{master:0}
kotetsu@bb01>
{master:0}
kotetsu@bb01> show bgp summary
Groups: 1 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.2 65001 4 5 0 0 26 1/1/1/0 0/0/0/0
192.0.2.6 65002 4 4 0 0 5 1/1/1/0 0/0/0/0
{master:0}
kotetsu@bb01> show bgp group BGP_UNDERLAY brief
Group Type: External Local AS: 65000
Name: BGP_UNDERLAY Index: 0 Flags: <Export Eval>
Export: [ POLICY_EXPORT_LO0 ]
Options: <AdvertisePeerAs>
Holdtime: 0
Total peers: 2 Established: 2
192.0.2.2+56114
192.0.2.6+49934
inet.0: 2/2/2/0
{master:0}
kotetsu@bb01> show bgp neighbor
Peer: 192.0.2.2+56114 AS 65001 Local: 192.0.2.1+179 AS 65000
Description: spine11
Group: BGP_UNDERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ POLICY_EXPORT_LO0 ]
Options: <Preference AddressFamily PeerAS Refresh>
Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
Address families configured: inet-unicast
Holdtime: 90 Preference: 170
Number of flaps: 0
Peer ID: 172.16.1.1 Local ID: 172.31.0.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 0 Peer index: 0
BFD: disabled, down
Local Interface: xe-0/0/0.0
NLRI for restart configured on peer: inet-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 65001)
Peer does not support Addpath
Table inet.0 Bit: 10000
RIB State: BGP restart is complete
Send state: in sync
Active prefixes: 1
Received prefixes: 1
Accepted prefixes: 1
Suppressed due to damping: 0
Advertised prefixes: 2
Last traffic (seconds): Received 23 Sent 5 Checked 56
Input messages: Total 179 Updates 2 Refreshes 0 Octets 3478
Output messages: Total 186 Updates 2 Refreshes 0 Octets 3663
Output Queue[0]: 0 (inet.0, inet-unicast)
Peer: 192.0.2.6+49934 AS 65002 Local: 192.0.2.5+179 AS 65000
Description: spine21
Group: BGP_UNDERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ POLICY_EXPORT_LO0 ]
Options: <Preference AddressFamily PeerAS Refresh>
Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
Address families configured: inet-unicast
Holdtime: 90 Preference: 170
Number of flaps: 0
Peer ID: 172.16.2.1 Local ID: 172.31.0.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 0 Peer index: 1
BFD: disabled, down
Local Interface: xe-0/0/1.0
NLRI for restart configured on peer: inet-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 65002)
Peer does not support Addpath
Table inet.0 Bit: 10000
RIB State: BGP restart is complete
Send state: in sync
Active prefixes: 1
Received prefixes: 1
Accepted prefixes: 1
Suppressed due to damping: 0
Advertised prefixes: 2
Last traffic (seconds): Received 54 Sent 19 Checked 15
Input messages: Total 92 Updates 2 Refreshes 0 Octets 1825
Output messages: Total 184 Updates 2 Refreshes 0 Octets 3625
Output Queue[0]: 0 (inet.0, inet-unicast)
{master:0}
kotetsu@bb01> show route protocol bgp
inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
172.16.1.1/32 *[BGP/170] 00:02:11, localpref 100
AS path: 65001 I, validation-state: unverified
> to 192.0.2.2 via xe-0/0/0.0
172.16.2.1/32 *[BGP/170] 00:01:49, localpref 100
AS path: 65002 I, validation-state: unverified
> to 192.0.2.6 via xe-0/0/1.0
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
{master:0}
kotetsu@bb01> ping 172.16.2.1
PING 172.16.2.1 (172.16.2.1): 56 data bytes
64 bytes from 172.16.2.1: icmp_seq=0 ttl=64 time=908.929 ms
64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=1249.259 ms
^C
--- 172.16.2.1 ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max/stddev = 908.929/1079.094/1249.259/170.165 ms
{master:0}
kotetsu@bb01> ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1): 56 data bytes
64 bytes from 172.16.1.1: icmp_seq=0 ttl=64 time=2595.413 ms
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=1406.501 ms
^C
--- 172.16.1.1 ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max/stddev = 1406.501/2000.957/2595.413/594.456 ms
spine11
{master:0}
kotetsu@spine11> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.1 65000 174 169 0 0 1:16:34 2/2/2/0 0/0/0/0
{master:0}
kotetsu@spine11> show bgp group BGP_UNDERLAY brief
Group Type: External Local AS: 65001
Name: BGP_UNDERLAY Index: 0 Flags: <Export Eval>
Export: [ POLICY_EXPORT_LO0 ]
Options: <AdvertisePeerAs>
Holdtime: 0
Total peers: 1 Established: 1
192.0.2.1+179
inet.0: 2/2/2/0
{master:0}
kotetsu@spine11> show bgp neighbor
Peer: 192.0.2.1+179 AS 65000 Local: 192.0.2.2+56114 AS 65001
Description: bb00
Group: BGP_UNDERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ POLICY_EXPORT_LO0 ]
Options: <Preference AddressFamily PeerAS Refresh>
Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
Address families configured: inet-unicast
Holdtime: 90 Preference: 170
Number of flaps: 0
Peer ID: 172.31.0.1 Local ID: 172.16.1.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 0 Peer index: 0
BFD: disabled, down
Local Interface: xe-0/0/0.0
NLRI for restart configured on peer: inet-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 65000)
Peer does not support Addpath
Table inet.0 Bit: 10000
RIB State: BGP restart is complete
Send state: in sync
Active prefixes: 2
Received prefixes: 2
Accepted prefixes: 2
Suppressed due to damping: 0
Advertised prefixes: 1
Last traffic (seconds): Received 9 Sent 7 Checked 50
Input messages: Total 182 Updates 3 Refreshes 0 Octets 3524
Output messages: Total 177 Updates 1 Refreshes 0 Octets 3459
Output Queue[0]: 0 (inet.0, inet-unicast)
{master:0}
kotetsu@spine11> show route protocol bgp
inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
172.16.2.1/32 *[BGP/170] 01:22:49, localpref 100
AS path: 65000 65002 I, validation-state: unverified
> to 192.0.2.1 via xe-0/0/0.0
172.31.0.1/32 *[BGP/170] 01:23:10, localpref 100
AS path: 65000 I, validation-state: unverified
> to 192.0.2.1 via xe-0/0/0.0
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
{master:0}
kotetsu@spine11> ping 172.31.0.1
PING 172.31.0.1 (172.31.0.1): 56 data bytes
64 bytes from 172.31.0.1: icmp_seq=0 ttl=64 time=1597.291 ms
64 bytes from 172.31.0.1: icmp_seq=1 ttl=64 time=1215.306 ms
^C
--- 172.31.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1215.306/1406.299/1597.291/190.993 ms
spine21
{master:0}
kotetsu@spine21> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.5 65000 173 88 0 0 38:05 2/2/2/0 0/0/0/0
{master:0}
kotetsu@spine21> show bgp group BGP_UNDERLAY brief
Group Type: External Local AS: 65002
Name: BGP_UNDERLAY Index: 0 Flags: <Export Eval>
Export: [ POLICY_EXPORT_LO0 ]
Options: <AdvertisePeerAs>
Holdtime: 0
Total peers: 1 Established: 1
192.0.2.5+179
inet.0: 2/2/2/0
kotetsu@spine21> show bgp neighbor
Peer: 192.0.2.5+179 AS 65000 Local: 192.0.2.6+49934 AS 65002
Description: bb00
Group: BGP_UNDERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ POLICY_EXPORT_LO0 ]
Options: <Preference AddressFamily PeerAS Refresh>
Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
Address families configured: inet-unicast
Holdtime: 90 Preference: 170
Number of flaps: 0
Peer ID: 172.31.0.1 Local ID: 172.16.2.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 0 Peer index: 0
BFD: disabled, down
Local Interface: xe-0/0/0.0
NLRI for restart configured on peer: inet-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 65000)
Peer does not support Addpath
Table inet.0 Bit: 10000
RIB State: BGP restart is complete
Send state: in sync
Active prefixes: 2
Received prefixes: 2
Accepted prefixes: 2
Suppressed due to damping: 0
Advertised prefixes: 1
Last traffic (seconds): Received 1 Sent 24 Checked 3
Input messages: Total 175 Updates 3 Refreshes 0 Octets 3391
Output messages: Total 88 Updates 1 Refreshes 0 Octets 1768
Output Queue[0]: 0 (inet.0, inet-unicast)
{master:0}
kotetsu@spine21> show route protocol bgp
inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
172.16.1.1/32 *[BGP/170] 00:41:43, localpref 100
AS path: 65000 65001 I, validation-state: unverified
> to 192.0.2.5 via xe-0/0/0.0
172.31.0.1/32 *[BGP/170] 00:41:43, localpref 100
AS path: 65000 I, validation-state: unverified
> to 192.0.2.5 via xe-0/0/0.0
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
{master:0}
kotetsu@spine21> ping 172.31.0.1
PING 172.31.0.1 (172.31.0.1): 56 data bytes
64 bytes from 172.31.0.1: icmp_seq=0 ttl=64 time=691.327 ms
64 bytes from 172.31.0.1: icmp_seq=1 ttl=64 time=1227.117 ms
64 bytes from 172.31.0.1: icmp_seq=2 ttl=64 time=238.494 ms
^C
--- 172.31.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 238.494/718.979/1227.117/404.077 ms
Overlay 設定～確認 (MP-BGP)
Underlay eBGP で経路交換した各 lo0 同士で iBGP を組みます。
まあ、長々となってしまいますが、絵にすると単純で以下のような感じです。
iBGP (MP-BGP )設定
ポイントは1つだけで family evpn signaling を設定して EVPN の NLRI を扱うってとこですかね。
bb01
set protocols bgp group BGP_OVERLAY type internal set protocols bgp group BGP_OVERLAY local-address 172.31.0.1 set protocols bgp group BGP_OVERLAY family evpn signaling set protocols bgp group BGP_OVERLAY cluster 172.31.0.1 set protocols bgp group BGP_OVERLAY local-as 64512 set protocols bgp group BGP_OVERLAY neighbor 172.16.1.1 description spine11 set protocols bgp group BGP_OVERLAY neighbor 172.16.2.1 description spine21
spine11
set protocols bgp group BGP_OVERLAY type internal set protocols bgp group BGP_OVERLAY local-address 172.16.1.1 set protocols bgp group BGP_OVERLAY family evpn signaling set protocols bgp group BGP_OVERLAY local-as 64512 set protocols bgp group BGP_OVERLAY neighbor 172.31.0.1 description bb01
spine21
set protocols bgp group BGP_OVERLAY type internal set protocols bgp group BGP_OVERLAY local-address 172.16.2.1 set protocols bgp group BGP_OVERLAY family evpn signaling set protocols bgp group BGP_OVERLAY local-as 64512 set protocols bgp group BGP_OVERLAY neighbor 172.31.0.1 description bb01
iBGP (MP-BGP) 確認
最初 neighbor IP アドレス設定をタイポしたせいで、Last Error: Open Message Error とか残っているのはご愛敬ということでひとつ...。
bb01
{master:0}
kotetsu@bb01> show bgp summary
Groups: 2 Peers: 4 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 64512 3 3 0 0 25 Establ
bgp.evpn.0: 0/0/0/0
172.16.2.1 64512 3 2 0 0 12 Establ
bgp.evpn.0: 0/0/0/0
192.0.2.2 65001 502 518 0 0 3:58:25 1/1/1/0 0/0/0/0
192.0.2.6 65002 257 517 0 0 3:58:04 1/1/1/0 0/0/0/0
{master:0}
kotetsu@bb01> show bgp group BGP_OVERLAY
Group Type: Internal AS: 64512 Local AS: 64512
Name: BGP_OVERLAY Index: 1 Flags: <Export Eval>
Options: <Cluster LocalAS>
Holdtime: 0 Local AS: 64512 Local System AS: 65000
Total peers: 2 Established: 2
172.16.1.1+52050
172.16.2.1+62794
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
bgp.evpn.0: 0/0/0/0
{master:0}
kotetsu@bb01> show bgp neighbor
Peer: 172.16.1.1+52050 AS 64512 Local: 172.31.0.1+179 AS 64512
Description: spine11
Group: BGP_OVERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Established (route reflector client)Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: Open Message Error
Options: <Preference LocalAddress Cluster AddressFamily LocalAS Rib-group Refresh>
Address families configured: evpn
Local Address: 172.31.0.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65000
Number of flaps: 0
Error: 'Open Message Error' Sent: 20 Recv: 0
Peer ID: 172.16.1.1 Local ID: 172.31.0.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 1 Peer index: 0
BFD: disabled, down
NLRI for restart configured on peer: evpn
NLRI advertised by peer: evpn
NLRI for this session: evpn
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: evpn
NLRI of received end-of-rib markers: evpn
NLRI of all end-of-rib markers sent: evpn
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 64512)
Peer does not support Addpath
Table bgp.evpn.0 Bit: 20000
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Advertised prefixes: 0
Last traffic (seconds): Received 1 Sent 9 Checked 0
Input messages: Total 6 Updates 1 Refreshes 0 Octets 169
Output messages: Total 6 Updates 0 Refreshes 0 Octets 188
Output Queue[1]: 0 (bgp.evpn.0, evpn)
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
Peer: 172.16.2.1+62794 AS 64512 Local: 172.31.0.1+179 AS 64512
Description: spine21
Group: BGP_OVERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Established (route reflector client)Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: Open Message Error
Options: <Preference LocalAddress Cluster AddressFamily LocalAS Rib-group Refresh>
Address families configured: evpn
Local Address: 172.31.0.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65000
Number of flaps: 0
Error: 'Open Message Error' Sent: 20 Recv: 0
Peer ID: 172.16.2.1 Local ID: 172.31.0.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 1 Peer index: 1
BFD: disabled, down
NLRI for restart configured on peer: evpn
NLRI advertised by peer: evpn
NLRI for this session: evpn
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: evpn
NLRI of received end-of-rib markers: evpn
NLRI of all end-of-rib markers sent: evpn
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 64512)
Peer does not support Addpath
Table bgp.evpn.0 Bit: 20000
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Advertised prefixes: 0
Last traffic (seconds): Received 46 Sent 0 Checked 9
Input messages: Total 10 Updates 1 Refreshes 0 Octets 245
Output messages: Total 18 Updates 0 Refreshes 0 Octets 416
Output Queue[1]: 0 (bgp.evpn.0, evpn)
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
...
spine11
{master:0}
kotetsu@spine11> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.31.0.1 64512 21 22 0 0 9:07 Establ
bgp.evpn.0: 0/0/0/0
192.0.2.1 65000 536 521 0 0 3:59:55 2/2/2/0 0/0/0/
{master:0}
kotetsu@spine11> show bgp group BGP_OVERLAY
Group Type: Internal AS: 64512 Local AS: 64512
Name: BGP_OVERLAY Index: 1 Flags: <Export Eval>
Options: <LocalAS>
Holdtime: 0 Local AS: 64512 Local System AS: 65001
Total peers: 1 Established: 1
172.31.0.1+179
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
bgp.evpn.0: 0/0/0/0
{master:0}
kotetsu@spine11> show bgp neighbor
Peer: 172.31.0.1+179 AS 64512 Local: 172.16.1.1+52050 AS 64512
Description: bb01
Group: BGP_OVERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Options: <Preference LocalAddress AddressFamily LocalAS Rib-group Refresh>
Address families configured: evpn
Local Address: 172.16.1.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65001
Number of flaps: 0
Peer ID: 172.31.0.1 Local ID: 172.16.1.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 1 Peer index: 1
BFD: disabled, down
NLRI for restart configured on peer: evpn
NLRI advertised by peer: evpn
NLRI for this session: evpn
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: evpn
NLRI of received end-of-rib markers: evpn
NLRI of all end-of-rib markers sent: evpn
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 64512)
Peer does not support Addpath
Table bgp.evpn.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Last traffic (seconds): Received 1 Sent 8 Checked 46
Input messages: Total 34 Updates 1 Refreshes 0 Octets 657
Output messages: Total 34 Updates 0 Refreshes 0 Octets 720
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
...
spine21
{master:0}
kotetsu@spine21> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.31.0.1 64512 34 19 0 0 7:30 Establ
bgp.evpn.0: 0/0/0/0
192.0.2.5 65000 549 273 0 0 2:02:08 2/2/2/0 0/0/0/0
{master:0}
kotetsu@spine21> show bgp group BGP_OVERLAY
Group Type: Internal AS: 64512 Local AS: 64512
Name: BGP_OVERLAY Index: 1 Flags: <Export Eval>
Options: <LocalAS>
Holdtime: 0 Local AS: 64512 Local System AS: 65002
Total peers: 1 Established: 1
172.31.0.1+179
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
bgp.evpn.0: 0/0/0/0
{master:0}
kotetsu@spine21> show bgp neighbor
Peer: 172.31.0.1+179 AS 64512 Local: 172.16.2.1+62794 AS 64512
Description: bb01
Group: BGP_OVERLAY Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Options: <Preference LocalAddress AddressFamily LocalAS Rib-group Refresh>
Address families configured: evpn
Local Address: 172.16.2.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65002
Number of flaps: 0
Peer ID: 172.31.0.1 Local ID: 172.16.2.1 Active Holdtime: 90
Keepalive Interval: 30 Group index: 1 Peer index: 1
BFD: disabled, down
NLRI for restart configured on peer: evpn
NLRI advertised by peer: evpn
NLRI for this session: evpn
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: evpn
NLRI of received end-of-rib markers: evpn
NLRI of all end-of-rib markers sent: evpn
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 64512)
Peer does not support Addpath
Table bgp.evpn.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Last traffic (seconds): Received 14 Sent 17 Checked 58
Input messages: Total 36 Updates 1 Refreshes 0 Octets 695
Output messages: Total 20 Updates 0 Refreshes 0 Octets 454
Trace options: normal
Trace file: /var/log/bgp.log size 10240 files 30
...
Overlay設定～確認(EVPN+VXLAN)
EVPN の設定と、その Dataplane として使う VXLAN 周りの設定をしていきます。(息切れ)
EVPN+VXLAN 設定
冒頭の全体構成に書いたのですが bb01 はこの辺の挙動に関しては土管に徹しているので登場しません。
VNI は 1..16777214 まで使えるので、単純に全 spine で VLAN ID と 1:1 で同じ対応付けをさせて、一律で VLAN ID + 10000 の値を振ってます。
あ、VNI 10100 以外は本項ではまだ使いません。
spine11
set vlans VLAN0100 vlan-id 100 set vlans VLAN0100 vxlan vni 10100 set vlans VLAN0100 vxlan ingress-node-replication set vlans VLAN0300 vlan-id 300 set vlans VLAN0300 vxlan vni 10300 set vlans VLAN0300 vxlan ingress-node-replication set protocols evpn encapsulation vxlan set protocols evpn extended-vni-list all set protocols evpn multicast-mode ingress-replication set protocols evpn vni-options vni 10100 vrf-target export target:1:10100 set protocols evpn vni-options vni 10300 vrf-target export target:1:10300 set policy-options community COM_10100 members target:1:10100 set policy-options community COM_10300 members target:1:10300 set policy-options community COM_LEAF_ESI members target:9999:9999 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 from community COM_10100 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_10300 from community COM_10300 set policy-options policy-statement POLICY_VRF_IMPORT term T_10300 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 from community COM_LEAF_ESI set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99999 then reject set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 64512:11 set switch-options vrf-import POLICY_VRF_IMPORT set switch-options vrf-target target:9999:9999 set switch-options vrf-target auto set interfaces xe-0/0/1 description "DEV=torSW11 IF=1" set interfaces xe-0/0/1 unit 0 family ethernet-switching interface-mode trunk set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members all
spine21
set vlans VLAN0100 vlan-id 100 set vlans VLAN0100 vxlan vni 10100 set vlans VLAN0100 vxlan ingress-node-replication set vlans VLAN0200 vlan-id 200 set vlans VLAN0200 vxlan vni 10200 set vlans VLAN0200 vxlan ingress-node-replication set protocols evpn encapsulation vxlan set protocols evpn extended-vni-list all set protocols evpn multicast-mode ingress-replication set protocols evpn vni-options vni 10100 vrf-target export target:1:10100 set protocols evpn vni-options vni 10200 vrf-target export target:1:10200 set policy-options community COM_10100 members target:1:10100 set policy-options community COM_10200 members target:1:10200 set policy-options community COM_LEAF_ESI members target:9999:9999 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 from community COM_10100 set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_10200 from community COM_10200 set policy-options policy-statement POLICY_VRF_IMPORT term T_10200 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 from community COM_LEAF_ESI set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 then accept set policy-options policy-statement POLICY_VRF_IMPORT term T_99999 then reject set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 64512:21 set switch-options vrf-import POLICY_VRF_IMPORT set switch-options vrf-target target:9999:9999 set switch-options vrf-target auto set interfaces xe-0/0/1 description "DEV=torSW21 IF=1" set interfaces xe-0/0/1 unit 0 family ethernet-switching interface-mode trunk set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members all
動作確認
node 同士の疎通確認
- node11
$ ip a show dev ens4
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:86:2b:8d:10:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global ens4
valid_lft forever preferred_lft forever
inet6 fe80::286:2bff:fe8d:1001/64 scope link
valid_lft forever preferred_lft forever
$ ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=975 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=949 ms
$ ip n show dev ens4
192.168.1.2 lladdr 00:86:2b:5c:0d:01 STALE
- node21
$ ip a show dev ens4
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:86:2b:5c:0d:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.1.255 scope global ens4
valid_lft forever preferred_lft forever
inet6 fe80::286:2bff:fe5c:d01/64 scope link
valid_lft forever preferred_lft forever
$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=639 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=926 ms
$ ip n show dev ens4
192.168.1.1 lladdr 00:86:2b:8d:10:01 STALE
vQFX のテーブル確認
前述の Juniper公式日本語資料 のページ 36 によると
bgp.evpn.0 = Junos OS ルーティング プロトコル プロセス（RPD）内のグローバル EVPN ルーティング テーブル default.switch.evpn.0（QFX5100 の場合） = Junos OS RPD 内のスイッチ レベル EVPN 転送テーブル \<virtual-switch-name>.evpn.0（MX の場合） = Junos OS RPD 内の仮想スイッチ レベル EVPN 転送テーブル
らしいですよ。
spine11
こいつにとっての vtep.32769 てのは spine21 (の VTEP)
{master:0}
kotetsu@spine11> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Logical Active
name address flags interface source
VLAN0100 00:86:2b:5c:0d:01 D vtep.32769 172.16.2.1
VLAN0100 00:86:2b:8d:10:01 D xe-0/0/1.0
{master:0}
kotetsu@spine11> show interfaces vtep.32769
Logical interface vtep.32769 (Index 566) (SNMP ifIndex 537)
Flags: Up SNMP-Traps Encapsulation: ENET2
VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 172.16.2.1, L2 Routing Instance: default-switch, L3 Routing Instance: default
Input packets : 4
Output packets: 4
Protocol eth-switch, MTU: Unlimited
Flags: Trunk-Mode
EVPN の Type2(MAC/IP Advertisement route) と Type3(Inclusive Multicast Ethernet Tag route) を spine21 (RD 64512:21)から学習して bgp.evpn.0 から default-switch.evpn.0 にロードされている様子
{master:0}
kotetsu@spine11> show route
...
:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.0.0.0/24 *[Direct/0] 00:23:24
> via em0.0
10.0.0.201/32 *[Local/0] 00:06:15
Local via em0.0
169.254.0.0/24 *[Direct/0] 00:23:24
> via em1.0
169.254.0.2/32 *[Local/0] 00:23:24
Local via em1.0
172.16.1.1/32 *[Direct/0] 00:23:24
> via lo0.0
172.16.2.1/32 *[Static/1] 00:13:22, metric2 0
> to 192.0.2.1 via xe-0/0/0.0
192.0.2.0/30 *[Direct/0] 00:23:24
> via xe-0/0/0.0
192.0.2.2/32 *[Local/0] 00:23:24
Local via xe-0/0/0.0
bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:64512:21::10100::00:86:2b:5c:0d:01/304
*[BGP/170] 00:08:29, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.1 via xe-0/0/0.0
3:64512:21::10100::172.16.2.1/304
*[BGP/170] 00:13:23, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.1 via xe-0/0/0.0
default-switch.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:64512:11::10100::00:86:2b:8d:10:01/304
*[EVPN/170] 00:08:33
Indirect
2:64512:21::10100::00:86:2b:5c:0d:01/304
*[BGP/170] 00:08:29, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.1 via xe-0/0/0.0
3:64512:11::10100::172.16.1.1/304
*[EVPN/170] 00:23:27
Indirect
3:64512:11::10300::172.16.1.1/304
*[EVPN/170] 00:15:11
Indirect
3:64512:21::10100::172.16.2.1/304
*[BGP/170] 00:13:23, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.1 via xe-0/0/0.0
{master:0}
kotetsu@spine11> show route forwarding-table
...
Routing table: :vxlan.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 rjct 1724 1
0.0.0.0/32 perm 0 dscd 1722 1
169.254.0.0/24 user 0 rtbl 1 5
169.254.0.2/32 user 0 169.254.0.2 locl 338 3
172.16.1.1/32 user 0 rtbl 1 5
172.16.2.1/32 user 0 indr 131070 3
192.0.2.1 ucst 1719 7 xe-0/0/0.0
192.0.2.0/30 user 0 rtbl 1 5
192.0.2.2/32 user 0 192.0.2.2 locl 1717 3
224.0.0.0/4 perm 0 mdsc 1723 1
224.0.0.1/32 perm 0 224.0.0.1 mcst 1726 1
255.255.255.255/32 perm 0 bcst 1727 1
...
Routing table: default-switch.bridge
Bridging domain: VLAN0100.bridge
VPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
00:86:2b:5c:0d:01/48 user 0 comp 1707 6
00:86:2b:8d:10:01/48 user 0 ucst 1712 6 xe-0/0/1.0
0x30001/51 user 0 comp 1739 2
0x30006/51 user 0 comp 1732 2
0x30002/51 user 0 comp 1733 2
Routing table: default-switch.bridge
Bridging domain: VLAN0300.bridge
VPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
0x30007/51 user 0 comp 1741 2
0x30005/51 user 0 comp 1714 2
0x30003/51 user 0 comp 1734 2
...
spine21
こいつにとっての vtep.32769 てのは spine11 (の VTEP)
{master:0}
kotetsu@spine21> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Logical Active
name address flags interface source
VLAN0100 00:86:2b:5c:0d:01 D xe-0/0/1.0
VLAN0100 00:86:2b:8d:10:01 D vtep.32769 172.16.1.1
EVPN の Type2(MAC/IP Advertisement route) と Type3(Inclusive Multicast Ethernet Tag route) を spine11 (RD 64512:11)から学習して bgp.evpn.0 から default-switch.evpn.0 にロードされている様子
{master:0}
kotetsu@spine21> show route
...
:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.0.0.0/24 *[Direct/0] 00:18:41
> via em0.0
10.0.0.202/32 *[Local/0] 00:11:20
Local via em0.0
169.254.0.0/24 *[Direct/0] 00:18:41
> via em1.0
169.254.0.2/32 *[Local/0] 00:18:41
Local via em1.0
172.16.1.1/32 *[Static/1] 00:18:39, metric2 0
> to 192.0.2.5 via xe-0/0/0.0
172.16.2.1/32 *[Direct/0] 00:18:41
> via lo0.0
192.0.2.4/30 *[Direct/0] 00:18:41
> via xe-0/0/0.0
192.0.2.6/32 *[Local/0] 00:18:41
Local via xe-0/0/0.0
bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:64512:11::10100::00:86:2b:8d:10:01/304
*[BGP/170] 00:12:17, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.5 via xe-0/0/0.0
3:64512:11::10100::172.16.1.1/304
*[BGP/170] 00:18:40, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.5 via xe-0/0/0.0
default-switch.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:64512:11::10100::00:86:2b:8d:10:01/304
*[BGP/170] 00:12:17, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.5 via xe-0/0/0.0
2:64512:21::10100::00:86:2b:5c:0d:01/304
*[EVPN/170] 00:12:16
Indirect
3:64512:11::10100::172.16.1.1/304
*[BGP/170] 00:18:40, localpref 100, from 172.31.0.1
AS path: I, validation-state: unverified
> to 192.0.2.5 via xe-0/0/0.0
3:64512:21::10100::172.16.2.1/304
*[EVPN/170] 00:14:43
Indirect
3:64512:21::10200::172.16.2.1/304
*[EVPN/170] 00:14:43
Indirect
{master:0}
kotetsu@spine21> show route forwarding-table
...
Routing table: :vxlan.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 rjct 1716 1
0.0.0.0/32 perm 0 dscd 1714 1
169.254.0.0/24 user 0 rtbl 1 5
169.254.0.2/32 user 0 169.254.0.2 locl 334 3
172.16.1.1/32 user 0 indr 131070 3
192.0.2.5 ucst 1711 7 xe-0/0/0.0
172.16.2.1/32 user 0 rtbl 1 5
192.0.2.4/30 user 0 rtbl 1 5
192.0.2.6/32 user 0 192.0.2.6 locl 1709 3
224.0.0.0/4 perm 0 mdsc 1715 1
224.0.0.1/32 perm 0 224.0.0.1 mcst 1718 1
255.255.255.255/32 perm 0 bcst 1719 1
...
Routing table: default-switch.bridge
Bridging domain: VLAN0100.bridge
VPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
00:86:2b:5c:0d:01/48 user 0 ucst 1730 6 xe-0/0/1.0
00:86:2b:8d:10:01/48 user 0 comp 1723 6
0x30006/51 user 0 comp 1731 2
0x30004/51 user 0 comp 1725 2
0x30001/51 user 0 comp 1726 2
Routing table: default-switch.bridge
Bridging domain: VLAN0200.bridge
VPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
0x30007/51 user 0 comp 1734 2
0x30005/51 user 0 comp 1729 2
0x30002/51 user 0 comp 1728 2
bb01
{master:0}
kotetsu@bb01> show ethernet-switching table
単純に EVPN 経路を reflect する土管なので bgp.evpn.0 にしか載っていない様子
{master:0}
kotetsu@bb01> show route
...
bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:64512:11::10100::00:86:2b:8d:10:01/304
*[BGP/170] 00:31:03, localpref 100, from 172.16.1.1
AS path: I, validation-state: unverified
> to 192.0.2.2 via xe-0/0/0.0
2:64512:21::10100::00:86:2b:5c:0d:01/304
*[BGP/170] 00:31:01, localpref 100, from 172.16.2.1
AS path: I, validation-state: unverified
> to 192.0.2.6 via xe-0/0/1.0
3:64512:11::10100::172.16.1.1/304
*[BGP/170] 00:46:21, localpref 100, from 172.16.1.1
AS path: I, validation-state: unverified
> to 192.0.2.2 via xe-0/0/0.0
3:64512:11::10300::172.16.1.1/304
*[BGP/170] 00:37:52, localpref 100, from 172.16.1.1
AS path: I, validation-state: unverified
> to 192.0.2.2 via xe-0/0/0.0
3:64512:21::10100::172.16.2.1/304
*[BGP/170] 00:36:05, localpref 100, from 172.16.2.1
AS path: I, validation-state: unverified
> to 192.0.2.6 via xe-0/0/1.0
3:64512:21::10200::172.16.2.1/304
*[BGP/170] 00:35:45, localpref 100, from 172.16.2.1
AS path: I, validation-state: unverified
> to 192.0.2.6 via xe-0/0/1.0
{master:0}
kotetsu@bb01> show route forwarding-table
...
パケットキャプチャ
いくつか特徴的なパケットを拾って眺めていきます。
前述の通り、今回は RE 同士を接続するところで直結構成にしてしまい bb01 が xe-0/0/2 から captureSW 方面にポートミラーリングして、GNS3 の bb01(RE) と captureSW 間のリンクを右クリックして start capture とかやっていきます。
なので bb01 に以下のような設定をして、xe-0/0/0 (spine11 側)の出入りを xe-0/0/2 にミラーリングしときます。
set interfaces xe-0/0/2 description "DEV=captureSW IF=1" set interfaces xe-0/0/2 unit 0 family ethernet-switching set forwarding-options analyzer ANAL_PORT input ingress interface xe-0/0/0.0 set forwarding-options analyzer ANAL_PORT input egress interface xe-0/0/0.0 set forwarding-options analyzer ANAL_PORT output interface xe-0/0/2.0
見返していて思ったんすが、アナルポートって...ちょっと...
あと、vQFX でも start shell して tcpdump とか使えますよ。Control Plane 宛のトラフィックだけとりたい時とかには。
DataPlane
bb01 で拾った VXLAN のパケットを見ていきます。
node11 から node21 に ping うった時、最初に出る ARP Request と ARP reply
VXLAN カプセルの IP ヘッダ的にはマルチキャストではなく、spine11 VTEP と spine21 VTEP 間のユニキャストになってます。
node11 と node21 間の ICMP echo Request と ICMP echo Reply
これは EVPN 使おうが関係なく、ただの VXLAN ヘッダついたパケットです。
ControlPlane
bb01 で拾った EVPN のパケットを見ていきます。
EVPN NLRI Type2(MAC/IP Advertisement route) Update
まずは node11 からの ARP Request を受信した spine11 が advertise する様子
次に node21 からの ARP Reply を受信した spine21 が advertise したのを bb01 が spine11 方面に reflect する様子
EVPN NLRI Type2(MAC/IP Advertisement route) Withdrawn
spine11 で clear ethernet-switching table して node11 の MAC を消す
EVPN NLRI Type3(Inclusive Multicast Ethernet Tag route) Withdrawn
spine21 で以下のような感じで、雑に VLAN 100 = VNI10100 at RD 64512:21 が消えたことを advertise させた時の様子
{master:0}[edit]
kotetsu@spine21# deactivate vlans VLAN0100
{master:0}[edit]
kotetsu@spine21# show | compare
[edit vlans]
! inactive: VLAN0100 { ... }
{master:0}[edit]
kotetsu@spine21# commit
次に spine21 で VLAN 100 = VNI10100 at RD 64512:21 ができたことを advertise させた時の様子
{master:0}[edit]
kotetsu@spine21# activate vlans VLAN0100
{master:0}[edit]
kotetsu@spine21# show | compare
[edit vlans]
! active: VLAN0100 { ... }
{master:0}[edit]
kotetsu@spine21# commit
