kakkotetsu

vQFX10000 で VXLAN+EVPN (L2 over L3 編) (original : 2017/01/03)

この記事は某所で 2017/01/03 に書いた記事のコピーです。 そのため 2017/05/11 時点ではやや古い情報も含まれています。(以下一例)

  • Juniper さんの vQFX ダウンロード規約が変わっていそう (詳細は未確認)
  • 201705 に GNS3 ver2.0 stable が出たので KVM 間のパケットキャプチャ用に GNS3 のハブを挟む必要がなくなった

.

最初に

前回(vQFX10000 を KVM+GNS3 で動かす)、Juniper vQFX10000(以降 vQFX) の DL 権限を個人で得て GNS3 で軽く動作確認がとれました。 今回は「仮想版って L2全般/L2VPN 系機能が動かなかったりするけど、vQFX はどうなんだ?…お、EVPN もちゃんと動くやんけ!」ってところまでを見ていきます。

本項でやること

以下をやります。

個人的には、2014/12 時点で VXLAN のマルチキャスト実装を確認した時 (VyOS と Arista で VXLAN 相互接続)に微妙だと思った点にも着目しておきたいです。今回、データプレーン周りはほぼ同じ構成を組んでますので。

  • 上の記事の〆が「マルチキャストルーティングを勉強しよう。」だったのですが、この2年間以下のようにウダウダやっていて、特に進捗はありませんでした。
    • マルチキャスト?えー…?うーん…
    • え?OVSDB でゴリゴリ?あー…hmm…
    • あ?OVSDBを使ったコントローラ導入でユニキャスト?え?NSX?買えるかよ…
    • お?自前でコントローラ実装?ははぁ…
    • ん?MP-BGP使ったEVPN?いかにもネットワーク屋が考えそう&食いつきそうですねぇ…でもまあ釣られてみるかあ(今ここ)
  • 上の記事では VTEP 間の到達性をもたせるのに「VyOS が認識しないといけない Arista 側の VTEP IP アドレスは、Arista の Loopback アドレスになるので、StaticRoute を追加しておきます。#これだと拡張が面倒なので、実環境では DefaultRoute か動的ルーティングを使うことが多い気がします。」とか書いてますが、今回は実環境をある程度想定した例示ができそうです。

概要構成図 / 環境

以下のような構成を組んでいきます。node11 と node21 が L2overL3 で通信するやつです。

f:id:kakkotetsu:20170511232921p:plain

環境は、いずれも前回の通りです。

参考資料

前述の通り、各要素技術の詳細な解説は放棄していますが…こんなニッチな記事を読む人向けなので、まあね?

構築~動作確認

GNS3 で仮想マシンのデプロイ ~ 結線 ~ 起動

前回の通りに、3ペアの vQFX をデプロイして接続していきます。 以下の感じで。

f:id:kakkotetsu:20170511233004p:plain

torSW101atorSW201a というのは、GNS3Ethernet Switch を使っていますが、各環境に合わせて適当な dot1Q 食えるスイッチ置けば良いです。 設定はそれぞれ以下の感じです。(今回は Port 3-4 は使いませんが)

f:id:kakkotetsu:20170511233054p:plain

f:id:kakkotetsu:20170511233114p:plain

node11node21 は、インターフェースに IP アドレス付与するだけなので、適当な疎通確認用ノードなので好きなのをどうぞ。(node12node22 は今回使わないです)

あと captureSW というのは GNS3 1.5.2 では github gns3-gui issues | QEMU link Packet Captures の通り、qemu 同士の結線をパケットキャプチャできないので、「bb01 でポートミラーリング」する手法をとっているために置いているものです。
未試行ですが、RE 同士の結線をしている箇所(bb01spine[12]1)に全て Ethernet Switch を挟めば、ポートミラーリング不要でパケットキャプチャできると思います。(そちらの方がやりやすい筈)

できたら起動して待ちます。

基本設定

各環境に合わせて syslog なり NTP なり ssh key 登録なりしておいて下さい。 あ、今回は BGP のログを /var/log/bgp.log に残すために、以下をやっておくと良いですよ。

set protocols bgp traceoptions file bgp.log
set protocols bgp traceoptions file size 10k
set protocols bgp traceoptions file files 30
set protocols bgp traceoptions flag normal

Underlay 設定~確認 (物理IF と eBGP)

まずは物理 Interface と eBGP 周りを設定して、各 lo0 の IP アドレスを経路交換できたことを確認します。 まあ、長々となってしまいますが、絵にすると単純で以下のような感じです。

f:id:kakkotetsu:20170511233155p:plain

物理IF設定

bb01

set interfaces xe-0/0/0 description "DEV=spine11 IF=xe-0/0/0"
set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.1/30
delete interfaces xe-0/0/0 unit 0 family inet dhcp

set interfaces xe-0/0/1 description "DEV=spine21 IF=xe-0/0/0"
set interfaces xe-0/0/1 unit 0 family inet address 192.0.2.5/30
delete interfaces xe-0/0/1 unit 0 family inet dhcp

set protocols lldp port-id-subtype interface-name
set protocols lldp interface xe-0/0/0
set protocols lldp interface xe-0/0/1

spine11

set interfaces xe-0/0/0 description "DEV=bb01 IF=xe-0/0/0"
set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.2/30
delete interfaces xe-0/0/0 unit 0 family inet dhcp

set protocols lldp port-id-subtype interface-name
set protocols lldp interface xe-0/0/0

spine21

set interfaces xe-0/0/0 description "DEV=bb01 IF=xe-0/0/1"
set interfaces xe-0/0/0 unit 0 family inet address 192.0.2.6/30
delete interfaces xe-0/0/0 unit 0 family inet dhcp

set protocols lldp port-id-subtype interface-name
set protocols lldp interface xe-0/0/0

物理IF疎通確認

リソースをケチったからか RTT 長すぎですね…。

bb01

kotetsu@bb01> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
xe-0/0/0           -                   02:05:86:71:84:00   DEV=bb01 IF=xe-0/0/0 spine11            
xe-0/0/1           -                   02:05:86:71:ff:00   DEV=bb01 IF=xe-0/0/1 spine21


kotetsu@bb01> show route

inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.0/24        *[Direct/0] 19:47:39
                    > via em0.0
10.0.0.191/32      *[Local/0] 19:47:39
                      Local via em0.0
169.254.0.0/24     *[Direct/0] 19:57:21
                    > via em1.0
169.254.0.2/32     *[Local/0] 19:57:21
                      Local via em1.0
192.0.2.0/30       *[Direct/0] 00:17:12
                    > via xe-0/0/0.0
192.0.2.1/32       *[Local/0] 00:17:12
                      Local via xe-0/0/0.0
192.0.2.4/30       *[Direct/0] 00:07:04
                    > via xe-0/0/1.0
192.0.2.5/32       *[Local/0] 00:07:04
                      Local via xe-0/0/1.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

fe80::286:2b0f:fc44:ab00/128
                   *[Direct/0] 19:56:54
                    > via lo0.0

{master:0}
kotetsu@bb01> ping 192.0.2.2
PING 192.0.2.2 (192.0.2.2): 56 data bytes
64 bytes from 192.0.2.2: icmp_seq=1 ttl=64 time=2420.112 ms
64 bytes from 192.0.2.2: icmp_seq=2 ttl=64 time=1027.245 ms
64 bytes from 192.0.2.2: icmp_seq=3 ttl=64 time=1525.667 ms
^C
--- 192.0.2.2 ping statistics ---
5 packets transmitted, 3 packets received, 40% packet loss
round-trip min/avg/max/stddev = 1027.245/1657.675/2420.112/576.246 ms

{master:0}
kotetsu@bb01> ping 192.0.2.6
PING 192.0.2.6 (192.0.2.6): 56 data bytes
64 bytes from 192.0.2.6: icmp_seq=0 ttl=64 time=3378.582 ms
64 bytes from 192.0.2.6: icmp_seq=1 ttl=64 time=1374.159 ms
64 bytes from 192.0.2.6: icmp_seq=2 ttl=64 time=1474.743 ms
^C
--- 192.0.2.6 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1374.159/2075.828/3378.582/922.101 ms

spine11

{master:0}
kotetsu@spine11> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
xe-0/0/0           -                   02:05:86:71:55:00   DEV=spine11 IF=xe-0/0/0 bb01


{master:0}
kotetsu@spine11> show route

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.0/24        *[Direct/0] 19:23:06
                    > via em0.0
10.0.0.201/32      *[Local/0] 19:23:06
                      Local via em0.0
169.254.0.0/24     *[Direct/0] 19:22:26
                    > via em1.0
169.254.0.2/32     *[Local/0] 19:22:26
                      Local via em1.0
192.0.2.0/30       *[Direct/0] 00:16:05
                    > via xe-0/0/0.0
192.0.2.2/32       *[Local/0] 00:16:05
                      Local via xe-0/0/0.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

fe80::286:2b0f:fca1:f500/128
                   *[Direct/0] 19:22:25
                    > via lo0.0


{master:0}
kotetsu@spine11> ping 192.0.2.1
PING 192.0.2.1 (192.0.2.1): 56 data bytes
64 bytes from 192.0.2.1: icmp_seq=0 ttl=64 time=2839.663 ms
64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=2463.433 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=2269.077 ms
64 bytes from 192.0.2.1: icmp_seq=3 ttl=64 time=1548.765 ms
64 bytes from 192.0.2.1: icmp_seq=4 ttl=64 time=844.779 ms
64 bytes from 192.0.2.1: icmp_seq=5 ttl=64 time=505.582 ms
64 bytes from 192.0.2.1: icmp_seq=6 ttl=64 time=852.323 ms
64 bytes from 192.0.2.1: icmp_seq=7 ttl=64 time=1724.594 ms
^C
--- 192.0.2.1 ping statistics ---
9 packets transmitted, 8 packets received, 11% packet loss
round-trip min/avg/max/stddev = 505.582/1631.027/2839.663/795.890 ms

spine21

{master:0}
kotetsu@spine21> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
xe-0/0/0           -                   02:05:86:71:55:00   DEV=spine21 IF=xe-0/0/0 bb01


{master:0}
kotetsu@spine21> show route

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.0/24        *[Direct/0] 08:41:29
                    > via em0.0
10.0.0.202/32      *[Local/0] 08:41:30
                      Local via em0.0
169.254.0.0/24     *[Direct/0] 08:46:47
                    > via em1.0
169.254.0.2/32     *[Local/0] 08:46:47
                      Local via em1.0
192.0.2.4/30       *[Direct/0] 00:03:53
                    > via xe-0/0/0.0
192.0.2.6/32       *[Local/0] 00:03:53
                      Local via xe-0/0/0.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

fe80::286:2b0f:fcd0:9b00/128
                   *[Direct/0] 08:46:47
                    > via lo0.0


{master:0}
kotetsu@spine21> ping 192.0.2.5
PING 192.0.2.5 (192.0.2.5): 56 data bytes
64 bytes from 192.0.2.5: icmp_seq=0 ttl=64 time=427.225 ms
64 bytes from 192.0.2.5: icmp_seq=1 ttl=64 time=709.372 ms
64 bytes from 192.0.2.5: icmp_seq=2 ttl=64 time=1231.701 ms
64 bytes from 192.0.2.5: icmp_seq=3 ttl=64 time=732.131 ms
^C
--- 192.0.2.5 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 427.225/775.107/1231.701/289.684 ms

lo0 + eBGP 設定

今回、いずれもシングル構成なのでマルチパス関係の設定は入れていませんよ。

bb01

set interfaces lo0 unit 0 family inet address 172.31.0.1/32

set policy-options policy-statement POLICY_EXPORT_LO0 from family inet
set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct
set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32
set policy-options policy-statement POLICY_EXPORT_LO0 then accept

set routing-options router-id 172.31.0.1
set routing-options autonomous-system 65000

set protocols bgp group BGP_UNDERLAY type external
set protocols bgp group BGP_UNDERLAY advertise-peer-as
set protocols bgp group BGP_UNDERLAY family inet unicast loops 2
set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.2 description spine11
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.2 peer-as 65001
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.6 description spine21
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.6 peer-as 65002

spine11

set interfaces lo0 unit 0 family inet address 172.16.1.1/32

set policy-options policy-statement POLICY_EXPORT_LO0 from family inet
set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct
set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32
set policy-options policy-statement POLICY_EXPORT_LO0 then accept

set routing-options router-id 172.16.1.1
set routing-options autonomous-system 65001

set protocols bgp group BGP_UNDERLAY type external
set protocols bgp group BGP_UNDERLAY advertise-peer-as
set protocols bgp group BGP_UNDERLAY family inet unicast loops 2
set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.1 description bb00
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.1 peer-as 65000

spine21

set interfaces lo0 unit 0 family inet address 172.16.2.1/32

set policy-options policy-statement POLICY_EXPORT_LO0 from family inet
set policy-options policy-statement POLICY_EXPORT_LO0 from protocol direct
set policy-options policy-statement POLICY_EXPORT_LO0 from route-filter 0.0.0.0/0 prefix-length-range /32-/32
set policy-options policy-statement POLICY_EXPORT_LO0 then accept

set routing-options router-id 172.16.2.1
set routing-options autonomous-system 65002

set protocols bgp group BGP_UNDERLAY type external
set protocols bgp group BGP_UNDERLAY advertise-peer-as
set protocols bgp group BGP_UNDERLAY family inet unicast loops 2
set protocols bgp group BGP_UNDERLAY export POLICY_EXPORT_LO0
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.5 description bb00
set protocols bgp group BGP_UNDERLAY neighbor 192.0.2.5 peer-as 65000

eBGP確認

それぞれ lo0IPアドレスを相互学習したことを確認します。 この後やる Overlay 用の MP-BGP (iBGP) で bb01 が Route Reflector になるので、実際には bb01spine[12]1 で交換できていれば十分な筈ですが。

bb01

{master:0}
kotetsu@bb01> show bgp summary
Groups: 1 Peers: 2 Down peers: 1
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.2             65001          4          4       0       0          20 1/1/1/0              0/0/0/0
192.0.2.6             65002          1          2       0       0        5:57 OpenConfirm

{master:0}
kotetsu@bb01> show bgp summary
Groups: 1 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.2             65001          4          5       0       0          25 1/1/1/0              0/0/0/0
192.0.2.6             65002          4          4       0       0           4 1/1/1/0              0/0/0/0

{master:0}
kotetsu@bb01>

{master:0}
kotetsu@bb01> show bgp summary
Groups: 1 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.2             65001          4          5       0       0          26 1/1/1/0              0/0/0/0
192.0.2.6             65002          4          4       0       0           5 1/1/1/0              0/0/0/0


{master:0}
kotetsu@bb01> show bgp group BGP_UNDERLAY brief
Group Type: External                               Local AS: 65000
  Name: BGP_UNDERLAY    Index: 0                   Flags: <Export Eval>
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <AdvertisePeerAs>
  Holdtime: 0
  Total peers: 2        Established: 2
  192.0.2.2+56114
  192.0.2.6+49934
  inet.0: 2/2/2/0


{master:0}
kotetsu@bb01> show bgp neighbor
Peer: 192.0.2.2+56114 AS 65001 Local: 192.0.2.1+179 AS 65000
  Description: spine11
  Group: BGP_UNDERLAY          Routing-Instance: master
  Forwarding routing-instance: master
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <Preference AddressFamily PeerAS Refresh>
  Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
  Address families configured: inet-unicast
  Holdtime: 90 Preference: 170
  Number of flaps: 0
  Peer ID: 172.16.1.1      Local ID: 172.31.0.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 0    Peer index: 0
  BFD: disabled, down
  Local Interface: xe-0/0/0.0
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 65001)
  Peer does not support Addpath
  Table inet.0 Bit: 10000
    RIB State: BGP restart is complete
    Send state: in sync
    Active prefixes:              1
    Received prefixes:            1
    Accepted prefixes:            1
    Suppressed due to damping:    0
    Advertised prefixes:          2
  Last traffic (seconds): Received 23   Sent 5    Checked 56
  Input messages:  Total 179    Updates 2       Refreshes 0     Octets 3478
  Output messages: Total 186    Updates 2       Refreshes 0     Octets 3663
  Output Queue[0]: 0            (inet.0, inet-unicast)

Peer: 192.0.2.6+49934 AS 65002 Local: 192.0.2.5+179 AS 65000
  Description: spine21
  Group: BGP_UNDERLAY          Routing-Instance: master
  Forwarding routing-instance: master
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <Preference AddressFamily PeerAS Refresh>
  Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
  Address families configured: inet-unicast
  Holdtime: 90 Preference: 170
  Number of flaps: 0
  Peer ID: 172.16.2.1      Local ID: 172.31.0.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 0    Peer index: 1
  BFD: disabled, down
  Local Interface: xe-0/0/1.0
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 65002)
  Peer does not support Addpath
  Table inet.0 Bit: 10000
    RIB State: BGP restart is complete
    Send state: in sync
    Active prefixes:              1
    Received prefixes:            1
    Accepted prefixes:            1
    Suppressed due to damping:    0
    Advertised prefixes:          2
  Last traffic (seconds): Received 54   Sent 19   Checked 15
  Input messages:  Total 92     Updates 2       Refreshes 0     Octets 1825
  Output messages: Total 184    Updates 2       Refreshes 0     Octets 3625
  Output Queue[0]: 0            (inet.0, inet-unicast)
{master:0}
kotetsu@bb01> show route protocol bgp

inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.1.1/32      *[BGP/170] 00:02:11, localpref 100
                      AS path: 65001 I, validation-state: unverified
                    > to 192.0.2.2 via xe-0/0/0.0
172.16.2.1/32      *[BGP/170] 00:01:49, localpref 100
                      AS path: 65002 I, validation-state: unverified
                    > to 192.0.2.6 via xe-0/0/1.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)


{master:0}
kotetsu@bb01> ping 172.16.2.1
PING 172.16.2.1 (172.16.2.1): 56 data bytes
64 bytes from 172.16.2.1: icmp_seq=0 ttl=64 time=908.929 ms
64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=1249.259 ms
^C
--- 172.16.2.1 ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max/stddev = 908.929/1079.094/1249.259/170.165 ms


{master:0}
kotetsu@bb01> ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1): 56 data bytes
64 bytes from 172.16.1.1: icmp_seq=0 ttl=64 time=2595.413 ms
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=1406.501 ms
^C
--- 172.16.1.1 ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max/stddev = 1406.501/2000.957/2595.413/594.456 ms

spine11

{master:0}
kotetsu@spine11> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.1             65000        174        169       0       0     1:16:34 2/2/2/0              0/0/0/0


{master:0}
kotetsu@spine11> show bgp group BGP_UNDERLAY brief
Group Type: External                               Local AS: 65001
  Name: BGP_UNDERLAY    Index: 0                   Flags: <Export Eval>
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <AdvertisePeerAs>
  Holdtime: 0
  Total peers: 1        Established: 1
  192.0.2.1+179
  inet.0: 2/2/2/0


{master:0}
kotetsu@spine11> show bgp neighbor
Peer: 192.0.2.1+179 AS 65000   Local: 192.0.2.2+56114 AS 65001
  Description: bb00
  Group: BGP_UNDERLAY          Routing-Instance: master
  Forwarding routing-instance: master
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <Preference AddressFamily PeerAS Refresh>
  Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
  Address families configured: inet-unicast
  Holdtime: 90 Preference: 170
  Number of flaps: 0
  Peer ID: 172.31.0.1      Local ID: 172.16.1.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 0    Peer index: 0
  BFD: disabled, down
  Local Interface: xe-0/0/0.0
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 65000)
  Peer does not support Addpath
  Table inet.0 Bit: 10000
    RIB State: BGP restart is complete
    Send state: in sync
    Active prefixes:              2
    Received prefixes:            2
    Accepted prefixes:            2
    Suppressed due to damping:    0
    Advertised prefixes:          1
  Last traffic (seconds): Received 9    Sent 7    Checked 50
  Input messages:  Total 182    Updates 3       Refreshes 0     Octets 3524
  Output messages: Total 177    Updates 1       Refreshes 0     Octets 3459
  Output Queue[0]: 0            (inet.0, inet-unicast)
{master:0}
kotetsu@spine11> show route protocol bgp

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.2.1/32      *[BGP/170] 01:22:49, localpref 100
                      AS path: 65000 65002 I, validation-state: unverified
                    > to 192.0.2.1 via xe-0/0/0.0
172.31.0.1/32      *[BGP/170] 01:23:10, localpref 100
                      AS path: 65000 I, validation-state: unverified
                    > to 192.0.2.1 via xe-0/0/0.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)


{master:0}
kotetsu@spine11> ping 172.31.0.1
PING 172.31.0.1 (172.31.0.1): 56 data bytes
64 bytes from 172.31.0.1: icmp_seq=0 ttl=64 time=1597.291 ms
64 bytes from 172.31.0.1: icmp_seq=1 ttl=64 time=1215.306 ms
^C
--- 172.31.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1215.306/1406.299/1597.291/190.993 ms

spine21

{master:0}
kotetsu@spine21> show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.5             65000        173         88       0       0       38:05 2/2/2/0              0/0/0/0


{master:0}
kotetsu@spine21> show bgp group BGP_UNDERLAY brief
Group Type: External                               Local AS: 65002
  Name: BGP_UNDERLAY    Index: 0                   Flags: <Export Eval>
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <AdvertisePeerAs>
  Holdtime: 0
  Total peers: 1        Established: 1
  192.0.2.5+179
  inet.0: 2/2/2/0


kotetsu@spine21> show bgp neighbor
Peer: 192.0.2.5+179 AS 65000   Local: 192.0.2.6+49934 AS 65002
  Description: bb00
  Group: BGP_UNDERLAY          Routing-Instance: master
  Forwarding routing-instance: master
  Type: External    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Export: [ POLICY_EXPORT_LO0 ]
  Options: <Preference AddressFamily PeerAS Refresh>
  Options: <AdvertisePeerAs PeerSpecficLoopsAllowed>
  Address families configured: inet-unicast
  Holdtime: 90 Preference: 170
  Number of flaps: 0
  Peer ID: 172.31.0.1      Local ID: 172.16.2.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 0    Peer index: 0
  BFD: disabled, down
  Local Interface: xe-0/0/0.0
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 65000)
  Peer does not support Addpath
  Table inet.0 Bit: 10000
    RIB State: BGP restart is complete
    Send state: in sync
    Active prefixes:              2
    Received prefixes:            2
    Accepted prefixes:            2
    Suppressed due to damping:    0
    Advertised prefixes:          1
  Last traffic (seconds): Received 1    Sent 24   Checked 3
  Input messages:  Total 175    Updates 3       Refreshes 0     Octets 3391
  Output messages: Total 88     Updates 1       Refreshes 0     Octets 1768
  Output Queue[0]: 0            (inet.0, inet-unicast)
{master:0}
kotetsu@spine21> show route protocol bgp

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.1.1/32      *[BGP/170] 00:41:43, localpref 100
                      AS path: 65000 65001 I, validation-state: unverified
                    > to 192.0.2.5 via xe-0/0/0.0
172.31.0.1/32      *[BGP/170] 00:41:43, localpref 100
                      AS path: 65000 I, validation-state: unverified
                    > to 192.0.2.5 via xe-0/0/0.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)


{master:0}
kotetsu@spine21> ping 172.31.0.1
PING 172.31.0.1 (172.31.0.1): 56 data bytes
64 bytes from 172.31.0.1: icmp_seq=0 ttl=64 time=691.327 ms
64 bytes from 172.31.0.1: icmp_seq=1 ttl=64 time=1227.117 ms
64 bytes from 172.31.0.1: icmp_seq=2 ttl=64 time=238.494 ms
^C
--- 172.31.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 238.494/718.979/1227.117/404.077 ms

Overlay 設定~確認 (MP-BGP)

Underlay eBGP で経路交換した各 lo0 同士で iBGP を組みます。 まあ、長々となってしまいますが、絵にすると単純で以下のような感じです。

f:id:kakkotetsu:20170511233224p:plain

iBGP (MP-BGP )設定

ポイントは1つだけで family evpn signaling を設定して EVPN の NLRI を扱うってとこですかね。

bb01

set protocols bgp group BGP_OVERLAY type internal
set protocols bgp group BGP_OVERLAY local-address 172.31.0.1
set protocols bgp group BGP_OVERLAY family evpn signaling
set protocols bgp group BGP_OVERLAY cluster 172.31.0.1
set protocols bgp group BGP_OVERLAY local-as 64512
set protocols bgp group BGP_OVERLAY neighbor 172.16.1.1 description spine11
set protocols bgp group BGP_OVERLAY neighbor 172.16.2.1 description spine21

spine11

set protocols bgp group BGP_OVERLAY type internal
set protocols bgp group BGP_OVERLAY local-address 172.16.1.1
set protocols bgp group BGP_OVERLAY family evpn signaling
set protocols bgp group BGP_OVERLAY local-as 64512
set protocols bgp group BGP_OVERLAY neighbor 172.31.0.1 description bb01

spine21

set protocols bgp group BGP_OVERLAY type internal
set protocols bgp group BGP_OVERLAY local-address 172.16.2.1
set protocols bgp group BGP_OVERLAY family evpn signaling
set protocols bgp group BGP_OVERLAY local-as 64512
set protocols bgp group BGP_OVERLAY neighbor 172.31.0.1 description bb01

iBGP (MP-BGP) 確認

最初 neighbor IP アドレス設定をタイポしたせいで、Last Error: Open Message Error とか残っているのはご愛敬ということでひとつ…。

bb01

{master:0}
kotetsu@bb01> show bgp summary
Groups: 2 Peers: 4 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.evpn.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1            64512          3          3       0       0          25 Establ
  bgp.evpn.0: 0/0/0/0
172.16.2.1            64512          3          2       0       0          12 Establ
  bgp.evpn.0: 0/0/0/0
192.0.2.2             65001        502        518       0       0     3:58:25 1/1/1/0              0/0/0/0
192.0.2.6             65002        257        517       0       0     3:58:04 1/1/1/0              0/0/0/0


{master:0}
kotetsu@bb01> show bgp group BGP_OVERLAY
Group Type: Internal    AS: 64512                  Local AS: 64512
  Name: BGP_OVERLAY     Index: 1                   Flags: <Export Eval>
  Options: <Cluster LocalAS>
  Holdtime: 0 Local AS: 64512 Local System AS: 65000
  Total peers: 2        Established: 2
  172.16.1.1+52050
  172.16.2.1+62794
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30
  bgp.evpn.0: 0/0/0/0


{master:0}
kotetsu@bb01> show bgp neighbor
Peer: 172.16.1.1+52050 AS 64512 Local: 172.31.0.1+179 AS 64512
  Description: spine11
  Group: BGP_OVERLAY           Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Established  (route reflector client)Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: Open Message Error
  Options: <Preference LocalAddress Cluster AddressFamily LocalAS Rib-group Refresh>
  Address families configured: evpn
  Local Address: 172.31.0.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65000
  Number of flaps: 0
  Error: 'Open Message Error' Sent: 20 Recv: 0
  Peer ID: 172.16.1.1      Local ID: 172.31.0.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 1    Peer index: 0
  BFD: disabled, down
  NLRI for restart configured on peer: evpn
  NLRI advertised by peer: evpn
  NLRI for this session: evpn
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: evpn
  NLRI of received end-of-rib markers: evpn
  NLRI of all end-of-rib markers sent: evpn
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 64512)
  Peer does not support Addpath
  Table bgp.evpn.0 Bit: 20000
    RIB State: BGP restart is complete
    RIB State: VPN restart is complete
    Send state: in sync
    Active prefixes:              0
    Received prefixes:            0
    Accepted prefixes:            0
    Suppressed due to damping:    0
    Advertised prefixes:          0
  Last traffic (seconds): Received 1    Sent 9    Checked 0
  Input messages:  Total 6      Updates 1       Refreshes 0     Octets 169
  Output messages: Total 6      Updates 0       Refreshes 0     Octets 188
  Output Queue[1]: 0            (bgp.evpn.0, evpn)
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30

Peer: 172.16.2.1+62794 AS 64512 Local: 172.31.0.1+179 AS 64512
  Description: spine21
  Group: BGP_OVERLAY           Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Established  (route reflector client)Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: Open Message Error
  Options: <Preference LocalAddress Cluster AddressFamily LocalAS Rib-group Refresh>
  Address families configured: evpn
  Local Address: 172.31.0.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65000
  Number of flaps: 0
  Error: 'Open Message Error' Sent: 20 Recv: 0
  Peer ID: 172.16.2.1      Local ID: 172.31.0.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 1    Peer index: 1
  BFD: disabled, down
  NLRI for restart configured on peer: evpn
  NLRI advertised by peer: evpn
  NLRI for this session: evpn
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: evpn
  NLRI of received end-of-rib markers: evpn
  NLRI of all end-of-rib markers sent: evpn
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 64512)
  Peer does not support Addpath
  Table bgp.evpn.0 Bit: 20000
    RIB State: BGP restart is complete
    RIB State: VPN restart is complete
    Send state: in sync
    Active prefixes:              0
    Received prefixes:            0
    Accepted prefixes:            0
    Suppressed due to damping:    0
    Advertised prefixes:          0
  Last traffic (seconds): Received 46   Sent 0    Checked 9
  Input messages:  Total 10     Updates 1       Refreshes 0     Octets 245
  Output messages: Total 18     Updates 0       Refreshes 0     Octets 416
  Output Queue[1]: 0            (bgp.evpn.0, evpn)
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30

...

spine11

{master:0}
kotetsu@spine11> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.evpn.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.31.0.1            64512         21         22       0       0        9:07 Establ
  bgp.evpn.0: 0/0/0/0
192.0.2.1             65000        536        521       0       0     3:59:55 2/2/2/0              0/0/0/


{master:0}
kotetsu@spine11> show bgp group BGP_OVERLAY
Group Type: Internal    AS: 64512                  Local AS: 64512
  Name: BGP_OVERLAY     Index: 1                   Flags: <Export Eval>
  Options: <LocalAS>
  Holdtime: 0 Local AS: 64512 Local System AS: 65001
  Total peers: 1        Established: 1
  172.31.0.1+179
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30
  bgp.evpn.0: 0/0/0/0


{master:0}
kotetsu@spine11> show bgp neighbor
Peer: 172.31.0.1+179 AS 64512  Local: 172.16.1.1+52050 AS 64512
  Description: bb01
  Group: BGP_OVERLAY           Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Options: <Preference LocalAddress AddressFamily LocalAS Rib-group Refresh>
  Address families configured: evpn
  Local Address: 172.16.1.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65001
  Number of flaps: 0
  Peer ID: 172.31.0.1      Local ID: 172.16.1.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 1    Peer index: 1
  BFD: disabled, down
  NLRI for restart configured on peer: evpn
  NLRI advertised by peer: evpn
  NLRI for this session: evpn
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: evpn
  NLRI of received end-of-rib markers: evpn
  NLRI of all end-of-rib markers sent: evpn
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 64512)
  Peer does not support Addpath
  Table bgp.evpn.0
    RIB State: BGP restart is complete
    RIB State: VPN restart is complete
    Send state: not advertising
    Active prefixes:              0
    Received prefixes:            0
    Accepted prefixes:            0
    Suppressed due to damping:    0
  Last traffic (seconds): Received 1    Sent 8    Checked 46
  Input messages:  Total 34     Updates 1       Refreshes 0     Octets 657
  Output messages: Total 34     Updates 0       Refreshes 0     Octets 720
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30

...

spine21

{master:0}
kotetsu@spine21> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.evpn.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.31.0.1            64512         34         19       0       0        7:30 Establ
  bgp.evpn.0: 0/0/0/0
192.0.2.5             65000        549        273       0       0     2:02:08 2/2/2/0              0/0/0/0


{master:0}
kotetsu@spine21> show bgp group BGP_OVERLAY
Group Type: Internal    AS: 64512                  Local AS: 64512
  Name: BGP_OVERLAY     Index: 1                   Flags: <Export Eval>
  Options: <LocalAS>
  Holdtime: 0 Local AS: 64512 Local System AS: 65002
  Total peers: 1        Established: 1
  172.31.0.1+179
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30
  bgp.evpn.0: 0/0/0/0


{master:0}
kotetsu@spine21> show bgp neighbor
Peer: 172.31.0.1+179 AS 64512  Local: 172.16.2.1+62794 AS 64512
  Description: bb01
  Group: BGP_OVERLAY           Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Options: <Preference LocalAddress AddressFamily LocalAS Rib-group Refresh>
  Address families configured: evpn
  Local Address: 172.16.2.1 Holdtime: 90 Preference: 170 Local AS: 64512 Local System AS: 65002
  Number of flaps: 0
  Peer ID: 172.31.0.1      Local ID: 172.16.2.1        Active Holdtime: 90
  Keepalive Interval: 30         Group index: 1    Peer index: 1
  BFD: disabled, down
  NLRI for restart configured on peer: evpn
  NLRI advertised by peer: evpn
  NLRI for this session: evpn
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Restart flag received from the peer: Notification
  NLRI that restart is negotiated for: evpn
  NLRI of received end-of-rib markers: evpn
  NLRI of all end-of-rib markers sent: evpn
  Peer does not support LLGR Restarter functionality
  Peer supports 4 byte AS extension (peer-as 64512)
  Peer does not support Addpath
  Table bgp.evpn.0
    RIB State: BGP restart is complete
    RIB State: VPN restart is complete
    Send state: not advertising
    Active prefixes:              0
    Received prefixes:            0
    Accepted prefixes:            0
    Suppressed due to damping:    0
  Last traffic (seconds): Received 14   Sent 17   Checked 58
  Input messages:  Total 36     Updates 1       Refreshes 0     Octets 695
  Output messages: Total 20     Updates 0       Refreshes 0     Octets 454
  Trace options: normal
  Trace file: /var/log/bgp.log size 10240 files 30

...

Overlay設定~確認(EVPN+VXLAN)

EVPN の設定と、その Dataplane として使う VXLAN 周りの設定をしていきます。(息切れ)

EVPN+VXLAN 設定

冒頭の全体構成に書いたのですが bb01 はこの辺の挙動に関しては土管に徹しているので登場しません。 VNI1..16777214 まで使えるので、単純に全 spine で VLAN ID と 1:1 で同じ対応付けをさせて、一律で VLAN ID + 10000 の値を振ってます。 あ、VNI 10100 以外は本項ではまだ使いません。

spine11

set vlans VLAN0100 vlan-id 100
set vlans VLAN0100 vxlan vni 10100
set vlans VLAN0100 vxlan ingress-node-replication

set vlans VLAN0300 vlan-id 300
set vlans VLAN0300 vxlan vni 10300
set vlans VLAN0300 vxlan ingress-node-replication

set protocols evpn encapsulation vxlan
set protocols evpn extended-vni-list all
set protocols evpn multicast-mode ingress-replication
set protocols evpn vni-options vni 10100 vrf-target export target:1:10100
set protocols evpn vni-options vni 10300 vrf-target export target:1:10300

set policy-options community COM_10100 members target:1:10100
set policy-options community COM_10300 members target:1:10300
set policy-options community COM_LEAF_ESI members target:9999:9999

set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 from community COM_10100
set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 then accept
set policy-options policy-statement POLICY_VRF_IMPORT term T_10300 from community COM_10300
set policy-options policy-statement POLICY_VRF_IMPORT term T_10300 then accept
set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 from community COM_LEAF_ESI
set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 then accept
set policy-options policy-statement POLICY_VRF_IMPORT term T_99999 then reject

set switch-options vtep-source-interface lo0.0
set switch-options route-distinguisher 64512:11
set switch-options vrf-import POLICY_VRF_IMPORT
set switch-options vrf-target target:9999:9999
set switch-options vrf-target auto 

set interfaces xe-0/0/1 description "DEV=torSW11 IF=1"
set interfaces xe-0/0/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members all

spine21

set vlans VLAN0100 vlan-id 100
set vlans VLAN0100 vxlan vni 10100
set vlans VLAN0100 vxlan ingress-node-replication

set vlans VLAN0200 vlan-id 200
set vlans VLAN0200 vxlan vni 10200
set vlans VLAN0200 vxlan ingress-node-replication

set protocols evpn encapsulation vxlan
set protocols evpn extended-vni-list all
set protocols evpn multicast-mode ingress-replication
set protocols evpn vni-options vni 10100 vrf-target export target:1:10100
set protocols evpn vni-options vni 10200 vrf-target export target:1:10200

set policy-options community COM_10100 members target:1:10100
set policy-options community COM_10200 members target:1:10200
set policy-options community COM_LEAF_ESI members target:9999:9999

set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 from community COM_10100
set policy-options policy-statement POLICY_VRF_IMPORT term T_10100 then accept
set policy-options policy-statement POLICY_VRF_IMPORT term T_10200 from community COM_10200
set policy-options policy-statement POLICY_VRF_IMPORT term T_10200 then accept
set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 from community COM_LEAF_ESI
set policy-options policy-statement POLICY_VRF_IMPORT term T_99900 then accept
set policy-options policy-statement POLICY_VRF_IMPORT term T_99999 then reject

set switch-options vtep-source-interface lo0.0
set switch-options route-distinguisher 64512:21
set switch-options vrf-import POLICY_VRF_IMPORT
set switch-options vrf-target target:9999:9999
set switch-options vrf-target auto 

set interfaces xe-0/0/1 description "DEV=torSW21 IF=1"
set interfaces xe-0/0/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members all

動作確認

node 同士の疎通確認

  • node11
$ ip a show dev ens4
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:86:2b:8d:10:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::286:2bff:fe8d:1001/64 scope link
       valid_lft forever preferred_lft forever

$ ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=975 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=949 ms

$ ip n show dev ens4
192.168.1.2 lladdr 00:86:2b:5c:0d:01 STALE
  • node21
$ ip a show dev ens4
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:86:2b:5c:0d:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::286:2bff:fe5c:d01/64 scope link
       valid_lft forever preferred_lft forever

$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=639 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=926 ms

$ ip n show dev ens4
192.168.1.1 lladdr 00:86:2b:8d:10:01 STALE

vQFX のテーブル確認

前述の Juniper公式日本語資料 のページ 36 によると

bgp.evpn.0 = Junos OS ルーティング プロトコル プロセス(RPD)内のグローバル EVPN ルーティング テーブル default.switch.evpn.0(QFX5100 の場合) = Junos OS RPD 内のスイッチ レベル EVPN 転送テーブル \<virtual-switch-name>.evpn.0(MX の場合) = Junos OS RPD 内の仮想スイッチ レベル EVPN 転送テーブル

らしいですよ。

spine11

こいつにとっての vtep.32769 てのは spine21 (の VTEP)

{master:0}
kotetsu@spine11> show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
   Vlan                MAC                 MAC      Logical                Active
   name                address             flags    interface              source
   VLAN0100            00:86:2b:5c:0d:01   D        vtep.32769             172.16.2.1
   VLAN0100            00:86:2b:8d:10:01   D        xe-0/0/1.0


{master:0}
kotetsu@spine11> show interfaces vtep.32769
  Logical interface vtep.32769 (Index 566) (SNMP ifIndex 537)
    Flags: Up SNMP-Traps Encapsulation: ENET2
    VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 172.16.2.1, L2 Routing Instance: default-switch, L3 Routing Instance: default
    Input packets : 4
    Output packets: 4
    Protocol eth-switch, MTU: Unlimited
      Flags: Trunk-Mode

EVPN の Type2(MAC/IP Advertisement route) と Type3(Inclusive Multicast Ethernet Tag route) を spine21 (RD 64512:21)から学習して bgp.evpn.0 から default-switch.evpn.0 にロードされている様子

{master:0}
kotetsu@spine11> show route

...

:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.0/24        *[Direct/0] 00:23:24
                    > via em0.0
10.0.0.201/32      *[Local/0] 00:06:15
                      Local via em0.0
169.254.0.0/24     *[Direct/0] 00:23:24
                    > via em1.0
169.254.0.2/32     *[Local/0] 00:23:24
                      Local via em1.0
172.16.1.1/32      *[Direct/0] 00:23:24
                    > via lo0.0
172.16.2.1/32      *[Static/1] 00:13:22, metric2 0
                    > to 192.0.2.1 via xe-0/0/0.0
192.0.2.0/30       *[Direct/0] 00:23:24
                    > via xe-0/0/0.0
192.0.2.2/32       *[Local/0] 00:23:24
                      Local via xe-0/0/0.0


bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:64512:21::10100::00:86:2b:5c:0d:01/304
                   *[BGP/170] 00:08:29, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.1 via xe-0/0/0.0
3:64512:21::10100::172.16.2.1/304
                   *[BGP/170] 00:13:23, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.1 via xe-0/0/0.0


default-switch.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:64512:11::10100::00:86:2b:8d:10:01/304
                   *[EVPN/170] 00:08:33
                      Indirect
2:64512:21::10100::00:86:2b:5c:0d:01/304
                   *[BGP/170] 00:08:29, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.1 via xe-0/0/0.0
3:64512:11::10100::172.16.1.1/304
                   *[EVPN/170] 00:23:27
                      Indirect
3:64512:11::10300::172.16.1.1/304
                   *[EVPN/170] 00:15:11
                      Indirect
3:64512:21::10100::172.16.2.1/304
                   *[BGP/170] 00:13:23, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.1 via xe-0/0/0.0
{master:0}
kotetsu@spine11> show route forwarding-table

...

Routing table: :vxlan.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct     1724     1
0.0.0.0/32         perm     0                    dscd     1722     1
169.254.0.0/24     user     0                    rtbl        1     5
169.254.0.2/32     user     0 169.254.0.2        locl      338     3
172.16.1.1/32      user     0                    rtbl        1     5
172.16.2.1/32      user     0                    indr   131070     3
                              192.0.2.1          ucst     1719     7 xe-0/0/0.0
192.0.2.0/30       user     0                    rtbl        1     5
192.0.2.2/32       user     0 192.0.2.2          locl     1717     3
224.0.0.0/4        perm     0                    mdsc     1723     1
224.0.0.1/32       perm     0 224.0.0.1          mcst     1726     1
255.255.255.255/32 perm     0                    bcst     1727     1

...

Routing table: default-switch.bridge
Bridging domain: VLAN0100.bridge
VPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
00:86:2b:5c:0d:01/48 user     0                  comp     1707     6
00:86:2b:8d:10:01/48 user     0                  ucst     1712     6 xe-0/0/1.0
0x30001/51         user     0                    comp     1739     2
0x30006/51         user     0                    comp     1732     2
0x30002/51         user     0                    comp     1733     2

Routing table: default-switch.bridge
Bridging domain: VLAN0300.bridge
VPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
0x30007/51         user     0                    comp     1741     2
0x30005/51         user     0                    comp     1714     2
0x30003/51         user     0                    comp     1734     2

...

spine21

こいつにとっての vtep.32769 てのは spine11 (の VTEP)

{master:0}
kotetsu@spine21> show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
   Vlan                MAC                 MAC      Logical                Active
   name                address             flags    interface              source
   VLAN0100            00:86:2b:5c:0d:01   D        xe-0/0/1.0
   VLAN0100            00:86:2b:8d:10:01   D        vtep.32769             172.16.1.1

EVPN の Type2(MAC/IP Advertisement route) と Type3(Inclusive Multicast Ethernet Tag route) を spine11 (RD 64512:11)から学習して bgp.evpn.0 から default-switch.evpn.0 にロードされている様子

{master:0}
kotetsu@spine21> show route

...

:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.0/24        *[Direct/0] 00:18:41
                    > via em0.0
10.0.0.202/32      *[Local/0] 00:11:20
                      Local via em0.0
169.254.0.0/24     *[Direct/0] 00:18:41
                    > via em1.0
169.254.0.2/32     *[Local/0] 00:18:41
                      Local via em1.0
172.16.1.1/32      *[Static/1] 00:18:39, metric2 0
                    > to 192.0.2.5 via xe-0/0/0.0
172.16.2.1/32      *[Direct/0] 00:18:41
                    > via lo0.0
192.0.2.4/30       *[Direct/0] 00:18:41
                    > via xe-0/0/0.0
192.0.2.6/32       *[Local/0] 00:18:41
                      Local via xe-0/0/0.0

bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:64512:11::10100::00:86:2b:8d:10:01/304
                   *[BGP/170] 00:12:17, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.5 via xe-0/0/0.0
3:64512:11::10100::172.16.1.1/304
                   *[BGP/170] 00:18:40, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.5 via xe-0/0/0.0

default-switch.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:64512:11::10100::00:86:2b:8d:10:01/304
                   *[BGP/170] 00:12:17, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.5 via xe-0/0/0.0
2:64512:21::10100::00:86:2b:5c:0d:01/304
                   *[EVPN/170] 00:12:16
                      Indirect
3:64512:11::10100::172.16.1.1/304
                   *[BGP/170] 00:18:40, localpref 100, from 172.31.0.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.5 via xe-0/0/0.0
3:64512:21::10100::172.16.2.1/304
                   *[EVPN/170] 00:14:43
                      Indirect
3:64512:21::10200::172.16.2.1/304
                   *[EVPN/170] 00:14:43
                      Indirect
{master:0}
kotetsu@spine21> show route forwarding-table

...

Routing table: :vxlan.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct     1716     1
0.0.0.0/32         perm     0                    dscd     1714     1
169.254.0.0/24     user     0                    rtbl        1     5
169.254.0.2/32     user     0 169.254.0.2        locl      334     3
172.16.1.1/32      user     0                    indr   131070     3
                              192.0.2.5          ucst     1711     7 xe-0/0/0.0
172.16.2.1/32      user     0                    rtbl        1     5
192.0.2.4/30       user     0                    rtbl        1     5
192.0.2.6/32       user     0 192.0.2.6          locl     1709     3
224.0.0.0/4        perm     0                    mdsc     1715     1
224.0.0.1/32       perm     0 224.0.0.1          mcst     1718     1
255.255.255.255/32 perm     0                    bcst     1719     1

...

Routing table: default-switch.bridge
Bridging domain: VLAN0100.bridge
VPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
00:86:2b:5c:0d:01/48 user     0                  ucst     1730     6 xe-0/0/1.0
00:86:2b:8d:10:01/48 user     0                  comp     1723     6
0x30006/51         user     0                    comp     1731     2
0x30004/51         user     0                    comp     1725     2
0x30001/51         user     0                    comp     1726     2

Routing table: default-switch.bridge
Bridging domain: VLAN0200.bridge
VPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
0x30007/51         user     0                    comp     1734     2
0x30005/51         user     0                    comp     1729     2
0x30002/51         user     0                    comp     1728     2


bb01

{master:0}
kotetsu@bb01> show ethernet-switching table

単純に EVPN 経路を reflect する土管なので bgp.evpn.0 にしか載っていない様子

{master:0}
kotetsu@bb01> show route

...

bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:64512:11::10100::00:86:2b:8d:10:01/304
                   *[BGP/170] 00:31:03, localpref 100, from 172.16.1.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.2 via xe-0/0/0.0
2:64512:21::10100::00:86:2b:5c:0d:01/304
                   *[BGP/170] 00:31:01, localpref 100, from 172.16.2.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.6 via xe-0/0/1.0
3:64512:11::10100::172.16.1.1/304
                   *[BGP/170] 00:46:21, localpref 100, from 172.16.1.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.2 via xe-0/0/0.0
3:64512:11::10300::172.16.1.1/304
                   *[BGP/170] 00:37:52, localpref 100, from 172.16.1.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.2 via xe-0/0/0.0
3:64512:21::10100::172.16.2.1/304
                   *[BGP/170] 00:36:05, localpref 100, from 172.16.2.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.6 via xe-0/0/1.0
3:64512:21::10200::172.16.2.1/304
                   *[BGP/170] 00:35:45, localpref 100, from 172.16.2.1
                      AS path: I, validation-state: unverified
                    > to 192.0.2.6 via xe-0/0/1.0


{master:0}
kotetsu@bb01> show route forwarding-table

...

パケットキャプチャ

いくつか特徴的なパケットを拾って眺めていきます。

前述の通り、今回は RE 同士を接続するところで直結構成にしてしまい bb01xe-0/0/2 から captureSW 方面にポートミラーリングして、GNS3 の bb01(RE)captureSW 間のリンクを右クリックして start capture とかやっていきます。
なので bb01 に以下のような設定をして、xe-0/0/0 (spine11 側)の出入りを xe-0/0/2ミラーリングしときます。

set interfaces xe-0/0/2 description "DEV=captureSW IF=1"
set interfaces xe-0/0/2 unit 0 family ethernet-switching

set forwarding-options analyzer ANAL_PORT input ingress interface xe-0/0/0.0
set forwarding-options analyzer ANAL_PORT input egress interface xe-0/0/0.0
set forwarding-options analyzer ANAL_PORT output interface xe-0/0/2.0

見返していて思ったんすが、アナルポートって…ちょっと…

あと、vQFX でも start shell して tcpdump とか使えますよ。Control Plane 宛のトラフィックだけとりたい時とかには。

DataPlane

bb01 で拾った VXLAN のパケットを見ていきます。

node11 から node21ping うった時、最初に出る ARP Request と ARP reply
VXLAN カプセルの IP ヘッダ的にはマルチキャストではなく、spine11 VTEPspine21 VTEP 間のユニキャストになってます。

f:id:kakkotetsu:20170511233328p:plain

f:id:kakkotetsu:20170511233339p:plain

node11node21 間の ICMP echo Request と ICMP echo Reply
これは EVPN 使おうが関係なく、ただの VXLAN ヘッダついたパケットです。

f:id:kakkotetsu:20170511233355p:plain

f:id:kakkotetsu:20170511233404p:plain

ControlPlane

bb01 で拾った EVPN のパケットを見ていきます。

EVPN NLRI Type2(MAC/IP Advertisement route) Update

まずは node11 からの ARP Request を受信した spine11 が advertise する様子
f:id:kakkotetsu:20170511233423p:plain

次に node21 からの ARP Reply を受信した spine21 が advertise したのを bb01spine11 方面に reflect する様子
f:id:kakkotetsu:20170511233441p:plain

EVPN NLRI Type2(MAC/IP Advertisement route) Withdrawn

spine11clear ethernet-switching table して node11MAC を消す
f:id:kakkotetsu:20170511233456p:plain

EVPN NLRI Type3(Inclusive Multicast Ethernet Tag route) Withdrawn

spine21 で以下のような感じで、雑に VLAN 100 = VNI10100 at RD 64512:21 が消えたことを advertise させた時の様子

{master:0}[edit]
kotetsu@spine21# deactivate vlans VLAN0100

{master:0}[edit]
kotetsu@spine21# show | compare
[edit vlans]
!    inactive: VLAN0100 { ... }

{master:0}[edit]
kotetsu@spine21# commit

f:id:kakkotetsu:20170511233515p:plain

次に spine21 で VLAN 100 = VNI10100 at RD 64512:21 ができたことを advertise させた時の様子

{master:0}[edit]
kotetsu@spine21# activate vlans VLAN0100

{master:0}[edit]
kotetsu@spine21# show | compare
[edit vlans]
!    active: VLAN0100 { ... }

{master:0}[edit]
kotetsu@spine21# commit

f:id:kakkotetsu:20170511233529p:plain

おしまい

  • 仮想版でも Juniper vQFX で L2VPN 機能が動くことを確認できました
  • linux kernel で EVPN しゃべるようになりませんかね(MPLS は入ったのだし)
    • 冒頭の絵でいうところの オレンジ領域(L3) は出来るだけエンドまで降りてきて欲しくないですか
      • Juniper の例示では、ToR スイッチ(QFX5100)まではきていますよね
      • そしたら linux kernel まで降りてきて、物理スイッチは MP-BGP の土管になれば…とか思ったわけです