Making RSVP work over DVMPN

When using RSVP Call Admission Control (CAC) for VoIP, DMVPN and RSVP have limitations that prevent RSVP from working over DMVPN. If you have VoIP and you can’t use location based CAC, RSVP is the only answers. So what’s the problem with RSVP over DMVPN? The root of the problem is RSVP’s loop prevention mechanism. In this post I’ll describe an original solutions to make RSVP CAC work over DMVPN.

RSVP Split-Horizon on Tunnels

RSVP has a little known behavior used for loop prevention. It is similar to the split-horizon rule of many Distance Vector routing protocols and is described in RFC2205:

“[S]tate that is received through a particular interface must never be forwarded out the same interface.” ^[1]

When RSVP is set to be mandatory for call setup between two locations, RSVP has to successfully establish a reservation for each one way RTP audio stream. That reservation is done by voice gateways acting as the RSVP agents for IP phones. IP phones do not have RSVP running, but rely on voice gateways for that functionality.

Normally, when using DMVPN Phase 3, the initial packets sent between two spoke sites, match a route with the next-hop of the DMVPN hub and are sent first to the hub [2]. The hub routes these packets to the destination spoke and sends a NHRP redirect message which initiates the spoke-to-spoke NHRP resolution request to resolve external addressing. Once external addressing is resolve, IPSec tunnels are negotiated between the spokes. The important detail is that resolution requests for spokes are sent after the initial packet traverses the hub and leaves the same mGRE interface. Without NHRP resolution IPSec tunnels can’t be established.

With RSVP, when a call is placed, the RSVP PATH message should be routed to the hub. Then these messages should be send back out the tunnel interface to the destination spoke and trigger the NHRP redirect message. Once the spoke receives a redirect message it starts the resolution request process, which in effect established the spoke-to-spoke tunnel. The RSVP loop prevention mechanism prevents this behavior. It stops the triggering of the NHRP redirect message.

What happens here is that when the initial packet is a RSVP PATH message, the RSVP’s loop prevention mechanism stops forwarding it out the same mGRE interface on the hub. DMVPN hub is not able to send the RSVP message and because of that it is not able to send the NHRP redirect message and consequently establish the tunnel. VoIP calls fail and all successive calls fail as well.

For the remainder of this post, I’ll use output from the below topology :

How do you resolve this issue? There is no way to disable RSVP loop prevention in IOS. What if the next-hop wasn’t the hub, but rather the spoke itself, just like in DMVPN phase 2? Would that eliminate the loop prevention problem? This can be accomplished with a simple command on the d-ro03 hub’s tunnel:

 d-ro03(config-if) # no ip next-hop-self eigrp 90

One additional setting prevented the call setup and testing. By default, the CallManger has a RSVP Response Timer which limits RSVP reservation setup to 2 seconds. After two seconds, the RSVP tear-down messages are sent from voice gateways. This maximum timer can be set to 5 seconds. There is one way to disable it by setting it to 0 seconds. When this timers is disabled, each voice gateway can have its own timer set up to 60 seconds with the command “call rsvp-sync resv-timer 60”. Unless this option is disabled, it is hard to see what is really going on and figure out the solutions. For demonstration purpose, I set the call setup to the maximum timer of 60 seconds. This timer might have to be tuned depending on network’s WAN latency.

To test this solution I looked at two debug message: “debug nhrp packet” and “debug ip rsvp messages”. These debugs only show the actual messages sent and received without any processing data. For simplicity only one way path reservation is displayed, but both direction were reserved.

With DMVPN routes pointing directly to the spokes, not the hub, the original RSVP PATH packet triggers a NHRP resolution request. During the resolution, packets are forwarded through the hub but still run into the RSVP loop prevention.

The DMVPN Spoke receives a RSVP PATH from the voice gateway and routes out the tunnel 2 interface. The next-hop is set directly to the spokes’s tunnel interface instead of the hub (NHOP 10.196.1.222).

 

a-ro02#21:23:49.044: RSVP: session [TBD]
Incoming Path, I/F=Gi0/0, Layer=IP, Link=IPv4
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, Len=236, TTL=254, RA=Y
RSVP HDR: RRC=N, TTL=255, Len=236, Cksum=0xFE84

a-ro02#21:23:49.056: RSVP: session 10.255.222.1_18080[0.0.0.0]:
Outgoing Path, I/F=Tu2, Layer=IP, Link=IPv4, NHOP=10.196.1.222, Prerouted=Y
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, V6_FL=0, Len=260, TTL=254, RA=Y
RSVP HDR: RRC=N, TTL=254, Len=236, Cksum=0x6EDF

The DVMPN Hub receives the PATH message, but cannot forward it out due to the loop prevention mechanism.

d-ro03#21:23:49.061: RSVP: session [TBD]
Incoming Path, I/F=Tu2, Layer=IP
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, Len=260, TTL=253, RA=Y
RSVP HDR: RRC=N, TTL=254, Len=236, Cksum=0x6EDF

Loop prevention debug message.

d-ro03#21:23:49.065: RSVP: can't forward Path out received interface

Even thought the hub is dropping the initial packets, the originating spoke sends out the NHRP Resolution Requests for the destination spoke’s next-hop. It does not know the external IP of the next-hop and sends the request to the NHS. The resolution request comes back with the address 11.2.24.2.

a-ro02#

21:23:49.068: NHRP: Send Resolution Request via Tunnel2 vrf 0, packet size: 87
21:23:49.068:  src: 10.196.1.221, dst: 10.196.1.3
21:23:49.068:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
21:23:49.068:      shtl: 4(NSAP), sstl: 0(NSAP)
21:23:49.068:      pktsz: 87 extoff: 52
21:23:49.068:  (M) flags: "router auth src-stable nat ", reqid: 150
21:23:49.068:      src NBMA: 11.1.24.2
21:23:49.068:      src protocol: 10.196.1.221, dst protocol: 10.196.1.222
21:23:49.068:  (C-1) code: no error(0)
21:23:49.068:        prefix: 32, mtu: 17916, hd_time: 180
21:23:49.068:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0,pref:0

a-ro02#
21:23:49.080: NHRP: Receive Resolution Reply via Tunnel2 vrf 0, packet size: 115
21:23:49.080:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
21:23:49.080:      shtl: 4(NSAP), sstl: 0(NSAP)
21:23:49.080:      pktsz: 115 extoff: 60
21:23:49.080:  (M) flags:"router auth dst-stable unique src-stable nat",reqid: 150
21:23:49.080:      src NBMA: 11.1.24.2
21:23:49.080:      src protocol: 10.196.1.221, dst protocol: 10.196.1.222
21:23:49.080:  (C-1) code: no error(0)
21:23:49.080:        prefix: 32, mtu: 17916, hd_time: 69
21:23:49.080:        addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4,pref:0
21:23:49.080:        client NBMA: 11.2.24.2
21:23:49.080:        client protocol: 10.196.1.222

Thirty seconds pass and the call has not been setup yet, even through the tunnels were established. The original PATH message was sent at 21:23:49.056 and the next PATH refresh is sent at 21:24:21.096

a-ro02#
21:24:21.096: RSVP: session 10.255.222.1_18080[0.0.0.0]:
Outgoing Path, I/F=Tu2, Layer=IP, Link=IPv4, NHOP=10.196.1.222, Prerouted=Y
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, V6_FL=0, Len=260, TTL=254, RA=Y
RSVP HDR: RRC=N, TTL=254, Len=236, Cksum=0x6EDF

Finally a RESV message comes back from the spoke on the tunnel 2’s interface directly from the destination spoke and not the hub.

a-ro02#
21:24:21.104: RSVP: session [TBD]
Incoming Resv, I/F=Tu2, Layer=IP, Link=IPv4
IP   HDR: 10.196.1.222->10.196.1.221, TOS=0x60, Len=196, TTL=253, RA=N
RSVP HDR: RRC=N, TTL=255, Len=196, Cksum=0xA277

The RSVP RESV message is sent out the LAN interface towards the voice gateway.

a-ro02#
21:24:21.108: RSVP: session 10.255.222.1_18080[0.0.0.0]:
Outgoing Resv, I/F=Gi0/0, Layer=IP, Link=IPv4, NHOP=10.221.1.5, Prerouted=N
IP   HDR: 10.221.1.2->10.221.1.5, TOS=0x60, V6_FL=0, Len=216, TTL=255, RA=N
RSVP HDR: RRC=N, TTL=255, Len=196, Cksum=0x0000

The call setup took 30 seconds, which is excessively long. The 30-second timer that controlled it was the RSVP state refresh timer[3]. It is used to ensure that the reservation is still valid. The minimal interval for this timer can be set to 5 seconds, which is still too long. In this case the 5 seconds is the time a caller has to wait before a ring tone is heard.

This time the tunnel is setup, but the problem is that initial RSVP PATH message is dropped at the hub and another one comes after 30 seconds or minimum 5 seconds, which is still too long. Consecutive calls don’t have any delay setting up the RTP stream.

Is there way to send another PATH message right away or retransmit the first one? The next section describes the solutions using RSVP Refresh Reduction and Reliable Messaging.

 

RSVP Refresh Reduction and Reliable Messaging

The second part of the solution to the loop prevention rule is RSVP Refresh Reduction and Reliable Messaging[4]. This feature requests RSVP ACKs for each message, in a similar matter as the TCP protocol. Reliable RSVP Messaging was introduced by Cisco[5] in IOS version 12.0(26)S. It will attempt to use this feature per each hop and revert if unsupported. To enable use the command “ip rsvp signalling refresh reduction”. One parameter was tuned in the lab “ip rsvp signalling initial-retransmit-delay 750” which corresponds to the retransmit timer. Retransmission delay doubles until an ACK is received or it exceeds the refresh interval. For this to work the spokes have this feature enabled, while the hubs do not. If the hubs have it enabled the initial messages will have to wait to be flushed from retransmitting attempts and delay call setup even longer. Not enabling refresh reduction on the hub was the key to establishing the RTP stream within less than a second.

Debugs used for these outputs were “debug nhrp packet” and “debug ip rsvp messages”.

DMVPN spoke a-ro02 receives a RSVP Path message from the RSVP agent 10.255.221.1

a-ro02#Jul 23 15:12:55.568: RSVP: session [TBD]
Incoming Path, I/F=Gi0/0, Layer=IP, Link=IPv4
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, Len=248, TTL=254, RA=Y
RSVP HDR: RRC=Y, TTL=255, Len=248, Cksum=0xAF4C

A-ro02 processes the message and forwards it directly to the spoke, which is routed over the hub.

a-ro02#Jul 23 15:12:55.580: RSVP: session 10.255.222.1_18610[0.0.0.0]:
Outgoing Path, I/F=Tu2, Layer=IP, Link=IPv4, NHOP=10.196.1.222, Prerouted=Y
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, V6_FL=0, Len=272, TTL=254, RA=Y
RSVP HDR: RRC=Y, TTL=254, Len=248, Cksum=0xF00A

The hub d-ro03 receives it but does not support RSVP (unknown object (23)), per the design. It sends back a RSVP PathError message.

d-ro03#Jul 23 15:12:55.581: RSVP: session [TBD]
Incoming Path, I/F=Tu2, Layer=IP
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, Len=272, TTL=253, RA=Y
RSVP HDR: RRC=Y, TTL=254, Len=248, Cksum=0xF00A

d-ro03#Jul 23 15:12:55.585: RSVP: Found unknown object (23) when parsing message

d-ro03#Jul 23 15:12:55.585: RSVP: session 10.255.222.1_18610[0.0.0.0]:
Outgoing PathError, I/F=Tu2, Layer=IP, NHOP=10.196.1.221, Prerouted=N
IP   HDR: 10.196.1.3->10.196.1.221, TOS=0x00, Len=148, TTL=255, RA=N
RSVP HDR: RRC=N, TTL=255, Len=128, Cksum=0x8F50

A-ro02’s initial Path message triggered a NHRP Resolution Request and d-ro03 responds creating the spoke-to-spoke tunnel.

Jul 23 15:12:55.592: NHRP: Send Resolution Request via Tunnel2 vrf 0,
Jul 23 15:12:55.592:  src: 10.196.1.221, dst: 10.196.1.3
Jul 23 15:12:55.592:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
Jul 23 15:12:55.592:      shtl: 4(NSAP), sstl: 0(NSAP)
Jul 23 15:12:55.592:      pktsz: 8 extoff: 52
Jul 23 15:12:55.592:  (M) flags: "router auth src-stable nat ", reqid: 20
Jul 23 15:12:55.592:      src NBMA: 11.1.24.2
Jul 23 15:12:55.592:      src protocol: 10.196.1.221, dst protocol: 10.196.1.222
Jul 23 15:12:55.592:  (C-1) code: no error(0)
Jul 23 15:12:55.592:        prefix: 32, mtu: 17916, hd_time: 180
Jul 23 15:12:55.592:        addr_len:0(NSAP),subaddr_len:0(NSAP),proto_len:0

Jul 23 15:12:55.592: NHRP: Receive Resolution Reply via Tunnel2 vrf 0
Jul 23 15:12:55.592:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
Jul 23 15:12:55.592:      shtl: 4(NSAP), sstl: 0(NSAP)
Jul 23 15:12:55.596:      pktsz: 115 extoff: 60
Jul 23 15:12:55.596:  (M) flags:"router auth dst-stable unique src-stable nat"
Jul 23 15:12:55.596:      src NBMA: 11.1.24.2
Jul 23 15:12:55.596:      src protocol: 10.196.1.221, dst protocol: 10.196.1.222
Jul 23 15:12:55.596:  (C-1) code: no error(0)
Jul 23 15:12:55.596:        prefix: 32, mtu: 17916, hd_time:169
Jul 23 15:12:55.596:        addr_len:4(NSAP),subaddr_len:0(NSAP),proto_len:4
Jul 23 15:12:55.596:        client NBMA: 11.2.24.2
Jul 23 15:12:55.596:        client protocol: 10.196.1.222

Because A-ro02 received the PathError, it resends the original Path message but without the reliable object request attached. This message is sent directly to the spoke since the tunnel was already created.

a-ro02#Jul 23 15:12:56.332: RSVP: session 10.255.222.1_18610[0.0.0.0]:
Outgoing Path, I/F=Tu2, Layer=IP, Link=IPv4, NHOP=10.196.1.222, Prerouted=Y
IP   HDR: 10.255.221.1->10.255.222.1, TOS=0x60, V6_FL=0, Len=260, TTL=254, RA=Y
RSVP HDR: RRC=Y, TTL=254, Len=236, Cksum=0xB14D

The delay added to error out the first message is only 752 ms in this particular network and but nowhere close to the 5 seconds or 30 seconds as with the previous configuration. Almost immediately the RSVP Resv message arrive and the call’s reservation is established.[6]

a-ro02#Jul 23 15:12:56.336: RSVP: session [TBD]
Incoming Resv, I/F=Tu2, Layer=IP, Link=IPv4
IP   HDR: 10.196.1.222->10.196.1.221, TOS=0x60, Len=208, TTL=253, RA=N
RSVP HDR: RRC=Y, TTL=255, Len=208, Cksum=0xA926

a-ro02#Jul 23 15:12:56.344: RSVP: session 10.255.222.1_18610[0.0.0.0]:
Outgoing Resv, I/F=Gi0/0, Layer=IP, Link=IPv4, NHOP=10.221.1.5, Prerouted=N
IP   HDR: 10.221.1.2->10.221.1.5, TOS=0x60, V6_FL=0, Len=240, TTL=255, RA=N
RSVP HDR: RRC=N, TTL=255, Len=220, Cksum=0x0000

Verifying the configuration, the interface on spokes have Refresh Reduction and Reliable Messaging enabled on the tunnel interface, but the hub neighbor 10.196.1.3 is not capable of it.

a-ro02#sh ip rsvp neighbor detail
Neighbor: 10.196.1.3 (Local 10.196.1.221)
Encapsulation: Raw IP
Rate-Limiting:
Dropped messages: 0
Refresh Reduction: configured
Refresh reduction capable: no
Reliable messaging in use: no
Remote epoch: 0xF27749
Out of order messages: 0
Retransmitted messages: 0
Highest rcvd message id: 0x7
<..>