Thursday, March 1, 2018

Troubleshooting NTP Synchronization on a Cisco WLC

I encountered an NTP issue on a WLC wherein the APs couldn't join the WLC even though the country and NTP settings were correct. Troubleshooting WLC via CLI is more flexible compared using the GUI and by issuing the show time command, it showed the WLC time is incorrect and observed the NTP status flapped between Not Sycnched or In Progress.
 
(Cisco Controller) >show time

Time............................................. Mon Apr 14 10:53:34 2036


Timezone delta................................... 0:0
Timezone location................................

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------------------
       1              0                                63.12.7.2     Not Synched          AUTH DISABLED
       2              0                                10.12.3.4      In Progress            AUTH DISABLED



I tried removing and adding different NTP servers that are reachable. I even add the upstream router as NTP but the APs still couldn't join the WLC.

(Cisco Controller) >show time

Time............................................. Mon Apr 14 11:02:31 2036

Timezone delta................................... 0:0
Timezone location................................

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------------------
       1              0                               172.27.11.5     Not Synched         AUTH DISABLED   //ROUTER

(Cisco Controller) >ping 63.12.7.2

Send count=3, Receive count=3 from 63.12.7.2

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 14

Base Mac             AP EthernetMac       AP Name       IP Address         Status
1c:1d:86:30:31:23   N A                      ap10              172.27.11.6       Not Joined
1c:1d:86:30:52:34    N A                     ap11              172.27.11.7       Not Joined
34:62:88:0f:43:56    N A                     ap05              172.27.11.1       Not Joined
34:62:88:0f:44:78    N A                     ap06              172.27.11.2       Not Joined
50:67:ae:c3:25:99    N A                     ap04              172.27.11.10     Not Joined
50:67:ae:c3:36:11    N A                     ap07              172.27.11.3       Not Joined
5c:a4:8a:be:b7:22   N A                      ap02              172.27.11.8       Not Joined
5c:a4:8a:ee:a8:33    N A                     ap15              172.27.11.16      Not Joined
5c:a4:8a:ee:d9:44   N A                      ap01              172.27.11.17      Not Joined
5c:a4:8a:ee:11:55    N A                     ap08              172.27.11.4        Not Joined
bc:16:f5:3e:22:66    N A                     ap14              172.27.11.27      Not Joined
bc:16:f5:3e:33:77    N A                     ap12              172.27.11.18      Not Joined
f0:9e:63:c1:44:88    N A                     ap013            172.27.11.9        Not Joined
f0:9e:63:c1:55:99    N A                     ap03              172.27.11.11      Not Joined


Here's a snippet of the debug capwap events enable on the WLC. You can turn off the debug by issuing a debug capwap events disable command.

*spamApTask2: Apr 14 11:03:00.148: apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.148: apType = 28 apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.148: apType: Ox1c bundleApImageVer: 8.2.130.0
*spamApTask2: Apr 14 11:03:00.148: version:8 release:2 maint:130 build:0
*spamApTask2: Apr 14 11:03:00.149: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2 port 33154
*spamApTask2: Apr 14 11:03:00.149: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.150: 34:62:88:0f:41:23 Discovery Request from 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.151: 34:62:88:0f:41:23 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 25, joined Aps =0
*spamApTask2: Apr 14 11:03:00.151: apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.151: apType = 28 apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.152: apType: Ox1c bundleApImageVer: 8.2.130.0
*spamApTask2: Apr 14 11:03:00.152: version:8 release:2 maint:130 build:0
*spamApTask2: Apr 14 11:03:00.153: 34:62:88:0f:4e:90 Discovery Response sent to 172.27.11.2 port 33154
*spamApTask2: Apr 14 11:03:00.153: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.153: 34:62:88:0f:41:23 Discovery Request from 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.154: 34:62:88:0f:41:23 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 25, joined Aps =0
*spamApTask2: Apr 14 11:03:00.154: apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.154: apType = 28 apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.155: apType: Ox1c bundleApImageVer: 8.2.130.0
*spamApTask2: Apr 14 11:03:00.155: version:8 release:2 maint:130 build:0
*spamApTask2: Apr 14 11:03:00.155: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2 port 33154
*spamApTask2: Apr 14 11:03:00.156: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2:33154
*spamApTask2: Apr 14 11:04:09.094: 58:f3:9c:b8:18:06 No AP entry exist in temporary database for 172.27.11.2:33154
*spamApTask2: Apr 14 11:04:15.276: 58:f3:9c:b8:18:06 DTLS connection not found, creating new connection for 172:27:11:2 (33154) 172:27:11:66 (5246)



To resolve, you have to force the NTP to sync by manually configure the date and time on the WLC under Commands > Set Time.


Remove the NTP server first by clicking on the blue arrow on the right > select Remove.


Re-add the NTP server by clicking New > type the NTP IP Address > Apply > Save Configuration.


The NTP status stayed In Sync and APs joined the WLC afterwards.

(Cisco Controller) >show time

Time............................................. Fri Feb 23 11:00:45 2018

Timezone delta................................... 0:0
Timezone location................................ (GMT +8:00) HongKong, Bejing, Chongquing

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------------------
       1              0                                        63.12.7.2                 In Sync              AUTH DISABLED