Cisco WLC Software Upgrade via CLI

I had to upgrade a Cisco WLC 2504 from 8.2.166 to 8.2.170 but for some reason, my web browser (both IE and FF) doesn't prompt to reboot the WLC. So I performed a WLC software upgrade using the CLI instead.

Here's a link where I previously performed an upgrade on using the web GUI.

(Cisco Controller) >transfer download datatype code

(Cisco Controller) >transfer download mode ftp

(Cisco Controller) >transfer download serverip 172.27.5.2

(Cisco Controller) >transfer download filename AIR-CT2500-K9-8-2-170-0.aes

(Cisco Controller) >transfer download start

Mode............................................. FTP  

Data Type........................................ Code         

FTP Server IP.................................... 172.27.5.2

FTP Server Port.................................. 21

FTP Path......................................... ./

FTP Filename..................................... AIR-CT2500-K9-8-2-170-0.aes

FTP Username..................................... ftpuser

FTP Password..................................... *********

This may take some time.

Are you sure you want to start? (y/N) y

FTP Code transfer starting.

FTP receive complete... extracting components.

Image version check passed.

Executing backup script.

Writing new RTOS to flash disk.

Writing new FP to flash disk.

Writing new AP Image Bundle to flash disk.

Executing fini script.

File transfer is successful. 
Reboot the controller for update to complete. 
Optionally, pre-download the image to APs before rebooting to reduce network downtime.


(Cisco Controller) >config ap image predownload primary all      // PRE-DOWNLOAD IMAGE TO AP (LESS DOWNTIME)


(Cisco Controller) >reset system     // REBOOT WLC FOR NEW OS TO TAKE EFFECT


The system has unsaved changes.
Would you like to save them now? (y/N) y

Activating Cisco WLC 3504 AP License

Cisco has announced the End-of-Life (EoL) for the Cisco WLC 2504 this year so we're getting the new Cisco 3504 wireless controller. Below are photos of the WLC 3504 and this is a helpful link for the information on chassis ports and LEDs

I had to troubleshoot a brand new WLC 3504 and for some reason the 1852 AP wasn't able to join the wireless controller even though the country (for the AP) and date/time were correctly configured. The logs showed a looping CAPWAP State: DTLS Teardown error when consoled to one of the AP.

[*10/30/2018 02:57:49.0222] CAPWAP State: DTLS Teardown

[*10/30/2018 02:57:49.0322] Dropping dtls packet since session is not established. Peer 202.7.3.4-5246, Local 202.7.3.30-5264, conn (nil)

[*10/30/2018 02:57:53.8807]

[*10/30/2018 02:57:53.8807] CAPWAP State: Discovery

[*10/30/2018 02:57:53.8807] IP DNS query for CISCO-CAPWAP-CONTROLLER.local.net

[*10/30/2018 02:57:53.9206] Discovery Request sent to 202.7.3.4, discovery type STATIC_CONFIG(1)

[*10/30/2018 02:57:53.9206] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)

[*10/30/2018 02:57:53.9206] Discovery Response from 202.7.3.4

[*10/30/2018 02:57:53.9406] Discovery Response from 202.7.3.4

[*10/30/2018 02:58:03.0000]

[*10/30/2018 02:58:03.0000] CAPWAP State: DTLS Setup

[*10/30/2018 02:58:03.0399]

[*10/30/2018 02:58:03.0399] CAPWAP State: Join

[*10/30/2018 02:58:03.0399] Sending Join request to 202.7.3.4 through port 5264

[*10/30/2018 02:58:47.0262] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Join(5).

[*10/30/2018 02:59:00.0722]

[*10/30/2018 02:59:00.0722] CAPWAP State: DTLS Teardown

[*10/30/2018 02:59:00.0922] Dropping dtls packet since session is not established. Peer 202.7.3.4-5246, Local 202.7.3.30-5264, conn (nil)

There were no APs that have joined the WLC per checking on the Monitor dashboard and under Wireless > Access Points.

In order for the APs to join the WLC, you'll need to activate the ap_count license. Go to Management > Software Activation > Licenses > click ap_count (hyperlink).

Set the License Status to Activate (default) > click Set Status.

An End User License Agreement (EULA) page will pop-up. Click I Accept.

The APs started to join afterwards. The box comes with the AP Adder License (or Entitlement). You'll need to specify the AP count under MANAGEMENT > Software Activation > Licenses > License Count > Add > type a number > Set Count. You'll be prompted again to accept the EULA.


For this scenario, I ordered a WLC 3504 with 15 AP license.

The Permanent AP Count (Adder) license will appear afterwards.

[*10/30/2018 03:01:37.0299] CAPWAP State: Join

[*10/30/2018 03:01:37.0399] Sending Join request to 202.7.3.4 through port 5264

[*10/30/2018 03:01:37.0399] Join Response from 202.7.3.4

[*10/30/2018 03:01:37.1499]

[*10/30/2018 03:01:37.1499] CAPWAP State: Image Data

[*10/30/2018 03:01:37.1999] do NO_UPGRADE, part2 is active part

[*10/30/2018 03:01:37.1999]

[*10/30/2018 03:01:37.1999] CAPWAP State: Configure

[*10/30/2018 03:01:37.2099] DOT11_CFG[0] Radio Mode is changed from Local to Local

[*10/30/2018 03:01:37.2099] DOT11_CFG[1] Radio Mode is changed from Local to Local

[*10/30/2018 03:01:37.2799] DOT11_DRV[0]: Start Radio0

[*10/30/2018 03:01:37.2899] DOT11_DRV[0]: Stop Radio0

[*10/30/2018 03:01:37.2999] DOT11_DRV[0]: Start Radio0

[*10/30/2018 03:01:37.4598] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).

[*10/30/2018 03:01:37.4598] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).

[*10/30/2018 03:01:37.4598] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).

[*10/30/2018 03:01:37.4798] DOT11_DRV[0]: Stop Radio0

[*10/30/2018 03:01:37.4898] DOT11_DRV[0]: Start Radio0

[*10/30/2018 03:01:37.5498]

[*10/30/2018 03:01:37.5498] CAPWAP State: Run

[*10/30/2018 03:01:37.7198] AP has joined controller Cisco_03:ab:cd

Transferring Configuration File on a Cisco WLC

You can upload or download files to and from a Cisco WLC by going to Commands > Download File or Upload File. In this case, I needed to backup the Configuration file from WLC to a remote TFTP server.

To upload the Configuration file to a remote server, choose File Type: Configuration > optionally tick Configuration File Encryption > Transfer Mode: TFTP > type the TFTP server IP Address > type the File Path: ./ > type a File Name (BACKUP-1).

WLC will pop-up a warning. Click OK.

The config file transfer completed just within a few seconds.

Here's a snippet of the WLC configuration file.

# WLC Config Begin <Tue Feb 20 02:57:56 2018>

config wlan apgroup wlan-radio-policy BUSINESS_WIFI 2 none
config wlan apgroup add BUSINESS_WIFI
config wlan apgroup interface-mapping add BUSINESS_WIFI 2 corporate
config wlan apgroup qinq tagging eap-sim-aka BUSINESS_WIFI enable
config wlan dhcp_server 4 172.27.3.2 

<OUTPUT TRUNCATED>

transfer upload path ./
transfer upload serverip 172.27.5.17
transfer upload datatype config
transfer upload filename BACKUP-1

# WLC Config End <Tue Feb 20 02:58:02 2018>
 

Troubleshooting NTP Synchronization on a Cisco WLC

I encountered an NTP issue on a WLC wherein the APs couldn't join the WLC even though the country and NTP settings were correct. Troubleshooting WLC via CLI is more flexible compared using the GUI and by issuing the show time command, it showed the WLC time is incorrect and observed the NTP status flapped between Not Sycnched or In Progress.
 
(Cisco Controller) >show time

Time............................................. Mon Apr 14 10:53:34 2036

Timezone delta................................... 0:0
Timezone location................................

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------------------
       1              0                                63.12.7.2     Not Synched          AUTH DISABLED
       2              0                                10.12.3.4      In Progress            AUTH DISABLED


I tried removing and adding different NTP servers that are reachable. I even add the upstream router as NTP but the APs still couldn't join the WLC.

(Cisco Controller) >show time

Time............................................. Mon Apr 14 11:02:31 2036

Timezone delta................................... 0:0
Timezone location................................

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------------------
       1              0                               172.27.11.5     Not Synched         AUTH DISABLED   //ROUTER

(Cisco Controller) >ping 63.12.7.2

Send count=3, Receive count=3 from 63.12.7.2

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 14

Base Mac             AP EthernetMac       AP Name       IP Address         Status
1c:1d:86:30:31:23   N A                      ap10              172.27.11.6       Not Joined
1c:1d:86:30:52:34    N A                     ap11              172.27.11.7       Not Joined
34:62:88:0f:43:56    N A                     ap05              172.27.11.1       Not Joined
34:62:88:0f:44:78    N A                     ap06              172.27.11.2       Not Joined
50:67:ae:c3:25:99    N A                     ap04              172.27.11.10     Not Joined
50:67:ae:c3:36:11    N A                     ap07              172.27.11.3       Not Joined
5c:a4:8a:be:b7:22   N A                      ap02              172.27.11.8       Not Joined
5c:a4:8a:ee:a8:33    N A                     ap15              172.27.11.16      Not Joined
5c:a4:8a:ee:d9:44   N A                      ap01              172.27.11.17      Not Joined
5c:a4:8a:ee:11:55    N A                     ap08              172.27.11.4        Not Joined
bc:16:f5:3e:22:66    N A                     ap14              172.27.11.27      Not Joined
bc:16:f5:3e:33:77    N A                     ap12              172.27.11.18      Not Joined
f0:9e:63:c1:44:88    N A                     ap013            172.27.11.9        Not Joined
f0:9e:63:c1:55:99    N A                     ap03              172.27.11.11      Not Joined


Here's a snippet of the debug capwap events enable on the WLC. You can turn off the debug by issuing a debug capwap events disable command.

*spamApTask2: Apr 14 11:03:00.148: apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.148: apType = 28 apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.148: apType: Ox1c bundleApImageVer: 8.2.130.0
*spamApTask2: Apr 14 11:03:00.148: version:8 release:2 maint:130 build:0
*spamApTask2: Apr 14 11:03:00.149: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2 port 33154
*spamApTask2: Apr 14 11:03:00.149: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.150: 34:62:88:0f:41:23 Discovery Request from 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.151: 34:62:88:0f:41:23 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 25, joined Aps =0
*spamApTask2: Apr 14 11:03:00.151: apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.151: apType = 28 apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.152: apType: Ox1c bundleApImageVer: 8.2.130.0
*spamApTask2: Apr 14 11:03:00.152: version:8 release:2 maint:130 build:0
*spamApTask2: Apr 14 11:03:00.153: 34:62:88:0f:4e:90 Discovery Response sent to 172.27.11.2 port 33154
*spamApTask2: Apr 14 11:03:00.153: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.153: 34:62:88:0f:41:23 Discovery Request from 172.27.11.2:33154
*spamApTask2: Apr 14 11:03:00.154: 34:62:88:0f:41:23 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 25, joined Aps =0
*spamApTask2: Apr 14 11:03:00.154: apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.154: apType = 28 apModel: AIR-CAP1602E-A-K9
*spamApTask2: Apr 14 11:03:00.155: apType: Ox1c bundleApImageVer: 8.2.130.0
*spamApTask2: Apr 14 11:03:00.155: version:8 release:2 maint:130 build:0
*spamApTask2: Apr 14 11:03:00.155: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2 port 33154
*spamApTask2: Apr 14 11:03:00.156: 34:62:88:0f:41:23 Discovery Response sent to 172.27.11.2:33154
*spamApTask2: Apr 14 11:04:09.094: 58:f3:9c:b8:18:06 No AP entry exist in temporary database for 172.27.11.2:33154
*spamApTask2: Apr 14 11:04:15.276: 58:f3:9c:b8:18:06 DTLS connection not found, creating new connection for 172:27:11:2 (33154) 172:27:11:66 (5246)


To resolve, you have to force the NTP to sync by manually configure the date and time on the WLC under Commands > Set Time.

Remove the NTP server first by clicking on the blue arrow on the right > select Remove.

Re-add the NTP server by clicking New > type the NTP IP Address > Apply > Save Configuration.

The NTP status stayed In Sync and APs joined the WLC afterwards.

(Cisco Controller) >show time

Time............................................. Fri Feb 23 11:00:45 2018

Timezone delta................................... 0:0
Timezone location................................ (GMT +8:00) HongKong, Bejing, Chongquing

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------------------
       1              0                                        63.12.7.2                 In Sync              AUTH DISABLED

Configuring NTP Server on a Cisco WLC via CLI

You'll need to configure a Cisco WLC using the CLI in case the routing to the network where you HTTPS or access the GUI is unreachable. In my case only the management server is able to SSH remotely to the WLC.

You can verify if the AP has successfully joined the WLC using the show ap join stats summary all command.

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 5

Base Mac             AP EthernetMac       AP Name                 IP Address         Status
c4:b9:cd:1f:02:ab    N A                  AP2c5a.0f5a.fabc        10.16.25.24      Not Joined
c4:b9:cd:1f:08:cd   N A                  AP2c5a.0f5a.fdef         10.16.25.30      Not Joined
c4:b9:cd:1f:17:ef    N A                  AP2c5a.0f5a.f123       10.16.25.26      Not Joined
c4:b9:cd:1f:1c:12    N A                  AP2c5a.0f5a.f456       10.16.25.27      Not Joined
c4:b9:cd:1f:22:34    N A                  AP2c5a.0f5a.f789        10.16.25.25      Not Joined


Use the show time command to verify if an NTP server is configured and synchronized time on the WLC.

(Cisco Controller) >show time

Time............................................. Sat Jan  1 11:54:25 2000     // DATE AND YEAR NOT SYNC'D
Timezone delta................................... 0:0
Timezone location................................ (GMT +8:00) HongKong, Bejing, Chongquing

NTP Servers
    NTP Polling Interval.........................     3600

     Index     NTP Key Index                  NTP Server                  NTP Msg Auth Status
    -------  ----------------------------------------------------------------------------------
       1              0                                63.12.7.2                           AUTH DISABLED


I tried to ping the configured NTP server and Google's public NTP server but it's not reachable.

(Cisco Controller) >ping 63.12.7.2    

Send count=3, Receive count=0 from 63.12.7.2

(Cisco Controller) >ping 216.239.35.4      // GOOGLE NTP

Send count=3, Receive count=0 from 216.239.35.4


I've configured another internal NTP server that is reachable from the site using the config time ntp server command.

(Cisco Controller) >ping 10.12.3.4

Send count=3, Receive count=3 from 10.12.3.4

(Cisco Controller) >config time ?
              
manual         Configures the system time.
ntp            Configures the Network Time Protocol.
timezone       Configures the system's timezone.
              
(Cisco Controller) >config time ntp ?
              
auth           Configures the NTP authentication
interval       Configures the Network Time Protocol Polling Interval.
key-auth       Configures the NTP authentication key.
server         Configures the Network Time Protocol Servers.
              
(Cisco Controller) >config time ntp server ?
              
<index>        Enter NTP server index.
              
(Cisco Controller) >config time ntp server 1 ?
              
<IP Address>   Enter NTP server's IP address. Use 0.0.0.0 to delete entry
              
(Cisco Controller) >config time ntp server 1 10.12.3.4


The DTLS tunnel on the AP were established and it was able to join the WLC.

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 5

Base Mac             AP EthernetMac       AP Name                 IP Address         Status
c4:b9:cd:1f:02:ab    N A                  AP2c5a.0f5a.fabc        10.16.25.24         Joined
c4:b9:cd:1f:08:cd   N A                  AP2c5a.0f5a.fdef         10.16.25.30         Joined
c4:b9:cd:1f:17:ef    N A                  AP2c5a.0f5a.f123       10.16.25.26          Joined
c4:b9:cd:1f:1c:12    N A                  AP2c5a.0f5a.f456       10.16.25.27         Joined
c4:b9:cd:1f:22:34    N A                  AP2c5a.0f5a.f789        10.16.25.25        Joined


(Cisco Controller) >save config       // SAVE THE CONFIG IN NVRAM

Are you sure you want to save? (y/n) y

Configuration Saved!