Friday, August 27, 2021

Cisco WLC DTLS Failed Due to Expired Manufacturer Installed Certificate (MIC)

I was troubleshooting a Cisco WLC configured as FlexConnect (or H-REAP) to a remote AP 1242 that wouldn't join. You can view WLC logs under Management > Logs > Message Logs.

*spamApTask2: Aug 26 13:37:28.876: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.10.14 for AP d4:6d:50:88:12:34

*spamApTask3: Aug 26 13:36:13.824: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.10.14 for AP d4:6d:50:88:12:34

*spamApTask2: Aug 26 13:35:03.936: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 192.168.10.14 for AP d4:6d:50:88:12:34

<OUTPUT TRUNCATED>

 

 

(Cisco Controller) >show ap join stats summary all

 

Number of APs.............................................. 1

 

Base Mac             AP EthernetMac       AP Name                 IP Address         Status

00:3a:99:12:ab:cd    58:8d:09:03:12:34   ap01               192.168.10.14     Not Joined

 

Since DTLS need accurate date and time, I tried to manually configure the WLC date/time, timezone under Commands > Set Time and also removed NTP but still no joy.

I also tried to upgrade the WLC AireOS 7.4.121.0 > 7.4.150.0 but still AP won't join the WLC.


My search led me to an expired WLC Manufacturer Installed Certificate (MIC) which has reached its 10 year expiration date. Below are some WLC commands as a workaround.

 

(Cisco Controller) >config ap ?

              

802.1Xuser     Configures the dot1x user for AP.

SSH            Enables/Disables SSH.

add            Adds a Foreign Access Point.

bhrate         Configures Cisco Bridge Backhaul Tx Rate.

bridgegroupname Sets/Deletes bridge group name.

bridging       Enables/Disables Ethernet-to-Ethernet bridging.

cdp            Enable/Disable CDP on Cisco AP.

cert-expiry-ignore Configures cert-expiry-ignore check operation.

core-dump      Configures the AP's memory core dump.

country        Configures the country of operation.

crash-file     Manages crash data and radio core files.

delete         Deletes a Foreign Access Point.

disable        Disables Cisco APs

enable         Enables Cisco APs

ethernet       Configures Ethernet Port of the AP

flexconnect    Enables/Disables VLAN on the flexconnect.

group-name     Configures the group name.

hotspot        Configures Hotspot configs on the AP.

image          Configure image.

led-state      Enables/Disables the LED-State or configure LED flash

link-encryption Capwap Data Link encryption options

 

--More-- or (q)uit

link-latency   Configures Link Latency Feature.

location       Configures the Location.

logging        Configures the AP logging.

mgmtuser       Configures the user for AP management.

mode           Configures an AP's  mode of operation.

monitor-mode   Configures an AP's monitor-mode channel optimization.

name           Configures the Name of an AP.

packet-dump    Configure AP Packet Capture parameters

port           Configures the port for a Foreign Access Point.

power          Configures Cisco Power over Ethernet (PoE) feature for an AP

primary-base   Configures the Primary Cisco Switch for an AP.

priority       Configure the AP's priority.

reporting-period Configures the AP's rogue/error reporting period

reset          Resets an AP

retransmit     Configures AP Control packet retransmission parameters.

role           Configures an AP's  Bridge role of operation.

rst-button     Enables or disables the Reset Button.

secondary-base Configures the Secondary Cisco Switch for an AP.

sniff          Enables/Disables sniffing on a radio.

static-IP      Enables/Disables/Changes an AP's static IP address configuration

stats-mode     Configures the mode(realtime vs normal) in which statistics are sent from an AP

stats-timer    Configures the frequency at which statistics are sent from an AP

syslog         Configures the system logging settings for an AP

 

--More-- or (q)uit

tcp-mss-adjust Configures TCP on Cisco AP.

telnet         Enables/Disables telnet.

tertiary-base  Configures the Tertiary Cisco Switch for an AP.

tftp-downgrade Initiates AP's image downgrade from a TFTP server

              

(Cisco Controller) >config ap cert-expiry-ignore ?

              

mic            Configures cert-expiry-ignore check operation for MIC.

ssc            Configures cert-expiry-ignore check operation for SSC.

              

(Cisco Controller) >config ap cert-expiry-ignore mic ?

              

enable         Enabling will ignore the lifetime-check for MIC.

disable        Disabling will do the lifetime-check for MIC.

              

(Cisco Controller) >config ap cert-expiry-ignore mic enable

 

(Cisco Controller) >config ap cert-expiry-ignore ssc ?

              

enable         Enabling will ignore the lifetime-check for SSC.

disable        Disabling will do the lifetime-check for SSC.

 

(Cisco Controller) >config ap cert-expiry-ignore ssc enable

 

(Cisco Controller) >config auth-list ?

              

add            Creates an authorized AP entry.

ap-policy      Configures an AP authorization policy.

delete         Delete an existing AP entry.

              

(Cisco Controller) >config auth-list ap-policy ?

                         

authorize-ap   Configures AP authorization policies.

authorize-lsc-ap Use auth-list to Authorize APs with Locally Significant Certificate.

ssc            Configures authorization of APs with self-signed certificate.

mic            Configures authorization of APs with manufacturing-installed certificates.

lsc            Configures authorization of APs with locally significant certificates.

              

(Cisco Controller) >config auth-list ap-policy ssc ?

              

enable         Allows APs with self-signed certificates to connect.

disable        Disallows APs with self-signed certificates to connect.

              

(Cisco Controller) >config auth-list ap-policy ssc enable

 

(Cisco Controller) >config certificate ?

              

compatibility  Configure certificate compatibility mode.

generate       Generates new certificates.

lsc            Configure Locally Significant Certificates (LSC)

ssc            Configure Self Signed Certificates (SSC)

use-device-certificate Use device certificate.

              

(Cisco Controller) >config certificate ssc ?

              

hash           Configure Self Signed Certificates hash

              

(Cisco Controller) >config certificate ssc hash ?

              

validation     Configures validation of SSC Hash

                      

(Cisco Controller) >config certificate ssc hash validation ?

 

enable         Enable hash validation of SSC certificate

              

disable        Disable hash validation of SSC certificate

              

(Cisco Controller) >config certificate ssc hash validation enable

 

(Cisco Controller) >save config

 

Are you sure you want to save? (y/n) y

 

Configuration Saved!

 

After the said commands were configured, the AP 1242 joined the WLC again.

(Cisco Controller) >show ap join stats summary all

 

Number of APs.............................................. 1

 

Base Mac             AP EthernetMac       AP Name                 IP Address         Status

00:3a:99:12:ab:cd    58:8d:09:03:12:34   ap01               192.168.10.14     Joined    

 

Saturday, April 17, 2021

Manual Radio Channel Assignment in a Cisco WLC

It's rainy season here in Singapore (April 2021) and I was craving for some Japanese food. So I went to Don Don Donki in Orchard Central and they sell a wide range of Japanese products like snacks, sweets, drinks, bento set, sushi, etc.



These are the items I brought home: Dango balls (sweet soy), California maki roll, Takoyaki (octopus) balls, Katsudon pork and sweet potato. They all taste authentic and almost have the same quality from a Japanese restaurant.

I headed over to Menbaka Fire Ramen in Cineleisure Orchard, which is a famous Ramen shop in Kyoto, Japan. They use burning negi (green) onion oil which creates the fire effect when poured into the bowl. This brings out the flavor of the green onions. The restaurant allow diners to seat in front of the kitchen to experience the "theatrical" fire display.


I ordered the Shoyu Fire Ramen and Ocha tea (cold) which is perfect for the rainy weather.

Here's a nice link regarding the Radio Resource Management (RRM) in a Cisco WLC. You can manually assign or override the AP's radio channel in a Cisco WLC under Wireless > Access Points > Radios > select the specific 802.11 radio (in this case 802.11b/g/n) > click the blue arrow icon (far right) on the specific AP (ap03 in this case) > click Configure.

Under RF Channel Assignment > Assignment Method > select Custom > select the specific channel (ap03 current channel is 1 so I chose channel 6) > click Apply > Save Configuration.

 

The ranges are as follows:
802.11a - 36, 40, 44, 48, 52, 56, 60, 64, 100, 104, 108, 112, 116, 132, 136, 140, 149, 153, 157, 161, 165, 190, 196

802.11b/g - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11

The defaults are as follows:
802.11a - 36, 40, 44, 48, 52, 56, 60, 64, 100, 104, 108, 112, 116, 132, 136, 140, 149, 153, 157, 161 

802.11b/g - 1, 6, 11

It's recommend that you use only non-overlapping channels: i.e. 1, 6, 11, and so on.

 

Monday, February 22, 2021

Cisco WLC 2504 Field Upgrade Software (FUS) 2.0

I needed to upgrade a Cisco WLC 2504 to AireOS version 8.5 and there's a note on the Cisco.com download site that a Field Upgrade Software (FUS) version 1.9 or higher is needed.

Cisco 2500 Series Wireless Controllers Release 8.5 Software.In order to use 8.4 or higher code, you must upgrade the 2504 Wireless Lan Controllers to FUS version 1.9 or higher, this must be done before installing the new AireOS version.

There's a separate link or download area for the WLC FUS firmware.

You can't find the FUS version in the WLC web GUI and the only way to get this info is via the CLI.

You use the WLC show sysinfo command to get the FUS version info. Look under the Bootloader Version (1.0.20) and Firmware Version (PIC 20.0). Since I'm running the latest FUS version on the WLC (as of this writing), I don't need to perform an FUS upgrade. 

Refer to Table 2 on this link for the FUS release notes. This link provides the steps in performing an FUS upgrade.

 

(Cisco Controller) >show sysinfo

 

Manufacturer's Name.............................. Cisco Systems Inc.

Product Name..................................... Cisco Controller

Product Version.................................. 8.2.130.0

Bootloader Version............................... 1.0.20

Field Recovery Image Version..................... 7.6.101.1

Firmware Version................................. PIC 20.0

 

Build Type....................................... DATA + WPS

 

System Name...................................... WLC01

System Location..................................

System Contact...................................

System ObjectID.................................. 1.3.6.1.4.1.9.1.1279

IP Address....................................... 192.168.7.10

IPv6 Address..................................... ::

Last Reset....................................... Software reset

System Up Time................................... 0 days 1 hrs 41 mins 35 secs

System Timezone Location.........................

System Stats Realtime Interval................... 5

System Stats Normal Interval..................... 180

 

Configured Country............................... SG  - Singapore

Operating Environment............................ Commercial (0 to 40 C)

Internal Temp Alarm Limits....................... 0 to 65 C

Internal Temperature............................. +47 C

External Temperature............................. +50 C

Fan Status....................................... 3800 rpm

 

State of 802.11b Network......................... Enabled

State of 802.11a Network......................... Enabled

Number of WLANs.................................. 0

Number of Active Clients......................... 0

 

Burned-in MAC Address............................ DC:EB:94:95:12:34

Maximum number of APs supported.................. 75

System Nas-Id.................................... WLC01

WLC MIC Certificate Types........................ SHA1/SHA2


Sunday, December 29, 2019

Configuring Application Visibility and Control (AVC) on a Cisco WLC

Cisco Application Visibility and Control (AVC) provides network visibility of classified traffic and gives the admin the option to control either by performing a Drop or Mark (DSCP) action. AVC utilizes several components such as Network Based Application Recognition (NBAR2), Quality of Service (QoS) and NetFlow which allows deep-packet inspection.

I had to block video streaming as well as block smartphone updates on a Cisco WLC 3504 in order to conserve bandwidth on a particular SSID. To configure AVC, to to WIRELESS > Application Visibility and Control > Applications.


Add a new AVC Profile under WIRELESS > Application Visibility and Control > AVC Profiles > click New.


Type the AVC Profile Name: BLOCK_STREAMING > click Apply.


Click on the AVC Profile (hyperlink): BLOCK_STREAMING.

Click Add New Rule (far right).

Select Application Group: voice-and-video > Application Name: youtube > Action: Drop > Apply.




Apply the AVC Profile under WLANs > select a WLAN ID > QoS > enable Application Visibility > select the created AVC Profile > Apply > Save Configuration. 


To monitor AVC, go to MONITOR > Applications > WLAN > select a WLAN ID.

Notice the AVC Profile (far right) applied to the WLAN SSID.



View each tabs: Aggregate / Upstream / Downstream.

It's recommended to configure and view NetFlow statistics either on a collector server/analyzer or using Cisco Prime Infrastructure.




Friday, December 7, 2018

Cisco WLC Software Upgrade via CLI

I had to upgrade a Cisco WLC 2504 from 8.2.166 to 8.2.170 but for some reason, my web browser (both IE and FF) doesn't prompt to reboot the WLC. So I performed a WLC software upgrade using the CLI instead.

Here's a link where I previously performed an upgrade on using the web GUI.


(Cisco Controller) >transfer download datatype code

(Cisco Controller) >transfer download mode ftp

(Cisco Controller) >transfer download serverip 172.27.5.2

(Cisco Controller) >transfer download filename AIR-CT2500-K9-8-2-170-0.aes

(Cisco Controller) >transfer download start

Mode............................................. FTP  
Data Type........................................ Code         
FTP Server IP.................................... 172.27.5.2
FTP Server Port.................................. 21
FTP Path......................................... ./
FTP Filename..................................... AIR-CT2500-K9-8-2-170-0.aes
FTP Username..................................... ftpuser
FTP Password..................................... *********

This may take some time.
Are you sure you want to start? (y/N) y

FTP Code transfer starting.

FTP receive complete... extracting components.

Image version check passed.

Executing backup script.

Writing new RTOS to flash disk.

Writing new FP to flash disk.

Writing new AP Image Bundle to flash disk.

Executing fini script.

File transfer is successful. 
Reboot the controller for update to complete. 
Optionally, pre-download the image to APs before rebooting to reduce network downtime.


(Cisco Controller) >config ap image predownload primary all      // PRE-DOWNLOAD IMAGE TO AP (LESS DOWNTIME)


(Cisco Controller) >reset system     // REBOOT WLC FOR NEW OS TO TAKE EFFECT


The system has unsaved changes.
Would you like to save them now? (y/N) y

Friday, November 16, 2018

Activating Cisco WLC 3504 AP License

Cisco has announced the End-of-Life (EoL) for the Cisco WLC 2504 this year so we're getting the new Cisco 3504 wireless controller. Below are photos of the WLC 3504 and this is a helpful link for the information on chassis ports and LEDs



I had to troubleshoot a brand new WLC 3504 and for some reason the 1852 AP wasn't able to join the wireless controller even though the country (for the AP) and date/time were correctly configured. The logs showed a looping CAPWAP State: DTLS Teardown error when consoled to one of the AP.


[*10/30/2018 02:57:49.0222] CAPWAP State: DTLS Teardown
[*10/30/2018 02:57:49.0322] Dropping dtls packet since session is not established. Peer 202.7.3.4-5246, Local 202.7.3.30-5264, conn (nil)
[*10/30/2018 02:57:53.8807]
[*10/30/2018 02:57:53.8807] CAPWAP State: Discovery
[*10/30/2018 02:57:53.8807] IP DNS query for CISCO-CAPWAP-CONTROLLER.local.net
[*10/30/2018 02:57:53.9206] Discovery Request sent to 202.7.3.4, discovery type STATIC_CONFIG(1)
[*10/30/2018 02:57:53.9206] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*10/30/2018 02:57:53.9206] Discovery Response from 202.7.3.4
[*10/30/2018 02:57:53.9406] Discovery Response from 202.7.3.4
[*10/30/2018 02:58:03.0000]
[*10/30/2018 02:58:03.0000] CAPWAP State: DTLS Setup
[*10/30/2018 02:58:03.0399]
[*10/30/2018 02:58:03.0399] CAPWAP State: Join
[*10/30/2018 02:58:03.0399] Sending Join request to 202.7.3.4 through port 5264
[*10/30/2018 02:58:47.0262] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Join(5).
[*10/30/2018 02:59:00.0722]
[*10/30/2018 02:59:00.0722] CAPWAP State: DTLS Teardown
[*10/30/2018 02:59:00.0922] Dropping dtls packet since session is not established. Peer 202.7.3.4-5246, Local 202.7.3.30-5264, conn (nil)


There were no APs that have joined the WLC per checking on the Monitor dashboard and under Wireless > Access Points.



In order for the APs to join the WLC, you'll need to activate the ap_count license. Go to Management > Software Activation > Licenses > click ap_count (hyperlink).


Set the License Status to Activate (default) > click Set Status.


An End User License Agreement (EULA) page will pop-up. Click I Accept.


The APs started to join afterwards. The box comes with the AP Adder License (or Entitlement). You'll need to specify the AP count under MANAGEMENT > Software Activation > Licenses > License Count > Add > type a number > Set Count. You'll be prompted again to accept the EULA.


For this scenario, I ordered a WLC 3504 with 15 AP license.




The Permanent AP Count (Adder) license will appear afterwards.


[*10/30/2018 03:01:37.0299] CAPWAP State: Join
[*10/30/2018 03:01:37.0399] Sending Join request to 202.7.3.4 through port 5264
[*10/30/2018 03:01:37.0399] Join Response from 202.7.3.4
[*10/30/2018 03:01:37.1499]
[*10/30/2018 03:01:37.1499] CAPWAP State: Image Data
[*10/30/2018 03:01:37.1999] do NO_UPGRADE, part2 is active part
[*10/30/2018 03:01:37.1999]
[*10/30/2018 03:01:37.1999] CAPWAP State: Configure
[*10/30/2018 03:01:37.2099] DOT11_CFG[0] Radio Mode is changed from Local to Local
[*10/30/2018 03:01:37.2099] DOT11_CFG[1] Radio Mode is changed from Local to Local
[*10/30/2018 03:01:37.2799] DOT11_DRV[0]: Start Radio0
[*10/30/2018 03:01:37.2899] DOT11_DRV[0]: Stop Radio0
[*10/30/2018 03:01:37.2999] DOT11_DRV[0]: Start Radio0
[*10/30/2018 03:01:37.4598] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).
[*10/30/2018 03:01:37.4598] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).
[*10/30/2018 03:01:37.4598] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Configure(8).
[*10/30/2018 03:01:37.4798] DOT11_DRV[0]: Stop Radio0
[*10/30/2018 03:01:37.4898] DOT11_DRV[0]: Start Radio0
[*10/30/2018 03:01:37.5498]
[*10/30/2018 03:01:37.5498] CAPWAP State: Run
[*10/30/2018 03:01:37.7198] AP has joined controller Cisco_03:ab:cd

Sunday, April 8, 2018

Transferring Configuration File on a Cisco WLC

You can upload or download files to and from a Cisco WLC by going to Commands > Download File or Upload File. In this case, I needed to backup the Configuration file from WLC to a remote TFTP server.


To upload the Configuration file to a remote server, choose File Type: Configuration > optionally tick Configuration File Encryption > Transfer Mode: TFTP > type the TFTP server IP Address > type the File Path: ./ > type a File Name (BACKUP-1).


WLC will pop-up a warning. Click OK.



The config file transfer completed just within a few seconds.


Here's a snippet of the WLC configuration file.


# WLC Config Begin <Tue Feb 20 02:57:56 2018>

config wlan apgroup wlan-radio-policy BUSINESS_WIFI 2 none
config wlan apgroup add BUSINESS_WIFI
config wlan apgroup interface-mapping add BUSINESS_WIFI 2 corporate
config wlan apgroup qinq tagging eap-sim-aka BUSINESS_WIFI enable
config wlan dhcp_server 4 172.27.3.2 

<OUTPUT TRUNCATED>

transfer upload path ./
transfer upload serverip 172.27.5.17
transfer upload datatype config
transfer upload filename BACKUP-1


# WLC Config End <Tue Feb 20 02:58:02 2018>