Firepower Release 6.7

 I was part of Firepower 6.7 Beta Program and I really was very impressed how I was supported by Cisco and in the way the feedback was appreciated. Since Version 6.7 was released to Public I want to show some of the new Features. I focus on FMC managed FTDs, since this use case is from my perspective by far the most used.

Release Notes https://www.cisco.com/c/en/us/td/docs/security/firepower/670/relnotes/firepower-release-notes-670.html 

I tested with some Virtual Devices for trying things out and had a HA pair running mainly AnyConnect Access throughout the Program in kinda productive fashion. There were actually no concerns regarding stability.

The described points below are not 100% complete on all new features, just a summary of my experiences throghout the testing process. I wasn't able to go through all of them, for example I had no chance to test any of the new features of NGFW in public Clouds.

FMC GUI Improvments

Upgrade Management

Good News for GUI Lovers - Rediness Check for HA Pairs is now possible through the GUI. Acutally there is also a Compatibility Check, where FMC points you to open Policy Deployments or a needed Upgrade on the FXOS side.

Also the whole Upgrade Process improved from a GUI perspective. Checking the actual status of an Upgrade and even accessing Logfiles is possible through GUI. So the way of closing down CLI at some point was continued - we'll see if this finally happens in future.

FMC Change Management

I big step from an operational standpoint, especially for Teams with more Administrators working on FMC. You can check in detail what changes were made to policies or device configuration before deploying a policy. Like with many SDx Solutions you can check a config diff to see what changes gets pushed out on the device.

This is also true from an auditing perspective. Going to the Audit log gets a simple option to view the actual changes a FMC admin did, also in a config diff view.

Another nice thing is Selective Policy Deployment. Imagine you want to modify a Snort Rule but you are not sure what the pending change of a new route on the device will do. So you simple uncheck the device changes in the Policy Deployment process and just push the Snort changes.

There is also an option for Deployment Rollback in case you did something wrong and not sure how to go back.

Usability Improvements

In the Beta Release at least, there was to option to enable a dark mode - big improvement *gg*

Another really nice thing is performance monitoring. You know the issue where you wanted to check overall system load from a firewall, like CPU, RAM, HDD, throughput and so on?! Now there's an overview which can provide all the answers in a single overview.

Small things that will help API-haters - no offense - a little. Bulk import of objects through FMC GUI or copy rules between different Access Control Policies.

VPN Improvements

Some more features coming up to improve feature parity with good old ASA Software.

AnyConnect 

Two things to mention here - SAML and Modules

There are ways to deploy modules before Release 6.7 with FlexConfig etc., but know it's an easy way to do it through the known RA VPN configuration options. The only thing which is still missing maybe, is an integrated Profile editor. 

You want to use modern protocols like SAML for AnyConnect - 6.7 is the way to go. If you want to use the nicer way of Duo MFA through SAML for example, it works now also on FTDs.

S2S VPN 

Firepower 6.7 can deploy VTI/Route Based VPNs. There are still some limitation you need to check (static routing or BGP for example).

For monitoring there is now an option in the GUI to get typical CLI commands an adminstrator would run on CLI to check VPN status. So you can check Phase1/2 from the GUI - troubleshooting VPN establishments is still a CLI thing though.

Other Nice Features to mention

SGT Support for using as Source and Destionation Match Criteria

pxGrid 2.0 Support

HTTP/2 Support

Snort3 - well, not for FMC managed devices in the Release Version

Remote Branch Deployment

It's now possible to connect FMC to a remote FTD through a Data Interface (ex. Outside Interface), so there is no need to workaround to get the Management Interface connected. Downside at the moment is, that High Availability is not yet supported.

Duo Radius Proxy on CentOS

Duo is super fast and easy to deploy - well it depends ;)

Got two boxes of CentOS Linux to setup Duo Radius Proxies, started the setup process with some prerequisites

[root@v***duo01 ~]# yum install gcc make liffi-devel per zlib-devel
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/4): base/7/x86_64/group_gz | 153 kB 00:00:00
(2/4): extras/7/x86_64/primary_db | 194 kB 00:00:00
(3/4): updates/7/x86_64/primary_db | 1.3 MB 00:00:00
(4/4): base/7/x86_64/primary_db | 6.1 MB 00:00:01
Determining fastest mirrors
* base: centos.anexia.at
* extras: centos.anexia.at
* updates: centos.anexia.at
No package liffi-devel available.
No package per available.
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed
------------------------------------snip--------------------------
Dependency Updated:
glibc.x86_64 0:2.17-307.el7.1 glibc-common.x86_64 0:2.17-307.el7.1 libgcc.x86_64 0:4.8.5-39.el7 libgomp.x86_64 0:4.8.5-39.el7
zlib.x86_64 0:1.2.7-18.el7
Complete! 

Next step is downloading the latest package and start the make process

[root@v***duo01 ~]# wget https://dl.duosecurity.com/duoauthproxy-latest-src.tgz
--2020-05-26 17:14:00-- https://dl.duosecurity.com/duoauthproxy-latest-src.tgz
Connecting to 10.133.21.140:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 43046700 (41M) [application/x-tar]
Saving to: ‘duoauthproxy-latest-src.tgz’
100%[===============================================================================================================>] 43,046,700 93.0MB/s in 0.4s
2020-05-26 17:14:00 (93.0 MB/s) - ‘duoauthproxy-latest-src.tgz’ saved [43046700/43046700]
[root@v***duo01 ~]# tar xzf duoauthproxy-latest-src.tgz
[root@v***duo01 ~]# cd duoauthproxy-4.0.0-3ff5a4b-src/
[root@v***duo01 duoauthproxy-4.0.0-3ff5a4b-src]# ls
conf config.mk doc _fipscustomize.py Makefile pkgs pkgs.mk sitecustomize.py
[root@v***duo01 duoauthproxy-4.0.0-3ff5a4b-src]# make
pushd /root/duoauthproxy-4.0.0-3ff5a4b-src/pkgs/openssl-fips-2.0.16 && \
make -f Makefile.duo all && \
make -f Makefile.duo install && \
------------------------------------snip--------------------------
~/duoauthproxy-4.0.0-3ff5a4b-src/pkgs/setuptools-42.0.2
~/duoauthproxy-4.0.0-3ff5a4b-src
adding minimal entry_points
Regenerating egg_info
Traceback (most recent call last):
File "setup.py", line 9, in <module>
import setuptools
File "/root/duoauthproxy-4.0.0-3ff5a4b-src/pkgs/setuptools-42.0.2/setuptools/__init__.py", line 20, in <module>
from setuptools.dist import Distribution, Feature
File "/root/duoauthproxy-4.0.0-3ff5a4b-src/pkgs/setuptools-42.0.2/setuptools/dist.py", line 36, in <module>
from setuptools import windows_support
File "/root/duoauthproxy-4.0.0-3ff5a4b-src/pkgs/setuptools-42.0.2/setuptools/windows_support.py", line 2, in <module>
import ctypes
File "/root/duoauthproxy-4.0.0-3ff5a4b-src/duoauthproxy-build/usr/local/lib/python3.8/ctypes/__init__.py", line 7, in <module>
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
Traceback (most recent call last):
File "bootstrap.py", line 64, in <module>
__name__ == '__main__' and main()
File "bootstrap.py", line 61, in main
run_egg_info()
File "bootstrap.py", line 54, in run_egg_info
subprocess.check_call(cmd)
File "/root/duoauthproxy-4.0.0-3ff5a4b-src/duoauthproxy-build/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/root/duoauthproxy-4.0.0-3ff5a4b-src/duoauthproxy-build/usr/local/bin/python3', 'setup.py', 'egg_info']' returned non-zero exit status 1.
make: *** [duoauthproxy-build/usr/local/lib/python3.8/site-packages/setuptools-42.0.2-py3.8.egg] Error 1

Seems there is an issue with Python. I figured out that CentOS runs on Python2 per default, but Duo needs to run Python3.
I quickly fixed that with

[root@v***duo01 ~]# python --version
Python 2.7.5
[root@v***duo01 ~]# yum install -y python3
------------------------------------snip--------------------------
[root@v***duo01 ~]# ln -fs /usr/bin/python3 /usr/bin/python

Now the make process finishes without errors, let's install

[root@v***duo01]# cd duoauthproxy-build/
[root@v***duo01]# ./install
In what directory do you wish to install the Duo Authentication Proxy?
[/opt/duoauthproxy]
Enter the name of a user account under which the Authentication Proxy should be run. We recommend a non-privileged and locked down account.
Or you can press <Enter> and our default locked down user will be created for you:
[duo_authproxy_svc]
Enter the name of a group under which the Authentication Proxy logs will be readable. Or press <Enter> and a default group will be created for you:
[duo_authproxy_grp]
Copying files... Done.
Create an initialization script to run the proxy upon startup? [Yes/no] yes
Created symlink from /etc/systemd/system/multi-user.target.wants/duoauthproxy.service to /etc/systemd/system/duoauthproxy.service.
Created service script at /etc/systemd/system/duoauthproxy.service
Installation completed. Before starting the Authentication Proxy,
Please edit the configuration file at:
/opt/duoauthproxy/conf/authproxy.cfg

Stealthwatch hints

Upgrade Issues (6.x -> 7.x)

Read the Upgrade Guide in detail and follow every step mentioned there. In short: Rollup Patch, Pre-SWU Patch, Upgrade.
If you still fail, and that happened to me 2 out of 3 upgrades, TAC will help. I tried to write down some of the tips below.

Upgrade Logs

/lancope/var/admin/upgrade/upgradeOutput.log

Finalize pre-SWU Installation from CLI

/lancope/services/secrets-service/finalize.sh

Cleanup unnecessary files

find / -type f -name "*.pcap"
find / -type f -name "*.swu"
find / -type f -name "*.tgz.gpg"

Refresh System Image

this cleans some diskspace and help if you enxperience a slow SMC GUI after the Upgrade to 7.0.
Additional it removes rollup patches.

1. Login as root or sysadmin via ssh on the appliance to use the System Configuration Menu.
2. Root user will have to launch the menu manually at the cli using the 'SystemConfig' command.
3. Select Advanced options
4. Select Refresh the System Image
5. Select yes to continue
6. Select yes to continue
7. Select ok to reboot

FC fails to authenticate to SMC during Upgrade?

update-fcnf-7.0.2.2019.07.05.1356-01 - INFO - >SAFETY CHECK: Partition Patch Satisfied
update-fcnf-7.0.2.2019.07.05.1356-01 - INFO - >Error in authenticating to the appliance 10.1.2.3: 400 Client Error: Bad Request
Remove the Certificate from SMC Admin interface and upload the new one from the FC.

SMC is not in Central Management after Upgrde?

Go to appliance Administration (https://10.2.3.4/smc/index.html), there you should see the note to go to Appliance Setup Tool

Run through the initial Setup Dialog an confirm every settings which should be already there (IP addresses, DNS, NTP etc.). The Appliance will reboot afterwards.

Stealthwatch GUI Admin Passwort Reset

SSH as root to the FC appliance then run these commands:

• systemctl stop LCOrchestrate.service
• systemctl stop lc-tomcat.service
• cd /lancope/var/database/dbs/hsqldb/admin
• rm -rf admin.*
• systemctl start LCOrchestrate.service
• systemctl start lc-tomcat.service

FC fails to re-add to CM due to Certificate Issues

Check Logs for CM

fc01: tail -f /lancope/var/logs/containers/svc-cm-agent.log

find things like

unable to find valid certification path to requested target

fc01:/lancope/var/nginx/ssl# rm -rf *
fc01:/lancope/var/nginx/ssl# /lancope/services/secrets-service/identity_service.py 
Lancope default certificate not detected
A10 certificate not detected
Appliance does not yet have a server identity. Generating one now...
Generating self-signed server identity certificate...
Self-signed server identity certificate generated.
Inserting newly generated server identity into appropriate locations...
New server identity is now active.
fc01:/lancope/var/nginx/ssl#  
fc01:/lancope/var/nginx/ssl# ls -l
total 16
-rw-r--r-- 1 root root 3186 Oct  3 06:50 client.crt
-rw-r--r-- 1 root root 3186 Oct  3 06:50 server.crt
-rw-r--r-- 1 root root 6363 Oct  3 06:50 server.key
fc01:/lancope/var/nginx/ssl# chown tomcat:swadmin *
fc01:/lancope/var/nginx/ssl# chmod 660 *
fc01:/lancope/var/nginx/ssl# ls -l
total 16
-rw-rw---- 1 tomcat swadmin 3186 Oct  3 06:50 client.crt
-rw-rw---- 1 tomcat swadmin 3186 Oct  3 06:50 server.crt
-rw-rw---- 1 tomcat swadmin 6363 Oct  3 06:50 server.key
fc01:/lancope/var/nginx/ssl# 
fc01:/lancope/var/nginx/ssl# systemctl restart lc-tomcat
fc01:/lancope/var/nginx/ssl# systemctl restart sw-nginx.service 

Cisco Cloud Email Security CES Bulk Import SMTP Routes, RAT entries

You want to bulk edit things like SMTP Routes or RAT entries on CES environments? You need CLI access to your ESA instances inside CES?

That's not that simple, there is no direct Access to CLI. There are SSH proxies provided for that usecase. First you need to ask for getting access there. (Some information below was provided by Cisco TAC)

Ask CES Activation Team ces-activations@cisco.com or Cisco TAC for CLI Access

Following are the steps for setting up SSH access.

Access to your IronPort appliances is provided through an SSH Proxy using key authentication. CLI access to your hosted appliances should be limited to key individuals within your organization.

Overview of the Process:

     1) Generate Private/Public keys

     1a) If you require multiple users, you may generate and submit up to 10 keys.

     2) Reply to this email with your *Public* key -attached-.

     3) We will apply your key(s) to our SSH Proxy.

     4) We will then provide you a guide on accessing your machine.

HOW TO: Generate a Private/Public keypair for Hosted CLI access.

For UNIX OSes such as Linux, OS X, Solaris, etc.

SSH-keygen -t rsa -b <bitstrength> -f <filename>

<bitstrength> is an integer number.

<filename> is the path name to a file where the key(s) will be saved to.

Example: SSH-keygen -t rsa -b 2048 -f ~/.SSH/my_key

*Please ensure that you safeguard your private keys.

*DO NOT send us your private keys.

*If you are submitting multiple keys, please provide names and email address associated to each key.

There are numerous SSH client applications available for all major operating systems. Windows users can use puttygen: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html 

Howto establish Proxytunnel

PUTTY CONNECTION GUIDE TEMPLATE:

Accessing your IronPort appliances is made through an SSH proxy.

You will initiate a local port forwarding proxy on your workstation.

Your workstation will be listening on a localhost port.

You will then SSH to your localhost, which will forward the SSH traffic to your IronPort appliance.

Enter in the Proxy hostname - Make sure you choose the correct one for your region, US or EU:

There are two US SSH proxies:

f4-ssh.iphmx.com (68.232.128.202)

f5-ssh.iphmx.com (68.232.134.202)

And two EU SSH proxies:

f10-ssh.c3s2.iphmx.com

f11-ssh.c3s2.iphmx.com

Click Data and add dh-user to the auto-login

Click SSH and check Don't start a shell or Comm...

Click Auth and Browse to your Private key.

Click Tunnels supply a Source Port, Destination (Your IronPort Appliance)

Click Add, and it should look like this.

example 

1. esa1.hc***.c3s2.iphmx.com

2. esa2.hc***.c3s2.iphmx.com

You can save this session for future use. Click Session, you may supply a name,

and Save the session. Go Ahead and Open the session to initiate the local port forwarding proxy.

If all goes well you will automatically be logged on to the proxy server.

You won't get a command prompt.

Now you will need to open a new putty window. Use the hostname 127.0.0.1

and use the source port number in the tunnel configuration above.

(2200) Click open to connect to your appliance.

When prompted use your appliance username and password.

Please confirm that you are able to access your appliances via the command line. If you have further questions please feel free to contact us.

Copy Files through Proxytunnel

c:\Program Files\PuTTY>pscp -P 2200 partner_syseng@127.0.0.1:configuration/smtp_routes_export c:\smtp_routes.txt

Using keyboard-interactive authentication.

partner_syseng@esa1.hc***.c3s2.iphmx.com's password:

smtp_routes.txt       | 0 kB |   0.3 kB/s | ETA: 00:00:00 | 100%

c:\Program Files\PuTTY>pscp -P 2200 partner_syseng@127.0.0.1:configuration/RAT_export c:\RAT_export.txt

Using keyboard-interactive authentication.

partner_syseng@esa1.hc***.c3s2.iphmx.com's password:

RAT_export.txt        | 0 kB |   0.2 kB/s | ETA: 00:00:00 | 100%

c:\Program Files\PuTTY>pscp.exe -P 2200 c:\Users\alth\Documents\smtp_routes_import.txt partner_syseng@127.0.0.1:configuration/smtp_routes_import

Using keyboard-interactive authentication.

partner_syseng@esa1.hc***.c3s2.iphmx.com's password:

smtp_routes_import.tx | 13 kB |  13.5 kB/s | ETA: 00:00:00 | 100%

c:\Program Files\PuTTY>pscp.exe -P 2200 c:\Users\alth\Documents\smtp_routes_import.txt partner_syseng@127.0.0.1:configuration/smtp_routes_import

Using keyboard-interactive authentication.

partner_syseng@esa1.hc***.c3s2.iphmx.com's password:

smtp_routes_import.tx | 13 kB |  13.4 kB/s | ETA: 00:00:00 | 100%