FTP security with Firewall ISA 2006 (Part 1)

Network administration - The ISA Firewall does not officially support the FTPS protocol because the protocol agreements are encrypted and therefore no application filters can handle these agreements.

This article will introduce a method to overcome this problem. However, one disadvantage of this method is that unencrypted connections will not work. Another solution to use is to create a customized Server Protocol Definition for FTP, then use the Pesach Shelnizt tool to prevent possible conflicts with the default FTP Server Protocol Definition.

One of the important new features built into IIS 7.0 (also included on Windows Server 2008) is FTP security. FTP security uses TLS encryption to secure data while transmitting via FTP data channels. This is a significant improvement over previous versions of the IIS FTP service (only unencrypted connections are supported).

However, if you've ever used the FTP service of IIS 7.0 through an ISA firewall, you'll probably find that things don't work as expected. When connecting to the server from the workstation, you will encounter a 550 Access Denied error.

Next we will explore the nature of this error and consider the remedy. To do that, we need to do the following:

1. Check the FTP workstation
   
2. Monitor ISA server system
   
3. Check the FTP server

The installation model of workstation and server system is as follows:

Suppose the FTP client's IP address is 10.10.10.100 . The IP address on the external interface of the ISA firewall system is ISA 10.10.10.254 and the server's IP address is 10.10.20.200 .

---

Step 1: Connect to the FTP server from the FTP client

When checking information you will know what is happening on the system. First, check Trace Packet because it allows us to monitor the FTP stream.

Packet View

According to information in Packet View, FTP client displays TCP connection information in the first 3 Packets (including SYN, SYN ACK and ACK). After the TCP connection is completed, the FTP communication will start. The first FTP command is the server command used to identify it as an FTP MS server.

Then the workstation will respond to an AUTH TLS message. Normally the client will display a message saying 'hello FTP server, i would like to start an encrypted TLS session' . The client will respond to this message twice and then receive an Access is Denied response from the server.

When you check the FTP server's Packets, you can check the 3 Packets for the first connection information, including a self-signed server package, but we won't be able to see the storage Packet FTP information of FTP server during AUTH TLS session.

The window displays information

If we look at the TCP information window, we can easily see that the client is sending AUTH SSL information and the server will never receive this information.

Step 2: Check the Access is Denied error message

The above information shows a 550 Access is Denied response being sent to the FTP client. However, there is no information confirming that the feedback is sent by the server. The FTP server can accept the request to perform TLS connection security. So, what is the problem here? Obviously, there is an intermediate device that created this response and sent it back to the workstation.

To use this information, check the ISA Firewall's log file and check if there is a level of denial of access. The illustration below shows the related log entries.

At first glance, the traffic seems very normal. ISA accepts the first packet and then disconnects it as it usually does when a normal TCP connection session ends.

So where is the Access is Denied error message sent from? For a closer inspection, we need to add a column to the ISA log file window named Result Code .

After adding this column, we will see what the ISA Firewall is doing.

Note: You can check the meaning of the Result Code here.

In the Result Code above, 0x80074E24 FWX_E_CONNECTION_KILLED indicates that the ISA server has disconnected.

---

Although this disconnection operation is not normal, it seems that this is the method of disconnecting the ISA firewall's session, so we can confirm that the Access is Denied error message is sent to the machine. station from ISA firewall system.

We then have to check which components of the ISA firewall system have disconnected. You should first check the application filter to handle FTP connections. It is possible that ISA's firewall system is a system for checking application layer status and information, configuring it to perform application layer testing for class protocols. One of these protocols is FTP. You can see this protocol in the Add-ins section in the left pane of the ISA Firewall console.

This means ISA knows the FTP protocol at the application layer and will differentiate the different commands with an FTP communication stream. We can use the dll application filter to configure the entries or not.

For example, the SMTP filter supports adding and removing commands approved by the firewall system. However, the FTP filter does not perform this function. What you can do is enable or disable it. You can cancel the filter here and then the filter will be turned off in the entire ISA firewall rule, or you can turn it off on certain rules.

FTP application filter and SMTP application filter

Actually the FTP application filter of ISA 2006 does not support AUTH TLS, so the default response from the ISA firewall system with such requests is to return an access denied message. ISA firewall system dumps the default FTP command group and you will not be able to insert accessible scripts.

Note: This response does not display the words 'filtered by ISA 2006' (filtered by ISA 2006), as it may provide unwanted information to hackers.

---

The only method to fix this problem in ISA 2006 is to turn off the FTP application filter in the Access Rule. Do the following:

1. Open the Properties Properties dialog box of the FTP Server Publishing Rule in the Access Rule .
2. Click the Traffic tab.
3. Clear the FTP Access Filter check box in the Application Filters frame.

Properties window of FTP server

If after you have done the reconnection and changed the traffic on the FTP server, you can first Packet display verification string information, the second packet shows the client's message 'Hi, I want to do TLS authentication '(Hello, I want to perform authentication for TLS rights), and the third information package is responding from server' ok let's by TLS '(approval). Then this information will go into encryption mode.

However, the authentication channel is only the first segment of the connection. Once you have authenticated a FTP server for which we want to list directories and file transfers, this process will be performed on the secondary data channel. In part two of this article we will learn how to transfer data over a secure channel.

Conclude

In this section of the article, we will focus on the problems with the FTP server when securing with the ISA 2006 firewall system. When using a network adapter, we can see that connecting to the client rejected but not rejected by the FTP server, but by the ISA firewall system. ISA firewall system integrates an application layer filter that supports FTP connection. However, the user cannot configure this filter (as opposed to the SMTP filter that allows users to perform some basic configuration). This is because the filter does not support the TLS agreement so users need to disable it in the entire rule or in certain rules. It is best to turn off this filter in some rules because if you disable it in the entire rule, the SecureNAT temporary machine cannot use the FTP protocol to access the network.