bill’s blog

At some point in your career as a system administrator you will be called upon to gather network traffic to find out where a system is failing. Sure ping and traceroute are wonderful tools that are down and dirty. They will tell you whether or not you have a basic IP connection and where it is failing BUT it doesn’t tell you much more than that. We at times will need to find out what is failing. Is it a mis-configured application? Are we sending data in cleartext when it should be encrypted? Are we “speaking” to the right DNS server? There are many ways to skin a cat in our profession BUT getting a packet dump really gives you insight to what is being put out onto the wire. There are a couple of things to consider before you actually start to collect data.

1. What data are you trying to collect?
2. What does the network topology look like?

It’s important to know what data you’re looking to collect because it will determine where to place your network-monitoring tool. Are you looking to collect the traffic between your web server (in a DMZ) and your internal network? Or are you looking to collect the traffic from that web server out to the world? If you’re using 2 NICs to separate out your traffic the monitoring tool will need to be placed on the interface that you are looking to capture the data of. Now this may sound simple (and in this case it is) BUT your typical SMB (small or medium business) network is often a lot more complicated. Another scenario is the ever-popular WLAN. What traffic are you looking to collect? Do you want to capture beacon frames or are you only looking at higher layer traffic? Hopefully you get the idea…

Next… How is the network laid out? Is it a HUB based network or is it a switched environment. Why is this important? Well let’s look at the mechanics of each. A Hub takes a packet that is received on one port and passes it to every other port on that port regardless if the packet’s final destination lies connected to that port. So in essence, any NIC that is connected to a hub can see all the network traffic that passes through the hub. If the traffic you are looking to capture all takes place on that hub your job is made easy. Practically speak… this is not realistic. The network issues surrounding a hub based environment (packet collisions) and the ever-dropping cost of switch technology makes this scenario highly unlikely.

Switches have all but replaced hubs in SMB environments. This presents some obstacles in gathering the traffic we are looking for. The Data Link Layer of the OSI model is where we tie the Network layer (IP address) to a physical attribute (MAC) of the machine we are working with. And while there is one additional layer to the OSI, all traffic is really run through the Data Link layer. SO what does this have to do with the price of rice in China? Well, a switch builds what’s called ARP tables. ARP (or address resolution protocol) matches an IP address with a computers MAC address. The switch then directs traffic received on one port to the port where the final destination resides! That destination is based on the MAC address of the host in question.

Remember… we are talking about hubs and switches. These are pieces of hardware! That being said because the traffic is directed between the receiving port and the destination port, other ports are not privy to the conversation between the two. Because there is still a need to ‘see’ the traffic that is passed between two ports (think network monitoring!), switch manufacturers have come up with a solution can the SPAN port (Switched Port ANalysis). This allows traffic to destined or sent from one port to be mirrored on another. There is a down side to this. Under heavy loads, packets can and will be dropped! This could and often does cause for the misinterpretation for the data collected.

The last option is called a TAP. Taps are hardware devices that are placed on the network and used when the need arises. Adding or removing the TAP from the network will result in network outages. The benefit to a TAP is that it preserves the full-duplex nature of a switched environment! It will not drop packets (that is if the line being monitor is NOT over-utilized). TAPs need to be strategically placed. Remember… What traffic are we looking to capture? There is a downside to TAPs. They are more difficult to set up! In order to properly implement these devices special configuration for NIC and/or the purchase of specialized hardware is required to combine the trace together. TAPs are usually used in places where putting a switch doesn’t make sense and you want to maintain the full bandwidth of the line!

A bastion host is a computer on the internal network that is intentionally exposed to attack (vconlinecourses.com, 2009). The host may be internal to your network but it is also forward facing. It is intentionally placed in ‘harm’s’ way, exposed so that the hosts that actually provide the service can remain protected. The Bastion host provides a layer of protection that other devices such as a firewall or an intrusion detection system do not… It is the focus of attack. A firewall should provide rules that keep the attacker at bay while the IDS will warn and in some cases thwart attacks. BUT the Bastion host WILL be attacked. It’s only a matter of time.

Just because the Bastion host doesn’t mean that it should be put out there unprotected. The host still needs to be hardened! There are many things one can do to protect the Bastion host.

DMZ

Putting all of your Bastion hosts into a protected network is your first line of defense. Because of the increased potential of these hosts being compromised, they are placed into their own sub-network in order to protect the rest of the network if an intruder was to succeed (wikipedia.org, 2009). At no time should a Bastion host have direct access to your protected resources! Internal (or protected) computers should only have access out to the Bastion host. As part of properly configured DMZ, routers/firewalls must be configured with ACLs (or Access Control Lists) so that only those events you (as the administrator) deemed acceptable are allowed to happen. Destination and source addresses need to be evaluated and rules need to be set in place to allow or deny access. Additionally, services ports need to be looked at as well. It may be acceptable for a source address to access port 80 (http) but not port 22 (ssh).

OS & Patches & ACLs

One thing to keep in mind when running a Bastion host is the box itself needs to be hardened. The OS needs to be kept up to date. Many vendors progressively secure their OS through security update. This may or may not be the right move. Vendors often roll multiple fixes into their updates… Sometimes it’s best to compile your own binary to install thus addressing the one service that may be affected by the vulnerability. Services that are not being used by the host should be disabled (or better yet) not installed… certain OS’s provide for this (Linux) others don’t (Apple). If the host has a host based firewall… turn it on configure it… block services that must run but could compromise the safety of the host. Secure the box through the use of ACLs (both user based as well as service based). It is usually up to the system administrator to determine through testing what ACLs they need to modify to lock down the network application as thoroughly as possible without disabling the very features that make is a useful tool (sans.org, 2009).

Base-lining

Tools like Tripwire and Nessus all play a part in base-lining your system. Tripwire is an excellent tool for determining the state of a file system. In broad strokes, it does this through the use of MD5 checksums. In theory, no two files (or disk images) will have the same exact checksum. Any changes, will result in a different checksum being produced. File integrity monitoring helps IT ensure the files associated with devices and applications across the IT infrastructure are secure, controlled, and compliant by helping IT identify improper changes made to these files, whether made maliciously or inadvertently (tripwire.com. 2009). So if an administrator, runs md5sum against a file system and then goes back a week later, if the checksums don’t match either he’s not on top of change control OR the system has been compromised! Nessus is a penetration-testing tool. In the case of Nessus, it looks at a database of know vulnerabilities and compares them with versions of software running on your host. When it finds a version of software running on your host that has been compromised, it will alert you to that fact. Should you find a software defect on your system it is imperative that you address the vulnerability through OS or patching and re-baseline.

Log Files

Syslog servers and log analyzers play an important role. Network monitoring solutions fit into this category as well! Logs are a vital part of understanding how your system is running. During the course of a few days or weeks massive amounts of information can be collected. Log files can tell you who tried to log in and when (or perhaps more importantly who failed to log in). It can tell you which files were accessed and by whom! It can tell you when a binary is having problems, either through miss-configuration or perhaps a bug (Heese, 2009). A wonderful tool for analyzing your data/log files is Splunk. It’s fast and allows you the ability to drill down through your log files in a very intuitive manner. Splunk can be configured to send alerts when certain criteria have been met. Sure you could do all this through shell scripts BUT you’d only be looking at the log files on one host! Because Splunk has the ability to act as a warehouse for all you system logs to can be set to look at multiple events across various systems and when combined can give you a true picture of your network/hosts.

Summary

You don’t become strong if you don’t learn! Systems that are exposed to the world need to be monitored. If you don’t, compromises will happen and you may not even know about it. A compromise host is not a matter of ‘if’ but rather ‘when’. Learning how your host was compromised can lead to better methods of securing it. Why leave it unprotected. Monitoring systems are essential to the well being of your systems. Why not take advantage of these automated systems. Spend the time to tune them. The more effort you put into it, the better the result will be, and the less false positives your IDS will flag! Know when an event is happening puts you back in control!

Resources:

Dillard, K., (2009), Intrusion Detection FAQ: What is a bastion host? Retrieved on March 16th 2009 from http://www.sans.org/resources/idfaq/bastion.php

Heese, B., (2009, March 11), Log Management, Retrieved on March 17th 2009 from http://weblog.randomdog.net/?m=20090312

Unknown, (2009), Bastion Hosts, Retrieved on March 17th 2009 from http://www.vconlinecourses.com/ec/dcs/DocView.learn?CourseID=3272624&47=5440807&dt=3%2F17%2F2009+9%3A06%3A50+PM&DocID=18645971&DocCollab_PK=19010583&Name=%22Bastion+Hosts.pdf%22

Unknown, (2009), File Integrity Monitoring with Tripwire, Retrieved on March 17th 2009 from http://www.tripwire.com/solutions/security/file-integrity-monitoring.cfm

Various, (2009, March 11), DMZ (computing), Retrieved on March 17th 2009 from http://en.wikipedia.org/wiki/Demilitarized_zone_(computing)