bill’s blog

July 4th is one of my favorite holiday’s… hitting the beach… Barbecuing… cold beers… and fireworks! BUT working in IT brings with the possibility of having you’re holiday plans interrupted by server/network outages. I can’t remember a 4th of July where I didn’t get a call that something is up with one of my servers and 2010 would be no different.

It started at 6:30AM… The main website for our Canadian office was unreachable. SO I booted my laptop and checked the site. Hmmm… It came up fine for me. Perhaps the server’s admin got to it before me. Called corporate and fixed myself a cup of coffee. 45 minutes later the phone rang again. “The site is down again!” I walked back to the computer, coffee in hand and indeed my browser timed out. Hmmm… OK something’s not right. I VPN’ed into the box and pointed the servers browser to the website and the site loaded BUT not as fast as I would have expected. The load on the server looked a bit high to me but this wasn’t my box and didn’t know what the normal numbers for the box were! I started to pour through the APACHE server error logs looking of answers. Nothing there!  Back to the browser… The page loaded fine, the speed having returned. I turned to my wife’s computer (making sure I took the local network out of the equation) and the point her browser to the site. This time I got a strange error message in the browser window…

“Can’t connect to the database too many connections open.”

Hmmm. Strange? Let me refresh my browser… the site pops back up. OK… let’s jump on the box and have a look at what’s going on… CPU utilization looks normal… MySQL looks OK… Refresh the browser… the site is still up. OK let’s have a look at the MySQL logs… Still nothing. So I called the developer to confirm that no moves were made into Production on Friday. I rebooted the box and everything seemed to return to normal. 2 hours later the phone rings once more… The site is down again Bill. Man this isn’t even my server… This is really going to be bad if I have to reboot the server every 2 hours this weekend. Opened a browser window and got the database connection error message again. OK let’s take a look that the system logs… WOW that’s funny the kernel is error’ing out and throttling back the the network stack. OK… Let’s see what netstat turns up… ouch! There were hundreds of connections in a FIN-WAIT or a SYN_RECEIVED state? What’s going to? Did some one patch the OS on this box? Nope… Let’s check the throughput of this box… 75,000 requests per second… Now one could dream but I’d think this was pretty rare occasion for this domain! OK… Let’s see if I could get at the firewall logs… Sure enough there were thousands of connections open. WOW we were in the middle of a DDoS (Distributed Denial of Service) attack. I couldn’t believe it.

The point of this is that it doesn’t use fancy network tools to figure out what’s going wrong with a  machine. I didn’t use a network sniffer. The box was not one of mine so I didn’t know the state of the server.  I used what was on the machine and started by eliminating variables. But the really big lesson learned is, it doesn’t matter how small you think your site is, it could always be the target of something like a DDoS.

Preparing to introduce the fine employees of SCUNCI to eWorks!

People have lives outside of work and it’s important to take that into consideration when dealing with co-workers and ‘clients’. Often times we are heads down pushing out projects or dealing with the mundane while working help desk. It is repetitive and at times it takes every ounce of restraint to keep from screaming but at the end of the day one must realize that we in IT. We are here to serve. A network isn’t simply there to be put in place just because we can… people need to use it to get there work done.

If the network isn’t up and the resources are not available… we’re not doing our jobs. People are capable of some very stupid things both benign and malicious. Intrusion Detection Systems helps to keep us one step ahead of bad guys! These systems provide us with eyes in the back of our heads. We can’t be expected to be everywhere at one time. Why not use the computer technology to helps us perform our jobs. Intrusion detection/prevention is an absolute task that needs to be taken seriously. Security through obscurity doesn’t work any more. In an average month my home network is scanned 100’s of times! And I’m a nobody! Now put a dollar value behind the information you’re protecting and the motivation value goes way up.

Securing your network is the first step. Setting up a firewall correctly can go a long way to making your networks safe. Many firewalls come with an IDS built in. One must realize that this will only inform you of the traffic moving your private network and the Internet but that’s where most of the malicious activity is coming from. The nearly 82% of losses were attributable to insider threats at a cost of $293,890,505. This is a little misleading… first the data is from a survey that is more than 10 years old, and second they put a dollar value on the loss without putting hard facts as to how those numbers were reached. Times change quickly in 10 years The Internet is not the safe haven of academia anymore. There are a lot more bad guys out there today with a different set of drivers to motivate them. 10 years ago it was about the notoriety now it’s about the dollars. Viruses were the crimes of the day… try calculating how much employee time is wasted on the prevention and eradication of virus breaches. Let’s see how those numbers stack up. No, today’s crackers are driven by the dollars or backed by nations that see the advantages of controlling other nations networks. BUT I digress!

The thing about human beings is we can get distracted very easily and block out information that we don’t need. Computers only do what we ask them to do… they see patterns of ones and zeros and act on them based on instructions we provide. False positives are problematic at best. ID systems work on a set of rules or signatures and while logic is applied, computers don’t have the capacity of reason. Because of this, things like false positives or worse yet, false negative can be problematic. False positives (or the boy who cried wolf syndrome) are when an IDS alerts that there is a problem when in fact there isn’t. The system will send out SMSs and email (god-forbid pagers). People will all jump to attention and look at what’s going on. If the problem is negligible or non-existent and continues unchanged… eventually we will block out the message. Unfortunately, that could lead to a situation where a real emergency is NOT responded to. False negative in my opinion are worse than false positives. This is where the IDS fails to alert when the actually is a problem. Very often this can go on for long periods of time allowing the intruder to go unchecked. The quicker you can close the door on them the better. IDSs are like anything else in IT, it needs to be tweaked and cared for (updated regularly).

As for insider threats… well I don’t want to seem like I’m down playing this! It is very real but with regard to the above-mentioned survey, the loss was attributable to employees (who had legitimate access to the data) taking proprietary information or changing data in acts of fraud. This is not a situation where IDSs would come into play. In today’s modern network ACLs (or Access Control Lists) should be implemented and reports generated to see who is access what data at what times. Failed logins or file accesses need to need reported and polices need to be put into place in order to correct the behavior.

Setting up a VPN (or Virtual Private Networking) does not have to be difficult. In fact using Apple’s OSX, it can be down right easy.  VPNs should never be taken lightly. IT is the door to your protected network. If they’re not set up correctly it could leave you and your network assets at risk. There are two main types of VPNs that on can implement on OSX server, PPTP and L2TP. There are pluses and minuses to each and depending on how you/what you’re looking to support will determine which implementation you will use. It’s interesting to note that neither of the two mentioned VPN protocols provide encryption. They are considered tunneling protocols and thus need to rely on other methods to provide the encryption.

PPTP – Is the older of the two most popular tunneling protocols. It relies on either on either MSCHAP-v2 or EAP-TLS for authentication. Additionally, Apple has built in support for both Kerberos authentication and RADIUS. PPTP uses Microsoft Point-to-Point Encryption (MPPE) to encrypt the data that passes through the tunnel. Originally MPPE was only offered with support for a 40bit key. It was later expanded to a 128bit key!

L2TP – Is the newer comer, its latest version (RFC 3931) having been published in 2005. L2TPv3 makes use of IPSec for securing the connection. This is preformed through the use of pre-shared secrets, symmetrical keys or digital certificates. As with any secure connection the hardest part of maintaining the SA is the managing of the keys used. However, once the first connection is made and security confirmed. The passing of pre-shared secrets, keys or digital certificates becomes trivial.

NOTE: It should be noted that that PPTP and L2TP are not the only players in the VPN game. There is two other methods as well, PPP Over SSL and PPP Over SSH.

Configuring your server

Open Server Admin and select the host you wish to administer. Select VPN and click save.

Figure 1. Services Activation Pane

Turn down the triangle to reveal the VPN configuration pane.

Configuring L2TP Settings

It is as this point that you can decide which tunneling protocol you’re going to support. Setting up the server is pretty simple. Select the check box to enable L2TP. You need to allocate an IP range (remember this is still a point to point connection). Under PPP Authentication select if you want to use the built-in Directory Service plug-ins for user ID and password lookups (you can also chose between MS-CHAPv2 or Kerberos) or point the VPN service to look at a RADIUS server for authentication lookups. Lastly, you need to specify whether you want to use a pre-Share secret or a digital certificate for IPSec Authentication.

Figure 2. L2TP Configuration Pane

Configuring PPTP Settings

If you need to support older VPN clients PPTP may be a better choice for you. Many experts still contend the PPTP is vulnerable to compromise but with anything else strong passwords make for strong security. Depending on the client that you need to support you may need to allow 40bit encryption keys. This should be avoided if at all possible as 40 bit keys are easily cracked.

Figure 3. PPTP Configuration Pane

Configuring Client Information Settings

Lastly, you need to “tell” your clients about the network they have just connected to. This could be done on the client side, and may be desirable is some situations. In a lot of ways this is very similar to setting up a DHCP server.

NOTE: If you are running DHCP on the same subnet, make sure that the allocated IP address ranges do not conflict!

Figure 4. Client Information Settings

NOTE: If no information is added to Network Routing Definitions all traffic is routed through the VPN connection. This may not always be desirable. If bandwidth is a concern, define a network that is private and force all non-private traffic over the client’s Internet connection.

Ports on your Firewall

One thing you must make sure to perform before your VPN will work is to open the required ports on your firewall. Both protocols make use of different ports can it can be confusing which ports are actually needed. Not just on the host (if you’re running IPFW on the host) but on the network perimeter. So what ports are used?

500       UDP      ISAKMP/IKE
1701      UDP      L2TP
1723      TCP      PPTP
4500      UDP      IKE NAT Traversal *

* NOTE: Port 4500 is also used for Back to My Mac (MobileMe, Mac OS X 10.5 or later)

In Mac OSX Server 10.3 the VPN service uses the following:

1.    PPTP uses the IP-GRE protocol (IP protocol 47).
2.    L2TP/IPsec uses the IP-ESP protocol (IP protocol 50, ESP).

Resources:
http://manuals.info.apple.com/en_US/Network_Services_Admin_v10.5.pdf
http://support.apple.com/kb/TS1629

Saturday – 02/21/2009

What an amazing day! We were invited to spend the day with the China Creative Group. They took care of us during our entire stay in mainland China but today was magical. I wasn’t sure what to expect. We were told that in the morning we would go through training. I figured it would be a lot like paintball in the United States but it was so much more.

It was a group building experience and I truly felt like they wanted us to join their ‘team.’ The morning started off with activities designed to foster team work.

The morning culminated in a challenge to climb a 40 foot pole, stand on the top on it and jump to a swing that was ten feet away from the top. I had my concerns about my ankle and being able to make the transition from a kneeling position to standing. But my colleagues cheered me on and helped by manipulating the tension of the safety ropes. What an awesome feeling! I didn’t think I could do it yet through team work I was able to make the jump!

I’ve met my match!

After lunch was through it was time to get to the shooting! Fortunately, the field wasn’t ready for us and we got to spend some time exploring a tree museum. Some of the trees dated back before Buddha! But the really cool part was that all the displays were in Chinese and my colleagues can to my rescue and translated the information so that I could understand.

Now I thought we were into our war games here… BUT they were truly into it! Crawling on the ground… Running into the brush… very realistic! It will be a day I will always remember… Thank you Flower, Lily, Jelly, Amanda, Happy, Cold, Matthew, Joe, Nelson… Everyone!

So last night I was trying to stand up a new replica against my OpenDirectory Master but it kept erroring out with a 1077 error. It was complaining about my credentials being incorrect. At first I though I must have fat fingered it… but after entering in the password one character at a time it still didn’t take. Looking through the slapconfig.log file (located in Library/Logs), I got the following error:


2009-02-09 22:08:02 +0800 - slapconfig -setmacosxodpolicy
2009-02-09 22:08:02 +0800 - slapconfig -createreplica
2009-02-09 22:08:02 +0800 - command: ssh root@192.168.171.10 /usr/sbin/slapconfig -checkmaster diradmin 0 4 4
2009-02-09 22:08:13 +0800 - ssh command failed with status 77
2009-02-09 22:08:13 +0800 - Error: Incorrect username or password. You must enter a directory domain administrator username and password.
(error = 77)


Everything was correct. I could ssh into the server using the root account. I could modify the directory (add/delete/modify accounts) using the diradmin account. But I still couldn’t bind the server. Turns out there is a bug that doesn’t allow you to bind the replica if the diradmin password contains anything but alpha-numerics. Change the password to something simple the replica binds without issue. So much for strong passwords!

Well today is the start of everything that I’ve taken this trip for. We’ll be updating the servers in Hong Kong today.

Setting up DNS, OpenDirectory, AFP shares and then migrating the user accounts over. Hopefully if all goes well we’ll be done by 10PM… hopefully!

6:15PM – Start Time
6:30PM – Got all users off server
7:30PM – Finally got the machine to boot from DVD
7:45PM – Got McDonald’s for dinner
8:00PM – Config’d host
9:10PM – Finally got DNS working… Hate DNS!
9:15PM – Started patching machine
9:30PM – Still waiting for the updates to download… Moving user data!
9:40PM – Downloads are done… Let’s bind to OpenDirectory!
10:00PM – Anyone know what a 1077 error is?
10:25PM – This network sucks…
10:40PM – Oh hey let’s test the Riverbed Device…
11:09PM – Strong passwords? Why bother?
11:34PM – Setting up Network Homes!
11:36PM – Oh wait… the directory doesn’t like diradmin any more.
12:19AM – Finished patching server
12:40AM – Fixed a few login issues
1:10AM – Tested all logins… They work.. I’m out of here!

Data Encryption is an often-overlooked aspect of computer usage. For many years encryption was looked at as a technology to protect your data as it transverses the Internet. But what about the data that is at rest on your computer? We’ve all read about the VA’s data loss 26.5 million individuals were exposed. An analyst had taken home the database of veterans’ names, dates of birth, Social Security numbers, and some health records to work on a project, according to the VA (Gross, 2006). One key aspect to protecting data is employee education. Employees need to respect the data they are dealing with. Complacency is a big issue. Like anything else, the more you use something the more comfortable you become with using it. Picking up a chainsaw for the first time and using it you know the potential hazards of its misuse and treat it with kid gloves… the more you use a chainsaw the more comfortable you are. The device is no less hazardous but the precautions you took, as a novice seems to make way for more nonchalant use.

So what to do about this? Well There are varying schools of though on this. One way is to encrypt the entire hard drive. When the user first turns on their computer they need to enter a password to unlock the drive and begin the boot process. The nice thing about this is the end-user only needs to worry about unlocking the computer with a password and then everything stored on the computer is encrypted. The bad thing is it the password to unlock the drive is lost… So is everything on the computer. The latest release to the PGP® Encryption Platform, PGP Whole Disk Encryption 9.9 adds pre-boot authentication to the proven PGP Corporation data encryption technology for Intel-based Mac OS X systems “Tiger” and “Leopard,” providing protection for data on desktops, laptops, and removable media (pgp.com, 2008).

The other school of thought is to only encrypt the user space. There are various ways to accomplish this and Apple provides a number of solutions right out of the box. The Ponemon Institute is an advocacy group that deals with the information and privacy issues. According to their findings in 2007, the cost of a data breach was approximately $197 per record, an increase of more than 40 percent since 2005 (Bocek, 2008). Now that may not seem like much but if you figure that number into the amount of records exposed in the VA breach, that’s 5.2 trillion dollars. Ouch! SO how has Apple made it easy to protect data that resides on your computer? Apple has two technologies that can be used to both store and securely erase data on your hard drive. They are:

1. FileVault
2. Encrypted Disc Images

FileVault

The main premise behind File Vault is that each users’ home directory is stored on an encrypted disk image. The disc image is created using the users password. The image is only unlocked when the user logs in. This eliminates the possibility of accident data loss due to bad file permission of the users’ part in environments where users share machines. One feature that is different from traditional whole disk encryption schemes is that in addition to the users’ password being used to encrypt the image, you can set up a master password for all FileVault images stored on your machine. Some may see this as a security whole BUT in enterprise based Environments this is a godsend! How many times during a typical week are you called for a password reset?

Figure 1 Security Preference Pane

Turning on FileVault is extremely simple. In System Preferences, select the Security Pane; you are now presented with everything you need to get the process stated. Clicking on the “Set Master Password…” button with present out a dialogue sheet to set the master password for the machine. Fill is the password and then verify, as this dialogue will display “•” when entering character into the password fields. One may be tempted to add a password hint. This is generally NOT a good idea!

Figure 2

Additionally, Apple provides a password strength tool. By Clicking on the key next to the “Master Password:” field (see figure 3) the tool will be presented.

Figure 3. Password Assistant Tool

Note: The password is presented in clear text. The better the password the further to the right the green bar extends.
Once this is completed your all set up with encrypted home directories. When setting up FileVault accounts for the first time, some time is required to do the actual encryption. Depending on how large your existing home directories are will determine how much coffee you’ll need to drink.

Encrypted Disc Images

Encrypted disc images are very similar to FileVault directories with two major differences. One they are portable. You can copy the image from machine to machine. The contents of the images are encrypted, so if you happen to put the image onto a flash drive and loss it your data is protected. Two… There are no master passwords to help you out should you forget your password. So you can forget the magic bullet to help you out. Your data is lost!

To create an encrypted disc image open Disk Utility. It can be found in /Applications/Utilities. Select “New Image” from the toolbar across the top of the main window. This will present you with a dialogue box where you can indicate where you want the image saved, how big you want it, what type of file system to lay down and most importantly in terms of this discussion, how strong you want the security to be. If you’re in an environment that makes of PKI using PGP, you can leverage the power of PGP’s whole disc encryption to encode the entire flash drive. Then when you insert the flash drive into your machine PGP will automatically open the image and display it on your desktop. You can accomplish the same thing by adding the password of the encrypted image into you Keychain. This will yield the same results but it’s more tedious in so far as you need to load the password onto all the machines that the flash drive will be used on. This is very labor intensive if your dealing with 500 flash drives and 500 computers.

Figure 4

For all of those in the government sector, selecting 256-bit encryption will yield a FIPS -140-2 compliant disc image (see figure 4).
Encrypting data at rest is simple… And not as expensive as the loss of data can be. Recently, the case of the 2006 Department of Veterans Affairs data loss resulting from the theft of an unencrypted laptop containing the names, birth dates and Social Security numbers of approximately 26.5 million veterans was settled.
“The settlement with the Department’s members and families over their alleged invasion of privacy should be a severe warning to any organization that isn’t using encryption on its laptops and other portable devices capable of data storage,” said Michael Callahan, vice president at encryption specialist Credant (Thomson, 2009).

The cost… $20 million… certainly less than the cost of encryption.

Resources:

Bocek, K. &Ma, T., (2008), Data Encryption for Dummies, Indianapolis, IN: Wiley Publishing

Gross, G., (2006, May 5), VA data loss could prompt federal privacy law, Retrieved on Feb 3, 2009 from http://www.networkworld.com/news/2006/060506-va-data-loss-could-prompt.html

Thomson, I., (2009, Jan 28), US veterans win $20m payout over lost laptop, Retrieved on Feb 3, 2009 from http://www.vnunet.com/vnunet/news/2235300/va-fined-million-breach

Unknown, (2008, June), PGP Corporation Delivers Pre-Boot Authentication to PGP Whole Disk Encryption for Mac OS X Users Retrieved on Feb 3, 2009 from http://www.pgp.com/newsroom/mediareleases/wde_for_mac_osx.html