bill’s blog

Just another WordPress weblog

I was recently asked to redo the permissions on 5TBs worth of data. There were inherited permissons that conflicted with the users new requirements… it was just a mess! I figured the best way to deal with this was to start from scratch… remove all ACLs and start fresh.

The easiest way I’ve found to do this is…

sudo chmod -R -N ./*

Those who say it cannot be done should not interrupt the people doing it!

- Matt Schultz, 2010

Network scanning is an art! Sure there are skills involved and software that makes it easier for us to profile a target and one can go about it in various ways. BUT before we discuss that we need to understand a little something about TCP. Transmission Control Protocol (or TCP) is the protocol that drives the Internet. Sure one could say that without DNS (which for the most part uses UDP) the Internet would be unusable. BUT it is the passing of content that motivates us all to get on the Internet and search for our heart’s desire! TCP is a connection based protocol. It uses what is often referred to a three-way handshake. Simple put…

  1. Host A sends a TCP synchronize (SYN) packet to Host B.
  2. Host B then sends a synchronize-acknowledgement (SYN-ACK) packet back to Host A acknowledging the SYN packet setting up the connection.
  3. Host A sends acknowledge (ACK) packet back to Host B and TCP socket connection is ESTABLISHED.

We can gather a lot of information based on how these connections are set up. The more information you have, the better you can profile/attack a host! Let’s explore 3 different methods a bit…

TCP connect is like using a sledge-hammer to drive a nail. It will certainly get the job done BUT it isn’t the most stealthy way of foot-printing a target. It does however provide a lot of useful information about which services are actually running on the host. For example using telnet to open a connection on port 25…

billheese@corusant:~$ telnet 10.0.1.15 25
Trying 10.0.1.15...
Connected to 10.0.1.15.
Escape character is '^]'.
220 mail.somedomain.net ESMTP Postfix

We now know exactly what server somedomain.net is using for it’s SMTP server.

Next… is the Half-Open Scan. With this particular method of scanning, an SYN packet is sent from Host A to Host B. As is expected with the use of TCP, host B then sends a SYN-ACK packet back to Host A. In reality, if a SYN-ACK packet is returned then Host A now knows a service is running on that port. It does not however know which service is running. Some of the well known ports are next to impossible to change if the service is going to function correctly, SMTP being on example… but other such as HTTP can run on virtually any port. Very often system administrators will deliberately change the port on which a well know service is running merely to hide it from the casual network snooper. There particular method of scanning can be put to even more devious uses… The DDOS (or Distributed Denial of Service). This is where multiple hosts send multiple SYN packets to a host forcing the host to use resources setting up a connection that will never be completed. There is a down side to this method of scanning… it requires root access on the scanning host… but that shouldn’t be too hard to get!

Lastly, there’s the IP Protocol scanning method. This method to scanning for active host is pretty basic. It uses the ICMP (or Internet Control Message Protocol). It’s often used to send control/error messages indicating whether a host or router could be reached (or not)! Many operating systems have a known signature of how they react to an IP Protocol scan. This allows an attacker to know which OS a host is running. In our example above, we can see that the host is running Postfix… it does not tell us what OS it’s running on. Knowing the OS (and in many cases the version) would allow a hacker to try their hand at other known vulnerabilities for a particular version of OS. Sure Postfix may be secure on that host but it’s running OSX 10.4 and that OS shipped with a bug in APACHE that allows for a buffer overflow condition.

These are just a few example of methods to footprint a host. There are others but the point I’m trying to make is that just as a carpenter uses many tools to build a house, so too does the hacker use many tools to accurately profile a host.

Lying… trickery… duping… all words for the same thing. Words that have taken on a new meaning in the world of the on-line… connected… human! Words that have evolved into high stakes games on misinformation, fraud and identity theft. Words that have taken on the new moniker of social engineering!

Social engineering is the act of manipulating people into performing actions or divulging confidential information, rather than by breaking in or using technical cracking techniques; essentially a fancier, more technical way of lying (wikipedia.org, 2010). So what does this all mean? Well how many times have you answered the phone and the person on the other end of the line starts asking you questions about your mortgage? Wanting to help you reduce your rates! They start by asking benign questions and then move onto more personal information… such as your date of birth or heaven forbid your social security number. You’re happy to give away that information in exchange for $200 dollars off your monthly expenditures!

Or how about that cold call asking if you’re in charge of the network infrastructure at your place of employment? Or perhaps they want to know about what routers you use or the brand of toner purchase. Sure they may be mere cold calls… BUT they could be so much more. Social engineering in not about knocking at one door to see who answers but rather it’s about gathering as much information and using the information gathered in previous calls to further the manipulators efforts to make inroads into an organization.

In his book the Art of Deception, Kevin Metnick goes to great lengths to illustrate the ways in which we can be tricked into revealing information that may be common place within an organization but to an outsider can be very damaging if used inappropriately. In an interview in 2006 with Tom Espiner, Kevin Metnick shared his thoughts on what signs to looks for in a possible social-engineering attack.

Mostly, it’s gut instinct–if something doesn’t look or feel right. If someone is calling on the telephone, but they refuse to give any contact information, that’s a red flag. If they make a request that’s out of the ordinary, that’s a red flag. If they make a request for something sensitive, that’s when verification is necessary, depending on company policy (Espiner, 2006).

Honestly, the Art of Deception should be required reading for anyone responsible for security in any kind of organization… especially IT and HR departments! Social engineering needs to be addressed. Still and all, no matter what technical measures you introduce, people will do and say careless things under insecure conditions (Coffee, 2006). Employees need to be educated to the various forms phishing another social engineering practices both when using the Internet as well as answering the phones (Heese, 2007).

At the end of the day, humans have a need to help others. It ingrained within each of us. We have to get in touch with our inner selves… That part of us the screams out that something is wrong. We need to listen to that voice and heed its warning.

Resources:

Coffee, Peter (2006, August 14). Security Success Depends on Good Management, Retrieved on July, 6th, 2010, http://www.eweek.com/article2/0,1895,2001478,00.asp

Espiner, T., (2006, June 14th), Kevin Mitnick, the great pretender, Retrieved on July, 6th, 2010 from http://news.cnet.com/Kevin-Mitnick,-the-great-pretender/2008-1029_3-6083668.html

Heese, W., (2007, February 21), Computer system security policies – key trends, Retrieved on July 6th, 2010 from http://weblog.randomdog.net/?p=942

Various, (2010, July 4th), Social Engineering (security), Retrieved on July6th, 2010 from http://en.wikipedia.org/wiki/Social_engineering_(security)

July 4th is one of my favorite holiday’s… hitting the beach… Barbecuing… cold beers… and fireworks! BUT working in IT brings with the possibility of having you’re holiday plans interrupted by server/network outages. I can’t remember a 4th of July where I didn’t get a call that something is up with one of my servers and 2010 would be no different.

It started at 6:30AM… The main website for our Canadian office was unreachable. SO I booted my laptop and checked the site. Hmmm… It came up fine for me. Perhaps the server’s admin got to it before me. Called corporate and fixed myself a cup of coffee. 45 minutes later the phone rang again. “The site is down again!” I walked back to the computer, coffee in hand and indeed my browser timed out. Hmmm… OK something’s not right. I VPN’ed into the box and pointed the servers browser to the website and the site loaded BUT not as fast as I would have expected. The load on the server looked a bit high to me but this wasn’t my box and didn’t know what the normal numbers for the box were! I started to pour through the APACHE server error logs looking of answers. Nothing there!  Back to the browser… The page loaded fine, the speed having returned. I turned to my wife’s computer (making sure I took the local network out of the equation) and the point her browser to the site. This time I got a strange error message in the browser window…

“Can’t connect to the database too many connections open.”

Hmmm. Strange? Let me refresh my browser… the site pops back up. OK… let’s jump on the box and have a look at what’s going on… CPU utilization looks normal… MySQL looks OK… Refresh the browser… the site is still up. OK let’s have a look at the MySQL logs… Still nothing. So I called the developer to confirm that no moves were made into Production on Friday. I rebooted the box and everything seemed to return to normal. 2 hours later the phone rings once more… The site is down again Bill. Man this isn’t even my server… This is really going to be bad if I have to reboot the server every 2 hours this weekend. Opened a browser window and got the database connection error message again. OK let’s take a look that the system logs… WOW that’s funny the kernel is error’ing out and throttling back the the network stack. OK… Let’s see what netstat turns up… ouch! There were hundreds of connections in a FIN-WAIT or a SYN_RECEIVED state? What’s going to? Did some one patch the OS on this box? Nope… Let’s check the throughput of this box… 75,000 requests per second… Now one could dream but I’d think this was pretty rare occasion for this domain! OK… Let’s see if I could get at the firewall logs… Sure enough there were thousands of connections open. WOW we were in the middle of a DDoS (Distributed Denial of Service) attack. I couldn’t believe it.

The point of this is that it doesn’t use fancy network tools to figure out what’s going wrong with a  machine. I didn’t use a network sniffer. The box was not one of mine so I didn’t know the state of the server.  I used what was on the machine and started by eliminating variables. But the really big lesson learned is, it doesn’t matter how small you think your site is, it could always be the target of something like a DDoS.

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, That whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness. Prudence, indeed, will dictate that Governments long established should not be changed for light and transient causes; and accordingly all experience hath shewn, that mankind are more disposed to suffer, while evils are sufferable, than to right themselves by abolishing the forms to which they are accustomed. But when a long train of abuses and usurpations, pursuing invariably the same Object evinces a design to reduce them under absolute Despotism, it is their right, it is their duty, to throw off such Government, and to provide new Guards for their future security.

I wanted to take a moment to thank all the men and women that place themselves in harm’s way to protect myself and family… You are the backbone of freedom and liberty! Thank you!

My Roman Catholic upbringing taught that while Jesus Christ took on human form he was still God. This belief is fundamental to Catholicism. I’ve often heard my Asian friends say they go to the temple to give reverence to the ancestors. I always understood this to mean that they were paying their respect to immortal beings or gods. It wasn’t until today that I realized that in Taoism, mortals could be deified and worshipped as gods. Guess this makes sense, as a big part of Taoism is the harmony between humans and the universe!

Evidently this deification process was happening as late as the 12th century. Che Kung, who a great general during the Southern Song Dynasty (1127-1279) was deified for his devotion to the people of Sha Tin.

It is thought that he had the ability to suppress plagues and many believe that Che Kung was responsible for keeping the Sung Dynasty alive by providing safe passage for Emperor Bing and his brother during the rebellions in Southern China. It is because of this that many now considered him a god.

There are two temples dedicated to Che Kung in Hong Kong… the most famous being the Che Kung Miu near Tai Wai, in Sha Tin District, New Territories. The temple complex is once again undergoing renovations.

Throughout the temple are pinwheels. It is believed that good luck will come upon those that spin the pinwheel.

MTBF

2 comments

I work in IT and one of my job functions is to warehouse the image files of a corporate creative department. Translated… that means I buy a lot of storage. One of the things that storage admins are looking at is the failure rate of the disc drives that make up their SAN environments. The higher the failure rate of a particular drive the better your chances of having a catastrophic loss… Or in other words you’re restoring from tape if you loss a lot of drives at one time!

MTBF (or mean time before failure) is a standard measurement (in hours) we use to calculate the life of a disk drive before it fails. The other measurement we use is AFR (or the annualized failure rate), which is expressed as a percent based on the MTBF verse the amount of time that device is powered on and running. A couple of things to note… MTBF is not necessarily a devices useful life. And AFR is not meant to be applied to a single drive but rather it is the expected failure rate of any given drive within a particular production run (population).

So what does this all mean?

Well most vendors spec consumer-geared disk drives at about 300000 MTBF. That being said the key word in MRBF is M (or mean). So what we’re looking at is about half of the drive for a given population with fail in the first 300000 hours of use.

Translated again… and I got help on this one ;-)

If you had 600,000 drives with 300,000 hour MTBFs, you’d expect to see one drive failure per hour. In a year you’d expect to see 8,760 (the number of hours in a year) drive failures or a 1.46% Annual Failure Rate (AFR) (Harris, 2007).

Realizing that this is what a manufacturer quotes as the expected life, one has to ask how does that hold up in reality. Well Google did a bit of research on this and found that their failure rate was much different from that of the manufacturers. Why? Because there is no clear definition between what a manufacturer considers a failure and the real world’s expectation on these devise are.

In reality many factors will determine whether a drive should remain in production. Call is an IT admins intuition… Call is that odd clicking sound… calls it taking forever to save a file… Often time we (IT professionals) will replace a drive before it is completely unusable (or the point where we can no longer retrieve data from the device). Did the drive fail? Technically no… Practically yes! If we can’t rely on the drive to reliably save and retrieve data that it has fails for our purpose… guess some manufactures don’t see it the same way!

Resources:

Harris, R., (2007, February, 19th), Google’s Disk Failure Experience, retrieved on June 3rd 2010 from http://storagemojo.com/2007/02/19/googles-disk-failure-experience/

The Art of War is governed by five constant factors, to be taken into account in one’s deliberations, when seeking to determine the conditions obtaining in the field.

The Moral Law
Heaven
Earth
The Commander
Method and Discipline

- Sun Tzu, The Art of War