bill's blog

Just another WordPress weblog

Browsing Posts tagged game plan

During the course of our day, we are often asked to provide answers. What’s wrong with the server? Who’s going to take the late shift? How can we implement this? We rely on the skills that we have developed over years of work and provide quick answers! Most times we’re right but every so often because we haven’t put the time into thinking things through, we are wrong! And the thing is most times the powers that be don’t always consider the things that have went well, BUT rather focus on those few times things that went wrong!

So what are we to do? Look forward and anticipate next steps. Think logically about the problem. Come up with a game plan! Sure following your gut will help but only to a certain extent. While your initial response may be right most of the time it is not always on the money! Part of preparation is mitigating risk. If we are studying for a test, we concentrate on what we feeling are the important concepts… Studying the longest on those concepts! You may get some answers wrong BUT if you do your homework you may still walk away with an A even though you got a few answers wrong. You have to balance getting every question right verses getting most questions right. Acceptable losses!

Putting together a game plan often requires more time and effort than actually implementing a solution. Understanding the why is of the utmost importance! Knowing your audience and the reasons they need the technology rolled out is the key to a successful rollout. We are not here to simply implement the next coolest technology. We are here as enablers! We are here to leverage our understanding of technology with a clear business need!

Once you have the business need figured out, its time to start thinking about a rollout plan. What is the return on investment? How many man-hours will it take to get this done? Will this work in our environment? How will we do a proper pilot program? The list goes on and on. It is up to us to try and discover as may of the potential problems BEFORE we put our solution into production! Take copious notes! Remember you’ll need to reproduce this and introduce it into production. Focus on the important points. Understanding the scope of the project will help you determine whether or not an obstacle is truly an impediment to a successful rollout! Remember… there will always be problems; it’s just a matter of whether they are show stoppers!

This leads to how you are going to mitigate the problems you can’t get around. I’m in the middle of a fairly large migration from one directory service to another. It requires me to make changes to servers throughout the world. I can’t update all the servers globally in one weekend. SO… I’m going to have some users that will need to remember two passwords when they log in on Monday morning. One for their login the other to access resources not yet migrated.

So how are we going to break down this problem?

THINKING THINGS THROUGH – In an optimal situation I’d be able to coordinate the migration globally. That would assume I had fully qualified people situated at each location that had a server that needed to be upgraded. This will lead to users having to remember two passwords and which to use at the appropriate time because we will have two directories running at the same time.

UNDERSTANDING THE RISKS – Unfortunately this is not the case. For power users this shouldn’t really be a problem but for novices this could lead to multiple failed logins that would have the helpdesk resetting passwords the first day after the implementation. So how do we avoid all the phone calls about failed logins on Monday morning. We could have the two directories trust each other. BUT that would require a lot of extra work.

MITIGATING THE RISKS – We can use Keychain Access (yes we’re talking about an OpenDirectory migration) to securely store the passwords from the old directory. This gives the impression of single sign-on. Alleviating the need for users to remember multiple passwords.

Yes this is really simple and it doesn’t truly depict the actual planning of the directory migration… if it did I’d be out of a job! BUT the point is you need to think about what you’re trying to do. There really isn’t one right solution. The solution is dependent on how your organization is able to handle the situation at hand. Think through the issues ahead of time is most important. Create troubleshooting checklists to pass out to those individuals (both the end-user and the helpdesk staff. Test, test and test again. The more you plan, the better your chances of a successful rollout!

When working in IT one needs to have a game plan… a road map so to speak with regard to fixing problems. One needs to understand what is happening and look at the problem from a number of different perspectives (Our servers’ hard drives are filling at random intervals… it’s got to be a server problem). One needs to understand what is causing the problem… more often that not… What’s changed in the environment? (Well we installed the new version of Firefox onto everyone’s machine yesterday!) Then how to go about fixing the problem? Remove Firefox from everyone’s machine? But wait… problems within IT often aren’t that straight forward… often times one cannot address the problem directly… “We need to use Firefox because our WebApp requires it” BUT wait… it’s this feature that is causing the problem! “If we turn off that particular feature it will allow most of us to use Firefox although some users could still have other problems”. We’ve provided a fix for the greater good… but is it really a fix? It depends!

Having a game plan as to how you are going to attack the problem and sticking with the game plan can make the difference… finding a workable solution! Understanding what you are looking for (and that can include data that you don’t know is there) and why can only help to keep you focused. The game plan isn’t always the same…certainly the rules are different if you’re working in a corporate environment verses a government organization. They can be different depending on whether it’s a criminal matter. You as the technical expert need to understand that the suspect has rights that cannot be infringed upon or you may find that all your hard work is inadmissible in court. Make sure you have the company’s permission, in writing, before you start poking around on other employees’ computers. Know who is authorized to give the OK to begin your work. Don’t start the work until you have everything in place.

Be Professional! Stick to what you were hired to do! It doesn’t matter whether you’re a salaried employee or a consultant! Be objective! Don’t form opinions until you’ve done your homework. Forming opinions prior to starting your work could lead you down the wrong path and waste valuable time. Keep your mouth shut… you never know what you’re going to find… Confidentiality is often equated to trust. In IT we often have more access to information than our bosses! Don’t sneak a peak and their salary information. You may not like what you find! If people can’t trust you, you’ll find yourself unemployed.

Assessing the problem, formulating a game, determination of follow through, and the ability to compromise to get results one can live with… are all qualities that a good systems administrator needs to have. Without these skills one is just shooting in the dark and hoping for the best and often this is just not good enough. Things will pass you by. I recently had the opportunity to put these skills to the test and while results of this Endeavour has yet to see fruition only time will tell.

So how do these skills translate into tools for those dealing with intrusion detection? One must realize that it not a question of if but rather when you will need to put your skills to the test. Let’s take a look…

Assessing the problem

This might not be a current problem that needs to be solved right away. Very often in intrusion detection is a matter of understanding what a future threat may be. Networks are under continuous bombardment! Some of which are malicious acts with a particular goal in mind (DOS attack), while other may be normal (or not so normal) activities on your network. Having a baseline is the only real way to understand your network. Very often this is an impossible task. Short of a baseline, understanding what traffic is on your network will go a long way to understanding when something goes wrong. A network grows rapidly and in a complex environment other administrators in different faculties could be working against you. Not in a malicious way but rather we as administrators should take the role of enabler rather than policeman. Sure we can’t let our users run amok on the network, but we should understand the need and then figure out a way to make it happen.

Formulating a game

Once we understand what the threat is we need to come up with a game plan as to how we are going to deal with it. Risk assessment is key. Assigning a pain index to the threat is the next step. We have to determine which threats are most important to deal with and which ones should be back burnered. One needs to realize that not every threat can be effectively dealt with. Sometimes there’s nothing that can be done short of taking your machine off the network and locking it in a closet. Having a policy with clear steps of action is important. Know who to call and when. Don’t try to go it alone. It’s like any other emergency… one-person drives; the other is on the phone. Depending on how large the event is you may need to coordinate with other individuals in different locations to contain the incident.

Determination

This is the hard part because it centers on you! You’re the one that can see things through. It’s not always easy to be the last one in the office. Sometimes you have to put on blinders and focus on the issue at hand. Many people will offer suggestions, but sticking to the game plan is all that matters. The policies are in place. Deviating from the course will only add to the confusion.

Ability to compromise

The down side to determination is things don’t always go your way. There is a point of diminishing returns. Sometimes you have to cut bait. WHY? Because it may be easy and cheaper to rebuilt your system from scratch. Sure we’d all like to get to the root cause of the problem. In some cases we will be mandated by compliance to a standard or various laws… BUT sometimes not. One has to remember that the machines we’re trying to fix are not our play toys. They are business tools with users that need to get access to them. Knowing when to put the brakes on and figure out another solution is important. A good policy will have contingency plans and if not you should take the lessons learned from the compromise and figure that into the revision of the plan. YES… It’s important to review and update your plan from time to time.

So I’ve been on the plane for 8 1/2 hours… Slept a little but not really.

It’s funny ways people deal with flights of this nature. My game plan was to eat dinner, write a term paper and then float away for 8 hours in an Ambeim induced haze. Oh well for thought out plans, sometimes the body doesn’t do what you want it to. It’s has given me the opportunity to look around at my fellow passengers. Most are sleeping peacefully, eye masks on, ear plugs in.

In all my travels out to Asia I’ve never been awake for the Chinese noodle bowl… It was refreshing though it brings me back to all the meals I’ve had in places I don’t remember. The Chinese prepare there food a lot differently than in the west. We’re used to too much MSG and salt. The taste here is much simplier.

We finally coming back into daylight… Most of my trips over have been in total daylight. The sun never goes down during the entire flight. This is the earliest I’ve ever gone. February. Guess the sun hasn’t climbed high enough in the sky yet.

Were more there than home now. All my loved ones are home sleeping. I look out of the window and watch was it snakes around itself in unending coils. I guess we’re over land now. One would never know it looking over the blanket of white beneath us.

6 hours later after finally getting some sleep, the day over China is turning to night. It’s only an hour before landing and the cabin is alive again. It’s strange how time transforms itself. My family back home are still resting in their cold New England bed… Stirring for the fort time… And I’m getting ready to get off the plane, have dinner and get into bed.

We as network administrators need to understand that we provide a service both to the companies we work for and the end users we serve. Without them we would find ourselves unemployed. IT is a service organization and as such, end users are our customers. We must understand that their needs sometimes come before our own. Sometimes this dedication includes giving of ourselves and our family in terms of the many hours that we will miss because a server is down. Fortunately we can prepare ourselves and lower the risk of downtime (and time away from home) with continued education.

Know the basics!

Confidentiality, Integrity and Availability… the foundation of everything we do. While confidentiality didn’t play out this week, Integrity and availability certain did. I spent most of the past week (on the clock and off) getting an image database online. A number of things went wrong. From an integrity point of view, we had a database that went south. It contained a record of every image the company had captured in the last 10 years. Backups proved to be too old to be of use (though as a second option something to consider). Long story short, we were able to get the database back online BUT there was corruption that needed to be addressed. This is where dedication comes into play. It would be too easy to give up on the database recovery efforts. We did have backups (though not current). Piecing together various databases proved to be the answers. While not the most elegant method it did get the database online and intact. Additionally, many hours were put into the recreation of the database to shorten the time the users were without the database. Thus availability comes into place. The game plan to pull data from various backups and stitch them together was going to take time. One must balance your own time with that of the greater good. And thus this paper was late but my end users got their data sooner rather than later.

Know the Policy!

IT is about making sure that people can work. Everyone! Sometime one individual can bring down a network. Just take a look at any virus. One person writes and distributes the code… the rest of the world suffers. IT polices are there to protect everyone both the end user AND IT administrators. So what goes into IT policies?

1. Clear understandings – This pertains to everyone in the organization. The policies are written so that everyone in the organization knows what they can and can’t do. Does the company allow external USB thumb drives? Are smart phones allowed? Who is allowed to have smart phones? Password sharing and ramifications? There is a whole plethora of things that should be covered.

2. Emergency situations – What are the procedures or actions to be taken during an emergency? What should be done? Who should be informed? When is a Disaster Recovery plan implemented?

3. Access – Who should have access to which data? AND where does one go to get access they have. What are the steps to be taken?

One thing to keep in mind is that the above questions are distributed to everyone within the organization.

Education!

Without continued education we as IT professionals would go the way of the dinosaur, though perhaps not as dramatically. IT changes rapidly. If one were to ask about virtual machines 5 years ago, no one would understand what we are talking about. More and more IT professionals are asked to take on technologies in a production environment and to support them. While reading, toying, trial and error can bring you most of the way… formal training is needed to support these advance technologies.

IT is about putting your heart and soul into your work. One must have the desire and drive to succeed in this industry. Only a select few can truly excel here!