A useful article written 11 years ago for system administrators, but still very relevant today.
Source: TechTalk
Emmanuel Carabott
Administrators have a tough job on their hands to manage, maintain and protect the network they are responsible for. Armed with the latest tools, they do an excellent job; however, at times, often due to pressure, they make mistakes – sometimes easily avoidable.
In this post we are going to present the top 43 common mistakes administrators have made, as a reminder of what they shouldn’t do.
They are categorized under networking, configuration, security, and best practices, and are based on firsthand experiences.
Here is the list:
Networking
- Leave the trailing dot off a zone file in DNS
The first and most significant mistake a new BIND administrator can do is forget to end each zone with the trailing dot, leaving their zone dangling in the breeze as clients try to append their own domain name, and fail. - Implement HOSTS files instead of fixing DNS
HOSTS files may be necessary for troubleshooting, but should never be used in production to get around a DNS issue. Six months from now, no one will remember that server with the HOSTS file, until they spend a few hours troubleshooting why it keeps trying to connect to an old ip.addr. - Implement recursive forwarding in DNS
Forwarding is for when a DNS server doesn’t have the answer to a client query, so it can ask another server who might. Set two servers to forward to one another, and you will quickly take down your network with the resulting UDP traffic that the looping queries generate. - Allow unrestricted zone transfers
No sense making a potential attacker’s job any easier. Only permit zone transfers to your DNS servers. - Leave out WINS
Windows 2000 came out, Microsoft networks still rely on NetBIOS for several functions. A well designed WINS solution will greatly improve performance, while the lack of one can cause all kinds of client issues. - Implement LMHOSTS files instead of fixing WINS
Much like using HOSTS files for DNS, a LMHOSTS file should be used for troubleshooting a specific client, not because your WINS infrastructure doesn’t work. - Implement a disjoint namespace
There are many things you can do, but should not. This is one of them. The inconsistencies that can occur when you use a disjoint namespace outweigh any political or legacy reason to do so. - Bypass the firewall
Firewalls are there for a reason – to prevent bad things from happening and to separate security zones. Bypassing a firewall makes a bad guy’s job that much easier, and can provide them an express lane straight into your network. - Bridge networks
Whether it bypasses the firewall like in point eight, or just starts spewing internal wire traffic over your wireless network, bridging is a good idea that always turns bad. - NAT internal traffic
If you think NATing internal traffic to an external address is easier than using a split DNS, you need to reconsider. Between the protocols that can break when NATed and the user issues that can arise when trying to troubleshoot, it is far better in the long run to implement a split DNS. It also makes your firewall configure that much easier to manage.
Configuration
- Apply a patch without testing
No vendor can fully test a patch in your environment. That’s your job. Applying an untested patch is marginally safer than not patching, but eventually it will break a critical application. Bite the bullet and build a test environment. - Make a change without testing and having a backout plan
Here’s a similar concept. Untested changes will eventually break something, and not having a backout plan in place means downtime. - Make several changes concurrently
The first thing you ask when troubleshooting is “what changed?”, because often the easiest fix is to change it back. When the answer is ten items long, it’s much harder to do this. - Bounce a box figuring no one will notice
Trust me, they will, and they will scream to high heaven that they were right in the middle of something when you rebooted the server. If you cannot wait for a maintenance window, you need to at least send out an email giving them a couple of minutes notice. - Use unsupported characters in any name
Here’s another case of “just because you can do something, doesn’t mean you want to”. Whether it is “$”, underscores, “\”, or spaces, including anything other than letters and numbers, it will eventually break something – be it a script, or a new application. - Run services using their own user account
I once saw a case where an administrator installed a cluster service to run using his own account because it was the easiest way to get a new service running. 45 days later, when he went to change his password, the service died. He set his account to never expire, and six months later when he quit and his account was disabled, and the service died again. Give each service its own service account, and never use your own account for anything but your own login. - Enable anonymous FTP uploads
Unless you really do want to host illegal warez that will burn your bandwidth and use up all your disk space, never allow anonymous uploads on FTP servers. - Configure an open relay
Configuring an open relay is the easiest way to stop your users from sending email to anyone; which is also the fast path to having your mail servers put on every block list on the planet.
Security
- Leave default credentials intact
Default credentials are published, well known, and scanned for by free tools. One of the fastest ways to get hacked is to leave default credentials alone. - Use dictionary passwords
Here’s the second fastest way to get hacked; using dictionary words for passwords. It only takes the simplest of tools a few minutes to run through every word in the dictionary, making password cracking a trivial exercise. - Use non-expiring passwords
The main reason we expire passwords is so that, if they have been compromised, eventually that door is closed. Trust me, no matter how good you think your password is, it’s not that good. Change your password regularly the same way as you make your users do it. - Use shared/common credentials
Check the log to see who made that change. Who did? The administrator? Great, who knows the administrator password? Every administrative user should have their own set of credentials, and the administrator/root/supervisor account passwords should be locked up and only pulled out in case of emergency. Anything less breaks accountability and any hope for auditing who did what. - Run unverified downloads
Checksums are there so you can be sure what you think you downloaded is what you have. Always verify things you downloaded from the Internet before running them, especially when you are going to be running them using privileged accounts. - Use outbound permit ACLs instead of a proxy
Do you like herding cats? Me neither, and I also don’t like trying to tighten a screw with a hammer, or driving a nail with a wrench. Outbound ACLs should block things you don’t want all systems to do, like send SMTP or NetBIOS traffic to the Internet. If you want to control web access, use a proxy, which is purpose built for the task and can deal with FQDNs and URLs instead of ip.addrs. - Block PINGs
The PING of Death is over a decade old. Blocking ICMP echo and response does almost nothing for security, but breaks a tried and true method for testing connectivity and troubleshooting network issues. It also violates an RFC. - Deploy open Wi-Fi networks
Hiding your SSID does nothing to secure your network. Deploying an open Wi-Fi network is as dangerous as running an Ethernet drop into the alley out back of your shop, and is an open invitation for attackers to run amok on your network. At a minimum, implement WPA, and segment your wireless network from your wired.
Best practices
- Surf the Internet while logged on as an administrator
When you are logged on as an administrator, every program you run is a risk. Unless you have a sandboxed browser, a compromised website could lead to a compromised client, or worse, network. Surf the web using your regular account to reduce your risk from zero-day attacks. - Read email while logged on as an administrator
Much like surfing the web, running your mail client with your privileged account runs the risk of compromise from malware attachments, embedded scripts in email, etc. The best antivirus and antispam products in the world still rely on signatures, which can only be developed after a zero-day attack becomes a known attack. - Skip documentation
Show me an administrator who likes to document, and I will… well, I won’t have to do anything because no administrator on the planet likes to document, but it is a necessary part of the job. Even you won’t remember everything you did six months later, having documentation to refer to can make the difference between a simple task, and weeks of reverse engineering or reinventing the wheel. - Skip change logs
Much like documentation, change logs make it easy to answer that troubleshooting question “what changed?”. This is especially beneficial when that question is being asked by your assistant while you are on vacation. Unless you want to answer the phone while you’re on the beach, document changes. - Implement a new system without a scheduled maintenance window
Any new system you deploy, whether a simple file server or a complex application farm, needs to have a maintenance window established so you can do upgrades, patching, etc. Unless you like staying up until 02:00 on a Sunday morning, try to get that window approved for daylight hours. - Implement a new system without including redundancy
Having redundancy means never having to get that 02:00 call because a service went down. You may not be able to add redundancy to legacy systems, but anything new you deploy should include redundancy. - Run backups without verifying restores
“I don’t care what the backup logs say…” – until you take that tape, restore the data from it, and verify you can access the restored data, you don’t have a backup you can count on. Do you want to tell the CEO that you cannot restore his mailbox because of a bad tape? - Skip a patch
I have worked over one hundred security incidents; more than 90 of those have been hacks against known vulnerabilities for which a patch existed, but wasn’t applied. Patch regularly, patch often, and never skip a security patch. - Monitor too little
If you rely on users complaining about outages to let you know when a system has failed, you won’t last for long in this career field. Monitoring your critical systems is a vital part of administering a network. - Monitor too much
But monitoring too much leads to information overload, and pretty soon you are ignoring all the monitoring emails, which means you miss the important ones that warned you of an imminent failure. It’s going to take a lot of effort to get the right balance, and no two companies will be quite the same, but a good starting point is to get an email alert immediately only for those things that show an actual failure, or a condition that indicates an imminent failure. Anything else should be a daily summary. - Email when angry
Whether you are sending out an email bcc all, or replying to an upset user or clueless PM who has riled your feathers, emailing angry does no one any good and can damage your reputation. Take a deep breath, go grab a cup of coffee, or even put it off until the next day, but if you find yourself pounding on the keyboard while you are composing an email, don’t dare hit send. - Keep information a secret
If you are the only one who knows how something works, you are not creating your own job security; you are guaranteeing you will get called on your day off, while you are on vacation, and that you will never be able to pass it on to someone else. The best administrators are the ones who share information with others, and cross train them to reduce any human as a single point of failure. - Update information inconsistently
Any update is better than no update, but inconsistent information can be confusing, lead to mistakes, and generate even more questions that you will have to answer. Establish a format or template for any information, whether it is for your change log or for user accounts in Active Directory, and make sure all administrators follow it consistently. - Violate licensing agreements
Some risks are just too great to take, and knowingly violating licensing agreements not only exposes the company to legal action and financial penalties, it can quickly end your career. - Practice other than they preach
Users, junior administrators, and bosses alike, are not nearly as stupid as you may think. Telling them to do one thing, while you do something else, is a very easy way to lose their respect, as well as their trust. Follow the rules and lead by example. - Abuse their privileges
It doesn’t matter that you can access that file folder, should you? Administrators are in a position of very high trust, and violating that trust can quickly end your career. - Test in production
Even if the only testing you can do is in a VM running on your workstation, you need to test any changes before deploying them to production. Failing to do so is just asking for trouble, will kill your SLAs, and tarnish your reputation – it’s not worth it.
If you can avoid doing the 43 things in this list, you will save yourself, and others, time and money, avoid headaches, and minimize downtime. In other words, you will be doing a great job and taking your game to the next level. Almost anyone can be an administrator; but very few can be great administrators.
About The Author
Emmanuel Carabott (CISSP) Certified Information Systems Security Professional has been working in the IT field for the past 18 years. He has joined GFI in 1999 where he currently heads the security research team. Emmanuel is also a contributor to the GFI Blog where he regularly posts articles on various topics of interest to sysadmins and other IT professions focusing primarily on the area of information security.
