MYNITOR.COM

Sometimes when your website receives an unexpected flow of traffic, it’s a great feeling of joy but it can also cause virtual indigestion to servers that aren’t able to keep up with the demand.  For this reason, numerous caching methods are available to serve dynamic pages as static only content so the load relieves all stress from the database and application servers.  However, since all files get served as static files, you’re now putting all the stress on your file system and hard drives.

Depending on the i/o and TPS capability of your storage devices, it could determine whether your server can withstand the load.  If i/o is not able to withstand, you’d then start experiencing CPU performance degrade and system load averages hike up.  The result is a system hang and unresponsiveness and most importantly loss of all that nice traffic surge you were expecting.

So how do you get around these issues?  There are several ways and usually people just tend to scale up their hardware resources horizontally by adding more servers.  This is obviously an expensive solution for a short-term so you should try all caching and software based alternatives rather than jumping to hardware solutions.

Here is a method that has helped us greatly in the past.  Say you have a server that has 16GB of physical memory and your primary website size is around 2GB total.  You can create a TMPFS and mount it to be perceived as local files system and copy all your website content to it, then make adjustments in your web config to point to the new docroot.

To create a file system with 2GB borrowed from physical memory:

# mkdir /www/mywebsite.com

# mount -t tmpfs -o size=2G,nr_inodes=10k,mode=0775,noatime,nodiratime tmpfs /www/mywebsite.com

# rsync -v -a /home/myoldwebsitepath/mywebsite.com/ /www/mywebsite.com

# service httpd restart

Now your entire website will run from memory and you should notice considerable boost in performance and system load dropping like a rock and staying down.  This method only works until your system gets rebooted and then you’d have to follow the above steps once again.  To automate, simply copy and paste the above lines into a script.

Awesome tools that simply breaks your Linux server and puts you in a box to go ahead and figure out what went wrong.  This is a great method to even use on interviews or simply test your own ability of recovering a Linux operating system when it’s bad.  Perhaps you’re studying for a certification, test your ability by running these tools, not knowing how it’ll break your system but you’re job will be to fix it back to normal.

Trouble Maker

There are a lot of tools out there to make the system administrator’s life easier. However, no tool is a replacement for properly understanding the system and experience in troubleshooting unknown situations. This is where Trouble-Maker comes in. Unlike other projects, we do not attempt to solve problems — we cause them.

When installed and run, this project will randomly select a problem from its set of issues and make it happen on your system. This can give you experience dealing with:

• Dealing with partially accurate user reporting of problems
• Troubleshooting boot problems
• Troubleshooting service configuration problems
• Troubleshooting (simulated) hardware problems

By reading around on such tools, trouble-maker seem to be among the first ones that are mentioned by people.

• Download Trouble-Maker.

Damn Vulnerable Linux

This one is not exactly a tool that you can run on your existing environment.  DVL is a Linux distro and it comes packaged with everything that could have gone wrong with a Linux system.

Damn Vulnerable Linux (DVL) is everything a good Linux distribution isn’t. Its developers have spent hours stuffing it with broken, ill-configured, outdated, and exploitable software that makes it vulnerable to attacks. DVL isn’t built to run on your desktop – it’s a learning tool for security students.

The main idea behind DVL was to build up a training system that I could use for my university lectures. My goal was to design a Linux system that was as vulnerable as possible, to teach topics such as reverse code engineering, buffer overflows, shellcode development, Web exploitation, and SQL injection.

• Download DVL.

I’m sure there are others out there, so submit a link in comments area.

There are times when tcpdump more convenient to use than wireshark such as on a remote server where wireshark is not installed on.  Also, tcpdump is installed on many default Linux installation and is widely used for network troubleshooting.

On the other hand Wireshark has a great gui interface that is flexible and can be customized to narrow down and view network captures easily.  As a result, simply use tcpdump to capture data and bring it over to wireshark for processing.

The command to capture all data and not just the default packet size in tcpdump, type the following on your command line:

# tcpdump -i eth0 -s 65535 -w capture.out

Where…

• eth0 – Network interface
• capture.out – The file name tcpdump is capturing data in.

So I just found out there is a command line binary called watch on Linux and well it’s surprisingly helpful, especially if you’re looking to run a command at regular intervals.

Normally, when I want to check a status of a directory or periodically check the netstat table, I write something like the following:

$ while true; do netstat -an | grep ; done

Now that I know about the watch command, I can simply do this instead:

$ watch netstat -an | grep 

The above command runs every 2 seconds, refreshing your screen.

Type man watch to see more examples and what else you can use it for.

If you’re on a Mac OS using Cisco VPN, more than likely you have seen this error.  In order to fix that error, simply type this command:

sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN restart

And you’re set! Restart the VPN client and you should be set to use it again.

So you’re logged into your Linux server on console and have downloaded some patches and need to get to single user mode?  How do you do it without going through the trouble of rebooting again and selecting run level from startup options?

By default, Linux boot in run level 5 so if you want to get into a different run level such as single user-mode, just type:

$ sudo init 1

That’s it.  Do what you need to do at single user mode and then to get back to default run level, type:

$ sudo init 5

That’s it.

Say, you need to reboot anyway.  To quickly do a reboot, type:

$ sudo init 6

And we’re done here.

You’ve probably seen internal dummy connection in your Apache log file and must be wondering what the heck is it and why is happening so constantly?

Well, the reason you’re seeing that your Apache access_log is because it’s part of managing the child processes and wakes itself up by sending a request to itself.  This is purely harmless and by feature and if it’s annoying you, then simply disable it in your CustomLog directive in Apache config.

I came across a great iptables one liner that changed my overall approach on whether using root to run a web server or any other application is even necessary anymore.

I wanted to run an application on port 80 but without being root.  So after digging around a bit for a solution, I found that iptables can be used to setup a NAT to re-route requests coming on port 80 to a high port which a regular user has access to.

So I started my application to run on port 9001 and then setup iptables to route incoming port 80 requests to route to 9001.  Here is the awesome one-liner:

iptables -t nat -A PREROUTING -p tcp –dport 80 -j REDIRECT –to-port 9001

Just add the above line to your startup script so it runs BEFORE your application starts and binds to port 9001.  You can tell all external users to connect to port 80 of your host and as soon as it reaches the server, iptables re-routes the port 80 request to port 9001.

I’m sure many out there is actively using Cygwin because they can’t stand being on a Windows platform.  I for one am one of those people.  I install Cygwin on any Windows based servers to login to it remotely via SSH and do what I need to do.

Recently I came across an issue where I had to login from server to server within Cygwin environment via passwordless SSH.  I had already installed openssh server and it was running as a service.  HOWEVER!  It was installed under the default “SYSTEM” windows user and as a result I could never get it working.

Finally, I came across some sources that pretty much indicated that in order for passwordless SSH to work in Cygwin, SSH server must be installed under it’s own user such as sshd_server.

Once I did that, it all worked out!

To delete trailing spaces at the end of a line…

To do it against a single file do:

$ sed 's/[ \t]*$//' file.orig > file.out

To do all files in current directory do:

$ for i in `ls -l | awk '{print $9}'`; do sed 's/[ \t]*$//' $i > $i.out; done