gzip process won't die on Linux server

[[SCUM] McPhil]

Since everyone seems to be rubbing their twigs and berries over Linux, I thought I'd slowly start a knowledge base for real life Linux administration issues that I deal with on a daily basis and hope someone benefits from it. Speedy if you would create a Linux section of the forum, that might be a great place for these.

==========================

One of the CentOS 5.2.x Linux servers I administrate started crashing around 4am GMT this morning. Seeing how this is an ecom webserver, it needed to get online now. I had to reboot the server from the source location as it was completely locked (no shell access). Once the server came back online, the CPU started at 100% straight away. With patience I was able to bring up a shell and login as root. the first command you should type when trying to see what is running on the server is:

#top

This will show you a list of running processes, CPU usage, memory usage, disk swap usage, how many users are logged in, etc...

In this situation there were probably 20 or more instances of gzip running. My thoughts quickly went to wtf... Straight away I thought I need to stop all the gzip processes and figure out wth happened.

There are a few ways to stop a process.

#kill -9 PID

or

#kill -9 process_name

PID equals what ever the PID or process number is. process_name equals what ever the name of the process is you want to kill.

In my situation I used killall because if I used kill I would have to generate a list of all the gzip PIDs running and that would suck. So I typed:

#killall -9 gzip

So, the CPU usage dropped and all the gzip processes died. Great, time for tea. NO! slowly the gzip processes started up again. My thoughts quickly returned to wtf...

In Linux most processes are dependent on other processes. Called child processes. Similar to the process tree in winders based systems for you of the winders users. I was thinking that since gzip started cropping up again so quickly that it was a child of something else running. A great way to see all the processes running on your server with PIDs and the child process, is:

#pstree -p

This will print you a list of all currently running process on the server. Scroll through the list in the shell window and look for the offending process - in my case gzip.

I found gzip in a process tree of about 5 processes. i don't remember exactly what it was but it was similar to this:

ââcrond(1823)ââsh(4005)ââbackup.sh(3979)âââtar(4003)ââgzip(5605)

You can see gzip was a child of tar, tar was a child of backup.sh, backup.sh was a child of sh, sh was a child of crond. Suddenly my brain kicks into gear and I say AH!!!

There is a crontab set to run once a week to backup the entire site with the databases. In recent history this website had gotten so large with a massive database the backup script had incapacitated the 2 quad-core Xeon processors and the 4GB of server memory. So I ran the killall command on these processes so I can sort out a fix for backup very soon after.

#killall -9 crond

etc... The server is now running happily and I've manually archived a backup so my boss won't rip me a new arse if the site looses data. Now I've got to sort out an other way to backup the site without crashing the server!!! I hope this helps someone in the future and please ask your linux question here. I'll do my best to help!

[[SCUM] Leroy]

Cool stuff McPhil - nice to see the thought process.

One bug that is pissing me off atm is when in Add/Remove programs I get some annoying netmailer settings pop-up (can't remember exact name will check later) when install/uninstalling programs. It doesn't stop me from actually getting what I want just wastes time and gives me a needless error. Maybe I can check if its a "child" of something else and see if I can fix it

[[SCUM] Miginty]

howdy mcphil, this database idea of urs could come in extreemely handy for me as im doin security and forensics in uni so pls update when you can be bothered

but out of curiosity cud u have used the netcat and pipe it into grep to see what the server was running, would that have done the same as what you were saying?

[[SCUM] McPhil]

sure, but a bit overkill. I wouldn't have need that much info... but sure could have done.

[[SCUM] Miginty]

overkill u say hhhmmmm thanx, will keep that in mind