29 Dec 2011 @ 8:42 PM 

Today, something a bit more complicated. This is a fun way to use a trap function we have seen yesterday .

The problem

You receive a new requirement from your customer (I did receive it a couple of times). You need to develop a program that will :

– Not consume CPU or disk until it is necessary.

– Read the content of a directory, then process each file in a directory A (move it to another directory B ) one by one. Not two files at the same time.

– Once the file has been “taken” by another process (no file in directory B), the next file can be published.

– You can’t have the system polling every minute or so, because the files need to be copied as fast as possible.

– The process that provides the files can start another process, but can’t wait more that a millisecond before giving the hand back.

Good luck…

The Solution

Well, at least the one I found, was to have a main process going to sleep, then being awoken by another one (triggered by the “file-provider”), and starts polling the directory A until it is empty. Then it goes back to sleep.

How ? well, like this :

First, we need to trap the “KILLUSR” signal :

trap “wakeup” 30

Then, we need to create a script, on the fly, that will be called to signal the main script that there is something to process (the “trigger” process)

trigger=trigger.sh
echo “#!/bin/ksh” >$trigger
echo “#Created by $0 on “`date` >>$trigger
echo ‘if [ “$1” == STOP ]’>>$trigger
echo “then” >>$trigger
echo “kill $$” >>$trigger
echo “else” >>$trigger
echo “kill -30 $$” >>$trigger
echo “fi” >>$trigger
echo “#End of the script” >>$trigger
chmod 755 $trigger

Next step, we create the function that will be executed in case of a trap.

_From=/from
_To=/to

wakeup () {
trap ‘echo “\c”‘ 30
# This is basically doing “noop”, to avoid being killed.
echo “Awoken at “`date` >>$0.log
while [ -f ${_From}/* ]
do
while [ -f ${_To}/* ]
do
sleep 1
done
File2Move=`ls -tr ${_From}| head -1`
mv ${_From}/${File2Move} ${_To}
done
echo “${_From} is empty” >>$0.log
}

And, last step, we need the code that will be executed while waiting :

while true
do
sleep 1800&
wait
done

When the script is waiting, the main part is just looping on sleep. Them, once it receives a trap, the function that does the copy is executed.

Afterwards, we take the first file sent, move it over, then wait until the destination directory is empty, and look for the next file.

If you have another idea on how this process could be done, feel free to comment !

Thank you for reading, and I hope to see you on tuesday !

Posted By: Dimi
Last Edit: 30 Dec 2011 @ 03:15 PM

EmailPermalinkComments (0)
Tags
Tags: , ,
Categories: Bash, Signals, Snippets
 19 Dec 2011 @ 8:00 PM 

Background info.

As you know (and if you don’t, just say you do), when you quit a terminal, all the processes that have been started receive a signal type 1, SIGHUP, for SIGnal HangUP.

Usually, the processes that receive this signal, if they are not designed to trap it, will simply exit. This is why, at school, your teacher told you to start the background scripts with nohup, to protect the process.This is a well-known procedure and everybody uses it.

Your code should look like this :

nohup ./myGreatScript.sh >>./logs/mygreatlog 2>&1 &

The issue.

The issue we had was that some of the scripts and processes (Java and C)  we started on one of the new servers were crashing after we quit the session. The first reflex of the support guy was to start the scripts with nohup, which is correct but not very convenient.

The problem, you see, is that I have been using theses scripts for, like 10 years, without experiencing this behavior once. I assumed the main script contained this nohup command, and that, for some reason, someone did remove it… It was not the case, the nohup has never been into this script.

How did it work ?

This was a legitimate question… The script worked for ten years on different machines, but on this one, without using nohup.

I went then through the whole script, line by line, to try to understand what was happening. My first idea was that the processes were started using tee, which can be done to protect it.

Tee can be used to protect a service from a SIGINT process (using -i option), but not a SIGHUP

The solution ?

In fact, the issue was nowhere near the script, but lied in the default shell used by the user on each machine.

On all the machine where it worked, it was either ksh or bash. On the one where it crashed (all the machine were Solaris), it was sh.

Both ksh and bash seem to react differently on the exit of a session, ksh and bash do not send a signup signal to all the child processes unless the huponexit shell option is set. To check the option status, at least in bash, you can use the following command :

shopt

and look for the line that says huponexit. To change the value, you can use the -u or -s options.

The solution here was to change the default shell from /bin/sh to /usr/bin/bash.

Please note that the shell options are not passed on to the child processes for some reason. I tried to get an explanation on this, but could’t. If anyone has an idea, please make yourself heard !

 

Thank you.

Posted By: Dimi
Last Edit: 19 Dec 2011 @ 10:53 PM

EmailPermalinkComments (0)
Tags
Tags: , , , , , ,
Categories: Bash, Signals

 Last 50 Posts
Change Theme...
  • Users » 66
  • Posts/Pages » 25
  • Comments » 4
Change Theme...
  • VoidVoid « Default
  • LifeLife
  • EarthEarth
  • WindWind
  • WaterWater
  • FireFire
  • LightLight