23 Dec 2011 @ 8:00 PM 

A short one for today, then it’s week-end and I will only post on Tuesday (Monday is holiday).

In a script, one of the things which is usually overlooked is the validation of variables. And if you are developing a library of functions (for scripts), it is simply forgotten most of the times. This is one of the reason you are not re-using the functions made by your fellow developers : if you want them to work for your specific purpose, you have to specify a thousand of options that you will never use.

How can you make a function using optional variables (meaning that it will resort to default ones when nothing is specified) without writing dozens of lines of code ?

By using the little trick described in this article. This is how you define a default variable usually :

D_LOG_LEVEL=9

if [ “$1” = “” ]  #In case it is defined on the command line

then

LOG_LEVEL=${D_LOG_LEVEL}

else

LOG_LEVEL=$1

fi

 

Half your script went to write this wonderful code. Now, let’s make it faster.

First, we can replace the ifs with the one we’ve seen earlier this week :

 

D_LOG_LEVEL=9

([ “$1” = “” ] && LOG_LEVEL=${D_LOG_LEVEL}) || LOG_LEVEL=$1

Next step, we can get rid of the variable “D_LOG_LEVEL, as it is used only once.

([ “$1” = “” ] && LOG_LEVEL=9) || LOG_LEVEL=$1

Last step, let’s get rid of the not very nice “”=”” :

([ -z $1 ] && LOG_LEVEL=9) || LOG_LEVEL=$1

This allows you to define variable that are going to resort to default if it is not specified, without loosing too much time writing the code.

Note that if you define variables that need to be set before you send

([ -z ${LOG_LEVEL} ] && LOG_LEVEL=9)

Last thing about the variables in the function : You should test them before you run your function or script. This sounds obvious, but it is not usually the case (resulting is headache for people doing the support). If your script needs a filename as parameter and needs this file to exist, writing this line :

([ -z $1 ] || [ ! -f $1 ]) && (echo “File $1 Missing” && return 1)

is not going to take you much time and will help the next guy who is going to use your script.

Thank you for helping the Unix Scripting World to be a better place, and see you on Tuesday !

Posted By: Dimi
Last Edit: 23 Dec 2011 @ 04:02 PM

EmailPermalinkComments (0)
Tags
 21 Dec 2011 @ 8:00 PM 

Tomorrow, we will talk about pipes (and file descriptors). But today, let’s talk about the other ways to chain commands.

One of the easy way for you to have to commands running one after the other, without you staying in front of the screen (see you at Starbucks), is to chain them using a semi-column :

RunVeryLongProcess;PrintReportOnVeryLongProcess;SendMailToBoss

Once you will have defined the commands “RunVeryLongProcess“, “PrintReportOnVeryLongProcess“, and “SendMailToBoss“, you will get a pay rise because the three jobs have been running one after the other, for 9 hours, and you have send a mail to your boss at 11pm ! How dedicated are you ! (At least to a Venti Macchiato :-)).

Yep, but what if RunVeryLongProcess fails ? Oops… You have send an empty report to your customer, and a mail to your boss saying that everything’s OK…

To avoid this, you can use the logic operators “and” (“&&”) and “or” (“||”), e.g.

RunVeryLongProcess && PrintReportOnVeryLongProcess && SendMailToBoss

Now, if “RunVeryLongProcess” or  “PrintReportOnVeryLongProcess“, no mail will be sent to your boss. You might not be the employee of the week, but you will still have a job 🙂

This “and” operator means that if the first job succeeds, the next one will be kicked. The keyword here is if, as you can do the same with an if statement, like this :

RunVeryLongProcess

if [ $? -eq 0 ]

then

PrintReportOnVeryLongProcess

if [ $? -eq 0 ]

then

SendMailToBoss

fi

fi

I’m tired only to write this, and it is only for three processes ! So, you can now make an extensive use of the && operator.

You can also use it to replace an if statement

&& operator instead of if statement.

Let’s say you want to check the variable “a”, if it is set to “morning” you say hello, and if not, you say “goodbye”…

(you can write the if, i’ll skip it, i’m tired)

And using && operator :

([ “$a” = “morning” ] && echo “Hello” ) || echo “goodbye”

You must understand how this is working. It starts by executing the first function, in this case, a test :

[ “$a” = “morning” ]

Then, if the result is true (return code 0), it tries the other part of the equation, i.e. :

echo “hello”

If the first part of the equation is wrong, then there is no need to execute the second part. The block around the && operand will be false. It then goes to the third part, which is supposed to be executed if the && equation is false, i.e.

echo “Goodbye”

If you want to use this, you must be sure that whatever function or command you call (except the first part of course) is going to return 0. Otherwise, you can get strange results, as both the “then” and “else” commands will be executed.

There is a way to avoid it, but it is getting complicated :

([ “$a” = “morning” ] && (say “hello” || true)) || say “goodbye”

By doing this,  even if the “say” function returns a non-zero status code, the “true” is executed. True always return “0” upon execution, so this is a way to avoid bizarre results.

This is the further you should go. Starting from this, there is no point to use the && operand anymore. Just write the if statement, and the guy fixing your script later will be thankful (and it might be you).

Regarding the order of execution, the parenthesis do not change it. The operations are executed from left to right.

(btw, the “say” command exists under Mac OS X and is quite fun).

That’s all for today. I hope you have enjoyed it, and see you tomorrow for a refresh on the pipes !

 

Nota bene

The reason why the parenthesis are working is that everything inside parentheses is in a subprocess, and the result is thus the one of the whole subprocess (the whole command, with both commands inside).

Therefore, you should not use this to pass a variable, as the subprocess is not going to affect the context of the main process !

Posted By: Dimi
Last Edit: 05 Jan 2012 @ 07:51 PM

EmailPermalinkComments (0)
Tags
Tags: , ,
Categories: Bash, Snippets
 20 Dec 2011 @ 8:00 PM 

Ok, a quick one today.

If you have never used sed, it is time to do so.

sed stands for Stream EDitor. It is a way to make quick changes in a file or a stream (between pipes). It is extremely powerful, and very fast.

Its main function is to search for strings, then apply changes to it. A small example :

sed -e ‘s/hell/heaven/g’ file >file2

This would parse the file, and changes all the occurrences of “hell” by “heaven“. Try to do that without sed (even awk will make it quite difficult).

On the above example, the ‘-e‘ is not useful. You can forget it, but I prefer to put it all the time, because it becomes mandatory if you want to chain sed commands, i.e. :

sed -e ‘s/hell/heaven/g’ -e ‘1d’ file >file2

This would replace the “hell” with “heaven” in the file, then delete the first line of the file.

Some version of sed allow for in-place changes, meaning that you do not have to use a second file to write the output. However, this is not available in all versions and systems, so I would discourage the use if you are moving from one server to another.

The real power of sed are the REGEX, or REGular EXpressions.

Small example : Imagine you want to comment out all the scripts called “Boom.sh” that are called inside the other script “Badaboom.sh”, a script 123000 lines long… How do you do that ? vi ? vim ? notepad ?

sed -e ‘/Boom.sh/s/^/# /’ Badaboom.sh >Badaboom2.sh

Done !

The first part of the command ‘/Boom.sh/’ searches for all the lines that contains the string between the slashes.

The second part of the command ‘s/^/# /’ replaces, on the found line, the beginning of the line (^) with a hash-space string (# ).

REGEXs can become very useful, and if you are working with Unix, it is time for you to know the basics of this super tool sed.

 http://www.grymoire.com/Unix/Sed.html

Thank you for reading,

Dimitri

Posted By: Dimi
Last Edit: 21 Dec 2011 @ 09:34 AM

EmailPermalinkComments (0)
Tags
 19 Dec 2011 @ 8:00 PM 

Background info.

As you know (and if you don’t, just say you do), when you quit a terminal, all the processes that have been started receive a signal type 1, SIGHUP, for SIGnal HangUP.

Usually, the processes that receive this signal, if they are not designed to trap it, will simply exit. This is why, at school, your teacher told you to start the background scripts with nohup, to protect the process.This is a well-known procedure and everybody uses it.

Your code should look like this :

nohup ./myGreatScript.sh >>./logs/mygreatlog 2>&1 &

The issue.

The issue we had was that some of the scripts and processes (Java and C)  we started on one of the new servers were crashing after we quit the session. The first reflex of the support guy was to start the scripts with nohup, which is correct but not very convenient.

The problem, you see, is that I have been using theses scripts for, like 10 years, without experiencing this behavior once. I assumed the main script contained this nohup command, and that, for some reason, someone did remove it… It was not the case, the nohup has never been into this script.

How did it work ?

This was a legitimate question… The script worked for ten years on different machines, but on this one, without using nohup.

I went then through the whole script, line by line, to try to understand what was happening. My first idea was that the processes were started using tee, which can be done to protect it.

Tee can be used to protect a service from a SIGINT process (using -i option), but not a SIGHUP

The solution ?

In fact, the issue was nowhere near the script, but lied in the default shell used by the user on each machine.

On all the machine where it worked, it was either ksh or bash. On the one where it crashed (all the machine were Solaris), it was sh.

Both ksh and bash seem to react differently on the exit of a session, ksh and bash do not send a signup signal to all the child processes unless the huponexit shell option is set. To check the option status, at least in bash, you can use the following command :

shopt

and look for the line that says huponexit. To change the value, you can use the -u or -s options.

The solution here was to change the default shell from /bin/sh to /usr/bin/bash.

Please note that the shell options are not passed on to the child processes for some reason. I tried to get an explanation on this, but could’t. If anyone has an idea, please make yourself heard !

 

Thank you.

Posted By: Dimi
Last Edit: 19 Dec 2011 @ 10:53 PM

EmailPermalinkComments (0)
Tags
Tags: , , , , , ,
Categories: Bash, Signals

 Last 50 Posts
Change Theme...
  • Users » 66
  • Posts/Pages » 25
  • Comments » 4
Change Theme...
  • VoidVoid « Default
  • LifeLife
  • EarthEarth
  • WindWind
  • WaterWater
  • FireFire
  • LightLight