05 Jan 2012 @ 8:00 PM 

Hello all,

The last post before a long week-end (I will not be able to post tomorrow, and on Monday and Tuesday. I’ll see you on wednesday.

Today, the topic is “The difference between find -exec and find | xargs”

The situation

You have to look into a specific directory and subdirectories for all the files named “*log” because you are quite sure one of them contains the word “error”.

You have two ways to do this :

1 – using -exec

This is a parameter to the find commands that allows you to do an extra command on the found files. The syntax is as follow :

-exec <command> {} \;

<command> replaces your command

{} will be replaced by the name of the found file.

\; terminates the command

i.e. you type :

find . -name \*log -exec grep -i error {} \;

and it will return all the lines containing error, regardless of the case.

2 – using xargs.

xargs is a command that allows the piped data to be passed as parameter to another command. The syntax is as follow :

| xargs <command>

xargs simply puts at the end of the command the piped data. i.e. you type :

find . -name \*log | xargs grep -i

3 – Using a while loop.

Yes, you can do like that, but it is not the topic of this discussion.


The difference

What is the main difference between the two?

-exec is going to take each file, and execute the command with it. Using this, you will get a list of all the lines, but not the name of the file, as grep assumes you know which file he talks about, as you have passed the name as parameter !

A sub process will be spawn for each file to be checked.

xargs, on the other end, is going to pass the complete list of files to grep. The name of the files will be shown in the result list.

The separator for the parameters will be the space, and this is OK as long as there is no space in the names of the files (but who puts spaces in the names of the files ? Ah, ok, users…).

In fact, the fail proof  way to deal with this specific request is to make the while loop.


The conclusion

Both ways can be useful, depending of the request and the situation. It is important that you understand the different ways those tools work, so that you can choose which one is the best for your usage.

Thank you for reading, and see you next wednesday ! (told you, long week-end for me ūüôā )


Posted By: Dimi
Last Edit: 02 Jan 2012 @ 05:17 PM

EmailPermalinkComments (0)
Tags: , , , ,
Categories: Bash, basics, Snippets
 03 Jan 2012 @ 8:00 PM 

Today, we’ll see something interesting about the scope of variable in bash, and maybe more a bug than something else.

Before going further, I want to say that I have never had problem with the scope of variables in Bash, but this specific case is very strange.

The situation

Let’s say you have a file “tempo1”, containing three lines :

COMP 2 + 3 + 4
COMP 4 + 5 + 6

You want to compute all the lines that contain “COMP” and provide the total result.

You then write the following script (usually in one line) :

typeset -i i j
cat tempo1 | while read line; do if echo $line | grep COMP >/dev/null 2>&1; ¬†then ¬† i=`expr \`echo $line | cut -f2- -d’ ‘\“; ¬† echo $i; let j=j+i;fi; done

(if the script is unclear, I’ll detail later on)

You assume that, at the end, “j” will have the sum of all, and “i” is going to be the last processed number. Well, you’re wrong.

The problem

At least if you use bash. There seems to be an issue with the context copy for a piped while. i.e. both your variable will be defined, but empty.

$ echo $0 i is $i j is $j
-bash i is j is

Now, let’s re-run the whole lot in ksh :

echo $0 i is $i j is $i
ksh i is 15 j is 15

The reason

Well, after looking on the web, it seems that I am not the only one puzzled. This has been reported as a bug, and we’ll see if it changes in the future.

The solution

Change your code, to avoid the piped while :

while read line; do if echo $line | grep COMP >/dev/null 2>&1; ¬†then ¬† i=`expr \`echo $line | cut -f2- -d’ ‘\“; ¬† echo $i; let j=j+i;fi; done <tempo1

and the problem disappears. Or use ksh for this specific purpose.

Thanks for reading this, and see you tomorrow ! (Should you require more explanation about the script, feel free to read on.)


Explanation of the script

Let’s first explode the script on multiple lines :

typeset -i ij
cat tempo1 | while read line
if echo $line | grep COMP >/dev/null 2>&1;  then
i=`expr \`echo $line | cut -f2- -d’ ‘\“
echo $i
let j=j+i

So, the first line :

typeset -i i j

is there to define the variables i and j as integer (-i) (equivalent to int i,j; in c)

The next line is parsing the file, and looping each line between the “do” and “done”. The line is put in the variable “line”

If you had written

cat tempo1 | while read JamesTKirk

then the variable containing the line would have been JamesTKirk, which is a bit long to type and also a bit of an overkill to have one of the best Starfleet Captain as your variable.

Next line speaks by itself :

if echo $line | grep COMP >/dev/null 2>&1;  then

means “if the result of the command “echo $line | grep COMP” is 0, then execute the following” and of course, this will only be the case if the line in question contains the word “COMP” somewhere. We could have done maybe with “grep -e ‘^COMP'”, or with the funny test we’ll see tomorrow, but let’s not split the hair.

i=`expr \`echo $line | cut -f2- -d’ ‘\“

expr allows you to do basic arithmetical calculation. This line means “Take everything after the first field on the line, space being the separator, then compute it and store it in the variable “i”“.

let j=j+i

let is another way to do arithmetical operations, using integer  variables.

and voila, done.

Should you require more assistance, feel free to post in the comments or to man the command you want to know about.

Thank you for reading, and see you tomorrow for a special test operator.

Posted By: Dimi
Last Edit: 02 Jan 2012 @ 04:37 PM

EmailPermalinkComments (0)
Tags: , , ,
Categories: Bash, Snippets
 19 Dec 2011 @ 8:00 PM 

Background info.

As you know (and if you don’t, just say you do), when you quit a terminal, all the processes that have been started receive a signal type 1, SIGHUP, for SIGnal HangUP.

Usually, the processes that receive this signal, if they are not designed to trap it, will simply exit. This is why, at school, your teacher told you to start the background scripts with nohup, to protect the process.This is a well-known procedure and everybody uses it.

Your code should look like this :

nohup ./myGreatScript.sh >>./logs/mygreatlog 2>&1 &

The issue.

The issue we had was that some of the scripts and processes (Java and C)  we started on one of the new servers were crashing after we quit the session. The first reflex of the support guy was to start the scripts with nohup, which is correct but not very convenient.

The problem, you see, is that I have been using theses scripts for, like 10 years, without experiencing this behavior once. I assumed the main script contained this nohup¬†command, and that, for some reason, someone did remove it… It was not the case, the nohup has never been into this script.

How did it work ?

This was a legitimate question… The script worked¬†for ten years on different machines, but on this one, without using nohup.

I went then through the whole script, line by line, to try to understand what was happening. My first idea was that the processes were started using tee, which can be done to protect it.

Tee can be used to protect a service from a SIGINT process (using -i option), but not a SIGHUP

The solution ?

In fact, the issue was nowhere near the script, but lied in the default shell used by the user on each machine.

On all the machine where it worked, it was either ksh or bash. On the one where it crashed (all the machine were Solaris), it was sh.

Both ksh and bash seem to react differently on the exit of a session, ksh and bash do not send a signup signal to all the child processes unless the huponexit shell option is set. To check the option status, at least in bash, you can use the following command :


and look for the line that says huponexit. To change the value, you can use the -u or -s options.

The solution here was to change the default shell from /bin/sh to /usr/bin/bash.

Please note that the shell options are not passed on to the child processes for some reason. I tried to get an explanation on this, but could’t. If anyone has an idea, please make yourself heard !


Thank you.

Posted By: Dimi
Last Edit: 19 Dec 2011 @ 10:53 PM

EmailPermalinkComments (0)
Tags: , , , , , ,
Categories: Bash, Signals

 Last 50 Posts
Change Theme...
  • Users » 67
  • Posts/Pages » 25
  • Comments » 4
Change Theme...
  • VoidVoid « Default
  • LifeLife
  • EarthEarth
  • WindWind
  • WaterWater
  • FireFire
  • LightLight