Clicky

Hi all,
Can anyone tell me what makes it hard to automate testing a command line application, with simple io redirection?

asked 12/09/2011 10:02

crazy4s's gravatar image

crazy4s ♦♦


29 Answers:
It isn't hard.  Who told you it was?  If the command has some sort of interactivity, then it becomes slightly more problematic, but a problem that can often be overcome by using the 'expect' command.

Can you give us any more details about your problem/situation?
link

answered

nemws1's gravatar image

nemws1

initially i have a quadratic solver program which written in c++, this program takes 3 input values from the user and it will return the roots value into shell variables, and we have the echo command to print the values out (but i'm not sure why do we have to do this?) is this where the problem comes in?
interactivity? can you give me some example?
i never use "expect" before, can you explain more details on this too.
Thanks.
link

answered 2011-12-10 at 23:54:59

crazy4s's gravatar image

crazy4s

how if i compare the return values and the expected values, like this
diff output expected
is this what you call interactivity, correct me if i'm wrong.
link

answered 2011-12-11 at 05:08:45

crazy4s's gravatar image

crazy4s

Interactivity - like using the 'passwd' program - you run it, but then it requires somebody to be typing something.  This *might* be happening with your program.

Can you give me the *exact* way you are running your program and getting results (screenshot)?
link

answered 2011-12-11 at 05:19:04

nemws1's gravatar image

nemws1

my output is smt like this:

[email protected] cs490$ make
g++ -c quad_interface.cpp
g++ -c quad_validate.cpp
g++ -c quad_sqrt.cpp
g++ -c quad_roots.cpp
g++ quad_interface.o quad_validate.o quad_sqrt.o quad_roots.o -o quad
[email protected] cs490$ ./quad 1 2 1
Double Real Root: -1

however i'm still figuring out how to put the return value into shell variables...
link

answered 2011-12-11 at 08:44:59

crazy4s's gravatar image

crazy4s

this is an output if the quadratic returns 2 distinct roots

[email protected] cs490$ ./quad 3 5 2
Distinct Roots: -0.666667 -1
link

answered 2011-12-11 at 09:09:55

crazy4s's gravatar image

crazy4s

and actually in my program i have an optional argument if the user want to save the return values into shell variables but i'm doing all these manually:

[email protected] cs490$ ret="$(./quad 3 5 2 RET)"
[email protected] cs490$ root1="$(./quad 3 5 2 ROOT1)"
[email protected] cs490$ root2="$(./quad 3 5 2 ROOT2)"
[email protected] cs490$ echo $ret $root1 $root2
2 -0.666667 -1
link

answered 2011-12-11 at 09:11:59

crazy4s's gravatar image

crazy4s

Ah, I see.  The usual way to get a return value from a command is with backtics:
will put a list of the files in the current directory into the variable $files.  Although your "$()" method works very similarly.

As a test I created a simple "input.txt" file that contains 3 numbers on each line:
1:
2:
3:
4:
5:
6:
7:
8:
1 2 1
3 5 2
2 2 2
3 3 3
5 2 1
3 2 1
5 2 4
6 7 2
(note these numbers may or may not make sense - I just randomly generated them).  Here's my commented script to run all of these, and print their output:
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
#!/bin/bash

# Run as:  batch.sh < input.txt

while read line ; do            # read input from STDIN, line by line
    ret=`./quad $line`          # run "quad", giving it our input
                                # results are returned to $ret

    # look for "Double" in the results, ignore all output
    echo $ret | grep Double >/dev/null 2>&1
    if [ $? -eq 0 ] ; then      # $? is the return code from 'grep'
        type="Double"           # and returns 0 for match, 1 for didn't match
    else
        type="Distinct"
    fi

    clean=`echo $ret | sed 's/.*: //'`     # remove everything up to the
                                           # first colon and the space after it

    echo $type $line $clean             # print input & results to screen
done


Is there some reason you want the return values into shell variables?  As you're seeing, this is not really do-able.  A child process is allowed to *see* shell variables of its parent, but it *cannot* set them.  The only way is to return a value in some way and have the parent set the variable itself.
link

answered 2011-12-11 at 09:37:16

nemws1's gravatar image

nemws1

No. What you describe is perhaps the easiest situation to produce automatic testing. Any command line program that reads input from standard input and writes the result to standard output is easy to test this way. As you noted, simply supply a large number of test cases on the input and then do a diff on the actual output with the expected output.

By interactivity, if the program reads and writes to the terminal and not a file, and if what it produces and the process it goes through to produce it might change and the user is expected to provide input based on some output, then it becomes more difficult to test. And if the program is graphical in nature and uses dialog boxes and such to get the commands, it becomes more difficult still. But what you described is easy.
link

answered 2011-12-11 at 09:41:23

blu's gravatar image

blu

hmm i somehow got the permission denied
[email protected] cs490$ batch.sh < Input.txt
-bash: ./batch.sh: Permission denied
i'm new to bash script, do i need to do chmod or smt?
link

answered 2011-12-11 at 09:45:27

crazy4s's gravatar image

crazy4s

Yes, chmod:

chmod +x batch.sh
link

answered 2011-12-11 at 10:08:11

nemws1's gravatar image

nemws1

but i got this

[email protected] cs490$ batch.sh < Input.txt
: bad interpreter: No such file or directory
link

answered 2011-12-11 at 10:10:04

crazy4s's gravatar image

crazy4s

Ah. Your bash might be in a different location. Try changing the first line of the script to:

#!/usr/bin/bash

But... You should read and try to understand the script as well.
link

answered 2011-12-11 at 10:15:12

nemws1's gravatar image

nemws1

btw can u explain this line echo $ret | grep Double >/dev/null 2>&1
i don't really get this grep Double >/dev/null 2>&1
and what does -eq 0 meant?

link

answered 2011-12-11 at 10:18:25

crazy4s's gravatar image

crazy4s

Ok.. so I'm taking the value of $ret, which will be something like: "Double Real Root: -1" - I'm echo'ing that, but the output is sent to the 'grep' command (which searches for strings, in this case the string "Double") instead of echo'ing to the screen.  I'm taking any output and trashing it (the ">/dev/null 2>&1" part).  The shell will automatically set the variable $? for me to the return code from 'grep'.  As my comments in the code mention, grep will return a 0 if it finds the target string, and a 1 otherwise.  I'm then testing to see if $? is equal to 0 (in shell scripts, you use -eq, -lt, and -gt for equal to, less than, and greater than, respectively).  If it is 0, I know the string "Double" was found in $ret.
link

answered 2011-12-11 at 10:29:15

nemws1's gravatar image

nemws1

i made some changes to my program which when i hit ./quad 1 2 1 it will return 1 -1 -1(the first argument is the number of roots, the rest are the roots values)
but i can't get any output

[email protected] cs490$ ./quad 1 2 1
1 -1 -1
[email protected] cs490$ batch.sh < Input.txt
./batch.sh: line 6: ret: command not found

./batch.sh: line 6: ret: command not found

./batch.sh: line 6: ret: command not found

./batch.sh: line 6: ret: command not found

1:
2:
3:
4:
5:
6:
7:
8:
9:
#!/usr/bin/bash

# Run as:  batch.sh < input.txt

while read line ; do            # read input from STDIN, line by line
    ret = `./quad $line`        # run "quad", giving it our input
                                             # results are returned to $ret
    echo $ret                      # print input & results to screen
done




link

answered 2011-12-11 at 10:54:07

crazy4s's gravatar image

crazy4s

oopps not sure what i did but it works now but is that any possible that i can parsed the ret into 3 arguments so that i can put into three different variables

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
#!/usr/bin/bash

# Run as:  batch.sh < input.txt

while read line ; do            # read input from STDIN, line by line
    ret=`./quad $line`        # run "quad", giving it our input
                                # results are returned to $ret
    clean=`echo $ret`
    echo $clean                 # print input & results to screen
done


link

answered 2011-12-11 at 11:13:27

crazy4s's gravatar image

crazy4s

so by doing diff expect output is this considered as interactivity, but how can this relates to "expect" command?

1:
2:
3:
4:
5:
6:
7:
8:
[email protected] cs490$ make
[email protected] cs490$ batch.sh < Input.txt > output.txt
[email protected] cs490$ cat output.txt
1 -1 -1
2 -2 -3
1 -1 -1
2 -4 -5
[email protected] cs490$ diff output.txt expect.txt

link

answered 2011-12-11 at 11:18:43

crazy4s's gravatar image

crazy4s

From your line 6 errors from before - you can't have spaces before/after the "=" in a bash script.

Next, forget I ever mentioned the 'expect' command (which is actually a scripting language).

I'm not sure how you're getting 3 results - your examples earlier had only one or two results.  However, you can grab any specific word out of your results by using 'awk':
1:
2:
3:
4:
ret="1 -1 -1"
first=`echo $ret | awk '{print $1}'`
second=`echo $ret | awk '{print $2}'`
third=`echo $ret | awk '{print $3}'`
link

answered 2011-12-11 at 11:25:26

nemws1's gravatar image

nemws1

okay so is easy to do automate testing if i read the input from a file and output to a file and compare the output file with the expected file, but to automate test  input from the user and output result to the terminal, this is the one which causes it hard to test is it?
link

answered 2011-12-11 at 12:24:54

crazy4s's gravatar image

crazy4s

'hard' is relative.  I know how to do it, so its not hard (for me). ;-)
link

answered 2011-12-11 at 12:47:13

nemws1's gravatar image

nemws1

is it possible to teach me here or is there any tutorials cause i'm new to this.
thanks in advance.
link

answered 2011-12-11 at 12:59:47

crazy4s's gravatar image

crazy4s

The main download page for 'expect' as well as bunch of links to tutorials is here:

  http://www.nist.gov/el/msid/expect.cfm

Just glanced through this, and it looks good (examples, explainations, etc.):

  http://www.nist.gov/manuscript-publication-search.cfm?pub_id=821307

Be forewarned, though, as 'expect' uses TCL scripting syntax (ie - it isn't bash)
link

answered 2011-12-11 at 13:06:21

nemws1's gravatar image

nemws1

i didn't dl the tcl n try it but according to what i've read does it work something like this else how should it looks like?

send "./quad 1 2 1"    // this sends the command which executes the program
expect "1 -1 -1"           // this is what we expect to be output to the screen
link

answered 2011-12-11 at 13:23:35

crazy4s's gravatar image

crazy4s

Kinda... again, your script is NOT interactive.  Let's say when you ran 'quad' that it ignored everything on the command line and instead spat out:

Give me the first number:

And then waited for input from the user.  Then you would use something like:

send "./quad"
expect "first"
send "1"

And so on...
link

answered 2011-12-11 at 14:16:03

nemws1's gravatar image

nemws1

hmm don't really get it but my program needs at least 4 arguments to be entered to execute the program and get the return values ryte?
if i do send "./quad" isn't it will give me the error message? or did i mistaken anything?
link

answered 2011-12-11 at 19:33:38

crazy4s's gravatar image

crazy4s

No - you're not mistaken.  Only in that you *might* need expect.  You DO NOT need to use expect.  It will gain you nothing in your example.
link

answered 2011-12-11 at 19:49:30

nemws1's gravatar image

nemws1

so if only i have separate arguments lk this only need to use expect , is it?
send ./quad
expect "a"
send "1"
expect "b"
send "2"
expect "c"
send "1"
but i still can't get what's the use of this:(
link

answered 2011-12-11 at 19:51:37

crazy4s's gravatar image

crazy4s

Yup, exactly.

What's the use of expect?  Have you ever run the 'passwd' command to change your password on a *NIX system?  By default, it *requires* a human to interact with it.  But you can get around that with 'expect'.  Another oft-used program (of the past) that one would often use 'expect' with is FTP.

You don't need it for your program, but when interacting with other programs, you might need to use expect.
link

answered 2011-12-11 at 19:58:08

nemws1's gravatar image

nemws1

Your answer
[hide preview]

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Tags:

×1

Asked: 12/09/2011 10:02

Seen: 658 times

Last updated: 12/15/2011 11:23