Introduction
Operating systems recognize a couple special file descriptor IDs:
STDIN-0- Input usally coming in from keyboard.STDOUT-1- Output from the application that is meant to consumed by the user, stored in a file, or piped to another application for parsing. Typically goes to terminal.STDERR-2- Used for info, debug, and error messages to the user that are not intended to specifically be part of the application output. Typically goes to terminal.
This guide will look at how you can redirect and pipe these streams for better application development and system administration.
STDOUT versus STDERR
To give an example of when to use STDOUT versus STDERR, consider an application that generates a CSV file.
You want to make sure the CSV output is in the proper format so other applications can parse it
properly. Well, what if there are some warnings you want to inform the user about during the
CSV generation process. For example, the user is calling a deprecated method, or they
didn't provide enough values for a row, so you fill them in with empty values.
The warning messages should go to STDERR and the actual CSV contents should go
to STDOUT. This way the user can redirect STDOUT to a file like output.csv while
leaving STDERR to print out to the terminal for them to review, or piping
STDERR to debug.log.
Piping
Simple piping uses | character to send the STDOUT
from one application to the STDIN of the next application.
# Send STDOUT from `cat` to the STDIN of `grep`
cat file.txt | grep "some text"
# You can pipe together many commands
echo "hello world" | grep "world" | grep "wor" | grep "wo"
One nice thing about piping in Linux is that each application that is executed is run in parallel, so each application is processing its STDIN and sending its STDOUT as soon as it is received. It does not wait for the first application to completely finish before the next application runs. This makes it very efficient as all applications runs concurrently.
Another benefit is that you can avoid creating intermediary files.
For example, to create a .tar.gz file, you can do it all it one
step.
# Tarball up the `.py` and `.txt` files, gzip them
# and send the final gzipped contents to a `.tar.gz` file.
# The dash `-` means output to STDOUT instead of a file
tar cf - *.py *.txt | gzip > text_and_py_files.tar.gz
To pipe both STDOUT and STDERR you can use the |& operator.
echo "both stdout and stderr will get piped out" |& cat
# Equivalent to:
echo "both stdout and stderr going to same place" 2>&1 | cat
Redirect STDOUT
To redirect the standard output to a file, you can run a program like this with the > angle bracket
followed by the file name. This works in Windows, Mac, and Linux.
# Redirect STDOUT to a file
python hello_world.py > output.txt
The example above will overwrite and re-create the file each time it runs. If you use two angle brackets, it will append instead of overwriting the file.
# Append to output.txt
python hello_world.py >> output.txt
Redirect STDERR
Redirecting STDERR is similar to redirect STDOUT except
you must specify the file descriptor ID of 2 when
redirecting.
For example:
# Redirect STDERR to `debug.log`
python hello.py 2>debug.log
# Redirect STDOUT to `stdout.log`
# Redirect STDERR to `debug.log`
python hello.py 1>stdout.log 2>debug.log
You can also redirect one stream to another.
In this example, we first redirect STDOUT (1)
to a file, and then we redirect to STDERR (2)
to the new address of 1 (the file).
Now both STDOUT and STDERR are going to all_output.log
# Redirect STDOUT to `all_output.log`
# then redirect STDERR to STDOUT (which goes to the file now)
python hello.py 1>all_output.log 2>&1
Redirect STDIN
You can also redirect the input, so instead of coming from your keyboard it comes from a file. Anything you entered in the file will be treated as if you typed it in yourself manually.
For example, if you had a Python 3 script like this:
# pyin.py
name = input("What is your name?")
email = input("What is your email?")
print("Your name is %s and email is %s" % (name, email))
Then in your input.txt file you had:
nanodano
nanodano@devdungeon.com
You could run the application like this:
python3 pyin.py < input.txt
This is essentially the same thing as running:
cat input.txt | python3 pyin.py
You can even redirect both STDIN and STDOUT at the same time.
python3 pyin.py < input.txt > output.txt
Conclusion
After reading this, you should know the difference between STDOUT and STDERR, and how to redirect them individually to output files. You should also know how to redirect STDIN to come from a file to provide input to an application.