How can I fix a Broken Pipe error?

Question

I recently reinstalled RVM (following the instructions at http://rvm.io) after a fresh install of Ubuntu 12.10 when I got an SSD Drive.

Now, when I type: type rvm | head -1

I receive the following error:

rvm is a function
-bash: type: write error: Broken pipe

But if I immediately repeat the command then I only receive:

rvm is a function

And it appears everything is ok? What's happening? What can I do to fix it? It doesn't happen always. It appears to be more sporadic. I've tried to find some kind of pattern to it but haven't yet.

Answer 1

Seeing "Broken pipe" in this situation is rare, but normal.

When you run type rvm | head -1, bash executes type rvm in one process, head -1 in another.¹ The stdout of type is connected to the "write" end of a pipe, the stdin of head to the "read" end. Both processes run at the same time.

The head -1 process reads data from stdin (usually in chunks of 8 kB), prints out a single line (according to the -1 option), and exits, causing the "read" end of the pipe to be closed. Since the rvm function is quite long (around 11 kB after being parsed and reconstructed by bash), this means that head exits while type still has a few kB of data to write out.

At this point, since type is trying to write to a pipe whose other end has been closed – a broken pipe – the write() function it caled will return an EPIPE error, translated as "Broken pipe". In addition to this error, the kernel also sends the SIGPIPE signal to type, which by default kills the process immediately.

(The signal is very useful in interactive shells, since most users do not want the first process to keep running and trying to write to nowhere. Meanwhile, non-interactive services ignore SIGPIPE – it would not be good for a long-running daemon to die on such a simple error – so they find the error code very useful.)

However, signal delivery is not 100% immediate, and there may be cases where write() returns EPIPE and the process continues to run for a short while before receiving the signal. In this case, type gets enough time to notice the failed write, translate the error code and even print an error message to stderr before being killed by SIGPIPE. (The error message says "-bash: type:" since type is a built-in command of bash itself.)

This seems to be more common on multi-CPU systems, since the type process and the kernel's signal delivery code can run on different cores, literally at the same time.

It would be possible to remove this message by patching the type builtin (in bash's source code) to immediately exit when it receives an EPIPE from the write() function.

However, it's nothing to be concerned about, and it is not related to your rvm installation in any way.

Answer 2

You can fix a broken pipe at the expense of another process by inserting tail -n +1 in your pipe, like this:

type rvm | tail -n +1 | head -1

The +1 tells tail to print the first line of input and everything that follows. Output will be exactly the same as if tail -n +1 wasn't there, but the program is smart enough to check standard output and closes the pipe down cleanly. No more broken pipes.

Answer 3

Let's try with yes, an endless process printing yes...

Before, the yes process was killed by SIGPIPE when reached limit.

➜ set -o pipefail
➜ yes | head -n 1
y
➜ echo $?        
141

My solution

➜ yes | (head -n 1;dd status=none of=/dev/null)
y


# the process will still running and output to null

You can replace yes with your program.

Answer 4

The write error: Broken pipe message refers to a writing process that tries to write to a pipe with no readers left on the reading end of that pipe and the special circumstance that the SIGPIPE signal is set to be ignored either by the current or the parent process. If it was the parent process that set SIGPIPE to be ignored, it is not possible for the child process to undo that again in a non-interacitive shell.

However, it is possible to kill type rvm when head -1 terminates by using explicit subshells. This way we can background type rvm, send typepid to the head -1 subshell and then implement a trap on EXIT there to kill type rvm explicitly.

trap "" PIPE        # parent process sets SIGPIPE to be ignored
bash                # start child process
export LANG=C
# create a fake rvm function
eval "
rvm() {
$(printf 'echo line of rvm code %s\n' {1..10000})
}
"

# rvm is a function
# bash: type: write error: Broken pipe
type rvm | head -1

# kill type rvm when head -1 terminates
# sleep 0: do nothing but with external command
( (sleep 0; type rvm) & echo ${!} ; wait ${!} ) | 
    (trap 'trap - EXIT; kill "$typepid"; exit' EXIT; typepid="$(head -1)"; head -1)

Answer 5

We can use awk to replace head, and not eagerly quits prematurely. (this should work on all variance of awk implementations):

type rvm | awk 'NR==1'

Or more generically, for the first N lines:

# First 10 lines:
type rvm | awk -v N=10 'NR<=N'

To explain the awk script: NR==1 is the condition, when met, the action should be taken. NR is the awk variable for the current Number of Records. The action usually is given after the condition enclosed in {...}. When the action is omitted, it is default to print the entire line {print}. So awk 'NR==1' means to print those lines whose "Number of Records" is 1, which is the first line.

Similarly, in awk -v N=10 'NR<=N', -v N=10 means passing N=10 as a variable to the following awk script. And 'NR<=N' means to print those lines whose "Number of Records" is less/eq N.

Why this doesn't trigger broken pipe error? It's because in above awk script, it processes every line until the end of file, so it won't close the reading end of the pipe prematurely.

Some performance considerations:

1. we don't cache the lines and print them out in an END section, so memory-wise it's OK to handle large number of lines
2. since we don't reference fields here (e.g., $1, $2, ...), field splitting should not happen, which further saves computation splitting potentially long lines and large amount of lines.
   • (Per POSIX doc[1]: "Before the first reference to a field in the record is evaluated, the record shall be split into fields, according to the rules in Regular Expressions, using the value of FS that was current at the time the record was read")

And finally, curiously, if we do want to emulate head's behavior of early exiting (and therefore the broken pipe sympton), we would add exit statement:

yes | awk -v N=10 'NR<=N; NR>N {exit}'  # print 10 "yes"

[1]: POSIX awk manual