0

I have an interesting scenario ...

EDIT: I am considering the scenario where there is a large amount of test flowing into stdin (like loop.sh) which is read at a slower rate in some other program (loop2.sh). I make the following assumption: only the latest data is important all previous data can be discarded. I have been considering real time text filtering on stdin, are there any solutions? This is was my experiment to test it the stdin buffer:

This is loop.sh

#! /bin/bash
i=0
for (( ; ; ))
do
    echo "$i" >&2
    echo "$i"
    ((i++))
done

And this is loop2.sh

#! /bin/bash
i=0
for (( ; ; ))
do
    echo "Count: $i"
    ((i++))
    sleep 1
done

i run the following: ./loop.sh | ./loop2.sh

and i get:

...
12762
12763
12764
12765
12766
12767
12768
12769
Count: 2
Count: 3
Count: 4
Count: 5
Count: 6
Count: 7
Count: 8
Count: 9
...

Essentially the stdin buffer from loop.sh has filled up :D

And that halted execution too!!!

My question is can i change this behaviour? ie) automatically overwrite stdin or change the size? additionally why 12769? how big is the default?

da_steve101
  • 283
  • 4
  • 16
  • 1
    Your ̀`loop2.sh` should read its *stdin* (from the pipe). Otherwise you are reaching the pipe buffer limit `PIPE_MAX`; read [pipe(7)](http://man7.org/linux/man-pages/man7/pipe.7.html) and [Advanced Linux Programming](http://advancedlinuxprogramming.com/) – Basile Starynkevitch Jan 17 '14 at 05:51
  • 2
    I know it should, this is just an experiment. I intentionally reached the stdin buffer limit to see what happened. How do i change PIPE_MAX? – da_steve101 Jan 17 '14 at 06:03
  • 1
    You should not do that. I think it is a kernel parameter. And it is not "the stdin buffer limit" but a pipe size. – Basile Starynkevitch Jan 17 '14 at 06:05
  • 2
    is there anyway of turning stdin to a circular buffer? – da_steve101 Jan 17 '14 at 06:12
  • 1
    This is a different question, and its answer is *no* – Basile Starynkevitch Jan 17 '14 at 06:12
  • hmmm ... and no other way to do something like that without reading in everything and implementing yourself? :/ thats disappointing ... but thanks :D – da_steve101 Jan 17 '14 at 06:17
  • You really should read [Advanced Linux Programming](http://advancedlinuxprogramming.com/). Looks like you are misunderstanding some basic concepts. If you are coding in C, consider polling with [poll(2)](http://man7.org/linux/man-pages/man2/poll.2.html) – Basile Starynkevitch Jan 17 '14 at 06:20
  • i am considering the problem when there is perpetually a large amount of data available at a rate higher than it can be read in and parsed. Poll is if its ready. That is not really the issue. it is more real time text filtering ... – da_steve101 Jan 17 '14 at 06:24
  • I'm not sure why you can't hack up a "circular buffer" with a shell script. Say you just want a program to read in stdin line by line, and then spit it back out (at a slower speed, possibly dropping various lines as you see fit, thereby munging the data feed) – Steven Lu Sep 01 '14 at 22:03

0 Answers0