What are some good Perl modules for flow-based programming on files?

Question

What are some good Perl modules to process files based on configurations?

Basically I am working on taking data files, split them into columns, remove some rows based on some columns, remove unnecessary columns, compare them to baseline (writes where changes have occured) and save a csv of the data and the comments as metadata.

Sample file is:

001SMSL22009032020090321024936
002XXXXX20090320102436               010000337 00051     
002XXXXX20090320103525               010000333 00090     
002XXXXX20090320103525               010000333 00090     
002XXXXX20090320103525               010000333 00090     
002XXXXX20090320103525               010000333 00090     
002XXXXX20090320103525               020000333 00090     
009000000009000000000271422122

it will compare row by row with another file (baseline) and some differing rows will be highlighted ( I am use Tk::DiffText).

Here is the pipeline where [is a pipe]

file -> [split] -> [remove production] -> [sort] -> [compare] -> {user jumps in and writes comments, edits file as needed} -> [save csv] -> [save comments]

The real question is what perl module helps to model and make a pipeline flow like this? After more research I have found this http://en.wikipedia.org/wiki/Flow-based_programming.

You should adjust your question to show a full example of the data file, the input you expect, and the output that you desire. Don't add to the question through comments: make the question better :) — brian d foy, Jul 31 '09 at 17:51
He comes onto Stack Overflow is spurts. So I wouldn't expect him around anytime soon. — Brad Gilbert, Jul 31 '09 at 23:17

score 1 · Answer 1 · edited Jul 31 '09 at 17:53

1

Hmmm, seems that it's nothing Perl cannot handle almost by itself :

taking data files

while (<>)

split them into columns,

my @row = split(/,/);

remove some rows based on some columns,

next if @row[5] =~ m/black_list_data/;

remove unnecessary columns

@row = ($row[1], $row[4]);

remove unnecessary columns

@row = ($row[1], $row[4]);

compare them to baseline (writes where changes have occured)

Ok, here you might use Algorithm::Diff

and save a csv of the data and the comments as metadata.

Class::CSV or DBD::CSV ?

edited Jul 31 '09 at 17:53

Brad Gilbert

33,846
11
78
129

answered Jul 31 '09 at 15:44

Steve Schnepp

4,620
5
39
54

Hmm.. the quote CSS style is quite similar to the code style :-( – Steve Schnepp Jul 31 '09 at 15:45
I normally leave the quotes as normal text or make them comments in the code. – Chas. Owens Jul 31 '09 at 17:36

score 1 · Answer 2 · edited Apr 03 '19 at 10:59

1

Take a look at Sprog. It is a visual programming engine written in Perl (using Gtk2). You can create Perl programs by dragging and dropping "gears". You can also add your own gears (written in Perl, of course).

edited Apr 03 '19 at 10:59

Glorfindel

21,988
13
81
109

answered Aug 01 '09 at 01:46

Chas. Owens

64,182
22
135
226

score 0 · Answer 3 · answered Aug 03 '09 at 15:24

I am not aware of any Perl implementations of Flow-Based Programming, but I believe Perl 5.8 has made interpreter threads available to Perl coders (someone correct me if I'm wrong!), so it should be relatively straightforward to build an FBP implementation on Perl. See http://perldoc.perl.org/threads.html

score 0 · Accepted Answer · answered Aug 04 '09 at 14:46

0

This is what I was looking for:

Text::Pipe

Text::Pipe::Stackable

Thank you for helping me clarify my ideas!

answered Aug 04 '09 at 14:46

kthakore

1,566
3
17
32

What are some good Perl modules for flow-based programming on files?

4 Answers4