0

Before:

main 2>&1 | tee -a $log_file;

This is working fine but it is throwing stderr in $log_file as shown below. I want to replace:

"ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)"

With:

"NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)"

I want the version and date to be in regex format.

After:

main 2>&1 | tee -a | sed -i 's|ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)|NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)|g' $log_file;

Error which are coming at downloading time

Few lines coming at the end(The lines are jumbled)

SRK
  • 21
  • 1
  • 9
  • 2
    So you want to disguise an error as a note. Big deal. _Why_? – arkascha Oct 14 '16 at 12:14
  • `tee -a` won't do much on its own. It should be something like `main | sed | tee`. – Biffen Oct 14 '16 at 12:14
  • main 2>&1 | sed -i 's|ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)|NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)|g' | tee -a $log_file . Is this correct? – SRK Oct 14 '16 at 12:17
  • 1
    @SRK How should we know?! Does it work? – Biffen Oct 14 '16 at 12:18
  • The command is not working for me. It is not replacing the ERROR line with NOTE line – SRK Oct 14 '16 at 12:19
  • Show some example output. – Biffen Oct 14 '16 at 12:20
  • It is throwing an error like this. ** sed: no input files ** – SRK Oct 14 '16 at 12:28
  • 1
    @SRK: Drop the `-i` from the sed options. That only works when using files, as it is for inplace editing – user000001 Oct 14 '16 at 12:29
  • After removing -i it is working but I need regex in place version number(5.3.0.0-213) and date formats(2015-02-02_12-17-08). They might change in future even then the code should work fine. – SRK Oct 14 '16 at 12:53

2 Answers2

1

You should precise your need, it's pretty hard to read your code by now.

There is two option here :

  • You get the mainstream and alternate him before saving into your log file
  • You format your log file at the end

First option

I can't test it, however, it should be like this : 

main 2>&1 | SED COMMAND | tee -a $log_file

Second option

Tested, it works.

sed -i "s/ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)/NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)/g" $log_file

Sed will edit the file specified inline because of -i option.

REGEX

If you want to change all ERROR by NOTE, then, you should just use this sed command

sed -i "s/ERROR/NOTE/g" $log_file;

And if you want to be more specific, take a look at this answer : using SED with wildcard

Community
  • 1
  • 1
Aks
  • 395
  • 3
  • 11
  • I don't want ERROR to be replaced everywhere. It should be replaced if it is stderr only and regex should only for (5.3.0.0-213) and (2015-02-02_12-17-08) because they may change in future. – SRK Oct 14 '16 at 12:46
  • Then the answer of Mustafa should do the tricks, you can find more information about it in the link I gave you. :) – Aks Oct 14 '16 at 12:50
  • Thanks. And for the reader, don't forget `sed -u` (--unbuffered load minimal amounts of data from the input files and flush the output buffers more often), if you think that sed doesn't work: the pipe stream is in the buffer. – phili_b Apr 16 '19 at 15:17
0

Could you try this;

main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)/ s/ERROR/INFO/" | tee -a $log_file

To find version, the regex; 

[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}  :

To find date, regex : for example (2015-02-02_12-17-08)

20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}

if date is like 2015-12-03 11.37.25, you should use 

20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}.[0-9]\{2\}.[0-9]\{2\}
Mustafa DOGRU
  • 3,994
  • 1
  • 16
  • 24
  • "main 2>&1 |sed 's|ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)|NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr)|g' | tee -a $log_file" was my previous command, so what are the changes to be done. – SRK Oct 14 '16 at 14:37
  • could you try this ; "main 2>&1 | sed "/ERROR.*5.3.0.0-213.*$(date +'%Y-%m-%d_%H-%M-%S').*/ s/ERROR/INFO/" | tee -a $log_file" – Mustafa DOGRU Oct 14 '16 at 14:45
  • if 2015-02-02_12-17-08 is not change; "main 2>&1 | sed "/ERROR.*5.3.0.0-213.*2015-02-02_12-17-08.*/ s/ERROR/INFO/" | tee -a $log_file" – Mustafa DOGRU Oct 14 '16 at 14:48
  • Timestamp and version will change if a new version of Pentaho comes. – SRK Oct 17 '16 at 05:01
  • Example : ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : – SRK Oct 17 '16 at 05:02
  • 1)ERROR (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : 2)ERROR(version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) . Only the 2nd one should be replaced. But by using the current regex, both of them are replaced. The regex should contain : (stderr). – SRK Oct 17 '16 at 07:21
  • main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*: (stderr).*/ s/ERROR/INFO/" | tee -a $log_file – SRK Oct 17 '16 at 07:25
  • Yeah.It is working perfectly. I need small concern. Is there any regex for this. After (stderr) also there are some digits and characters. Can we have a regex for them also? – SRK Oct 17 '16 at 08:28
  • @SRK : could you give a sample digits and characters after (stderr). – Mustafa DOGRU Oct 17 '16 at 08:33
  • INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) % Total % Received % Xferd Average Speed Time Time Time Current INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) Dload Upload Total Spent Left Speed INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 – SRK Oct 17 '16 at 08:34
  • INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 INFO (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) 8 176M 8 14.4M 0 0 6766k 0 0:00:26 0:00:02 0:00:24 6764k – SRK Oct 17 '16 at 08:34
  • @SRK : after stderr, there are many difference values, So I tried to find a correct regex, but I can't – Mustafa DOGRU Oct 17 '16 at 08:41
  • I have attached the exact screen shot of it. Can you please look into it? – SRK Oct 17 '16 at 08:45
  • While running the script getting an error like this. SCRIPT::main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)/ s/ERROR/NOTE/" | tee -a $log_file ERROR::2016/10/17 05:19:17 - file_reports_urls - NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) curl: (3) Illegal characters found in URL – SRK Oct 17 '16 at 12:23
  • SCRIPT::main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{‌​3\}.*20[0-9]\{2\}-[0‌​-9]\{2\}-[0-9]\{2\}_‌​[0-9]\{2\}-[0-9]\{2\‌​}-[0-9]\{2\}.*(stder‌​r)/ s/ERROR/NOTE/" | tee -a $log_file – SRK Oct 17 '16 at 12:23
  • ERROR::2016/10/17 05:19:17 - file_reports_urls - NOTE (version 5.3.0.0-213, build 1 from 2015-02-02_12-17-08 by buildguy) : (stderr) curl: (3) Illegal characters found in URL – SRK Oct 17 '16 at 12:23
  • @SRK; I think, this is not sed problem, probably, curl command in main throw this, due to illegal characters found in URL. – Mustafa DOGRU Oct 17 '16 at 12:29
  • Exactly,thanks for your information. But here few lines are skipping because of the sed command. Example: 1) INFO 2016-10-14 02:51:57 - YT_BULK_API ETL MASTER LOAD – SRK Oct 17 '16 at 12:41
  • INFO 2016-10-14 02:51:57 - Set working directory: /home/pentahodev/repository/yt_bulk_api – SRK Oct 17 '16 at 12:41
  • INFO 2016-10-14 02:51:57 - Cleaning log files older than 10 days – SRK Oct 17 '16 at 12:41
  • INFO 2016-10-14 02:51:57 - YT_BULK_API ETL Master Load will run from Run Seq - 1 – SRK Oct 17 '16 at 12:41
  • The sed command is skipping all the outcome of main 2>&1 except which are starting from ERROR. Can you please change the regex? – SRK Oct 17 '16 at 12:44
  • @SRK: date and time connected space or _ char so, you can use this; main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}[_, ][0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)/ s/ERROR/INFO/" | tee -a $log_file – Mustafa DOGRU Oct 17 '16 at 12:50
  • Can I ask you a question if you don't mind? main 2>&1 is also returning other lines which are starting with "INFO" and "Running" lines along with ERROR lines. Can we change Regex to allow INFO and Running lines? – SRK Oct 17 '16 at 12:58
  • maybe this help. But I cant test this; main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{‌​3\}.*20[0-9]\{2\}-[0‌​-9]\{2\}-[0-9]\{2\}[‌​_, ][0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)/ s/ERROR/INFO/;/\(RUNNING\|INFO\)/p" | tee -a $log_file – Mustafa DOGRU Oct 17 '16 at 13:18
  • By using the previous command also INFO and Running lines are coming but they are present at somewhere in the middle. Few of @ERROR lines coming at exact place and few are at the end. – SRK Oct 17 '16 at 13:31
  • I have attached the screen shot. After job got executed successfully it is showing downloading status. – SRK Oct 17 '16 at 13:34
  • main 2>&1 | sed "/\(Running\|INFO\|ERROR.*version.*build.*(stderr)\)/ s/ERROR/#INFO/g" | tee -a $log_file Will the above command work? – SRK Oct 18 '16 at 09:03