0

I have a python script called script.py. When I run this script, it creates a logs folder on the Desktop and downloads all the necessary logs from a website and writes them as .log files in this logs folder. I want Fluentd to run this script every 5 minutes and do nothing more. The next source I have on the config file does the real job of sending this log data to another place. If I already have the logs folder on the Desktop, this log files are uploaded correctly to the next destination. But the script never runs. If I delete my logs folder locally, this is the output fluentd gives:

2020-07-27 10:20:42 +0200 [trace]: #0 plugin/buffer.rb:350:enqueue_all: enqueueing all chunks in buffer instance=47448563172440
2020-07-27 10:21:09 +0200 [trace]: #0 plugin/buffer.rb:350:enqueue_all: enqueueing all chunks in buffer instance=47448563172440
2020-07-27 10:21:36 +0200 [debug]: #0 plugin_helper/child_process.rb:255:child_process_execute_once: Executing command title=:exec_input spawn=[{}, "python /home/zohair/Desktop/script.py"] mode=[:read] stderr=:discard

This never gives a logs folder on my Desktop which the script normally does output if run locally like python script.py

If I already have the logs folder, I can see the logs on the stdout normally. Here is my config file:

<source>
@type exec
command python /home/someuser/Desktop/script.py
run_interval 5m
<parse>
@type none
keys none
</parse>
<extract> 
tag_key none
</extract>
</source>
 
 
<source>
@type tail
read_from_head true
path /home/someuser/Desktop/logs/*
tag sensor_1.log-raw-data
refresh_interval 5m
<parse>
@type none
</parse>
</source>
 
<match sensor_1.log-raw-data>
@type stdout
</match>

I just need fluentd to run the script and do nothing else, and let the other source take this data and send it to somewhere else. Any solutions?

  • This log (`2020-07-27 10:21:36 +0200 [debug]: #0 plugin_helper/child_process.rb:255:child_process_execute_once: Executing command title=:exec_input spawn=[{}, "python /home/zohair/Desktop/script.py"] mode=[:read] stderr=:discard`) means that the script is executing. Please check the folder for file updates from command-line or Files Explorer to verify that the files are being updated or not. Observe the files' timestamps. – Azeem Jul 27 '20 at 09:25
  • The folder ```logs``` on the directory is never being created. If I run the script normally, it does create the folder. The script in Python creates a path in /home/someuser/Desktop/logs and then saves the files there. –  Jul 27 '20 at 09:31
  • Did you check by creating the folder manually? – Azeem Jul 27 '20 at 09:34
  • I created an empty folder there manually called ```logs```. Ran fluentd config file, no logs show up inside. If I run the python script, it does add the log files inside that folder. –  Jul 27 '20 at 09:36
  • Just tested this scenario with [this](https://godbolt.org/z/3oPxnK) configuration and it is working. Observe logs at the end of the snippet. – Azeem Jul 27 '20 at 09:45
  • Can you try it with a python script like this one? https://hastebin.com/opezovaxit.py –  Jul 27 '20 at 10:47
  • You need to add a check for directory existence. See this: https://stackoverflow.com/a/18973480/7670262 – Azeem Jul 27 '20 at 10:50
  • Yes, in my original script, there is an existence check, that should not be a problem. This simple script just creates a folder on desktop and a file in it. When running this simple script on fluentd, it doesnt create anything. –  Jul 27 '20 at 10:55
  • Changed the script. Working for me:https://hastebin.com/cigafisoxe.coffeescript – Azeem Jul 27 '20 at 12:07
  • I used the same exact code that you have provided above, weirdly my output is different (It didnt create any logs folder and no output to stdout). I have attached the output and the config file. Can you take a look into what may be going wrong? https://hastebin.com/gopikabiqi.bash –  Jul 27 '20 at 12:40
  • Try this configuration: https://hastebin.com/vitifujuje.bash – Azeem Jul 27 '20 at 12:43
  • With the same config file and the script, it still gives the output without creating any folder for logs. https://hastebin.com/igadudocow.bash –  Jul 27 '20 at 13:02
  • 1
    What is your `fluentd` version? You're running it as root. Are you sure that it is not a permissions issue? How about `chmod +x script.py` and then run it directly? – Azeem Jul 27 '20 at 13:13
  • Found out what the problem was. As fluentd was running as superuser, there was a missing import of requests from my original script. Now I will try to add another command on fluentd for executing pip install -r and try it from there. –  Jul 30 '20 at 08:36
  • 1
    Right. So, a permission/privilege issue. Though, I'm glad you figured it out. :) – Azeem Jul 30 '20 at 08:39
  • Yes, Thank you so much. It is weird that fluentd never gave an error even when using -vv about the module not found. –  Jul 30 '20 at 09:06
  • You're welcome! What is that module's name? BTW, you ought to [create an issue](https://github.com/fluent/fluentd/issues/new/choose) for this. Might be helpful for others as well. – Azeem Jul 30 '20 at 09:28
  • 1
    Hello, I opened an issue there. It was the 'requests' and 'beautifulsoup4' module. –  Jul 30 '20 at 10:51

1 Answers1

0

Problem was solved by creating another @type exec for pip install -r requirements.txt which fulfilled the missing module error which was not being shown on the fluentd error log (Was running fluentd as superuser).