Log Jenkins Console Output to S3

Question

So i have to Log my Jenkins Job Console Output to S3 Bucket and this involves many jobs to be configured in a way such that the console output log of the Jenkins job is pushed to S3 post build.

Adiii · Answer 1 · 2020-04-22T10:49:54.443

1

One way is to simply upload logs file to s3 using AWS CLI. You can try below approach.

echo "hello from job"
echo "job logs to s3"

cat ${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log >> ${BUILD_NUMBER}.log

/usr/bin/aws s3 cp  ${BUILD_NUMBER}.log s3://s3_bucket_name/${BUILD_NUMBER}.log

Or you can set cron job to sync local logs folder with s3.

for example, you can set cron job that will run every minutes and sync local log file to s3 or copy and have versioning on s3 side, or the other option is to trigger cronjob script after completion of each job.

for logs_file in $(ls /var/lib/jenkins/jobs/*/builds/lastSuccessfulBuild/log); 
#now logs_file contain all job logs file path


do  
   echo "copy logs file to s3 $logs_file
   # to identify file unique
   s3_file_name=$(date +'%Y%-m%d%s')
   aws s3 cp  $logs_file s3://s3_bucket_name/"${s3_file_name}".log
done

#Note there are different directories you can copy accordingly
#lastFailedBuild
#lastSuccessfulBuild
# lastUnstableBuild
#lastUnsuccessfulBuild
# and logs by build numbers

edited Apr 22 '20 at 10:49

answered Apr 21 '20 at 17:36

Adiii

54,482
7
145
148

Hey @Adiii Thanks for the input but what if i need to apply the same on atleast 100+ Jobs – Ramakrishnan_S Apr 22 '20 at 09:41
you mean an old job or you want to configure for other similar jobs too? – Adiii Apr 22 '20 at 10:23
Other similar jobs and old jobs too – Ramakrishnan_S Apr 22 '20 at 10:50
you can check the update script and change it accordingly, for the old job you need `$(ls /var/lib/jenkins/jobs/*/builds/*/log);` – Adiii Apr 22 '20 at 11:27
Hey @Adii how can i add a timestamp value to my log file in groovy when i actually create the plan file before pushing it to the s3. – Ramakrishnan_S May 11 '20 at 16:01
`Add timestamps to the Console Output` you can find this option under *Build Environment* – Adiii May 11 '20 at 17:24

score 0 · Answer 2 · answered Apr 21 '20 at 18:29

See tee in Utility steps:

script {
    tee("$WORKSPACE/my_output.txt") {
       sh "echo Hello"
    }
}

You may then upload the results to s3:

def path = env.JOB_NAME.replace('%2F','-').replace('/', "-").replace('_', "-").toLowerCase()

withAWS(endpointUrl:"...", credentials:"...") {
  s3Upload bucket: "jenkins",
    path:"${path}/${env.BUILD_NUMBER}/",
    file: "my_output.txt",
    workingDir: "${env.WORKSPACE}"
}

Log Jenkins Console Output to S3

2 Answers2