delete all log streams of a log group using aws cli

Question

In order to delete a log stream from a log group using the CLI command , individual log stream names are required . Is there a way to delete all log streams belonging to a log group using a single command?

Stephen · Answer 1 · 2018-01-29T14:52:44.163

You can achieve this through using --query to target the results of describe-log-streams. This allows you to loop through and delete the results.

aws logs describe-log-streams --log-group-name $LOG_GROUP_NAME --query 'logStreams[*].logStreamName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-stream --log-group-name $LOG_GROUP_NAME --log-stream-name $x; done

You can use --query to target all or specific groups or streams.

Delete streams from a specific month

aws logs describe-log-streams --log-group-name $LOG_GROUP --query 'logStreams[?starts_with(logStreamName,`2017/07`)].logStreamName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-stream --log-group-name $LOG_GROUP --log-stream-name $x; done

Delete All log groups - Warning, it deletes EVERYTHING!

aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-group --log-group-name $x; done

Clearing specific log groups

aws logs describe-log-groups --query 'logGroups[?starts_with(logGroupName,`$LOG_GROUP_NAME`)].logGroupName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-group --log-group-name $x; done

Credit

This works but deletes all of the log _groups_. The OP is asking how to delete the log _streams_ within a group. — brianz, Jan 19 '18 at 19:15
@brianz good point, I have updated the answer to correctly answer the question asked — Stephen, Jan 29 '18 at 14:54
Instead of outputting the list of streams in a visual table format and then using awk and grep to extract the contents of the table, you can just output in text format like this `aws logs describe-log-streams --log-group-name $LOG_GROUP_NAME --query 'logStreams[*].[logStreamName]' --output text | while read x; do aws logs delete-log-stream --log-group-name $LOG_GROUP_NAME --log-stream-name $x; done` — gene_wood, Jul 20 '20 at 20:32

score 13 · Answer 2 · answered Mar 15 '18 at 11:55

Implemented script with command from @Stephen's answer. The script shows summary before deletion and tracks progress of deletion.

#!/usr/bin/env bash

LOG_GROUP_NAME=${1:?log group name is not set}

echo Getting stream names...
LOG_STREAMS=$(
    aws logs describe-log-streams \
        --log-group-name ${LOG_GROUP_NAME} \
        --query 'logStreams[*].logStreamName' \
        --output table |
        awk '{print $2}' |
        grep -v ^$ |
        grep -v DescribeLogStreams
)

echo These streams will be deleted:
printf "${LOG_STREAMS}\n"
echo Total $(wc -l <<<"${LOG_STREAMS}") streams
echo

while true; do
    read -p "Prceed? " yn
    case $yn in
    [Yy]*) break ;;
    [Nn]*) exit ;;
    *) echo "Please answer yes or no." ;;
    esac
done

for name in ${LOG_STREAMS}; do
    printf "Delete stream ${name}... "
    aws logs delete-log-stream --log-group-name ${LOG_GROUP_NAME} --log-stream-name ${name} && echo OK || echo Fail
done

Github link

@Kilhoffer I'm a bit late for the party, but wouldn't it be easier to set a shorter retention for your loggroups so old logentries will be deleted automatically? — derpirscher, Apr 18 '22 at 08:36

score 11 · Answer 3 · edited Nov 03 '17 at 13:24

11

To delete all log streams associated with a specific log group, run the following command, replacing NAME_OF_LOG_GROUP with your group:

aws logs describe-log-streams --log-group-name NAME_OF_LOG_GROUP --output text | awk '{print $7}' | while read x; 
do aws logs delete-log-stream --log-group-name NAME_OF_LOG_GROUP --log-stream-name $x
done

edited Nov 03 '17 at 13:24

Stephen Rauch

47,830
31
106
135

answered Nov 03 '17 at 12:54

Caleb Clark

111
1
2

score 10 · Answer 4 · edited Jan 11 '19 at 07:23

10

Here is Script to delete all logs in a log group using python. Just change the logGroupName to match your logGroup.

import boto3

client = boto3.client('logs')

response = client.describe_log_streams(
    logGroupName='/aws/batch/job'
)


def delete_stream(stream):
    delete_response = client.delete_log_stream(
        logGroupName='/aws/batch/job',
        logStreamName=stream['logStreamName']
    )

    print(delete_response)


results = map(lambda x: delete_stream(x), response['logStreams'])

edited Jan 11 '19 at 07:23

German Lashevich

2,193
27
38

answered May 21 '17 at 12:54

Will Humphreys

2,677
3
25
26

1

there is however a very small issue with the above script. It would still not delete `all` the streams in the specified log group. That's because `describe_log_streams` has a limitation of pulling up the 1st 50 records *at max* alone. So we could add another loop over this and get all the streams deleted IMHO. – qre0ct Oct 16 '18 at 05:20
1

Or still better, *pagination* could be used like this: ```paginator = client.get_paginator('describe_log_streams')``` ```pg_response = paginator.paginate(logGroupName='/aws/batch/job', PaginationConfig={'MaxItems': 100000})``` ```response = pg_response.build_full_result()``` ```results = map(lambda x: delete_stream(x), response['logStreams'])``` – qre0ct Dec 23 '19 at 16:20

score 4 · Answer 5 · answered Nov 19 '19 at 21:34

Based on @german-lashevich's answer

If you have thousands of log streams, you will needed to parallelize.

#!/usr/bin/env bash

LOG_GROUP_NAME=${1:?log group name is not set}

echo Getting stream names...
LOG_STREAMS=$(
    aws logs describe-log-streams \
        --log-group-name ${LOG_GROUP_NAME} \
        --query 'logStreams[*].logStreamName' \
        --output table |
        awk '{print $2}' |
        grep -v ^$ |
        grep -v DescribeLogStreams
)

echo These streams will be deleted:
printf "${LOG_STREAMS}\n"
echo Total $(wc -l <<<"${LOG_STREAMS}") streams
echo

while true; do
    read -p "Prceed? " yn
    case $yn in
    [Yy]*) break ;;
    [Nn]*) exit ;;
    *) echo "Please answer yes or no." ;;
    esac
done

step() {
    local name=$1
    printf "Delete stream ${name}... "
    aws logs delete-log-stream --log-group-name ${LOG_GROUP_NAME} --log-stream-name ${name} && echo OK || echo Fail
}

N=20
for name in ${LOG_STREAMS}; do ((i=i%N)); ((i++==0)) && wait ; step "$name" & done

score 3 · Answer 6 · answered Dec 27 '19 at 08:17

For Windows users this powershell script could be usefull, to remove all the log streams in a log group:

#Set your log group name
$log_group_name = "/production/log-group-name"

aws logs describe-log-streams --log-group-name $log_group_name --query logStreams --output json | ConvertFrom-json | ForEach-Object {$_.logStreamName} | ForEach-Object {
    aws logs delete-log-stream --log-group-name $log_group_name --log-stream-name $_
    Write-Host ($_ + " -> deleted") -ForegroundColor Green
}

Just save it as your_script_name.ps1 and execute it in powershell.

score 2 · Answer 7 · answered Mar 07 '17 at 08:09

2

This cannot be done using a single aws Cli command. Hence we achieved this using a script where we first retrieved all the log streams of a log group and then deleted them in a loop.

answered Mar 07 '17 at 08:09

akhila

720
2
7
17

1

I will appreciate if you could provide the script. – KurioZ7 Aug 29 '19 at 08:19

score 2 · Answer 8 · answered Dec 20 '18 at 16:07

An alternative version using Powershell CLI on Windows, launch powershell command line and use:

$LOG_GROUP_NAME="cloud-watch-group-name";
$LOG_STREAM_NAMEP="cloud-watch-log-stream-name";

Set-DefaultAWSRegion -Region us-your-regions;
Set-AWSCredential -AccessKey ACCESSKEYEXAMPLE -SecretKey sEcReTKey/EXamPLE/xxxddddEXAMPLEKEY -StoreAs MyProfileName

Get-CWLLogStream -loggroupname $LOG_GROUP_NAME -logstreamnameprefix $LOG_GROUP_NAMEP | Remove-CWLLogStream -LogGroupName $LOG_GROUP_NAME;

You may use -Force parameter on the Remove-CWLogStream Cmdlet in case you don´t want to confirm one by one.

References https://docs.aws.amazon.com/powershell/latest/reference/Index.html

score 1 · Answer 9 · answered Sep 02 '20 at 19:45

The others have already described how you can paginate through all the log streams and delete them one by one.

I would like to offer two alternative ways that have (more or less) the same effect, but don't require you to loop through all the log streams.

Deleting the log group, then re-creating it has the desired effect: All the log streams of the log group will be deleted.

delete-log-group

followed by:

create-log-group

CAVEAT: Deleting a log group can have unintended consequences. For example, subscriptions and the retention policy will be deleted as well, and those have to be restored too when the log group is re-created.

Another workaround is to set a 1 day retention period.

put-retention-policy

It won't have an immediate effect, you will have to wait ca. a day, but after that all the old data will be deleted. The name of the old streams and their meta data (last event time, creation time, etc.) will remain though, but you won't be charged for that (as far as I can tell based on my own bill).

So it is not exactly what you asked for. However, probably the most important reason why one would want to delete all the log streams is to delete the logged data (to reduce costs, or for compliance reasons), and this approach achieves that.

WARNING: Don't forget to change the retention policy after the old data is gone, or you will continually delete data after 1 day, and chances are, it is not what you want in the long run.

grepit · Answer 10 · 2021-01-14T17:40:21.927

If you are doing this in zshell /zsh and you only need simple one liner command then just update the values :

* Pattern
AWS_SECRET_ACCESS_KEY
AWS_ACCESS_KEY_ID
AWS_DEFAULT_REGION

Pattern can any text , you can also add ^ for begging of the line or $ for end of the line.

run the below command !

Pattern="YOUR_PATTERN" && setupKeys="AWS_ACCESS_KEY_ID=YOUR_KEY AWS_SECRET_ACCESS_KEY=YOUR_KEY AWS_DEFAULT_REGION=YOUR_REGION" && 
eval "${setupKeys} aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table | sed 's/|//g'| sed 's/\s//g'| grep -i ${Pattern} "| while read x; do  echo "deleting $x" && $setupKeys aws logs delete-log-group --log-group-name $x; done

score 0 · Answer 11 · answered Feb 13 '17 at 18:30

0

--log-group-name is not optional in aws cli, you can try using an * for --log-group-name value (in test environment)

aws logs delete-log-group --log-group-name my-logs

Reference URL: http://docs.aws.amazon.com/cli/latest/reference/logs/delete-log-group.html

answered Feb 13 '17 at 18:30

Aty

269
3
10

1

I would like to delete all log streams under a log-group using the CLI. Not the log group – akhila Feb 14 '17 at 07:22
This is an answer to a different question. – kmarsh Apr 30 '21 at 18:43

score 0 · Answer 12 · answered May 04 '20 at 04:23

If you are using a prefix, you could use the following command.

aws logs describe-log-streams --log-group-name <log_group_name> --log-stream-name-prefix"<give_a_log_group_prefix>" --query 'logStreams[*].logStreamName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-stream --log-group-name <log_group_name> --log- stream-name $x;done;

score 0 · Answer 13 · answered May 12 '23 at 00:36

A quick dirty py script with pagination to delete all steams in a group.

Set log_group_name from terminal input or un comment to set in code.

import boto3

# take input
print('log group name:')
log_group_name = input()

# or set in code
# log_group_name = '' # set your log name

client = boto3.client('logs')


def run(next_token: str = None):
    print(f'batch w/token: {next_token}')
    params = {
        'logGroupName': log_group_name
    }
    if next_token:
        params['nextToken'] = next_token
    response_streams = client.describe_log_streams(**params)
    streams = response_streams['logStreams']
    for stream in streams:
        log_stream_name = stream['logStreamName']
        response_delete = client.delete_log_stream(
            logGroupName=log_group_name,
            logStreamName=log_stream_name
        )
        success = response_delete.get('ResponseMetadata', {}).get('HTTPStatusCode') == 200
        print(f'delete {log_stream_name} - {"success" if success else "failed..."}')
    return response_streams.get('nextToken')


# process
next_token = run()
while next_token:
    next_token = run(next_token)

delete all log streams of a log group using aws cli

13 Answers13