AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running.
With Lambda, you can run code for virtually any type of application or backend service – all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.
In my last post, i wrote about Uses and event triggers
We are working on a Cloud data protection and DR related produce, usually, the developer in my team does unit testing with create_snapshot API and they forgot to delete the snapshot or they test the scheduler with 5-minute interval and leaves it. End of the day we will be having tons of useless snapshots from different Amazon EBS volumes in different regions which add more to the monthly bill.
I created a simple AWS Lambda function which deletes the EBS snapshots older than one day
Create a Lambda Service
- Log in to the AWS console
- Click on Lambda
- Skip the blueprint and create a new function
- Set some name, description
- Runtime to Python 3.6
- Paste the code, save and Test.
In delete_snapshots days value is hardcoded to 1 that can be replaced using events. Events are the key-value pair that can be accessed in function using event[‘key’].
In the above code I set the filter description “Created by Nimesa” , you can set filter by tags or any other parameter
Setup Execution role
- You can choose the existing role or create new
- When accessing EBS snapshots Lambda should have EC2 access to list and delete snapshots
Timeout Settings
In basic settings setup the timeout value and memory, these setting allows you to control the code execution performance and costs for your Lambda function. Changing your resource settings (by selecting memory) or changing the timeout may impact your function cost.
Add Triggers
- Create a cloud watch event
- Rule: Fixed rate of 1 Days
- The function will be triggered by the schedule of 1 day
- You can add API Gateway also to call like web API
Cool 😎
Where is the code?
You could able to see the code embedded from GIST.
Raw code link https://gist.githubusercontent.com/asvignesh/77c709cb86759d9c297d7a997b12d02a/raw/e53df15acda0f79af13251cdcedae25233a0518c/delete_Snapshots.py
Hi Vignesh
It’s very useful post, i have one requirement, I’m making daily backup snapshots for production servers, i want to delete more than 15 days old snapshots from particular EBS Volumes, how to mention that particular volume id’s in the above code?
Thanks in advance!
You can use the filter by “volume-id” , try this if it didn’t work reply here i’ll post some sample
Great write up! I have a few questions…As you can see in my code, I look for the description “Created by Lambda backup function ebs-snapshots”. But when I test this, I see no snapshots have been removed.
from datetime import datetime, timedelta, timezone
import boto3
class Ec2Instances(object):
def __init__(self, region):
print(“region “+ region)
self.ec2 = boto3.client(‘ec2’, region_name=region)
def delete_snapshots(self, older_days=0):
delete_snapshots_num = 0
snapshots = self.get_lambda_created_snapshots()
for snapshot in snapshots[‘Snapshots’]:
fmt_start_time = snapshot[‘StartTime’]
if (fmt_start_time < self.get_delete_data(older_days)):
self.delete_snapshot(snapshot['SnapshotId'])
delete_snapshots_num+1
return delete_snapshots_num
def get_lambda_created_snapshots(self):
snapshots = self.ec2.describe_snapshots(Filters=[{'Name': 'description', 'Values': ['Created by Lambda backup function ebs-snapshots']}])
return snapshots
def get_delete_data(self, older_days):
delete_time = datetime.now(tz=timezone.utc) – timedelta(days=older_days)
return delete_time;
def delete_snapshot(self, snapshot_id):
self.ec2.delete_snapshot(SnapshotId=snapshot_id)
def lambda_handler(event, context):
print("event " + str(event))
print("context " + str(context))
ec2_reg = boto3.client('ec2')
regions = ec2_reg.describe_regions()
for region in regions['Regions']:
region_name = region['RegionName']
instances = Ec2Instances(region_name)
deleted_counts = instances.delete_snapshots(1)
print("deleted_counts for region "+ str(region_name) +" is " + str(deleted_counts))
return 'completed'
___
START RequestId: 2c29ab3d-8113-11e8-bf58-f5dde8cf91a3 Version: $LATEST
event {'key3': 'value3', 'key2': 'value2', 'key1': 'value1'}
context
region ap-south-1
deleted_counts for region ap-south-1 is 0
region eu-west-3
deleted_counts for region eu-west-3 is 0
region eu-west-2
deleted_counts for region eu-west-2 is 0
region eu-west-1
deleted_counts for region eu-west-1 is 0
region ap-northeast-2
deleted_counts for region ap-northeast-2 is 0
region ap-northeast-1
deleted_counts for region ap-northeast-1 is 0
region sa-east-1
deleted_counts for region sa-east-1 is 0
region ca-central-1
deleted_counts for region ca-central-1 is 0
region ap-southeast-1
deleted_counts for region ap-southeast-1 is 0
region ap-southeast-2
deleted_counts for region ap-southeast-2 is 0
region eu-central-1
deleted_counts for region eu-central-1 is 0
region us-east-1
deleted_counts for region us-east-1 is 0
region us-east-2
deleted_counts for region us-east-2 is 0
region us-west-1
deleted_counts for region us-west-1 is 0
region us-west-2
deleted_counts for region us-west-2 is 0
END RequestId: 2c29ab3d-8113-11e8-bf58-f5dde8cf91a3
REPORT RequestId: 2c29ab3d-8113-11e8-bf58-f5dde8cf91a3 Duration: 33069.49 ms Billed Duration: 33100 ms Memory Size: 128 MB Max Memory Used: 39 MB
Do you have snapshots with the provided description ( Created by Lambda backup function ebs-snapshots )
If you have snapshots older than 1 day and with provided description still not deleting it, please paste the code in https://pastebin.com/ and share the link.
Yes. I have one snapshot with that description created on 7/6. Here is the code for the lambda function I am using: https://pastebin.com/gtQt6Cmd
Here is the test event: https://pastebin.com/5wG6Z1Q1
I can see it going through each region, but not deleting the snap thats in the us-east-1 region.
I didn’t find anything wrong with the code, I tested your code in my IDE by creating the snapshot with the description “Created by Lambda backup function ebs-snapshots” and it works fine.
here is the screenshot of the debug console
http://prntscr.com/k44vl9
Hi Tyler,
Did your issue got fixed, because I am also getting the same issue.
Hello-
I am filtering by description: “Created by Lambda backup function ebs-snapshots”, however, when I test the function, I can see nothing gets removed.
Here is my code and results of the test:
START RequestId: cb7982ed-8113-11e8-9ad5-25207a29664d Version: $LATEST
event {‘key3’: ‘value3’, ‘key2’: ‘value2’, ‘key1’: ‘value1’}
context
region ap-south-1
deleted_counts for region ap-south-1 is 0
region eu-west-3
deleted_counts for region eu-west-3 is 0
region eu-west-2
deleted_counts for region eu-west-2 is 0
region eu-west-1
deleted_counts for region eu-west-1 is 0
region ap-northeast-2
deleted_counts for region ap-northeast-2 is 0
region ap-northeast-1
deleted_counts for region ap-northeast-1 is 0
region sa-east-1
deleted_counts for region sa-east-1 is 0
region ca-central-1
deleted_counts for region ca-central-1 is 0
region ap-southeast-1
deleted_counts for region ap-southeast-1 is 0
region ap-southeast-2
deleted_counts for region ap-southeast-2 is 0
region eu-central-1
deleted_counts for region eu-central-1 is 0
region us-east-1
deleted_counts for region us-east-1 is 0
region us-east-2
deleted_counts for region us-east-2 is 0
region us-west-1
deleted_counts for region us-west-1 is 0
region us-west-2
deleted_counts for region us-west-2 is 0
END RequestId: cb7982ed-8113-11e8-9ad5-25207a29664d
REPORT RequestId: cb7982ed-8113-11e8-9ad5-25207a29664d Duration: 32700.97 ms Billed Duration: 32800 ms Memory Size: 128 MB Max Memory Used: 39 MB
Hi asvignesh ,
I added volime-id to delete snaps by volume id but it didnot delete as per
def delete_snapshots(self, older_days=10):
snapshots = self.ec2.describe_snapshots(Filters=[{‘Name’: ‘volume-id’, ‘Values’: [‘vol-06a4cddddb626fbd6df5’]}])
It deleted all except one snapshot.
Nikhil
Hi Nikhil,
In filters you should use volume-id key instead of Name and list of volume ids or single volume id in Values .
Thanks Vignesh for getting back quickly on this. Sorry i am new to python .
Do you mean this way ,
snapshots = self.ec2.describe_snapshots(Filters=[{‘volume-id key’: ‘volume-id’, ‘Values’: [‘vol-06a4cddddbfbd6df5’]}])
And for retention to keep for 10days
def delete_snapshots(self, older_days=10)
is that write ?
Added – snapshots = self.ec2.describe_snapshots(Filters=[{‘volume-id key’: ‘volume-id’, ‘Values’: [‘vol-06a4cdb626fbd6df5’]}])
but run came with error
{
“errorMessage”: “Parameter validation failed:\nUnknown parameter in Filters[0]: \”volume-id key\”, must be one of: Name, Values”,
“errorType”: “ParamValidationError”,
“stackTrace”: [
[
“/var/task/lambda_function.py”,
44,
“lambda_handler”,
“deleted_counts = instances.delete_snapshots(1)”
],
[
“/var/task/lambda_function.py”,
13,
“delete_snapshots”,
“snapshots = self.get_nimesa_created_snapshots()”
],
[
“/var/task/lambda_function.py”,
25,
“get_nimesa_created_snapshots”,
“snapshots = self.ec2.describe_snapshots(Filters=[{‘volume-id key’: ‘volume-id’, ‘Values’: [‘vol-06a4cdb626fbd6df5’]}])”
],
[
“/var/runtime/botocore/client.py”,
314,
“_api_call”,
“return self._make_api_call(operation_name, kwargs)”
],
[
“/var/runtime/botocore/client.py”,
586,
“_make_api_call”,
“api_params, operation_model, context=request_context)”
],
[
“/var/runtime/botocore/client.py”,
621,
“_convert_to_request_dict”,
“api_params, operation_model)”
],
[
“/var/runtime/botocore/validate.py”,
291,
“serialize_to_request”,
“raise ParamValidationError(report=report.generate_report())”
]
]
}
use snapshots = self.ec2.describe_snapshots(
Filters=[{‘Name’: ‘volume-id’, ‘Values’: [‘vol-242342342343’]}])
Replace vol-242342342343 with your volume id
Hello Vignesh!
Thanks for writing this awesome blog, I have followed the steps given but somehow it’s not deleting the snapshot volume (though I am even used the same tags)
I’m not getting any error but as you can see below no snapshot volumes are deleted , i do have more than 1 older volumes.
“`
START RequestId: fdnh343-d2dc-11e8-82d8-6fddcfffa078 Version: $LATEST
event {‘key1’: ‘value1’, ‘key2’: ‘value2’, ‘key3’: ‘value3’}
context
region ap-south-1
deleted_counts for region ap-south-1 is 0
region eu-west-3
deleted_counts for region eu-west-3 is 0
region eu-west-2
deleted_counts for region eu-west-2 is 0
region eu-west-1
deleted_counts for region eu-west-1 is 0
region ap-northeast-2
deleted_counts for region ap-northeast-2 is 0
region ap-northeast-1
deleted_counts for region ap-northeast-1 is 0
region sa-east-1
deleted_counts for region sa-east-1 is 0
region ca-central-1
deleted_counts for region ca-central-1 is 0
region ap-southeast-1
deleted_counts for region ap-southeast-1 is 0
region ap-southeast-2
deleted_counts for region ap-southeast-2 is 0
region eu-central-1
deleted_counts for region eu-central-1 is 0
region us-east-1
deleted_counts for region us-east-1 is 0
region us-east-2
deleted_counts for region us-east-2 is 0
region us-west-1
deleted_counts for region us-west-1 is 0
region us-west-2
deleted_counts for region us-west-2 is 0
END RequestId: fdnh343-d2dc-11e8-82d8-6fddcfffa078
REPORT RequestId: fdnh343-d2dc-11e8-82d8-6fddcfffa078 Duration: 30503.52 ms Billed Duration: 30600 ms Memory Size: 128 MB Max Memory Used: 51 MB
“`
Hi,
I would need to create ebs snapshot every minutes and it should be deleted every hour. Can you guide me how to modify the script?
Thank you.
Hi Minnu,
use ec2.create_snapshot(VolumeId=, Description=) to create a snapshot for the volume, have the proper naming convention ( or store in DB, I feel it’s not much needed) and create another Cloudwatch Schedule expression to trigger
Have some logic in Tag to identify and delete the oldest snapshot and run it on another schedule expression.
Hello Vignesh,
I have a requirement to delete the snapshots which are older than 7 days from the current date. What piece of code should I modify from the above provided code.
In L: 40 instances.delete_snapshots(1) 1 represents the snapshots older than 1 day
change the 1 to 7
In L: 40 instances.delete_snapshots(1) 1 represents the snapshots older than 1 day
change the 1 to 7
Thanks.
Do i also need to change the same value in def delete_snapshots(self, older_days=1): ?
The above code is working fine, only issue is while triggering lambda status is showing as failed with below error:
Task timed out after 3.00 seconds.
When i am increasing the timeout to 4 or 5 sec, it is still giving the error as : Task timed out after 4.00 or 5.00 seconds.
Any idea how can i fix this?
I changed the timeout to 1 minute and it’s not showing the timeout error now. Can you just reply for the hard-code region issue.
How can i hard-code the region value to only one region?
Currently it is trying to get the snapshot details from all the regions which is not my requirement.
for region in regions[‘Regions’]:
region_name = region[‘RegionName’]
instances = Ec2Instances(region_name)
Remove the for loop, and hardcode the region like instances = Ec2Instances(‘us-east-1’)
Will this code not work for encrypted snapshot.
No seperate API from encrypted snapshot, this code will work for encrypted snapshots.
HI Vignesh,
I want delete all the snapshots in the account older than 30 days, and there is no Name or Description been added to the snapshots being created right now.
What changes should I make to the above Python script? Sorry I do not know Python. 🙂