Getting Started / Hands-on / ...
Optimize Costs and Gain Visibility into Usage with Amazon S3 Storage Lens
TUTORIAL
Overview
Amazon S3 Storage Lens delivers organization-wide visibility into object storage usage, activity trends, and makes actionable recommendations to optimize costs and apply data protection best practices. S3 Storage Lens is the first cloud storage analytics solution to provide a single view of object storage usage and activity across hundreds, or even thousands, of accounts in an AWS Organization, with drill-downs to generate insights at multiple aggregation levels. Using the contextual recommendations that S3 Storage Lens derives from analysis of metrics across your organization, you can take immediate steps to optimize your storage.
Amazon S3 is used to store large shared datasets across tens to hundreds of accounts and buckets, multiple Regions, and thousands of prefixes. With S3 Storage Lens, you can easily understand your storage and analyze it to detect outliers. S3 Storage Lens delivers more than 60 metrics (free metrics and advanced metrics) on S3 storage usage and activity to an interactive dashboard in the S3 console. All customers have access to free metrics and can enable advanced metrics for a monthly per-object monitoring charge to receive prefix-level insights, extended data retention, recommendations, and the option to publish metrics to Amazon CloudWatch.
What you will accomplish
- Understand the difference between free and advanced metrics
- Create, configure, and navigate an S3 Storage Lens dashboard
- Explore use cases for S3 Storage Lens
Prerequisites
Access this support page for more information on how to create and activate a new AWS account.
For help creating your first S3 bucket, visit the Amazon S3 User Guide.
AWS experience
Time to complete
30 minutes
Cost to complete (with free metrics)
Free
Cost to complete (with advanced metrics)
Prices may vary: $0.20 per million objects monitored per month (Amazon S3 pricing page)
Requires
AWS account and a populated Amazon S3 bucket
Services used
Last updated
November 17, 2022
Implementation
Free metrics
All Amazon S3 customers can access an interactive dashboard in the S3 console that contains preconfigured views of storage usage and activity trends. With S3 Storage Lens free metrics, you receive 28 metrics across various categories at the bucket level, and 14 days of historical data in the dashboard. You can filter your dashboard by summary, cost optimization, data protection, access management, performance, or events to analyze specific metrics for your intended use cases. In addition to the dashboard in the S3 console, you can export metrics in CSV or Apache Parquet format to an S3 bucket of your choice for further use. For more information about what usage metrics are aggregated by S3 Storage Lens, see the Amazon S3 Storage Lens metrics glossary.
Advanced metrics and recommendations
S3 Storage Lens offers free metrics for all dashboards and configurations with the option to upgrade to advanced metrics and recommendations for an additional charge. For more information, see the Management & analytics tab on the Amazon S3 pricing page.
By upgrading to Storage Lens advanced metrics, you receive 35 additional metrics with 15 months of historical data. The advanced metrics provide insights related to activity (such as request counts), deeper cost optimization (such as S3 Lifecycle rule counts), additional data protection (such as S3 Replication rule counts), and detailed status codes (such as 403 authorization errors). You can also drill down to the prefix level to identify your largest and fastest growing prefixes within a bucket. Additionally, you can access S3 Storage Lens metrics through Amazon CloudWatch to create alarms, or the CloudWatch API to send metrics to integrated observability partners. You can use advanced insights to expand Lifecycle rules usage for greater cost optimization, identify buckets with insufficient data protection policies, and improve the performance of your application workloads.
S3 Storage Lens provides automated recommendations to help you optimize your storage. Recommendations are placed contextually alongside relevant metrics in the S3 Storage Lens dashboard. Historical data is not eligible for recommendations because recommendations are relevant to what is happening in the most recent period. Recommendations only appear when they are relevant.
S3 Storage Lens recommendations come in the following forms:
- Suggestions
Suggestions alert you to trends within your storage usage and activity that might indicate an opportunity to optimize your storage cost or apply data protection best practice. - Call-outs
Call-outs are recommendations that alert you to interesting anomalies within your storage usage and activity over a period that might need further attention or monitoring. - Reminders
Reminders provide insights into how Amazon S3 works. They can help you learn more about ways to use S3 features to reduce storage costs or apply data protection best practices.
S3 Storage Lens collects metrics daily, and data is available for queries for 15 months. For more information about the storage metrics aggregated by S3 Storage Lens, see the Amazon S3 Storage Lens metrics glossary.
Step 1: Create an S3 Storage Lens dashboard
1.1 — Sign in to the AWS Management Console using your account credentials. From the AWS console services search bar, enter S3. Under the services search results, select S3.
1.2 — Navigate to the Dashboards menu item under the Storage Lens section on the left panel. Next, choose Create dashboard.
1.3 — Under the General panel, enter a descriptive name for your dashboard and choose a Home Region. Next, choose the Enable option under Status for updated daily metrics.
1.4 — A dashboard can analyze storage across accounts, Regions, buckets, and prefix. Under Dashboard scope, choose whether you would like to include or exclude certain Regions, buckets, or both to change the scope of your dashboard.
If you select the Include Regions and buckets button, you will have the option to include all Regions and buckets, or select which Regions and buckets you would like to include from a dropdown. Otherwise, if you select the Exclude Regions and buckets button, you will have a dropdown option to choose which Regions and buckets to exclude.
1.5 — Under Metrics selection, choose the options you want to use. For this tutorial, we selected Advanced metrics and recommendations and included Advanced metrics, CloudWatch publishing, and Prefix aggregation. Further, you can choose the Advanced metrics categories that you want to include in the dashboard. For this tutorial, we selected Activity metrics, Detailed status code metrics, Advanced cost optimization metrics, and Advanced data protection metrics. You can find a detailed list of metrics in the metrics glossary in the Amazon S3 User Guide.
1.6 — Under Metrics export, you may choose Enable to have your dashboard metrics exported to a specified S3 bucket every 24 hours.
If you choose to enable this, you will then have to select your preferred output format and your destination bucket.
To learn more, see the documentation on S3 Storage Lens data export.
During this time, feel free to leave and come back to this tutorial once the metrics have been generated.
Step 2: Navigate to the S3 Storage Lens dashboard
2.1 — Navigate back to the dashboard by accessing the Amazon S3 console, and then go to the Dashboards menu item, as you did in Step 1.2. Once you open the dashboard, you may expand the Filters panel to temporarily filter the dashboard data by Accounts, Regions, Storage classes, Buckets, and Prefixes.
2.2 — The next section is a snapshot of a variety of metrics. You can see a trend line that shows the trend of each metric over the last 30 days, if using advanced metrics, and a percentage change (14 days if using free metrics or if you enabled advanced metrics less than 30 days ago). The number in the % change comparison column shows the Day/day percentage change by default. You may select to compare by Week/week or Month/month.
Moreover, you can select different metric groups to view on the snapshot overview from the dropdown. The selections include Summary, Cost optimization, Data protection, Activity, Access management, Events, and Performance.
2.3 — Under the Snapshot panel, you will see the Trends and distributions section. In this section, you can compare two metrics over a date range, which you can specify, to view trends over time.
2.4 — Right below, the dashboard also shows those two metrics and how they are distributed across Storage class and AWS Regions. You can click on any value in this graph and Drill down to filter the entire dashboard on that value, or select Analyze by to navigate to a new dashboard view for that dimension.
2.5 — The last section on the Overview tab allows you to perform a Top N analysis of a metric over a date range, where N is between 1 and 25. In the example below, we have selected the top three items in descending order for the Total storage metric.
You can then see the top three accounts, Regions, buckets, and prefixes given the chosen metric, along with the associated trends.
You can view the other tabs on the dashboard for more specific metrics on your Accounts, Regions, Storage classes, Buckets, and Prefixes.
Step 3: Explore use cases for S3 Storage Lens
Identify large buckets that you are not aware of (Free tier)
The first step to managing your storage costs is to gain a detailed understanding of your S3 usage by bucket. With S3 Storage Lens, you can access a centralized view of all buckets in your account. You can even configure a dashboard at the AWS Organization level to see all buckets in all your accounts. Using S3 Storage Lens makes it easy to get visibility into all your buckets, which can reveal unexpected findings such as buckets with more objects than you expected.
3.1 — At the top of the S3 Storage Lens dashboard, navigate to the Bucket tab.
3.3 — On the Trend of buckets graph, you can visualize not only which bucket has the highest total storage, but also which buckets have had the most growth. On this graph, it is clear that bucket1 has had consistent growth while also having the largest storage.
We can drill down on this bucket to gather more insights, such as the average object size, percentage of noncurrent version bytes, or the largest prefixes.
3.4 — Then, you can navigate to the bucket within the Amazon S3 console to understand the associated workload and identify internal owners of the bucket based on the account number. You can then find out from the bucket owners whether this growth is expected, or if it is unexpected growth that you can now place under proper monitoring and control.
Increase use of S3 storage classes
One of the clearest paths to storage cost savings is through optimizing your storage costs based on frequency of access and performance needs via Amazon S3 Storage Classes. Amazon S3 offers a range of storage classes that you can choose from based on the data access, resiliency, and cost requirements of your workloads. These storage classes include:
- S3 Standard for general-purpose storage of frequently accessed data
- S3 Intelligent-Tiering for data with unknown or changing access patterns
- S3 Standard-Infrequent Access (S3 Standard-IA) and S3 One Zone-Infrequent Access (S3 One Zone-IA) for long-lived, but less frequently accessed data
- Amazon S3 Glacier Flexible Retrieval, Amazon S3 Glacier Instant Retrieval, and Amazon S3 Glacier Deep Archive for long-term archive and digital preservation
If you see that all, or nearly all, of your storage bytes are in the S3 Standard storage class, it means that you may be able to optimize your usage by exploring additional S3 storage classes to best align to your use case. If you see a view like this, you can likely benefit from exploring cost optimization design patterns.
First, you can have cost optimization automated for you by using the S3 Intelligent-Tiering storage class, which is ideal for unknown or changing access patterns. Second, for known access patterns, you can configure Amazon S3 lifecycle policies to reduce your storage costs by transitioning your data to more cost-effective storage classes as the access frequency slows over time. View the Amazon S3 pricing page for more details on exact savings, and note additional costs for transitions and using S3 Glacier storage classes per object overhead.
You can then continue your analysis in S3 Storage Lens to explore storage class usage at greater depths, drilling down to see storage class distributions for specific Regions or buckets (or prefixes if you have upgraded to the advanced tier). It is common to have a subset of buckets that are not optimally configured, which is where you can benefit from using different or additional S3 storage classes. S3 Storage Lens is an effective tool to screen for these buckets before moving on to take further action.
Uncover buckets that have gone cold
If you have buckets that have gone cold, meaning that the storage in those buckets is no longer accessed (or rarely accessed), it is often an indicator that the related workload is no longer in use. If you have enabled S3 Storage Lens advanced metrics, you have access to activity metrics to understand how hot (or cold) your buckets are. There are metrics like GET requests and download bytes that indicate how often your buckets are accessed each day. You can trend this data over several months (extended data retention is available with the advanced tier) to understand the consistency of the access patterns and to spot buckets that are no longer being accessed. The % retrieval rate metric, computed as Download bytes / Total storage, is a useful metric to understand the proportion of storage in a bucket that is accessed daily. Keep in mind that the download bytes are duplicated in cases where the same object is downloaded multiple times during the day.
The best way to visualize buckets that have gone cold is through the Bubble analysis graph on the Bucket tab of the dashboard. The bubble analysis graph enables you to plot your buckets on multiple dimensions using any three metrics to represent the x-axis, y-axis, and size of the bubble.
3.6 — Navigate to the Bucket tab and down to the Bubble analysis graph. Select Total storage, % retrieval rate, and Avg. object size.
If you drill-down on any buckets with a retrieval rate of zero (or near zero) and a larger relative storage size, you can find buckets that have gone cold and where the storage cost is likely large enough to warrant taking action. For this example, we would look at bucket10 and bucket1.
From here, you can identify the bucket owners in your organization to confirm the purpose of the workload and find out if the storage is still needed. If it’s not needed, you can remediate costs by configuring lifecycle expiration policies, or by archiving the data in the Amazon S3 Glacier storage classes. And to avoid the problem of cold buckets in the future, you can apply one of the recommended design patterns previously mentioned in this guide, to automatically transition your data using S3 lifecycle policies or enable auto-archiving with S3 Intelligent-Tiering.
Step 4: Clean up
For this tutorial, we recommend that if you find value in the insights you are retrieving, keep advanced metrics and recommendations on. Note that you will keep incurring costs if you choose this option.
Alternatively, you may choose to stop incurring costs by turning off advanced metrics and recommendations. You will still have access to Free metrics on your dashboard.
4.2 — Choose the Edit button on the top right of the dashboard configuration page.
4.3 — Scroll down to the metrics selection panel, and choose Free metrics. Next, select Save changes.
Conclusion
Congratulations! You have learned how to create, configure, and navigate an Amazon S3 Storage Lens dashboard, and explored three use cases to optimize your object storage costs.
Next steps
To learn more about Amazon S3 Storage Lens, visit the Amazon S3 User Guide and product page.