AWS Announces General Availability of UltraWarm for Amazon Elasticsearch Service

Amazon Web Services (AWS), announced the general availability of UltraWarm for Amazon Elasticsearch Service, a new, highly performant, fully managed, low-cost warm storage tier that provides fast, interactive analytics of log data at one-tenth the cost of existing storage options.

Amazon Elasticsearch Service makes it simple to collect, analyze, and visualize machine-generated log data from websites, mobile devices, and sensors. UltraWarm for Amazon Elasticsearch Service gives Elasticsearch customers a warm storage tier that both stores large amounts of data cost-effectively and provides the type of snappy, interactive experience that Elasticsearch customers expect.

There are no up-front investments required to use UltraWarm, and customers pay a simple hourly rate for the storage provisioned.

As more and more applications are built using microservices, containers, and purpose-built data stores, the volume of machine-generated log data is growing exponentially.

Real-time analysis of this data has become essential to customers in order to quickly resolve operational and security issues. Amazon Elasticsearch Service is a popular service for log analytics because of its ability to ingest high volumes of log data and analyze it interactively.

With this explosive growth of log data, storing and analyzing months’ or even years’ worth of data is cost-prohibitive at scale. This has led customers to use multiple analytics tools, or delete valuable data, missing out on important insights that the longer-term data could yield.

To solve for this customer challenge, AWS built UltraWarm, which gives Elasticsearch customers a warm storage tier that both stores large amounts of data cost-effectively, and provides the type of snappy, interactive experience that Elasticsearch customers expect.

Now Amazon Elasticsearch Service supports two storage tiers, hot and UltraWarm. The hot tier is used for indexing, updating, and providing the fastest access to data.

UltraWarm provides a distributed cache for more frequently accessed data, while using advanced placement techniques to determine the blocks of data that are accessed less frequently, and can be moved outside of the cache to Amazon Simple Storage Service (Amazon S3).

UltraWarm stores data in Amazon S3, providing up to 50% faster query execution versus competing for warm-tier solutions, and 80% lower cost than the warm-tier storage from other managed Elasticsearch offerings.

With UltraWarm, customers can manage current and historical log data for interactive operational analysis and visualization in a single cluster. UltraWarm is a seamless extension of the Amazon Elasticsearch Service.

Customers can easily visualize search results across both their recent and longer-term operational data, all from their Kibana interface.

Additionally, UltraWarm supports all of the Elasticsearch Application Programming Interfaces (APIs), tools, and features, including enterprise-grade security with fine-grained access control, encryption at rest and in flight, integrated alerting, SQL querying, and more.

This allows developers, DevOps engineers, and InfoSec experts to use Amazon Elasticsearch Service for the analysis of recent (weeks) and longer-term (months or years) operational data without needing to spend days restoring data from archives (Amazon S3 or Amazon Glacier) to an active searchable state in an Elasticsearch cluster.

“Our customers tell us that log data offers a wealth of operational and security insights, but that the storage of log data quickly adds up, and proves cost-prohibitive over the medium and long term,” said Raju Gulabani, VP of Databases and Analytics, AWS.

“UltraWarm is the most cost-effective Elasticsearch-compatible storage solution available. It is also performance-optimized, so customers can investigate and interactively visualize their data while they embrace data at scale.”

UltraWarm can be enabled on existing or new domains using the AWS Management Console, CLI, or SDK. UltraWarm is available today on Amazon Elasticsearch version 6.8 and above in US East (N. Virginia, Ohio), US West (Oregon, N. California), AWS GovCloud (US-Gov-East, US-Gov-West), Canada (Central), South America (Sao Paulo), EU(Ireland, London, Frankfurt, Paris, Stockholm), Asia Pacific (Singapore, Sydney, Tokyo, Seoul, Mumbai, Hong Kong), China (Beijing, Ningxia), and Middle East (Bahrain), with additional regions coming soon.

Sophos is a worldwide leader in next-generation cybersecurity, protecting organizations of all sizes in more than 150 countries from today’s most advanced cyber threats.

“Sophos uses Amazon Elasticsearch Service to run a large-scale security monitoring and alerting system because it is highly performant and scalable,” said Prakash Talreja, Architect, Sophos.

“We are excited that UltraWarm will enable us to retain log data for much longer in a cost-effective way. We see great value in leveraging UltraWarm to bring down cost and reduce operational overhead.”

Asurion is a leading provider of device insurance, warranty, and support services for cell phones, consumer electronics, and home appliances.

“All our application and infrastructure logs are loaded into Amazon Elasticsearch Service for real-time data analysis, API performance metrics, and alerting based on log events," said Shyam Rayaprolu, Principal Architect, Retail and SBX Platforms, Asurion.

“We are always looking to bring costs down. Even though we automated our data backup process operationally, it has become overhead for our DevOps and Support teams.

We are really excited that UltraWarm removes the operational overhead, and reduces cost significantly. We were able to easily migrate our data from the hot nodes to UltraWarm nodes, and use our existing Kibana dashboards, without the need to make any changes.”

SparkPost is the world’s first and only predictive email intelligence platform, helping brands predict and optimize email performance with data analytics solutions.

“Amazon Elasticsearch Service powers one of our most used features – Event Search – which indexes billions of email messages and engagement events in real-time every day.

Our customers can quickly resolve support issues by easily looking up the delivery and open status for an email sent to a particular recipient,” says Chris McFadden, VP of Engineering, SparkPost.

“We love the simplicity and performance of Amazon Elasticsearch Service and are excited to use UltraWarm to further reduce our operational costs, while simultaneously offering our customers access to raw event data beyond our current ten-day retention period.

We believe offering up to 30 days of data will make our Event Search feature even more valuable as a go-to troubleshooting tool for our customers.”