AWS is making a significant push towards Apache Iceberg, adopting the open table format across its analytics, machine learning, and storage services. This move is driven by customer demand and the format's suitability for large-scale data analysis. AWS is actively involved in shaping Iceberg's future and has introduced S3 Tables, a new type of storage bucket built on Iceberg, to enhance performance and ease of use.
AWS has placed its bet on the Apache Iceberg open table format ( OTF ) across its analytics, machine learning , and storage stack. This move is a direct response to the growing demand from customers already utilizing its popular S3 object storage. While there is a growing consensus around Iceberg, the future of its rival, Delta Lake , created by Databricks and now open source under the Linux Foundation, remains uncertain.
Delta Lake is currently the preferred format for software giants like Microsoft and SAP. However, for AWS, the world's largest cloud platform provider, the choice is clear – Iceberg, at least until its S3 customers indicate otherwise. This commitment to Iceberg extends beyond mere adoption. AWS is actively involved in shaping the format's future. They have core committers contributing to the Iceberg open source stack, directly influencing APIs and collaborating with other developers. This deep involvement stems from observing the trend among their largest analytics customers on S3, who were already gravitating towards Iceberg. AWS remains open to exploring support for other formats if customer demand shifts, but for now, Iceberg's design and growing popularity make it the most attractive option for building structured support on storage.A key element of AWS's Iceberg strategy is the introduction of S3 Tables, a new type of storage bucket described by Warfield as a 'managed Iceberg table.' These tables provide an Iceberg catalog where users can create namespaces and tables, each treated as a first-class resource. Users can even define access control and security policies at the table level. AWS claims S3 Tables will deliver a 10x performance boost for access due to their pre-partitioning nature. Additionally, AWS automatically handles all maintenance and optimization tasks in the background. This move is a significant step in AWS's data storage evolution, addressing the challenges faced by companies like Netflix, which migrated from on-premises data warehouses to AWS S3 object storage. Netflix encountered performance issues and unexpected behavior when trying to query their data via Hive Tables. This led to the development of Iceberg, designed for large-scale analytical workloads and compatible with various query engines like Spark, Trino, Flink, Presto, Hive, and Impala. Iceberg aimed to empower organizations to analyze their data without the need for costly and cumbersome data store migrations
Data Management Big Data AWS Apache Iceberg OTF S3 Tables Data Warehousing Machine Learning Analytics Delta Lake
United Kingdom Latest News, United Kingdom Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Chris Pratt Joins DEF Reset to Shed 'Holiday Fluff'Chris Pratt embraces a fitness challenge after holiday break to regain his physique.
Read more »
Nelly Furtado 'Keeps it Real' in 2025, Debunks Beauty MythsNelly Furtado embraces body positivity and authenticity in a revealing Instagram post.
Read more »
AWS Tackles AI 'Hallucinations' with Automated ReasoningAmazon Web Services (AWS) is introducing Amazon Bedrock Automated Reasoning checks to address the issue of AI 'hallucinations,' where AI models generate plausible but inaccurate responses. This new technology aims to prevent factual errors by verifying the accuracy of statements made by AI models.
Read more »
Ransomware crew abuses AWS native encryption, sets data-destruct timer for 7 days'Codefinger' crims on the hunt for compromised keys
Read more »
China's Salt Typhoon spies spotted on US govt networks before telcos, CISA boss saysWe are only seeing 'the tip of the iceberg,' Easterly warns
Read more »
AWS Embraces Apache Iceberg as Default Open Table FormatAWS has chosen Apache Iceberg as its preferred open table format (OTF) across its analytics, machine learning, and storage services. This decision is driven by customer demand, particularly those utilizing AWS S3 object storage. While Iceberg enjoys growing support, the future of rival OTF Delta Lake remains uncertain. AWS plans to actively shape Iceberg's development through its core committers and is confident in this direction based on the needs of its largest analytics customers.
Read more »