Last Week in a Byte on Delta Lake | 2023-04-18
Delta Lake
Delta Lake is an open-source storage framework that enables building a Lakehouse architecture
You can watch or read the latest #DeltaLake news a week late (2023-04-18 edition)!
delta-rs 0.9.0 released!
We're happy to announce the release of delta-rs 0.9.0 with various enhancements, including #HDFS support, upgrade #datafusion, and write support for additional #Arrow data types. There is more to come, as we’re already working on the 0.10 release! cc #apachearrow #deltalake
https://github.com/delta-io/delta-rs/releases/tag/rust-v0.9.0¸
Summary slides for delta-spark 2.3
Do you want to know more about Delta Lake? Follow Will Girten and get the latest Delta Lake information, including these handy slides showcasing what is new in the delta-spark 2.3 release! cc #apachespark
Introducing Kotosiro
We love seeing the growth of the #deltasharing community and the #rustlang community - with Kotosiro, we have both! Kotosiro is a minimalistic Rust implementation of a Delta Sharing server that currently supports both #AWS and #GCP environments.
It’s really cool because there are callouts to the original Rust Delta Sharing implementation known as Riverbank - by Delta maintainer R. Tyler Croy - as well as callouts to #ROAPI which allows you to create full-fledged APIs for slowly moving datasets without writing a single line of code by Delta maintainer Qingping (QP) Hou. Thanks very much, Shingo OKAWA, for your awesome contribution!
beam-datalake for Apache Beam and Delta Lake integration
We’re happy to announce another #ApacheBeam -based project called beam-datalake which has an IO connector named DataLakeIO, which connect Beam and data lakes such as Apache Hudi, Apache Iceberg, and of course, Delta Lake!
Last week's publications
There were a lot of great publications last week, including, but not limited to:
GeekCoders: How I use MACK Library in Delta Lake using Databricks/PySpark
Lakehouse by the sea: Migrating Seafowl storage layer to delta-rs by Marko Grujic
Matthew Powers, CFA published two great Delta Lake blogs - How to use Delta Lake generated columns and How to create and append to Delta Lake tables with pandas #pandas #generatedcolumns
Khuyen Tran posted How Delta Lake simplifies pandas DataFrame versioning and allows access to prior versions for auditing and debugging using delta-rs cc #pandas #DataFrame
Long time Apache Spark contributor and Databricks Beacon Bartosz Konieczny recently published Table file-formats - Z-Order compaction: Delta Lake.
Connect with us!
Want to learn more about Delta Lake and chat with other users and contributors? Join us at delta.io, Slack, LinkedIn, and GitHub.
Sr. Developer Advocate
1yGreat recap! Really appreciate Will Girten's delta-spark 2.3 slides, they're a perfect cheat sheet to all the new features and how to use them! Also cool to see all the work around Delta Sharing, including new Rust implementations, we love to see it 😀
Staff Developer Advocate
1yI love these recaps!