Skip to content

Commit

Permalink
Re-enabled the iceberg module using iceberg-1.3 on spark 3.4
Browse files Browse the repository at this point in the history
## Description
Re-enabled the iceberg module using iceberg-1.3 on spark 3.4

Existing unit tests.

## Does this PR introduce _any_ user-facing changes?
No

Closes #1820

Signed-off-by: Scott Sandre <scott.sandre@databricks.com>
GitOrigin-RevId: c48bfb7c3f4bc3d28b99f31d1d3d6d60afd2fbad
  • Loading branch information
tdas authored and vkorukanti committed Jun 20, 2023
1 parent cebedd9 commit d9a5f9f
Showing 1 changed file with 3 additions and 6 deletions.
9 changes: 3 additions & 6 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -242,9 +242,7 @@ lazy val storageS3DynamoDB = (project in file("storage-s3-dynamodb"))
)
)

// Requires iceberg release on 3.4
/**
lazy val deltaIceberg = (project in file("delta-iceberg"))
lazy val iceberg = (project in file("iceberg"))
.dependsOn(spark % "compile->compile;test->test;provided->provided")
.settings (
name := "delta-iceberg",
Expand All @@ -253,15 +251,14 @@ lazy val deltaIceberg = (project in file("delta-iceberg"))
releaseSettings,
libraryDependencies ++= Seq( {
val (expMaj, expMin, _) = getMajorMinorPatch(sparkVersion)
("org.apache.iceberg" % s"iceberg-spark-runtime-$expMaj.$expMin" % "1.1.0" % "provided")
("org.apache.iceberg" % s"iceberg-spark-runtime-$expMaj.$expMin" % "1.3.0" % "provided")
.cross(CrossVersion.binary)
},
// Fix Iceberg's legacy java.lang.NoClassDefFoundError: scala/jdk/CollectionConverters$ error
// due to legacy scala.
"org.scala-lang.modules" %% "scala-collection-compat" % "2.1.1"
)
)
*/

/**
* Get list of python files and return the mapping between source files and target paths
Expand Down Expand Up @@ -291,7 +288,7 @@ val createTargetClassesDir = taskKey[Unit]("create target classes dir")

// Don't use these groups for any other projects
lazy val sparkGroup = project
.aggregate(spark, contribs, storage, storageS3DynamoDB)
.aggregate(spark, contribs, storage, storageS3DynamoDB, iceberg)
.settings(
// crossScalaVersions must be set to Nil on the aggregating project
crossScalaVersions := Nil,
Expand Down

0 comments on commit d9a5f9f

Please sign in to comment.