Adding a test use case for creating #33860
Conversation
| SPARK_USER: tester | ||
| HADOOP_USER_NAME: tester | ||
| SPARK_HOME: /usr/lib/spark | ||
| PATH: $PATH:${SPARK_HOME}/bin |
There was a problem hiding this comment.
We can't extend the environment here. This is no-op. Move to an export
| - bash | ||
| - wolfi-base | ||
| - libstdc++-11 | ||
| - libavcodec61 | ||
| environment: | ||
| LANG: en_US.UTF-8 | ||
| JAVA_HOME: /usr/lib/jvm/${{range.value}} |
There was a problem hiding this comment.
This is also no-op, has to be done in the pipeline
| - bash | ||
| - wolfi-base | ||
| - libstdc++-11 | ||
| - libavcodec61 |
| @@ -21,7 +21,7 @@ environment: | |||
| - maven | |||
| - openjdk-11 | |||
| - openjdk-17 | |||
| # Only 8 is used during the build process | |||
| - openjdk-17-default-jvm | |||
There was a problem hiding this comment.
Since we explicitly set JAVA_HOME, we don't need this. Also error prone as we build with two different Java versions and don't want to use the wrong one
| @@ -120,7 +131,7 @@ subpackages: | |||
| val sum = rdd.reduce(_ + _) | |||
| assert(sum == 15) | |||
| EOF | |||
| cat SimpleJob.scala | /usr/lib/spark/bin/spark-shell | |||
| cat SimpleJob.scala | /usr/lib/spark/bin/spark-shell --conf spark.jars.ivy=/tmp/.ivy | |||
There was a problem hiding this comment.
I'm not sure I understand why we need pass --conf
| dependencies: | ||
| runtime: | ||
| - openjdk-17-default-jvm |
There was a problem hiding this comment.
We build with 8 and 17. We should drop this
| mkdir -p ${{targets.contextdir}}/usr/lib/spark/work-dir | ||
| mv bin/ ${{targets.contextdir}}/usr/lib/spark | ||
| mv sbin/ ${{targets.contextdir}}/usr/lib/spark | ||
| mv target ${{targets.contextdir}}/usr/lib/spark | ||
| mv assembly ${{targets.contextdir}}/usr/lib/spark |
There was a problem hiding this comment.
Why does this deviate from the build process for 2.12 above?
| mv assembly ${{targets.contextdir}}/usr/lib/spark | ||
| cp resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh ${{targets.contextdir}}/usr/lib/spark/ | ||
| - name: ${{package.name}}-scala-2.13-compat |
There was a problem hiding this comment.
Why do we need another compat package?
There was a problem hiding this comment.
@EyeCantCU The entrypoint.sh is sourced from a different location for 2.12 which I suspect is as a result of the build process.
There was a problem hiding this comment.
If we loop over a range of supported configurations (scala 2.12-2.13, jdk8-17), we can accommodate that in the same subpackage definition without creating a new one with much of the same logic
| SPARK_LOCAL_IP: 127.0.0.1 | ||
| SPARK_LOCAL_HOSTNAME: localhost | ||
| SPARK_HOME: /usr/lib/spark | ||
| PATH: $PATH:${SPARK_HOME}/bin |
There was a problem hiding this comment.
This doesn't work, use an export
Fixes:
Related:
Pre-review Checklist
For new package PRs only
endoflife.date)For new version streams
name: ${{package.name}}-compat)provides:logical unversioned forms of the package (e.g.nodejs,nodejs-lts)For package updates (renames) in the base images
When updating packages part of base images (i.e. cgr.dev/chainguard/wolfi-base or ghcr.io/wolfi-dev/sdk)
apk upgrade --latestsuccessfully upgrades packages or performs no actionsFor security-related PRs
For version bump PRs
epochfield is reset to 0For PRs that add patches