Snap improvements #4298

brlin-tw · 2024-08-09T11:23:37Z

This pull request introduces various improvements to the snap recipe, please review.

This patch mitigates risk of supply chain attack via tainted source release packages(e.g. CVE-2024-3094). Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

This patch addresses the following Snapcraft linter warning: ``` - library: libopenjp2.so.7: unused library 'usr/lib/x86_64-linux-gnu/libo penjp2.so.2.4.0'. (https://snapcraft.io/docs/linters-library) ``` Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

…a part Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

* Reduce part pull time by limiting the history depth to clone * Reduce part build time by disable components that we don't need. * Reduce overall snap size by only shipping files that are needed in runtime. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

This is the current latest version. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

The application is still runnable due to some of the packages are already staged by the Leptonica part, however it should still be declared for clarity. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

This change reduces package size by 44%(73637888 -> 40894464). Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Currently this feature isn't exposed to the host, dropping the dependencies to reduce built package size. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Previously this is implicitly configured as the build dependency isn't available and the training tools build target isn't called. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

The zlib library is Leptonica's dependency, not Tesseract's. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

These should be Leptonica part's build dependencies. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

… tesseract and leptonica part Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

stweil · 2024-08-09T12:45:17Z

Maybe libcurl and libarchive could be added, too. Both add more features to tesseract.

brlin-tw · 2024-08-09T13:02:38Z

@stweil

Maybe libcurl and libarchive could be added, too. Both add more features to tesseract.

I have plans to add them as well, will do.

…ble snap publishing This scriptlet allows stable builds to be always available for the downstream snap publisher to promote to the `stable` channel on the Snap Store. Higher risk builds(e.g. beta/edge) will only occur when the lower risk builds(rc/stable) are promoted. Refer the following Snapcraft Forum topic for more info: Selective-checkout: Check out the tagged release revision if it isn't promoted to the stable channel - doc - snapcraft.io https://forum.snapcraft.io/t/selective-checkout-check-out-the-tagged-release-revision-if-it-isnt-promoted-to-the-stable-channel/10617 Here's a snippet of the scriptlet in action: ``` :: selective-checkout: INFO: Detecting stable releases... :: selective-checkout: determine_stable_release_details: DEBUG: Last stable release tag determines to be "5.4.1". :: selective-checkout: determine_stable_release_details: DEBUG: Last stable version determines to be "5.4.1". :: selective-checkout: snap_query_version: DEBUG: Checking what snap revisions are available for the tesseract snap at the stable release channel... :: selective-checkout: determine_stable_release_details: DEBUG: Last stable version on the snap store determines to be "". :: selective-checkout: INFO: Detecting release candidate releases... :: selective-checkout: DEBUG: Last release candidate release tag determines to be "5.4.0-rc2". :: selective-checkout: DEBUG: Last release candidate version determines to be "5.4.0-rc2". :: selective-checkout: snap_query_version: DEBUG: Checking what snap revisions are available for the tesseract snap at the candidate release channel... :: selective-checkout: DEBUG: Last release candidate version on the snap store determines to be "". :: selective-checkout: INFO: Detecting beta releases... :: selective-checkout: DEBUG: Last beta release tag determines to be "5.0.0-beta-20210916". :: selective-checkout: DEBUG: Last beta version determines to be "5.0.0-beta-20210916". :: selective-checkout: snap_query_version: DEBUG: Checking what snap revisions are available for the tesseract snap at the beta release channel... :: selective-checkout: DEBUG: Last beta version on the snap store determines to be "". :: selective-checkout: determine_which_version_to_build: DEBUG: Stable version(5.4.1) is newer than the one on the Snap Store(none), stable release will be built. :: selective-checkout: Info: The last tagged stable release(5.4.1) hasn't been promoted to the stable channel() on the Snap Store yet, checking out 5.4.1. ``` Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw · 2024-08-09T16:59:25Z

@stweil

Maybe libcurl and libarchive could be added, too. Both add more features to tesseract.

I have made the requested change, please review.

brlin-tw · 2024-08-09T17:04:37Z

I noticed that the snap would fail to launch due to link failure:

/snap/tesseract/x1/usr/local/bin/tesseract: error while loading shared libraries: libtesseract.so.5: cannot open shared object file: No such file or directory

Will push a fix soon.

Libraries under $SNAP/usr/local/lib isn't searched by the dynamic linker by default. Refer-to: Staging - Build and staging dependencies - doc - snapcraft.io <https://forum.snapcraft.io/t/build-and-staging-dependencies/11451> Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

This patch addresses the following linter warnings: ``` - library: libicuio.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicuio.so.70.1'. (https://snapcraft.io/docs/linters-library) - library: libicutest.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicutest.so.70.1'. (https://snapcraft.io/docs/linters-library) - library: libicutu.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicutu.so.70.1'. (https://snapcraft.io/docs/linters-library - library: libicui18n.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicui18n.so.70.1'. (https://snapcraft.io/docs/linters-library) ``` Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

…act part libcurl functionality requires network access. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

egorpugin · 2024-08-09T19:03:49Z

Do we really need this selective-checkout file?

brlin-tw · 2024-08-09T19:04:33Z

I noticed that the snap would fail to launch due to link failure:

/snap/tesseract/x1/usr/local/bin/tesseract: error while loading shared libraries: libtesseract.so.5: cannot open shared object file: No such file or directory

This problem is fixed in the current branch tip.

I also fixed broken libcurl functionality due to missing network snapd security confinement interface connection, please review.

brlin-tw · 2024-08-09T20:15:15Z

@egorpugin

Do we really need this selective-checkout file?

This is required for the downstream snap publisher(currently me, but it can be anyone that has store collaborator access) to have stable(r) snap builds available to promote to the stable(r) channels on the Snap Store. Currently, the snap build is done by periodically(every 6 hours IIRC) importing this repository to the mirror repo on Launchpad, which has a snap package recipe configured to be built automatically whenever the content of the mirror changes. As the build trigger isn't based on release tags we cannot ensure that beta/rc/stable builds will be available to be promoted to the beta/candidate/stable channels on the store backstage unless the Snapcraft part pull logic is customized.

Here is an example of what the snap promote operation looks like:

It is fine to drop this file if the snap maintainer/collaborator has the Git repository's write access and is willing to manually set the release tag to build(e.g. via the source-tag property of the tesseract part) in the snapcraft.yaml as part of the release process.

brlin-tw · 2024-08-10T05:01:52Z

I found a problem that the Tesseract config files shipped within the snap are not loaded due to the lack of data directory fallback logic(the TESSDATA_PREFIX environment variable is set to $SNAP_USER_COMMON to allow the user to install trained data files, however the config files are installed in the readonly $SNAP/usr/local/share/tessdata directory).

tesseract ../eurotext.png - -l eng hocr
read_params_file: Can't open hocr

openat(AT_FDCWD, "/home/brlin/snap/tesseract/common/configs/hocr", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/brlin/snap/tesseract/common/tessconfigs/hocr", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "hocr", O_RDONLY)      = -1 ENOENT (No such file or directory)
write(2, "read_params_file: Can't open hoc"..., 34read_params_file: Can't open hocr
) = 34

Related code: CCUtil::main_setup function of tesseract/src/ccutil/ccutil.cpp at main · tesseract-ocr/tesseract.

I'll make a patch to handle the fallback in the meantime, but it may be desired to implement the fallback logic in the main code(User specified path -> TESSDATA_PREFIX -> hardcoded datadir during the build configuration process).

This patch fixes configuration files in the readonly snap filesystem not loadable by Tesseract due to missing datadir fallback logic. If the config file with the same name doesn't appear in the $SNAP_USER_COMMON directory, the one in the $SNAP/usr/local/share/tessdata directory will be loaded, allow Tesseract to work properly. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw · 2024-08-10T17:41:42Z

I'll make a patch to handle the fallback in the meantime

A patch(bdccad) is implemented to workaround this issue, I can confirm read-only config files can be accessed normally. I don't really speak C++ so please do review.

brlin-tw · 2024-08-10T17:58:07Z

I noticed several new problems while researching the config loading issue:

Currently the snap hardcodes the TESSDATA_PREFIX environment variable to SNAP_USER_COMMON, we should avoid doing so to allow users to set the variable to any accessible, preferable path to have a consistent experience with the native distribution.
The TESSDATA_PREFIX hardcoded datadir macro isn't set to a valid value(should be $SNAP/usr/local/share instead of /usr/local/share, this isn't currently used as it is overridden by the hardcoded TESSDATA_PREFIX environment variable, but fixing it can simplify the aforementioned fallback config loading patch which hardcodes the entire path.)

I'll address these issues in the next couple of days.

…t variable This patch addresses the following Snapcraft linter warnings: ``` CRAFT_ARCH_TRIPLET is deprecated, use CRAFT_ARCH_TRIPLET_BUILD_{ON|FOR} ``` Refer-to: core22 | Parts environment variables | Snapcraft documentation <https://snapcraft.io/docs/parts-environment-variables#heading--snapcraft-configuration-core22> Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

egorpugin · 2024-08-10T20:37:18Z

@brlin-tw Is it possible to make tesseract-snap repo and put all snap related files there?

@stweil What do you think?

brlin-tw · 2024-08-11T08:19:09Z

@egorpugin

@brlin-tw Is it possible to make tesseract-snap repo and put all snap related files there?

It's definitely possible, though in this case, I'll have to figure out how to trigger an automatic snap build whenever the main repo has new changes. I'll look into it.

This patch fixes the incorrect compiled-in datadir path and as a side-effect, simplifies the fallback config loading patch. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

This patch allows the user to store Tesseract data in custom paths using the TESSDATA_PREFIX environment variable. Previously it is hardcoded to $SNAP_USER_COMMON. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw · 2024-08-11T08:33:27Z

Currently the snap hardcodes the TESSDATA_PREFIX environment variable to SNAP_USER_COMMON, we should avoid doing so to allow users to set the variable to any accessible, preferable path to have a consistent experience with the native distribution.

The TESSDATA_PREFIX hardcoded datadir macro isn't set to a valid value(should be $SNAP/usr/local/share instead of /usr/local/share, this isn't currently used as it is overridden by the hardcoded TESSDATA_PREFIX environment variable, but fixing it can simplify the aforementioned fallback config loading patch which hardcodes the entire path.)

These issues are fixed in the current tip of the branch.

brlin-tw · 2024-08-23T05:40:43Z

Hello, apologies for the ignorance. Here are some progress I'm doing at the moment:

I'm now experimenting with the automation required to effectively publish Tesseract snaps in a separate repo in my brlin-tw/tesseract-snap repo, it involves borrowing some of the Snapcrafters automation implementation(and slightly customized a bit) but is still a work-in-progress.
I previously had a problem publishing the snap and accessing the store's backstage due to a configuration issue at the store end, fortunately, it is finally resolved and is no longer a blocker.

This also means I can now add you as the snap's collaborator which will allow you to access the store backstage and manage application releases, simply message me the mail address associated with your Ubuntu One account.
I'm also looking into the possibility of letting the Snapcrafters organization adopt this snap if you don't prefer to manage the snap distribution as the snap would be better maintained in the long run at the cost of some flexibility(stricter contribution inclusion policy, etc.), let me know what you think.

Update: Not an option right now as the Snapcrafters organization isn't ready to adopt new snaps.

brlin-tw added 16 commits August 9, 2024 18:38

snap: Switch Leptonica source to Git repository

b613c6d

This patch mitigates risk of supply chain attack via tainted source release packages(e.g. CVE-2024-3094). Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Fix missing {build,stage}-packages declaration for the Leptonic…

0649d0f

…a part Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Ensure Leptonica features we need are built

7e9dd2d

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Update Leptonica to 1.84.1 version

8d9174a

This is the current latest version. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Fix missing WebP support in the Leptonica part

b28ab15

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Fix missing stage-packages of the Tesseract part

ef199ef

The application is still runnable due to some of the packages are already staged by the Leptonica part, however it should still be declared for clarity. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Reduce package size by stripping out files not used in runtime

8782ee4

This change reduces package size by 44%(73637888 -> 40894464). Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Drop unused training tools build/runtime dependencies

19270b4

Currently this feature isn't exposed to the host, dropping the dependencies to reduce built package size. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Explicitly declare features we don't want to be build

db818ac

Previously this is implicitly configured as the build dependency isn't available and the training tools build target isn't called. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Disable unused features to reduce snap size and built time

7a7a984

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Fix incorrect dependency declaration for the Tesseract part

7d2ba18

The zlib library is Leptonica's dependency, not Tesseract's. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Drop incorrect build-packages declaration of the Tesseract part

23b31c6

These should be Leptonica part's build dependencies. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Add cosmetic blank line separator between the definition of the…

d05286d

… tesseract and leptonica part Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

snap: Improve order of the after part property

53f2b00

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw force-pushed the snap-improvements branch from ec39230 to 89a673c Compare August 9, 2024 13:11

snap: Fix missing libarchive and libcurl support of the Tesseract part

9717555

Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw added 3 commits August 10, 2024 02:34

fixup! snap: Fix missing libarchive and libcurl support of the Tesser…

3a2cbc1

…act part libcurl functionality requires network access. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw added 2 commits August 11, 2024 16:25

snap: Fix incorrect compiled-in datadir

88c3e33

This patch fixes the incorrect compiled-in datadir path and as a side-effect, simplifies the fallback config loading patch. Signed-off-by: 林博仁(Buo-ren Lin) <buo.ren.lin@gmail.com>

brlin-tw mentioned this pull request Aug 11, 2024

Investigate on how to maintain the snap recipe in a separate repo, and trigger build whenever the main repo changes brlin-tw/tesseract-snap#1

Open

brlin-tw mentioned this pull request Aug 22, 2024

feat(release-to-candidate): Support projects that doesn't specify the version property snapcrafters/ci#46

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snap improvements #4298

Snap improvements #4298

brlin-tw commented Aug 9, 2024

stweil commented Aug 9, 2024

brlin-tw commented Aug 9, 2024 •

edited

Loading

brlin-tw commented Aug 9, 2024 •

edited

Loading

brlin-tw commented Aug 9, 2024

egorpugin commented Aug 9, 2024

brlin-tw commented Aug 9, 2024

brlin-tw commented Aug 9, 2024 •

edited

Loading

brlin-tw commented Aug 10, 2024 •

edited

Loading

brlin-tw commented Aug 10, 2024 •

edited

Loading

brlin-tw commented Aug 10, 2024 •

edited

Loading

egorpugin commented Aug 10, 2024

brlin-tw commented Aug 11, 2024

brlin-tw commented Aug 11, 2024

brlin-tw commented Aug 23, 2024 •

edited

Loading

Snap improvements #4298

Are you sure you want to change the base?

Snap improvements #4298

Conversation

brlin-tw commented Aug 9, 2024

stweil commented Aug 9, 2024

brlin-tw commented Aug 9, 2024 • edited Loading

brlin-tw commented Aug 9, 2024 • edited Loading

brlin-tw commented Aug 9, 2024

egorpugin commented Aug 9, 2024

brlin-tw commented Aug 9, 2024

brlin-tw commented Aug 9, 2024 • edited Loading

brlin-tw commented Aug 10, 2024 • edited Loading

brlin-tw commented Aug 10, 2024 • edited Loading

brlin-tw commented Aug 10, 2024 • edited Loading

egorpugin commented Aug 10, 2024

brlin-tw commented Aug 11, 2024

brlin-tw commented Aug 11, 2024

brlin-tw commented Aug 23, 2024 • edited Loading

brlin-tw commented Aug 9, 2024 •

edited

Loading

brlin-tw commented Aug 9, 2024 •

edited

Loading

brlin-tw commented Aug 9, 2024 •

edited

Loading

brlin-tw commented Aug 10, 2024 •

edited

Loading

brlin-tw commented Aug 10, 2024 •

edited

Loading

brlin-tw commented Aug 10, 2024 •

edited

Loading

brlin-tw commented Aug 23, 2024 •

edited

Loading