Original rubyforge ticket

As stated in the previous ticket:

I don't know of a good way to do this that works across platforms and makes sense.

Rubygems is for ruby. If a file is in your gem's bindir and listed as a gem executable, it should be ruby. If your gem has "special needs" then it should do it out of band, like create a Rakefile that does your special steps for you and list the Rakefile as an extension in the spec. Or just use a post-install message telling people to install with --no-wrapper.

Lemme know if I didn't understand something.

I haven't heard back in months and I still think this is a bad idea. Closing.

One use case:

Shoes 4 is written in JRuby. We need to pass options on startup to the JVM on the OSX platform only. Normally we'd do this via a bash script, but we can't, and since it's Ruby, the best we could do is spin up another Ruby, which would make the terrible JVM startup time absolutely unbearable.

@steveklabnik @zenspider @olbrich @hugogiraudel I just bumped into another valid reason in favor of allowing gem developers to install non-ruby executables automatically.

I am unaware of any way that a child process can affect the parent's process environment on UNIX, in particular, user's current directory. This is no different from Ruby scripts. However, were it a shell script, you can simply eval a string containing commands, and affect the parent shell process, including changing your current directory.

I am building a ruby version of the wd (warp directory) tool that's been available for zsh for quite some time now. I love the concept, and want to expand on it, but the core functionality of this tool is to be able to change the directory of the user that runs wd in the first place. It's easy to do in a shell, but impossible to do in Ruby.

I am currently shipping two executables with the gem: warp-dir – is a ruby file that runs the gem's code. To work around the limitation of not being able to change the directory of the parent process, I am having the ruby code generate shell compatible executable code (even for printing help). Talk about mete-meta programming :) So warp-dir ruby script is not meant to be run directly by humans. You are supposed to do something like this: eval $(warp-dir --help). So the wd executable is a tiny script that provides this functionality. And it would be lovely to have it be installed without ruby wrappers automatically. Gem user should really not worry about this type of thing.

I am glad that it looks like I can choose --no-wrapper flag while installing the gem, but it would be much more preferable if the gem author could explicitly tell rubygems not to create wrappers regardless.

One possible way to do this in a backwards compatible way, is by introducing a new config option for such scripts:

  gem.files         = `git ls-files`.split($\).reject{ |f| f =~ /^doc\// }
  gem.executables   << 'warp-dir'
  gem.non_ruby_executables   << 'wd'
  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})

By using non_ruby_executables we could add a new feature without breaking any prior expectations.

My 2c.

If this is the case, it would be great to verify this at packaging time and reject the executable. It was somewhat unexpected surprise that it blew up later after installation.

Sorry for commenting on such an old issue, but would it be possible to add something like this non_ruby_executables option?

I'm working on this PR for rubocop-daemon, which makes RuboCop much faster. It's incredibly fast when I use a bash script to talk to the daemon server using netcat, so I wanted to install this bash script as an executable.

In the meantime, how I could I provide this bash script to people who install the rubocop-daemon gem? Maybe a "post-install" script? I don't want to wrap it with Ruby, because this adds some extra latency.

I think this is unlikely to be implemented at the moment, but I don't mind reopening this ticket 🤷‍♂️.

As a workaround, I suggest you ship the script in a different location, and tell users to copy it to a project binstub in a post install message.

@deivid-rodriguez, sure, I'm working on such a workaround and others with the same issues have implemented their own. But allowing the packaging of helper non-ruby executables has the potential to make gems much more powerful and easy to use

@deivid-rodriguez

Implementing support for non_ruby_executable would probably take an experienced Rubyist a couple of hours at most.

• Clearly, enough people think this is a useful feature.
• Clearly, shell wrappers represent the lion share of such executables.
• Why not embrace this, and make everyone's life easier?

IMO from the proposed solutions there are three which I would qualify as almost identical in terms of how useful they are:

• Force --no-wrapper from gemspec
• Check shebang for each provided executable and don't wrap if it's not ruby
• I just thought of a combination of the above: Look for a second comment line after the shebang and if it's # no-wrap, don't wrap that particular executable
• non_ruby_executables in gemspec

They can probably satisfy 99% of the needs (and the needs are in <1% of all gems). However this means that for a multi-platform gem (0.01% of all cases), we'd end-up with /some/path/bin/gem.bat or gem.cmd which are the windows versions of the executable. This is clutter and not very pleasant. Plus it would be hard to provide special different custom executables for, say, mac, linux, and unix (you need to work around this with a shell script which first identifies the system and then executes the real code)

I think a truly powerful solution will allow us to provide custom install code. A hook to be executed after gem install has been called and all "classic" behavior is complete.

Still, if one of the first list is easier, even if there is will to implement the Truly Powerful Solution ™, we could absolutely start with one of the first. I can see value in having both the powerful solution and one of the first ones

@iboB From the solutions proposed, the one most appealing to me is "Check shebang for each provided executable and don't wrap if it's not ruby", since it doesn't require us adding any new feature or changing the gemspec specification. If someone can work on that, I'm happy to review such a change.

If no one starts or completes this by next week, I think I'll have time to try to whip up a PR. I'll write here if I start working on this

So, I started here: https://github.com/iboB/rubygems/tree/issue_88

The functionality is done in the first commit. It's like 5 lines of code. Now all that's left is the remaining 99% in tests and docs 🤣

@iboB Thanks for working on this 🎉. Two early comments:

• I believe already some "ruby shebang" detection code around. It'd be great if you could reuse it somehow.
• Let's focus on the bug fix for now, and leave the feature (that "rubygems: no-wrap" magic comment) for a separate PR and discussion).

@deivid-rodriguez

For a ruby shebang, I couldn't find anything appropriate. Closest I could was: https://github.com/rubygems/rubygems/blob/53a1673b8edae04f0190c30506facd6b4c1e1f72/bundler/lib/bundler/cli/exec.rb#L76_L91

... but that only checks several variants and there are many other ways to have a ruby shebang.

The pattern in my code captures anything which mentions ruby in the shebang line and this will likely match every existing and working gem executable. The thing is that after shutting down my "rubygems: no-wrap" magic, I think my pattern may be too broad 😄

I closed the PR. I think it has the potential of breaking existing gems which rely on wrapping executables with no shebang.

So I propose a different approach which guarantees not to break ANY existing gems.

Don't wrap if:

• The first line is a shebang which does not contain "ruby", or:
• The first line contains the magic string rubygems: no-wrap which is for windows batch and other scripts

(later as a separate issue I'll propose to allow the second line to contain the magic string, so authors can manually opt-out of wrap even for ruby exectuables)

@deivid-rodriguez what do you think?

Since it was a fairly simple change which doesn't affect any existing tests, I just went ahead and submitted a draft PR for this. If it's ok, I can modify the associated docs and submit the PR

I understand how the previous approach could cause incompatibilities, since not all libraries provide shebangs on top of their ruby executables. Good call on cancelling it 👍.

What's the problem with the reverse approach, though? Namely, only not wrapping the executable if the executable provides a non ruby shebang. In your pull request, you're also covering the case of "executable doesn't provide a shebang" with a custom magic comment. But doesn't that accomplish the same thing as a shebang? If as a gem author you are providing a non ruby executable, what would be the case for using rubygems: no-wrap on top of the executable instead of a shebang?

@deivid-rodriguez Windows

batch files on windows can't have a shebang. It's invalid batch code. Thus they need to have some other way of announcing their "non-rubyness". With this suggestion they would begin with either :: rubygems: no-wrap or @REM rubygems: no-wrap Both valid batch

Starting the line with :: is invald ruby as well, so I guess we could add a check whether the line starts with :: and thus prevent wrapping of batch files, but some people dislike :: and prefer @REM. The magic string works for both

(Plus, as I said, it can be reused in the future to even allow opt-out of wrap for ruby exectuables)

Sorry I missed the Windows part, thanks for explaining. I'll have another look at this soon with that in mind 👍.

Gemspec already has a way to define platform specificity and potentially exclude itself from windows.

That said, I believe you can install bash on windows which means that some of the gems expecting UNIXy things will work on windows unchanged.

I still think the same decision tree applies to gems on windows: I might want to include a .bat file as part of my gem, as well as bash script on Unix.

The wrapper/no wrapper logic should work the same. But ideally it should be possible to choose a platform specific executable for installing in PATH

@deivid-rodriguez it's been over a month

any news? :)

No, sorry @iboB, I didn't have time yet.

Hi @iboB, I'm trying to catch up with old issues. I'm in general in favour of this, but I would fix only the "non ruby shebang case" for now and wait for anybody to request the other feature. As far as I know, none of the use cases mentioned here are trying to ship a batch script with a gem, so I'd say we can attend that problem when it becomes a problem for someone? I'm just overwhelmed by the amount of features we already need to deal with, so I'd like to minimize the changes we make.

@deivid-rodriguez I am. It is a problem for me. The whole reason I got involved with this issue is because I am trying to ship batch files with my gem.

@deivid-rodriguez @iboB I thought I was in the same boat, with my gem warp-dir. It's a gem that's compatible with ZSH's wd. It installs a command wd (which is a BASH function that invokes gem's executable) and allows you to create and navigate straight to the directory bookmarks on the file system. I initially thought that I needed to ship wd as a shell script because I realized that on UNIX a sub-process can not change the directory for the parent process. But then I realized that if wd is a shell script, and it's executed as a shell script, it will still run in a sub-process, so the same restriction applies. Therefore the only way I solved this is to ask the user to install the shell function with warp-dir install --dotfile ~/.bashrc or whatever dotfile.

TL;DR

1. I no longer feel I have the need to install BASH files into gem's bin folder.
2. I suspect that any use-case that claims this requirement, will find an alternative elegant solution that only installs ruby into the Gems' bin folder.
3. Furthermore, I suspect that allowing arbitrary executables to be installed with gems will create all sorts of new attack vectors for malicious software. Can I build and install a C-binary together with gem as well?
4. With the recent addition of ruby rules to Bazel Build System, adding noni-ruby binaries would likely complicate the integration. The system is already very complicated since the goal is to create a completely hermetically sealed installation of a Ruby command with all of its supporting dependencies, including Ruby itself, standard library, and the gems, in the temporary build folder.

I still think it's ridiculous that adding a custom executable is impossible with gems and I definitely wouldn't call the workarounds and jumps through hoops people have created because of this "elegant".

I would also allow shipping binary executables, yes. In most cases it's a bad idea but certain uses may exist. More power to the developers!

As for this being an attack vector... how? You can ship absolutely anything right now. Anything except non-ruby executables in the bin folder. How is also allowing other types of executables more dangerous? It's only more convenient to do certain things, but a malicious attacker won't be stopped by inconvenience.

@iboB I think I'd rather use the rules for detecting Windows Batch scripts that you mentioned before rather than introducing a rubygems specific magic comment. The rules you mentioned exactly match Apache Tika's rules: https://github.com/apache/tika/blob/41a99cce34f90259ad8775cb8abbdbcc19e7ca27/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml#L80-L90. So it seems like a common criteria to use. What do you think?

These particular rules don't match the second-most common first line for batch files which is colon-colon. Valid ruby programs whose first line begins with rem or @echo off are also possible.

Still I don't quite understand what's wrong with the magic string. What exactly are we trying to prevent by not allowing it? I also don't understand why not allow optional opt-out of wrap even for ruby scripts

• It's future proof: Not all shells are batch or bash
• It's backwards compatible: There are no unwrapped scripts, so there's nothing to break
• It allows gems to be used for all kinds of packages instead of forcing users to jump through hoops.
• It's not more insecure: It doesn't introduce new attack vectors. You can easily ship your malicious code and call it from a by-the-book-honestly-wrapped script today. What's the difference?

I fail to see a single objective reason to be against the magic string.

The reason might be "I, the dictator of rubygems, am against it because the stars tell me so!". If this is the case, I will not whip up some broken batch detector, which will be full of false positives, or false negatives, or both, and will instead resign from this issue.

I'm just discussing and thinking what's best, there's no need for that tone.

Apologies for the snark reply, but I'm really exasperated here.

There is a proposed solution and there are arguments put forward for that solution. The response, after months, is "how about something else" with no actual counter arguments, or even mention how something else is better or how the original proposal is bad.

Trying to match shebangs or typical batch first liners is fine and makes the gem author's job easier at times, but it really makes no sense to prevent them from having a way to deal with false positives and false negatives. It will only lead to another level of hoop jumps and hacky workarounds.

The magic string is the best idea I have for this.

Well, this is a really really niche use case, so nobody cares much about it. Previously this was closed and there was no reply for years, now I'm trying to loop back every few weeks. Sorry but this is all I have.

One reason for auto-detection instead of a magic string feature is that it would just work by default. With the magic string the gem author will run into the weird issue, and might not have a clue on how to solve it and have trouble finding about hidden feature. She might even think: this is clearly a batch file, why didn't rubygems do the right thing, it's probably broken, and just give up.

Something else I'm thinking is: have you tried using a Rakefile extension for this? You can put your script at, say, lib/scripts/foo.cmd, and then add some ruby code as a Rakefile extension that will run at install time and put the script in the proper location.

I'm not familiar with that solution. I'll give it a shot.

After coming across this issue and the times it was referenced with no workarounds added here, I assumed there was no workaround.

Also non_ruby_executables in the gemspec is an alternative idea from above, which should not be forgotten. This satisfies all needs and there is no room for false positives or false negatives. (could be also be no_wrap_executables or something else)

It's an idea I just had, let me know how it goes!

Also, On the "niche-ness" of this use case, I honestly think it's a feedback loop of softs. Every other package manager allows shipping of arbitrary shell scripts or executables so people which have this use case just use alternative package managers. Shipping ruby software with npm is not unheard of.

You may be right, and "niche-ness" is very subjective anyways, so I retract that. What we know for fact is that it hasn't gauged much interest between the different rubygems maintainers.

Hello to you all. I think this is a good moment to encourage you to continue with that thing!!
I still feel as a beginner with ruby and all that, so I usually just follow the discussion - you would never know of my interest.
But I think there are many more out there like me - just listening silently - waiting for the "no-wrapper" or "this-should-be-installed-in-that-way-under-these-conditions" feature.
Hm, sounds like a lot of work - sorry.