`gem update` fetches gems that are already installed

[voxik]

I have following gems available on my system:

$ gem list

*** LOCAL GEMS ***

abbrev (default: 0.1.1)
base64 (default: 0.1.1)
benchmark (default: 0.2.1)
bigdecimal (3.1.3)
bundler (2.4.6)
cgi (default: 0.3.6)
coderay (1.1.3)
csv (default: 3.2.6)
date (default: 3.3.3)
debug (1.7.1)
delegate (default: 0.3.0)
did_you_mean (default: 1.6.3)
digest (default: 3.1.1)
drb (default: 2.1.1)
english (default: 0.7.2)
erb (default: 4.0.2)
error_highlight (default: 0.5.1)
etc (default: 1.4.2)
fcntl (default: 1.0.2)
fiddle (default: 1.1.1)
fileutils (default: 1.7.0)
find (default: 0.1.1)
forwardable (default: 1.3.3)
getoptlong (default: 0.2.0)
invoice_printer (2.2.0)
invoice_printer_fonts (2.2.0)
io-console (0.6.0)
io-nonblock (default: 0.2.0)
io-wait (default: 0.3.0)
ipaddr (default: 1.2.5)
irb (1.6.2)
json (2.6.3)
logger (default: 1.5.3)
matrix (0.4.2)
method_source (1.0.0)
mutex_m (default: 0.1.2)
net-ftp (0.2.0)
net-http (default: 0.3.2)
net-imap (0.3.4)
net-pop (0.1.2)
net-protocol (default: 0.2.1)
net-smtp (0.3.3)
nkf (default: 0.1.2)
observer (default: 0.1.1)
open-uri (default: 0.3.0)
open3 (default: 0.1.2)
openssl (default: 3.1.0)
optparse (default: 0.3.1)
ostruct (default: 0.5.5)
pathname (default: 0.2.1)
pdf-core (0.9.0)
pp (default: 0.4.0)
prawn (2.4.0)
prawn-table (0.2.2)
prettyprint (default: 0.1.1)
prime (0.1.2)
pry (0.14.1)
pstore (default: 0.1.2)
psych (5.0.1)
racc (default: 1.6.2)
rdoc (6.5.0)
readline (default: 0.0.3)
readline-ext (default: 0.1.5)
reline (default: 0.3.2)
resolv (default: 0.2.2)
resolv-replace (default: 0.1.1)
rinda (default: 0.1.1)
ruby2_keywords (default: 0.0.5)
rubyzip (2.3.2)
securerandom (default: 0.2.2)
set (default: 1.0.3)
shellwords (default: 0.1.0)
singleton (default: 0.1.1)
stringio (default: 3.0.4)
strscan (default: 3.0.5)
syntax_suggest (default: 1.0.2)
syslog (default: 0.1.1)
tempfile (default: 0.1.3)
time (default: 0.2.1)
timeout (default: 0.3.1)
tmpdir (default: 0.1.3)
tsort (default: 0.1.1)
ttfunk (1.7.0)
un (default: 0.2.1)
uri (default: 0.12.0)
weakref (default: 0.1.2)
xsv (1.1.1)
yaml (default: 0.2.1)
zlib (default: 3.0.0)

Now I want to update the invoice_printer:

$ gem update invoice_printer --conservative --minimal-deps
Updating installed gems
Updating invoice_printer
Fetching ttfunk-1.7.0.gem
Fetching pdf-core-0.9.0.gem
Fetching prawn-2.4.0.gem
Fetching matrix-0.4.2.gem
Fetching prawn-table-0.2.2.gem
Successfully installed ttfunk-1.7.0
Successfully installed pdf-core-0.9.0
Successfully installed prawn-2.4.0
Successfully installed prawn-table-0.2.2
Successfully installed matrix-0.4.2
Building native extensions. This could take a while...
ERROR:  Error installing invoice_printer:
	ERROR: Failed to build gem native extension.

    current directory: /home/vondruch/.gem/ruby/gems/json-2.6.3/ext/json/ext/generator
/usr/bin/ruby -I /usr/share/rubygems extconf.rb
mkmf.rb can't find header files for ruby at /usr/share/include/ruby.h

You might have to install separate package for the ruby development
environment, ruby-dev or ruby-devel for example.

extconf failed, exit code 1

Gem files will remain installed in /home/vondruch/.gem/ruby/gems/json-2.6.3 for inspection.
Results logged to /home/vondruch/.gem/ruby/extensions/x86_64-linux/3.2.0/json-2.6.3/gem_make.out
Gems updated: ttfunk pdf-core prawn prawn-table matrix json
Gems already up-to-date: invoice_printer

Despite having the matrix, ttfunk, pdf-core, prawn and prawn-table already installed and despite additional --conservative --minimal-deps options, RubyGems are fetching them again. And I wonder why?

Please note that the gems in question are installed via RPM, so they live in the system gem location. OTOH the invoice_printer was installed via gem install and lives in my home directory. I can imagine that this mixture might be problematic, while I am quite sure it used to work. What has changed?

I am testing this scenario on my Fedora Rawhide. Initially, I have hit this issue with:

ruby-3.1.2-169.fc38.x86_64
rubygems-3.3.22-201.fc38.noarch

while later tried with

ruby-3.2.1-179.fc39.x86_64
rubygems-3.4.6-179.fc39.noarch

[voxik]

It is also ironical, that the output claims Gems already up-to-date: invoice_printer, while that is hardly true.

[deivid-rodriguez]

Thanks for reporting. I have no idea, feel free to investigate!

[voxik]

First issue, I don't think that the --conservative option is propagated into DependencyInstaller. All options which are not know to initialize are dropped

[voxik]

Secondly, I am looking at the @requests. I'd expect, that I'll see there just the invoice_printer there (I have already preinstalled all the required dependencies to simplify the situation a bit) so the result would look like:

 [Activation request
   [APISpecification
     name: invoice_printer
     version: 2.3.0
     platform: ruby
     dependencies:
     [Gem::Dependency.new("e2mmap", Gem::Requirement.new([">= 0"]), :runtime),
      Gem::Dependency.new("json", Gem::Requirement.new(["~> 2.1"]), :runtime),
      Gem::Dependency.new("matrix", Gem::Requirement.new([">= 0"]), :runtime),
      Gem::Dependency.new("prawn", Gem::Requirement.new(["~> 2.4"]), :runtime),
      Gem::Dependency.new("prawn-table", Gem::Requirement.new(["~> 0.2.2"]), :runtime)]
     set uri: https://index.rubygems.org/info/]
    for [Dependency request  invoice_printer (= 2.3.0)  requested by nil]]]

But there is the whole tree. Even if the tree should be there, then all the entries have InstalledSpecification except for JSON:

[Activation request
   [APISpecification name: json version: 2.6.3 platform: ruby dependencies: [] set uri: https://index.rubygems.org/info/]
    for [Dependency request 
     json (~> 2.1)
      requested by [Activation request
       [APISpecification
         name: invoice_printer
         version: 2.3.0
         platform: ruby
         dependencies:
         [Gem::Dependency.new("e2mmap", Gem::Requirement.new([">= 0"]), :runtime),
          Gem::Dependency.new("json", Gem::Requirement.new(["~> 2.1"]), :runtime),
          Gem::Dependency.new("matrix", Gem::Requirement.new([">= 0"]), :runtime),
          Gem::Dependency.new("prawn", Gem::Requirement.new(["~> 2.4"]), :runtime),
          Gem::Dependency.new("prawn-table", Gem::Requirement.new(["~> 0.2.2"]), :runtime)]
         set uri: https://index.rubygems.org/info/]
        for [Dependency request  invoice_printer (= 2.3.0)  requested by nil]]]],

Why would the API be queried for JSON, when the JSON is installed? Contrary to upstream Ruby, the JSON is not default gem in Fedora, but why it should matter? Default or not, it is installed and the .spec file is around.

[voxik]

I think this was introduced by #2901 and specifically 78c6b18. The issue is that the call to Gem::Specification.dirs wipes all the gem directories except the --user-install specific one. These are the directories prior the line:

["/home/vondruch/.gem/ruby/specifications", "/usr/share/gems/specifications", "/usr/local/share/gems/specifications"]

And after:

["/home/vondruch/.gem/ruby/specifications"]

and I don't think this is correct at all. I am not sure about the intent of #2886, but I am quite sure it should not result in installing additional packages.

[voxik]

Please note that I hit this issue due to JSON dependency and this case is specific to Fedora. But if there was some package depending on any bundled gem (e.g. RDoc or Minitest), such package would be installed for the second time.

Also, we should not forget that GEM_PATH allows to specify multiple locations for gems and if these are considered during runtime, they should be considered also during updates.

[voxik]

Also, we should not forget that GEM_PATH allows to specify multiple locations for gems and if these are considered during runtime, they should be considered also during updates.

Just thinking further, what I said above might be considered controversial. But what should be IMHO non-controversial is that gem update invoice_printer should not behave different to gem install invoice_printer. If the latter has not installed json dependency, neither should the former.

[deivid-rodriguez]

I was able to reproduce this with rubygems 3.4.6, but also confirmed that more recent versions behave just fine. Did not bothering bisecting the change that fixed this, though. In any case, closing!