Optimize Backend::Flatten#flatten_keys allocations

[lovro-bikic]

Refactors Flatten#flatten_keys logic so it allocates less objects in memory, and is slightly faster.

Reduction in memory depends on the size of the locale file; the bigger the file, the fewer the allocations when storing translations. Some examples (smaller to bigger locales):

• example.yml (115 keys): 25% less memory allocated (40 kB -> 30 kB)
• Redmine en.yml (1.4k keys): ~39% reduction (527 kB -> 322 kB)
• a Rails monolith (8 locales, 13.5k keys each): ~78% reduction (26 MB -> 5.66 MB) 🚀
  • this is a client project I'm working on, so unfortunately I can only share its structure: it's a JSON file with no nesting (the keys are already flattened)

Memory profiling

Script

# profile.rb
require 'bundler/setup'
require 'i18n'
require 'memory_profiler'
require 'yaml'

locale, data = YAML.load_file('./en.yml').to_a.first

I18n.backend = I18n::Backend::KeyValue.new({}, true)
MemoryProfiler.report(allow_files: /flatten\.rb/) do
  I18n.backend.store_translations(locale, data)
end.pretty_print(allocated_strings: 0, to_file: 'report.txt', scale_bytes: true)

Make sure to download the Redmine file, and add memory_profiler to Gemfile. Place the script in repo root and run with ruby profile.rb.

Profile report before

Total allocated: 527.02 kB (8372 objects)
Total retained:  0 B (0 objects)

allocated memory by gem
-----------------------------------
 527.02 kB  i18n/lib

allocated memory by file
-----------------------------------
 527.02 kB  ~/i18n/lib/i18n/backend/flatten.rb

allocated memory by location
-----------------------------------
 372.24 kB  ~/i18n/lib/i18n/backend/flatten.rb:62
  97.27 kB  ~/i18n/lib/i18n/backend/flatten.rb:39
  57.50 kB  ~/i18n/lib/i18n/backend/flatten.rb:75

allocated memory by class
-----------------------------------
 301.68 kB  String
 111.76 kB  Array
  58.02 kB  Hash
  55.56 kB  Symbol

allocated objects by gem
-----------------------------------
      8372  i18n/lib

allocated objects by file
-----------------------------------
      8372  ~/i18n/lib/i18n/backend/flatten.rb

allocated objects by location
-----------------------------------
      6972  ~/i18n/lib/i18n/backend/flatten.rb:62
      1399  ~/i18n/lib/i18n/backend/flatten.rb:39
         1  ~/i18n/lib/i18n/backend/flatten.rb:75

allocated objects by class
-----------------------------------
      4186  String
      2794  Array
      1389  Symbol
         3  Hash

Profile report after

Total allocated: 322.34 kB (4317 objects)
Total retained:  0 B (0 objects)

allocated memory by gem
-----------------------------------
 322.34 kB  i18n/lib

allocated memory by file
-----------------------------------
 322.34 kB  ~/i18n/lib/i18n/backend/flatten.rb

allocated memory by location
-----------------------------------
 167.56 kB  ~/i18n/lib/i18n/backend/flatten.rb:62
  97.27 kB  ~/i18n/lib/i18n/backend/flatten.rb:39
  57.50 kB  ~/i18n/lib/i18n/backend/flatten.rb:75

allocated memory by class
-----------------------------------
 208.76 kB  String
  58.02 kB  Hash
  55.56 kB  Symbol

allocated objects by gem
-----------------------------------
      4317  i18n/lib

allocated objects by file
-----------------------------------
      4317  ~/i18n/lib/i18n/backend/flatten.rb

allocated objects by location
-----------------------------------
      2917  ~/i18n/lib/i18n/backend/flatten.rb:62
      1399  ~/i18n/lib/i18n/backend/flatten.rb:39
         1  ~/i18n/lib/i18n/backend/flatten.rb:75

allocated objects by class
-----------------------------------
      2925  String
      1389  Symbol
         3  Hash

Summary:

Memory
  before: 527.02 kB
  after: 322.34 kB

Objects
  before: 8372
  after: 4317

Notice that fewer allocations are due to no Array instances after the optimization, and also fewer Strings.

Benchmark

I used the already provided benchmark/run.rb on example.yml from the repo.

Report (5 runs in total)

before:

unmemoized:
store                   0.73 ms         0 objects
store                   0.72 ms         0 objects
store                   0.75 ms         0 objects
store                   0.77 ms         0 objects
store                   0.72 ms         0 objects

memoized:
store                   1.08 ms         0 objects
store                   1.06 ms         0 objects
store                   1.12 ms         0 objects
store                   1.10 ms         0 objects
store                   1.08 ms         0 objects


after:

unmemoized
store                   0.68 ms         0 objects
store                   0.68 ms         0 objects
store                   0.76 ms         0 objects
store                   0.68 ms         0 objects
store                   0.74 ms         0 objects

memoized
store                   1.06 ms         0 objects
store                   1.06 ms         0 objects
store                   1.15 ms         0 objects
store                   1.06 ms         0 objects
store                   1.14 ms         0 objects

before unmemoized avg: 	0.74 ms
after unmemoized avg:	0.71 ms (-4%)

before memoized avg:	1.09 ms
after memoized avg:	1.09 ms (same)

Best case scenario, performance is slightly improved. Worst case, it's the same as before. On other locale files, I get similar benchmark results.