CFE-4341: Adjusted package module inventory to include quotes around fields when needed

[craigcomstock]

On FreeBSD where package versions can have commas such as joe-4.6,1 this was causing errors with the use of packagesmatching() function.

   error: Line from package inventory 'joe,4.6,1,amd64,pkg' did not yield correct number of elements.

Ticket: CFE-4341
Changelog: title

[craigcomstock]

@cf-bottom jenkins with exotics

[cf-bottom]

Alright, I triggered a build:

(with exotics)

Jenkins: https://ci.cfengine.com/job/pr-pipeline/10516/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-10516/

[olehermanse]

👍

[craigcomstock]

@cf-bottom jenkins with exotics please, thanks.

[cf-bottom]

Sure, I triggered a build:

(with exotics)

Jenkins: https://ci.cfengine.com/job/pr-pipeline/10518/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-10518/

[craigcomstock]

Looks like the architecture is coming out as amd64\r and not amd64 as expected.
I am looking into it.

In the test failure:

R: Packagesmatching test: [{"arch":"amd64\r","method":"test_module","name":"cfe-present-package","version":"1"}] [{"arch":"amd64\r","method":"test_module","name":"cfe-present-package","version":"1"}] [{"arch":"amd64\r","method":"test_module","name":"cfe-present-package","version":"3"}] [{"arch":"amd64\r","method":"test_module","name":"cfe-present-package","version":"3"}]

and in the test policy:

67            "module_installed_ok" expression => strcmp("$(test.test_module_packages_print)", '[{"arch":"amd64","method":"test_module","name":"cfe-present-package","version":"1"}]');

[larsewi]

Looks like the architecture is coming out as amd64\r and not amd64 as expected.

The CsvWriterNewRecord(idcw); probably adds the CRLF characters (\r\n) to the string. That's one \r character more than what was before. Maybe you don't need to call it at all, otherwise you can try to replace \r\n with \n at the end of the string?

[gettoknowmii]

Any update on this or a workflow for me to apply the change manually while waiting on this PR to get approved

[larsewi]

Hi @gettoknowmii,

@craigcomstock is taking some personal time off, thus I made an attempt to fix it while he is gone. @cf-bottom can you run this through Jenkins?

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/10537/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-10537/

[larsewi]

Retry Ubuntu 18

[larsewi]

# cat /home/vagrant/test.cf
bundle agent __main__
{
  vars:
    "my_package"
      data => packagesmatching("bogus", ".*", ".*", ".*");
  reports:
    "$(with)"
      with => storejson("my_package");
}

I was able to reproduce on master:

# cf-agent -KIf /home/vagrant/test.cf
   error: Line from package inventory 'bogus,1.4.8,1,amd64,apt_get' did not yield correct number of elements.

Tested with this branch:

# cf-agent -KIf /home/vagrant/test.cf
R: [
  {
    "arch": "amd64",
    "method": "apt_get",
    "name": "bogus",
    "version": "1.4.8,1"
  }
]

and it seems to work 🎉

[larsewi]

Reproduced on Ubuntu 22 by adding these three lines

    sys.stdout.write("Name=bogus\n")
    sys.stdout.write("Version=1.4.8,1\n")
    sys.stdout.write("Architecture=amd64\n")

to end of list_installed() function in /var/cfengine/masterfiles/modules/packages/vendored/apt_get.mustache

[larsewi]

@craigcomstock can you squash the commits on Monday? I'm not allowed to :(

[craigcomstock]

Tested this on freebsd 13.2 and looks good.

cf-check dump /var/cfengine/state/packages_installed_pkg.lmdb | grep inventory

gives

        "<inventory>": "autoconf,2.71,amd64\nautoconf-switch,20220527,amd64\nautomake,1.16.5,amd64\nbash,5.2.15,amd64\nbinutils,\"2.40_4,1\",amd64\nboost-libs,1.83.0_1,amd64\ncurl,8.5.0,amd64\nexpat,2.5.0,amd64\nfirstboot-freebsd-update,1.3,amd64\nfirstboot-pkgs,1.6,amd64\ngcc,12_5,amd64\ngcc12,12.2.0_7,amd64\ngdb,13.2_1,amd64\ngettext-runtime,0.21.1,amd64\ngit,2.43.0,amd64\ngmake,4.3_2,amd64\ngmp,6.3.0,amd64\nicu,\"74.2,1\",amd64\nindexinfo,0.3.1,amd64\nlibffi,3.4.4,amd64\nlibiconv,1.17,amd64\nlibidn2,2.3.4,amd64\nliblz4,\"1.9.4,1\",amd64\nlibnghttp2,1.58.0,amd64\nlibpsl,0.21.2_4,amd64\nlibssh2,\"1.11.0,3\",amd64\nlibtool,2.4.7_1,amd64\nlibunistring,1.1,amd64\nlibxml2,2.10.4_2,amd64\nlibyaml,0.2.5,amd64\nlmdb,\"0.9.31,1\",amd64\nm4,\"1.4.19,1\",amd64\nmpc,1.3.1_1,amd64\nmpdecimal,2.5.1,amd64\nmpfr,\"4.2.1,1\",amd64\np5-Authen-SASL,2.17,amd64\np5-CGI,4.60,amd64\np5-Clone,0.46,amd64\np5-Digest-HMAC,1.04,amd64\np5-Encode-Locale,1.05,amd64\np5-Error,0.17029,amd64\np5-GSSAPI,0.28_2,amd64\np5-HTML-Parser,3.81,amd64\np5-HTML-Tagset,3.20_1,amd64\np5-HTTP-Date,6.06,amd64\np5-HTTP-Message,6.45,amd64\np5-IO-HTML,1.004,amd64\np5-IO-Socket-IP,0.42,amd64\np5-IO-Socket-SSL,2.084,amd64\np5-LWP-MediaTypes,6.04,amd64\np5-Mozilla-CA,20231213,amd64\np5-Net-SSLeay,1.92,amd64\np5-TimeDate,\"2.33,1\",amd64\np5-URI,5.21,amd64\npcre,8.45_3,amd64\npcre2,10.42,amd64\nperl5,5.36.3_1,amd64\npkg,1.20.9,amd64\npython3,3_3,amd64\npython39,3.9.18,amd64\nreadline,8.2.0,amd64\nrsync,3.2.7,amd64\nsource-highlight,3.1.9_7,amd64\nsudo,1.9.15p5,amd64\nvirtualbox-ose-additions-nox11,6.1.50,amd64\nxxhash,0.8.2,amd64\nzstd,1.5.5,amd64\n",

Which includes quotes where commas are present in versions.

[craigcomstock]

Any update on this or a workflow for me to apply the change manually while waiting on this PR to get approved

@gettoknowmii are you able to clone, build and install? I have merged this PR so the fix is available in master branch.

Instructions are here: https://github.com/cfengine/core/blob/master/docs/BSD.md

[craigcomstock]

cherry picks
#5475
#5474

[gettoknowmii]

@craigcomstock thank you. I will go ahead and build and install and test this.

[gettoknowmii]

This works and solves my issue. Thank you so much @craigcomstock. I will be opening a PR with https://bugs.freebsd.org/ so they can patch it upstream. Thanks to everyone who worked on this.

[larsewi]

@gettoknowmii, thanks for testing 🚀