Skip to content

Conversation

cdalvaro
Copy link
Contributor

@cdalvaro cdalvaro commented Aug 31, 2023

  • Have you followed the guidelines for contributing?
  • Have you ensured that your commits follow the commit style guide?
  • Have you checked that there aren't other open pull requests for the same formula update/change?
  • Have you built your formula locally with HOMEBREW_NO_INSTALL_FROM_API=1 brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
  • Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
  • Does your build pass brew audit --strict <formula> (after doing HOMEBREW_NO_INSTALL_FROM_API=1 brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

@github-actions github-actions bot added the new formula PR adds a new formula to Homebrew/homebrew-core label Aug 31, 2023
Comment on lines 33 to 38
Copy link
Contributor Author

@cdalvaro cdalvaro Aug 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the version to the binary name because when you ask catboost for its version, you get this:

Arc info:
    Branch: unknown-vcs-branch
    Commit: 0577215664901532860606512090082402431042
    Author: ordinal
    Summary: No VCS

So, no info about its version is reported.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we pass the version information to the build somehow?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed the strategy to download the source code.

By cloning the git repository, the VCS information is properly added:

Git info:
    Commit: d03b246cae23490dcf991cf822be110d6f818665
    Branch: tags/v1.2.1
    Author: akhropov <[email protected]>
    Summary: Release 1.2.1..
    git-svn info:
    Last Changed Rev: 9293085

So now, I'm testing:

assert_match %r{Branch: tags/v#{version}}, shell_output("#{bin}/catboost --version")

@cdalvaro cdalvaro force-pushed the catboost-cli branch 4 times, most recently from 8c92721 to fcaa17b Compare August 31, 2023 07:33
@cdalvaro cdalvaro changed the title catboost-cli: 1.2.1 (new formula) catboost-cli 1.2.1 (new formula) Aug 31, 2023
@chenrui333
Copy link
Member

Since we have already sharded all the formulae, this formula should go to the /c folder.

Comment on lines 9 to 13
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to add livecheck, the default one just works out fine.

Suggested change
livecheck do
url :stable
strategy :github_latest
end

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed!

Comment on lines 17 to 20
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
on_linux do
depends_on "llvm" => :build
end
uses_from_macos "llvm" => :build

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need a better test


We need a test that exercises the some of the functionality of the app. Version checks or usage checks (foo --version or foo --help) are not sufficient, as explained in the formula cookbook.

In most cases, a good test would involve running a simple test case: run #{bin}/foo input.txt.

  • Then you can check that the output is as expected (with assert_equal or assert_match on the output of shell_output)
  • You can also check that an output file was created, if that is expected: assert_predicate testpath/"output.txt", :exist?

Some advice for specific cases:

  • If the formula is a library, compile and run some simple code that links against it. It could be taken from upstream's documentation / source examples.
  • If the formula is for a GUI program, try to find some function that runs as command-line only, like a format conversion, reading or displaying a config file, etc.
  • If the software cannot function without credentials, a test could be to try to connect with invalid credentials (or without credentials) and confirm that it fails as expected.
  • Same if the software requires a virtual machine, docker instance, etc. to be running.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've managed to add a test that checks the installed binary is working as expected.

Comment on lines 16 to 28
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is cleaner than adding a ninja dependency. What is the use of CMAKE_TOOLCHAIN_FILE?

Suggested change
extra_cmake_args = [
"-S",
buildpath.to_s,
"-B",
"#{buildpath}/brew-build",
"-G",
"Ninja",
"-DCATBOOST_COMPONENTS=app",
"-DHAVE_CUDA=NO",
"-DCMAKE_TOOLCHAIN_FILE=#{buildpath}/build/toolchains/clang.toolchain",
]
system "cmake", *extra_cmake_args, *std_cmake_args
system "ninja", "-C", "#{buildpath}/brew-build", "catboost"
args = [
"-DCATBOOST_COMPONENTS=app",
"-DHAVE_CUDA=NO",
"-DCMAKE_TOOLCHAIN_FILE=#{buildpath}/build/toolchains/clang.toolchain",
]
system "cmake", "-S", ".", "-B", "build", *args, *std_cmake_args
system "cmake", "--build", "build"
system "cmake", "--install", "build"

Copy link
Contributor Author

@cdalvaro cdalvaro Sep 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Although system "cmake", "--install", "build" does nothing. So I've replaced this line by:

bin.install "catboost"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted this change, and now the formula is compiling right using ninja...

@cdalvaro cdalvaro force-pushed the catboost-cli branch 8 times, most recently from 90080a3 to e3f45dd Compare September 1, 2023 13:53
@cdalvaro
Copy link
Contributor Author

cdalvaro commented Sep 1, 2023

Thank you @chenrui333 and @SMillerDev for all your comments!

@cdalvaro cdalvaro force-pushed the catboost-cli branch 2 times, most recently from be040a8 to 27fb92a Compare September 1, 2023 14:37
@cdalvaro
Copy link
Contributor Author

cdalvaro commented Sep 1, 2023

I'm not sure why the formula is installing on my Mac but not passing the CI...

Apparently, when compiling the binary inside the CI environment, the following error arises:

[ 20%] Building CXX object library/cpp/getopt/small/CMakeFiles/cpp-getopt-small.dir/wrap.cpp.o
cd /tmp/catboost-cli-20230901-6591-5wg5ww/cmake-build/library/cpp/getopt/small && /opt/homebrew/Library/Homebrew/shims/mac/super/clang++ -DCATBOOST_OPENSOURCE=yes -I/tmp/catboost-cli-20230901-6591-5wg5ww -I/tmp/catboost-cli-20230901-6591-5wg5ww/cmake-build -I/tmp/catboost-cli-20230901-6591-5wg5ww/contrib/libs/cxxsupp/libcxx/include -I/tmp/catboost-cli-20230901-6591-5wg5ww/contrib/libs/cxxsupp/libcxxrt/include -I/tmp/catboost-cli-20230901-6591-5wg5ww/contrib/libs/clang14-rt/include -I/tmp/catboost-cli-20230901-6591-5wg5ww/contrib/libs/double-conversion -I/tmp/catboost-cli-20230901-6591-5wg5ww/contrib/libs/libc_compat/reallocarray -isystem /private/tmp/catboost-cli-20230901-6591-5wg5ww/.brew_home/.conan/data/zlib/1.2.13/_/_/package/82146580b4483a8651b14897b7c57cead95f4235/include -fexceptions   -fno-common   -fcolor-diagnostics   -faligned-allocation   -fdebug-default-version=4   -ffunction-sections   -fdata-sections   -Wall   -Wextra   -Wno-parentheses   -Wno-implicit-const-int-float-conversion   -Wno-unknown-warning-option   -pipe   -D_THREAD_SAFE   -D_PTHREADS   -D_REENTRANT   -D_LARGEFILE_SOURCE   -D__STDC_CONSTANT_MACROS   -D__STDC_FORMAT_MACROS   -D__LONG_LONG_SUPPORTED  -DLIBCXX_BUILDING_LIBCXXRT -D_FILE_OFFSET_BITS=64 -fsigned-char   -Woverloaded-virtual   -Wimport-preprocessor-directive-pedantic   -Wno-undefined-var-template   -Wno-return-std-move   -Wno-defaulted-function-deleted   -Wno-pessimizing-move   -Wno-deprecated-anon-enum-enum-conversion   -Wno-deprecated-enum-enum-conversion   -Wno-deprecated-enum-float-conversion   -Wno-ambiguous-reversed-operator   -Wno-deprecated-volatile  -O3 -DNDEBUG -std=c++20 -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -nostdinc++ -DLIBCXX_BUILDING_LIBCXXRT -D_libunwind_ -MD -MT library/cpp/getopt/small/CMakeFiles/cpp-getopt-small.dir/wrap.cpp.o -MF CMakeFiles/cpp-getopt-small.dir/wrap.cpp.o.d -o CMakeFiles/cpp-getopt-small.dir/wrap.cpp.o -c /tmp/catboost-cli-20230901-6591-5wg5ww/library/cpp/getopt/small/wrap.cpp
  In file included from /tmp/catboost-cli-20230901-6591-5wg5ww/cmake-build/catboost/libs/model/flatbuffers/features.fbs.cpp:1:
  /tmp/catboost-cli-20230901-6591-5wg5ww/cmake-build/catboost/libs/model/flatbuffers/features.fbs.h:16:10: fatal error: 'catboost/libs/helpers/flatbuffers/guid.fbs.h' file not found
  #include "catboost/libs/helpers/flatbuffers/guid.fbs.h"
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  make[2]: *** [catboost/libs/model/flatbuffers/CMakeFiles/libs-model-flatbuffers.dir/features.fbs.cpp.o] Error 1
  make[1]: *** [catboost/libs/model/flatbuffers/CMakeFiles/libs-model-flatbuffers.dir/all] Error 2

But I don't understand why I'm not seeing this issue when building the formula from source on local.

HOMEBREW_NO_INSTALL_FROM_API=1 brew install --build-from-source catboost-cli
==> Fetching dependencies for catboost-cli: pygments, python-certifi and conan@1
==> Fetching pygments
==> Downloading https://ghcr.io/v2/homebrew/core/pygments/manifests/2.16.1
Already downloaded: /Users/Carlos/Library/Caches/Homebrew/downloads/633acc99909b42dd99c46451135139a8e7e2f8da03deb0700b46ee66f17ae660--pygments-2.16.1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/pygments/blobs/sha256:559c9e5f2e02ab5f1ba7d613a8b34db2f838e1858feede8c9fd56128006
Already downloaded: /Users/Carlos/Library/Caches/Homebrew/downloads/9b6a9533a66410d529dea8fcd12cc407027c46868635968e0149936c48a41c7e--pygments--2.16.1.arm64_ventura.bottle.tar.gz
==> Fetching python-certifi
==> Downloading https://ghcr.io/v2/homebrew/core/python-certifi/manifests/2023.7.22
Already downloaded: /Users/Carlos/Library/Caches/Homebrew/downloads/d6e47db3a5fd2594b39a1e30647428b52baf34d8f1af5a13f647f2a63c59d35a--python-certifi-2023.7.22.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/python-certifi/blobs/sha256:f7878fa87dfc11f074a00e30691dc749135c15fbb999ec16fdb3d
Already downloaded: /Users/Carlos/Library/Caches/Homebrew/downloads/406789a686ca05b75465473ace9800d51520b441b0a92c79bb75b45e00324cd2--python-certifi--2023.7.22.arm64_ventura.bottle.tar.gz
==> Fetching conan@1
==> Downloading https://ghcr.io/v2/homebrew/core/conan/1/manifests/1.60.2-1
Already downloaded: /Users/Carlos/Library/Caches/Homebrew/downloads/44d89d397910bcb5d81a109157b985eac180db42b4eaf21e968d74cb18d4a82d--conan@1-1.60.2-1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/conan/1/blobs/sha256:e22deb9dfc4311be500f17b1c69c17ba75a47d39b709c9e68b04a9a801ce
Already downloaded: /Users/Carlos/Library/Caches/Homebrew/downloads/00ebe5109b53ebecb6644518cd3043be805cfaa947e1765e694e60fe0aee1101--conan@1--1.60.2.arm64_ventura.bottle.1.tar.gz
==> Fetching catboost-cli
==> Cloning https://github.com/catboost/tutorials.git
Cloning into '/Users/Carlos/Library/Caches/Homebrew/catboost-cli--testdata--git'...
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
==> Cloning https://github.com/catboost/catboost.git
Cloning into '/Users/Carlos/Library/Caches/Homebrew/catboost-cli--git'...
Updating files: 100% (53448/53448), done.
==> Checking out tag v1.2.1
HEAD is now at d03b246cae Release 1.2.1..
==> Installing dependencies for catboost-cli: pygments, python-certifi and conan@1
==> Installing catboost-cli dependency: pygments
==> Pouring pygments--2.16.1.arm64_ventura.bottle.tar.gz
🍺  /opt/homebrew/Cellar/pygments/2.16.1: 656 files, 8.3MB
==> Installing catboost-cli dependency: python-certifi
==> Pouring python-certifi--2023.7.22.arm64_ventura.bottle.tar.gz
🍺  /opt/homebrew/Cellar/python-certifi/2023.7.22: 34 files, 27.8KB
==> Installing catboost-cli dependency: conan@1
==> Pouring [email protected]_ventura.bottle.1.tar.gz
🍺  /opt/homebrew/Cellar/conan@1/1.60.2: 747 files, 6MB
==> Installing catboost-cli
==> cmake -S . -B /private/tmp/catboost-cli-20230901-77863-kpflpo/cmake-build -DCATBOOST_COMPONENTS=app -DHAVE_CUDA=NO -DCMAKE_TOO
==> cmake --build .
🍺  /opt/homebrew/Cellar/catboost-cli/1.2.1: 6 files, 26MB, built in 3 minutes 48 seconds
==> Running `brew cleanup catboost-cli`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

catboost --version
Git info:
    Commit: d03b246cae23490dcf991cf822be110d6f818665
    Branch: tags/v1.2.1
    Author: akhropov <[email protected]>
    Summary: Release 1.2.1..
    git-svn info:
    Last Changed Rev: 9293085

Other info:
    Build by: carlos.alvaro
    Top src dir: /tmp/catboost-cli-20230901-77863-kpflpo

Edit: Using Ninja as compiler the formula is working

@bayandin
Copy link
Contributor

bayandin commented Sep 3, 2023

Catboost bundles a bunch of third-party libraries (see https://github.com/catboost/catboost/tree/master/contrib/libs).
Does this formula use them? If yes we should replace them with Homebrews formulae.

@cdalvaro
Copy link
Contributor Author

cdalvaro commented Sep 3, 2023

Catboost bundles a bunch of third-party libraries (see https://github.com/catboost/catboost/tree/master/contrib/libs).
Does this formula use them? If yes we should replace them with Homebrews formulae.

These are the third-party packages and libraries installed with Conan:

conanfile.txt: Installing package
Requirements
    libiconv/1.15 from 'conancenter' - Downloaded
    openssl/1.1.1s from 'conancenter' - Downloaded
    zlib/1.2.13 from 'conancenter' - Downloaded
Packages
    libiconv/1.15:8872922bbc2111658d66264358cebe53cb98c77e - Build
    openssl/1.1.1s:82146580b4483a8651b14897b7c57cead95f4235 - Build
    zlib/1.2.13:82146580b4483a8651b14897b7c57cead95f4235 - Build
Build requirements
    bzip2/1.0.8 from 'conancenter' - Downloaded
    pcre/8.45 from 'conancenter' - Downloaded
    ragel/6.10 from 'conancenter' - Downloaded
    swig/4.0.2 from 'conancenter' - Downloaded
    yasm/1.3.0 from 'conancenter' - Downloaded
Build requirements packages
    bzip2/1.0.8:fe6bc445938d37721aaee5d1131a84487f6d375f - Build
    pcre/8.45:e44576d02fc7b9dd302b5205a3ba6386d7bea824 - Build
    ragel/6.10:3eb32b976fb0626f3c40a8b87ace72156aa44204 - Download
    swig/4.0.2:f230c9ff93e75f7f31a9ddd287cf970e1e2303b0 - Download
    yasm/1.3.0:3eb32b976fb0626f3c40a8b87ace72156aa44204 - Download

Although I don't known how Conan works and if there is a method to force Conan to use system (homebrew) libraries instead of downloading them.

Full log attached.

cmake-build.log

@bayandin
Copy link
Contributor

bayandin commented Sep 3, 2023

These are the third-party packages and libraries installed with Conan

Right, and along with this, ninja -C ... catboost compiles a bunch of stuff from contrib/libs/...

I didn't find a way to override these to system dependencies in the documentation (https://catboost.ai/en/docs/installation/build-native-artifacts), or an obvious way in cmake files.

I think, at this point catboost-cli doesn't meet the acceptance criteria:
https://docs.brew.sh/Acceptable-Formulae#stuff-that-requires-vendored-versions-of-homebrew-formulae

@cdalvaro
Copy link
Contributor Author

cdalvaro commented Sep 3, 2023

These are the third-party packages and libraries installed with Conan

Right, and along with this, ninja -C ... catboost compiles a bunch of stuff from contrib/libs/...

I didn't find a way to override these to system dependencies in the documentation (https://catboost.ai/en/docs/installation/build-native-artifacts), or an obvious way in cmake files.

I think, at this point catboost-cli doesn't meet the acceptance criteria: https://docs.brew.sh/Acceptable-Formulae#stuff-that-requires-vendored-versions-of-homebrew-formulae

OK, I understand.

I'll keep using catboost-cli from my custom tap:

brew install cdalvaro/tap/catboost-cli

@cdalvaro cdalvaro closed this Sep 3, 2023
@cdalvaro cdalvaro deleted the catboost-cli branch September 3, 2023 17:27
@github-actions github-actions bot added the outdated PR was locked due to age label Oct 4, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
new formula PR adds a new formula to Homebrew/homebrew-core outdated PR was locked due to age
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants