[DNM] GH-46736: [CI][Dev] conan-minimum build failure with RapidJSON missing error

[hiroyuki-sato]

Rationale for this change

conan-minimum can't find RapidJSON package.

https://github.com/ursacomputing/crossbow/actions/runs/15484349076/job/43595828887#step:6:1006

-- Conan: Including build module from '/root/.conan2/p/opens291bcb59a86d6/p/lib/cmake/conan-official-openssl-variables.cmake'
-- Providing CMake module for FindThriftAlt as part of Parquet CMake package
-- pkg-config package for thrift that is used by parquet for static link isn't found
CMake Warning at cmake_modules/FindRapidJSONAlt.cmake:29 (find_package):
  By not providing "FindRapidJSON.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "RapidJSON", but CMake did not find one.

  Could not find a package configuration file provided by "RapidJSON"
  (requested version 1.1.0) with any of the following names:

    RapidJSONConfig.cmake
    rapidjson-config.cmake

  Add the installation prefix of "RapidJSON" to CMAKE_PREFIX_PATH or set
  "RapidJSON_DIR" to a directory containing one of the above files.  If
  "RapidJSON" provides a separate development package or SDK, be sure it has
  been installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:310 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:2538 (resolve_dependency)
  CMakeLists.txt:618 (include)


CMake Error at /root/.conan2/p/cmakee7c682f30a1a9/p/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find RapidJSONAlt (missing: RAPIDJSON_INCLUDE_DIR) (Required is
  at least version "1.1.0")
Call Stack (most recent call first):
  /root/.conan2/p/cmakee7c682f30a1a9/p/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)

What changes are included in this PR?

Build RapidJSON from source if necessary.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

• GitHub Issue: [CI] conan-minimum build failure with RapidJSON missing error #46736

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[hiroyuki-sato]

@github-actions crossbow submit -g conan

[github-actions]

Revision: 9692eed

Submitted crossbow builds: ursacomputing/crossbow @ actions-b471051ec7

Task	Status
conan-maximum
conan-minimum

[hiroyuki-sato]

@github-actions crossbow submit -g conan

[github-actions]

Revision: d76fb6b

Submitted crossbow builds: ursacomputing/crossbow @ actions-a066ea7665

Task	Status
conan-maximum
conan-minimum

[hiroyuki-sato]

@github-actions crossbow submit -g conan

[github-actions]

Revision: c751f99

Submitted crossbow builds: ursacomputing/crossbow @ actions-721add8033

Task	Status
conan-maximum
conan-minimum

[hiroyuki-sato]

@github-actions crossbow submit -g conan

[github-actions]

Revision: cbda3b4

Submitted crossbow builds: ursacomputing/crossbow @ actions-3d10d87fac

Task	Status
conan-maximum
conan-minimum

[hiroyuki-sato]

Hello, @kou

IIUC, RapidJSON must build from source code due to INBOX RapidJSON too old(0.12) on the conan-image(Ubuntu 16.04).
I don't have 100% sure correctly change, But CI passed. What do you think this change?

Here are the results of my investigation.
#46736 (comment)

[github-actions]

⚠️ GitHub issue #46736 has been automatically assigned in GitHub to PR creator.

[kou]

We should not change conanfile.py only for our CI.
We have ci/conan/ to prevent a new release that breaks Conan packaging. So conanfile.py must be synchronized with upstream ( https://github.com/conan-io/conan-center-index/tree/master/recipes/arrow ).

Can we update Docker image for Conan instead?

[hiroyuki-sato]

@github-actions crossbow submit -g conan

[github-actions]

Revision: ec495bf

Submitted crossbow builds: ursacomputing/crossbow @ actions-ebe8a68077

Task	Status
conan-maximum
conan-minimum

[hiroyuki-sato]

@kou Thank you for your advice.

Could you tell me "docker image" means?
IIUC, this build run on conanio/gcc11-ubuntu16.04: gcc 11 on Ubuntu 16.04.
https://github.com/apache/arrow/blob/main/.env#L102-L106
This image doesn't belongs to this project.
What image should we modify?

I understand ci/conan directory is the copy of conan-center-index like homebrew #45460.
It seems that the current directory doesn't synchronize the latest.
I synchronized and latest. After the synchronization, both test has been failed.

Should we modify upstream side?

https://github.com/conan-io/conan-docker-tools/blob/master/images/README.md#official-docker-images

[kou]

Could you tell me "docker image" means? IIUC, this build run on conanio/gcc11-ubuntu16.04: gcc 11 on Ubuntu 16.04. https://github.com/apache/arrow/blob/main/.env#L102-L106 This image doesn't belongs to this project. What image should we modify?

Ah, conan-io/conan-docker-tools doesn't provide images that are based on newer Ubuntu yet?

BTW, why is RapidJSON provided by Ubuntu related? Conan has RapidJSON 1.1.0: https://github.com/conan-io/conan-center-index/blob/master/recipes/rapidjson/config.yml

We should use it instead of RapidJSON provided by Ubuntu.

I understand ci/conan directory is the copy of conan-center-index like homebrew #45460. It seems that the current directory doesn't synchronize the latest. I synchronized and latest. After the synchronization, both test has been failed.

How did you do? We have a script for it: https://github.com/apache/arrow/blob/main/ci/conan/merge_upstream.sh

[hiroyuki-sato]

Ah, conan-io/conan-docker-tools doesn't provide images that are based on newer Ubuntu yet?

Yes. (Ubuntu 16.04 is the latest).
https://hub.docker.com/u/conanio

BTW, why is RapidJSON provided by Ubuntu related? Conan has RapidJSON 1.1.0: https://github.com/conan-io/conan-center-index/blob/master/recipes/rapidjson/config.yml

Before creating this PR. the status was the below.

group	status	RapidJSON
conan-maximum:	passed	Build 1.1.0
conan-minimum:	failed	Not build

conan-maximum built RapidJSON from source code. So, conan-maximum build passed.
conan-maximum build wihth ARROW_CONAN_WITH_JSON=ON, It set arrow/*with-json=true in the conan command)

However, conam-minumum doesn't set ARROW_CONAN_WITH_JSON option. All of settings use default.
#46736 (comment)
As a reusult, It try to use the RapidJSON provided by Ubuntu. But It use RapidJSON 0.12.

arrow/ci/conan/all/conanfile.py

Line 374 in 0e5249b

tc.variables["RapidJSON_SOURCE"] = "SYSTEM"

I think that this is the reson of build failure.

IIUC, both(conon-maximum and conan-minimum ) must be build RapidJSON from source.

Changing tc.variables["RapidJSON_SOURCE"] = "SYSTEM" to tc.variables["RapidJSON_SOURCE"] = "AUTO" solve
conan-minimum issue. I think AUTO means build from source if needed.
But you told me it's not proper way.

I don't know how to fix yet. It seems I need fix upstream side.

How did you do? We have a script for it: https://github.com/apache/arrow/blob/main/ci/conan/merge_upstream.sh

I'll try this.

[hiroyuki-sato]

Hello, @amoeba. Do you have any idea how to fix the problem. I found your name from the commit log.
conan-io/conan-center-index@69d04bd

I'm not familiar with conan.

[kou]

IIUC, both(conon-maximum and conan-minimum ) must be build RapidJSON from source.

Does it mean "we use bundled RapidJSON in apache/arrow"?

We must use Conan's RapidJSON not bundled RapidJSON.

[kou]

In general, we must not use bundled dependencies with Conan. We must use dependencies in Conan instead with Conan.

[hiroyuki-sato]

Does conan-maximum(not minimum) use bundled RapidJSON?

Does it mean "we use bundled RapidJSON in apache/arrow"?

I meant conan-minimul currently, try to use the RapidJSON provided by Ubuntu, (SYSTEM)
Neither not bundled RapidJSON or the original source.
My point was conan-minimum must build RapidJSON from source.

I understand the follwoing

The goal,

group	RapidJSON	RapidJSON type
conan-minimum	Build from source	RapidJson using Conan's RapidJSON
conam-maximum	Build from source	Bundled RapidJSON??

But I don't know what configuration file should we change for use Conan's RapidJSON in the conan-manimum yet.
Do we need to modify upstream side? (not in the arrow project)

[kou]

Does conan-maximum(not minimum) use bundled RapidJSON?

I haven't checked build log but it must not use bundled RapidJSON.

Does it mean "we use bundled RapidJSON in apache/arrow"?

I meant conan-minimul currently, try to use the RapidJSON provided by Ubuntu, (SYSTEM) Neither not bundled RapidJSON or the original source. My point was conan-minimum must build RapidJSON from source.

You must not use "build RapidJSON from source". I think that you use it as "we use bundled RapidJSON" or "we use RapidJSON in Conan". But we can build RapidJSON from source OR use pre-built RapidJSON with Conan.

I understand the follwoing

The goal,
group RapidJSON RapidJSON type
conan-minimum Build from source RapidJson using Conan's RapidJSON
conam-maximum Build from source Bundled RapidJSON??

We must always use RapidJSON in Conan when RapidJSON is needed. We can build RapidJSON or pre-built RapidJSON in Conan.

But I don't know what configuration file should we change for use Conan's RapidJSON in the conan-manimum yet. Do we need to modify upstream side? (not in the arrow project)

We need check full CMake log to consider how to fix this problem. If our CMake configuration couldn't find RapidJSON provided by Conan, we must fix our CMake configuration.

[kou]

#46744

[hiroyuki-sato]

Thanks! resolved by #46744.