[Feature Request]: Better project dependency management

[yingfeng]

Is there an existing issue for the same feature request?

• I have checked the existing issues.

Describe the feature you'd like

Discussed in https://github.com/orgs/infiniflow/discussions/1269

^{Originally posted by http403 June 1, 2024}
Hi community and members of infiniflow,

Will the community be interested in pausing the development of RAGFlow a bit, and restructure the project using better dependency management tool? I'm trying to make the Docker image smaller by cutting out the GPU specific dependencies, which I use Poetry to aid me, and I discover few dependencies conflicts:

• volcengine need pycryptodome==3.9.9 got pycryptodome=3.20.0
• volcengine need pytz==2020.5 got pytz==2024.1
• bcembedding need transformers>=4.35.0,<4.37.0 got transformer==4.38.1

Note: bcembedding and volcengine aren't version pinned

It will be nice to use some form of dependency management tools like Poetry or Pipenv to avoid such issues. Not to mention PyCryptodome 3.9.9 and pytz both released in 2020, which are very old which PyCryptodome have CVE-2023-52323 vulnerability before version 3.19.1.

Again, I don't mind chime in my time to make it happen.

[http403]

The author of the discussion is here. Which project dependency management would the InfiniFlow/RAGFlow team prefer? I'm more experienced with Pipenv and less with Poetry. I don't mind learn another one if that is more suitable to your workflow. There is a comparison blog which you can reference.

[KevinHuSh]

The author of the discussion is here. Which project dependency management would the InfiniFlow/RAGFlow team prefer? I'm more experienced with Pipenv and less with Poetry. I don't mind learn another one if that is more suitable to your workflow. There is a comparison blog which you can reference.

Let's use Poetry.

[CamusGao]

Besides, is there any possibilities to separating the model provider and the main program into different projects, and provide a thin version of the program that only supports the OpenAI-like API?

[http403]

@KevinHuSh I will start working on it, no specific timeline though. Expect dependency version conflicts.