Feat: Variable-Beam-Width-Search (VBWS) part4

[wili-65535]

Part 4 of VBWS support.

• Previous PRs:
  • Part 1: link
  • Part 2: link
  • Part 3: link

The progress of this PR

• Complete end-to-end support of VBWS.
• Remove all "useVariableBeamWidthSearch" flags in the runtime since we mark it in the DecodeConfig.
• Replace llmReq->mSamplingConfig.beamWidth with llmReq->getBeamWidthByIter() in the runtime to get corresponding beam width of current generation step.
• Adjust MicroBatchScheduler to schedule requests with same beam width into a batch.
• Collect generation steps of each request in the decoder for later use by beam search layer.
• Simplify members in beam search layer / kernels (needed in Diverse-Beam-Search, DBWS but needless in VBWS).
• Add related unit tests (CPP + Python).
• Update related document.
• Other small refactor of comments and format of the code.

[tensorrt-cicd]

PR_Github #4039 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2869 completed with status: 'SUCCESS'

[shiqingzhangCSU]

When running beam_width_array=[6,8,128], an error is reported.

[TensorRT-LLM][ERROR] Encountered an error in forwardSync function: [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaEventSynchronize(get()): an illegal memory access was encountered (/code/tensorrt_llm/cpp/include/tensorrt_llm/runtime/cudaEvent.h:66)
1 0x7f588642df3b void tensorrt_llm::common::check(cudaError, char const*, char const*, int) + 139
2 0x7f58871053bb tensorrt_llm::batch_manager::TrtGptModelInflightBatching::decoderSync(tensorrt_llm::batch_manager::ScheduledRequests const&, std::optional<tensorrt_llm::runtime::CudaEvent> const&) + 299
3 0x7f588710618c tensorrt_llm::batch_manager::TrtGptModelInflightBatching::forwardSync() + 716
4 0x7f58871e0228 tensorrt_llm::executor::Executor::Impl::forwardSync(std::__cxx11::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > >&) + 72
5 0x7f58871eb539 tensorrt_llm::executor::Executor::Impl::executionLoop() + 585
6 0x7f59db6c7db4 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xecdb4) [0x7f59db6c7db4]
7 0x7f5a72c9caa4 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7f5a72c9caa4]
8 0x7f5a72d29c3c /usr/lib/x86_64-linux-gnu/libc.so.6(+0x129c3c) [0x7f5a72d29c3c]

[wili-65535]

When running beam_width_array=[6,8,128], an error is reported.

[TensorRT-LLM][ERROR] Encountered an error in forwardSync function: [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaEventSynchronize(get()): an illegal memory access was encountered (/code/tensorrt_llm/cpp/include/tensorrt_llm/runtime/cudaEvent.h:66) 1 0x7f588642df3b void tensorrt_llm::common::check(cudaError, char const*, char const*, int) + 139 2 0x7f58871053bb tensorrt_llm::batch_manager::TrtGptModelInflightBatching::decoderSync(tensorrt_llm::batch_manager::ScheduledRequests const&, std::optional<tensorrt_llm::runtime::CudaEvent> const&) + 299 3 0x7f588710618c tensorrt_llm::batch_manager::TrtGptModelInflightBatching::forwardSync() + 716 4 0x7f58871e0228 tensorrt_llm::executor::Executor::Impl::forwardSync(std::__cxx11::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > >&) + 72 5 0x7f58871eb539 tensorrt_llm::executor::Executor::Impl::executionLoop() + 585 6 0x7f59db6c7db4 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xecdb4) [0x7f59db6c7db4] 7 0x7f5a72c9caa4 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7f5a72c9caa4] 8 0x7f5a72d29c3c /usr/lib/x86_64-linux-gnu/libc.so.6(+0x129c3c) [0x7f5a72d29c3c]

@shiqingzhangCSU I am not able to reproduce the error since I can run with "--beam_width_array=[6,8,128]" successfully, and my output is in the following file.
log-6_8_128.log

I use output length 8 to reduce file size, but output length 1024 is also OK to run. You can search "Using Optimization Profile: 0" in the file to check the input shape of TRT engine in each generation step.

Furthermore, could you provide more detailed information to reproduce the issue? (model name and size, build commands, run commands, etc.)

[tensorrt-cicd]

PR_Github #4709 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3396 completed with status: 'SUCCESS'

[wili-65535]

/bot run

[tensorrt-cicd]

PR_Github #4818 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #4818 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3488 completed with status: 'SUCCESS'