Feature/shepherd

[xzyaoi]

No description provided.

[Copilot]

Copilot reviewed 12 out of 24 changed files in this pull request and generated 1 comment.

Files not reviewed (12)

• meta/requirements.txt: Language not supported
• tools/simulator/core/engine.py: Evaluated as low risk
• scratchpad/utils/client/req.py: Evaluated as low risk
• scratchpad/utils/ui/init.py: Evaluated as low risk
• tools/simulator/core/request.py: Evaluated as low risk
• tools/simulator/core/policies/routing/random.py: Evaluated as low risk
• scratchpad/utils/client/init.py: Evaluated as low risk
• tools/simulator/core/policies/routing/_base.py: Evaluated as low risk
• tools/simulator/core/policies/init.py: Evaluated as low risk
• tools/client/req.py: Evaluated as low risk
• scratchpad/utils/ui/ui.py: Evaluated as low risk
• tools/shepherd/prompt_builder.py: Evaluated as low risk

Comments suppressed due to low confidence (6)

tools/shepherd/data_gen.py:18

• The print statement should use parentheses to correctly call the len function: f"[{len(questions)}/{args.num_samples}] Score: {score}, Question: {datum}"

f"[len(questions)/{args.num_samples}] Score: {score}, Question: {datum}"

tools/shepherd/data_gen.py:23

• Use os.path.join for better cross-platform compatibility: with open(os.path.join('.local', 'calibration.jsonl'), 'w') as f:

with open(".local/calibration.jsonl", "w") as f:

tools/simulator/cli/start_simulator.py:39

• The make_table function assumes that data is not empty and that data[0] has a keys method. Add a check to ensure data is not empty and that data[0] has a keys method.

console.print(make_table("Summary", server.summary))

tools/shepherd/response_collector.py:12

• The response is checked only for None. Consider checking for other invalid responses like empty strings or malformed JSON.

if response is None:

tools/shepherd/response_collector.py:30

• The LLM class instantiation is repeated within the loop. Consider moving it outside the loop for optimization.

for i, datum in tqdm(enumerate(data)):

tools/shepherd/response_collector.py:30

• The tqdm progress bar is not properly utilized. It should be used with the for loop directly.

for i, datum in tqdm(enumerate(data)):