Running the TrackML_example with TrainTrack

[a-akram]

Hello All,

I am testing the pipeline for TrackML data following the Quick Start guide. After some effort I was able to run the pipeline partially. Here are some observations:

• First, the default pipeline_test.yaml is missing, so I use the pipeline_quickstart.yaml instead ($ traintrack configs/pipeline_quickstart.yaml)
• Wandb have to be installed explicitly

The processing stage was successful. But I am having issues with the Embedding stage. I have CPU version of PyTorch, during the validation stage when build_edges() is called, FAISS should be used for CPU and FRNN in case of GPU. It seems like even in CPU, FRNN is called which wasn't imported in the first place. Now the error is as follows:

Running from top with args: ['/home/adeel/anaconda/envs/exatrkx-test/bin/traintrack', 'configs/pipeline_quickstart.yaml']
run_pipeline.start()...
Namespace(inference=False, pipeline_config='configs/pipeline_quickstart.yaml', run_stage=False, slurm=False, verbose=False)
current stage:  {'set': 'Embedding', 'name': 'LayerlessEmbedding', 'config': 'train_quickstart_embedding.yaml', 'batch_config': 'batch_gpu_default.yaml', 'resume_id': None}
Running stage, with args, and model library: LightningModules
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
wandb: Currently logged in as: aakram (use `wandb login --relogin` to force relogin)
wandb: Tracking run with wandb version 0.12.10
wandb: Syncing run dashing-armadillo-5
wandb: ⭐️ View project at https://wandb.ai/aakram/CodaEmbeddingStudy
wandb: 🚀 View run at https://wandb.ai/aakram/CodaEmbeddingStudy/runs/24kbmduf
wandb: Run data is saved locally in /home/adeel/current/data_sets/exatrkx-test/lightning_models/lightning_checkpoints/wandb/run-20220211_205202-24kbmduf
wandb: Run `wandb offline` to turn off syncing.


  | Name    | Type       | Params
---------------------------------------
0 | network | Sequential | 3.2 M 
---------------------------------------
3.2 M     Trainable params
0         Non-trainable params
3.2 M     Total params
12.726    Total estimated model params size (MB)
Validation sanity check: 0it [00:00, ?it/s]/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/torch_geometric/deprecation.py:13: UserWarning: 'data.DataLoader' is deprecated, use 'loader.DataLoader' instead
  warnings.warn(out)
/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 8 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
Validation sanity check:   0%|                                                                                                                                     | 0/2 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/adeel/anaconda/envs/exatrkx-test/bin/traintrack", line 8, in <module>
    sys.exit(main())
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/traintrack/command_line_pipe.py", line 71, in main
    run_pipeline.start(run_args)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 152, in start
    run_stage(**config)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/traintrack/utils/data_utils.py", line 51, in wrapped
    return dFxn(*cp, **dp)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 81, in run_stage
    train_stage(model, model_config)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 96, in train_stage
    trainer.fit(model)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
    self._call_and_handle_interrupt(
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
    self._dispatch()
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
    return self._run_train()
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1311, in _run_train
    self._run_sanity_check(self.lightning_module)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1375, in _run_sanity_check
    self._evaluation_loop.run()
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
    dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 122, in advance
    output = self._evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 217, in _evaluation_step
    output = self.trainer.accelerator.validation_step(step_kwargs)
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 239, in validation_step
    return self.training_type_plugin.validation_step(*step_kwargs.values())
  File "/home/adeel/anaconda/envs/exatrkx-test/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 219, in validation_step
    return self.model.validation_step(*args, **kwargs)
  File "/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Embedding/embedding_base.py", line 301, in validation_step
    outputs = self.shared_evaluation(
  File "/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Embedding/embedding_base.py", line 257, in shared_evaluation
    e_spatial = build_edges(spatial, spatial, indices=None, r_max=knn_radius, k_max=knn_num)
  File "/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Embedding/utils.py", line 173, in build_edges
    import frnn
ModuleNotFoundError: No module named 'frnn'

wandb: Waiting for W&B process to finish, PID 21911... (failed 1). Press ctrl-c to abort syncing.
wandb:                                                                                
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Synced dashing-armadillo-5: https://wandb.ai/aakram/CodaEmbeddingStudy/runs/24kbmduf
wandb: Find logs at: /home/adeel/current/data_sets/exatrkx-test/lightning_models/lightning_checkpoints/wandb/run-20220211_205202-24kbmduf/logs/debug.log
wandb: 

I hope this will help to fix the problem, but I will try to find a solution.

[murnanedaniel]

Thanks very much for opening this issue @a-akram, and apologies for the delay in replying. I'm addressing this issue today, and will have an update for you within the next couple of hours.

[murnanedaniel]

Could you try to run the pipeline again (after a fresh install according to the new simplified instructions) and let me know if you have any issues?

[a-akram]

@murnanedaniel I have a quick run. I couldn't figure out the source of the error.

Running from top with args: ['/home/adeel/anaconda/envs/exatrkx-cpu/bin/traintrack', 'configs/pipeline_quickstart.yaml']
Namespace(slurm=False, inference=False, verbose=False, run_stage=False, pipeline_config='configs/pipeline_quickstart.yaml')
Running stage, with args, and model library: LightningModules
Traceback (most recent call last):
  File "/home/adeel/anaconda/envs/exatrkx-cpu/bin/traintrack", line 8, in <module>
    sys.exit(main())
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/command_line_pipe.py", line 69, in main
    run_pipeline.start(run_args)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/run_pipeline.py", line 149, in start
    run_stage(**config)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/utils/data_utils.py", line 51, in wrapped
    return dFxn(*cp, **dp)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/run_pipeline.py", line 75, in run_stage
    model_class = build_model(model_config)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/utils/model_utils.py", line 68, in build_model
    model_class = find_model(model_set, model_name, model_library)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/utils/model_utils.py", line 40, in find_model
    imported_module_list = [
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/traintrack/utils/model_utils.py", line 41, in <listcomp>
    importlib.import_module(".".join([model_set, "Models", module]))
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Processing/Models/feature_construction.py", line 18, in <module>
    from ..utils.event_utils import prepare_event
  File "/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Processing/utils/event_utils.py", line 23, in <module>
    from torch_geometric.data import Data
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch_geometric/__init__.py", line 4, in <module>
    import torch_geometric.data
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch_geometric/data/data.py", line 3, in <module>
    from torch_geometric.typing import OptTensor, NodeType, EdgeType
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch_geometric/typing.py", line 4, in <module>
    from torch_sparse import SparseTensor
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch_sparse/__init__.py", line 19, in <module>
    torch.ops.load_library(spec.origin)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch/_ops.py", line 110, in load_library
    ctypes.CDLL(path)
  File "/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.9/site-packages/torch_sparse/_version_cpu.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs

I have a question and a suggestion. The question is, what does Architectures dir do? In the quickstart guide it is mentioned that the Processing runs Architectures/Processing/prepare_small_feature_store.yaml but I think it should run the Pipelines/TrackML_Example/LightningModules/Processing/prepare_quickstart.yaml (maybe its just a mistake in the documentation). My understanding is that Pipelines/TrackML_Example contains everything for a successful run, so I only fix paths in this directory.

The suggestion is the paths inside the config file should be relative to EXATRKX_DATA variable, and one should know what the Architectures does?

[murnanedaniel]

Thanks for trying to run this @a-akram. To the error: This kind of error appears when you have multiple installations of a library. To test this, try to pip uninstall torch-sparse both in your environment and outside of it. Then reinstall it only in your environment (conda install pytorch-sparse -c pyg). That should clear the error. If you get similar errors (probably for the two or three other similar packages used in Pytorch Geometric), you just need to repeat this fix.

To the Architectures directory: At the moment, it's a place I'm trying to put general examples of each model. But you're right, any actual application has everything it needs in the Pipelines directories. If there is a mention of Architectures in the Quickstart, that is indeed an error.

[a-akram]

Your suggestion worked and I managed to run the Processing stage. But while running the Embedding stage, after training and validation, the test step got Killed. I couldn't figure out why the teststep failed. Both the train and val steps were successful.

Running from top with args: ['/home/adeel/anaconda/envs/exatrkx-cpu/bin/traintrack', 'configs/pipeline_quickstart.yaml']
Namespace(inference=False, pipeline_config='configs/pipeline_quickstart.yaml', run_stage=False, slurm=False, verbose=False)
Running stage, with args, and model library: LightningModules
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs

  | Name    | Type       | Params
---------------------------------------
0 | network | Sequential | 3.2 M 
---------------------------------------
3.2 M     Trainable params
0         Non-trainable params
3.2 M     Total params
12.726    Total estimated model params size (MB)
Validation sanity check: 0it [00:00, ?it/s]/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/torch_geometric/deprecation.py:13: UserWarning: 'data.DataLoader' is deprecated, use 'loader.DataLoader' instead
  warnings.warn(out)
Validation sanity check:   0%|                                                                                                                                                                                                                     | 0/2 [00:00<?, ?it/s]/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Embedding/embedding_base.py:300: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  eff = torch.tensor(cluster_true_positive / cluster_true)
/home/adeel/current/5_exatrkx/exatrkx-hsf/Pipelines/TrackML_Example/LightningModules/Embedding/embedding_base.py:301: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  pur = torch.tensor(cluster_true_positive / cluster_positive)
/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py:59: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 1. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.
  warning_cache.warn(
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90/90 [14:53<00:00,  9.93s/it, loss=0.1, v_num=1]
Testing: 0it [00:00, ?it/s]Killed

Moreover, in embedding_base.py the test_dataloader() checks for valid testset.

def val_dataloader(self):
        if len(self.valset) > 0:
            return DataLoader(self.valset, batch_size=1, num_workers=8)
        else:
            return None

def test_dataloader(self):
        if len(self.testset):
            return DataLoader(self.testset, batch_size=1, num_workers=1)
        else:
            return None

Isn't one should also set if len(self.testset) > 0: similar to the if len(self.valset) > 0: as in the val_dataloader()?

[murnanedaniel]

You're welcome to change this line, but it will act the same. I think the issue with your testing is not clear - Killed can mean many possible things. Since you are using this as a research library (I understand you will want to use the metric learning and GNN pipeline for dark matter work), I would actually recommend you step through the pipeline with the example notebooks. In particular, the Example_Embedding and GNN_Example notebooks will show you what is actually happening under the hood, and then you can play more directly with the data and model architectures. It will also be much easier to debug why the pipeline is not working on your system.
One last thing: if you have access to a GPU (e.g. even for free on Google colab), the process of training and testing will be much faster. These models (particularly the GNN) can be very slow to train on CPU.

[a-akram]

I managed to run the embedding stage. The issue was indeed the memory so I water down my data set to just a few events, reduce batch size etc, just to look for other errors.

The error occurred in the EmbeddingTelemetry callback in get_pt_metrics(). Here, one is trying to concatenate two NumPy arrays but only one array is provided. I don't understand the trick here.

pt_true_pos = np.concatenate(self.pt_true_pos, axis=1)  
pt_true = np.concatenate(self.pt_true, axis=1)

The complete stack trace is the following:

Testing: 100%|█████████████████████████████████████| 5/5 [31:29<00:00, 371.22s/it]
--------------------------------------------------------------------------------                                                                                                                                                                                         
DATALOADER:0 TEST RESULTS                                                                                                                                                                                                                                                
{}                                                                                                                                                                                                                                                                       
--------------------------------------------------------------------------------                                                                                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                                                                       
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/bin/traintrack", line 8, in <module>                                                                                                                                                                                
    sys.exit(main())                                                                                                                                                                                                                                                     
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/traintrack/command_line_pipe.py", line 69, in main                                                                                                                                      
    run_pipeline.start(run_args)                                                                                                                                                                                                                                         
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 149, in start                                                                                                                                         
    run_stage(**config)                                                                                                                                                                                                                                                  
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/traintrack/utils/data_utils.py", line 51, in wrapped                                                                                                                                    
    return dFxn(*cp, **dp)                                                                                                                                                                                                                                               
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 81, in run_stage                                                                                                                                      
    train_stage(model, model_config)                                                                                                                                                                                                                                     
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 105, in train_stage                                                                                                                                   
    trainer.test(model)                                                                                                                                                                                                                                                  
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 911, in test                                                                                                                                
    return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)                                                                                                                                                          
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt                                                                                                          
    return trainer_fn(*args, **kwargs)                                                                                                                                                                                                                                   
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 954, in _test_impl                                                                                                                          
    results = self._run(model, ckpt_path=self.tested_ckpt_path)                                                                                                                                                                                                          
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run                                                                                                                               
    self._dispatch()                                                                                                                                                                                                                                                     
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1275, in _dispatch                                                                                                                          
    self.training_type_plugin.start_evaluating(self)                                                                                                                                                                                                                     
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 206, in start_evaluating                                                                                         
    self._results = trainer.run_stage()                                                                                                                                                                                                                                  
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1286, in run_stage                                                                                                                          
    return self._run_evaluate()                                                                                                                                                                                                                                          
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1334, in _run_evaluate                                                                                                                      
    eval_loop_results = self._evaluation_loop.run()                                                                                                                                                                                                                      
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 151, in run                                                                                                                                      
    output = self.on_run_end()                                                                                                                                                                                                                                           
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 140, in on_run_end                                                                                                         
    self._on_evaluation_end()                                                                                                                                                                                                                                            
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 200, in _on_evaluation_end                                                                                                 
    self.trainer.call_hook("on_test_end", *args, **kwargs)                                                                                                                                                                                                               
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1495, in call_hook                                                                                                                          
    callback_fx(*args, **kwargs)                                                                                                                                                                                                                                         
  File "/export/home/adeel/anaconda/envs/exatrkx-cpu/lib/python3.8/site-packages/pytorch_lightning/trainer/callback_hook.py", line 231, in on_test_end                                                                                                                   
    callback.on_test_end(self, self.lightning_module)                                                                                                                                                                                                                    
  File "/export/home/adeel/current/stt-hsf/LightningModules/Embedding/Models/inference.py", line 76, in on_test_end                                                                                                                                                      
    metrics = self.calculate_metrics()                                                                                                                                                                                                                                   
  File "/export/home/adeel/current/stt-hsf/LightningModules/Embedding/Models/inference.py", line 124, in calculate_metrics                                                                                                                                               
    centers, ratio_hist = self.get_pt_metrics()                                                                                                                                                                                                                          
  File "/export/home/adeel/current/stt-hsf/LightningModules/Embedding/Models/inference.py", line 84, in get_pt_metrics                                                                                                                                                   
    pt_true_pos = np.concatenate(self.pt_true_pos, axis=1)                                                                                                                                                                                                               
  File "<__array_function__ internals>", line 5, in concatenate                                                                                                                                                                                                          
ValueError: need at least one array to concatenate

There are a few interesting warnings as well:

.../python3.8/site-packages/torch_geometric/deprecation.py:13: UserWarning: 'data.DataLoader' is deprecated, use 'loader.DataLoader' instead
.../LightningModules/Embedding/embedding_base.py:301: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor
(sourceTensor).
  eff = torch.tensor(cluster_true_positive / cluster_true)
.../LightningModules/Embedding/embedding_base.py:302: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor
(sourceTensor).
  pur = torch.tensor(cluster_true_positive / cluster_positive)
...python3.8/site-packages/pytorch_lightning/utilities/data.py:59: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 1. To avoid any miscalculations, use `self.log(..
., batch_size=batch_size)`.