Skip to content

Conversation

@samuelstanton
Copy link
Collaborator

The primary objective of this PR is to implement four new node types within the pytorch-cortex library, leveraging the Hugging Face transformers library as the backend. These nodes will enable the use of Transformer architectures within the NeuralTree framework. The implementation is focused only on adding the new modules and their configuration, adhering to existing patterns where possible, and avoiding unrelated refactoring.

The goal is to add:

  1. TransformerEncoderRoot: Wraps a Hugging Face encoder model (e.g., BERT, RoBERTa, LBSTER encoders).
  2. TransformerDecoderRoot: Wraps a Hugging Face decoder-only model (e.g., GPT-2, LBSTER decoders).
  3. TransformerEncoderBranch: Applies additional Transformer encoder layers to features from the trunk.
  4. TransformerDecoderBranch: Applies additional Transformer decoder layers (causal self-attention only) to features from the trunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants