Skip to content

Add TranslationTask #396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 55 commits into from
Dec 11, 2023
Merged

Add TranslationTask #396

merged 55 commits into from
Dec 11, 2023

Conversation

rmitsch
Copy link
Collaborator

@rmitsch rmitsch commented Dec 5, 2023 •
edited
Loading

Description

Add TranslationTask.

Corresponding documentation PR

explosion/spaCy#13183

Types of change

New feature.

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran all tests in tests and usage_examples/tests, and all new and existing tests passed. This includes
    • all external tests (i. e. pytest ran with --external)
    • all tests requiring a GPU (i. e. pytest ran with --gpu)
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

rmitsch added 30 commits October 17, 2023 12:28
@rmitsch
Add context length info. Refactor BuiltinTask and models to facilitat…
3aba660 
…e this.
@rmitsch
Merge branch 'develop' into feat/inf-doc-len
5699773 
# Conflicts:
#	spacy_llm/tasks/builtin_task.py
@rmitsch
Add token count estimator plumbing.
4213372
@rmitsch
Add plumbing for mapper and reducer.
f440ca4
@rmitsch
Add ShardMapper prototype.
e47f762
@rmitsch
Integrating mapping into prompt generation workflow.
89a5510
@rmitsch
Update response parsing and component to support sharding (WIP).
086dec9
@rmitsch
Fix shard & prompt flow.
23718fc
@rmitsch
Fix shard & prompt flow.
7ce670d
@rmitsch
Remove todo comments.
0d75ea8
@rmitsch
Fix Anthropic, Cohere, NoOp model tests.
9da7098
@rmitsch
Merge branch 'develop' into feat/inf-doc-len
0cb9afd
@rmitsch
Fix test_llm_pipe().
f368412
@rmitsch
Fix type checking test.
b1f111d
@rmitsch
Fix span parsing tests.
44a2787
@rmitsch
Fix internal tests.
6d8cdc7
@rmitsch
Fix _CountTask.
e712f41
@rmitsch
Fix sentiment and summarization tasks and tests.
985fd68
@rmitsch
Fix Azure connection URL. Fix Model test pings.
98842a2
@rmitsch
Fix Lemma parsing.
b54a3d9
@rmitsch
Start work on doc-to-shard property copying.
9bf365d
@rmitsch
Fix REL doc preprocessing.
dddfaab
@rmitsch
Remove comment on doc attribute handling during sharding, as this is …
3af21b5 
…done by spaCy's slicing directly.
@rmitsch
Add reducer implementations.
fee9ca7
@rmitsch
Implement outstanding task reducers.
e508499
@rmitsch
Resolve merge conflicts.
3218541
@rmitsch
Add shardable/non-shardable LLM task typing distinction. Add support …
c104387 
…for handling both types of tasks. Update tests.
@rmitsch
Merge branch 'develop' into feat/inf-doc-len
2c6d899 
# Conflicts:
#	spacy_llm/tests/tasks/test_rel.py
@rmitsch
Fix EL task.
2502c4d
@rmitsch
Fix EL tokenization and highlighting partially.
03055c5
@rmitsch rmitsch added feat/new New feature feat/task Feature: tasks labels Dec 5, 2023
@rmitsch rmitsch self-assigned this Dec 5, 2023
@rmitsch rmitsch changed the base branch from main to feat/inf-doc-len December 5, 2023 14:47
@rmitsch rmitsch mentioned this pull request Dec 5, 2023
Merged
3 tasks
@rmitsch rmitsch marked this pull request as ready for review December 5, 2023 15:08
@rmitsch rmitsch added the Test external Run external tests label Dec 5, 2023
svlandeg
svlandeg approved these changes Dec 7, 2023
View reviewed changes
Copy link
Contributor

@svlandeg svlandeg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Looking good. I suggest we merge this after the infinite-docs one, so this one will show up by itself in the commit history.

svlandeg
svlandeg reviewed Dec 7, 2023
View reviewed changes
Base automatically changed from feat/inf-doc-len to develop December 11, 2023 14:27
@rmitsch
Merge branch 'develop' into feat/translation-task
9ffa746 
# Conflicts:
#	spacy_llm/models/hf/base.py
#	spacy_llm/models/hf/falcon.py
#	spacy_llm/models/hf/llama2.py
#	spacy_llm/models/hf/mistral.py
#	spacy_llm/models/hf/openllama.py
#	spacy_llm/models/hf/stablelm.py
#	spacy_llm/models/langchain/model.py
#	spacy_llm/pipeline/llm.py
#	spacy_llm/tasks/entity_linker/util.py
#	spacy_llm/tests/models/test_dolly.py
#	spacy_llm/tests/models/test_langchain.py
#	spacy_llm/tests/pipeline/test_llm.py
#	spacy_llm/tests/sharding/test_sharding.py
#	spacy_llm/tests/tasks/legacy/test_ner.py
#	spacy_llm/tests/tasks/legacy/test_spancat.py
#	spacy_llm/tests/tasks/test_textcat.py
#	spacy_llm/tests/test_combinations.py
#	spacy_llm/ty.py
#	usage_examples/tests/test_readme_examples.py
@rmitsch
Resolve merge conflicts.
c840fda
@rmitsch
Merge branch 'develop' into feat/translation-task
2c691b3 
# Conflicts:
#	spacy_llm/tests/sharding/test_sharding.py
@rmitsch
Fix test_llm_task_factories.
e3f4d73
@rmitsch rmitsch merged commit f628afc into develop Dec 11, 2023
@svlandeg svlandeg deleted the feat/translation-task branch December 11, 2023 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@svlandeg svlandeg svlandeg approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@rmitsch rmitsch

Labels

feat/new New feature feat/task Feature: tasks Test external Run external tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rmitsch @svlandeg