feat: add load process for components and their versions #390

dwong2708 · 2025-09-18T22:57:58Z

Resolves: #383

Changes

Implemented load process for components
Implemented load process for component versions (both draft and published)
Skipped component static files during load

End-to-End Testing Output

Input:
Given this dump file:
test.zip

Result:
We obtained the following output:

Components:

Publish Logs:

Draft Change Logs:

openedx-webhooks · 2025-09-18T22:58:02Z

Thanks for the pull request, @dwong2708!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

ormsbee

First pass complete. Thank you!

ormsbee · 2025-09-19T17:30:07Z

openedx_learning/apps/authoring/backup_restore/toml.py

+    if "entity" not in pe_data:
+        raise ValueError("Invalid publishable entity TOML: missing 'entity' section")
+    if "version" not in pe_data:
+        raise ValueError("Invalid publishable entity TOML: missing 'version' section")
+    if "key" not in pe_data["entity"]:
+        raise ValueError("Invalid publishable entity TOML: missing 'key' field")
+    if "can_stand_alone" not in pe_data["entity"]:
+        raise ValueError("Invalid publishable entity TOML: missing 'can_stand_alone' field")


Assume that these error messages will go into a log file, and try to surface everything that's wrong with the publishable entity at once, not just one at a time. Also include identifying information for which publishable entity is missing this information if possible, so there's some way to identify it.

I implemented serializers to collect those errors for a future log file. Thank you.

ormsbee · 2025-09-19T18:28:56Z

openedx_learning/apps/authoring/backup_restore/zipper.py

+        component_content = self._read_file_from_zip(zipf, component_file_path)
+        component_data, component_version_data = parse_publishable_entity_toml(component_content)
+
+        with publishing_api.bulk_draft_changes_for(learning_package.id):


If it's straightforward to do, we probably want two big bulk_draft_changes_for blocks--the first loads the published versions of everything and publishes them. The second would create the draft versions of everything.

Change applied, thanks!

ormsbee · 2025-09-19T18:30:42Z

openedx_learning/apps/authoring/backup_restore/zipper.py

+                A tuple of (ComponentType, local_key) if valid, else (None, None).
+        """
+        if not entity_key:
+            return None, None


Why return this instead of raising an error?

Yeah it was weird. I changed it. Thanks

Is there a newer version of this that you haven't pushed up yet?

Oh, I see. I forgot to remove it. It’s no longer needed since I added this logic in the components API layer. I'll remove it now.

Applied. Thank you

ormsbee · 2025-09-19T18:31:49Z

openedx_learning/apps/authoring/backup_restore/zipper.py

+
+        # Extract component type information
+        namespace, component_type_name, filename = parts
+        local_key = filename.rsplit(".", 1)[0]  # Remove .toml extension


The component key parts should be derived from the [entity] key, not the file path, since the entity key is the canonical source for that data. Also, the logic for deriving the key parts from the entity key is something that could go into the components app api.py file.

It was already using the entity key. I applied a change to move it to the component API file. Thanks.

ormsbee · 2025-09-19T18:34:55Z

openedx_learning/apps/authoring/backup_restore/zipper.py

+
+            # Only assign draft if it’s not the same as published
+            if version_num == draft_version_num and version_num != published_version_num:
+                draft_version = version


The draft_version should exist regardless of whether or not it's the same as the published version.

Fixed. Thanks

- Add Component and ComponentVersion serializers. The goal is to validate inputs and capture errors consistently. - Refactor the component saving process: it now uses two separate blocks with the bulk draft changes context manager.

dwong2708 · 2025-09-20T18:38:40Z

First round of addressing comments is done.

In a nutshell, I implemented these changes:

Added Component and ComponentVersion serializers to validate inputs and capture errors consistently, supporting future error logging.
Refactored the component saving process: it now uses two separate blocks with the bulk draft changes context manager.

Results derived from these changes:

Input:

Output:
test.zip

Publishable entities

Component versions

Draft Change Log

Publish Log

Thank you!

ormsbee

Can you please investigate using DRF Serializers to do the validation? It seems like if we're at the point where we're using serializer classes, we might as well use standard ones.

You'd probably want to subclass rest_framework.serializers.Serializer for this work.

Thank you.

openedx_learning/apps/authoring/components/api.py

dwong2708 · 2025-09-22T23:35:38Z

I agree with using DRF Serializer, it was straightforward to implement here.
I was looking at this thread: https://openedx.slack.com/archives/C05NT2YN820/p1758574908253769, and Pydantic would have been nice to use since it’s less coupled. However, our environment is Django-based, so I agree that using a DRF serializer is the right choice.
Thank you

ormsbee

Question and a small request for a test, but otherwise looks good to merge. Thank you.

ormsbee · 2025-09-23T19:39:14Z

openedx_learning/apps/authoring/backup_restore/zipper.py

+            if not file.endswith(".toml"):
+                # Skip non-TOML files
+                continue


How do non-TOML files end up here in the first place?

The non-TOML files are static files. Here’s an example list

file entities/xblock.v1/drag-and-drop-v2/4d1b2fac-8b30-42fb-872d-6b10ab580b27/component_versions/v2/block.xml file entities/xblock.v1/html/e32d5479-9492-41f6-9222-550a7346bc37/component_versions/v5/static/me.png file entities/xblock.v1/html/e32d5479-9492-41f6-9222-550a7346bc37/component_versions/v5/block.xml file entities/xblock.v1/html/e32d5479-9492-41f6-9222-550a7346bc37/component_versions/v4/block.xml file entities/xblock.v1/html/e32d5479-9492-41f6-9222-550a7346bc37/component_versions/v4/static/me.png file entities/xblock.v1/openassessment/1ee38208-a585-4455-a27e-4930aa541f53/component_versions/v2/block.xml file entities/xblock.v1/problem/256739e8-c2df-4ced-bd10-8156f6cfa90b/component_versions/v2/block.xml file entities/xblock.v1/survey/6681da3f-b056-4c6e-a8f9-040967907471/component_versions/v1/block.xml file entities/xblock.v1/video/22601ebd-9da8-430b-9778-cfe059a98568/component_versions/v3/block.xml

This has not been implemented yet, but it will be included in the next steps

ormsbee · 2025-09-23T19:54:48Z

openedx_learning/apps/authoring/components/api.py

+            f"Invalid entity_key format: {entity_key!r}. "
+            "Expected format: '{namespace}:{type_name}:{local_key}'"
+        ) from exc
+    return get_or_create_component_type(namespace, type_name), local_key


Please write a test for this.

Applied. Thanks

feat: add load process for components and their versions

32dc910

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Sep 18, 2025

openedx-webhooks added this to Contributions Sep 18, 2025

github-project-automation bot moved this to Needs Triage in Contributions Sep 18, 2025

dwong2708 marked this pull request as ready for review September 18, 2025 23:08

dwong2708 requested a review from ormsbee September 18, 2025 23:08

mphilbrick211 added the mao-onboarding Reviewing this will help onboard devs from an Axim mission-aligned organization (MAO). label Sep 19, 2025

mphilbrick211 moved this from Needs Triage to Ready for Review in Contributions Sep 19, 2025

ormsbee requested changes Sep 19, 2025

View reviewed changes

dwong2708 requested a review from ormsbee September 20, 2025 18:38

ormsbee requested changes Sep 22, 2025

View reviewed changes

openedx_learning/apps/authoring/components/api.py Outdated Show resolved Hide resolved

dwong2708 added 2 commits September 22, 2025 17:28

feat: use DRF serializers to handle restore library data

72dae16

fix: minor adjustment

e745137

dwong2708 requested a review from ormsbee September 22, 2025 23:31

mphilbrick211 moved this from Ready for Review to In Eng Review in Contributions Sep 23, 2025

fix: remove _get_component_type method

b5cac7e

ormsbee requested changes Sep 23, 2025

View reviewed changes

test: add unit tests for component type utility functions

9a6fb4e

dwong2708 requested a review from ormsbee September 23, 2025 22:36

ormsbee approved these changes Sep 24, 2025

View reviewed changes

ormsbee merged commit 3104fe8 into openedx:main Sep 24, 2025
11 checks passed

github-project-automation bot moved this from In Eng Review to Done in Contributions Sep 24, 2025

feat: add load process for components and their versions #390

feat: add load process for components and their versions #390

Uh oh!

Conversation

dwong2708 commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

End-to-End Testing Output

Uh oh!

openedx-webhooks commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ormsbee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dwong2708 commented Sep 20, 2025

Uh oh!

ormsbee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dwong2708 commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ormsbee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dwong2708 commented Sep 18, 2025 •

edited

Loading

openedx-webhooks commented Sep 18, 2025 •

edited

Loading

dwong2708 commented Sep 22, 2025 •

edited

Loading