ST6RI-682 Implement loading of models from repository #616

TheKorpos · 2025-02-04T12:25:03Z

This PR adds basic model download capability using the standard API.

Overview

The model download functionality can be used

In Jupyter using magic commands
In Eclipse IDE as a "right click" action on projects
As command line java application

In Jupyter, the downloaded models are stored in memory and can be used throughout the work session. The Eclipse and command-line versions serialize the model into separate XMI files (.sysmlx). In the Eclipse IDE .kermlx and .sysmlx files are now added to the Xtext index so their content can be referenced using the textual syntax.

Common

.kermlx parsing and indexing:
- Added org.omg.kerml.xtext.xmi, org.omg.kerml.xtext.ui.xmi
.sysmlx parsing and indexing:
- Added org.omg.sysml.xtext.xmi, org.omg.sysml.xtext.ui.xmi
APIs for model download, JSON to EMF transformation
- org.omg.sysml.util.repository.ProjectRepository: repository access, download
- org.omg.sysml.xtext.ui.util.RepositoryContentFetcher: JSON to EMF transformation, XMI serialization (.sysmlx)

Jupyter

New magic commands:

%projects

Print the name and identifier of all projects in the repository.

%load [--id=<PROJECT ID>] [--name=<NAME>] [--branchid=<BRANCH ID>] [--branch=<BRANCH_NAME>] [<NAME>]

Download previously published models from a project in the repository. (Use %projects to view repository contents.)
Named elements of the downloaded models may then be referenced by models in the notebook.
<NAME> is the full name of the project.
If <PROJECT ID> is given, then the project with that UUID is loaded. In this case, the <NAME> must not be given.
If <BRANCH NAME> or <BRANCH ID> is given, then the model is loaded from this branch of the project.
If no <BRANCH NAME> or <BRANCH ID> is given, the default branch is used.
If <BRANCH ID> is given, then a <BRANCH NAME> must not be given.

Eclipse

Added SysMLv2 menu to the popup menu for projects, with the following options:
- Generate Library Index: moved under SysMLv2 menu.
- Pull model from API: download models from a repository and saves in .sysmlx
  - Requires that .settings/org.omg.sysml.remote.properties exists with the following contents:
```
base.url=repository url
remote.projectId=project id
remote.branchId=branch id
```
    (remote.branchId is optional; if not provided, the default branch is used)

NOTE:
A specific tester was implemented to hide the "Pull model from API" action from projects that don't have the required properties file (org.omg.sysml.remote.properties). However, the sysml.ui plugin must be activated for this tester to take effect. The plugin activation happens the first time the sysml editor is opened.

Command Line Application

org.omg.sysml.xtext.util.SysMLRepositoryLoadUtil
- arguments:
  -p, --project: name of the project in the repository
  -l, --library: location of the standard library
  -t, --target: target location for saved resources
  -b, --base (optional): base URL of the repository, default: http://localhost:9000

- added command to list repository contents - added command to download and index repository project - refactored code

- Introduces a new SysMLv2 menu for all contributions - Renames repository upload feature - Command framework should check for the existence of repository descriptor for showing the command

seidewitz

Instead of %publications and %download use %projects and %load.
The %load command (and utility) should have an option to use the project ID instead of the name.

- changed %publications to %projects - changed %download to %load - added optional arguments to %load --id=<ID> --name=<NAME> - changed SysMLRepositoryLoad util arguments -n, -name to reference project by name -id to reference project by id

TheKorpos · 2025-02-05T15:34:58Z

Instead of %publications and %download use %projects and %load.

The %load command (and utility) should have an option to use the project ID instead of the name.

@seidewitz
I've changed the magic command names as you requested and added a load by id option.
The magic command %load works like this now
%load --id=<PROJECT ID> loads the model by id
%load --name=<PROJECT NAME>, %load <PROJECT NAME> loads the model by name

as for the utility project name is now passed with -n <PROJECT NAME> or -name <PROJECT NAME>.
alternatively -id <PROJECT ID> can be used

- added ability to add any element as root element to the APIModel

seidewitz · 2025-05-12T16:03:05Z

@TheKorpos

The PR description contains the following:

NOTE: the action shows up regardless for now. The file is needed to perform the operation.
NOTE: a specific tester was implemented to hide the action from projects without the required properties file. However, this tester is not called before the sysml.ui plugin is activated, e.g. before a sysml editor is opened.

I don't really understand this note. Could you please removed the crossed-out line and clarify the remaining note?

publish help

magic command

seidewitz · 2025-05-13T15:57:55Z

When I try to pull a model from the API in Eclipse using the following settings:

base.url=https://sysml2-dev.intercax.com:9000
remote.projectId=02336c2a-4c7b-4518-b980-addb71d9d378

I get the following exception:

java.util.ConcurrentModificationException
	at org.eclipse.emf.common.util.AbstractEList$EIterator.checkModCount(AbstractEList.java:751)
	at org.eclipse.emf.common.util.AbstractEList$EIterator.doNext(AbstractEList.java:699)
	at org.eclipse.emf.common.util.AbstractEList$EIterator.next(AbstractEList.java:685)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1$SynchronizedEIterator.next(SynchronizedXtextResourceSet.java:196)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1$SynchronizedEIterator.next(SynchronizedXtextResourceSet.java:1)
	at org.omg.sysml.util.repository.EObjectUUIDTracker.trackUUIDSFromResources(EObjectUUIDTracker.java:69)
	at org.omg.sysml.util.repository.EObjectUUIDTracker.trackLibraryUUIDs(EObjectUUIDTracker.java:55)
	at org.omg.sysml.xtext.ui.handlers.PullRepositoryProject.tryPullRemoteFor(PullRepositoryProject.java:130)
	at org.omg.sysml.xtext.ui.handlers.PullRepositoryProject$1.runInWorkspace(PullRepositoryProject.java:97)
	at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:43)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

model upload

TheKorpos · 2025-05-14T14:02:55Z

Hi @seidewitz

java.util.ConcurrentModificationException
at org.eclipse.emf.common.util.AbstractEList$EIterator.checkModCount(AbstractEList.java:751)
at org.eclipse.emf.common.util.AbstractEList$EIterator.doNext(AbstractEList.java:699)
at org.eclipse.emf.common.util.AbstractEList$EIterator.next(AbstractEList.java:685)
at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1$SynchronizedEIterator.next(SynchronizedXtextResourceSet.java:196)
at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1$SynchronizedEIterator.next(SynchronizedXtextResourceSet.java:1)
at org.omg.sysml.util.repository.EObjectUUIDTracker.trackUUIDSFromResources(EObjectUUIDTracker.java:69)
at org.omg.sysml.util.repository.EObjectUUIDTracker.trackLibraryUUIDs(EObjectUUIDTracker.java:55)
at org.omg.sysml.xtext.ui.handlers.PullRepositoryProject.tryPullRemoteFor(PullRepositoryProject.java:130)
at org.omg.sysml.xtext.ui.handlers.PullRepositoryProject$1.runInWorkspace(PullRepositoryProject.java:97)
at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:43)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

I couldn't reproduce this issue. I do get some erros sometimes when the library is not properly built but this time around I cound't break it enough to get an error like this.

Https doens't work right now but I can reach the repo through http.

magic command

Previously loading required using UUID or name to reference projects and their branches but there were no interface that allowed mixing the two e.g reference the project by UUID and the branch by name. This rework makes load parameterization more flexible to allow more cases to be covered.

- Also corrected the incrementing of "counter" in the %projects and %load commands.

- The "next" method should only be used to create a resource for a model entered in a Jupyter notebook cell.

if possible.

seidewitz · 2025-05-19T19:44:03Z

I couldn't reproduce this issue. I do get some erros sometimes when the library is not properly built but this time around I couldn't break it enough to get an error like this.

Https doesn't work right now but I can reach the repo through http.

Oops, I knew that the Intercax repo deployment didn't work with https. The pull from repository works when I change the URL to http.

seidewitz · 2025-05-19T19:46:30Z

I also updated EMFModelDelta (on the ST6RI-178 branch, PR #633) to use the declared name of a root element (if there is one) as the name of the .sysmlx file created when pulling a project from the repository.

ST6RI-178 Implement versioned publishing

- org.omg.kerml.xmi -> org.omg.kerml.xtext.xmi - org.omg.sysml.xmi -> org.omg.sysml.xtext.xmi - org.omg.sysml.xtext.ui.utilities -> org.omg.sysml.xtext.ui.util

TheKorpos · 2025-05-20T12:31:26Z

I've moved the .gitignores to the new packages (of course I mistyped the issue number...) and updated the package exports in the manifest

seidewitz · 2025-05-20T16:25:58Z

I've moved the .gitignores to the new packages (of course I mistyped the issue number...) and updated the package exports in the manifest

It turns out that I actually had already added the new package exports to the manifests, but I had somehow forgotten to delete the old packages that were replaced. While I don't think having redundant exports causes any real problems, I just removed the redundancies for clarity.

TheKorpos added 14 commits January 27, 2025 14:58

ST6RI-682 Added PoC for model download capability

6f4d422

ST6RI-682 Fixed library id collection and refactor

0a7d3c5

ST6RI-682 Added Membership import workaround and minor fixes

43a48a7

ST6RI-763 Added standalone setups for .kermlx and .sysmlx XMIs

b762971

ST6RI-682 Registered sysmlx in Eclipse environment

57a829a

ST6RI-682 Registered .kermlx in Eclipse environment

c315a19

ST6RI-682 Added Pull project download capability to Eclipse IDE

c365e18

ST6RI-682 Transforming Xtext resources before pulling remote project

ecd87e6

ST6RI-682 Fixed MANIFEST.MF files

46d5d9c

ST6RI-682 Using Platform URI in Eclipse IDE

11da1e8

ST6RI-682 Fixed URI calculation in ProjectDelta

edf12fb

ST6RI-682 Fixed misspell in Launch Configuration

f36638f

ST6RI-682 Added model download functionality to jupyter kernel

23fd314

- added command to list repository contents - added command to download and index repository project - refactored code

ST6RI-682 Added javadoc, minor fixes and refactor

60a04a7

TheKorpos requested review from seidewitz and ujhelyiz February 4, 2025 12:25

ujhelyiz and others added 3 commits February 4, 2025 16:52

ST6RI-682 Updates Eclipse UI contributions

791f814

- Introduces a new SysMLv2 menu for all contributions - Renames repository upload feature - Command framework should check for the existence of repository descriptor for showing the command

ST6RI-682 Disabled SysMLv2 menu on closed projects

74ff13d

ST6RI-682 Ordered menu items and added separator

89209a3

seidewitz requested changes Feb 5, 2025

View reviewed changes

TheKorpos added 8 commits February 5, 2025 17:22

ST6RI-682 Made --id and --name mutually exclusive in %load command

633e761

ST6RI-178 Updated project versioning capability

a4d3e12

ST6RI-178 WIP - pushing incremental changes

09a120d

ST6RI-178 Cleaned up imports and simplified class name

af9dfb1

ST6RI-178 Added workarounds to fix upload/download in Jupyter

fe8ecfd

- added ability to add any element as root element to the APIModel

ST6RI-178 Added missing change to fix CI build

5f37567

ST6RI-178 Added basic branching capability

4a4449e

ST6RI-178 Added missing javadoc

6e61024

ST6RI-178 Owner derived feature is always included in the uploaded model

9a7f29e

TheKorpos added 3 commits May 13, 2025 10:56

ST6RI-178 Removed timestamp from project name and minor correction in

0a4d336

publish help

ST6RI-178 Added error message for missing branches in load command

c5cf3cd

ST6RI-178 Changed messages of SysMLRepositoryLoadUtil to match more with

16b95d6

magic command

ST6RI-178 isLibraryElement derived feature is always included in the

b945136

model upload

TheKorpos and others added 10 commits May 15, 2025 11:07

Merge remote-tracking branch 'origin/master' into ST6RI-178

97c6ead

ST6RI-178 Revert unplanned line ending changes to ease review

bf1bfea

ST6RI-178 Added alphabetic ordering of projects listed by %projects

7a7312f

magic command

ST6RI-178 Added --branchid to %load magic command

957cfe8

ST6RI-178 Added %projects and %load to SysMLInteractive::run().

c602522

- Also corrected the incrementing of "counter" in the %projects and %load commands.

ST6RI-178 Revised load(RemoteBranch) to not use the "next" method.

67e7173

- The "next" method should only be used to create a resource for a model entered in a Jupyter notebook cell.

ST6RI-178 Fixed root namespace insertion in model loading

6fa6377

ST6RI-178 Added base path and branch ID to messages for %load.

d73ece4

ST6RI-178 Updated EMFModelDelta to use root element name for file name,

dd437c3

if possible.

seidewitz added 3 commits May 19, 2025 17:42

Merge pull request #633 from Systems-Modeling/ST6RI-178

6994d25

ST6RI-178 Implement versioned publishing

ST6RI-682 Updated .gitignore for new xtend-gen subdirectories.

7f1d0fe

ST6RI-682 Renamed some packages relating to pull from repo in Xtext.

1dae64f

- org.omg.kerml.xmi -> org.omg.kerml.xtext.xmi - org.omg.sysml.xmi -> org.omg.sysml.xtext.xmi - org.omg.sysml.xtext.ui.utilities -> org.omg.sysml.xtext.ui.util

seidewitz approved these changes May 20, 2025

View reviewed changes

ST6RI-682 Moved .gitignores to new packages, fixed exported packages

ebdd395

TheKorpos force-pushed the ST6RI-682 branch from 3b011fa to ebdd395 Compare May 20, 2025 12:28

ST6RI-682 Removed a redundant exported packages in a couple manifests.

9fd6560

seidewitz merged commit f2dfa92 into master May 20, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ST6RI-682 Implement loading of models from repository #616

ST6RI-682 Implement loading of models from repository #616

Uh oh!

TheKorpos commented Feb 4, 2025 •

edited by seidewitz

Loading

Uh oh!

seidewitz left a comment

Uh oh!

TheKorpos commented Feb 5, 2025 •

edited

Loading

Uh oh!

seidewitz commented May 12, 2025

Uh oh!

seidewitz commented May 13, 2025

Uh oh!

TheKorpos commented May 14, 2025

Uh oh!

seidewitz commented May 19, 2025

Uh oh!

seidewitz commented May 19, 2025 •

edited

Loading

Uh oh!

TheKorpos commented May 20, 2025

Uh oh!

seidewitz commented May 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ST6RI-682 Implement loading of models from repository #616

ST6RI-682 Implement loading of models from repository #616

Uh oh!

Conversation

TheKorpos commented Feb 4, 2025 • edited by seidewitz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Common

Jupyter

Eclipse

Command Line Application

Uh oh!

seidewitz left a comment

Choose a reason for hiding this comment

Uh oh!

TheKorpos commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seidewitz commented May 12, 2025

Uh oh!

seidewitz commented May 13, 2025

Uh oh!

TheKorpos commented May 14, 2025

Uh oh!

seidewitz commented May 19, 2025

Uh oh!

seidewitz commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheKorpos commented May 20, 2025

Uh oh!

seidewitz commented May 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TheKorpos commented Feb 4, 2025 •

edited by seidewitz

Loading

TheKorpos commented Feb 5, 2025 •

edited

Loading

seidewitz commented May 19, 2025 •

edited

Loading