Store UIDs in a single centralized file to reduce filesystem clutter

[MikeSchulze]

Describe the project you are working on

This is a followup proposal about how Godot should store uid's

Describe the problem or limitation you are having in your project

I wrote this proposal based on the outcome of #4603 (comment)
With Godot 4.4-dev5, the UID files are created for each resource in the file system, which clutters the file system.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Background

The main reason is to support users who deliberately or unknowingly destroy their projects by moving or renaming resources that are dependent on scenes.

dev5 state

Moreover, introducing UIDs files won't help when users improperly handle their project resources.
Here's a small example that reduces this solution to absurdity:
To hide the UID files in VS or Rider as suggested here, moving individual resources would result in the UID files remaining in their old location.
This completely breaks this solution, and worse, when Godot restarts, a new UID is created while the original UID remains as a corpse in the filesystem.
Additionally, UIDs are being created for all resources and scripts, regardless of whether they are referenced by a scene.

Case study

I use Rider and restructure my project by moving scenes and associated scripts to other directories and renaming them if necessary.
This has no effect within Rider, the scripts are still valid and no error message about invalid script references in the scenes is displayed.
To test the scene, a Godot instance is started to execute the scene, but this would end with an error on invalid script path reference.
This means that Godot must ALWAYS first check all references in scenes when starting.
To resolve missing or invalid references, Godot must scan the file system and redefine the references.
So far so correct?

I would therefore suggest a service that A is executed when Godot is started and B is scheduled as a task at regular intervals.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Implementation Strategy

The service should maintain file consistency through similarity matching:

File Tracking

• Create hash index of all script/resource files
• Parse scenes to extract script references
• Store original paths and relationships in one meta file (+ save copy)

Change Detection

• Scan the filesystem periodically
• Compare file contents using Jaccard similarity:
  similarity = |A ∩ B| / |A ∪ B|
  where A,B are sets of code lines
• Threshold of 0.8 (80%) for match detection
• Prioritize paths with similar names/structure

Reference Management

• Update internal mapping on changes are detected
• Modify scene references automatically
• Keep backup of original paths
• Log all modifications

Error Handling

• Validate new references
• Enable rollback if needed
• Notify users of changes
• Provide manual override option

The service runs silently in editor background, only surfacing when user attention is required for ambiguous cases.

If this enhancement will not be used often, can it be worked around with a few lines of script?

--

Is there a reason why this should be core and not an add-on in the asset library?

handling uids are part of the core

[ydeltastar]

There are similar cache files in .godot folder like global_script_class_cache.cfg. I agree it could be a list in a single centralized file, something like uids.cfg or project.uids next to project.godot.

Periodical scans shouldn't be necessary. There are editor and OS filesystem-watching features to do it on demand.

[sockeye-d]

I think automatically repairing references is a good idea, but before the editor actually does it it should show a pop-up that displays which resources are being remapped where, in case it fails catastrophically

[RebelliousX]

Sorry to say this, but having a separate service for maintaining this is bad in my opinion. Many things can go wrong, and the idea of scanning the project periodically basically we are asking for trouble and stutter. Instead, I think a database file that can use fast query like SQL or an open source alternative would do just fine. More on that towards the end.

The whole implementation of UID is bad in my opinion, not the idea itself per se. Since they are useful to track changes of the locations of the files for example (prevent multiple duplicate resources? Copying tscn files using file explorer into other directories can lead to this problem and get spammed in the output log about it when the project opens).

Instead of polluting the project directory and include a reference to the UID file path in the resource file like tscn or others, Godot should have a single, fast and reliable database in the .godot directory that manages all of these shenanigans. Any time the file is moved or deleted, the database should be updated accordingly. If file can not be found, scan entire project directory and if not then recreate a new UID reference in the database.

This approach will:

• Organize everything into a single database hidden in the .godot directory.
• Eliminate the need to have irrelevant files to the user of Godot exposed to the project.
• Less clutter when dealing with source control / version control like Git, less unwanted spamming of changes.
• Less disk space, allocation of hundreds of thousands of these small files even if each one takes like 20 bytes of data, they will occupy 4kb minimum depending the cluster size on the disk drive. Larger drives can have larger cluster sizes and possibly 64kb. They add up.
• Database can be compressed and optimized, queried fast using SQL or anything similar, less seeking time for those who are still using HDDs
• Faster performance in general.

[Calinou]

Godot should have a single, fast and reliable database in the .godot directory

UIDs must be committed to version control, otherwise there's no guarantee that they will be consistent across different copies of the same project.

This is also where committing a centralized file can be problematic. It'll lead to merge conflicts much more often than separate files, as adding or removing files is an operation that doesn't lead to difficult-to-resolve merge conflicts.

There are ways to engineer file formats to cause merge conflicts less often (since Git deals with merge conflicts in terms of paragraphs separated by a line of whitespace), but I don't think we'll find a way to prevent 100% of merge conflicts.

[SeremTitus]

This is also where committing a centralized file can be problematic. It'll lead to merge conflicts much more often than separate files, as adding or removing files is an operation that doesn't lead to difficult-to-resolve merge conflicts.

I think this can be address by implementing a git hook that manages fully the centralized file (e.g project.uids), similar to pre-commit used in the engine's development to maintain code style.

On change detection, instead of going for full automated healing using Jaccard similarity algorithm as suggested by the proposal, a manual alternative would be better where a pop up shows files that are no longer mapped to there uids and file that are yet to be assigned to help the user resolve externally moved and renamed files. In this case new files are not automatically assigned uids if there is an existing uid with missing file the resolve popup first appears, can be further designed to also take file extension into consideration.

We can also have both centralized file (project.uids) and the current solution, leaving it as a stylistic preference on the developer.

[AThousandShips]

My two cents on this is that I don't think the issue of clutter is remotely serious enough to justify a highly complicated, potentially fragile point of failure, especially if ensuring it doesn't cause major frustration with VCS requires even more complex code to be added.

Separate .uid files are very limited in occurrence, and most files don't use them

Adding a central file to track this might also, depending on implementation and solutions, become a bottleneck if it's supposed to be highly up-to-date, with management of pushing UID entries to this file

Also, what happens if you commit only part of some new files? What if you add a menu system to your project and you're just working on that and you don't commit that to VCS? How are the UIDs handled then? That sounds like an additional point of friction with merge conflicts, unlike .uid files which you wouldn't include if you don't include their associated file

And in the end the reason to go with .uid files was to avoid having to handle moved files with various detection methods for out-of-editor changes, the main purpose of these files are to not be handled globally but locally and simply not caring about trying to figure out where file_x went after we restarted

[RebelliousX]

@Calinou The thing is, if a file resides in the project sub-directories, that the user has no control over editing that file, then that file should be considered metadata, extra fluff.

I personally set my git to ignore .uid files, since they can be recreated if needed, no?

@AThousandShips

And in the end the reason to go with .uid files was to avoid having to handle moved files with various detection methods for out-of-editor changes, the main purpose of these files are to not be handled globally but locally and simply not caring about trying to figure out where file_x went after we restarted

How does Git recognize if a file has moved to another directory instead of considering it being deleted and a new file created?
Monitoring file changes can be done by:

• creating hash for files
• get modified time and date
• only check during project starting or during changes in editor like saving a file or moving or renaming a file.

If there are mismatches (file path doesn't exist, modified date and time not the same on database) then mark that
entry in the database and see if other hashes match with different file path and fix the links from there.

If a file path is valid, but mismatching modified date and time during saving or project loading then recreate hash for those files.

This might be unrelated, but I thought of using a database similar to how Visual Studio uses a database .sdf for auto-complete and other things. While I certainly don't know how they implement that or how it works. That database can get corrupted and that is why Visual Studio allows you to refresh or reset that database by deleting the .vs folder in the same folder as the solution, it can be recreated when solution starts.

[Calinou]

I personally set my git to ignore .uid files, since they can be recreated if needed, no?

This will break any paths in scene/resource/script files that load paths by UID. When the UID is recreated, there's no guarantee it'll have the same value as before.

[RebelliousX]

This will break any paths in scene/resource/script files that load paths by UID. When the UID is recreated, there's no guarantee it'll have the same value as before.

Then, this is exactly why the current implementation is so bad that it can break things easily. Putting the UID inside scene/resource/script files is bad. What if I want to move the scene file externally (outside of editor), do I need to move the UID file too since it resides in the same directory?

At this point, it would be better to make a drastic changes and create new file types that hold both file types in one file. The scene, resource or script file and uid file in one file like .gdz instead of .gd, .tscnz instead of .tscn..etc.

[MikeSchulze]

Sorry to say this, but having a separate service for maintaining this is bad in my opinion.

I deliberately called it a service to keep the implementation open. Personally, I would prefer a database for storage anyway, as this solves several problems by itself.

UIDs must be committed to version control, otherwise there's no guarantee that they will be consistent across different copies of the same project.

I think this approach is terrible as already described and doesn't help in the slightest if the developers don't stick to it in a disciplined way

This is also where committing a centralized file can be problematic. It'll lead to merge conflicts much more often than separate files, as adding or removing files is an operation that doesn't lead to difficult-to-resolve merge conflicts.

There are ways to engineer file formats to cause merge conflicts less often (since Git deals with merge conflicts in terms of paragraphs separated by a line of whitespace), but I don't think we'll find a way to prevent 100% of merge conflicts.

For me, these are simply pretextual arguments, if you use a database and the information is tabulated with an unique id, which the uid is, there are no problems here.
Therefore, the merge conflicts argument is obsolete

My two cents on this is that I don't think the issue of clutter is remotely serious enough to justify a highly complicated, potentially fragile point of failure, especially if ensuring it doesn't cause major frustration with VCS requires even more complex code to be added.

Separate .uid files are very limited in occurrence, and most files don't use them

I don't know how you came to that conclusion, it's extremely nerdy to have all those uid files.
In my plugin I have almost 400 additional files ending with “uid” and I find that a huge burden.

And in the end the reason to go with .uid files was to avoid having to handle moved files with various detection methods for out-of-editor changes, the main purpose of these files are to not be handled globally but locally and simply not caring about trying to figure out where file_x went after we restarted

This statement is not correct. Godot already has a service/tool that runs in the background. It searches for “missing” uid files and restores them. According to the statement, Godot should also recognize the external move at runtime, so in the end it comes down to the same thing.

This will break any paths in scene/resource/script files that load paths by UID. When the UID is recreated, there's no guarantee it'll have the same value as before.

This is also not true, if the uid files are missing they are created again and only if the .godot folder is missing.
And by the way, there is a uid cache file!

[arkology]

This is why I was against this uids zerg rush and stay on dev build without uids.
You have to commit this files. And this complicates repository navigation in browser significantly (yes, there was metadata files before for images and so on, but not for scripts - and when you browse repository, 99% of time you are looking for scripts). Or in external editor, doesn't matter.
Unfortunately I don't see any way this could be done to make everyone happy.
And one of the "coolest" part of current implementation:

When the UID is recreated, there's no guarantee it'll have the same value as before.