When Git commits are missed, the files are not included with the next commit

[ljhaywar]

Here is what I know:
Stream A and Stream B have common history.
I migrated Stream A. The end result of File 1 in Git matched the end result in RTC, so all was well.
I migrated the portion of Stream B from the branching point forward. Then I rebased the history so that Stream A had a complete set of history. The end result of File 1 in Git did not match the end result in RTC.

It turns out that some of the Git commits were missed in the migration of Stream A prior to the branch point. I expected that any code changes that were not migrated as part of a Git commit (for example, if the script stopped because of a merge conflict and then was manually restarted) would be included in the very next Git commit. This would result in the code changes being associated with the wrong comment, but I was ok with that if it only happened occasionally. Upon investigation, that is not what actually happens. The code changes that are missed do not go into the very next Git commit--they go into the next Git commit that touches that file. In my case, the missed Git commit happened before the branching point. The catchup of the missed code change happened after the branching point. Unfortunately, in my case, this means that Stream B never got this code change--resulting in an incorrect file.

I'm going to think this through some more...would committing file changes whenever the script stops fix the problem?

[ljhaywar]

I just went back through my logs for the migration of Stream A and found this:
The following resources had uncommitted changes and were overwritten:
A copy of each file is available in "C:\temp...".
File1
Other files
Successfully loaded items into the sandbox.

Continuing to investigate...

[ljhaywar]

I had several files that didn't match and were listed in the logs as I described above. I had a few files not match due to the capitalization change as I described in Issue 31.

I just discovered one file that was listed differently in the logs:
The following resources had uncommitted changes and were overwritten:
A copy of each file is available in "C:\temp...".
pathToADifferentFile
Successfully loaded items into the sandbox.
The flow target was updated.
Saved working directory and index state WIP on mybranchname: hash1 commitComment1
HEAD is now at hash1 commitComment2
diff --git pathToMyFile
index 0f24917..3edcb7c 100644
--- pathToMyFile
+++ pathToMyFile

I'm not sure yet what the Git ouptut means.
@@ -2539,15 +2539,10 moreComments

[WtfJoke]

The code changes that are missed do not go into the very next Git commit--they go into the next Git commit that touches that file.

Well that almost cant be the case... After every accept the command "git add -A" will be executed (this means all changed files) will be added to git with the following commit.
You can test this, when you do a change (for example an accept a change) using eclipse and using resume in the script you will see that the change gets added.
If you miss some changes there needs to be another explanation.

would committing file changes whenever the script stops fix the problem?

I don't think so, usually when you hit issues while accepting changesets, your workspace doesnt change. So there is nothing changed on the filesystem and therefore you cant commit anything.

So at the moment I am clueless, how this could happen... Sorry

EDIT:
Ah I just saw something. This line is interesting:
Saved working directory and index state WIP on mybranchname: hash1 commitComment1
HEAD is now at hash1 commitComment2

This is the output of the git stash command. Well I need to rethink this. But maybe the reason is that the git checkout reverts these files? Very unlickely, but just maybe... This would be at least an explanation.

Can you check what kind of stuff is in your stash (use git stash -list)?

[ljhaywar]

The error message saying the resources had uncommitted changes and were overwritten is a RTC message (at least that is what I think since the path where the file is saved has ".jazzshed" in it.

The details are still fuzzy to me, but here is what I think is happening:

• The script has issues when trying to resume Delivery A and stops.
• I manually resume Delivery A.
• Without doing a manual "git add -A", I restart the script.
• RTC realizes it has files in the workspace that it doesn't know how it got there and prints the message "The following resources had uncommitted changes and were overwritten." (This confuses me a bit since I'm using the same repository workspace for both, so I'm not sure why it would think it has uncommitted changes.) RTC removes the changes from Delivery A.
• The script continues and accepts Delivery B. Since the only changes in the filesystem are from Delivery B, "git add -A" only picks up the changes from Delivery B. Delivery A is completely gone.
• The script continues and accepts deliveries C-E and adds them appropriately to Git.
• Delivery F touches the same file that had issues in Delivery A. In the script, RTC resumes Delivery F. The resume pulls in all changes to the file to bring it to current with Delivery F, essentially pulling in changes from both Delivery A and Delivery F. When the script does "git add -A", the changes from Delivery A and Delivery F are added to the Git repo.

I'm running a test now to see if doing the manual "git add -A" when the script stops fixes the problem.

[WtfJoke]

More interesting would be if it fixes the problem when you do the git add -A after you reload the workspace manually

[WtfJoke]

When you test, do a git status after each action... So you know when git add -A does something. :)

[ljhaywar]

Good news! Running "git add -A" seems to have fixed my migration problems. I ran a test of migrating a complete stream and rebasing, and this time the contents of every file was correct. (The only issue I hit was with the capitalization of some of the file names.) Details are below.

My script stopped 7 times. One time I accidentally stopped the script as I thought it was stuck on a conflict but it actually was not, so we'll throw away this case. 5 of the cases behaved in the same way. 1 case behaved strangely.

5 cases that behaved the same way:

• Script output showed a conflict it could not recover from.
• "git status" showed changes not staged for commit
• the History view in Eclipse showed a changeset that was resumed, but this delivery had not been converted to a commit in the Git repo's history yet
• I manually resolved the conflict in Eclipse
• "git config --replace-all user.name 'My Users Name'"
• "git config --replace.all user.email myusersemailaddress"
• "git add -A"
• "git commit -m "myComment" --date "myDate" --author="My Users Name "
• "git status" returned "nothing to commit, working directory clean"
• I restarted the script
• Script output did NOT show "The following resources had uncommitted changes and were overwritten."

1 case that behaved strangely:

• Script output showed a conflict it could not recover from.
• "git status" showed "nothing to commit, working directory clean"
• the History view in Eclipse showed a changeset that was resumed, but this delivery had not been converted to a commit in the Git repo's history yet
• I manually resolved the conflict in Eclipse
• "git config --replace-all user.name 'My Users Name'"
• "git config --replace.all user.email myusersemailaddress"
• "git add -A"
• "git commit -m "myComment" --date "myDate" --author="My Users Name "
• "git status" returned "nothing to commit, working directory clean"
• The commit was not made as Git thinks there is nothing to commit.
• I restarted the script
• Script output DID show "The following resources had uncommitted changes and were overwritten."
  I can't figure out why Git thought there was nothing to commit in this case. Luckily this was after the branch point, so it did not create a problem when the branch was rebased. Was it user error and I was somehow running git status on the wrong branch?

I've asked about the changes being overwritten in https://jazz.net/forum/questions/198772/what-causes-the-message-the-following-resources-had-uncommitted-changes-and-were-overwritten-to-be-displayed

[WtfJoke]

"git config --replace-all user.name 'My Users Name'"
"git config --replace.all user.email myusersemailaddress"

These two are not really necessary but they wont hurt as well. :)
Usually I do a git add -A (mostly just "." instead of -A) and then copy the commit command from the output of the rtc2git (last executed command... appropriate git command). The same like you described.

Do I understand right, that until now you didnt figured out what caused to have some missing files? Something needs to make them disappear....

I still think the checkout/branch does revert them.... Could you test this somehow?
Neverthless I think im going to add a condition when resuming, that you need to have a clean branch. Probably I just call this method: https://github.com/WtfJoke/rtc2git/blob/develop/rtcFunctions.py#L212 (after renaming the method of course 😄 )

[WtfJoke]

Script output DID show "The following resources had uncommitted changes and were overwritten."

If you remove the --force (in load) or --overwrite-uncommited (in other commands like set component) you might be able to figure out why this message gets displayed

[ljhaywar]

I just reread through this entire thread and realized you asked me to run "git stash -list." I was able to run "git stash list" and I am seeing WIP listed. I initially added the "stash" command for when the script was switching branches. Since it's not doing that anymore, do you think I should remove it? Do you think that's causing problems?

Do I understand right, that until now you didnt figured out what caused to have some missing files?

I'm not sure what you mean? I never had missing files--some of my files were missing updates to their content.

[WtfJoke]

some of my files were missing updates to their content.

Yeah I meant that. Are those missing updates in one of your stashes?

Just due the fact, that the current version doesnt support multiple stream I think im going to delete that stash command before the checkout. User probably doesnt even realize something is stashed, I think its better that the checkout fails in this case.

[ljhaywar]

Unfortunately, I didn't keep good enough notes to remember which commit had the missing content, but it's very likely.

[WtfJoke]

I think im going to delete that stash command before the checkout

I think im going to add a condition when resuming, that you need to have a clean branch.

I implemented these two changes. Im pretty sure, with these there shouldnt be any loose changes or "missed" commits.

As usual, in case you hit this issue with those changes again, let me know and reopen this issue. :)