Processing step fails on chstream transactions

[ghost]

The second stage, where it creates the branches, seems to fail a lot for us when processing chstream transactions that remove a timelock. I do not know what information to provide for this issue...

It looks like the script is trying to rebase the stream. Is there a reason it does not do a merge from the parent stream instead? Can the script differentiate between a timelock add/remove and reparenting a stream?

[orao]

The chstream code has been refactored to handle both mkstream and chstream transactions... It has been a while since I've done it but I remember that the code grew a bit unwieldy... You can see it here.

The removal of a timelock is a tricky one. If there have been no promotes into this stream, then we can "fast-forward" the child stream, otherwise we need to do a merge from the parent into the child stream. I think that the script tries to do this, but without more details about the error you're getting I'm not sure I can help.

What kind of failure is it? Is there a warning that is printed? Is it an exception? Or does the graph just look weird?

[ghost]

What kind of failure is it?

The script stops with an exception.

One error was:

2018-01-19 11:46:14,968 - ac2git - INFO - promote 163314. Cherry pick into stream1 2c0dc673. Source stream stream2 (id: 1180) is not tracked.
2018-01-19 11:46:15,188 - ac2git - DEBUG - Transaction #163315 - chstream by user1 to stream3 at 2015-01-06 08:44:25 local time (2015-01-06 13:44:25 UTC)
2018-01-19 11:46:24,214 - ac2git - ERROR - Failed to retrieve last git commit hash. Command git log -1 --format=format:%H stream3 failed.
2018-01-19 11:46:24,215 - ac2git - INFO - Running time was 22:53:35.23
2018-01-19 11:46:24,215 - ac2git - ERROR - The script has encountered an exception, aborting!
Traceback (most recent call last):
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 4075, in AccuRev2GitMain
rv = state.Start(isRestart=args.restart, isSoftRestart=args.softRestart)
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 3347, in Start
self.ProcessTransactions()
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 3123, in ProcessTransactions
self.ProcessTransaction(streamMap=state["stream_map"], trId=tr, affectedStreamMap=transactionsMap[tr], prevAffectedStreamMap=prevAffectedStreamMap)
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 2622, in ProcessTransaction
raise Exception("Error! Failed to get the last commit hash for branch {b} (stream: {s}), transaction {trType} {trId}!".format(trType=tr.Type, trId=tr.id, b=branchName, s=stream.name))
Exception: Error! Failed to get the last commit hash for branch stream3 (stream: stream3), transaction chstream 163315!

I increased the start transaction number to try to get past it and got this error:

2018-01-19 12:00:01,272 - ac2git - INFO - promote 163332. Cherry pick into stream4 1f3ada73. Source stream stream5 (id: 8) is not tracked.
2018-01-19 12:00:01,518 - ac2git - DEBUG - Transaction #163340 - promote by user1 to stream3 at 2015-01-06 09:39:28 local time (2015-01-06 14:39:28 UTC)
2018-01-19 12:00:10,547 - ac2git - ERROR - Failed to retrieve last git commit hash. Command git log -1 --format=format:%H stream3 failed.
2018-01-19 12:00:10,549 - ac2git - INFO - Running time was 0:00:44.61
2018-01-19 12:00:10,549 - ac2git - ERROR - The script has encountered an exception, aborting!
Traceback (most recent call last):
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 4075, in AccuRev2GitMain
rv = state.Start(isRestart=args.restart, isSoftRestart=args.softRestart)
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 3347, in Start
self.ProcessTransactions()
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 3123, in ProcessTransactions
self.ProcessTransaction(streamMap=state["stream_map"], trId=tr, affectedStreamMap=transactionsMap[tr], prevAffectedStreamMap=prevAffectedStreamMap)
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 2825, in ProcessTransaction
isAncestor = self.GitMergeBase(refs=[ lastSrcBranchHash, parents[0] ], isAncestor=True)
File "/opt/sandbox/Accurev2GitMigratrionScripts/ac2git/ac2git.py", line 2296, in GitMergeBase
assert None not in refs, "None is not an accepted value for a ref. Given refs are {refs}".format(refs=refs)
AssertionError: None is not an accepted value for a ref. Given refs are ['1f3ada732e733cb5d857e18b72ac221af620aad1', None]

Increasing the start transaction number by a couple weeks avoids the error.

[ghost]

The big problem now is that after much pain and frustration and pulling of hair, I think I have a complete conversion as of 2 weeks ago. But I try to run it again to get the changes in the last couple weeks and it won't update the branches with the latest changes. It downloads them, because I see the changes listed in the refs (e.x. by running git log refs/ac2git/depots/4/streams/2202/info or git log refs/ac2git/depots/4/streams/2202/data), but the latest changes are not in the branch nor in git log refs/ac2git/sate/depots/4/streams/2202/commit_history.

What might cause it to stop updating the branches?

The end transaction is set to "now", and I tried rerunning the script as I ran it the first time, and also with --soft-restart. I do not want to completely restart because it took over 2 weeks to download everything.

[ghost]

If I change the merge-strategy to orphanage and rerun the script (with --soft-restart), then it includes the most recent commits in the branches. I reran the script again with merge-strategy changed back to normal and the branches are missing their latest commits.

[orao]

The chstream transactions have been a bit complicated. A while back I've attempted to refactor the code for them in order to combine it with the mkstream transaction code, that looked much the same, but I fear that it had just made the whole thing more complicated.

The first exception you posted is triggered here. The line above is trying to get the last transaction on stream3 which has failed for some reason. It would be interesting to know if the command git log -1 --format=format:%H stream3 actually returns anything, but I suspect it won't.

Clearly my assumption is that this branch or commit would exist but for some reason that is wrong. From your message I can't really infer why that is the case but there is something you could do to try and get past this error. Instead of raising the exception simply set the parents list to be empty ([]). This should make a new orphaned branch instead of rebasing the existing branch and should allow the script to continue.

[orao]

For whatever reason it seems that you have some branch that either doesn't exist any more or something else is making the GetLastCommitHash() function to return None. Your second error too is pointing to the same issue, as the error you've posted is triggered on these lines which also look for an existing commit that can't be found. Identifying what branch it is trying to find and figuring out why it no longer exists would help us figure out what went wrong, but for that we need the logs from what the script did prior to logging the error. In their absence we can try and make the code paths that are affected try and ignore this error hoping for a good result...

[ghost]

Sorry, I was not clear that

Increasing the start transaction number by a couple weeks avoids the error

is an acceptable solution for us. I had picked an arbitrary start date far enough back to make sure that it grabs everything we need. Moving the start date forward by a couple weeks is going to be fine for us.

I probably should have created a new issue for the following:

I need to figure out why it is not updating branches with the latest changes when I run the script again, or at least find some kind of workaround. I see there is an option to run the script repeatedly to pick up new changes, but it is not working. The new transaction data gets downloaded (and can be found by running git log refs/ac2git/depots/4/streams/2202/info or git log refs/ac2git/depots/4/streams/2202/data), but it is not added to the existing branches.

[orao]

What is the last log message that it prints? Knowing the path the program took in the Start(), ProcessTransactions(), and RetrieveStreams() would be off interest... Even a general idea would help narrow down the possible cause.

[ghost]

When trying to update, there are no errors. The script looks like it runs fine, but does not add the additional commits to the branches.

[orao]

I didn't mean the errors, I meant the INFO lines that tell you what code path it is taking. If possible, it would be good to add some print statements to help diagnose why it isn't working. But first, did you see this message printed? Or this message? Or this message?

Are you running the script with the verbose and debug output?