use the repo tool to sync repository leads to inability to create history

[jwehler]

Describe the bug
Running OG 1.7.7 in Docker (and same in 1.7.8 verified just now), one of my git repos under a project fails to render any file that I click on with the following dump:

There was an error!
Walk failure.

org.eclipse.jgit.errors.RevWalkException: Walk failure.
	at org.eclipse.jgit.revwalk.RevWalk.nextForIterator(RevWalk.java:1380)
	at org.eclipse.jgit.revwalk.RevWalk.iterator(RevWalk.java:1402)
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:520)
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:461)
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:456)
	at org.opengrok.indexer.history.FileHistoryCache.get(FileHistoryCache.java:608)
	at org.opengrok.indexer.history.HistoryGuru.getHistory(HistoryGuru.java:264)
	at org.opengrok.web.PageConfig.getLastRevFromHistory(PageConfig.java:1296)
	at org.opengrok.web.PageConfig.getLatestRevision(PageConfig.java:1288)
	at org.apache.jsp.list_jsp._jspService(list_jsp.java:286)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:71)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:467)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:223)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
	at org.opengrok.web.CookieFilter.doFilter(CookieFilter.java:50)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
	at org.opengrok.web.AuthorizationFilter.doFilter(AuthorizationFilter.java:95)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
	at org.opengrok.web.StatisticsFilter.doFilter(StatisticsFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:353)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:870)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1696)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing commit d803f74934a46848b448b71420dbe21a9cc39a67
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:961)
	at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:126)
	at org.eclipse.jgit.revwalk.TreeRevFilter.include(TreeRevFilter.java:113)
	at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:108)
	at org.eclipse.jgit.revwalk.BlockRevQueue.<init>(BlockRevQueue.java:40)
	at org.eclipse.jgit.revwalk.FIFORevQueue.<init>(FIFORevQueue.java:37)
	at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:133)
	at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:443)
	at org.eclipse.jgit.revwalk.RevWalk.nextForIterator(RevWalk.java:1378)
	... 45 more

I don't see the referenced missing commit in the project at all (via git show ), but I also don't see any obvious historical issue running local git log commands, etc, in that git project. This problem opengrok project has about a dozen git projects under it and two of the git repos have this error (unique missing commit hash in the two separate projects of course, but same error signature). A separate similarly indexed project that happens to point at another branch in the same git repo is fine.

[vladak]

Could you be more specific about the project/repository layout and how the error happens w.r.t. the layout ? I assume there are nested repositories.

I wonder if there are similar errors in the indexer log. (when creating the history cache)

[jwehler]

Certainly. This is a project that is populated using Google's repo tool to manage a collection of git repositories (14 at the moment). The layout is like what you'd see with an Android-like setup, just path/to/some/git, path/to/another/git, etc.

When viewing in OG webapp, I see top-level project name, then the list of sub git projects under that. 2 of those 14 encounter this issue when browsing the files shown. I can see the all the files listed, but if you click on any file to view it I'll get the error described. You are correct in assuming the indexer is showing similar problems. I dumped the log from docker (over 15MB of text due to exceptions) and do see a similar format of failure:

Jun 04, 2021 3:58:56 PM org.opengrok.indexer.history.HistoryGuru createCache
WARNING: An error occurred while creating cache for /opengrok/src/<myproject/path/to/some/git> (GitRepository)
org.eclipse.jgit.errors.RevWalkException: Walk failure.
	at org.eclipse.jgit.revwalk.RevWalk.nextForIterator(RevWalk.java:1380)
	at org.eclipse.jgit.revwalk.RevWalk.iterator(RevWalk.java:1402)
	at org.opengrok.indexer.history.GitRepository.accept(GitRepository.java:477)
	at org.opengrok.indexer.history.BoundaryChangesets.getBoundaryChangesetIDs(BoundaryChangesets.java:73)
	at org.opengrok.indexer.history.RepositoryWithPerPartesHistory.doCreateCache(RepositoryWithPerPartesHistory.java:77)
	at org.opengrok.indexer.history.Repository.createCache(Repository.java:385)
	at org.opengrok.indexer.history.HistoryGuru.createCache(HistoryGuru.java:548)
	at org.opengrok.indexer.history.HistoryGuru.lambda$createCacheReal$3(HistoryGuru.java:595)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing commit d803f74934a46848b448b71420dbe21a9cc39a67
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:961)
	at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:126)
	at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:121)
	at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:167)
	at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:443)
	at org.eclipse.jgit.revwalk.RevWalk.nextForIterator(RevWalk.java:1378)
	... 12 more

and many more of the format:

Jun 04, 2021 3:59:24 PM org.opengrok.indexer.analysis.AnalyzerGuru populateDocument
WARNING: An error occurred while reading history: 
org.opengrok.indexer.history.HistoryException: failed to get history for ''/opengrok/src/<myproject/path/to/some/git/apath/to/a/file>''
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:540)
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:461)
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:456)
	at org.opengrok.indexer.history.FileHistoryCache.get(FileHistoryCache.java:608)
	at org.opengrok.indexer.history.HistoryGuru.getHistory(HistoryGuru.java:264)
	at org.opengrok.indexer.history.HistoryGuru.getHistory(HistoryGuru.java:224)
	at org.opengrok.indexer.history.HistoryGuru.getHistoryReader(HistoryGuru.java:208)
	at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:594)
	at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:821)
	at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$4(IndexDatabase.java:1352)
	at java.base/java.util.stream.Collectors.lambda$groupingByConcurrent$59(Collectors.java:1297)
	at java.base/java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:575)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
	at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 6c4f56b01364093406700362c12c18aaf3f74ad4
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:135)
	at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
	at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:917)
	at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:827)
	at org.opengrok.indexer.history.GitRepository.getHistory(GitRepository.java:502)
	... 21 more

I don't see anything labeled SEVERE or ERROR, all just WARNING class. Though possible when I dumped the log to a file last week some output was stderr and didn't go to the log I suppose.

[vladak]

Looks like even simple revision walk (in GitRepository.accept()) hits the problem. The error in populateDocument() is the same thing, just delayed to the 2nd indexing phase (due to #747).

I wonder what is special about these 2 repositories.

[jwehler]

It is unclear to me what is unique . I have the same 2 git projects indexed successfully on a different branch. The one that fails has content from the other successful one merged into it at well. Feels like some sort of merge commit type issue, maybe when walking history it takes the wrong path parent-wise? Just reaching here...

Are there some equivalent git commands I can run locally to see if I can reproduce the output? I the jgit revwalk should be handed commits to look at...can I generate a similar data set locally to see if I find these unknown commits in my list?

[vladak]

Perhaps this is not so much about the project/repository structure rather than what is inside these repositories. What happens if you copy these failing 2 repositories and make them separate projects ? Will that fail too ?

[jwehler]

Ok, I think I understand the trigger. First, to answer your questions.

1. I copied the source separately, removed the .git and indexed. Works fine, so content itself is ok.
2. I cloned just the problem project(s) and indexed them. Also works fine. This should have been the same type of setup and triggered the problem if it was purely the git content.

That means it is repo tool related, which then it dawned on me that for these instances of the code, I had enabled a local repo mirror. The pattern for creating that mirror:

repo init <normal options> --mirror
repo sync

Then using it:

repo init <normal options> --reference=/path/to/mirror
repo sync -c

The problem projects, since they were the most recently added ones, had started using my newly setup mirror. It provides significant space saving and sync time improvements, given I was just looking at a different branch of the same codebase. I did this all the time prior to using Docker (along much older versions of OG) in the distant past.

I synced a new version of the problem project, this time without usage of the mirror. While OG was still running, I swapped the mirror and non-mirror versions of the same codebase under the hood in the opengrok/src dir. Now I can browse the files and the content shows up properly. My second problem project was setup the same way, I fixed that one as well. I'm reindexing now, but I am quite confident all will work as expected.

Long story short, something with the repo mirror is messing with the way the git history is determined with jgit.

[vladak]

There is some remote chance that the underlying cause is similar to #3666, I mean in the way how the Repository JGit object is created.