burndown with granularity != sampling panics

[bobheadxi]

There doesn't seem to be anything in the documentation explicitly saying that granularity can't be different from burndown, so could this be a bug introduced by #245 ? I read through the burndown code and where it panics (when grouping histories), but can't seem to decipher what might be going on.

I'm using V10:

❯ hercules version
Version: 10
Git:     bf6626c0210a1446d6b494c0af25736c83bc01c0

• sampling != burndown, default tick size

hercules --sampling=7 --burndown https://github.com/bobheadxi/calories.git       
finalizing...2019/03/25 14:14:08 Failed to run the pipeline on [https://github.com/bobheadxi/calories.git]
panic: runtime error: index out of range

goroutine 1 [running]:
gopkg.in/src-d/hercules.v10/leaves.(*BurndownAnalysis).groupSparseHistory(0xc0001d6700, 0xc000b75410, 0xffffffffffffffff, 0x20300000000000, 0x65fffff, 0xc000da34c0, 0x4017a9a)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/leaves/burndown.go:1509 +0x681
gopkg.in/src-d/hercules.v10/leaves.(*BurndownAnalysis).Finalize(0xc0001d6700, 0x4d82580, 0xc000104600)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/leaves/burndown.go:504 +0x55
gopkg.in/src-d/hercules.v10/internal/core.(*Pipeline).Run(0xc000da3cc8, 0xc00048b400, 0x61, 0x80, 0x0, 0x0, 0x0)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/internal/core/pipeline.go:852 +0x6d1
main.glob..func3(0x57c9820, 0xc000a5f5c0, 0x1, 0x3)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/cmd/hercules/root.go:270 +0x865
gopkg.in/src-d/hercules.v10/vendor/github.com/spf13/cobra.(*Command).execute(0x57c9820, 0xc0000e6010, 0x3, 0x3, 0x57c9820, 0xc0000e6010)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/vendor/github.com/spf13/cobra/command.go:766 +0x2ae
gopkg.in/src-d/hercules.v10/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x57c9820, 0xc0000e99e0, 0xc0001bbf88, 0x400821f)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/vendor/github.com/spf13/cobra/command.go:852 +0x2c0
gopkg.in/src-d/hercules.v10/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v10/cmd/hercules/root.go:510 +0x32

• sampling == burndown works fine

❯ hercules --sampling=7 --granularity=7 --burndown https://github.com/bobheadxi/calories.git 
hercules:
  version: 10
  hash: bf6626c0210a1446d6b494c0af25736c83bc01c0
  repository: https://github.com/bobheadxi/calories.git
  begin_unix_time: 1509843192
  end_unix_time: 1522909723
  commits: 97
  run_time: 34
Burndown:
  granularity: 7
  sampling: 7
  tick_size: 24h0m0s
  "project": |-

• sampling == burndown, with custom tick size works fine as well

❯ hercules --tick-size=7 --sampling=7 --granularity=7 --burndown https://github.com/bobheadxi/calories.git
hercules:
  version: 10
  hash: bf6626c0210a1446d6b494c0af25736c83bc01c0
  repository: https://github.com/bobheadxi/calories.git
  begin_unix_time: 1509843192
  end_unix_time: 1522909723
  commits: 97
  run_time: 40
Burndown:
  granularity: 7
  sampling: 7
  tick_size: 7h0m0s
  "project": |-

[bobheadxi]

Just tried it with v9 and can confirm it was an issue back then, so this might not be related to #245 :

❯ hercules version
Version: 9
Git:     c8fd37d1bcc1163e829d20a8e818b36e4930dcc5

❯ hercules --sampling=7 --burndown https://github.com/bobheadxi/calories.git                
finalizing...2019/03/25 14:24:44 Failed to run the pipeline on [https://github.com/bobheadxi/calories.git]
panic: runtime error: index out of range

goroutine 1 [running]:
gopkg.in/src-d/hercules.v9/leaves.(*BurndownAnalysis).groupSparseHistory(0xc00104eb40, 0xc000f2e2d0, 0xffffffffffffffff, 0xc000def468, 0x40384d4, 0xc000000180, 0x200000003)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/leaves/burndown.go:1471 +0x67e
gopkg.in/src-d/hercules.v9/leaves.(*BurndownAnalysis).Finalize(0xc00104eb40, 0x4d7ae00, 0xc000f0bd10)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/leaves/burndown.go:489 +0x55
gopkg.in/src-d/hercules.v9/internal/core.(*Pipeline).Run(0xc000defcc8, 0xc0004a3000, 0x61, 0x80, 0x0, 0x0, 0x0)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/internal/core/pipeline.go:852 +0x6d1
main.glob..func3(0x57b7800, 0xc000a68b40, 0x1, 0x3)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/cmd/hercules/root.go:270 +0x865
gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra.(*Command).execute(0x57b7800, 0xc0000320d0, 0x3, 0x3, 0x57b7800, 0xc0000320d0)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra/command.go:766 +0x2ae
gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x57b7800, 0xc000a98350, 0xc000199f88, 0x400787f)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra/command.go:852 +0x2c0
gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/cmd/hercules/root.go:510 +0x32

[vmarkovtsev]

Sampling and burndown are not bound together and can be arbitrary. According to the stack trace, there is an invalid commit "from the future". Actually, groupSparseHistory is trivial compared to addBurndownMatrix so it could be much worse 😄

You should debug how this record "from the future" appeared.

[bobheadxi]

Hm, the same issue occurs with a burndown on Hercules itself on V9 of the CLI:

❯ hercules version
Version: 9
Git:     c8fd37d1bcc1163e829d20a8e818b36e4930dcc5

❯ hercules --sampling=7 --burndown https://github.com/src-d/hercules.git
finalizing...2019/03/25 14:36:50 Failed to run the pipeline on [https://github.com/src-d/hercules.git]
panic: runtime error: index out of range

goroutine 1 [running]:
gopkg.in/src-d/hercules.v9/leaves.(*BurndownAnalysis).groupSparseHistory(0xc00538ab40, 0xc00538ee40, 0xffffffffffffffff, 0xc00667f468, 0x40384d4, 0xc000000180, 0x200000003)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/leaves/burndown.go:1471 +0x67e
gopkg.in/src-d/hercules.v9/leaves.(*BurndownAnalysis).Finalize(0xc00538ab40, 0x4d7ae00, 0xc00538ab40)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/leaves/burndown.go:489 +0x55
gopkg.in/src-d/hercules.v9/internal/core.(*Pipeline).Run(0xc00667fcc8, 0xc0051f0000, 0x33a, 0x400, 0x0, 0x0, 0x0)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/internal/core/pipeline.go:852 +0x6d1
main.glob..func3(0x57b7800, 0xc000a55530, 0x1, 0x3)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/cmd/hercules/root.go:270 +0x865
gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra.(*Command).execute(0x57b7800, 0xc0000c4010, 0x3, 0x3, 0x57b7800, 0xc0000c4010)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra/command.go:766 +0x2ae
gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x57b7800, 0xc000079650, 0xc00017df88, 0x400787f)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra/command.go:852 +0x2c0
gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /Users/robertlin/go/src/gopkg.in/src-d/hercules.v9/cmd/hercules/root.go:510 +0x32

[vmarkovtsev]

Yep, crashes for me too. Any combination with granularity > sampling crashes. Can it be a feature, not a bug 🤔