feat: Add Incremental Flag to Proof Generation #2337
Conversation
felixjordandev
requested review from
ctrlc03,
0xmad and
NicoSerranoP
as code owners
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
@crisgarner This PR aims to fix #1895 |
There was a problem hiding this comment.
@felixjordandev thanks, left some comments and please fix all the jobs (they all must be green before merging).
|
@0xmad I have made the changes you requested and all the jobs are passing. Please merge if everything looks ok. Thanks |
There was a problem hiding this comment.
Thanks - however this does not solve the issue and will end up storing more proofs than needed as when running this.poll.tallyVotes() it will just start from zero. The logic would need to be applied to the Poll class so that it can:
- either skip the batch(or more) that is already processed and continue processing (but it would require a way to load the previous results)
- load the already used salts (which should be stored alongside proofs) and recalculate the proofs which will end up with the same commitments due to using the same salts (and not requiring to add a way to reload previous results)
Also it should work for both processMessages and tallyVotes
I implemented the second approach: loading the already used salts and recalculating the proofs. Please let me know if the changes look good to you. Thanks. |
|
|
||
| logMagenta({ text: info(`Progress: ${this.poll.numBatchesProcessed} / ${totalMessageBatches}`) }); | ||
| const batchIndex = this.poll.numBatchesProcessed; | ||
| const proofPath = path.join(this.outputDir, `process_${batchIndex}.json`); |
There was a problem hiding this comment.
| const proofPath = path.join(this.outputDir, `process_${batchIndex}.json`); | |
| const proofPath = path.resolve(this.outputDir, `process_${batchIndex}.json`); |
| const exists = fs.existsSync(proofPath); | ||
| if (exists) { | ||
| // Read file synchronously to avoid await in loop | ||
| const existingProof = JSON.parse(fs.readFileSync(proofPath, "utf8")) as Proof; |
There was a problem hiding this comment.
Still sync read here (should be async)
| const batchIndex = this.poll.numBatchesTallied; | ||
| const proofPath = path.join(this.outputDir, `tally_${batchIndex}.json`); | ||
|
|
||
| let shouldGenerateNewProof = true; | ||
|
|
||
| // Check if proof exists and incremental flag is set | ||
| if (this.incremental) { |
There was a problem hiding this comment.
This code is duplicated in proof generation for mp, makes sense to reuse it.
| const foundPoll = poll.polls.get(pollId); | ||
| if (!foundPoll) { |
There was a problem hiding this comment.
| const foundPoll = poll.polls.get(pollId); | |
| if (!foundPoll) { | |
| const foundPoll = poll.polls.get(pollId); | |
| if (!foundPoll) { |
| const statePath = path.join(this.outputDir, `poll_${pollId}_state.json`); | ||
| if (fs.existsSync(statePath)) { |
There was a problem hiding this comment.
| const statePath = path.join(this.outputDir, `poll_${pollId}_state.json`); | |
| if (fs.existsSync(statePath)) { | |
| const statePath = path.resolve(this.outputDir, `poll_${pollId}_state.json`); | |
| if (fs.existsSync(statePath)) { |
| @@ -49,6 +49,8 @@ import type { PathElements } from "@maci-protocol/crypto"; | |||
| import { STATE_TREE_ARITY, VOTE_OPTION_TREE_ARITY } from "./utils/constants"; | |||
| import { ProcessMessageErrors, ProcessMessageError } from "./utils/errors"; | |||
|
|
|||
| import fs from "fs"; | |||
There was a problem hiding this comment.
Poll shouldn't contain this module
| throw new ProcessMessageError(ProcessMessageErrors.InsufficientVoiceCredits); | ||
| } | ||
| // Calculate vote start and end indices | ||
| const voteStartIndex = 2; |
| * Create circuit input for poll results | ||
| * @returns The circuit input |
There was a problem hiding this comment.
There is no poll results circuit.
| * Create circuit input for poll state | ||
| * @returns The circuit input |
There was a problem hiding this comment.
There is no poll state circuit
| // (In a real implementation, we would retrieve the stored circuit inputs) | ||
| return {} as IProcessMessagesCircuitInputs; |
There was a problem hiding this comment.
This can't be merged like this
Prevents regeneration of existing proofs by adding an --incremental flag that reuses existing proofs when available, avoiding commitment mismatches due to random salts.