Open
Description
Evaluator objects shouldn't be reused- once we've evaluated a checkpoint or model, we want to save the score and move on to the next. This motivates a reasonable design change, implementing something like:
class MMLUEvaluator(Evaluator):
def __init__(self):
# optional empty initialization
...
def run(self):
...
@staticmethod
def run(self, model, tasks, few_shot, batch):
...