Metric names in additional info are confusing

[nabenabe0928]

• I'm submitting a ...
  • bug report
  • feature request
  • support request => Please do not submit support request here, see note at the top of this template.

Issue Description

• When Issue Happens
  The additional_info in the BaseTask has xxx_loss key and it has various metrics.
  However, when the metric property is greater is better, then the metric is actually transformed.
  For example, accuracy is not accuracy, but is handled as error rate.
  The naming rule should be re-considered in the future.

[ravinkohli]

Definitely we can improve the naming rule. But the error rate is exactly the idea behind xxx_loss. Additional info contains information about the loss. And to keep it general, the loss is always greater_is_better=False. I do agree that it can be confusing for people and I am not sure how useful it is to tell the error rate for all the other supported metrics. I suggest we store only 'xxx_loss':{<optimize_metric>: <transformed_metric_value>}. Additionally, we can add the score (so raw metric values) for all_supported_metrics to the additional_info. But this is something that we have in common with auto-sklearn, so @eddiebergman how is this handled there?

[eddiebergman]

I don't think we do any renaming of metrics. We don't have anything in line with BaseTask so the only additional_info I'm aware of is from SMAC's runhistory items. We don't actually give much information back to the user about metrics at the moment though so I wouldn't look to us for guidance. Here's some pointer I can give you if it helps:

• Here's an example on how we report back scoring functions, needs a nicer interface for sure.
• You can use leaderboard as a reference for what we take out of runhistory. We report back the 'train_loss' and the 'cost', where cost is caluclated using greater_is_better and max values etc..
• Are metrics are defined here

[nabenabe0928]

I did not mean we need to completely rename them because it is painstaking in several ways.

1. Usage in other methods/functions
2. Need to check two names everytime (==> more if statements)
3. Vague which states the values are (==> new codes do not work easily although it is a good aspect in terms of safe coding)

I would recommend to put prefix or suffix so that we can easily judge which states those values are now.
For example, as suggested, transformed_accuracy is one example. (or accuracy_loss. This is not an intuitive naming though)

But somehow the transformation is performed by worst_val - metric_val, we cannot just say negative_accuracy, which we can refer if we took - metric_val.