quantitative analysis of memory profiling data

[Sashan]

tickets #1149 and #1148 show a way how to obtain peak memory usage for each openssl application we use during make test. The make test potentially produces ~8k of data entries per run. The very basic idea is to store those results periodically (for example every day). Once there will be enough data collected (after a month or so) we want to plot those data into charts. However there are some challenges ahead: how to present 8k charts/tables in some meaningful dashboard?

Even if we will reduce the number of data collected/presented we still talk about 1k of charts worth to look at time to time. Perhaps we can define some kind of alert triggers like: if particular test increases its memory use since last build then the test should appear on a dashboard.

this feels like a job prometheus (https://prometheus.io/) (or something similar) can do. The idea is to set up unit test such memory peak usage is collected (#1149, #1148) If make test completes successfully then collect all result files in test-run directory and save them to prometheus DB. this is the first step.

The next step is to figure out how to create a dashboard which will show test which memory usage increased since last time. We need to keep in mind that if particular test (foo-12345) gets reported on Tuesday it won't get reported on Wednesday because its memory usage has not changed since Wednesday. I have no experience with creating such dashboards so will be happy if anyone else can step in.