libmprofile simple memory profiler for openssl

[Sashan]

unlike libtcmalloc which comes with its own memory allocator, thelibmprofile uses CRYPTO_set_mem_functions(3ossl) to override CRYPTO_malloc(), CRYPTO_realloc() and CRYPTO_free() used in OpenSSL. After creating a profiling record the libmprofile dispatches the call further to libc to allocate/free memory. The libmprofile is not suitable for profiling long-time running processes because of massive overhead to keep profiling data. The libmprofile saves data to file when application exits. Those records are kept in heap memory allocated by libc.

The libmprofile currently lives here: https://github.com/sashan/mprofile. There is also pull request which adds libmprofile to openssl's make test: openssl/openssl#27139 Because the library uses CRYPTO_set_mem_functions(3ossl) we must also deal with static versions of libcrypto/libssl which both are used by many test tools. The preliminary plan is to make libmprofile part of libtestutil so the tests which use static libcrypto/libssl find libmprofile in libtestutil. This way we minimize the change to existing sources/build infrastructure. However it's not ideal the proper way is to make sure the test which use static libcrypto/libssl link with static version of libmprofile. The tests which use dynamic version of libcrypto/libssl should rely on LD_PRELOAD like we do for libtcmalloc (#1148). The current way in PR is less than ideal because test binaries which link dynamically with libcrypto/libssl get two copies of libmprofile: once comes via LD_PRELOAD, the other via libtestutil.

By default libmprofile just calculates max memory use by libcrypto/libssl in profiled application. This comes with zero memory overhead. It is safe to use for any process. The report is written when profiled application exits. The current .json form reads as follows:

{
        "annotation" : "../../util/wrap.pl ../../test/lhash_test",
        "total_allocated_sz" : 1053240143,
        "total_released_sz" : 1053240143,
        "allocs" : 30835940,
        "releases" : 30835940,
        "reallocs" : 44,
        "max" : 257292503,
        "tstart" : {
                "sec" : 1743008818,
                "nsec" : 758328505
        },
        "tfinish" : {
                "sec" : 1743008886,
                "nsec" : 543549332
        }
}

To obtain data above one can run lhash_test as follows:

export LD_LIBRARY_PATH=`pwd`
export LD_PRELOAD=path/to/libmprofile.so
export MPROFILE_OUTF=lhash.json
test/lhash_test

Building libmprofile should be straigtforward. Its Makefile assumes gcc tools. Editing the makefile and commentin/uncommenting lines it can be adjusted to clang. The call stack collection is facilitated by libraries which come with compiler. To profile simple application which does OPENSSL_malloc() followed by OPENSSL_free() one does:

export MPROFILE_OUTF=main.json
export MPROFILE_MODE=5
LD_PRELOAD=/path/to/libmprofile.so ./main

the main.json file then reads as follows:

{ "start_time" : {
        "s" : 1743501339,
        "ns" : 535086498
  },
        "annotation" : "FooBar annotation",
  "allocations" : [
        {
                "id" : 1,
                "addr" : 8095174935824,
                "realloc" : 0,
                "delta_sz" : 256,
                "state" : "allocated",
                "next_id" : 2,
                "prev_id" : 0,
                "stack_id" : 1,
                "time" : {
                        "s" : 1743501339,
                        "ns" : 535578591
                }
        },
        {
                "id" : 2,
                "addr" : 8095174935824,
                "realloc" : 0,
                "delta_sz" : -256,
                "state" : "free",
                "next_id" : 0,
                "prev_id" : 1,
                "stack_id" : 2,
                "time" : {
                        "s" : 1743501339,
                        "ns" : 535659365
                }
        }
],
 "stacks" : [
        {
                "id" : 2,
                "stack_count" : 1,
                "thread_id" : 0,
                "stack_trace" : [ "collect_backtrace+0x28", "mp_CRYPTO_free_trace_with_stack+0x5c", "main+0x5c", "CRYPTO_free+0x183b", "" ]
        },
        {
                "id" : 1,
                "stack_count" : 1,
                "thread_id" : 0,
                "stack_trace" : [ "collect_backtrace+0x28", "mp_CRYPTO_malloc_trace_with_stack+0xc1", "CRYPTO_malloc+0x42", "main+0x3d", "CRYPTO_free+0x183b", "" ]
        }
]

The magic MPROFILE_MODE is eplained in README