matrix-rust-sdk/benchmarks/README.md

Benchmarks for the rust-sdk crypto layer

This directory contains various benchmarks that test critical functionality in the crypto layer in the rust-sdk.

We're using Criterion for the benchmarks, the full documentation for Criterion can be found here.

Running the benchmarks

The benchmark can be simply run by using the bench command of cargo:

$ cargo bench

This will work from the workspace directory of the rust-sdk.

If you want to pass options to the benchmark you'll need to specify the name of the benchmark:

$ cargo bench --bench crypto_bench -- # Your options go here

If you want to run only a specific benchmark, simply pass the name of the benchmark as an argument:

$ cargo bench --bench crypto_bench "Room key sharing/"

After the benchmarks are done, a HTML report can be found in target/criterion/report/index.html.

Using a baseline for the benchmark

The benchmarks will by default compare the results to the previous run of the benchmark. If you are improving the performance of a specific feature and run the benchmark many times, it may be useful to store a baseline to compare against instead.

The --save-baseline switch can be used to create a baseline for the benchmark.

$ cargo bench --bench crypto_bench -- --save-baseline libolm

After you make your changes you can use the baseline to compare the results like so:

$ cargo bench --bench crypto_bench -- --baseline libolm

Generating Flame Graphs for the benchmarks

The benchmarks support profiling and generating Flame Graphs while they run in profiling mode using pprof.

Profiling usually requires root permissions, to avoid the need for root permissions you can adjust the value of perf_event_paranoid, e.g. the most permisive value is -1:

$ echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid

To generate flame graphs feature simply enable the profiling mode using the --profile-time command line flag:

$ cargo bench --bench crypto_bench -- --profile-time=5

After the benchmarks are done, a flame graph for each individual benchmark can be found in target/criterion/<name-of-benchmark>/profile/flamegraph.svg.