XProf

XProf is a visual tracer and profiler that allows you to track execution of Elixir / Erlang functions in real-time.

Goal

XProf was originally created to help solving performance problems of live, highly concurrent and utilized back-end systems. It's often the case that high latency or big CPU usage is caused by very specific requests that are triggering inefficient code. Finding this code is usually pretty difficult.

In this original usage one would first inspect execution time statistics to get an overview of the system. Then capture arguments and results (return value or exception) of function calls that lasted longer than given number of milliseconds.

With the introduction of xprof-commands via an extened query syntax in 2.0 more versatile stats, filters and other features became available.

How does it look like

Click the image below to watch a short demo investigating the TryMe application with XProf. The function nap sometimes takes way too much time (as you would guess from the name, it takes a bit of sleep). In the video we:

  • observe call count and duration percentiles
  • capture arguments and return values of a few long calls
  • apply a match spec to filter out "long" calls
  • compare two functions

XProf Demo

How to use it

  1. Add xprof to your build tool config file (and optionally also to the release config file such as reltool.config in order to include it in your release).
  2. Build your project.
  3. Start xprof by executing xprof:start(). in Erlang shell, or :xprof.start in Elixir shell.
  4. Go to http://localhost:7890 (replace localhost with your server’s hostname if you connect to a remote host).
  5. Type in function that you would like to start tracing.
  6. Start tracing clicking green button.

The preferred way is to add the xprof Hex package as a dependency to you rebar3 config or Mix project file:

%% rebar.config (at least version `3.3.3` is required):

{deps, [
       ...
       {xprof, "2.0.0-rc.4"}
]}.
# `mix.exs`:

defp deps do
    [
      ...
      {:xprof, "~> 2.0.0-rc.4"}
    ]
  end

You can also fetch from the github repository (not recommended, only for development): (TODO: finalize below example, needs raw and rsync resouce plugins)

{deps, [
       ...
       {xprof_umbrella, {raw, {git, "https://github.com/appliscale/xprof.git", {tag, "2.0.0-rc.4"}}}},
       {xprof_core, {rsync, "_build/default/lib/xprof_umbrella/apps/xprof_core"}},
       {xprof_gui, {rsync, "_build/default/lib/xprof_umbrella/apps/xprof_gui"}},
       {xprof, {rsync, "_build/default/lib/xprof_umbrella/apps/xprof"}},
]}.

Supported Versions

XProf currently supports Erlang/OTP R16B - 20. Newer OTP versions might work but are not tested.

Syntax mode

XProf supports both Erlang and Elixir syntax. If the elixir application is running it will use Elixir syntax and Erlang syntax otherwise to read the function to trace and to print captured arguments. It is also possible to manually set the preferred mode.

XProf flavoured match-spec funs

In the function browser apart from simple module-function-arity you can also specify further filters in the form of a match-spec functions (similar to recon or redbug) as well as xprof commands with options. For details see the page on Query syntax

Recursive functions

By default XProf only measures the outermost call to a recursive function. For example lists:map(fun f/1, [1, 2, 3]). will only register one call to lists:map/2. This is also true for indirectly recursive functions (such as when a calls b and b calls a again). This behaviour can be undesireable so it can be disabled by setting the ignore_recursion environment variable to false.

Erlang records

Erlang record syntax is supported in the queries and works similar to the Erlang shell. XProf keeps a single global list of loaded record definitions. Record definitions can be loaded at startup time from modules listed in app env load_records or at runtime calling xprof_core:rr(Module) (see documentation of xprof_core for more details). The record definitions are extracted from debug_info of the beam files belonging to the loaded modules. As the list is global there can be only one record with the same name loaded at a time and records loaded later might override previously loaded ones.

Configuration

You can configure XProf by changing the below application variables:

ApplicationKeyDefaultDescription
xprof_guiipanyListen address of the web interface (in tuple format, see inet:ip_address())
xprof_guiport7890Port for the web interface
xprof_coremax_tracer_queue_len1000Overflow protection. If main tracer proccess will have more than 1000 messages in its process queue tracing will be stopped and one needs to use trace button to resume. The purpose of this is to prevent out of memory crashes when tracer process is not able to process incomming traces fast enough. This may happen when we trace very "hot" function.
xprof_coremax_duration30000The largest duration value in ms. In case a call takes even longer, this maximum value is stored instead.
xprof_coreignore_recursiontrueWhether to only measure the outermost call to a recursive function or not (ie. measure all calls).
xprof_coremodeSyntax mode (erlang or elixir)
xprof_coreload_records[]List of modules from which to load record definitions at startup.

Compile-time configuration

XPROF_ERL_HIST - By default XProf uses the hdr_histogram_erl NIF library. If you have compilation problems you can choose to use a native Erlang histogram implementation by defining the OS env var XPROF_ERL_HIST when compiling xprof_core.

COWBOY_VERSION - By default XProf uses Cowboy version 2.x. This version is only supported from Erlang/OTP 19 and is not backwards compatible with older Cowboy versions. If for some reason you would like to use Cowboy version 1.x you can define the OS env var COWBOY_VERSION=1 when compiling xprof_gui.

Web Interface

XProf's web interface supports a lot of small but convenient features as query autocomplition, list of called functions, collapsing graphs, multiple graphs in a row and so on.

Keyboard shortcuts

  • UP/DOWN arrows (cursor in query box): scroll through recent queries
  • UP/DOWN arrows (cursor in suggestion list below query box): scroll through auto-completion suggestions
  • TAB: if no suggetion is selected yet auto-complete to longest common prefix of dropdown list items. Otherwise copy the selected item to the search box and refresh the dropdown list.
  • ENTER: start tracing either the selected suggestion if there is any or the expression in the search box.

Contributing

All improvements, fixes and ideas are very welcomed!

Project uses rebar3 for building and testing Erlang code. WebUI part resides in xprof_gui app's priv directory and it's already precompiled so there is no need to build JS sources in order to run xprof.

Running tests

make test

Working with JS sources

The WebUI uses

  • React.js
  • ECMAScript 6 (with elements from 7th version).
  • Bootstrap
  • Webpack

All resources are in apps/xprof_gui/priv directory. The src folder contains the sources and the build folder is a placeholder for final JS generated by webpack and then served by cowboy server (XProf's dependency).

Starting XProf in development mode

To develop xprof in a convenient way the following setup is recommended.

You have to invoke following command once, if you do not have dependencies or you need to update them:

$ make bootstrap_front_end

Then going with normal development flow - in the first terminal window start Erlang xprof. The sync app will be started, It automatically reloads erlang modules that have changed, so you don't need to recompile every time something changed.

$ make dev_back_end

In the second window install all the assets and start webpack in development mode which is also going to recompile all JS files into apps/xprof_gui/priv/build-dev directory when they are modified. To achieve that use following command:

$ name dev_front_end