region_profiler package

Submodules

region_profiler.chrome_trace_listener module

class region_profiler.chrome_trace_listener.ChromeTraceListener(trace_filename)[source]

Bases: region_profiler.listener.RegionProfilerListener

This listener produces a log, suitable for Chrome Trace Viewer.

Learn more about Chrome Trace Viewer.

finalize()[source]

Hook ‘Finish profiling’ event.

region_canceled(profiler, region)[source]

Hook ‘Cancel region’ event.

Parameters:
region_entered(profiler, region)[source]

Hook ‘Enter region’ event.

Parameters:
region_exited(profiler, region)[source]

Hook ‘Exit region’ event.

Parameters:

region_profiler.debug_listener module

class region_profiler.debug_listener.DebugListener[source]

Bases: region_profiler.listener.RegionProfilerListener

Log profiler events to console.

This listener log enter/exit events in real time. Sample output:

RegionProfiler: Entered <main> at 0 ns
RegionProfiler: Entered fetch_mnist() at 641 us
RegionProfiler: Exited fetch_mnist() at 643 ms after 642 ms
RegionProfiler: Entered train at 743 ms
RegionProfiler: Entered fetch_next at 743 ms
RegionProfiler: Exited fetch_next at 744 ms after 1.183 ms
RegionProfiler: Entered forward at 744 ms
RegionProfiler: Entered loss_fn() at 744 ms
RegionProfiler: Entered NN at 745 ms
RegionProfiler: Exited NN at 764 ms after 19.40 ms
RegionProfiler: Exited loss_fn() at 765 ms after 20.55 ms
RegionProfiler: Exited <main> at 1.066 s after 1.066 s
RegionProfiler: Finalizing profiler
finalize()[source]

Log ‘Finish profiling’ event.

region_canceled(profiler, region)[source]

Log ‘Exit region’ event.

region_entered(profiler, region)[source]

Log ‘Enter region’ event.

region_exited(profiler, region)[source]

Log ‘Exit region’ event.

region_profiler.global_instance module

region_profiler.global_instance.func(name=None, asglobal=False)[source]

Decorator for entering region on a function call.

Examples:

@rp.func()
def foo():
    ...
Parameters:
  • name (str, optional) – region name. If None, the name is deducted from region location in source
  • asglobal (bool) – enter the region from root context, not a current one. May be used to merge stats from different call paths
Returns:

a decorator for wrapping a function

Return type:

Callable

region_profiler.global_instance.install(reporter=<region_profiler.reporters.ConsoleReporter object>, chrome_trace_file=None, debug_mode=False, timer_cls=None)[source]

Enable profiling.

Initialize a global profiler with user arguments and register its finalization at application exit.

Parameters:
region_profiler.global_instance.iter_proxy(iterable, name=None, asglobal=False)[source]

Wraps an iterable and profiles next() calls on this iterable.

This proxy may be useful, when the iterable is some data loader, that performs data retrieval on each iteration. For instance, it may pull data from an asynchronous process.

Such proxy was used to identify that when receiving a batch of 8 samples from a loader process, first 5 samples were loaded immediately (because they were computed asynchronously during the loop body), but then it stalled on the last 3 iterations meaning that loading had bigger latency than the loop body.

Examples:

for batch in rp.iter_proxy(loader):
    ...
Parameters:
  • iterable (Iterable) – an iterable to be wrapped
  • name (str, optional) – region name. If None, the name is deducted from region location in source
  • asglobal (bool) – enter the region from root context, not a current one. May be used to merge stats from different call paths
Returns:

an iterable, that yield same data as the passed one

Return type:

Iterable

region_profiler.global_instance.region(name=None, asglobal=False)[source]

Start new region in the current context.

This function implements context manager interface. When used with with statement, it enters a region with the specified name in the current context on invocation and leaves it on with block exit.

Examples:

with rp.region('A'):
    ...
Parameters:
  • name (str, optional) – region name. If None, the name is deducted from region location in source
  • asglobal (bool) – enter the region from root context, not a current one. May be used to merge stats from different call paths
Returns:

node of the region.

Return type:

region_profiler.node.RegionNode

region_profiler.listener module

class region_profiler.listener.RegionProfilerListener[source]

Bases: object

Base class for profiler listeners, that can augment profiler functionality.

Profiler listeners can execute custom hooks on the following events:

  • Enter region
  • Exit region
  • Cancel region
  • Finish profiling
finalize()[source]

Hook ‘Finish profiling’ event.

region_canceled(profiler, region)[source]

Hook ‘Cancel region’ event.

Parameters:
region_entered(profiler, region)[source]

Hook ‘Enter region’ event.

Parameters:
region_exited(profiler, region)[source]

Hook ‘Exit region’ event.

Parameters:

region_profiler.node module

class region_profiler.node.RegionNode(name, timer_cls=<class 'region_profiler.utils.Timer'>)[source]

Bases: object

RegionNode represents a single entry in a region tree.

It contains a builtin timer for measuring the time, spent in the corresponding region. It keeps track of the count, sum, max and min measurements. These statistics can be accessed through stats attribute.

In addition, RegionNode has a dictionary of its children. They are expected to be retrieved using get_child(), that also creates a new child if necessary.

name

Node name.

Type:str
stats

Measurement statistics.

Type:SeqStats
cancel_region()[source]

Cancel current region timing.

Stats will not be updated with the current measurement.

enter_region()[source]

Start timing current region.

exit_region()[source]

Stop current timing and update stats with the current measurement.

get_child(name, timer_cls=None)[source]

Get node child with the given name.

This method creates a new child and stores it in the inner dictionary if it has not been created yet.

Parameters:
  • name (str) – child name
  • timer_cls (class, optional) – override child timer class
Returns:

new or existing child node with the given name

Return type:

RegionNode

timer_is_active()[source]

Return True if timer is currently running.

class region_profiler.node.RootNode(name='<root>', timer_cls=<class 'region_profiler.utils.Timer'>)[source]

Bases: region_profiler.node.RegionNode

An instance of RootNode is intended to be used as the root of a region node hierarchy.

RootNode differs from RegionNode by making RegionNode.stats property returning the current measurement values instead of the real stats of previous measurements.

cancel_region()[source]

Prevents root region from being cancelled.

exit_region()[source]

Instead of RegionNode.exit_region() it does not reset timer attribute thus allowing it to continue timing on reenter.

region_profiler.profiler module

class region_profiler.profiler.RegionProfiler(timer_cls=None, listeners=None)[source]

Bases: object

RegionProfiler handles code regions profiling.

This is the central class in region_profiler package. It is responsible for maintaining the hierarchy of timed regions as well as providing facilities for marking regions for timing using

Normally it is expected that the global instance of RegionProfiler is used for the profiling, see package-level function region_profiler.install(), region_profiler.region(), region_profiler.func(), and region_profiler.iter_proxy().

ROOT_NODE_NAME = '<main>'
current_node

Return current region node.

Returns:node of the region as defined above
Return type:region_profiler.node.RegionNode
finalize()[source]

Perform profiler finalization on application shutdown. Finalize all associated listeners.

func(name=None, asglobal=False)[source]

Decorator for entering region on a function call.

Examples:

@rp.func()
def foo():
    ...
Parameters:
  • name (str, optional) – region name. If None, the name is deducted from region location in source
  • asglobal (bool) – enter the region from root context, not a current one. May be used to merge stats from different call paths
Returns:

a decorator for wrapping a function

Return type:

Callable

iter_proxy(iterable, name=None, asglobal=False, indirect_call_depth=0)[source]

Wraps an iterable and profiles next() calls on this iterable.

This proxy may be useful, when the iterable is some data loader, that performs data retrieval on each iteration. For instance, it may pull data from an asynchronous process.

Such proxy was used to identify that when receiving a batch of 8 samples from a loader process, first 5 samples were loaded immediately (because they were computed asynchronously during the loop body), but then it stalled on the last 3 iterations meaning that loading had bigger latency than the loop body.

Examples:

for batch in rp.iter_proxy(loader):
    ...
Parameters:
  • iterable (Iterable) – an iterable to be wrapped
  • name (str, optional) – region name. If None, the name is deducted from region location in source
  • asglobal (bool) – enter the region from root context, not a current one. May be used to merge stats from different call paths
  • indirect_call_depth (int, optional) – adjust call depth to correctly identify the callsite position for automatic naming
Returns:

an iterable, that yield same data as the passed one

Return type:

Iterable

region(name=None, asglobal=False, indirect_call_depth=0)[source]

Start new region in the current context.

This function implements context manager interface. When used with with statement, it enters a region with the specified name in the current context on invocation and leaves it on with block exit.

Examples:

with rp.region('A'):
    ...
Parameters:
  • name (str, optional) – region name. If None, the name is deducted from region location in source
  • asglobal (bool) – enter the region from root context, not a current one. May be used to merge stats from different call paths
  • indirect_call_depth (int, optional) – adjust call depth to correctly identify the callsite position for automatic naming
Returns:

node of the region.

Return type:

region_profiler.node.RegionNode

region_profiler.reporter_columns module

Define columns that may be used for configuring reporters (see region_profiler.reporters).

Each column is defined as a function, that takes a current region_profiler.reporters.Slice and a list of all slices and returns the requested metrics of the current slice.

Each column stores its name in column_name attribute.

region_profiler.reporter_columns.as_column(print_name=None, name=None)[source]

Mark a function as a column provider.

Parameters:
  • print_name (str, optional) – column name without underscores. If None, name with underscores replaced is used
  • name (str, optional) – column name. If None, function name is used
region_profiler.reporter_columns.average(this_slice, all_slices)[source]

Column provider. Retrieves average.

region_profiler.reporter_columns.average_us(this_slice, all_slices)[source]

Column provider. Retrieves average us.

region_profiler.reporter_columns.count(this_slice, all_slices)[source]

Column provider. Retrieves count.

region_profiler.reporter_columns.indented_name(this_slice, all_slices)[source]

Column provider. Retrieves name.

region_profiler.reporter_columns.max(this_slice, all_slices)[source]

Column provider. Retrieves max.

region_profiler.reporter_columns.max_us(this_slice, all_slices)[source]

Column provider. Retrieves max us.

region_profiler.reporter_columns.min(this_slice, all_slices)[source]

Column provider. Retrieves min.

region_profiler.reporter_columns.min_us(this_slice, all_slices)[source]

Column provider. Retrieves min us.

region_profiler.reporter_columns.name(this_slice, all_slices)[source]

Column provider. Retrieves name.

region_profiler.reporter_columns.node_id(this_slice, all_slices)[source]

Column provider. Retrieves id.

region_profiler.reporter_columns.parent_id(this_slice, all_slices)[source]

Column provider. Retrieves parent id.

region_profiler.reporter_columns.parent_name(this_slice, all_slices)[source]

Column provider. Retrieves parent name.

region_profiler.reporter_columns.percents_of_total(this_slice, all_slices)[source]

Column provider. Retrieves % of total.

region_profiler.reporter_columns.total(this_slice, all_slices)[source]

Column provider. Retrieves total.

region_profiler.reporter_columns.total_inner(this_slice, all_slices)[source]

Column provider. Retrieves total inner.

region_profiler.reporter_columns.total_inner_us(this_slice, all_slices)[source]

Column provider. Retrieves total inner us.

region_profiler.reporter_columns.total_us(this_slice, all_slices)[source]

Column provider. Retrieves total us.

region_profiler.reporters module

class region_profiler.reporters.ConsoleReporter(columns=(<function indented_name>, <function total>, <function percents_of_total>, <function count>, <function min>, <function average>, <function max>), stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)[source]

Bases: object

Print profiler state in a human-readable way to console.

Data is printed in a table with a configurable set of columns (default columns: name, total duration, % of total, hit count, min, average, max).

Nodes are printed in a depth-first order with siblings processed sorted by the total time descending.

By default, these column are reported:

  • name
  • total time inside region
  • percents of total application time
  • number of timer region was hit
  • min time inside region
  • average time inside region
  • max time inside region

Example output:

name                           total  % of total  count       min   average       max
--------------------------  --------  ----------  -----  --------  --------  --------
<root>                        925 ms     100.00%      1    925 ms    925 ms    925 ms
. bar()                     494.5 ms      53.48%      1  494.5 ms  494.5 ms  494.5 ms
. . loop                    408.1 ms      44.14%      1  408.1 ms  408.1 ms  408.1 ms
. . . iter                  400.9 ms      43.36%      4  100.2 ms  100.2 ms  100.2 ms
. . init                    12.85 ms       1.39%      2  5.776 ms  6.426 ms  7.076 ms
. . bar() <example2.py:40>  7.866 ms       0.85%      1  7.866 ms  7.866 ms  7.866 ms
dump_profiler(rp)[source]

Dump the profiler state.

Parameters:rp (region_profiler.profiler.RegionProfiler) – region profiler
class region_profiler.reporters.CsvReporter(columns=(<function node_id>, <function name>, <function parent_id>, <function parent_name>, <function total_us>, <function total_inner_us>, <function count>, <function min_us>, <function average_us>, <function max_us>), stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)[source]

Bases: object

Print profiler state in a CSV format.

Data is printed in a table with a configurable set of columns (default columns: id, name, parent_id, parent_name, total_us, total_inner_us, count, min_us, average_us, max_us).

Nodes are printed in a depth-first order with siblings processed sorted by the total time descending.

By default, these column are reported:

  • id
  • name
  • parent node id
  • parent node name
  • total time inside region in us
  • total time inside region in us excluding time inside its child regions
  • number of timer region was hit
  • min time inside region in us
  • average time inside region in us
  • max time inside region in us

Example output:

id, name, parent_id, parent_name, total_us, total_inner_us, count, min_us, average_us, max_us
0, <root>, , , 966221, 443352, 1, 966221, 966221, 966221
1, bar(), 0, <root>, 522868, 68080, 1, 522868, 522868, 522868
2, loop, 1, bar(), 410395, 9517, 1, 410395, 410395, 410395
3, iter, 2, loop, 400877, 400877, 4, 100208, 100219, 100227
4, init, 1, bar(), 35456, 35456, 2, 16589, 17728, 18867
5, bar() <example2.py:42>, 1, bar(), 8935, 8935, 1, 8935, 8935, 8935
dump_profiler(rp)[source]

Dump the profiler state.

Parameters:rp (region_profiler.profiler.RegionProfiler) – region profiler
region_profiler.reporters.DEFAULT_CONSOLE_COLUMNS = (<function indented_name>, <function total>, <function percents_of_total>, <function count>, <function min>, <function average>, <function max>)

Default column list for ConsoleReporter.

region_profiler.reporters.DEFAULT_CSV_COLUMNS = (<function node_id>, <function name>, <function parent_id>, <function parent_name>, <function total_us>, <function total_inner_us>, <function count>, <function min_us>, <function average_us>, <function max_us>)

Default column list for CsvReporter.

class region_profiler.reporters.SilentReporter(columns)[source]

Bases: object

Dummy test reporter. It stores rows in its attribute rows.

Nodes are printed in a depth-first order with siblings processed sorted by the total time descending.

dump_profiler(rp)[source]

Dump the profiler state.

Parameters:rp (region_profiler.profiler.RegionProfiler) – region profiler
class region_profiler.reporters.Slice(id, name, parent, call_depth, count, total_time, total_inner_time, min_time, max_time)[source]

Bases: object

Slice is an entry in a serialized list of region_profiler.node.RegionNode.

When a profiler report is generated, a slice of the current profiler data is taken (using get_node_slice()). Each entry in this slice is an instance of Slice, that contains all data about a corresponding node, that may be useful for reporting. Data from a slice is not accessed directly, but using column providers, defined in region_profiler.reporter_columns.

id

unique slice id

Type:int
name

node name

Type:str
parent

parent node in the hierarchy

Type:Slice, optional
call_depth

node depth in the hierarchy

Type:int
count

number of region hits

Type:int
total_time

total time spent in the corresponding region

Type:float
total_inner_time

total time spent in the corresponding region minus total time of all node ancestors

Type:float
min_time

minimal duration, spent in the corresponding region

Type:float
max_time

maximal duration, spent in the corresponding region

Type:float
parent_name

Convenience method for retrieving parent node name.

region_profiler.reporters.get_node_slice(slices, node, parent_slice, call_depth)[source]

Serialize a node and its descendants data in a list of Slice.

Descendants are serialized sorted by their total time in decreasing order.

Parameters:
  • slices (list of Slice) – global list of slices
  • node (region_profiler.node.RegionNode) – current node that is to be serialized
  • parent_slice (Slice, optional) – link to a slice of the parent node
  • call_depth (int) – depth of the node in the hierarchy
region_profiler.reporters.get_profiler_slice(rp)[source]

Serialize a profiler state in a list of Slice.

Descendants are serialized sorted by their total time in decreasing order.

Parameters:rp (region_profiler.profiler.RegionProfiler) – profiler
Returns:serialized nodes of the profiler
Return type:list of Slice

region_profiler.utils module

class region_profiler.utils.CallerInfo(file, line, name)

Bases: tuple

file

Alias for field number 0

line

Alias for field number 1

name

Alias for field number 2

class region_profiler.utils.NullContext[source]

Bases: object

Empty context manager.

class region_profiler.utils.SeqStats(count=0, total=0, min=0, max=0)[source]

Bases: object

Helper class for calculating online stats of a number sequence.

SeqStats records the following parameters of a number sequence:
  • element count
  • sum
  • average
  • min value
  • max value

SeqStats does not store the sequence itself, statistics are calculated online.

add(x)[source]

Update statistics with the next value of a sequence.

Parameters:x (number) – next value in the sequence
avg

Calculate sequence average.

class region_profiler.utils.Timer(clock=<function default_clock>)[source]

Bases: object

Simple timer.

Allows to measure duration between start and stop events.

By default, measurement is done on a fractions of a second scale. This can be changed by providing a different clock in constructor.

The duration can be retrieved using current_elapsed() or total_elapsed().

begin_ts()[source]

Start event timestamp.

Returns:timestamp
Return type:int or float
current_elapsed()[source]

Return duration between start and stop events or duration from last start event if no pairing stop event occurred.

Returns:duration
Return type:int or float
elapsed()[source]

Return duration between start and stop events.

If timer is running (no stop() has been called after last start() invocation), 0 is returned.

Returns:duration
Return type:int or float
end_ts()[source]

Stop event timestamp.

Returns:timestamp
Return type:int or float
is_running()[source]

Check if timer is currently running.

Returns:
Return type:bool
mark_aux_event()[source]

Update last_event_time.

start()[source]

Start new timer measurement.

Call this function again to continue measurements.

stop()[source]

Stop timer and add current measurement to total.

Returns:duration of the last measurement
Return type:int or float
region_profiler.utils.default_clock()[source]

Default clock provider for Timer class.

Returns:value (in fractional seconds) of a performance counter
Return type:float
region_profiler.utils.get_caller_info(stack_depth=1)[source]

Return caller function name and call site filename and line number.

Parameters:stack_depth (int) –

select caller frame to be inspected.

  • 0 corresponds to the call site of the get_caller_info() itself.
  • 1 corresponds to the call site of the parent function.
Returns:information about the caller
Return type:CallerInfo
region_profiler.utils.get_name_by_callsite(stack_depth=1)[source]

Get string description of the call site of the caller.

Parameters:stack_depth

select caller frame to be inspected.

  • 0 corresponds to the call site of the get_name_by_callsite() itself.
  • 1 corresponds to the call site of the parent function.
Returns:string in the following format: 'function<filename:line>'
Return type:str
region_profiler.utils.null_decorator()[source]

Empty decorator.

region_profiler.utils.pretty_print_time(sec)[source]

Get duration as a human-readable string.

Examples

  • 10.044 => ‘10.04 s’
  • 0.13244 => ‘132.4 ms’
  • 0.0000013244 => ‘1.324 us’
Parameters:sec (float) – duration in fractional seconds scale
Returns:human-readable string representation as shown above.
Return type:str

Module contents

region_profiler – profile custom regions of code