The Monitor object¶

To begin with, we will consider a very simple scenario.

Scenario

You have monitored some tests, and now you want to read collected metrics.

Your data are stored on a monitor server whose address is at https://my.monitor.server.org:5555/

For doing so, let’s introduce the Monitor object. It will be your only entry point to every interaction with either your remote server or a local database.

from monitor_api import Monitor

URL = 'https://my.monitor.server.org:5555/'
mon = Monitor(URL)

You now have an object able to interact with the remote server. Note if you wanted to query the database generated by pytest-monitor, the URL would have been the path to the database on your filesystem.

The Monitor object uses a small semantic around actions you can perform

count is used to obtain integers and roughly aims at counting entities.
list aims at providing an exhaustive listing of entities for specific resources such as components, variants and so on.
get is used to obtain a fully described, single entity.

Counting and retrieving entities¶

The Monitor interface allows you to perform two basic operations on entities: counting and retrieving detailed information. The table below details operation support for each entity:

Entity	Countable	Details
Metric	Yes	No
Session	Yes	Yes
Context	Yes	Yes
Component	Yes	No
Builds & pipelines	No	No

The Monitor provides 4 methods for counting entities:

Monitor.count_components() will give the number of different components (empty values are ignored)
Monitor.count_sessions() will count the number of distinct sessions
Monitor.count_contexts() does the same as the previous method except that it works for contexts
Monitor.count_metrics() offers the ability to count all metrics with the possibility to add some restrictions:
- metrics with the same SCM reference
- metrics of a session
- metrics of a contexts.

Scenario: The storage holds 3 components, 10 sessions (5 different SCM reference each run twice), 1 context and 200 metrics (20 metrics per session). Count all entities.

print(mon.count_components()) # Prints 3
print(mon.count_contexts()) # Prints 1
print(mon.count_metrics()) # Prints 200
print(mon.count_sessions()) # Prints 10
print(mon.count_metrics(session_h='deadbeef0badc0ffee')) # Prints 20
print(mon.count_metrics(context_h='badc0ffee0deadbeef')) # Prints 200
print(mon.count_metrics(scm_ref='badbeef')) # Prints 20

Retrieving details is available only for Session and Context as Metrics are not intended to be retrieved individually. Both Session and Context have an hash string that can be used to uniquely identify the entity. Through this key you can access any of these entities individually by using

Monitor.get_session(hash: str) to retrieve details about a Session
Monitor.get_context(hash: str) to retrieve details about a Context

session = mon.get_session('deadbeef0badc0ffee')
print(f'Session {session.h}, SCM@{session.scm}') # Prints "Session deadbeef0badc0ffee, SCM@ABCDEF"
assert session.start_date == datetime.datetime(2021, 2, 9, 12, 12, 4, 136861)
print(session.tags) # Prints {'pipeline_branch': 'master-py36', 'pipeline_build_no': '6', '__ci__': 'circleci'}

context = mon.get_context('badc0ffee0deadbeef')
print(f'Context {context.h}, node {context.machine_node}') # Prints "Context badc0ffee0deadbeef, node@Rotondu"
assert context.total_ram >= 10000

Collecting entities¶

To assist you, the Monitor interface proposes various methods to collect multiple entities at once through objects described in monitor_api.core.collections.

Basically, 3 collections are available:

Sessions behaves like a dict structure using Session’s hash as keys and Session as value holders.
Contexts works the same as Sessions except Context object are used for storing values.
Metrics acts as a list with access to any element using its position in the list.

They share a similar interface: data load/dump, filtering, context managers…

Scenario: List all entities (200 metrics, 1 context and 10 sessions) known by the server

metrics = mon.list_metrics()
print(len(metrics)) # Print 200
sessions = mon.list_sessions()
print(len(sessions)) # Print 10
print(session.keys()) # Prints all session identifiers
contexts = mon.list_contexts()
print(len(contexts)) # Print 1
print(contexts.keys()) # Print all context identifiers

See more at Api Reference.

Associating metrics with CI runs¶

pytest-monitor since version 1.5.0 automatically detects builds and pipelines when run from a specific Software Factory. To quickly focus on metrics issued by these runs, the Monitor object offers a way to collect metrics and sessions associated to these pipelines and/or builds.

Basically, a pipeline sets all degrees of freedom so that tests can be run in a repeatable and reproducible way. A build is simply an instance of a pipeline. In other words, it represents a single run of a pipeline and has no meaning without a pipeline.

The interesting thing about that is that when you run your test in parallel, a single run of your build will lead to multiple Session. This way, pipelines and builds offers quite a neat way to access all metrics of a given build regardless of all parallel aspects.

To get access to these information, you have 4 main entry points:

list_component_pipelines(component: str) will give you access to all known pipelines of a specific component.
list_component_pipeline_builds(component: str, pipeline: str) will in turn list all builds for the pipeline of your choice for the given components.
list_pipelines() will basically lists all pipelines in the storage.
list_pipeline_builds(pipeline: str) will list all builds of a pipeline.

This last method may not allow you to perfectly distinguish pipelines for different component. This is quite obvious if you consider two components being run in two different pipelines but named in the same way.

Note

Pipelines and Builds, although entities does not have dedicated objects for now.

Monitoring resource usage¶

Last but not least, the Monitor object provides dedicated entry points for listing tests which are amongst top (resp. lowest) resource consumers. To do so, monitor_api provides 2 specific types: ResourceMethod and ResourceType.

The ResourceType aims at standardizing the resource you want to analyze whereas ResourceMethod represents the sorting order

The following entry points are provided:

list_metrics_resources(resource: ResourceType, method: ResourceMethod, max_element: int)
list_metrics_resources_from_component(component: str, resource: ResourceType, method: ResourceMethod, max_element: int)
list_metrics_resources_from_pipeline(pipeline: str, resource: ResourceType, method: ResourceMethod, max_element: int)
list_metrics_resources_from_build(pipeline: str, build: str, resource: ResourceType, method: ResourceMethod, max_element: int)

To illustrate the behavior let’s consider the following scenario.

Scenario: Identify top 15 tests which consumes most the memory on build 123 of pipeline “master-py38”. Prints for each variant the amount of memory used.

metrics = mon.list_metrics_resources_from_build('master-py38', '123', ResourceType.MEMORY,
                                                ResourceMethod.TOP, 15)
for metric in metrics:
    print(f'{metric.variant} : {metric.memory_usage}MB')