NOTE: these properties can be used on their own or in conjunction with others e.g. Checks.

Input

Provides a common way to define input. Takes one or more filesystem path or command. When a command is provided, its output is written to a temporary file and input.path is set to the path of that file.

This property is required by and used as input to the Search property.

Usage:

input:
  command: hotsos.core.host_helpers.CLIHelper.<command>
  path: <path>
  options:
    disable-all-logs: <bool>
    args: [arg1]
    kwargs:
      key1: val1
    args-callback: import.path.to.method

The path and command settings are mutually exclusive. A path can be a single or list of filesystem paths that must be relative to Data Root.

PathFinder

For applications that have more than one install method this can result in log files existing in different locations. To support this, some plugins provide a PathFinder which takes a relative log file path and returns the absolute path if it exists based on one or more choice. To use a PathFinder do the following:

input: '<plugin>:app.log'

Where “plugin” results in hotsos.core.plugins.<name>.PathFinder being imported and called with app.log as a value. If the application uses either of e.g. /var/log and /snap/app/common/log then both are checked until one is found. This is useful so that we don’t have to explicitly list all possible paths for every search.

Logrotate Depth

By default if –all-logs is provided to the hotsos client that will apply to every path but there may be cases where this is not desired and –disable-all-logs can be used to disable this behaviour for a specific path.

Using Command Output as Input

If we want to take input as the output of a command, we can use the `command:` stanza to specify a command that will be executed and its output captured.

If the command requires some input, both args and kwargs can be set as a list and/or dictionary respectively and will be providing as input to the CLIHelper command.

Sometimes we may way to dynamically generate the input args to command. For this purpose we can use args-callback which is set to the import path of a method whose return value must be a tuple of the form (<list>, <dict>) where the list is used as args to the command and dict is kwargs.

Cache keys: * cmd_tmp_path

Search

Used to define a search using expression(s) and constraints. Different types of search expressions that can be used depending on the data being searched and how the results will be interpreted:

simple search (SearchDef) - a single pattern used to match single lines.

sequence search (SequenceSearchDef) - used to match (non-overlapping) sequences.

Search results are passed to their handler as a raw SearchResultsCollection.

This property is implemented as a mapped property so the search name is optional.

IMPORTANT: do not use global search properties. If you do this, the same search tag will be used for all searches and it will not be possible to distinguish results from more than one leaf node.

Usage:

search:
  # the following are used to define a "simple" search
  expr: <str>
  hint: <str>
  # the following are used to define a "sequence" search
  start: <str>
  body: <str>
  end: <str>
  # If this is set to True it enables a passthrough sequence
  passthrough-results: True|False
  constraints:
    # Epoch (to current date i.e. CLIHelper.date()) that
    # results must fall within. Default is infinite.
    search-result-age-hours: <int>
    # Period of time within which we expect to find results.
    # Default is infinite.
    search-period-hours: <int>
    # Minimum number of search results required. If a search
    # period is defined, these must occur within that period.
    # Default is 1.
    min-results: <int>
    # Search result must be at least this number of hours
    # after the last boot time.
    min-hours-since-last-boot: <int>

Search expressions can be defined as a single string or list of strings. If the provided string value starts with “@” it will be considered as Python import path and imported. This allows for using expressions stored in code.

If you want to analyse logs that contain overlapping sequences, perhaps from multiple threads running concurrently, a passthrough sequence search is used by setting passthrough-results to True. This will leverage LogEventStats and requires a callback method to be implemented to process the results.

Constraints are used to filter search results and are typically used in conjunction with Checks. In order to use constraints, search expressions must match a timestamp using result group 1. The format of timestamps e.g. in logs and command outputs will vary and there are handlers in the code to support common formats.

Cache keys:

• simple_search - a searchkit.SearchDef object

• sequence_search - a searchkit.SequenceSearchDef object

• sequence_passthrough_search - a list of searchkit.SearchDef objects

The above keys are mostly used for internal purposes and the following extra entries are added to provide a way to access search results in Raises (also see PropertyCache):

• search.results_group_<int> - extract the value from result group <int>

• search.num_results - the number of results found by this search

In the following example we demonstrate how to use these keys. A file called var/log/myapp.log has contents:

2023-10-12 13:22:01 ERROR: queue 'small_queue' is full
2023-10-12 14:12:33 ERROR: queue 'small_queue' is full

And we have a scenario like:

checks:
  errorsfound:
    input: var/log/myapp.log
    expr: '\S+ \S+ ERROR: queue ''(\S+)'' is full'
conclusions:
  haserrors:
    decision: errorsfound
    raises:
      type: SomeWarning
      message: >-
        found {count} reports of queue full for queue(s): {queues}
      format-dict:
        count: '@checks.errorsfound.search.num_results'
        queues: '@checks.errorsfound.search.results_group_1:unique_comma_join'

The message string output would look like:

found 2 "queue full" error(s) for queue(s): small_queue