Configure

To go further...

This page is about configuring existing stlog objects. To go further, you can also read the Extend page.

The `setup()` function

stlog library seems great but I don't want to use a special way to configure it!

Yes, you can configure stlog library without custom shortcuts like Output or setup(). See FAQ for details about that.

The log level

First of all, you can tune the (minimal) log level as a global parameter for all outputs/loggers.

from stlog import setup, getLogger

# Filter messages < WARNING
setup(level="WARNING") 
getLogger().info("ignored message")

You can also override this in the setup function for a specific logger name:

from stlog import setup, getLogger

# Filter messages < WARNING
setup(level="WARNING", extra_levels={"bar": "DEBUG"}) 
getLogger("foo").info("ignored message")
getLogger("bar").debug("not ignored message thanks to the extra_levels override")

Of course, you can still use setLevel() on logger instances.

Tip

You can also use logging levels as integer. Example: logging.DEBUG

The outputs

The main configuration option of the setup function is the outputs parameter which takes a list of output.Output objects.

You can see how to create your own outputs in the extend page.

For now, you can use two kind of outputs:

a output.StreamOutput object which represents a standard stream output (for example on the console stdout or stderr)
a output.RichStreamOutput object which represents a "rich" stream output for a real and modern terminal emulator (with colors and fancy stuff)

rich library

To use a output.RichStreamOutput, you must install the rich library by yourself. It's a mandatory requirement for this ouput.

If you don't know which one to use or if you need an automatic behavior (depending on the fact that the rich library is installed or not or if we are writing to a real terminal and not to a filter redirected to a file for example), you can use a very handy factory: output.make_stream_or_rich_stream_output which will automatically choose for you.

Each Output can provide custom options but there are two common ones:

formatter which can be used to override the default logging.Formatter object
level which can be used to override the default logging level (for this specific output if this level is not already filtered at Logger level)
filters which can be used to add some logging.Filter for this specific output (note: you can also add filters at the logger level with addFilter() method)

Here is an example to configure two outputs:

one classic stream output to stderr with an overridden formatter and an overridden log level
one classic stream output to stdout with a JSON formatter

import sys
from stlog import setup
from stlog.output import StreamOutput
from stlog.formatter import HumanFormatter, JsonFormatter

setup(
    level="INFO",
    outputs=[
        StreamOutput(
            stream=sys.stderr,
            formatter=HumanFormatter(exclude_extras_keys_fnmatchs=["*"]),
            level="WARNING",
        ),
        StreamOutput(stream=sys.stdout, formatter=JsonFormatter(indent=4)),
    ]
)

Formatters?

You have a dedicated section about Formatters and stlog custom Formatters bellow.

Default output is and "automatic rich or not rich" stream on stderr with a HumanFormatter as formatter.

Filters?

Here is an example:

import sys
import logging
from stlog import setup, getLogger
from stlog.output import StreamOutput
from stlog.formatter import HumanFormatter, JsonFormatter

def this_is_a_filter(record: logging.LogRecord) -> False:
    if record.getMessage() == "message to filter":
        # We don't want this message
        return False
    return True

setup(
    level="INFO",
    outputs=[
        StreamOutput(
            stream=sys.stderr,
            formatter=HumanFormatter(exclude_extras_keys_fnmatchs=["*"]),
            level="WARNING",
        ),
        StreamOutput(stream=sys.stdout, formatter=JsonFormatter(indent=4)),
        filters=[this_is_a_filter],
    ]
) 

getLogger("foo").info("message to filter") # this message will be filtered

Warnings and "not catched" exceptions

Python warnings are automatically captured with the stlog logging infrastructure. You can disable this with capture_warnings=False parameter:

from stlog import setup

setup(
    capture_warnings=False
)

Moreover, stlog capture your "not catched" exceptions to emit them as ERROR messages. You can disable this or tune this with:

from stlog import setup

setup(
    logging_excepthook=None
)

You can also provide your own hook. See this python documentation for details.

Standard logging compatibility options

By default, stlog is going to inject global contexts into standard logging calls.

If you want to disable this behavior:

from stlog import setup

setup(
    reinject_context_in_standard_logging=False
)

You can go further by connecting the standard logging extra kwargs to stlog:

from stlog import setup
import logging

setup(read_extra_kwargs_from_standard_logging=True)
logging.warning(
    "standard logger with extra kwargs", extra={"foo": "bar", "foo2": "bar2"}
)

rich output

Formatters

In stlog we have two kinds of formatters:

standard/classic formatters in stlog.formatters: they are compatible with python logging Formatters
kvformatters in stlog.kvformatters: these custom classes build a new extras placeholder (with all key/values from contexts) you can use in your standard formatters (as a part on the format string)

KV formatters

Let's start with "KV formatters". They are completely specific to stlog but optional. They format extra key values into a single {extras} placeholder you can use classically with "standard formatters". They are optional because you can also use individual placeholders for each key/value.

Let's say, you have to extra key values:

foo="bar"
foo2="bar2"

In the standard formatter format, you can use {foo} and {foo2} placeholders without a "KV formatter" at all. But you need to know the name of all key values you want to add to your log messages (because you have to list them in the format string):

Example:

if the formatter format string is {levelname}: {message} (foo={foo}, foo2={foo2})" and if you log with stlog.getLogger().warning("this is a warning", foo="bar", foo2="bar2"), you are going to get: warning: this is a warning (foo=bar, foo2=bar2)
but if you use a LogFmtKVFormatter and a format string: {levelname}: {message} {extras}, you are going to get: warning: this is a warning {foo=bar, foo2=bar2}

So "KV formatters" are only responsible of serializing all extra key values to an {extras} string placeholder. If you don't use this placeholder in your format string, you don't need a "KV formatter" at all.

what about %(extras)s placeholder instead?

stlog use new formatter strings (with style="{"). So placeholders are using the form: {name}. If you prefer old-style placeholders with style="%" you can of course switch to the old-style and you use %(extras)s placeholder instead.

Available "KV formatters"

Here are available "KV formatters". You can of course write yours (see extend page in documentation).

`EmptyKVFormatter`

This one is not very interesting as it always returns an empty string as {extras}.

`TemplateKVFormatter`

This one formats {extras} with string templating.

See kvformatter.TemplateKVFormatter for details.

Example:

import sys
from stlog import setup, getLogger
from stlog.kvformatter import TemplateKVFormatter
from stlog.output import StreamOutput
from stlog.formatter import HumanFormatter


setup(
    outputs=[
        StreamOutput(
            stream=sys.stderr,
            formatter=HumanFormatter(
                fmt="{asctime}: ***{levelname}*** {message}{extras}",
                kv_formatter=TemplateKVFormatter(
                    template="{key} => {value}",
                    separator=", ",
                    prefix="\n    ",
                ),
            ),
        )
    ]
)

getLogger().warning("this is a warning", foo="bar", foo2="bar2")

2023-03-29T14:48:37Z: ***WARNING*** this is a warning
    foo => bar, foo2 => bar2}

`LogFmtKVFormatter`

This class formats {extras} with logfmt format.

See kvformatter.LogFmtKVFormatter for details.

Example:

import sys
from stlog import setup, getLogger
from stlog.kvformatter import LogFmtKVFormatter
from stlog.output import StreamOutput
from stlog.formatter import HumanFormatter

setup(
    outputs=[
        StreamOutput(
            stream=sys.stderr,
            formatter=HumanFormatter(
                fmt="{asctime}: ***{levelname}*** {message}{extras}",
                kv_formatter=LogFmtKVFormatter(
                    prefix=" [",
                    suffix="]",
                ),
            ),
        )
    ]
)

getLogger().warning("this is a warning", foo="foo bar", foo2="bar2")

2023-03-29T14:48:37Z: ***WARNING*** this is a warning [foo="foo bar" foo2=bar2]

What about compound types in extras?

FIXME

Standard formatters

Common options

All stlog formatters have common options you can find on the API reference.

These options are common logging.Formatter options extended with some stlog custom options.

Extra options are mainly about context key/values formatting. We are going to see some real examples after.

`HumanFormatter`

This formatter can be used to get a human friendly output (without the rich library).

Let's start with a simple example:

import sys
from stlog import setup, getLogger
from stlog.output import StreamOutput
from stlog.formatter import HumanFormatter


setup(outputs=[StreamOutput(stream=sys.stderr, formatter=HumanFormatter())])

getLogger(__name__).warning("this is a warning", foo="bar", foo2="bar2")

2023-03-29T14:48:37Z __main__ [ WARNING  ] this is a warning {foo=bar foo2=bar2}

Default format is:

{asctime} {name} [{levelname:^10s}] {message}{extras}

Of course, you can change this format with for example:

import sys
from stlog import setup, getLogger
from stlog.output import StreamOutput
from stlog.formatter import HumanFormatter

format = "{asctime} {levelname} from process #{process}: {message}\n    => {extras}"
asctime_format = "%H:%M:%SZ"
setup(
    outputs=[
        StreamOutput(
            stream=sys.stderr,
            formatter=HumanFormatter(fmt=format, datefmt=asctime_format),
        )
    ]
)

getLogger(__name__).warning("this is a warning", foo="bar", foo2="bar2")

14:48:37Z WARNING from process #6789: this is a warning
    =>  {foo=bar foo2=bar2}

{extras} format?

By default, HumanFormatter use a LogFmtKVFormatter for formatting the {extras} placeholder. Of course, you can change or tune that by providing your own instance of KVFormatter to HumanFormatter object.

`RichHumanFormatter`

This formatter can be used to get a human friendly output (with the rich library library installed).

Default format is:

:arrow_forward: [log.time]{asctime}[/log.time] {name} [{rich_level_style}]{levelname:^8s}[/{rich_level_style}] [bold]{rich_escaped_message}[/bold]{extras}

This formatter provides some custom placeholders:

{rich_escaped_message}: the standard {message} placeholder but escaped to avoid some accidental rich markup rendering inside message
{rich_escaped_extras}: same thing but for {extras} placeholder
{rich_level_style}: contain a smart rich markup style depending on the log level

{extras}/{rich_escaped_extras} format?

By default, RichHumanFormatter use a LogFmtKVFormatter for formatting the {extras} placeholder with custom attributes:

prefix="\n :arrow_right_hook: "
suffix=""
template="[repr.attrib_name]{key}[/repr.attrib_name][repr.attrib_equal]=[/repr.attrib_equal][repr.attrib_value]{value}[/repr.attrib_value]"

Of course, you can change or tune that by providing your own instance of KVFormatter to HumanFormatter object.

`LogFmtFormatter`

This formatter will format your logs with the logfmt format.

Default format is:

time={asctime} logger={name} level={levelname} message={message}{extras}

But the use of this format is special with this formatter as placeholders can be encoded to be valid for logfmt format. So to configure an alternate format:

use the key={placeholder} syntax (with space separated key/values blocs)
don't try to escape or quote your values , it will be done automatically and dynamically, so don't use quotes in your format
use the placeholder {extras} alone at the end (without leading space) to get all extra key/values

import sys
from stlog import setup, getLogger
from stlog.output import StreamOutput
from stlog.formatter import LogFmtFormatter


fmt = "time={asctime} logger={name} process={process} level={levelname} message={message}{extras}"
setup(outputs=[StreamOutput(stream=sys.stderr, formatter=LogFmtFormatter(fmt=fmt))])

getLogger(__name__).warning("this is a warning", foo="bar", foo2="bar2")
getLogger(__name__).critical(
    'this is a critical message with\nmultiple\nlines and "quotes"'
)

rich output

`JsonFormatter`

This formatter will format your logs in the JSON format.

Default format is:

{{
    "time": {asctime},
    "logger": {name},
    "level": {levelname},
    "message": {message},
    "source": {{
        "path": {pathname},
        "lineno": {lineno},
        "module": {module},
        "funcName": {funcName},
        "process": {process},
        "processName": {processName},
        "thread": {thread},
        "threadName": {threadName}
    }}
}}

But the use of this format is special with this formatter as placeholders can be encoded to be valid for JSON. So to configure an alternate format:

provide a nearly JSON valid format (except for placeholders, see next item)
don't try to escape your values, it will be done automatically and dynamically, so don't use quotes in your format around placeholders
there is no {extras} placeholder (see below)

What about {extras} placeholder in JSON format?

Extra key/values are automatically injected in the JSON as root keys if include_extras_in_key="" (default).

You can include all extras keys as a child dict or another root key by using for example: includes_extras_in_key="extras" to get something like:

{
    "time": "2023-01-01T12:13:14Z",
    "logger": "foo",
    "level": "CRITICAL",
    "message": "Houston we have a problem",
    "extras": {
        "foo": "bar",
        "foo2": "bar2",
    }
}

Note: you can also use `include_extras_in_key=None` to remove all extras key/values from output.

Available Environment variables

As we love Twelve-Factor App plenty of default behavior of stlog can be configured with environment variables.

configuration priority?

In order of importance (the last one wins):

default values (set in the code)
environment variables
explicit configuration in the code (always wins)

`STLOG_LEVEL`

This variable can tune the default "log level". Use CRITICAL, FATAL, ERROR, WARN, WARNING, INFO, DEBUG or NOTSET as value.

`STLOG_DESTINATION`

This variable can tune the default destination. Use stdout or stderr as value. Default to stderr.

`STLOG_OUTPUT`

This variable can set the default output and format. You can use:

console: the output will be sent to stdout or stderr (depending on STLOG_DESTINATION value) with a "human formatter" and will be use a rich (colors...) output depending in STLOG_USE_RICH value. By default, rich output will be automatically used if the terminal support it and if the rich library is installed.
json: the output will be sent to stdout or stderr (depending on STLOG_DESTINATION value) with a generic "JSON formatter" in a compact mode (without indentation)
json-human: the output will be sent to stdout or stderr (depending on STLOG_DESTINATION value) with a generic "JSON formatter" in an indented way
json-gcp: the output will be sent to stdout or stderr (depending on STLOG_DESTINATION value) with a GCP "JSON formatter" (to be automatically decoded and displayed in GCP logging products)

`STLOG_USE_RICH`

This variable can tune the behavior of output.make_stream_or_rich_stream_output function:

if empty or set to NONE or AUTO => nothing (the function makes automatically a StreamOuput or a RichStreamOutput depending on the terminal support and if the rich library is installed)
if set to 1, TRUE, YES => the function will always return a RichStreamOutput (even the log stream is redirected to a file!)
else (0, FALSE, NO...) => the function will always return a standard StreamOutput (with colors and fancy things)

use_rich parameter?

This can be overriden by the use_rich parameter when calling output.make_stream_or_rich_stream_output

`STLOG_CAPTURE_WARNINGS`

This variable can change the default value of capture_warnings parameter of the setup function:

if set to 0, FALSE, NO: default value of capture_warnings is set to False

`STLOG_REINJECT_CONTEXT_IN_STANDARD_LOGGING`

This variable can change the default value of reinject_context_in_standard_logging parameter of the setup function:

if set to 0, FALSE, NO: default value of reinject_context_in_standard_logging is set to False

`STLOG_READ_EXTRA_KWARGS_FROM_STANDARD_LOGGING`

This variable can change the default value of read_extra_kwargs_from_standard_logging parameter of the setup function:

if set to 1, TRUE, YES: default value of read_extra_kwargs_from_standard_logging is set to True

`STLOG_ENV_JSON_CONTEXT` and `STLOG_ENV_CONTEXT_*`

These variables can be used to inject a global context. See usage documentation for details.

`STLOG_DEFAULT_IGNORE_COMPOUND_TYPES`

if set to 1, TRUE, YES: compound types (dict, list...) are silently ignored in LogFmtKVFormatter (used by default by "human" outputs)
else (0, FALSE, NO...): compound type will be displayed

The default is to ignore.

What about JSON outputs?

This variable has no effect on JSON outputs.

`STLOG_UNIT_TESTS_MODE`

Private feature!

This is a private feature (DON'T USE IT) to get always the same output (fixed date, fixed process number...)

`STLOG_PROGRAM_NAME`

Default program name when getting a logger without name. If not set, we will try to guess.

Configure

The setup() function

The log level

The outputs

Warnings and "not catched" exceptions

Standard logging compatibility options

Formatters

KV formatters

Available "KV formatters"

EmptyKVFormatter

TemplateKVFormatter

LogFmtKVFormatter

Standard formatters

Common options

HumanFormatter

RichHumanFormatter

LogFmtFormatter

JsonFormatter

Available Environment variables

STLOG_LEVEL

STLOG_DESTINATION

STLOG_OUTPUT

STLOG_USE_RICH

STLOG_CAPTURE_WARNINGS

STLOG_REINJECT_CONTEXT_IN_STANDARD_LOGGING

STLOG_READ_EXTRA_KWARGS_FROM_STANDARD_LOGGING

STLOG_ENV_JSON_CONTEXT and STLOG_ENV_CONTEXT_*

STLOG_DEFAULT_IGNORE_COMPOUND_TYPES

STLOG_UNIT_TESTS_MODE

STLOG_PROGRAM_NAME

The `setup()` function

`EmptyKVFormatter`

`TemplateKVFormatter`

`LogFmtKVFormatter`

`HumanFormatter`

`RichHumanFormatter`

`LogFmtFormatter`

`JsonFormatter`

`STLOG_LEVEL`

`STLOG_DESTINATION`

`STLOG_OUTPUT`

`STLOG_USE_RICH`

`STLOG_CAPTURE_WARNINGS`

`STLOG_REINJECT_CONTEXT_IN_STANDARD_LOGGING`

`STLOG_READ_EXTRA_KWARGS_FROM_STANDARD_LOGGING`

`STLOG_ENV_JSON_CONTEXT` and `STLOG_ENV_CONTEXT_*`

`STLOG_DEFAULT_IGNORE_COMPOUND_TYPES`

`STLOG_UNIT_TESTS_MODE`

`STLOG_PROGRAM_NAME`