Can I get an email on alerts?

In this article, we’ll look at how to set up email notifications when an alert is raised in the Avassa platform. This can help users that lack an operation center instantly track alerts through their inbox.

No alert shall go unnoticed

A recent conversation with a user revealed that in one of their customer locations the cameras collected dust and blurred the image. The dust build up is detected by their software but needs manual cleaning. Hence their software will generate alerts in the Avassa platform and someone can go on-site and clean the lenses.

If you are a large enterprise or a service provider, you will probably integrate our alarm topics into you operation center where you have 24/7 staffing. You can easily do this by for example using our Fluent Bit plugin and subscribe to alerts on the sites and then let FluentBit forward to your log and alert system. But when having conversations with our smaller customers, they rarely have the resources nor the need for a 24/7 SOC. They do however have SLAs to meet and can’t ever let an alert go unnoticed. One idea we had was to consume the alarm topic and generate an email when an alert is raised. So we did what we usually do when we have good ideas, we start coding.

Activating email notifications on alerts

The idea is simple, setup a Volga consumer on the system:alerts topic, possibly filter what types of alerts should generate an email and then send the email. Volga is our built-in pub/sub system where, among other things, alarms are published.

A very simple python program will do. The python snippet will run on a server with access to the Control Tower, it doesn’t require being run on the Avassa system itself (although it can if packaged as a container).

To get started, install the Avassa python client library.

$ python3 -m pip install --upgrade avassa-client

#!/usr/bin/env python3

import asyncio
import json

async def async_main():
    import avassa_client
    import avassa_client.volga as volga
    import datetime
    import json
    import socket
    # Load config file
    with open('config.json') as config_file:
        file_contents = config_file.read()
        config = json.loads(file_contents)

    # Login user
    session = avassa_client.login(
            host=config['ct-api'],
            username=config['ct-user'],
            password=config['ct-password'])

    # Use hostname as consumer name
    myhostname = socket.gethostname()
    # The system:alerts topic should always be available
    create_opts = volga.CreateOptions.fail()
    topic = volga.Topic.local('system:alerts')

    # Just get new messages, make sure you ack, see below
    position = volga.Position.unread()
    async with volga.Consumer(session=session,
                              consumer_name=myhostname,
                              mode='exclusive',
                              position=position,
                              topic=topic,
                              on_no_exists=create_opts) as consumer:
        # Kick it off with one message
        await consumer.more(1)
        while True:
            msg = await consumer.recv()
            payload = msg['payload']
            # Only mail on non cleared alerts
            # Here you may want to filter on more stuff
            dt = datetime.datetime.fromisoformat(payload['time'])
            now = datetime.datetime.now(datetime.timezone.utc)

            # Only send message if it's not cleared nor older than a day
            if not payload['cleared'] and (now-dt).days <= 1:
                print(f'payload: {payload}')
                await send_mail(config, str(payload))
            # Ack we have handled this message
            await consumer.ack(msg['seqno'])

async def send_mail(config, alert_message):
    # see <https://realpython.com/python-send-email/>
    import smtplib
    import ssl
    from email.message import EmailMessage

    sender = config['sender']
    receiver = config['receiver']

    # Create RFC 5322 compliant message
    msg = EmailMessage()
    msg['Subject'] = "Alert"
    msg['From'] = sender
    msg['To'] = receiver
    alert_message = json.dumps(alert_message)
    message = f"An alert was raised, details:\\n{alert_message}"
    msg.set_content(message)

    context = ssl.create_default_context()

    with smtplib.SMTP(host=config['smtp-server'],
                      port=config['smtp-port']) as server:
        server.starttls(context=context)
        server.login(sender, config['smtp-password'])
        print("sending mail")
        server.send_message(msg)

def main():
    asyncio.run(async_main())

if __name__ == "__main__":
    main()

The code above loads a config file with the following JSON format.

{
  "ct-api": "<https://api.proc-env.acme.avassa.net>",
  "ct-user": "alerts@acme.com",
  "ct-password": "secret-password",
  "sender": "alerts@acme.com",
  "receiver": "alerts@acme.com",
  "smtp-server": "send.smtp.com",
  "smtp-port": 25,
  "smtp-password": "secret-smtp-password"
}

With this in place you will get an email like:

An alert was raised, details:
"{'alert': 'application-error', 'time': '2023-05-16T12:10:29.858Z', 'id': 'application-error/home-cluster-3/visitor-counter', 'site': 'home-cluster-3', 'severity': 'critical', 'description': 'Application failure.', 'expiry-time': '2023-05-16T16:10:29.858Z', 'cleared': False, 'data': {'application': 'visitor-counter', 'application-version': '2.0', 'application-deployment': 'visitors', 'error-message': 'failed to load image into docker cache'}}"

A bit crude, but can of course be refined.

Conclusion not under separate…

As you can see above, with a few lines of python code we can consume alerts and send emails when alerts are raised.

Keep reading: Programming with Volga.