October 2022: Feature releases & highlights

• Site observability enhancements
  • Site tenant and application view
  • Logs and metrics search on sites
  • Site alerts
  • Host metrics
• Site and host life-cycle management
• Application site variables
• Documentation enhancements

This month’s release relates very much to the lasagne at an edge site.

In many cases, different organisations are responsible for different layers, and blame games are easily started across the layers. In the October release, we have addressed features that will help you make the perfect lasagne for your edge system. The features will help you communicate across the layers and work proactively. You will also see features that make life easier across the layers, such as enabling OS upgrades with minimum application impact.

The main three highlights for October are:

1. Site observability enhancements: you can now search logs and metrics per site, receive proactive site alerts, and easily see any applications and tenants that are allocated to the site
2. Site and host lifecycle management: support for administrative actions and state relating to host replacement and OS upgrade procedures
3. Site-specific application variables: an application specification can refer to site-specific variables, for example controlling the number of replicas

Site observability enhancements

Site tenant and application view

Service impact is sometimes considered the holy grail for monitoring. In the Avassa system, you can easily see which applications and tenants are allocated to a site. If you select the site, you will see two new tabs (visible if you have applications and tenants, respectively)

The screenshot above illustrates the case where you have a tenant sw that has deployed an application popper to the site. The operational state of the application is also shown. This way, you can easily see the service impact from sites/hosts to tenants and applications.

Logs and metrics search on sites

The Avassa system has a built-in pub/sub bus, “Volga,” that runs on each edge site and centrally in the Control Tower. Relevant metrics and logs are published on the bus. Examples are host metrics, application metrics, scheduler, and deployment events. These categories of logs and metrics are called “topics” and are now available in the User Interface when you select a site:

On the site, you can then pick the topic you are interested in:

Finally select a time window, and, optionally, a string, and the search will be proxied to the site:

You will find this very useful to analyze what is happening on a site from various aspects. It is also a good way to start to learn about available topics and possibly integrate the data over APIs. For example, picking up host metrics for your monitoring system. The Web UI utilizes the Volga query API. By clicking the </> button above the query, you can see the question and experiment yourself using the Avassa command line and Rest API.

This functionality has been available in the UI for only the container logs since before. Now all topics are available., In addition, you can now also select specific time intervals when searching your container logs:

Site alerts

As a site provider, you will now get proactive alerts on non-desired states regarding the hosts; examples are disc warnings and host down. These are published on a Volga topic system:site-alerts.

As described above, you can inspect these using the general UI functionality, supctl, or Rest API. But they will now also show up in the UI as alerts:

The top right corner of the Web UI has an alert icon that will become red if there is a new alert. Clicking the bell icon will show the eight latest alerts.

You can inspect all alerts by clicking “View all” here or using the leftmost “Alerts” menu.

These alerts are also represented as states on the corresponding entities, in this case the host. The screenshot below shows the site list to the left where we have an indication that the Bergakungen site has issues. Selecting the site will display the list of hosts and corresponding states.

Coming releases will include application-related alerts.

Host metrics

For each host on the sites, Avassa collects host metrics regarding disc, memory, and load. These metrics are now available in the UI after you have selected a host on a site:

For the host, you get the following metrics

• Disc: disc metrics from mount points related to Avassa Edge Enforcer, mounted volumes, Edge Enforcer state directory, and the Edge Enforcer container root. Not the complete host
• Memory: total, available and free memory from /proc/meminfo
• Load: Average load of processes last minute, last 5 minutes, and last 15 minutes from /proc/loadavg

Site and host life-cycle management

Maintenance activities of the edge hosts need to be performed with minimal impact on the running applications. These activities can, for example, be host replacements and OS upgrade. You can now manage how applications are scheduled on the site in order to prepare for this kind of activities

• You can set a host in maintenance mode; this means that no new services will be scheduled for this host
• You can drain a host; this means that services are moved to other hosts on the site

The latter is for example, useful before performing an OS upgrade:

Drain a host:

$ supctl do --site sthlm system cluster hosts sthlm-003 drain

This command will block and return when all service instances on the host are stopped.

The maintenance-mode is configuration on the site and can for example be set by the convenient supctl edit operation:

supctl edit system sites gothenburg-bergakungen

And you can inspect the state on the site:

supctl show --site gothenburg-bergakungen system cluster hosts --fields cluster-hostname,maintenance-mode

cluster-hostname: gothenburg-bergakungen-001
maintenance-mode: blocked

Application site variables

In many cases, you would like the application to have site-specific behavior. And rather than having different application specifications for different sites, you would like that to be expressed as variables in your application specification. One pattern is to use different vaults for the different sites and use site variables to pick the right vault as described in Site specific secrets and configuration.

We have now also added explicit variable support for replicas and container log size in the application specification. This means that you can have for example different replicas on different site on a site label. Read more in the REST documentation on replicas and container-log-size for the application.

Assume you add a popcorn-replicas site label to your sites.

This can now be used as a variable in your application specification:

name: popcorn-controller
version: "1.0"
services:
  - name: popcorn-controller-service
    mode: replicated
    replicas: ${SYS_SITE_LABELS[popcorn-replicas]}

In this way you can have one single application specification but it will have different number of replicas on the sites.

💡 The best lasagne?
Massimo Bottura, Osteria Francescana, Modena

Documentation enhancements

• New tutorial on Volga and supctl