One edge, many application teams: Mastering multi-tenancy at the edge

I will kick off this article with a quotation from a conference I once attended. There was a discussion around the complexity of a large open source project with many sub-projects and components. One speaker stood up and said:

“Multiple tenants? We will just wrap it in a tenancy framework”

Before reading on, let’s just stop for a second and spend a few moments on reality-checking that statement. At the distributed on-site edge, what are best practices for managing multiple tenants across sites and edge applications?

Well, let’s dig into the topic. As you will realize when reading the below, multi-tenancy is actually something that needs to be designed into the product from the get-go, and is not something that can be bolted on as an afterthought, no matter how many micro-services you use.

First of all lets define tenants and the different roles a tenant can have in a real-life example of an edge environment.

The edge infrastructure is managed by one tenant, let’s call it the site provider. The site provider is responsible for making sure that the edge sites and compute resources are available for running applications. Additionally, we want let the site provider manage a second category of tenants: the application owners. The application owners are responsible for the deployment and management of applications, on the edge infrastructure provided by the site provider. In this case, we’re looking at a shared multi-tenancy approach, where application owners share the available resources on sites, according to the resource slicing managed by the site provider. This is a typical scenario for large enterprise users that might cater several internal and/or external application teams, utilizing the very same edge infrastructure.

“Why not provide completely new and fresh compute (such as a VM) to each application owner?”

You might ask. In fact, we frequently see how such a model where each tenant would get a complete new compute or even VMs at the edge conflicts the very nature of edge infrastructure, where resources are constrained.

So, what are the things that need to be shared (and consequently: isolated) amongst tenants?

This would include infrastructure resources such as:

• Edge sites, two application tenants can for example share a single edge site

Also, it can include resources within a host in a shared site:

• Memory, storage and CPU: which need to sliced amongst the application owners on the site
• External devices, like cameras: that in some cases are uniquely allocated to a certain tenant, for example reserving a camera for the security team.
• GPUs

While infrastructure resources are sliced and need to be shared amongst application tenants, the reverse is true for the applications.

Application resources:

• Container logs and application metrics
• Application networking
• Application data and secrets

Each tenant application needs to be fully isolated from other application tenants. One application team should not be able to read another teams container logs. The challenge is to give the infrastructure owner the right amount of insight into the application owner tenants, without ever leaking any sensitive data. This is where the concepts of adding a tenancy framework as an add-on solution is really starting to wobble.

What about the other side of the very same coin: The application owners insight into the edge site infrastructure. It is important to find the right abstraction so that detailed infrastructure information does not appear into the application layer. For example: Do the application owners really need to be bothered with information on which physical host has a camera mounted? Or are they more interested in deploying to where-ever that is. Instead, there needs to be a “contract” between the site provider and application owners. That contract should contain requirements from the application owners and offered capabilities of the site providers. The site scheduler should then strive to fulfill the obligations between the two tenants. This is illustrated below:

The illustration highlights the following main requirements:

• Application owner tenants do not have insight into the infrastructure details
• Application owners define application requirements
• The site provider publishes capabilities and defines resource limits
• The edge site cluster matches requirements against capabilities
• The edge site cluster limits resources amongst tenants
• Different application owners should have no insight into other applications resources
• The site provider should have very limited insight into application owners, for example not be able to read container logs

So, what about our multi-tenancy framework then? Look at the requirements above and include the requirements for sharing and isolating resoruces in one unified way, one API, one UI. And if you then start to pick various projects to solve each of the components; a container scheduler, a storage component, a logging component, a service mesh component etc. How would you make it all come together?

Wrap all of them in a multi-tenancy framework?

Probably not.

You can watch a video that illustrates these principles in the Avassa multi-tenancy demo.