Calico Integration

Calico is one of the most popular CNI’s in the Kubernetes ecosystem.  Offering a simple method to connect nodes, this container networking function is easy to implement for small-to-medium sized clusters, and has strong community and paid support options.

 

However, as more nodes are added to the cluster, there is an exponential growth in the number of BGP peers that must be maintained by each node.  This limits the scalability of your on-demand infrastructure.

 

Project Calico recommends establishing local BGP peering with the physical network to offload this task to network switches and routers.  The implementation guide for this change (available from legacy hardware vendors) is sometimes 100+ pages long!  Surely there must be an easier way.

Enabling BGP peering between all Kubernetes nodes and network switches and routers requires detailed IP and AS number planning as well as properly configured BGP policies on every single device. Every time DevOps engineers add, move or delete Kubernetes cluster nodes — network engineers will need to repeat the planning and implementation of these BGP policies. This takes lot of time and creates the potential for human error that can bring the network down.

 

With Netris, a one-line annotation command triggers Netris to automatically configure both Calico and network switches and routers.  The necessary BGP peers are configured on both sides (nodes to leaf/TOR switches), the peering is established in a fully automatic fashion.  Once this new network routing domain converges, Netris turns off the original full-mesh mode, without interrupting application traffic.  From that point forward, Netris monitors the CNI control plane for any changes, and if necessary, automatically updates all affected devices in realtime.

Calico Integration Demo

In this video, the Calico annotation is applied and the BGP configurations are created and applied by Netris automatically.  Once the new routing topology is stable, the full-mesh BGP configuration running on the k8s nodes is shut down.  There is no impact to application traffic during this process.

Integration Design

Before

Full Mesh BGP

  • BGP works out of the box
  • Good for small clusters
  • Doesn’t scale
  • Exponential number of BGP neighbors
  • High resource consumption in large clusters

After

Local BGP peering with Leaf/TOR

  • BGP config is automatically managed by Netris
  • Architecture complements Calico requirements
  • Decreased number of BGP peers on k8s servers
  • Improved performance of k8s nodes
  • Maximum cluster scalability

One Kubernetes Annotation to Apply!

Seamless automatic integration of physical network with CNI network, based on best practices and recommendations of Tigera/Calico.