Context

Currently, there are two different K8s CNI architectures which are possible for integrating Tungsten Fabric (TF) as a CNI in K8s.

Option 1: K8s CNI running via Docker Compose

Architecture

Description

In this architecture, the TF containers which deliver the K8s CNI are run via Docker Compose on one of the hosts in the deployment.   This architecture is what is deployed when the `Ansible Deployer` is used.   There are a few key limitations with this architecture: It is very hard to support a Highly Available CNI control plane in this design. It is hard to integrate with other deployment types (such as RKE) because the CNI is not deployed inside K8s (which is how the majority of the other CNIs in the market are deployed).

Option 2: K8s CNI running inside K8s

Architecture

Description

In this architecture, the TF containers which deliver the K8s CNI are running inside K8s and are controlled by the K8s scheduler.   This is the architecture which is used when deploying via the `1-line deployment` for both CentOS and Ubuntu.     There are a few benefits from running this architecture: Because K8s manages the lifecycle and scheduling of the containers, it is easier to create an HA deployment. The CNI implementation is entirely contained in a single deployment manifest, making it easier to implement in K8s clusters which are installed outside the control of TF. Because the CNI implementation is self contained, it makes it easier to build integrations with other deployers, such as Rancher's RKE.

Current State

This section is going to be a bit dense, but I am going to attempt to provide a snapshot of the current state and the technical implications in play.

• Currently, there are two different manifests available for deploying Option 2, one for [CentOS|https://tungstenfabric.github.io/website/Tungsten-Fabric-Centos-one-line-install-on-k8s.html] ([manifest|https://raw.githubusercontent.com/Juniper/contrail-kubernetes-docs/master/install/kubernetes/templates/contrail-single-step-cni-install-centos.yaml]) and one for [Ubuntu|https://tungstenfabric.github.io/website/Tungsten-Fabric-Ubuntu-one-line-install-on-k8s.html] (manifest).  
  Unfortunately, from what I can tell, these manifests are not actively maintained and there does not appear to be a formal process to build these manifests in a generic way with the current tooling.  Here is the source for the currently [maintained manifests|https://github.com/Juniper/contrail-container-builder/tree/master/kubernetes/manifests].
• It is possible to build these manifests using the [resolve-manifest.sh|https://github.com/Juniper/contrail-container-builder/blob/master/kubernetes/manifests/resolve-manifest.sh] script, however, this script is expected to be run on the target system where the CNI will be installed as it populates many of the variables by inspecting the host on which the script is run on.
• The [resolve-manifest.sh|https://github.com/Juniper/contrail-container-builder/blob/master/kubernetes/manifests/resolve-manifest.sh] script calls the [parse-env.sh|https://github.com/Juniper/contrail-container-builder/blob/master/parse-env.sh], which defines a bunch of the default settings and will use the current host on which it is run to populate some variables.  The [parse-env.sh|https://github.com/Juniper/contrail-container-builder/blob/master/parse-env.sh] script references a [common.env|https://github.com/Juniper/contrail-container-builder/blob/master/common.env.sample] script, which enables the manual override of different variables in the produced manifest.
• In the current process, there are defaults which are associated with the manifest (which is required and important), however, the defaults which are used in the manifest generation process defined in [parse-env.sh|https://github.com/Juniper/contrail-container-builder/blob/master/parse-env.sh] are not the same defaults which are used to apply defaults to the actual containers.  The defaults which are applied to the containers are actually pulled from [common.sh|https://github.com/Juniper/contrail-container-builder/blob/master/containers/base/common.sh].  Unfortunately, the [common.sh|https://github.com/Juniper/contrail-container-builder/blob/master/containers/base/common.sh] script, while maintaining the most comprehensive list of defaults (which I have found), currently depends on pulling some information from the environment on which the script is run.  This means that we are not able to leverage it, in its current form, as the common source of the default values which are to be injected into the containers and the generated manifests.

The Proposal

I think it is very important that the Tungsten Fabric community starts officially supporting the Option 2 deployment architecture as a first class citizen as this architecture is the default / best practice approach for installing a CNI in Kubernetes and enables the opportunity to deploy TF in a much wider range of K8s deployments.  I would like to build an integration with Rancher's RKE so we can use TF as a first class CNI in Rancher and this deployment architecture is a requirement for that work to be effective.

Since we are already advertising this deployment architecture as a viable deployment strategy on both [CentOS|https://tungstenfabric.github.io/website/Tungsten-Fabric-Centos-one-line-install-on-k8s.html] ([manifest|https://raw.githubusercontent.com/Juniper/contrail-kubernetes-docs/master/install/kubernetes/templates/contrail-single-step-cni-install-centos.yaml]) and [Ubuntu|https://tungstenfabric.github.io/website/Tungsten-Fabric-Ubuntu-one-line-install-on-k8s.html] (manifest), I think it is important that we properly support those documented deployment architectures with the means to support the generation of the required manifests from the [latest upstream manifests|https://github.com/Juniper/contrail-container-builder/tree/master/kubernetes/manifests].

I would like to create a PR which would do the following:

• Extract the default values from [common.sh|https://github.com/Juniper/contrail-container-builder/blob/master/containers/base/common.sh] and put them into a new `defaults.sh` file in the same folder.  The `defaults.sh` script would only specify defaults which do not need any environment context to populate.  This new `defaults.sh` script would then be `sourced` into the current [common.sh|https://github.com/Juniper/contrail-container-builder/blob/master/containers/base/common.sh] so the currently maintained build process would remain unchained.  The [entrypoint.sh|https://github.com/Juniper/contrail-container-builder/blob/master/containers/analytics/alarm-gen/entrypoint.sh] (an example, there are a lot of them) scripts will not need to change because it only references the [common.sh|https://github.com/Juniper/contrail-container-builder/blob/master/containers/base/common.sh].  However, all of the [.externals|https://github.com/Juniper/contrail-container-builder/blob/master/containers/controller/webui/base/.externals] scripts will need to be updated (my understanding) to handle the new `defaults.sh` script so it can be referenced via `/defaults.sh` instead of the full contextual path.
• Introduce a new script in the [manifests|https://github.com/Juniper/contrail-container-builder/tree/master/kubernetes/manifests] directory which references the new `defaults.sh` script and also takes a [common.env|https://github.com/Juniper/contrail-container-builder/blob/master/common.env.sample] file which can be used for passing in any contextual variables into the script to define the variables specific to the environment on which this manifest will be used on.  The execution of this script will be completely independent from the host on which it runs on (does not try to set variables based on the host the script is run on).