Using Jsonnet mixins for configurations

name: inverse
class: title,inverse

## anarcher@gmail.com
## 2019-08-20

---
class: inverse

# jsonnet
A data templating language for app and tool developers.

A simple extension of JSON.

- https://jsonnet.org/
- https://twitter.com/sparkprime
- [Introduction to Jsonnet](https://github.com/ksonnet/ksonnet-lib/blob/master/docs/jsonnetIntro.md)

---
class: pic

![](./venn.svg)

---

# Jsonnet Demo

---

# K8S Deployment example: `helm`

```yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
 name: {{ template "nginx.fullname" . }}
 labels:
 app: {{ template "nginx.name" . }}
spec:
 replicas: {{ .Values.replicaCount }}
 selector:
 matchLabels:
 app: {{ template "nginx.name" . }}
 template:
 metadata:
 labels:
 app: {{ template "nginx.name" . }}
 spec:
 containers:
 - name: {{ template "nginx.name" . }}
 image: {{ Values.Image }}
 ports:
 - containerPort: 80
```

---

# K8S Deployment example: `jsonnet`

`nginx-base.jsonnet `
```yaml
local deploy = import "kube-deployment.libsonnet";
deploy + {
 name: "my-nginx",
 container: {
 image: "nginx:1.12",
 },
}
```

```sh
$kubectl apply -h
Apply a configuration to a resource by filename or stdin. The resource name must be specified. This resource will be
created if it doesn't exist yet. To use 'apply', always create the resource initially with either 'apply' or 'create
--save-config'.

JSON and YAML formats are accepted.
```

[GOTO:#11](#11)

???
- Kustomize [Proposal: Support overlaying guide comment](https://github.com/kubernetes-sigs/kustomize/issues/1125)

---

# K8S Deployment example: `jsonnet`

`kube-deployment.libsonnet`
<pre><code class="yaml hljs remark-code" style="font-size: 80%;">{
 `name`:: error "name is required",
 `container`:: error "container is required",
 apiVersion: "apps/v1beta1",
 kind: "Deployment",
 metadata: {
    name: $.name,
    labels: { app: $.name },
 },
 spec: {
    selector: { matchLabels: $.metadata.labels },
    template: {
        metadata: { labels: $.metadata.labels },
        spec: {
            containers: [
                `$.container` { name: `$.name` },
            ],
        },
    },
 },
}
</code></pre>

???
# References
- self refers to the current object.
- `$` refers to the outer-most object.
- `['foo']` looks up a field.
- `.f` can be used if the field name is an identifier.

---
class: pic

![](standards.png)

- https://xkcd.com/927/
- [[Public] Kubernetes application management tools
](https://docs.google.com/spreadsheets/u/1/d/1FCgqz1Ci7_VCz_wdh8vBitZ3giBtac_H8SBw4uxnrsE/htmlview#gid=0)

---

# Using Jsonnet with Kubernetes

- https://jsonnet.org/articles/kubernetes.html
- [An Example of Real Kubernetes: Bitnami](https://engineering.bitnami.com/articles/an-example-of-real-kubernetes-bitnami.html)
    - `kubecfg` : https://github.com/bitnami/kubecfg
- [ksonnet](https://ksonnet.io/)
    - The team behind ksonnet is stepping back from the project. As a result, work on ksonnet will end and the GitHub repositories will be archived. [link](https://blogs.vmware.com/cloudnative/2019/02/05/welcoming-heptio-open-source-projects-to-vmware/)

<pre><code class="sh hljs remark-code" style="font-size: 60%;"># View all available commands
ks --help

# Create an app directory (and reference your cluster with $KUBECONFIG)
KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config}"

ks init ks-example    # Add '--api-spec=version:v1.8.0' to this command if you're running Kubernetes 1.8!
cd ks-example

# Autogenerate a basic manifest
ks generate deployed-service guestbook-ui \
  --image gcr.io/heptio-images/ks-guestbook-demo:0.1 \
  --type ClusterIP

# Deploy the manifest to your cluster, and view results
ks apply default
kubectl get svc guestbook-ui

# Teardown (Don't worry, this only removes resources defined in your ksonnet app,
# and should not affect any other resources in your 'default' namespace)
ks delete default
</code></pre>

---

# Using Jsonnet with Prometheus

- [kube-prometheus](https://github.com/coreos/kube-prometheus#customizing-kube-prometheus)
    - Use Prometheus to monitor Kubernetes and applications running on Kubernetes
- [Prometheus Monitoring Mixin for Kubernetes](https://github.com/kubernetes-monitoring/kubernetes-mixin)
    - A set of Grafana dashboards and Prometheus alerts for Kubernetes.
- [2018-02 Prometheus Monitoring Mixins Design Document](https://docs.google.com/document/d/1A9xvzwqnFVSOZ5fD3blKODXfsat5fg6ZhnKu9LK3lB4/edit#heading=h.gt9r2h2gklj3)
    - A monitoring mixin is __a package__ of configuration containing __Prometheus alerts__, __Prometheus recording rules__ and __Grafana dashboards__. 
    - Mixins will be maintained in __version controlled repos__ (eg __git__) as a set of files. Versioning of mixins will be provided by the version control system; mixins themselves should not contain multiple versions.
    - __Jsonnet__ offers the ability to __parameterise configuration__, allow for basic customisation.  Furthermore, in Jsonnet one can reference another part of the data structure, __reducing repetition__.   
- __Mixin__ Conceptually, Mixins provide dynamic inheritance, at runtime instead of compile time, which lets you combine them freely to modify objects or create new ones.

---

# Prometheus Monitoring Mixins
 
`mixin.libsonnet`
```yaml
{
_config+:: {...},
grafanaDashboards+:: {
   “dashboard-name.json”: {...},
},
prometheusAlerts+:: [...],
prometheusRules+:: [...],
} 
```
[GOTO:#6](#6)
- We’d like to suggest some __standardisation__ of how configuration is supplied to mixins.
- A top level `_config` dictionary has __various parameters__ for substitution 
- [`jsonnet-bundler`](https://github.com/jsonnet-bundler/jsonnet-bundler) : A jsonnet package manager.
    <pre><code class="bash hljs remark-code" style="font-size: 70%;">jb install https://github.com/anguslees/kustomize-libsonnet
    jb install https://github.com/coreos/prometheus-operator/jsonnet/prometheus-operator
    </code></pre>

---

# Prometheus Monitoring Mixins

`mixin.libsonnet`

```yaml
local kubernetes = import "kubernetes-mixin/mixin.libsonnet";

kubernetes {
  _config+:: {
    kubeStateMetricsSelector: 'job="kube-state-metrics"',
    cadvisorSelector: 'job="kubernetes-cadvisor"',
    nodeExporterSelector: 'job="kubernetes-node-exporter"',
    kubeletSelector: 'job="kubernetes-kubelet"',
    grafanaK8s+:: {
      dashboardNamePrefix: 'Mixin / ',
      dashboardTags: ['kubernetes', 'infrastucture'],
    },
  },
}
```

```sh
$ jb init
$ jb install github.com/kubernetes-monitoring/kubernetes-mixin
```

---

# Prometheus Monitoring Mixins

```sh
$ jsonnet -J vendor -m dashboards -e '(import "config.libsonnet").grafanaDashboards'
dashboards/kube-apiserver.json
dashboards/kube-controller-manager.json
dashboards/kube-proxy.json
dashboards/kube-scheduler.json
dashboards/kubelet.json
```

```sh
$ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' > alerts.yml
alerts.yml
$ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' >files/rules.yml
rules.yml
```

__Next:__ Adding Dashboards to Grafana & Prometheus
- [Provisioning Grafana](https://grafana.com/docs/administration/provisioning/#dashboards)

---

# Prometheus Monitoring Mixins

Prometheus mixin 
- https://github.com/prometheus/prometheus/pull/4474
- https://github.com/prometheus/prometheus/tree/master/documentation/prometheus-mixin

Node exporter mixin
- https://github.com/prometheus/node_exporter/pull/941
- https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin

---

# kube-prometheus

```sh
$ mkdir my-kube-prometheus; cd my-kube-prometheus
$ jb init  # Creates the initial/empty `jsonnetfile.json`
# Install the kube-prometheus dependency
$ jb install github.com/coreos/kube-prometheus/jsonnet/kube-prometheus@release-0.1 # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json`
```
`minxin.jsonnet`
```sh
local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +
{
    _config+:: {
               namespace: 'monitoring',
    },
};

{ ['00prometheus-operator-' + name + '.json']: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) }
{ ['0prometheus-' + name + '.json']: kp.prometheus[name] for name in std.objectFields(kp.prometheus) }
```

```sh
$kubecfg update -J vendor/ mixin.jsonnet
```

---

# kube-prometheus

```sh
$kubecfg update -J vendor/ mixin.jsonnet
$kubecfg show -o yaml -J vendor/ mixin.jsonnet
$jsonnet -J vendor -m manifests/ minxin.jsonnet
```

<pre><code class="shell hljs remark-code" style="font-size:70%;">
├── jsonnetfile.json
├── jsonnetfile.lock.json
├── manifests
├── mixin.jsonnet
└── vendor
    ├── etcd-mixin
    ├── grafana
    ├── grafana-builder
    ├── grafonnet
    ├── ksonnet
    ├── kube-prometheus
    ├── kubernetes-mixin
    ├── prometheus
    └── prometheus-operator
</code></pre>

---

# ksonnet

https://ksonnet.io/

```json
local k = import 'ksonnet/ksonnet.beta.4/k.libsonnet';
{
    serviceAccount:
      local serviceAccount = k.core.v1.serviceAccount;
      serviceAccount.new('prometheus-' + $._config.prometheus.name) +
      serviceAccount.mixin.metadata.withNamespace($._config.namespace),
}
```

- [Move away from ksonnet-lib since it is deprecated](https://github.com/coreos/kube-prometheus/issues/99)
> "I have created a PR to add ksonnet.beta.4 to ksonnet-lib that is based on Kubernetes 1.14 and we (as in maintainers of kube-prometheus, Promethues Operator and kubernetes-mixin) have no plans to abandon ksonnet any time soon."

---

# ksonnet-lib

https://github.com/ksonnet/ksonnet-lib

```json
local k = import "ksonnet.beta.2/k.libsonnet";

// Specify the import objects that we need
local container = k.extensions.v1beta1.deployment.mixin.spec.template.spec.containersType;
local containerPort = container.portsType;
local deployment = k.extensions.v1beta1.deployment;

local targetPort = 80;
local podLabels = {app: "nginx"};

local nginxContainer =
  container.new("nginx", "nginx:1.7.9") +
  container.ports(containerPort.containerPort(targetPort));

local nginxDeployment =
  deployment.new("nginx-deployment", 2, nginxContainer, podLabels);

k.core.v1.list.new(nginxDeployment)

```

---

# Jsonnet dashboard

- [Grafonnet](https://github.com/grafana/grafonnet-lib/) : a simple DSL using jsonnet to generate Grafana dashboards.

```json
local grafana = import 'grafonnet/grafana.libsonnet';
local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + {
    grafanaDashboards+:: {
     'my-dashboard.json':
        dashboard.new('My Dashboard v2')
        .addTemplate(template.datasource(
            'datasource',
            'prometheus',
            'Prometheus',
        ))
        .addRow(
            row.new().addPanel(
                graphPanel.new('My Panel', span=6, datasource='$datasource')
                .addTarget(prometheus.target('vector(1)'))
            ),
        ),
    }
}
```

---

# Jsonnet dashboard

```sh
$kubecfg -J vendor/ update mixin.jsonnet
INFO  Creating configmaps monitoring.grafana-dashboard-apiserver
INFO  Creating configmaps monitoring.grafana-dashboard-controller-manager
INFO  Creating configmaps monitoring.grafana-dashboard-k8s-resources-cluster
INFO  Creating configmaps monitoring.grafana-dashboard-k8s-resources-namespace
INFO  Creating configmaps monitoring.grafana-dashboard-k8s-resources-pod
INFO  Creating configmaps monitoring.grafana-dashboard-k8s-resources-workload
INFO  Creating configmaps monitoring.grafana-dashboard-k8s-resources-workloads-namespace
INFO  Creating configmaps monitoring.grafana-dashboard-kubelet
INFO  Creating configmaps monitoring.grafana-dashboard-my-dashboard
INFO  Creating configmaps monitoring.grafana-dashboard-persistentvolumesusage
INFO  Creating configmaps monitoring.grafana-dashboard-pods
INFO  Creating configmaps monitoring.grafana-dashboard-prometheus
INFO  Creating configmaps monitoring.grafana-dashboard-prometheus-remote-write
INFO  Creating configmaps monitoring.grafana-dashboard-proxy
INFO  Creating configmaps monitoring.grafana-dashboard-scheduler
INFO  Creating configmaps monitoring.grafana-dashboard-statefulset
INFO  Creating configmaps monitoring.grafana-dashboards
```

---

# Jsonnet dashboard

```sh
$k get cm
NAME                                                  DATA   AGE
grafana-dashboard-apiserver                           1      26m
grafana-dashboard-controller-manager                  1      26m
grafana-dashboard-k8s-resources-cluster               1      26m
grafana-dashboard-k8s-resources-namespace             1      26m
grafana-dashboard-k8s-resources-pod                   1      26m
grafana-dashboard-k8s-resources-workload              1      26m
grafana-dashboard-k8s-resources-workloads-namespace   1      26m
grafana-dashboard-kubelet                             1      26m
grafana-dashboard-my-dashboard                        1      26m
grafana-dashboard-persistentvolumesusage              1      26m
grafana-dashboard-pods                                1      26m
grafana-dashboard-prometheus                          1      26m
grafana-dashboard-prometheus-remote-write             1      26m
grafana-dashboard-proxy                               1      26m
grafana-dashboard-scheduler                           1      26m
grafana-dashboard-statefulset                         1      26m
grafana-dashboards                                    1      26m
```

---

# Jsonnet dashboard

![](./my-dashboard.png)
![](./grafana-menu.png)