name: inverse class: title,inverse Using Jsonnet mixins for configurations ## anarcher@gmail.com ## 2019-08-20 --- class: inverse # jsonnet A data templating language for app and tool developers. A simple extension of JSON. - https://jsonnet.org/ - https://twitter.com/sparkprime - [Introduction to Jsonnet](https://github.com/ksonnet/ksonnet-lib/blob/master/docs/jsonnetIntro.md) --- class: pic  --- # Jsonnet Demo --- # K8S Deployment example: `helm` ```yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: {{ template "nginx.fullname" . }} labels: app: {{ template "nginx.name" . }} spec: replicas: {{ .Values.replicaCount }} selector: matchLabels: app: {{ template "nginx.name" . }} template: metadata: labels: app: {{ template "nginx.name" . }} spec: containers: - name: {{ template "nginx.name" . }} image: {{ Values.Image }} ports: - containerPort: 80 ``` --- # K8S Deployment example: `jsonnet` `nginx-base.jsonnet ` ```yaml local deploy = import "kube-deployment.libsonnet"; deploy + { name: "my-nginx", container: { image: "nginx:1.12", }, } ``` ```sh $kubectl apply -h Apply a configuration to a resource by filename or stdin. The resource name must be specified. This resource will be created if it doesn't exist yet. To use 'apply', always create the resource initially with either 'apply' or 'create --save-config'. JSON and YAML formats are accepted. ``` [GOTO:#11](#11) ??? - Kustomize [Proposal: Support overlaying guide comment](https://github.com/kubernetes-sigs/kustomize/issues/1125) --- # K8S Deployment example: `jsonnet` `kube-deployment.libsonnet` <pre><code class="yaml hljs remark-code" style="font-size: 80%;">{ `name`:: error "name is required", `container`:: error "container is required", apiVersion: "apps/v1beta1", kind: "Deployment", metadata: { name: $.name, labels: { app: $.name }, }, spec: { selector: { matchLabels: $.metadata.labels }, template: { metadata: { labels: $.metadata.labels }, spec: { containers: [ `$.container` { name: `$.name` }, ], }, }, }, } </code></pre> ??? # References - self refers to the current object. - `$` refers to the outer-most object. - `['foo']` looks up a field. - `.f` can be used if the field name is an identifier. --- class: pic  - https://xkcd.com/927/ - [[Public] Kubernetes application management tools ](https://docs.google.com/spreadsheets/u/1/d/1FCgqz1Ci7_VCz_wdh8vBitZ3giBtac_H8SBw4uxnrsE/htmlview#gid=0) --- # Using Jsonnet with Kubernetes - https://jsonnet.org/articles/kubernetes.html - [An Example of Real Kubernetes: Bitnami](https://engineering.bitnami.com/articles/an-example-of-real-kubernetes-bitnami.html) - `kubecfg` : https://github.com/bitnami/kubecfg - [ksonnet](https://ksonnet.io/) - The team behind ksonnet is stepping back from the project. As a result, work on ksonnet will end and the GitHub repositories will be archived. [link](https://blogs.vmware.com/cloudnative/2019/02/05/welcoming-heptio-open-source-projects-to-vmware/) <pre><code class="sh hljs remark-code" style="font-size: 60%;"># View all available commands ks --help # Create an app directory (and reference your cluster with $KUBECONFIG) KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config}" ks init ks-example # Add '--api-spec=version:v1.8.0' to this command if you're running Kubernetes 1.8! cd ks-example # Autogenerate a basic manifest ks generate deployed-service guestbook-ui \ --image gcr.io/heptio-images/ks-guestbook-demo:0.1 \ --type ClusterIP # Deploy the manifest to your cluster, and view results ks apply default kubectl get svc guestbook-ui # Teardown (Don't worry, this only removes resources defined in your ksonnet app, # and should not affect any other resources in your 'default' namespace) ks delete default </code></pre> --- # Using Jsonnet with Prometheus - [kube-prometheus](https://github.com/coreos/kube-prometheus#customizing-kube-prometheus) - Use Prometheus to monitor Kubernetes and applications running on Kubernetes - [Prometheus Monitoring Mixin for Kubernetes](https://github.com/kubernetes-monitoring/kubernetes-mixin) - A set of Grafana dashboards and Prometheus alerts for Kubernetes. - [2018-02 Prometheus Monitoring Mixins Design Document](https://docs.google.com/document/d/1A9xvzwqnFVSOZ5fD3blKODXfsat5fg6ZhnKu9LK3lB4/edit#heading=h.gt9r2h2gklj3) - A monitoring mixin is __a package__ of configuration containing __Prometheus alerts__, __Prometheus recording rules__ and __Grafana dashboards__. - Mixins will be maintained in __version controlled repos__ (eg __git__) as a set of files. Versioning of mixins will be provided by the version control system; mixins themselves should not contain multiple versions. - __Jsonnet__ offers the ability to __parameterise configuration__, allow for basic customisation. Furthermore, in Jsonnet one can reference another part of the data structure, __reducing repetition__. - __Mixin__ Conceptually, Mixins provide dynamic inheritance, at runtime instead of compile time, which lets you combine them freely to modify objects or create new ones. --- # Prometheus Monitoring Mixins `mixin.libsonnet` ```yaml { _config+:: {...}, grafanaDashboards+:: { “dashboard-name.json”: {...}, }, prometheusAlerts+:: [...], prometheusRules+:: [...], } ``` [GOTO:#6](#6) - We’d like to suggest some __standardisation__ of how configuration is supplied to mixins. - A top level `_config` dictionary has __various parameters__ for substitution - [`jsonnet-bundler`](https://github.com/jsonnet-bundler/jsonnet-bundler) : A jsonnet package manager. <pre><code class="bash hljs remark-code" style="font-size: 70%;">jb install https://github.com/anguslees/kustomize-libsonnet jb install https://github.com/coreos/prometheus-operator/jsonnet/prometheus-operator </code></pre> --- # Prometheus Monitoring Mixins `mixin.libsonnet` ```yaml local kubernetes = import "kubernetes-mixin/mixin.libsonnet"; kubernetes { _config+:: { kubeStateMetricsSelector: 'job="kube-state-metrics"', cadvisorSelector: 'job="kubernetes-cadvisor"', nodeExporterSelector: 'job="kubernetes-node-exporter"', kubeletSelector: 'job="kubernetes-kubelet"', grafanaK8s+:: { dashboardNamePrefix: 'Mixin / ', dashboardTags: ['kubernetes', 'infrastucture'], }, }, } ``` ```sh $ jb init $ jb install github.com/kubernetes-monitoring/kubernetes-mixin ``` --- # Prometheus Monitoring Mixins ```sh $ jsonnet -J vendor -m dashboards -e '(import "config.libsonnet").grafanaDashboards' dashboards/kube-apiserver.json dashboards/kube-controller-manager.json dashboards/kube-proxy.json dashboards/kube-scheduler.json dashboards/kubelet.json ``` ```sh $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' > alerts.yml alerts.yml $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' >files/rules.yml rules.yml ``` __Next:__ Adding Dashboards to Grafana & Prometheus - [Provisioning Grafana](https://grafana.com/docs/administration/provisioning/#dashboards) --- # Prometheus Monitoring Mixins Prometheus mixin - https://github.com/prometheus/prometheus/pull/4474 - https://github.com/prometheus/prometheus/tree/master/documentation/prometheus-mixin Node exporter mixin - https://github.com/prometheus/node_exporter/pull/941 - https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin --- # kube-prometheus ```sh $ mkdir my-kube-prometheus; cd my-kube-prometheus $ jb init # Creates the initial/empty `jsonnetfile.json` # Install the kube-prometheus dependency $ jb install github.com/coreos/kube-prometheus/jsonnet/kube-prometheus@release-0.1 # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json` ``` `minxin.jsonnet` ```sh local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + { _config+:: { namespace: 'monitoring', }, }; { ['00prometheus-operator-' + name + '.json']: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } { ['0prometheus-' + name + '.json']: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } ``` ```sh $kubecfg update -J vendor/ mixin.jsonnet ``` --- # kube-prometheus ```sh $kubecfg update -J vendor/ mixin.jsonnet $kubecfg show -o yaml -J vendor/ mixin.jsonnet $jsonnet -J vendor -m manifests/ minxin.jsonnet ``` <pre><code class="shell hljs remark-code" style="font-size:70%;"> ├── jsonnetfile.json ├── jsonnetfile.lock.json ├── manifests ├── mixin.jsonnet └── vendor ├── etcd-mixin ├── grafana ├── grafana-builder ├── grafonnet ├── ksonnet ├── kube-prometheus ├── kubernetes-mixin ├── prometheus └── prometheus-operator </code></pre> --- # ksonnet https://ksonnet.io/ ```json local k = import 'ksonnet/ksonnet.beta.4/k.libsonnet'; { serviceAccount: local serviceAccount = k.core.v1.serviceAccount; serviceAccount.new('prometheus-' + $._config.prometheus.name) + serviceAccount.mixin.metadata.withNamespace($._config.namespace), } ``` - [Move away from ksonnet-lib since it is deprecated](https://github.com/coreos/kube-prometheus/issues/99) > "I have created a PR to add ksonnet.beta.4 to ksonnet-lib that is based on Kubernetes 1.14 and we (as in maintainers of kube-prometheus, Promethues Operator and kubernetes-mixin) have no plans to abandon ksonnet any time soon." --- # ksonnet-lib https://github.com/ksonnet/ksonnet-lib ```json local k = import "ksonnet.beta.2/k.libsonnet"; // Specify the import objects that we need local container = k.extensions.v1beta1.deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; local deployment = k.extensions.v1beta1.deployment; local targetPort = 80; local podLabels = {app: "nginx"}; local nginxContainer = container.new("nginx", "nginx:1.7.9") + container.ports(containerPort.containerPort(targetPort)); local nginxDeployment = deployment.new("nginx-deployment", 2, nginxContainer, podLabels); k.core.v1.list.new(nginxDeployment) ``` --- # Jsonnet dashboard - [Grafonnet](https://github.com/grafana/grafonnet-lib/) : a simple DSL using jsonnet to generate Grafana dashboards. ```json local grafana = import 'grafonnet/grafana.libsonnet'; local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + { grafanaDashboards+:: { 'my-dashboard.json': dashboard.new('My Dashboard v2') .addTemplate(template.datasource( 'datasource', 'prometheus', 'Prometheus', )) .addRow( row.new().addPanel( graphPanel.new('My Panel', span=6, datasource='$datasource') .addTarget(prometheus.target('vector(1)')) ), ), } } ``` --- # Jsonnet dashboard ```sh $kubecfg -J vendor/ update mixin.jsonnet INFO Creating configmaps monitoring.grafana-dashboard-apiserver INFO Creating configmaps monitoring.grafana-dashboard-controller-manager INFO Creating configmaps monitoring.grafana-dashboard-k8s-resources-cluster INFO Creating configmaps monitoring.grafana-dashboard-k8s-resources-namespace INFO Creating configmaps monitoring.grafana-dashboard-k8s-resources-pod INFO Creating configmaps monitoring.grafana-dashboard-k8s-resources-workload INFO Creating configmaps monitoring.grafana-dashboard-k8s-resources-workloads-namespace INFO Creating configmaps monitoring.grafana-dashboard-kubelet INFO Creating configmaps monitoring.grafana-dashboard-my-dashboard INFO Creating configmaps monitoring.grafana-dashboard-persistentvolumesusage INFO Creating configmaps monitoring.grafana-dashboard-pods INFO Creating configmaps monitoring.grafana-dashboard-prometheus INFO Creating configmaps monitoring.grafana-dashboard-prometheus-remote-write INFO Creating configmaps monitoring.grafana-dashboard-proxy INFO Creating configmaps monitoring.grafana-dashboard-scheduler INFO Creating configmaps monitoring.grafana-dashboard-statefulset INFO Creating configmaps monitoring.grafana-dashboards ``` --- # Jsonnet dashboard ```sh $k get cm NAME DATA AGE grafana-dashboard-apiserver 1 26m grafana-dashboard-controller-manager 1 26m grafana-dashboard-k8s-resources-cluster 1 26m grafana-dashboard-k8s-resources-namespace 1 26m grafana-dashboard-k8s-resources-pod 1 26m grafana-dashboard-k8s-resources-workload 1 26m grafana-dashboard-k8s-resources-workloads-namespace 1 26m grafana-dashboard-kubelet 1 26m grafana-dashboard-my-dashboard 1 26m grafana-dashboard-persistentvolumesusage 1 26m grafana-dashboard-pods 1 26m grafana-dashboard-prometheus 1 26m grafana-dashboard-prometheus-remote-write 1 26m grafana-dashboard-proxy 1 26m grafana-dashboard-scheduler 1 26m grafana-dashboard-statefulset 1 26m grafana-dashboards 1 26m ``` --- # Jsonnet dashboard  