chapter: Preparing the K8s Cluster

text:
*nplus* Charts bring some custom resources, *Application*, *Instance* and *Component*. they are created during deployment of a chart and then updated by the environment operator every time the status changes.
To make this work, you will need to have the *Custom Resource Definitions* applied to your cluster prior to deploying any environment or component. This deployment is handled by the *Cluster Chart*.
```bash
helm install nplus/nplus-cluster
```
The *CRDs* are grouped into *nscale* and *nplus* (both synonym), so that you can either query for
```bash
kubectl get instance
kubectl get component
kubectl get application
```
or simply all at once with
```bash
kubectl get nscale -A
```
the output looks like this (shortened output, showing the installed samples):
```bash
$ kubectl get nscale -A
NAMESPACE   NAME                                                    INSTANCE                     COMPONENT       TYPE            VERSION        STATUS
empty-sim   component.nplus.cloud/database                          empty-sim                    database        database        16             healthy
empty-sim   component.nplus.cloud/nappl                             empty-sim                    nappl           core            9.2.1302       healthy
lab         component.nplus.cloud/demo-centralservices-s3-nstl      demo-centralservices-s3      nstl            nstl            9.2.1302       healthy
lab         component.nplus.cloud/demo-ha-web                       demo-ha                      web             web             9.2.1300       redundant
lab         component.nplus.cloud/demo-ha-webdav                    demo-ha                      webdav          webdav          9.2.1000       redundant
lab         component.nplus.cloud/demo-ha-zerotrust-administrator   demo-ha-zerotrust            administrator   administrator   9.2.1300       healthy
lab         component.nplus.cloud/no-provisioner-nstl               no-provisioner               nstl            nstl            9.2.1302       healthy
lab         component.nplus.cloud/no-provisioner-rs                 no-provisioner               rs              rs              9.2.1201       starting
lab         component.nplus.cloud/no-provisioner-web                no-provisioner               web             web             9.2.1300       healthy
lab         component.nplus.cloud/sbs-nappl                         sbs                          nappl           core            9.2.1302       healthy
NAMESPACE   NAME                                                    INSTANCE            APPLICATION   VERSION        STATUS
empty-sim   application.nplus.cloud/application                     empty-sim           application   9.2.1303-123   healthy
empty-sim   application.nplus.cloud/prepper                         empty-sim           prepper       1.2.1300       healthy
lab         application.nplus.cloud/demo-ha-zerotrust-application   demo-ha-zerotrust   application   9.2.1303-123   healthy
lab         application.nplus.cloud/demo-shared-application         demo-shared         application   9.2.1303-123   healthy
lab         application.nplus.cloud/sbs-sbs                         sbs                 SBS           9.2.1303-123   healthy
lab         application.nplus.cloud/tenant-application              tenant              application   9.2.1303-123   healthy
NAMESPACE   NAME                                              HANDLER   VERSION    TENANT   STATUS
empty-sim   instance.nplus.cloud/empty-sim                    manual    9.2.1302            healthy
lab         instance.nplus.cloud/default                      manual    9.2.1302            healthy
lab         instance.nplus.cloud/demo-centralservices         manual    9.2.1302            healthy
lab         instance.nplus.cloud/rms                          manual    9.2.1302            healthy
lab         instance.nplus.cloud/sbs                          manual    9.2.1302            healthy
lab         instance.nplus.cloud/tenant                       manual    9.2.1302            healthy
```


chapter: K8s namespace aka *nplus environment*

text:
*nplus instances* are deployed into K8s namespaces. Always. even if you do not specify a namespace, it uses a namespace: `default`.
In order to use this namespace for *nplus instances*, you need to deploy some shared *nplus components* into it, which are used by the instances. This is done by the environment chart:
```
helm install \
--values demo.yaml \
demo nplus/nplus-environment
```
After that, the K8s namespace is a valid *nplus environment* that can house multiple *nplus instances*.


chapter: deploying assets into the environment

text:
Most likely, you will need assets to be used by your instances. Fonts for example: The *nscale Rendition Server* and die *nscale Server Application Layer* both require the Microsoft fonts, that are not allowed to be distributed by neither nscale nor nplus. So this example shows how to upload some missing pieces into the environment:
```
kubectl cp ./apps/app-installer-9.0.1202.jar nplus-toolbox-0:/conf/pool
kubectl cp ./fonts nplus-toolbox-0:/conf/pool
kubectl cp ./copy-snippet.sh nplus-toolbox-0:/conf/pool/scripts
kubectl cp ./test.md nplus-toolbox-0:/conf/pool/snippets
kubectl cp ./snc nplus-toolbox-0:/conf/pool
```
Alternatively, you can also use a `prepper` component, that you can activate on the environment chart, to download assets from any web site and deploy them into the environment:
```
components:
prepper: true
prepper:
download:
- "https://git.nplus.cloud/public/nplus/raw/branch/master/assets/sample.tar.gz"
```
Please see the prepper [README.md](../../charts/prepper/README.md) for more information.


chapter: Operator Web UI

text:
The environment comes with the operator, responsible for managing / controlling the [custom resources](../cluster/README.md). It has a Web UI, that can be enabled in the environment chart.
![screenshot operator](assets/operator.png)


chapter: *namespace*-less manifests

text:
Speaking of namespaces: Sometimes you want to drop the namespace from your manifest. This can be done by
```yaml
utils:
includeNamespace: false
```
when you then call
```bash
helm template myInstance nplus/nplus-instance > myInstance.yaml
```
the manifest in `myInstance.yaml` will **not** have a namespace set, so you can apply it to multiple namespaces later:
```bash
kubectl apply --namespace dev -f myInstance.yaml
kubectl apply --namespace qa -f myInstance.yaml
kubectl apply --namespace prod -f myInstance.yaml
```


chapter: Creating an empty document area while deploying an Instance

text:
This is the simplest sample, just the core services with an empty document area:
```
helm install \
--values samples/application/empty.yaml \
--values samples/environment/demo.yaml \
empty nplus/nplus-instance
```
The empty Document Area is created with
```yaml
components:
application: true
prepper: true

application:
docAreas:
- id: "Sample"
run:
- "/pool/downloads/sample.sh"
prepper:
download:
- "https://git.nplus.cloud/public/nplus/raw/branch/master/assets/sample.tar.gz"
```
This turns on the *prepper* component, used to download a sample tarball from git. It will also extract the tarball into the `downloads` folder that is created on the *pool* automatically.
Then, after the Application Layer is running, a document area `Sample` is created. The content of the sample script will be executed.
If you use **argoCD** as deployment tool, you would go with
```
helm install \
--values samples/application/empty.yaml \
--values samples/environment/demo.yaml \
empty-argo nplus/nplus-instance-argo
```


chapter: Deploying the *SBS* Apps to a new document area

text:
In the SBS scenario, some Apps are installed into the document area:
```bash
helm install \
--values samples/applications/sbs.yaml \
--values samples/environment/demo.yaml \
sbs nplus/nplus-instance
```
The values look like this:
```yaml
components:
application: true
application:
nameOverride: SBS
docAreas:
- id: "SBS"
name: "DocArea with SBS"
description: "This is a sample DocArea with the SBS Apps installed"
apps:
- "/pool/nstore/bl-app-9.0.1202.zip"
- "/pool/nstore/gdpr-app-9.0.1302.zip"
...
- "/pool/nstore/ts-app-9.0.1302.zip"
- "/pool/nstore/ocr-base-9.0.1302.zip"
```
This will create a document area `SBS` and install the SBS Apps into it.


chapter: Accounting in nstl

text:
To collect Accounting Data in *nscale Server Storage Layer*, you can enable the nstl accouting feature by setting `accounting: true`.
This will create the accounting csv files in *ptemp* under `<instance>/<component>/accounting`.
Additionally, you can enable a log forwarder printing it to stdout.
```
nstl:
accounting: true
logForwarder:
- name: Accounting
path: "/opt/ceyoniq/nscale-server/storage-layer/accounting/*.csv"
```


chapter: (auto-) certificates and the pitfalls of *.this*

text:
*nplus* will automatically generate certificates for your ingress. It either uses an issuer like *cert-manager* or generates a *self-signed-certificate*.
In your production environment though, you might want to take more control over the certificate generation process and don't leave it to *nplus* to automatically take care of it.
In that case, you want to switch the automation *off*.
To do so, you need to understand what is happening internally:
- if `.this.ingress.issuer` is set, the chart requests this issuer to generate a tls secret with the name `.this.ingress.secret`
by creating a certificate resource with the name of the domain `.this.ingress.domain`
- else, so no issuer is set, the chart checks wether the flag `.this.ingress.createSelfSignedCertificate` is set to `true` and
generates a tls secret with the name `.this.ingress.secret`
- else, so neither issuer nor createSelfSignedCertificate are set, the charts will not generate anything
The way how `.this` works is, that it gathers the key from `.Values.global.environment`, `.Values.global` and then `.Values` and flattens them merged into `.this`so that you can set your values
on different levels.
However, the *merge* function overwrites non exising values and also boolean `true` overwrites a boolean `false`, not just the nil values. So to make sure we still can cancel functionality
by setting `null`or `false`, there is a forth merge which is set to forcefully overwrite existing keys: `override`, which can also be set on *environment*, *global* or on the *component* level.
So the correct way to cancel the generation process is to force the issuer to null (which will cancel the *cert-manager* generation) and also force `createSelfSignedCertificate` to false (to cancel the *self-signed-certificate* generation):
```yaml
global:
override:
ingress:
enabled: true
secret: myCertificate
issuer: null
createSelfSignedCertificate: false
```
This makes sure, you will get an ingress, that uses the tls certificate in the secret `myCertificate` for encryption and does not generate anything.


chapter: Grouping Instances

text:
Sometimes Instances become quite large with many components. If you work on them with multiple team members, you end up having to synchronize the deployment of the Instances.
You can easily rip large Instances apart using the `group` tag, joining multiple Instances into one group and making sure the NetworkPolicies are opened to pods from other Instances within the Instance Group.
```yaml
global:
instance:
# -- despite the instance name, all components within this group will be prefixed
# with the group (unless the group name and the environment name are not identical)
# Also this makes sure the network policies are acting on the group, not on the instance.
group: "sample-group"
```
You can query the instance group in your code with `.instance.group`.
Example: We build multiple Instances in one group:
- sample-group-backend
- Database
- nstl
- rs
- sample-group-middleware
- nappl
- application(s)
- sample-group-frontend
- web
- cmis
Portainer is showing the group as if it were an single instance:
![Portainer](assets/portainer.png)
The nplus UI is showing the instances of the group:
![nplus Web Monitoring](assets/monitor.png)
And the nplus CLI is also showing single instances:
```
% kubectl get nscale
NAME                                                                 INSTANCE                         COMPONENT       TYPE            VERSION    STATUS
component.nplus.cloud/sample-group-cmis                              sample-group-frontend            cmis            cmis            9.2.1200   healthy
component.nplus.cloud/sample-group-database                          sample-group-backend             database        database        16         healthy
component.nplus.cloud/sample-group-nappl                             sample-group-middleware          nappl           core            9.2.1302   healthy
component.nplus.cloud/sample-group-rs                                sample-group-backend             rs              rs              9.2.1201   healthy
component.nplus.cloud/sample-group-web                               sample-group-frontend            web             web             9.2.1300   healthy
NAME                                                  HANDLER   VERSION    TENANT   STATUS
instance.nplus.cloud/sample-group-backend             manual    9.2.1302            healthy
instance.nplus.cloud/sample-group-frontend            manual    9.2.1302            healthy
instance.nplus.cloud/sample-group-middleware          manual    9.2.1302            healthy
```


chapter: Sharing Instances

text:
Some organisations have multiple tenants that share common services, like *nscale Rendition Server* or
have a common IT department, thus using only a single *nscale Monitoring Console* acress all tenants.
This is the Central Services Part:
```
helm install \
--values samples/shared/centralservices.yaml \
--values samples/environment/demo.yaml \
sample-shared-cs nplus/nplus-instance
```
And this is the tenant using the Central Services:
```
helm install \
--values samples/shared/shared.yaml \
--values samples/environment/demo.yaml \
sample-shared nplus/nplus-instance
```
If you enable security based on *Network Policies*, you need to add additional Policies to allow access. Please see `shared-networkpolicy.yaml` and `centralservices-networkpolicy.yaml` as an example.
You also want to set the *monitoringInstance* in the `global` section of the values file to enable the Network Policy for incoming monitoring traffic.
```yaml
global:
security:
cni:
monitoringInstance: sample-shared-cs
```


chapter: Using detached applications

text:
All the other samples use an application that is deployed **inside of an instance**. However, you can also deploy an application **detached** from the instance as a solo chart.
The reason for this is, that you
- can update the instance without running the application update
- update the application without touching the instance
- have multiple applications deployed within one instance
There are two major things you need to do:
1. make sure the application charts sets the instance name of the instance, it should connect to
2. take the default values of the application match the ones it would get by an instance deployment
This is a sample: (find the complete one in the [application.yaml](application.yaml))
```yaml
nameOverride: SBS
docAreas:
- id: "SBS"
name: "DocArea with SBS"
description: "This is a sample DocArea with the SBS Apps installed"
apps:
...
instance:
# this is the name of the instance, it should belong to
name: "sample-detached"


chapter: make sure it can wait for the nappl of the instance to be ready, before it deploys.

text:
waitImage:
repo: cr.nplus.cloud/subscription
name: toolbox2
tag: 1.2.1300
pullPolicy: IfNotPresent
waitFor:
- "-service {{ .component.prefix }}nappl.{{ .Release.Namespace }}.svc.cluster.local:{{ .this.nappl.port }} -timeout 1800"


chapter: Now we define where and what to deploy

text:
nappl:
host: "{{ .component.prefix }}nappl.{{ .Release.Namespace }}"
port: 8080
ssl: false
instance: "nscalealinst1"
account: admin
domain: nscale
password: admin
secret:
nstl:
host: "{{ .component.prefix }}nstl.{{ .Release.Namespace }}"
rs:
host: "{{ .component.prefix }}rs.{{ .Release.Namespace }}"
```


chapter: High Availability

text:
To gain a higher level of availability for your Instance, you can
- create more Kubernetes Cluster Nodes
- create more replicas of the *nscale* and *nplus* components
- distribute those replicas across multiple nodes using anti-affinities
This is how:
```
helm install \
--values samples/ha/values.yaml
--values samples/environment/demo.yaml \
sample-ha nplus/nplus-instance
```
The essents of the values file is this:
- We use three (3) *nscale Server Application Layer*, two dedicated to user access, one dedicated to jobs
- if the jobs node fails, the user nodes take the jobs (handled by priority)
- if one of the user nodes fail, the other one handles the load
- Kubernetes takes care of restarting nodes should that happen
- All components run with two replicas
- Pod anti-affinities handle the distribution
- any administration component only connects to the jobs nappl, leaving the user nodes to the users
- PodDisruptionBudgets are defined for the crutial components. These are set via `minReplicaCount` for the components that can support multiple replicas, and `minReplicaCountType` for the **first** replicaSet of the components that do not support replicas, in this case nstla.
```
web:
replicaCount: 2
minReplicaCount: 1
rs:
replicaCount: 2
minReplicaCount: 1
ilm:
replicaCount: 2
minReplicaCount: 1
cmis:
replicaCount: 2
minReplicaCount: 1
webdav:
replicaCount: 2
minReplicaCount: 1
nstla:
minReplicaCountType: 1
administrator:
nappl:
host: "{{ .component.prefix }}nappljobs.{{ .Release.Namespace }}"
waitFor:
- "-service {{ .component.prefix }}nappljobs.{{ .Release.Namespace }}.svc.cluster.local:{{ .this.nappl.port }} -timeout 600"
pam:
nappl:
host: "{{ .component.prefix }}nappljobs.{{ .Release.Namespace }}"
waitFor:
- "-service {{ .component.prefix }}nappljobs.{{ .Release.Namespace }}.svc.cluster.local:{{ .this.nappl.port }} -timeout 600"
nappl:
replicaCount: 2
minReplicaCount: 1
jobs: false
waitFor:
- "-service {{ .component.prefix }}nappljobs.{{ .Release.Namespace }}.svc.cluster.local:{{ .this.nappl.port }} -timeout 600"
nappljobs:
replicaCount: 1
jobs: true
disableSessionReplication: true
ingress:
enabled: false
snc:
enabled: true
waitFor:
- "-service {{ .component.prefix }}database.{{ .Release.Namespace }}.svc.cluster.local:5432 -timeout 600"
application:
nstl:
host: "{{ .component.prefix }}nstl-cluster.{{ .Release.Namespace }}"
nappl:
host: "{{ .component.prefix }}nappljobs.{{ .Release.Namespace }}"
```


chapter: Assigning CPU and RAM

text:
You **should** assign resources to your components, depending on the load that you expect.
In a dev environment, that might be very little and you may be fine with the defaults.
in a qa or prod environment, this should be wisely controlled, like this:
```yaml
nappl:
resources:
requests:
cpu: "100m"         # Minimum 1/10 CPU
memory: "1024Mi"    # Minimum 1 GB
limits:
cpu: "2000m"        # Maximum 2 Cores
memory: "4096Mi"    # Maximum 4 GB. Java will see this as total.
javaOpts:
javaMinMem: "512m"    # tell Java to initialize the heap with 512 MB
javaMaxMem: "2048m"   # tell Java to use max 2 GB of heap size
```
There are many discussions going on how much memory you should give to Java processes and how they react. Please see the internet for insight.


chapter: Our **current** opinion is:

text:
Do not limit ram. You are not able to foresee how much Java is really consuming as the heap is only part of the RAM requirement. Java also needs *metaspace*, *code cache* and *thread stack*. Also the *GC* needs some memory, as well as the *symbols*.
Java will crash when out of memory, so even if you set javaMaxMem == 1/2 limits.memory (what many do), that guarantees nothing and might be a lot of waste.
So what you can consider is:
```yaml
nappl:
resources:
requests:
cpu: "1000m"        # 1 Core guaranteed
memory: "4096Mi"    # 4GB guaranteed
limits:
cpu: "4000m"        # Maximum 4 Cores


chapter: memory:             # No Limit but hardware

text:
javaOpts:
javaMinMem: "1024m"   # Start with 1 GB
javaMaxMem: "3072m"   # Go up to 3GB (which is only part of it) but be able to take more (up to limit) without crash
```
Downside of this approach: If you have a memory leak, it might consume all of your nodes memory without being stopped by a hard limit.


chapter: A possible **Alternative**:

text:
You can set the RAM limit equal to the RAM request and leave the java Memory settings to *automatic*, which basically simulates a server. Java will *see* the limit as being the size of RAM installed in the machine and act accordingly.
```yaml
nappl:
resources:
requests:
cpu: "1000m"        # 1 Core guaranteed
memory: "4096Mi"    # 4GB guaranteed
limits:
cpu: "4000m"        # Maximum 4 Cores
memory: "4096Mi"    # No Limit but hardware


chapter: javaMinMem:           # unset, leaving it to java


chapter: javaMaxMem:           # unset, leaving it to java


chapter: In a **DEV** environment,

text:
you might want to do more **overprovisioning**. You could even leave it completely unlimited, as in **DEV** you want to see memory and cpu leaks, so a limit might hide them from your sight.
So this is a possible allocation for **DEV**, defining only the bare minimum requests:
```yaml
nappl:
resources:
requests:
cpu: "1m"           # 1/1000 Core guaranteed,
# but can consume all cores of the cluster node if required and available
memory: "512Mi"     # 512MB guaranteed,
# but can consume all RAM of the cluster node if required and available
```
In this case, Java will see all node RAM as the limit and use whatever it needs. But as you are in a **dev** environment, there is no load and no users on the machine, so this will not require much.


chapter: Resources you should calculate

text:
The default resources assigned by *nplus* are for demo / testing only and you should definitely assign more ressources to your components.
Here is a very rough estimate of what you need:
| Component       | Minimum (Demo and Dev) | Small            | Medium            | Large              | XL   | Remark                                                      |
| --------------- | ---------------------- | ---------------- | ----------------- | ------------------ | ---- | ----------------------------------------------------------- |
| ADMIN           | 1 GB RAM, 1 Core       | 2 GB RAM, 1 Core | 2 GB RAM, 1 Core  | 2 GB RAM, 1 Core   |      |                                                             |
| **Application** | -                      | -                | -                 | -                  |      | Resources required during deployment only                   |
| CMIS            | 1 GB RAM, 1 Core       | 2 GB RAM, 2 Core | 2 GB RAM, 2 Core  | 2 GB RAM, 2 Core   |      |                                                             |
| **Database**    | 2 GB RAM, 2 Core       | 4 GB RAM, 4 Core | 8 GB RAM, 6 Core  | 16 GB RAM, 8 Core  | open |                                                             |
| ILM             | 1 GB RAM, 1 Core       | 2 GB RAM, 2 Core | 2 GB RAM, 2 Core  | 2 GB RAM, 2 Core   |      |                                                             |
| MON             | 1 GB RAM, 1 Core       | 2 GB RAM, 2 Core | 2 GB RAM, 2 Core  | 2 GB RAM, 2 Core   |      | quite fix                                                   |
| **NAPPL**       | 2 GB RAM, 2 Core       | 4 GB RAM, 4 Core | 8 GB RAM, 6 Core  | 16 GB RAM, 8 Core  | open | CPU depending on Jobs / Hooks, RAM depending on amount user |
| **NSTL**        | 500 MB RAM, 1 Core     | 1 GB RAM, 2 Core | 1 GB RAM, 2 Core  | 1 GB RAM, 2 Core   |      | quite fix                                                   |
| PAM             |                        | 2 GB RAM, 1 Core | 2 GB RAM, 1 Core  | 2 GB RAM, 1 Core   |      |                                                             |
| PIPELINER       | 2 GB RAM, 2 Core       | 4 GB RAM, 4 Core | 4 GB RAM, 4 Core  | 4 GB RAM, 4 Core   | open | Depending on Core Mode *or* AC Mode, No Session Replication |
| **RS**          | 1 GB RAM, 1 Core       | 8 GB RAM, 4 Core | 32 GB RAM, 8 Core | 64 GB RAM, 12 Core | open | CPU depending on format type, RAM depending on file size    |
| SHAREPOINT      |                        | 2 GB RAM, 2 Core | 2 GB RAM, 2 Core  | 2 GB RAM, 2 Core   |      |                                                             |
| WEB             | 1 GB RAM, 1 Core       | 2 GB RAM, 2 Core | 4 GB RAM, 4 Core  | 8 GB RAM, 4 Core   | open |                                                             |
| WEBDAV          |                        | 2 GB RAM, 2 Core | 2 GB RAM, 2 Core  | 2 GB RAM, 2 Core   |      |                                                             |
**Bold** components are required by a *SBS* setup, so here are some estimates per Application:
| Component | Minimum (Demo and Dev) | Minimum (PROD)    | Recommended (PROD) | Remark             |
| --------- | ---------------------- | ----------------- | ------------------ | ------------------ |
| SBS       | 6 GB RAM, 4 Core       | 16 GB RAM, 8 Core | 24 GB RAM, 12 Core | Without WEB Client |
| eGOV      | TODO                   | TODO              | TODO               | eGOV needs much more CPU than a non eGOV system |
A word on **eGOV**: The eGOV App brings hooks and jobs, that require much more resources than a *normal* nscale system even with other Apps installed.


chapter: Real Resources in DEV Idle

text:
```
% kubectl top pods
...
sample-ha-administrator-0                                2m           480Mi
sample-ha-argo-administrator-0                           2m           456Mi
sample-ha-argo-cmis-5ff7d78c47-kgxsn                     2m           385Mi
sample-ha-argo-cmis-5ff7d78c47-whx9j                     2m           379Mi
sample-ha-argo-database-0                                2m           112Mi
sample-ha-argo-ilm-58c65bbd64-pxgdl                      2m           178Mi
sample-ha-argo-ilm-58c65bbd64-tpxfz                      2m           168Mi
sample-ha-argo-mon-0                                     2m           308Mi
sample-ha-argo-nappl-0                                   5m           1454Mi
sample-ha-argo-nappl-1                                   3m           1452Mi
sample-ha-argo-nappljobs-0                               5m           2275Mi
sample-ha-argo-nstla-0                                   4m           25Mi
sample-ha-argo-nstlb-0                                   6m           25Mi
sample-ha-argo-pam-0                                     5m           458Mi
sample-ha-argo-rs-7d6888d9f8-lp65s                       2m           1008Mi
sample-ha-argo-rs-7d6888d9f8-tjxh8                       2m           1135Mi
sample-ha-argo-web-f646f75b8-htn8x                       4m           1224Mi
sample-ha-argo-web-f646f75b8-nvvjf                       11m          1239Mi
sample-ha-argo-webdav-d69549bd4-nz4wn                    2m           354Mi
sample-ha-argo-webdav-d69549bd4-vrg2n                    3m           364Mi
sample-ha-cmis-5fc96b8f89-cwd62                          2m           408Mi
sample-ha-cmis-5fc96b8f89-q4nr4                          3m           442Mi
sample-ha-database-0                                     2m           106Mi
sample-ha-ilm-6b599bc694-5ht57                           2m           174Mi
sample-ha-ilm-6b599bc694-ljkl4                           2m           193Mi
sample-ha-mon-0                                          3m           355Mi
sample-ha-nappl-0                                        3m           1278Mi
sample-ha-nappl-1                                        4m           1295Mi
sample-ha-nappljobs-0                                    6m           1765Mi
sample-ha-nstla-0                                        4m           25Mi
sample-ha-nstlb-0                                        4m           25Mi
sample-ha-pam-0                                          2m           510Mi
sample-ha-rs-7b5fc586f6-49qhp                            2m           951Mi
sample-ha-rs-7b5fc586f6-nkjqb                            2m           1205Mi
sample-ha-web-7bd6ffc96b-pwvcv                           3m           725Mi
sample-ha-web-7bd6ffc96b-rktrh                           9m           776Mi
sample-ha-webdav-9df789f8-2d2wn                          2m           365Mi
sample-ha-webdav-9df789f8-psh5q                          2m           345Mi
...
```


chapter: Defaults

text: Check the file `default.yaml`. You can set default memory limits for a container. These defaults are applied if you do not specify any resources in your manifest.


chapter: Setting Resources for sidecar containers and init containers

text:
You can also set resources for sidecar containers and init containers. However, you should only set these if you know exactly what you are doing and what implications they have.
```yaml
nstl:
sidecarResources:
requests:
cpu: "100m"        # 0.1 Core guaranteed
memory: "1024Mi"   # 1GB guaranteed
limits:
memory: "2048Mi"   # Limit to 2 GB
# we do NOT limit the CPU (read [here](https://home.robusta.dev/blog/stop-using-cpu-limits) for details)
```
Init Resources can be set by using `initResources` key.


chapter: Single-Instance-Mode

text:
If you choose to separate tenants on your system not only by *nplus Instances* but also by *nplus Environments*, thus running each tenant in a separate Kubernetes *Namespace*, you do not need to create an *nplus Environment* first, but you can rather enable the *nplus Environment Components* within your instance:
```yaml
components:
sim:
dav: true
backend: true
operator: true
toolbox: true
```
Steps to run a SIM Instance:
1. Create the namespace and the necessary secrets to access the repo, registry as well as the nscale license file
```
SIM_NAME="empty-sim"
kubectl create ns $SIM_NAME
kubectl create secret docker-registry nscale-cr \
--namespace $SIM_NAME \
--docker-server=ceyoniq.azurecr.io \
--docker-username=$NSCALE_ACCOUNT \
--docker-password=$NSCALE_TOKEN
kubectl create secret docker-registry nplus-cr \
--namespace $SIM_NAME \
--docker-server=cr.nplus.cloud \
--docker-username=$NPLUS_ACCOUNT \
--docker-password=$NPLUS_TOKEN
kubectl create secret generic nscale-license \
--namespace $SIM_NAME \
--from-file=license.xml=$NSCALE_LICENSE
```
2. Deploy the Instance
```
helm install \
--values lab.yaml \
--values single-instance-mode.yaml \
--namespace $SIM_NAME \
$SIM_NAME nplus/nplus-instance
```
If you do not have any Application that requires assets such as scripts or apps, you are good to go with this.
However, if your Application does require assets, the *problem* is to get them into your (not existing) environment before the Applications is trying to access them.
There are three possible solutions:
1. You create an umbrella chart and have a job installing the assets into your Instance
2. You pull / download assets from your git server or an asset server before the Application deployment
3. You pull / download assets from your git server or an asset server before the Component deployment, including the Application
**Solution 1** obiously involes some implementation on your end. That is not covered in this documentation.
**Solution 2** can be achieved by defining a downloader in your application chart (see `empty-download.yaml`):
```yaml
components:
application: true
application:
docAreas:
- id: "Sample"
download:
- "https://git.nplus.cloud/public/nplus/raw/branch/master/samples/assets/sample.sh"
run:
- "/pool/downloads/sample.sh"
```
**Solutions 3** should be used if you have any assets that need to be available **before** the nscale Components start, like snippets for the web client etc.
You can use the *Prepper* for that purpose. The *Prepper* prepares everything required for the Instance to work as intended. It is very much like the *Application*, except that it does not connect to any nscale component (as they do not yet run by the time the prepper executes). But just like the Application, the Prepper is able to download assets and run scripts.
You can add this to your deployment:
```yaml
components:
prepper: true
prepper:
download:
- "https://git.nplus.cloud/public/nplus/raw/branch/master/assets/sample.tar.gz"
run:
- "/pool/downloads/sample/sample.sh"
```


chapter: the argo version of the instance

text:
Deployin with argoCD is straight forward, as there is a ready-to-run instance chart version for argo, that takes **exactly** the same values as the *normal* chart:
```bash
helm install \
--values samples/application/empty.yaml \
--values samples/environment/demo.yaml \
sample-empty-argo nplus/nplus-instance-argo
```


chapter: Using Waves

text:
The instance chart already comes with pre-defined waves. They are good to go with (can be modified though):
```yaml
nappl:
meta:
wave: 15
```
**But**: You might be annoyed by ArgoCD, when some services do not come up preventing other services to not be started at all since ArgoCD operates in Waves, so later services might not be deployed at all if an early wave services fails.
Especially in DEV, this can become a testing problem.
To turn *off* Waves completely for a Stage, Environment or Instance, go
```
global:
environment:
utils:
disableWave: true
```


chapter: Old Version

text:
If you want to test rolling updates and upgrades to a new minor version, check out the *e90* sample:
This sample installs version 9.0.1400 for testing. Because cluster node discovery changed with a newer jGroups version in nscale, the chart detects the older version and enables the legacy discovery mechanism so the Pod can find its peers on versions prior to 9.1.1200.
```
helm install \
--values samples/empty.yaml \
--values samples/demo.yaml \
--values versions/9.0.1400.yaml \
sample-e90 nplus/nplus-instance
```


chapter: New Version Sample

text:
Some nscale versions are license compatible, meaning a 9.1 license file can also run nscale 9.0 software. But that is not always the case.
So you may need to set individual licenses per instance:
```
kubectl create secret generic nscale-license-e10 \
--from-file=license.xml=license10.xml
```
Check that the license has been created:
```


chapter: kubectl get secret | grep license

text:
nscale-license          Opaque              1      7d22h
nscale-license-e10      Opaque              1      17s
```
Now install the instance:
```
helm upgrade -i \
--values samples/empty.yaml \
--values samples/demo.yaml \
--values versions/10.0.yaml \
--set global.license=nscale-license-e10 \
sample-e10 nplus/nplus-instance
```


chapter: Probes

text:
Older versions might not work with the newer/current probes and may not start even though they are actually running. You can override the probes to match older versions, for example:
```
nappl:
probes:
livenessProbe:
httpGet:
path: /index.html
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 3
failureThreshold: 5
readinessProbe:
httpGet:
path: /index.html
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 3
failureThreshold: 5
```


chapter: All the standards

text:
There are several features that will enhance the security of your system:
- all components are running rootless by default
- all components drop all privileges
- all components deny escalation
- all components have read only file systems
- Access is restricted by NetworkPolicies


chapter: Additional: The backend Protocol

text:
Additionally, you can increase security by encrypting communication in the backend. Depending on your network driver, this might already been done automatically beween the Kubernetes Nodes. But you can double that even within a single node by switching the backend Protocol to https:
```yaml
global:
nappl:
port: 8443
ssl: true


chapter: Web and PAM do not speak https by default yet, CRs have been filed.

text:
nappl:
ingress:
backendProtocol: https
cmis:
ingress:
backendProtocol: https
ilm:
ingress:
backendProtocol: https
webdav:
ingress:
backendProtocol: https
rs:
ingress:
backendProtocol: https
mon:
ingress:
backendProtocol: https
administrator:
ingress:
backendProtocol: https
```
This will turn every communication to https, **but** leave the unencrypted ports (http) **open** for inter-pod communication.


chapter: Zero Trust Mode

text:
This will basically do the same as above, **but** also turn **off** any unencrypted port (like http) and also implement NetworkPolicies to drop unencrypted packages.
This will also affect the way how *probes* are checking the pods health: *nplus* will switch them to use https instead, so even the very internal Healtch Check infrastructure will be encrypted in *zero trust mode*:
```yaml
components:
pam: false  #TODO: ITSMSD-8771: PAM does not yet support https backend.
global:
security:
zeroTrust: true
nappl:
port: 8443
ssl: true
```


chapter: (virtual-) Remote Management Server

text:
The *nplus RMS* creates a virtual IP Address in your subnet. On this IP, you will find an *nscale Remote Management Service* and a Layer 4 Proxy, forwarding the ports of the components to the
belonging pods.
The result is, that under this VIP, it looks as if there is a real server with a bunch of *nscale* components installed. So you can use the desktop admin client to connect to it and configure it. Including offline configuration.
The offline configuration writes settings to the configuration files of the components. These files are injected into the Pods by *nplus* making the legacy magic work again.
Also, Shotdown, Startup and Restart buttons in the Admin client will work, as that will by translated to Kubernetes commands by *nplus*
Anyways, there are some restrictions:
- In a HA scenario, you need multiple virtual server, as nscale does not allow some components to deploy more than one instance per server (like nstl) and they would then also block the default ports. So better to have more RMS
- Log Files are not written, so the Admin cannot grab them. So no log file viewing in Admin
> Please notice that this is a BETA Feature not released for Production use.
This is a sample of RMS in a HA environment with two virtual servers:
```yaml
components:
rmsa: true
rmsb: true
rmsa:
ingress:
domain: "server1.{{ .instance.group | default .Release.Name }}.lab.nplus.cloud"
comps:
nappl:
enabled: true
restartReplicas: 2
nstl:
enabled: true
name: nstla
restartReplicas: 1
host: "{{ .component.prefix }}nstla.{{ .Release.Namespace }}.svc.cluster.local"
rs:
enabled: true
restartReplicas: 2
web:
enabled: true
restartReplicas: 2
rmsb:
ingress:
domain: "server2.{{ .instance.group | default .Release.Name }}.lab.nplus.cloud"
comps:
nappl:
enabled: true
name: nappljobs
restartReplicas: 1
replicaSetType: StatefulSet
host: "{{ .component.prefix }}nappljobs.{{ .Release.Namespace }}.svc.cluster.local"
nstl:
name: nstlb
enabled: true
restartReplicas: 1
host: "{{ .component.prefix }}nstlb.{{ .Release.Namespace }}.svc.cluster.local"
```


chapter: Using Object Stores

text:
Blobstores aka Objectstores have a REST Interface that you can upload your Payload to and receive an ID for it. They are normally structured into *Buckets* or *Containers* to privide
some sort of pooling payload within the store.
The *nscale Server Storage Layer* supports multiple brands of objectstores, the most popular being Amazon S3 and Microsoft Azure Blobstore.
In order to use them, you need to
- get an account for the store
- configure the *nstl* with the url, credentials etc.
- Add firewall rules to access to store
Have a look at the sample files
- s3-env.yaml
for Amazon S3 compatible storage, and
- azureblob.yaml
for Azure Blobstore
For S3 compatible storage, there are multiple S3 flavours available.


chapter: Custom Environment Variables

text: There are multiple ways of how to set custom environment variables in addition to the named values, you set in helm:


chapter: Using `env`

text:
Please have a look at `s3-env.yaml` to see how custom environment variables can be injected into a component:
```
nstl:
env:
# Archivtyp
NSTL_ARCHIVETYPE_900_NAME: "S3"
NSTL_ARCHIVETYPE_900_ID: "900"
NSTL_ARCHIVETYPE_900_LOCALMIGRATION: "0"
NSTL_ARCHIVETYPE_900_LOCALMIGRATIONTYPE: "NONE"
NSTL_ARCHIVETYPE_900_S3MIGRATION: "1"
```
This will set the environment variables in the storage layer to add an archive type with id 900.


chapter: Using `envMap` and `envSecret`

text:
Alternatively to the standard `env`setting, you can also use configmaps and secrets for additional environment variables.
The file `s3-envres.yaml` creates a configmap and a secret with the same variables as used in the `s3-env.yaml` sample. `s3-envfrom.yaml` shows how to import them.
Please be aware, that data in secrets need to be base64 encoded:
```
echo "xxx" | base64
```
So in order to use the envFrom mechanism,
- prepare the resources (as in `s3-envres.yaml`)
- upload the resources to your cluster
```
kubectl apply -f s3-envres.yaml
```
- add it to your configuration
```
nstl:
# These resources are set in the s3-envres.yaml sample file
# you can set single values (envMap or envSecret) or lists (envMaps or envSecrets)
envMaps:
- env-sample-archivetype
- env-sample-device
envSecret: env-sample-device-secret
```


chapter: Specifics of the Sharepoint Connector

text: Normally, you will have different configurations if you want multiple Sharepoint Connectors. This makes the *nsp* somewhat special:


chapter: Multi Instance HA Sharepoint Connector

text:
This sample shows how to setup a sharepoint connector with multiple instances having **different** configurations for archival, but with **High Availability** on the retrieval side.
SharePoint is one of the few components for which is is quite common to have multiple instances instead of replicas. Replicas would include, that the configuration for all pods is identical. However, you might want to have multiple configurations as you also have multiple sharepoint sites you want to archive.
Running multiple instances with ingress enabled leads to the question, what the context path is for each instance. It cannot be the same as the load balancer would not be able to distinguish between them and thus refuses to add the configuration object - leading in a deadlock situation.
So *nplus* defined different context paths if you have multiple instances:
- sharepointa on `/nscale_spca`
- sharepointb on `/nscale_spcb`
- sharepointc on `/nscale_spcc`
- sharepointd on `/nscale_spcd`
If you only run one instance, it defaults to `/nscale_spc`.


chapter: HA on retrieval

text:
Once archived, you might want to use all instances for retrieval, since they share a common retrieval configuration (same nappl, ...). So in order to gain High Availability even across multiple instances, there are two options:
1. You turn off the services and ingresses on any sharepoint instance but sharepointa. Then you switch sharepointa's service selector to *type mode*, selecting all pods with type `sharepoint` instead of all pods of component `sharepointa`. Then you can access this one service to reach them all.
2. You can turn on the *clusterService*, which is an additional service that selects all `sharepoint` type pods and then adds an extra ingress on this service with the default context path `nscale_spc`
However, in both scenarios, beware that the sharepoint connector can only service one context path at a time, so you will need to change the context path accordingly.


chapter: Sample for solution 1

text:
On the instance, define the following:
```
components:
# -- First, we switch the default SharePoint OFF
sharepoint: false
# -- Then we enable two sharepoint instances to be used with different configurations
sharepointa: true
sharepointb: true
sharepointa:
service:
# -- Switching the service to "type" makes sure we select not only the component pods (in this case all replicas of sharepointa)
# but rather **any** pod of type sharepoint.
selector: "type"
ingress:
# -- The default contextPath for sharepointa is `nscale_spca` to make sure we have distinguishable paths for all sharepoint instances.
# however, in this case we re-use the service as cluster service and die ingress as cluster ingress, so we switch to the general
# contextPath, as if it was a single component deployment
contextPath: "/nscale_spc"
sharepointb:
service:
# -- The other SP Instance does not need a service any more, as it is selected by the cluster service above. So we switch off the component
# service which also switches off the ingress as it would not have a backing service any more
enabled: false
# -- The default contextPath for sharepointb is `nscale_spcb` to make sure we have distinguishable paths for all sharepoint instances.
# however, in this case we re-use the service as cluster service and die ingress as cluster ingress, so we switch to the general
# contextPath, as if it was a single component deployment
contextPath: "/nscale_spc"
```


chapter: Sample for Solution 2

text:
On the instance, define the following:
```
components:
# -- First, we switch the default SharePoint OFF
sharepoint: false
# -- Then we enable two sharepoint instances to be used with different configurations
sharepointa: true
sharepointb: true
sharepointa:
clusterService:
# -- This enabled the cluster service
enabled: true
# -- the cluster Ingress needs to know the context path it should react on.
contextPath: "/nscale_spc"
ingress:
# -- we turn off the original ingress as the common context path would block the deployment
enabled: false
# -- The default contextPath for sharepointa is `nscale_spca` to make sure we have distinguishable paths for all sharepoint instances.
# however, in this case we re-use the service as cluster service and die ingress as cluster ingress, so we switch to the general
# contextPath, as if it was a single component deployment
contextPath: "/nscale_spc"
sharepointb:
clusterService:
# -- on the second SharePoint Instance, we **disable** the cluster service, as it is already created by sharepointa.
enabled: false
# -- however, we need to set the context path, as this tells the networkPolicy to open up for ingress even though we switch die Ingress off in the
# next step
contextPath: "/nscale_spc"
ingress:
# -- we turn off the original ingress as the common context path would block the deployment
enabled: false
# -- The default contextPath for sharepointb is `nscale_spcb` to make sure we have distinguishable paths for all sharepoint instances.
# however, in this case we re-use the service as cluster service and die ingress as cluster ingress, so we switch to the general
# contextPath, as if it was a single component deployment
contextPath: "/nscale_spc"
```


chapter: Assigning PVs

text:
For security reasons, you might want to use a storage class that does not perform automatic provisioning of PVs.
In that case, you want to reference a pre-created volume in the PVC.
In nplus, you can do so by setting the volumeName in the values.
Please review `values.yaml` as an example:
```yaml
database:
mounts:
data:
volumeName: "pv-{{ .component.fullName }}-data"
nstl:
mounts:
data:
volumeName: "pv-{{ .component.fullName }}-data"
```
You can also set the environment config volume. Please refer to the environment documentation for that.
```
helm install \
--values samples/environment/demo.yaml \
--values samples/static/values.yaml
sample-static nplus/nplus-instance
```


chapter: Data Disk:

text:
1. Create a pool on your cep cluster
```
ceph osd pool create k-lab 64 64
```
2. Create a block device pool
```
rbd pool init k-lab
```
3. Create an image
```
rbd create -s 50G k-lab/pv-sample-static-database-data
rbd create -s 50G k-lab/pv-sample-static-nstl-data
rbd ls k-lab | grep pv-sample-static-
```
Resize:
```
rbd resize --size 50G k-lab/pv-no-provisioner-database-data --allow-shrink
```


chapter: File Share:

text:
1. Create a Subvolume (FS)
```
ceph fs subvolume create cephfs pv-no-provisioner-rs-file --size 53687091200
```
2. Get the path of the subvolume
```
ceph fs subvolume getpath  cephfs pv-no-provisioner-rs-file
```


chapter: Troubleshooting

text:
```
kubectl describe pv/pv-no-provisioner-rs-file pvc/no-provisioner-rs-file
kubectl get volumeattachment
```


chapter: PV Manifests

text:
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-no-provisioner-database-data
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 50Gi
csi:
driver: rook-ceph.rbd.csi.ceph.com
fsType: ext4
nodeStageSecretRef:
# node stage secret name
name: rook-csi-rbd-node
# node stage secret namespace where above secret is created
namespace: rook-ceph-external
volumeAttributes:
# Required options from storageclass parameters need to be added in volumeAttributes
clusterID: rook-ceph-external
pool: k-lab
staticVolume: "true"
imageFeatures: layering
#mounter: rbd-nbd
# volumeHandle should be same as rbd image name
volumeHandle: pv-no-provisioner-database-data
persistentVolumeReclaimPolicy: Retain
# The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
volumeMode: Filesystem
storageClassName: ceph-rbd
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-no-provisioner-nstl-data
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 50Gi
csi:
driver: rook-ceph.cephfs.csi.ceph.com
fsType: ext4
nodeStageSecretRef:
# node stage secret name
name: rook-csi-rbd-node
# node stage secret namespace where above secret is created
namespace: rook-ceph-external
volumeAttributes:
# Required options from storageclass parameters need to be added in volumeAttributes
clusterID: rook-ceph-external
pool: k-lab
staticVolume: "true"
imageFeatures: layering
#mounter: rbd-nbd
# volumeHandle should be same as rbd image name
volumeHandle: pv-no-provisioner-nstl-data
persistentVolumeReclaimPolicy: Retain
# The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
volumeMode: Filesystem
storageClassName: ceph-rbd
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-no-provisioner-rs-file
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 50Gi
csi:
driver: cephfs.csi.ceph.com
nodeStageSecretRef:
name: rook-csi-cephfs-secret
#rook-csi-cephfs-node
namespace: rook-ceph-external
volumeAttributes:
# Required options from storageclass parameters need to be added in volumeAttributes
clusterID: rook-ceph-external
fsName: cephfs
pool: cephfs_data
staticVolume: "true"
# rootPath kriegt man per ceph fs subvolume getpath  cephfs pv-no-provisioner-rs-file
rootPath: "/volumes/_nogroup/pv-no-provisioner-rs-file/3016f512-bc19-4bfb-8eb2-5118430fbbe5"
#mounter: rbd-nbd
# volumeHandle should be same as rbd image name
volumeHandle: pv-no-provisioner-rs-file
persistentVolumeReclaimPolicy: Retain
# The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
volumeMode: Filesystem
storageClassName: cephfs
```