330 lines
14 KiB
Markdown
330 lines
14 KiB
Markdown
|
# NATS Server
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
[NATS](https://nats.io) is a simple, secure and performant communications system for digital systems, services and devices.
|
||
|
|
NATS is part of the Cloud Native Computing Foundation ([CNCF](https://cncf.io)).
|
||
|
|
NATS has over [30 client language implementations](https://nats.io/download/), and its server can run on-premise, in the cloud, at the edge, and even on a Raspberry Pi.
|
||
|
|
NATS can secure and simplify design and operation of modern distributed systems.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
|
||
|
|
helm upgrade --install nats nats/nats
|
||
|
|
```
|
||
|
|
|
||
|
|
## Upgrade Nodes
|
||
|
|
|
||
|
|
- **Upgrading from 0.x**: The `values.yaml` schema changed significantly from 0.x to 1.x. Read [UPGRADING.md](UPGRADING.md) for instructions on upgrading a 0.x release to 1.x.
|
||
|
|
|
||
|
|
## Values
|
||
|
|
|
||
|
|
There are a handful of explicitly defined options which are documented with comments in the [values.yaml](values.yaml) file.
|
||
|
|
|
||
|
|
Everything in the NATS Config or Kubernetes Resources can be overridden by `merge` and `patch`, which is supported for the following values:
|
||
|
|
|
||
|
|
| key | type | enabled by default |
|
||
|
|
|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
|
||
|
|
| `config` | [NATS Config](https://docs.nats.io/running-a-nats-service/configuration) | yes |
|
||
|
|
| `config.cluster` | [NATS Cluster](https://docs.nats.io/running-a-nats-service/configuration/clustering/cluster_config) | no |
|
||
|
|
| `config.cluster.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
|
||
|
|
| `config.jetstream` | [NATS JetStream](https://docs.nats.io/running-a-nats-service/configuration#jetstream) | no |
|
||
|
|
| `config.jetstream.fileStore.pvc` | [k8s PVC](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core) | yes, when `config.jetstream` is enabled |
|
||
|
|
| `config.nats.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
|
||
|
|
| `config.leafnodes` | [NATS LeafNodes](https://docs.nats.io/running-a-nats-service/configuration/leafnodes/leafnode_conf) | no |
|
||
|
|
| `config.leafnodes.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
|
||
|
|
| `config.websocket` | [NATS WebSocket](https://docs.nats.io/running-a-nats-service/configuration/websocket/websocket_conf) | no |
|
||
|
|
| `config.websocket.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
|
||
|
|
| `config.websocket.ingress` | [k8s Ingress](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#ingress-v1-networking-k8s-io) | no |
|
||
|
|
| `config.mqtt` | [NATS MQTT](https://docs.nats.io/running-a-nats-service/configuration/mqtt/mqtt_config) | no |
|
||
|
|
| `config.mqtt.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
|
||
|
|
| `config.gateway` | [NATS Gateway](https://docs.nats.io/running-a-nats-service/configuration/gateways/gateway#gateway-configuration-block) | no |
|
||
|
|
| `config.gateway.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
|
||
|
|
| `config.resolver` | [NATS Resolver](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/jwt/resolver) | no |
|
||
|
|
| `config.resolver.pvc` | [k8s PVC](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core) | yes, when `config.resolver` is enabled |
|
||
|
|
| `container` | nats [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | yes |
|
||
|
|
| `reloader` | config reloader [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | yes |
|
||
|
|
| `promExporter` | prometheus exporter [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | no |
|
||
|
|
| `promExporter.podMonitor` | [prometheus PodMonitor](https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1.PodMonitor) | no |
|
||
|
|
| `service` | [k8s Service](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) | yes |
|
||
|
|
| `statefulSet` | [k8s StatefulSet](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#statefulset-v1-apps) | yes |
|
||
|
|
| `podTemplate` | [k8s PodTemplate](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core) | yes |
|
||
|
|
| `headlessService` | [k8s Service](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) | yes |
|
||
|
|
| `configMap` | [k8s ConfigMap](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#configmap-v1-core) | yes |
|
||
|
|
| `natsBox.contexts.default` | [NATS Context](https://docs.nats.io/using-nats/nats-tools/nats_cli#nats-contexts) | yes |
|
||
|
|
| `natsBox.contexts.[name]` | [NATS Context](https://docs.nats.io/using-nats/nats-tools/nats_cli#nats-contexts) | no |
|
||
|
|
| `natsBox.container` | nats-box [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | yes |
|
||
|
|
| `natsBox.deployment` | [k8s Deployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#deployment-v1-apps) | yes |
|
||
|
|
| `natsBox.podTemplate` | [k8s PodTemplate](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core) | yes |
|
||
|
|
| `natsBox.contextsSecret` | [k8s Secret](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core) | yes |
|
||
|
|
| `natsBox.contentsSecret` | [k8s Secret](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core) | yes |
|
||
|
|
|
||
|
|
### Merge
|
||
|
|
|
||
|
|
Merging is performed using the Helm `merge` function. Example - add NATS accounts and container resources:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
merge:
|
||
|
|
accounts:
|
||
|
|
A:
|
||
|
|
users:
|
||
|
|
- {user: a, password: a}
|
||
|
|
B:
|
||
|
|
users:
|
||
|
|
- {user: b, password: b}
|
||
|
|
natsBox:
|
||
|
|
contexts:
|
||
|
|
a:
|
||
|
|
merge: {user: a, password: a}
|
||
|
|
b:
|
||
|
|
merge: {user: b, password: b}
|
||
|
|
defaultContextName: a
|
||
|
|
```
|
||
|
|
|
||
|
|
## Patch
|
||
|
|
|
||
|
|
Patching is performed using [JSON Patch](https://jsonpatch.com/). Example - add additional route to end of route list:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
cluster:
|
||
|
|
enabled: true
|
||
|
|
patch:
|
||
|
|
- op: add
|
||
|
|
path: /routes/-
|
||
|
|
value: nats://demo.nats.io:6222
|
||
|
|
```
|
||
|
|
|
||
|
|
## Common Configurations
|
||
|
|
|
||
|
|
### JetStream Cluster on 3 separate hosts
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
cluster:
|
||
|
|
enabled: true
|
||
|
|
replicas: 3
|
||
|
|
jetstream:
|
||
|
|
enabled: true
|
||
|
|
fileStore:
|
||
|
|
pvc:
|
||
|
|
size: 10Gi
|
||
|
|
|
||
|
|
podTemplate:
|
||
|
|
topologySpreadConstraints:
|
||
|
|
kubernetes.io/hostname:
|
||
|
|
maxSkew: 1
|
||
|
|
whenUnsatisfiable: DoNotSchedule
|
||
|
|
```
|
||
|
|
|
||
|
|
### NATS Container Resources
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
container:
|
||
|
|
env:
|
||
|
|
# different from k8s units, suffix must be B, KiB, MiB, GiB, or TiB
|
||
|
|
# should be ~90% of memory limit
|
||
|
|
GOMEMLIMIT: 7GiB
|
||
|
|
merge:
|
||
|
|
# recommended limit is at least 2 CPU cores and 8Gi Memory for production JetStream clusters
|
||
|
|
resources:
|
||
|
|
requests:
|
||
|
|
cpu: "2"
|
||
|
|
memory: 8Gi
|
||
|
|
limits:
|
||
|
|
cpu: "2"
|
||
|
|
memory: 8Gi
|
||
|
|
```
|
||
|
|
|
||
|
|
### Specify Image Version
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
container:
|
||
|
|
image:
|
||
|
|
tag: x.y.z-alpine
|
||
|
|
```
|
||
|
|
|
||
|
|
### Operator Mode with NATS Resolver
|
||
|
|
|
||
|
|
Run `nsc generate config --nats-resolver` and replace the `OPERATOR_JWT`, `SYS_ACCOUNT_ID`, and `SYS_ACCOUNT_JWT` with your values.
|
||
|
|
Make sure that you do not include the trailing `,` in the `SYS_ACCOUNT_JWT`.
|
||
|
|
|
||
|
|
```
|
||
|
|
config:
|
||
|
|
resolver:
|
||
|
|
enabled: true
|
||
|
|
merge:
|
||
|
|
type: full
|
||
|
|
interval: 2m
|
||
|
|
timeout: 1.9s
|
||
|
|
merge:
|
||
|
|
operator: OPERATOR_JWT
|
||
|
|
system_account: SYS_ACCOUNT_ID
|
||
|
|
resolver_preload:
|
||
|
|
SYS_ACCOUNT_ID: SYS_ACCOUNT_JWT
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
## Accessing NATS
|
||
|
|
|
||
|
|
The chart contains 2 services by default, `service` and `headlessService`.
|
||
|
|
|
||
|
|
### `service`
|
||
|
|
|
||
|
|
The `service` is intended to be accessed by NATS Clients. It is a `ClusterIP` service by default, however it can easily be changed to a different service type.
|
||
|
|
|
||
|
|
The `nats`, `websocket`, `leafnodes`, and `mqtt` ports will be exposed through this service by default if they are enabled.
|
||
|
|
|
||
|
|
Example: change this service type to a `LoadBalancer`:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
service:
|
||
|
|
merge:
|
||
|
|
spec:
|
||
|
|
type: LoadBalancer
|
||
|
|
```
|
||
|
|
|
||
|
|
### `headlessService`
|
||
|
|
|
||
|
|
The `headlessService` is used for NATS Servers in the Stateful Set to discover one another. It is primarily intended to be used for Cluster Route connections.
|
||
|
|
|
||
|
|
### TLS Considerations
|
||
|
|
|
||
|
|
The TLS Certificate used for Client Connections should have a SAN covering DNS Name that clients access the `service` at.
|
||
|
|
|
||
|
|
The TLS Certificate used for Cluster Route Connections should have a SAN covering the DNS Name that routes access each other on the `headlessService` at. This is `*.<headless-service-name>` by default.
|
||
|
|
|
||
|
|
## Advanced Features
|
||
|
|
|
||
|
|
### Templating Values
|
||
|
|
|
||
|
|
Anything in `values.yaml` can be templated:
|
||
|
|
|
||
|
|
- maps matching the following syntax will be templated and parsed as YAML:
|
||
|
|
```yaml
|
||
|
|
$tplYaml: |
|
||
|
|
yaml template
|
||
|
|
```
|
||
|
|
- maps matching the follow syntax will be templated, parsed as YAML, and spread into the parent map/slice
|
||
|
|
```yaml
|
||
|
|
$tplYamlSpread: |
|
||
|
|
yaml template
|
||
|
|
```
|
||
|
|
|
||
|
|
Example - change service name:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
service:
|
||
|
|
name:
|
||
|
|
$tplYaml: >-
|
||
|
|
{{ include "nats.fullname" . }}-svc
|
||
|
|
```
|
||
|
|
|
||
|
|
### NATS Config Units and Variables
|
||
|
|
|
||
|
|
NATS configuration extends JSON, and can represent Units and Variables. They must be wrapped in `<< >>` in order to template correctly. Example:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
merge:
|
||
|
|
authorization:
|
||
|
|
# variable
|
||
|
|
token: << $TOKEN >>
|
||
|
|
# units
|
||
|
|
max_payload: << 2MB >>
|
||
|
|
```
|
||
|
|
|
||
|
|
templates to the `nats.conf`:
|
||
|
|
|
||
|
|
```
|
||
|
|
{
|
||
|
|
"authorization": {
|
||
|
|
"token": $TOKEN
|
||
|
|
},
|
||
|
|
"max_payload": 2MB,
|
||
|
|
"port": 4222,
|
||
|
|
...
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
### NATS Config Includes
|
||
|
|
|
||
|
|
Any NATS Config key ending in `$include` will be replaced with an include directive. Included files should be in paths relative to `/etc/nats-config`. Multiple `$include` keys are supported by using a prefix, and will be sorted alphabetically. Example:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
merge:
|
||
|
|
00$include: auth.conf
|
||
|
|
01$include: params.conf
|
||
|
|
configMap:
|
||
|
|
merge:
|
||
|
|
data:
|
||
|
|
auth.conf: |
|
||
|
|
accounts: {
|
||
|
|
A: {
|
||
|
|
users: [
|
||
|
|
{user: a, password: a}
|
||
|
|
]
|
||
|
|
},
|
||
|
|
B: {
|
||
|
|
users: [
|
||
|
|
{user: b, password: b}
|
||
|
|
]
|
||
|
|
},
|
||
|
|
}
|
||
|
|
params.conf: |
|
||
|
|
max_payload: 2MB
|
||
|
|
```
|
||
|
|
|
||
|
|
templates to the `nats.conf`:
|
||
|
|
|
||
|
|
```
|
||
|
|
include auth.conf;
|
||
|
|
"port": 4222,
|
||
|
|
...
|
||
|
|
include params.conf;
|
||
|
|
```
|
||
|
|
|
||
|
|
### Extra Resources
|
||
|
|
|
||
|
|
Enables adding additional arbitrary resources. Example - expose WebSocket via VirtualService in Istio:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
websocket:
|
||
|
|
enabled: true
|
||
|
|
extraResources:
|
||
|
|
- apiVersion: networking.istio.io/v1beta1
|
||
|
|
kind: VirtualService
|
||
|
|
metadata:
|
||
|
|
namespace:
|
||
|
|
$tplYamlSpread: >
|
||
|
|
{{ include "nats.metadataNamespace" $ }}
|
||
|
|
name:
|
||
|
|
$tplYaml: >
|
||
|
|
{{ include "nats.fullname" $ | quote }}
|
||
|
|
labels:
|
||
|
|
$tplYaml: |
|
||
|
|
{{ include "nats.labels" $ }}
|
||
|
|
spec:
|
||
|
|
hosts:
|
||
|
|
- demo.nats.io
|
||
|
|
gateways:
|
||
|
|
- my-gateway
|
||
|
|
http:
|
||
|
|
- name: default
|
||
|
|
match:
|
||
|
|
- name: root
|
||
|
|
uri:
|
||
|
|
exact: /
|
||
|
|
route:
|
||
|
|
- destination:
|
||
|
|
host:
|
||
|
|
$tplYaml: >
|
||
|
|
{{ .Values.service.name | quote }}
|
||
|
|
port:
|
||
|
|
number:
|
||
|
|
$tplYaml: >
|
||
|
|
{{ .Values.config.websocket.port }}
|
||
|
|
```
|