Project

General

Profile

Distributed Icinga2 » History » Version 6

Jon Goldberg, 09/18/2017 04:09 PM

1 3 Jon Goldberg
{{last_updated_at}} by {{last_updated_by}}
2 1 Jon Goldberg
# Distributed Icinga2
3
4
### Overview
5
6
With a single instance of Icinga2, we can check the status of remote services that are publicly accessible.  For instance, we can check if SSH responds on a remote host.  However, we can NOT check information that is not publicly accessible - for instance, whether the remote server has low disk space.
7
8
To handle this, we use a distributed Icinga2 infrastructure.  Our original Icinga instance becomes a **master** instance.  We also install a minimal version of Icinga on remote servers, known as **satellite** instances.  Icinga2 has a wizard that establishes encrypted communication links between servers.  We use a [top-down configuration](https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/distributed-monitoring#distributed-monitoring-top-down), which means that all checks are initiated from the master instance - but some service checks are configured to run on a different endpoint.
9
10
In our instance, that means that some checks run entirely on the master - e.g. an SSH check - but a "low disk space" check is scheduled on the master instance, which contacts the satellite, instructs it to run the plugin, and return the result.
11
### Installing Icinga2 as a Satellite
12
13
Source: https://www.olindata.com/blog/2015/03/monitoring-remote-systems-icinga-2
14
15
Run these commands on the satellite (assumes the satellite OS is Ubuntu)
16 2 Jon Goldberg
17 4 Jon Goldberg
```bash
18 1 Jon Goldberg
apt install software-properties-common
19
add-apt-repository ppa:formorer/icinga
20
apt update
21
apt install icinga2
22
systemctl enable icinga2.service
23
systemctl start icinga2.service
24
# Get the salt from the master node in /etc/icinga2/constants.conf
25
icinga2 pki ticket --cn 'icinga.jmaconsulting.biz' --salt <salt goes here>
26
icinga2 node wizard
27
```
28
Alternate instructions for Debian Jessie:
29 2 Jon Goldberg
30 4 Jon Goldberg
```bash
31 1 Jon Goldberg
# Note: Ensure you have jessie-backports enabled: https://backports.debian.org/Instructions/
32
apt install software-properties-common
33
apt-get -t jessie-backports install icinga2
34
# Debian installs some some host definitions that interfere with being a satellite; delete them
35 5 Joseph Lacey
rm /etc/icinga2/conf.d/hosts.conf
36 1 Jon Goldberg
systemctl enable icinga2.service
37
systemctl start icinga2.service
38
# Get the salt from the master node in /etc/icinga2/constants.conf
39
icinga2 pki ticket --cn 'orange.megaphonetech.com' --salt <salt goes here>
40
icinga2 node wizard
41
```
42
Here's an example of the node wizard setup for a red.megaphonetech.com:
43 2 Jon Goldberg
44 1 Jon Goldberg
```
45
Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: 
46
Starting the Node setup routine...
47
Please specify the common name (CN) [rh6.jmaconsulting.biz]: 
48
Please specify the master endpoint(s) this node should connect to:
49
Master Common Name (CN from your master setup): icinga.jmaconsulting.biz
50
Do you want to establish a connection to the master from this node? [Y/n]: 
51
Please fill out the master connection information:
52
Master endpoint host (Your master's IP address or FQDN): icinga.jmaconsulting.biz
53
Master endpoint port [5665]: 
54
Add more master endpoints? [y/N]: 
55
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
56
Host [icinga.jmaconsulting.biz]: 
57
Port [5665]: 
58
information/base: Writing private key to '/etc/icinga2/pki/rh6.jmaconsulting.biz.key'.
59
information/base: Writing X509 certificate to '/etc/icinga2/pki/rh6.jmaconsulting.biz.crt'.
60
information/cli: Fetching public certificate from master (icinga.jmaconsulting.biz, 5665):
61
62
Certificate information:
63
64
 Subject:     CN = icinga.jmaconsulting.biz
65
 Issuer:      CN = Icinga CA
66
 Valid From:  Jan 25 23:12:52 2017 GMT
67
 Valid Until: Jan 22 23:12:52 2032 GMT
68
 Fingerprint: B7 DC AF 0C F7 B2 EF DD C9 8F 34 99 5C C2 9A 86 57 9C D2 B0 
69
70
Is this information correct? [y/N]: Y
71
information/cli: Received trusted master certificate.
72
73
Please specify the request ticket generated on your Icinga 2 master.
74
 (Hint: # icinga2 pki ticket --cn 'rh6.jmaconsulting.biz'): #Run this command on the master to generate a ticket number
75
information/cli: Requesting certificate with ticket '00a2aff74b3b3145630504276912a9addd714810'.
76
77
information/cli: Created backup file '/etc/icinga2/pki/rh6.jmaconsulting.biz.crt.orig'.
78
information/cli: Writing signed certificate to file '/etc/icinga2/pki/rh6.jmaconsulting.biz.crt'.
79
information/cli: Writing CA certificate to file '/etc/icinga2/pki/ca.crt'.
80
Please specify the API bind host/port (optional):
81
Bind Host []: 
82
Bind Port []: 
83
Accept config from master? [y/N]: Y
84
Accept commands from master? [y/N]: Y
85
information/cli: Disabling the Notification feature.
86
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
87
information/cli: Enabling the Apilistener feature.
88
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
89
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
90
information/cli: Generating local zones.conf.
91
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
92
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
93
information/cli: Updating constants.conf.
94
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
95
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
96
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
97
Done.
98
```
99
Now run `service icinga2 checkconfig && service icinga2 restart` to finish the installation on the satellite.
100
### Create Zones and Endpoints
101
102
You must also set up the `Zone` and `Endpoint` objects, stored in `/etc/icinga2/zones.conf`. In our model, each zone contains one and only one endpoint.
103
#### Zones and Endpoints on the Satellite
104
105
The node wizard mostly handles this for you on the satellite, it will generate a file with these constants.  Note that NodeName and ZoneName are defined in /etc/icinga2/constants.conf, and should both be the name of the host.
106 2 Jon Goldberg
107 1 Jon Goldberg
```
108
object Endpoint "orange.megaphonetech.com" {
109
        host = "orange.megaphonetech.com"
110
        port = "5665"
111
}
112
113
object Zone "master" {
114
        endpoints = [ "orange.megaphonetech.com" ]
115
}
116
117
object Endpoint NodeName {
118
}
119
120
object Zone ZoneName {
121
        endpoints = [ NodeName ]
122
        parent = "master"
123
}
124
```
125
Add the following Zone object on your satellite.  Files in this zone are synced to all Icinga2 instances, so you can distribute many configuration files this way:
126 2 Jon Goldberg
127 1 Jon Goldberg
```
128
/* sync global commands */
129
object Zone "global-templates" {
130
        global = true
131
}
132
```
133
#### Zones and Endpoints on the Master
134
135
Edit `/etc/icinga2/zones.conf`.  Add the Zone and Endpoint of the new satellite.  For instance, if the satellite has a ZoneName and NodeName of database.lavabrooklyn.org defined in its constants /etc/icinga2/constants.conf then you should add these objects on the master:
136 2 Jon Goldberg
137 1 Jon Goldberg
```
138
object Endpoint "database.lavabrooklyn.org" {
139
}
140
141
object Zone "database.lavabrooklyn.org" {
142
        endpoints = [ "database.lavabrooklyn.org" ]
143
        parent = "master"
144
}
145
```
146 6 Jon Goldberg
#### Harden the TLS connection
147
Add this line to the bottom of the "api" object in `/etc/icinga2/features-available/api.conf`:
148
```
149
tls_protocolmin = "TLSv1.2"
150
```
151
152 1 Jon Goldberg
#### Testing the Zones/Endpoints
153
154
After reloading Icinga2 on both satellite and master, you should be able to see the new checks reflected in Icingaweb2 immediately, though they might be listed as "Pending".  Click "Check now" to ensure they're working as expected.
155
### How to change a satellite's hostname
156
157
    Re-run `icinga2 node wizard` on the satellite.  This will fix the satellite's "constants.conf" and "zones.conf".
158
    Edit `/etc/icinga2/zones.conf` Zone and Endpoint on the master to change the name.