Server Documentation: database.server.england

System Overview

• Hostname: database.server.england
• IP Address: 192.168.0.143
• Primary Role: Backend Data Store for File Encryption Software and License Management.
• Storage Engine: ZFS (VaultDB Pool) with MariaDB 10.11.

---

Core Infrastructure Services

This server manages sensitive cryptographic metadata and licensing keys. Due to high-security requirements, several custom controllers manage the application lifecycle.

Service	Component	Description
MariaDB	mariadb.service	Primary SQL database server listening on port 46317.
Fail2Ban	fail2ban.service	Intrusion prevention protecting SSH and database ports.
ZFS Event Daemon	zfs-zed.service	Monitors pool health and automates snapshot events.
License Sync	license-db-sync.service	Automates two-way synchronization between local MariaDB and Aiven Cloud DB.
SSH	ssh.service	Administrative access (Port 22).

---

The Heartbeat & Middleman Logic

A specialized coordination system is used to manage the server's availability and listener states.

• heartbeat-controller.service: A Java-based application that manages system state and potential shutdowns.
• heartbeat.service: A UDP-based listener providing connectivity signals.
• heartbeat-middleman.sh: A critical orchestration script used to switch between the controller and the heartbeat server. It ensures that port 46317 is properly released and re-bound during transitions to avoid bind failures.
• boot-manager.sh: Executed during the boot sequence to detect maintenance windows (08:00 and 22:00-23:00). If a window is detected, it preemptively stops the heartbeat-controller to allow for uninterrupted updates and backups.

---

Data Integrity & Backup Strategy

The database utilizes a tiered ZFS backup approach to ensure zero data loss.

1. Local ZFS Snapshots (db_zfs_backup)

A nightly cron job at 23:00 executes a differential check. It compares the current state of vaultdb/mariadb against the most recent snapshot. A new snapshot is only created if changes are detected, keeping the last three snapshots for recovery.

2. Cloud Synchronization (sync_license_db)

The system maintains a bidirectional sync with a remote Aiven Cloud MySQL instance. This ensures that license keys generated externally are pulled to the local lab, and local updates are pushed to the cloud.

3. Integrity Audits

• Monthly Scrub: A zpool scrub is triggered on the 1st of every month at 23:00 for both vaultdb and vaultbackup.
• Login Audit: zfs_login_check.sh provides an immediate health and capacity summary of ZFS pools and the /vaultbackup mount point upon every SSH login.

---

Scheduled Operations (Cron)

Schedule	Command	Function
@reboot	daily_apt_check.sh	Initial package audit on startup.
08:20	systemctl poweroff	Automated morning shutdown to conserve resources post-maintenance.
23:00	db_zfs_backup	Differential ZFS snapshot of the MariaDB dataset.
00:00	systemctl poweroff	Primary nightly shutdown.

---

Network Configuration and Listening Ports

Port	Protocol	Service	Note
46317	TCP	MariaDB	Primary Database Listener.
46317*	UDP	Heartbeat	*Subject to Middleman Orchestration. Conflict managed by heartbeat-middleman.sh.
22	TCP	SSH	Administrative Access.
53	TCP/UDP	systemd-resolved	Local Name Resolution.

---

Critical Configuration Paths

• MariaDB Data (ZFS): /vaultdb/mariadb
• Backup Archives: /vaultbackup/mariadb_backups
• Custom Binaries: /usr/local/bin/
• Sync Logs: /var/log/license_db_sync.log
• Backup Logs: /var/log/zfs_backup.log

Live Status

Raw Hardware Specifications

System:
  Kernel: redacted arch
  x86_64 bits: 64
  compiler:
  gcc v: 13.3.0
  Console: pty pts/0
    Distro: Ubuntu 24.04.4 LTS (Noble Numbat)
Machine:
  Type: Server System: HP product: ProLiant DL160 G6 v: N/A
  serial: redacted
  Mobo: N/A model: N/A serial: N/A
  BIOS: HP v: redacted
    date: redacted
CPU:
  Info: 2x 4-core model: Intel Xeon E5606 bits: 64 type: MCP SMP arch: Nehalem rev: 2
  cache:
    L1: 2x 256 KiB (512 KiB)
    L2: 2x 1024 KiB (2 MiB)
    L3: 2x 8 MiB (16 MiB)
  Speed (MHz): avg: 1283 high: 1600 min/max: 1200/2133 cores: 1: 1200 2: 1200 3: 1200 4: 1200 5: 1200 6: 1200 7: 1467 8: 1600 bogomips: 34134
  Flags: ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Graphics:
  Device-1: Matrox Systems MGA G200e [Pilot] ServerEngines vendor: Hewlett-Packard driver: mgag200 v: kernel bus-ID: 02:00.0
  Display: server: No display server data found. Headless machine? tty: 235x60
  API: EGL v: 1.5 drivers: swrast platforms: active: surfaceless,device inactive: gbm,wayland,x11
  API: OpenGL v: 4.5 vendor: mesa v: 25.2.8-0ubuntu0.24.04.1 note: console (EGL sourced)
    renderer: llvmpipe (LLVM 20.1.2 128 bits)
Audio:
  Message: No device data found.
  API: ALSA v: k6.8.0-110-generic status: inactive
Network:
  Device-1: Intel 82576 Gigabit Network vendor: Hewlett-Packard NC362i Dual port Server driver: igb v: kernel port: e880 bus-ID: 05:00.0
  IF: enp5s0f0 state: up speed: 1000 Mbps duplex: full
  mac: redacted
  Device-2: Intel 82576 Gigabit Network vendor: Hewlett-Packard NC362i Dual port Server driver: igb v: kernel port: ec00 bus-ID: 05:00.1
  IF: enp5s0f1 state: down
  mac: redacted
RAID:
  Hardware-1: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] driver: mpt3sas v: 43.100.00.00 bus-ID: 04:00.0
  Device-1: vaultbackup type: zfs status: ONLINE level: mirror-0
  raw:
    size: 2.72 TiB
    free: 2.72 TiB
  zfs-fs:
    size: 2.63 TiB
    free: 2.63 TiB
  Components: Online: 1: sdc 2: sdd1
  Device-2: vaultdb type: zfs status: ONLINE level: mirror-0
  raw:
    size: 2.72 TiB
    free: 2.72 TiB
  zfs-fs:
    size: 2.63 TiB
    free: 2.63 TiB
  Components: Online: 1: sda 2: sdb
Drives:
  Local Storage:
  total:
  raw: 11.03 TiB
  usable: 5.39 TiB
  used: 12.07 GiB (0.2%)
  ID-1: /dev/sda vendor: Hitachi model: HUS72403CLAR3000
    size: 2.73 TiB
  ID-2: /dev/sdb vendor: Hitachi model: HUS72403CLAR3000
    size: 2.73 TiB
  ID-3: /dev/sdc vendor: Hitachi model: HUS72403CLAR3000
    size: 2.73 TiB
  ID-4: /dev/sdd vendor: Hitachi model: HUS72403CLAR3000
    size: 2.73 TiB
  ID-5: /dev/sde model: P3-128
    size: 119.24 GiB
Partition:
  ID-1: /
  size: 116.81 GiB
  used: 12.06 GiB (10.3%)
  fs: ext4
  dev: /dev/sde2
Swap:
  ID-1: swap-1 type: file
  size: 4 GiB
  used: 0 KiB (0.0%)
  file: /swap.img
Sensors:
  Src: ipmi System Temperatures: cpu: N/A mobo: N/A
  Fan Speeds (rpm): cpu: 3806 mobo: 5329 fan-2: 3806 fan-3: 3747 fan-4: 3806 fan-6: 5329
  Power: 12v: 12.040 5v: 5.078 3.3v: 3.342 vbat: N/A
  Src: lm-sensors System Temperatures: cpu: 35.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Info:
  Memory:
  total: 64 GiB
  available: 62.88 GiB
  used: 1.34 GiB (2.1%)
  Processes: 320
  Uptime: 10m
  Init: systemd target: graphical (5)
  Packages: 965
  Compilers: gcc: 13.3.0
  Shell: Sudo v: redacted