15 min read Server Security & Hardening Part 3 (12 / 16)
Part 3 — CI/CD & IaC

Lesson 12.12 — Server Configuration, Secrets & Access Management

Production server hardening: Linux permissions, SSH authentication, host firewall policies, Cloud IAM, and enterprise secrets management

What we will master in this lesson:

Launching a cloud virtual machine and deploying code is only the beginning. If the server permits root password logins, exposes database passwords in source code, or leaves database ports open to the public internet, catastrophic compromise is inevitable. Here is what we will achieve:

  • Linux User & Group management and the hazards of root logins
  • File permissions (chmod, chown) and why chmod 777 is dangerous
  • Ed25519 SSH key authentication and disabling password logins
  • Dual-layer defense: Cloud Security Groups + Host-level UFW firewalls
  • Cloud IAM, Authentication vs. Authorization, and IAM Instance Roles
  • Principle of Least Privilege (PoLP), MFA, and temporary tokens (STS)
  • Environment variables vs. Secrets, and AWS Secrets Manager / Vault
  • Injecting secrets into Docker & CI/CD and automated security updates
Real-World Scenario

1. The Unhardened Server Dilemma & The Necessity of Access Management

Imagine you have launched a cloud Virtual Machine and successfully deployed your web application. A few weeks later, three software engineers and an SRE join your squad. They need terminal access to debug issues, run database migrations, and inspect logs.

Catastrophic Cloud Anti-Patterns:
  • Handing everyone the shared root password to log into the production server.
  • Committing raw database root credentials and Stripe API keys directly into Git repositories.
  • Leaving database ports (MySQL 3306, Postgres 5432) open to the public internet (0.0.0.0/0).
  • Failing to revoke access credentials when an employee departs the organization.

Every public IP address on the internet is continuously scanned by automated botnets probing for default passwords and unpatched vulnerabilities. Hardening your server is non-negotiable.

Security

Eliminating unauthorized logins and brute-force attacks.

Reliability

Preventing accidental system outages and file corruption.

Maintainability

Streamlining onboarding, auditing, and offboarding.

Consistency

Enforcing standardized security baselines across all fleets.

User Architecture

2. Linux User & Group Management: Role-Based Identity

Linux is a true multi-user operating system. Every human engineer and background process should possess a dedicated identity to preserve an unambiguous Audit Trail:

Standard User Roles in Production:

root (UID 0):

The omnipotent superuser. Direct remote login should be disabled completely.

ubuntu / admin:

Default cloud initialization user. Restricted to SSH key and sudo privileges.

deploy:

Service account utilized strictly by automated CI/CD runners.

salim / dev:

Individual engineer accounts bound to specific project workspace permissions.

Core Linux User & Group Commands:

# 1. Create a password-protected individual user:
sudo adduser developer

# 2. Create a collective functional group:
sudo groupadd developers

# 3. Append user to group (-aG):
sudo usermod -aG developers developer

# 4. Grant managed administrative privilege via sudo:
sudo usermod -aG sudo developer
Root vs. Sudo: Operating as raw root lacks safety rails; a misplaced character (like rm -rf /) obliterates the filesystem instantly. Using sudo enforces deliberate authorization and logs every privileged execution to /var/log/auth.log for historical forensics.
Data Protection

3. Linux File Permissions & Ownership: Mastering chmod and chown

Every filesystem object in Linux possesses permissions across three distinct entities: Owner, Group, and Others. Each scope supports three operations:

Read (r = 4): Authority to open and read file contents or list directory entries.
Write (w = 2): Authority to modify, overwrite, or delete file contents.
Execute (x = 1): Authority to run scripts/binaries or traverse inside directories.
Octal Code Symbolic Granted Permissions Standard Production Use Case
600 rw------- Owner can Read/Write; Group & Others have zero access. .env files, SSH Private Keys, TLS Certificates
644 rw-r--r-- Owner can Read/Write; Group & Others can only Read. Public assets, HTML, CSS, JavaScript, application files
700 rwx------ Owner has full Read/Write/Execute; Others locked out. ~/.ssh directory
755 rwxr-xr-x Owner can do all; Group & Others can Read & Execute. Standard web directories (/var/www/html) and binaries
777 rwxrwxrwx Anyone on the server or web can read, rewrite, or delete! Dangerous Anti-Pattern (NEVER use in production!)
Why chmod -R 777 is a Critical Flaw:

Junior developers frequently run chmod -R 777 to silence upload permission errors. This allows any compromised process on the host to inject malicious web shells or wipe your entire application directory. Solve permission bottlenecks by assigning appropriate user ownership (e.g., sudo chown -R www-data:www-data storage/) rather than unlocking permissions universally.

Remote Access

4. SSH Hardening: Ed25519 Cryptographic Keys & Banning Passwords

Password-based SSH access exposes servers to dictionary scans and credential stuffing. Production environments mandate Asymmetric Cryptographic Key Pairs:

Private Key (Secret):

Stored strictly on your local client machine (~/.ssh/id_ed25519). Never upload this to GitHub or any server!

Public Key (Open):

Appended to the target user's ~/.ssh/authorized_keys on the remote server.

Step-by-Step SSH Hardening Commands:

# 1. Generate modern, compact Ed25519 keypair on local machine:
ssh-keygen -t ed25519 -C "salim@byteion.com"

# 2. Safely copy public key to remote server:
ssh-copy-id -i ~/.ssh/id_ed25519.pub developer@YOUR_SERVER_IP

# 3. Secure file permissions on remote server:
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys

Hardened directives in /etc/ssh/sshd_config:

# Disable remote root logins:
PermitRootLogin no
# Disable password logins completely (SSH Key only):
PasswordAuthentication no
# Prohibit blank credentials:
PermitEmptyPasswords no
The Golden Safety Rule: Always validate syntax with sudo sshd -t before reloading. Never disconnect your active terminal session! Open a fresh terminal window, test logging in with your SSH key, and only terminate your prior session once the new connection succeeds.
Network Perimeter

5. Dual-Layer Firewall: Cloud Security Groups + Host UFW

Enterprise architectures employ Defense-in-Depth across networking. Relying on a single firewall layer creates a Single Point of Failure (SPOF):

1. Cloud Perimeter Firewall

AWS Security Groups / Cloudflare WAF. Filters malicious packets at the hypervisor network edge before they ever touch your virtual machine.

2. Host Firewall (UFW)

Linux kernel iptables filtering. Safeguards the operating system even if a cloud security group rule is accidentally opened.

Hardening UFW on Ubuntu Server:

# 1. Deny all incoming packets by default, permit outgoing:
sudo ufw default deny incoming
sudo ufw default allow outgoing

# 2. Restrict SSH (Port 22) exclusively to trusted Admin IPs:
sudo ufw allow from 103.145.22.10 to any port 22 proto tcp

# 3. Allow web traffic for HTTP and HTTPS:
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# 4. Enable firewall and verify status:
sudo ufw enable && sudo ufw status verbose
Docker & UFW Gotcha: Docker manipulates iptables directly and bypasses UFW by default. In production, never expose container ports directly to 0.0.0.0; bind web containers to loopback (127.0.0.1:8080:80) behind an Nginx reverse proxy.
Cloud Identity

6. Cloud IAM: Authentication, Authorization & Instance Roles

Managing operating system users is only half the battle. Controlling who can provision, reboot, or destroy cloud infrastructure is governed by IAM (Identity and Access Management):

Authentication (AuthN) — "Who are you?"

Verifies identity via passwords, biometric passkeys, and time-based MFA tokens.

Authorization (AuthZ) — "What can you do?"

Grants specific actions via JSON policies (e.g., an engineer can reboot EC2, but cannot view billing).

IAM Roles vs. Hardcoded Long-Term Access Keys

If your web application needs to read assets from an S3 bucket, never embed AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY inside configuration files. If leaked, your cloud account can be weaponized for crypto mining.

Instead, attach an IAM Role directly to the EC2 instance profile. The AWS Security Token Service (STS) injects ephemeral, automatically rotated Temporary Credentials that expire in hours.

Core Security Doctrine

7. Principle of Least Privilege (PoLP) & Multi-Factor Authentication

Principle of Least Privilege: Grant every user, script, and container only the absolute minimum permissions necessary to accomplish their explicit function—not a single permission more.

To Developers: Grant SSH access to Staging and log viewers; withhold master production credentials.
To Web Servers: Grant read-only access to specific S3 assets; withhold account administrative rights.
To Database Users: Grant SELECT, INSERT, UPDATE on application schemas; forbid DROP DATABASE.

Multi-Factor Authentication (MFA): Mandating hardware FIDO2 keys (YubiKey) or time-based one-time password (TOTP) authenticators ensures that even if credentials are stolen, unauthorized login remains blocked.

Secrets Engineering

8 & 9. Non-Sensitive Configuration vs. Secrets & Cloud Vaults

Software configuration must be bifurcated into public settings and cryptographic secrets:

1. Non-Sensitive Configuration
APP_ENV=production
APP_PORT=8080
LOG_LEVEL=info
CACHE_DRIVER=redis

Operational parameters; safe if committed to internal repositories.

2. Sensitive Cryptographic Secrets
DB_PASSWORD=SuperSecret99#
STRIPE_API_KEY=sk_live_8f3a1...
JWT_SECRET=x9Kq2Lm8...
AWS_SECRET_KEY=wJalrXUtn...

Leaking a single key can bankrupt a company or compromise user data!

Why Use Centralized Secrets Managers (AWS Secrets Manager / Vault)?

While .env files work locally, maintaining files across dozens of microservice nodes is unmanageable. Centralized vaults provide:

  • Hardware Encryption: Secrets are encrypted at rest with hardware KMS (AES-256-GCM).
  • Automated Rotation: Rotates database credentials every 30 days without downtime or human touch.
  • Comprehensive Audit Trails: CloudTrail logs every process and container reading secrets.
Pipeline Security

10. Managing Secrets in Docker & CI/CD Pipelines

Avoid these fatal anti-patterns during container builds and continuous delivery:

Fatal Dockerfile Anti-Patterns:
ENV DB_PASSWORD="mysecretpassword" # Burned permanently into image history layers!
COPY .env /app/.env # Pushing image to public Docker Hub leaks everything!
Recommended Practices:
1. Encrypt secrets inside GitHub Repository Settings → Secrets and variables.
2. Inject secrets into containers strictly at runtime: docker run -e DB_PASSWORD=$SECRET my-app
3. Use BuildKit mount secrets for temporary build tokens: docker build --secret id=mysecret
Integrated Defense

11 & 12. Security Patching & The Four Tiers of Access

Keeping operating systems updated against zero-day vulnerabilities requires continuous hygiene:

# Package updates and unattended security patches:
sudo apt update && sudo apt upgrade -y
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

The 4 Tiers of Cloud Access Architecture:

1. Infrastructure Layer (Cloud IAM): Authorizes who can provision VMs, modify firewalls, or manage storage.
2. Server OS Layer (Linux SSH & Sudo): Authorizes who can establish SSH sessions and execute systemd commands.
3. Application Layer (RBAC & JWT): Authorizes who can access administrative dashboards or billing records.
4. Service-to-Service Layer (mTLS & IAM Roles): Authorizes backend services to query databases or cloud queues securely.
Hands-on Lab

🎮 Interactive Server Hardening & Secrets Workbench

Experiment with Linux permissions, test SSH daemon brute-force defenses, configure dual-layer firewalls, and simulate automated secret rotations.

Interactive Linux Permission Calculator

Toggle bit masks to calculate octal codes, symbolic strings, and real-time security postures.

Owner (User)
Group (Team)
Others (Public World)
chmod command & representation:
chmod 600 myfile -rw-------
SAFE: Ideal for .env & Private Keys
Summary & Next Steps

Key Takeaways: What We Have Mastered

  • Eliminate Root Logins: Assign individual accounts with sudo logging to maintain accountability.
  • File Permissions Hygiene: Secure sensitive .env files with chmod 600 and never use 777.
  • SSH Daemon Hardening: Ban password authentication and root remote logins in favor of Ed25519 keypairs.
  • Dual-Layer Defense: Restrict Port 22 and database ports across both Cloud Security Groups and Host UFW.
  • Principle of Least Privilege: Grant users, scripts, and containers only the minimal permissions required.
  • IAM Roles Over Static Keys: Bind instance profiles to compute nodes to consume short-lived STS tokens.
  • Centralized Secret Vaults: Store production database passwords in KMS-backed Secrets Managers with auto-rotation.
  • Automated Patching: Apply security patches via unattended-upgrades to remediate zero-day exploits.
Next Lesson Preview

Lesson 12.13 — Monitoring, Logging & Observability

We have automated cloud infrastructure and hardened our servers with enterprise security. But when live traffic surges and thousands of users connect simultaneously, how do you detect CPU saturation, memory exhaustion, database bottlenecks, or HTTP 500 error spikes before your users do? In the next lesson, we enter Part 4 and master Metrics, Logs, and Traces (The Three Pillars of Observability) with Prometheus, Grafana, and automated alerting!

Authoritative References & Documentation:
  • Ubuntu Server Guide: OpenSSH Server & Key Management
  • Ubuntu Documentation: UFW (Uncomplicated Firewall) Security
  • AWS Well-Architected Framework: Security Pillar & IAM Best Practices
  • OWASP: Secrets Management Cheat Sheet & Credential Hygiene
  • CIS Benchmarks: Ubuntu Linux Hardening Standards