Linux Standards and Best Practices



 

 


Introduction

This document is intended to provide a Linux standards definition for Administrators who install, maintain, and administer Linux. It is not really intended to be all-inclusive, but rather a point from which to begin creation of a set of standards that meet the needs of your organization. This document is based upon the use of Red Hat Enterprise Linux, but can just as easily be modified for any other distribution especially RHEL related distributions such as Centos and Fedora.

Purpose

The document describes a minimum set of standards to be used for installing, maintaining and administering Linux on computers, both physical and Virtual in a commercial production environment. These standards can also be used as a guideline for home, Small Office/Home Office (SOHO) and hobby environments.

Details

Hardware

Console access must be provided to all hardware using one of the following options.

  • In a home, hobby or SOHO environment, this usually means a standard keyboard, mouse and display type of setup.
  • When installed in a typical commercial production environment all physical hardware should be provided with remote lights out management capabilities either with HP ILO or IBM HMC. Alternatively, especially if not using IBM or HP hardware, KVMs may be mounted in each row and KVM over TCP/IP may be used.
  • As a last alternative, a display, keyboard and mouse on a rolling “crash cart” may be used for physical access.

All physical hardware in a commercial datacenter environment should have hardware based RAID whenever possible.

Linux Distributions and Versions

The latest version and update level of Linux, such as Red Hat Enterprise Linux (RHEL), Fedora, or CentOS should always be used where possible. The only exceptions should be when applications require an earlier version of a given distribution to function correctly.

Earlier versions of any distribution should always be installed and updated to the latest update level.

CentOS is an acceptable alternative for RHEL in many production environments and for lab and testing environments in which the specific requirement for RHEL is not an issue. CentOS is a binary compatible, free, Open Source version of RHEL which has simply had all of the Red Hat logos and branding replaced by CentOS logos and branding. It’s release and update cycles are a few weeks to months later than the Red Hat cycles.

Linux Updates and Patches

All Linux patches and updates should be installed as outlined below, unless a particular patch or update would result in impairment of the application software and thereby the intended function of the computer. In such an event, all possible patches must be installed and only the patch causing the problem should be withheld from installation.

  • All critical security patches must be tested and installed within 14 days of release.
  • All other security patches must be tested and installed within 30 days of release.
  • All non-security patches should be tested and installed within 60 days of release.

System Reboots

Linux systems should never be rebooted except for the following circumstances.

  1. The kernel has been updated.
  2. Hardware maintenance which requires the host to be powered off.

Security

  1. IPTables, or, more recently, firewalld should be on and specifically configured for the needs of each machine. By default, only SSH should be allowed in. Failure to use one of these firewalls on each host, no matter where located, makes for a “crunchy” network, i.e., one that is crispy on the outside and soft on the inside. Other ports should be activated only as required by installed applications or administrative requirements.
  2. SELinux should be active, especially on any box within a DMZ. SELinux should be configured as “Enforcing” and “Strict.” This will prevent any crackers who do manage to access the computer from making any changes that will allow them to escalate privileges or to modify any executable or configuration files.
  3. For maximum security while still allowing Linux administrators access to machines, SSH Public/Private keypairs should be used. Remote access for root should be allowed, but only from certain “portal” machines in each zone. Portal machines should allow SSH access for root between zones by providing direct access between the portal machines themselves. The portal computers in any DMZ should only allow inbound SSH access from other portal computers located in internal non-DMZ zones. IPTables on each Linux computer should be used to restrict inbound SSH access to only those portal computers.
  4. All unused services should be turned off and removed from the host.
  5. Logwatch should be installed and enabled on all Linux computers. The emailed output from Logwatch should be scanned daily for anomalous entries.
  6. X11 forwarding is a security vulnerability and should be disallowed at all times.

Virtual Machine Configuration

When possible, only a single virtual CPU should be specified for Virtual Machines (VMs). This is due to internal functionality that makes a single virtual CPU work better than multiple virtual CPUs for most current applications. In order to take advantage of multiple virtual CPUs, the application must be well written and highly multithreaded.

Virtual RAM should be requested based on the maximum RAM expected to be utilized by the operating system and allocations. It is best to request the optimum amount required as overallocation of RAM in Virtual Machines can actually impair performance.

Installation

All installations shall be performed using Red Hat Satellite Server or some form of kickstart or other automated installation process. This provides for consistent, repeatable installation processes. It also provides a mechanism for distribution of updates and patches.

Where it is not possible to do an automated installation using Red Hat Satellite Server, some form of Linux Installation Checklist should be used as a guide to providing a standardized result. Such a checklist is best created as a shell script that performs all possible setup and configuration tasks after an initial basic installation. A second script should be used to verify correct functioning of the first script.

Boot and Startup Configuration

All Linux systems should be configured to display the GRUB boot menu to allow for selecting alternate kernels to boot from and to allow kernel boot parameters to be altered during the boot process.

All startup output should be displayed on the console. The Red Hat Graphical Boot (RHGB) should be disabled by removing the rhgb parameter from the kernel boot line in grub.conf or grub.cfg for GRUB2.

Start Scripts

All programs and services which have scripts added to the standard SysV start script directories must be added to chkconfig or systemd startup for ease of management, consistency, and compliance with evolving startup standards.

Note: All versions and distributions of Linux are in the process of changing from SysV start scripts to a new startup system called systemd. Applications will need to adapt their underlying installations to deal with start scripts in the future. Using chkconfig or systemd simplifies and standardizes service management before, during and after this transition, which will take multiple releases of each distribution. The conversion for Fedora has been completed by Fedora 17. Significant changes are also taking place within /etc/inittab as a result of these changes to the startup mechanism—in fact, the inittab file is no longer used by Fedora.

RAID Configuration

Whenever possible, Hardware RAID is to be used when configuring physical systems. Virtual Machines should always run on systems that are RAIDed at the hardware level and do not require software RAID at the OS level. If hardware RAID is not available on a physical computer, then software RAID should be configured during the Linux installation.

The specific RAID configuration will depend upon the number of physical HDDs installed. In general, the OS should be installed on a pair of mirrored drives (hardware RAID1). If only two additional drives are available, they should be mirrored also. If more than two drives are available, they should be combined into a single RAID 5 array.

Logical Volume Management (LVM) and Standard Filesystem Sizes

Standard Filesystem sizes are based on best practices for general purpose Linux systems and, except for /boot, are on LVM partitions to enhance future expandability. An LVM filesystem can be expanded at will, on the fly with little impact to the running system.

All physical disk space available after creating multi-disk devices such as hardware or software RAID (which should be mutually exclusive) should be created as part of a single Volume Group or at most two Volume Groups. Logical Volumes are then created for each file system. This provides for the most flexible use of all available disk space. With the setup below there will probably be plenty of free space in the Volume Group which can be allocated as needed at a later time.

Linux Virtual Machines should be configured with a single default virtual disk with a minimum size of about 45GB.

On physical servers, when only two physical disks are available and are configured in a RAID 1 configuration, all of the disk space should be configured as a single Volume Group. Additional space as required by applications should usually be added as part of this Volume Group. Space can be added to standard mount points such as /opt simply by expanding the Logical Volume using available free space in the Volume Group.

In some cases, such as when using two disks in a RAID 1 configuration for the OS and two or more disks in RAID 1 or RAID 5 configuration for applications, creation of a separate Volume Group, VolGroup01 should be created to contain the Logical Volumes for the applications and data.

Only a single swap partition should be created and it should reside in the same Volume Group as the OS.

Linux Filesystems
Filesystem LVM Type Size (GB) Comments
/boot No EXT3 0.50
/  (root) Yes EXT3 2.00
/var Yes EXT3 9.00 Expand as required by server purpose
/home Yes EXT3 3.00
/usr Yes EXT3 8.00 Expand as required by server purpose
/opt Yes EXT3 5.00 Expand as required by server purpose
/tmp Yes EXT3 5.00 Expand as required by server purpose
Swap Yes SWAP 4.00 Based on physical memory and application working set requirements. Generally can be between 1x to 2x physical memory, but if more than 4GB of physical memory, may only require <1x physical for swap space. 4GB used here for an example.Maximum swap size should be no more than 8GB. If you are having performance problems due to memory starvation with that amount of swap, additional swap space will not help and you should add RAM instead.
Total GB 42.50

Time

The NTP client must be installed and active on each computer. It should be synchronized to a local NTP time server which is, in turn, synchronized to the distribution time server pools. A pool can be used as an NTP time source for individual hosts if no other local sources are available but it is not recommended.

Network Interface Card Settings

All Network Interface Cards (NICs) should be configured for auto-negotiation to ensure automatic negotiation of the highest available speeds between the NIC and the switch. This implies that all switch ports be set to auto-negotiate also.

User Accounts and Groups

All standard Linux system level users are located in the UID range between 1 and 99 which is reserved for this purpose. All application level users should be added in the UID range between 101 and 999. All regular (human) users should be added starting at UID 1000 and above.

Group IDs should follow the same standards. GID 100 is reserved for the “Users” group. In some environments it is common for all regular users to be added to this group but this is not recommended as it constitutes a security risk.

For regular users the UID and GID should be identical, that is a user with UID 1001 should have a GID of 1001 as well. Since each user belongs to its own group, security is enhanced because files are not automatically shared between users. By default, users should not all belong to a single common primary group such as users (GID 100). Making files available to other users should be accomplished by using secondary group memberships to which the sharing users all belong and a directory where shared files can be stored and which has group ownership by the common group. Group membership should be limited to those who have a specific need to share related files. This allows for more granular management of shared files.

Additional Utilities

The following utilities are very useful and should be installed on every Linux server. These are all part of the RHEL, CENTOS and Fedora distributions.

Additional Useful Linux Utilities
Command or RPM Name Description
mc Midnight Commander Text mode file manager
screen screen Multiple terminal sessions through one SSH (or other) remote connection.
iptraf or iptraf-ng IP Traffic Provides multiple ways to display and analyze network traffic.
sysstat System Statistics Contains SAR and iostat.

Documentation

Each server must be separately documented. Any and all deviations from normal or standard installations must be noted in the documentation for each server. This documentation should include the following.

  • Installation date
  • Server hardware configuration
  • Initial software configuration
  • Firewall configuration

Any and all work performed on each server must be noted in a work log maintained for each server.