Part 2 - Troubleshooting Linux Scenarios ๐Ÿง‘โ€๐Ÿ’ป

Part 2 - Troubleshooting Linux Scenarios ๐Ÿง‘โ€๐Ÿ’ป

ยท

6 min read

Issue 1: Unable to connect to a website or an application

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ Ping the server by Hostname and IP Address
โ”‚ โ”œโ”€โ”€ False: Above Troubleshooting Diagram "Server is not reachable or cannot connect"
โ”‚ โ”œโ”€โ”€ True: Check the service availability by using telnet command with port
โ”‚ โ”‚ โ”œโ”€โ”€ True: Service is running
โ”‚ โ”‚ โ”œโ”€โ”€ False: Service is not reachable or running
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the service status using systemctl or other command
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the firewall/selinux
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the service logs
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the service configuration
โ””โ”€โ”€ ...

Issue 2: Unable to get IP Address

๐Ÿ› ๏ธ Approach / Solution:
โ”œโ”€โ”€ IP Assignment Methods
โ”‚ โ”œโ”€โ”€ DHCP
โ”‚ โ”‚ โ”œโ”€โ”€ Fixed Allocation
โ”‚ โ”‚ โ”œโ”€โ”€ Dynamic Allocation
โ”‚ โ”œโ”€โ”€ Static
โ”œโ”€โ”€ Troubleshooting
โ”‚ โ”œโ”€โ”€ check network setting from virtualization environment like VMware, VirtualBox, or etc
โ”‚ โ”œโ”€โ”€ check the IP address is assigned or not
โ”‚ โ”œโ”€โ”€ check the NIC status from the host side using #lspci, #nmcli, etc
โ”‚ โ”œโ”€โ”€ restart network service
โ””โ”€โ”€ ...

Issue 3: Server is not reachable or unable to connect

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ Ping the server by Hostname and IP Address
โ”‚ โ”œโ”€โ”€ Hostname/IP Address is pingable
โ”‚ โ”‚ โ”œโ”€โ”€ Issue might be on the client side as server is reachable
โ”‚ โ”œโ”€โ”€ Hostname is not pingable but IP Address is pingable
โ”‚ โ”‚ โ”œโ”€โ”€ Could be the DNS issue
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ check /etc/hosts
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ check /etc/resolv.conf
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ check /etc/nsswitch.conf
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ (Optional) DNS can also be defined in the /etc/sysconfig/network-scripts/ifcfg-<interface>
โ”‚ โ”œโ”€โ”€ Hostname/IP Address both are not pingable
โ”‚ โ”‚ โ”œโ”€โ”€ Check the other server on its same network to see if there is a Network side access issue or overall something bad
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ False: Issue is not overall network side but it's with that host/server
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ True: Might be an overall network side issue
โ”‚ โ”‚ โ”œโ”€โ”€ Logged into the server by Virtual Console, if the server is Powered ON. Check the uptime
โ”‚ โ”‚ โ”œโ”€โ”€ Check if the server has the IP and has UP status of Network interface
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ (Optional) Also check IP-related information from /etc/sysconfig/network-scripts/ifcfg-<interface>
โ”‚ โ”‚ โ”œโ”€โ”€ Ping the gateway, also check routes
โ”‚ โ”‚ โ”œโ”€โ”€ Check Selinux, Firewall rules
โ”‚ โ”‚ โ”œโ”€โ”€ Check physical cable connection

Issue 4: Unable to ssh as root or any other user

๐Ÿ› ๏ธ Approach / Solution:
โ”œโ”€โ”€ Ping the server by Hostname and IP Address
โ”‚ โ”œโ”€โ”€ False: Above Troubleshooting Diagram "Server is not reachable or cannot connect"
โ”‚ โ”œโ”€โ”€ True: Check the service availability by using telnet command with port
โ”‚ โ”‚ โ”œโ”€โ”€ True: Service is running
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Issue might be on the client side
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ User might be disabled, nologin shell, disabled root login, and other configurations
โ”‚ โ”‚ โ”œโ”€โ”€ False: Service is not reachable or running
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the service status using systemctl or other command
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the firewall/selinux
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the service logs
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“ Check the service configuration
โ””โ”€โ”€ ...

Issue 5: Disk Space is full issue or add/extend disk space

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ System Performance degradation detection
โ”‚ โ”œโ”€โ”€ Application getting slow/unresponsive
โ”‚ โ”œโ”€โ”€ Commands are not running (For Example: / disk space is full)
โ”‚ โ”œโ”€โ”€ Cannot do logging and other etc
โ”œโ”€โ”€ Analyse the issue
โ”‚ โ”œโ”€โ”€ df command to find the problematic filesystem space issue
โ”œโ”€โ”€ Action
โ”‚ โ”œโ”€โ”€ After finding the specific filesystem, use du command in that filesystem to get which files/directories are large
โ”‚ โ”œโ”€โ”€ Compress/remove big files
โ”‚ โ”œโ”€โ”€ Move the items to another partition/server
โ”‚ โ”œโ”€โ”€ Check the health status of the disks using badblocks command (For Example: #badblocks -v /dev/sda)
โ”‚ โ”œโ”€โ”€ Check which process is IO Bound (using iostat)
โ”‚ โ”œโ”€โ”€ Create a link to file/dir
โ”œโ”€โ”€ New disk addition
โ”‚ โ”œโ”€โ”€ Simple partition
โ”‚ โ”‚ โ”œโ”€โ”€ Add disk to VM
โ”‚ โ”‚ โ”œโ”€โ”€ Check the new disk with df/lsblk command
โ”‚ โ”‚ โ”œโ”€โ”€ fdisk to create partition. Better to have LVM partition
โ”‚ โ”‚ โ”œโ”€โ”€ Create filesystem and mount it
โ”‚ โ”‚ โ”œโ”€โ”€ fstab entry for persistence
โ”‚ โ”œโ”€โ”€ LVM Partition
โ”‚ โ”‚ โ”œโ”€โ”€ Add disk to VM
โ”‚ โ”‚ โ”œโ”€โ”€ Check the new disk with df/lsblk command
โ”‚ โ”‚ โ”œโ”€โ”€ fdisk to create LVM partition
โ”‚ โ”‚ โ”œโ”€โ”€ PV, VG, LV
โ”‚ โ”‚ โ”œโ”€โ”€ Create filesystem and mount it
โ”‚ โ”‚ โ”œโ”€โ”€ fstab entry for persistence
โ”‚ โ”œโ”€โ”€ Extend LVM partition
โ”‚ โ”‚ โ”œโ”€โ”€ Add disk, and create LVM partition
โ”‚ โ”‚ โ”œโ”€โ”€ Add LVM partition (PV) in the existing VG
โ”‚ โ”‚ โ”œโ”€โ”€ Extend LV and resize filesystem
โ””โ”€โ”€ ...

Issue 6: SSL/TLS Certificate Expiry

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ Check SSL/TLS certificate expiration date
โ”œโ”€โ”€ Confirm the correct certificate is being used
โ”œโ”€โ”€ Renew the SSL/TLS certificate before expiry
โ”œโ”€โ”€ Verify that the server's system time is accurate
โ”œโ”€โ”€ Restart the web server after certificate renewal
โ”œโ”€โ”€ Ensure the renewed certificate is properly configured
โ”œโ”€โ”€ Check for any errors in the web server logs
โ”œโ”€โ”€ Consider automating certificate renewal tasks
โ”œโ”€โ”€ Monitor SSL/TLS certificate expiry using alerts
โ””โ”€โ”€ ...

Issue 7: Database Connection Issue

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ Check if the database service is running
โ”œโ”€โ”€ Verify database connection parameters (username, password)
โ”œโ”€โ”€ Test database connectivity using command-line tools
โ”œโ”€โ”€ Inspect database logs for connection errors
โ”œโ”€โ”€ Review firewall rules for database port access
โ”œโ”€โ”€ Confirm network accessibility between application and database servers
โ”œโ”€โ”€ Check for any recent changes in the database configuration
โ”œโ”€โ”€ Monitor database resource usage and performance
โ”œโ”€โ”€ Investigate potential database server load issues
โ””โ”€โ”€ ...

Issue 8: Web Application 404 Error

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ Verify that the requested URL is correct
โ”œโ”€โ”€ Check web server logs for 404 error details
โ”œโ”€โ”€ Confirm that the file or resource exists on the server
โ”œโ”€โ”€ Review web server configuration for correct document root
โ”œโ”€โ”€ Inspect file and directory permissions
โ”œโ”€โ”€ Clear web browser cache and retry
โ”œโ”€โ”€ Consider URL rewriting rules if applicable
โ”œโ”€โ”€ Test with different browsers or devices
โ”œโ”€โ”€ Investigate if there are any recent website changes
โ””โ”€โ”€ ...

Issue 9: Slow Application Response Time

๐Ÿ› ๏ธ Approach / Solution:
โ”œโ”€โ”€ Identify bottlenecks using 'top' or 'htop'
โ”œโ”€โ”€ Monitor application-specific logs for errors or warnings
โ”œโ”€โ”€ Check for database connection and query performance
โ”œโ”€โ”€ Review application code for inefficient algorithms
โ”œโ”€โ”€ Optimize database queries and indexing
โ”œโ”€โ”€ Investigate potential issues with external API calls
โ”œโ”€โ”€ Monitor network latency between application components
โ”œโ”€โ”€ Consider implementing caching mechanisms
โ”œโ”€โ”€ Optimize server and network configurations
โ””โ”€โ”€ ...

Issue 10: High CPU Usage

๐Ÿ› ๏ธ Approach / Solution:

โ”œโ”€โ”€ Identify the process causing high CPU usage using top or htop
โ”œโ”€โ”€ Check if the issue is intermittent or continuous
โ”œโ”€โ”€ Review logs for any error messages or known issues
โ”œโ”€โ”€ Inspect running processes and their resource consumption
โ”œโ”€โ”€ Investigate potential malware or unauthorized processes
โ”œโ”€โ”€ Consider optimizing or scaling the application
โ”œโ”€โ”€ Monitor system metrics over time to identify patterns
โ”œโ”€โ”€ Apply performance tuning based on the specific application
โ””โ”€โ”€ ...

Did you find this article valuable?

Support Prasad Suman Mohan by becoming a sponsor. Any amount is appreciated!

ย