In the ever-evolving world of software development, we often find ourselves relying on third-party code to ease and speed up development. While these tools can offer significant benefits, they can also introduce unique challenges when it comes to troubleshooting. In this post we’ll cover some essential advice for debugging third-party software, enabling you to resolve issues efficiently and effectively.
Check logs and verbosity
One of the first steps in troubleshooting software is to check the logs. Logs are a treasure trove of information, providing crucial insights into what happened during the execution of a program. Make sure to set the appropriate log verbosity level, as this will ensure that you receive enough detailed information to identify the root cause of the issue. If the software allows you to configure verbosity levels, start with the highest level of detail and work your way down if necessary.
For example to adjust the verbosity level of nginx's error logs, you can modify the error_log directive in your nginx configuration file (usually located at /etc/nginx/nginx.conf):
error_log /var/log/nginx/error.log notice;In this example, the error log is set to the "notice" level, which provides a moderate amount of detail. The available log levels are: "debug", "info", "notice", "warn", "error", "crit", "alert", and "emerg" in the order of increasing severity. Setting a certain log level will cause all messages of the specified and more severe log levels to be logged. For example, the default level error will cause error, crit, alert, and emerg messages to be logged.
Setting a minimum log level is not specific to just nginx but is a common practice across various software applications and logging libraries.
Enable debug mode
As you might have noticed, there is a “debug” log level in the nginx example above. Many software applications come with a built-in debug mode or log level designed to provide additional information to help diagnose problems. Enabling debug mode often results in more detailed logs and error messages, making it easier to pinpoint the cause of any issues. Consult the software's documentation or user guides to learn how to enable and utilize debug mode effectively. You might even see that there are several sub-levels of the “debug” level.
For example, ssh has three debug levels:
-v Verbose mode. Causes ssh to print debugging messages about its progress. This is helpful in debugging connection, authentication, and configuration problems. Multiple -v options increase the verbosity. The maximum is 3.Here’s an example output with the most verbose option
$ ssh -vvv 127.0.0.1
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/victor/.ssh/config
debug1: /home/victor/.ssh/config line 1: Applying options for *
debug3: kex names ok: [diffie-hellman-group1-sha1]]
debug1: /home/victor/.ssh/config line 217: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 127.0.0.1 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/victor/.ssh/known_hosts'
debug3: ssh_connect_direct: entering
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/victor/.ssh/id_rsa type 0
debug1: identity file /home/victor/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
...Check the documentation for configuration options
As you work with third-party software, always remember to consult the documentation. It often happens that you’ve misread or entirely missed a configuration option that affects the application’s behavior. The documentation is your best friend when it comes to understanding the software's functionality, configuration options, and potential pitfalls. It sometimes includes a “Troubleshooting” section that provides valuable information on how to resolve common issues and how to configure the software to suit your specific needs. For command-line tools you can quickly pull up the documentation using man <command>
$ man ssh
Update to a newer compatible version
In some cases, the issue you're experiencing may have already been addressed in a newer version of the software/library. Before spending too much time troubleshooting, check if there's an update available. Keep in mind that it's essential to verify that the newer version is compatible with your project, as upgrading might introduce breaking changes. Minor version upgrades should be painless in most cases but as a precaution always review the release notes and perform thorough testing before deploying an updated version in your production environment.
Google it!
When you encounter an error message, don't hesitate to turn to Google or other search engines for help. Chances are, someone else has faced a similar issue and has shared their experience online. In addition to searching for specific error messages, review release logs for the software, as these often contain information about known issues and their corresponding fixes. Furthermore, directly searching or exploring forums, discussion boards, and community resources can provide valuable insights and potential solutions that didn’t show up in the Google search results. Try searching directly in StackOverflow, GitHub issues, Discord/Slack/ server of the community, etc.
Troubleshooting third-party software can be challenging and in most cases - frustrating as hell. I hope that by following the advice outlined in this post you’ll be well-equipped to solve the issues you encounter. Remember to leverage logs, debug mode, documentation and online resources to efficiently diagnose and resolve problems, ensuring the smooth operation of your own piece of software as well as your mental wellbeing.