Diagnosing Problems with rippled

If y'all are having bug with rippled, the first step is to collect more information to accurately characterize the problem. From there, it can exist easier to figure out a root cause and a fix.

Run across the following pages for some common categories of problems, their causes, and fixes:

  • If your server does not start (such every bit crashing or otherwise shutting down automatically), see rippled Server Won't Beginning.
  • If your server starts, only does not reliably sync or remain synced to the XRP Ledger network, come across rippled Server Doesn't Sync.

The remainder of this document suggests steps for diagnosing problems that happen while your server is up and running (including if the process is active but unable to sync with the network).

Get the server_info

You can employ the commandline to get server condition data from the local rippled instance. For example:

The response to this command has a lot of information, which is documented along with the server_info method. For troubleshooting purposes, the most important fields are (from most commonly used to least):

  • server_state - Most of the time, this field should evidence proposing for a server that is configured as a validator, or total for a non-validating server. The value connected ways that the server can communicate with the rest of the peer-to-peer network, but it does non yet have plenty data to rail progress of the shared ledger state. Normally, syncing to the state of the balance of the ledger takes about 5-fifteen minutes after starting.

    • If your server remains in the connected land for hours, or returns to the connected state after being in the total or proposing states, that usually indicates that your server cannot proceed upwards with the remainder of the network. The about common bottlenecks are disk I/O, network bandwidth, and RAM.

    • For example, the following server state information shows a healthy server that took less than 3 minutes to sync (split between the disconnected, connected, and syncing states), and is currently in the fully-synced proposing country, where it has remained for approximately 90 minutes:

                                                          $ ./rippled server_info Loading:                            "/etc/opt/ripple/rippled.cfg"                            2020-Jan-03                            22:49:32.834134358 HTTPClient:NFO Connecting to                            127.0.0.1:5005                            {                            "result"                            :                            {                            "info"                            :                            {                            ...                            (trimmed)                            ...                            "server_state"                            :                            "proposing",                            "server_state_duration_us"                            :                            "5183282365",                            "state_accounting"                            :                            {                            "connected"                            :                            {                            "duration_us"                            :                            "126164786",                            "transitions"                            :                            ane                            },                            "disconnected"                            :                            {                            "duration_us"                            :                            "2111321",                            "transitions"                            :                            ane                            },                            "total"                            :                            {                            "duration_us"                            :                            "5183282365",                            "transitions"                            :                            1                            },                            "syncing"                            :                            {                            "duration_us"                            :                            "5545604",                            "transitions"                            :                            1                            },                            "tracking"                            :                            {                            "duration_us"                            :                            "0",                            "transitions"                            :                            ane                            }                            },       ...                            (trimmed)                            ...                            }                            }                            }

      If your server shows multiple transitions between the same states, that indicates that your server was unable to stay synced. If yous do non accept a full or proposing state, and then your server has not nonetheless synced to the network. Over a long flow of time, it's likely your server may occasionally lose sync because internet connections fluctuate, so this is but a problem if the amount of time spent not in sync is a significant portion of your uptime. Afterwards about 24 hours of uptime, if less than 99% of your server'southward full runtime is spent in the full or proposing states, you may want to investigate possible sources of instability.

    • For help debugging syncing issues, see Server Doesn't Sync.

  • complete_ledgers - This field shows which ledger indexes your server has complete ledger data for. Healthy servers usually have a single range of recent ledgers, such as "12133424-12133858".

    • If yous have a disjoint set of complete ledgers such as "11845721-12133420,12133424-12133858", that could bespeak that your server has had intermittent outages or has temporarily fallen out of sync with the rest of the network. The nearly common causes for this are insufficient disk I/O or network bandwidth.

    • Normally, a rippled server downloads recent ledger history from its peers. If gaps in your ledger history persist for more than than a few hours, you lot may non be connected to any peers who have the missing data. If this occurs, you can force your server to try and peer with one of Ripple'due south full-history public servers by calculation the following stanza to your config file and restarting:

                                                                                      [ips_fixed]                                                        s2.ripple.com 51235

  • amendment_blocked - This field is commonly omitted from the server_info response. If this field appears with the value true, then the network has canonical an amendment for which your server doesn't have an implementation. Almost likely, you can set up this by updating rippled to the latest version. You lot can besides use the characteristic method to run across what subpoena IDs are currently enabled and which one(southward) your server does and does not back up.

  • peers - This field indicates how many other servers in the XRP Ledger peer-to-peer network your server is connected to. Good for you servers typically show between 5 and fifty peers, unless explicitly configured to connect only to certain peers.

    • If you accept 0 peers, your server may be unable to contact the network, or your system clock may be wrong. (Ripple recommends running an NTP daemon on all servers to keep their clocks synced.)

    • If you accept exactly 10 peers, that may indicate that your rippled is unable to receive incoming connections through a router using NAT . You can improve connectivity by configuring your router'south firewall to forwards the port used for peer-to-peer connections (port 51235 by default ).

No Response from Server

The rippled executable returns the following message if it wasn't able to connect as a client to the rippled server:

                                                        {                                                                                "error"                                                            :                                                            "internal"                    ,                                                                                "error_code"                                                            :                                                            71                    ,                                                                                "error_message"                                                            :                                                            "Internal error."                    ,                                                                                "error_what"                                                            :                                                            "no response from server"                                        }

This by and large indicates i of several problems:

  • The rippled server is starting upwardly, or is not running at all. Check the status of the service; if information technology is running, wait a few seconds and effort again.
  • You may demand to pass dissimilar parameters to the rippled commandline client to connect to your server.
  • The rippled server may be configured not to accept JSON-RPC connections.

Check the server log

By default, rippled writes the server's debug log to the file /var/log/rippled/debug.log. The location of the debug log can differ based on your server's config file. If you start the rippled service directly (instead of using systemctl or service to kickoff it), information technology also prints log messages to the console by default.

The default config file sets the log level to severity "alarm" for all categories of log messages by internally using the log_level method during startup. You can control the verbosity of the debug log using the --silent commandline option during startup and with the log_level method while the server is running. (Run into the [rpc_startup] stanza of the config file for settings.)

It is normal for a rippled the server to impress many warning-level (WRN) messages during startup and a few warning-level messages from time to time later on. You lot can safely ignore most warnings in the beginning 5 to 15 minutes of server startup.

For a more thorough explanation of various types of log messages, see Understanding Log Messages.

Info Drove Script

If you take problems diagnosing the trouble, or y'all are unable to resolve the problem with any of the mutual fixes, you may desire to ask for help in a support forum or the GitHub bug . When request for assist, you lot can use an info collection script to get together information nearly your organization to help others diagnose the effect.

The official package installation (for Ubuntu/Debian or CentOS/RedHat) installs such a script by default, to /opt/ripple/bin/getRippledInfo. New in: rippled 1.5.0 If you compiled rippled yourself, y'all can find the same script in the rippled source lawmaking repository .

To use the script:

  1. Run the script while rippled is running.

                                                $ /opt/ripple/bin/getRippledInfo                        ####################################################                        rippled info has been gathered. Please copy the   contents of /tmp/ripple_info.Xo8Xr/rippled_info.md   to a github gist at https://gist.github.com/    Delight REVIEW THIS FILE FOR ANY SENSITIVE Information   Earlier POSTING! We have tried our best to omit   any sensitive information from this file, but you   should verify before posting.                        ####################################################

    The script collects the output of many commands and writes them to a temporary file. The filename is randomized with a string of messages and numbers (case-sensitive), for example: /tmp/ripple_info.Xo8Xr/rippled_info.dr.

  2. Expect over the output file for sensitive information.

    The script attempts to scrub sensitive information from the output, such as validator keys or tokens. However, you should still check the output earlier posting publicly, as a precaution. For instance, the script outputs detailed data virtually your server hardware, and you may desire to remove some sections for privacy reasons. Use a text editor to read the output file and to remove anything you don't want to post.

                                                nano /tmp/ripple_info.Xo8Xr/rippled_info.md

  3. Upload the output file where others tin can see information technology.

    Yous can upload the file directly to GitHub Gist , Pastebin , or a similar service. If you are running rippled on a remote server, y'all may find information technology easier to start transfer the file to a machine with a spider web browser, using scp or a similar tool.

See As well

  • Concepts:
    • The rippled Server
    • Amendments
  • Tutorials:
    • Capacity Planning
    • Configure rippled
  • References:
    • rippled API Reference
      • rippled Commandline Usage
      • log_level method
      • server_info method