Merge remote-tracking branch 'upstream/develop' into hooks-merge

* upstream/develop:
  Set version to 2.3.0-b4
  feat(SQLite): allow configurable database pragma values (5135)
  refactor: re-order PRAGMA statements (5140)
  fix(book_changes): add "validated" field and reduce RPC latency (5096)
  chore: fix typos in comments (5094)
  Set version to 2.2.3
  Update SQLite3 max_page_count to match current defaults (5114)

API-CHANGELOG.md

@@ -26,7 +26,15 @@ In `api_version: 2`, the `signer_lists` field [will be moved](#modifications-to-
 
 The `network_id` field was added in the `server_info` response in version 1.5.0 (2019), but it is not returned in [reporting mode](https://xrpl.org/rippled-server-modes.html#reporting-mode).
 
-## XRP Ledger server version 2.0.0
+### Breaking change in 2.3
+
+- `book_changes`: If the requested ledger version is not available on this node, a `ledgerNotFound` error is returned and the node does not attempt to acquire the ledger from the p2p network (as with other non-admin RPCs).
+
+Admins can still attempt to retrieve old ledgers with the `ledger_request` RPC.
+
+### Addition in 2.3
+
+- `book_changes`: Returns a `validated` field in its response, which was missing in prior versions.
 
 ### Additions in 2.2
 

RELEASENOTES.md

@@ -6,6 +6,64 @@ This document contains the release notes for `rippled`, the reference server imp
 
 Have new ideas? Need help with setting up your node? [Please open an issue here](https://github.com/xrplf/rippled/issues/new/choose).
 
+# Version 2.2.3
+
+Version 2.2.3 of `rippled`, the reference server implementation of the XRP Ledger protocol, is now available. This release fixes a problem that can cause full-history servers to run out of space in their SQLite databases, depending on configuration. There are no new amendments in this release.
+
+[Sign Up for Future Release Announcements](https://groups.google.com/g/ripple-server)
+
+<!-- BREAK -->
+
+## Background
+
+The `rippled` server uses a SQLite database for tracking transactions, in addition to the main data store (usually NuDB) for ledger data. In servers keeping a large amount of history, this database can run out of space based on the configured number and size of database pages, even if the machine has disk space available. Based on the size of full history on Mainnet, servers with the default SQLite page size of 4096 may now run out of space if they store full history. In this case, your server may shut down with an error such as the following:
+
+```text
+Free SQLite space for transaction db is less than 512MB. To fix this, rippled
+  must be executed with the vacuum <sqlitetmpdir> parameter before restarting.
+  Note that this activity can take multiple days, depending on database size.
+```
+
+The exact timing of when a server runs out of space can vary based on a few factors. Server operators who encountered a similar problem in 2018 and followed steps to [increase the SQLite transaction database page size issue](../../../docs/infrastructure/troubleshooting/fix-sqlite-tx-db-page-size-issue) may not encounter this problem at all. The `--vacuum` commandline option to `rippled` from that time may work to free up space in the database, but requires extended downtime.
+
+Version 2.2.3 of `rippled` reconfigures the maximum number of SQLite pages so that the issue does not occur.
+
+Clio servers providing full history are not affected by this issue.
+
+
+## Action Required
+
+If you run an [XRP Ledger full history server](https://xrpl.org/docs/infrastructure/configuration/data-retention/configure-full-history), upgrading to version 2.2.3 may prevent the server from crashing when `transaction.db` exceeds approximately 8.7 terabytes.
+
+Additionally, five amendments introduced in version 2.2.0 are open for voting according to the XRP Ledger's [amendment process](https://xrpl.org/amendments.html), which enables protocol changes following two weeks of >80% support from trusted validators. If you operate an XRP Ledger server older than version 2.2.0, upgrade by Sep 23, 2024 to ensure service continuity. The exact time that protocol changes take effect depends on the voting decisions of the decentralized network.
+
+## Changelog
+
+### Bug Fixes
+
+- Update SQLite3 max_page_count to match current defaults ([#5114](https://github.com/XRPLF/rippled/pull/5114))
+
+### GitHub
+
+The public source code repository for `rippled` is hosted on GitHub at <https://github.com/XRPLF/rippled>.
+
+We welcome all contributions and invite everyone to join the community of XRP Ledger developers to help build the Internet of Value.
+
+
+## Credits
+
+The following people contributed directly to this release:
+
+J. Scott Branson <[email protected]>
+
+
+Bug Bounties and Responsible Disclosures:
+
+We welcome reviews of the `rippled` code and urge researchers to responsibly disclose any issues they may find.
+
+To report a bug, please send a detailed report to: <[email protected]>
+
+
 # Version 2.2.2
 
 Version 2.2.2 of `rippled`, the reference server implementation of the XRP Ledger protocol, is now available. This release fixes an ongoing issue with Mainnet where validators can stall during consensus processing due to lock contention, preventing ledgers from being validated for up to two minutes. There are no new amendments in this release.

cfg/rippled-example.cfg

@@ -1106,6 +1106,20 @@
 #                           This setting may not be combined with the
 #                           "safety_level" setting.
 #
+#       page_size           Valid values: integer (MUST be power of 2 between 512 and 65536)
+#                           The default is 4096 bytes. This setting determines
+#                           the size of a page in the transaction.db file.
+#                           See https://www.sqlite.org/pragma.html#pragma_page_size 
+#                           for more details about the available options.
+#
+#       journal_size_limit  Valid values: integer
+#                           The default is 1582080. This setting limits
+#                           the size of the journal for transaction.db file. When the limit is
+#                           reached, older entries will be deleted.
+#                           See https://www.sqlite.org/pragma.html#pragma_journal_size_limit
+#                           for more details about the available options.
+#
+#
 #-------------------------------------------------------------------------------
 #
 # 7. Diagnostics

src/libxrpl/protocol/BuildInfo.cpp

@@ -33,8 +33,9 @@ namespace BuildInfo {
 //  and follow the format described at http://semver.org/
 //------------------------------------------------------------------------------
 // clang-format off
-char const* const versionString = "2.3.0-b2"
+char const* const versionString = "2.3.0-b4"
     // clang-format on
+
     "-hooks"
 
 #if defined(DEBUG) || defined(SANITIZER)

src/libxrpl/protocol/tokens.cpp

@@ -91,7 +91,7 @@ algorithm that converts a number to coefficients from base B2.
 
 There is a useful shortcut that can be used if one of the bases is a power of
 the other base. If B1 == B2^G, then each coefficient from base B1 can be
-converted to base B2 independently to create a a group of "G" B2 coefficient.
+converted to base B2 independently to create a group of "G" B2 coefficient.
 These coefficients can be simply concatenated together. Since 16 == 2^4, this
 property is what makes base 16 useful when dealing with binary numbers. For
 example consider converting the base 16 number "93" to binary. The base 16

src/test/csf/README.md

@@ -144,7 +144,7 @@ sim.collectors.add(simDur);
 ```
 
 The next lines add a single collector to the simulation.  The
-`SimDurationCollector` is a a simple example collector which tracks the total
+`SimDurationCollector` is a simple example collector which tracks the total
 duration of the simulation.  More generally, a collector is any class that
 implements `void on(NodeID, SimTime, Event)` for all [Events](./events.h)
 emitted by a Peer.  Events are arbitrary types used to indicate some action or

src/test/csf/random.h

@@ -121,7 +121,7 @@ makeSelector(Iter first, Iter last, std::vector<double> const& w, Generator& g)
 }
 
 //------------------------------------------------------------------------------
-// Additional distributions of interest not defined in in <random>
+// Additional distributions of interest not defined in <random>
 
 /** Constant "distribution" that always returns the same value
  */

src/test/ledger/Directory_test.cpp

@@ -29,7 +29,7 @@ namespace test {
 
 struct Directory_test : public beast::unit_test::suite
 {
-    // Map [0-15576] into a a unique 3 letter currency code
+    // Map [0-15576] into a unique 3 letter currency code
     std::string
     currcode(std::size_t i)
     {

src/test/nodestore/Database_test.cpp

@@ -439,6 +439,115 @@ class Database_test : public TestBase
                 BEAST_EXPECT(found);
             }
         }
+        {
+            // N/A: Default values
+            Env env(*this);
+            auto const s = setup_DatabaseCon(env.app().config());
+            if (BEAST_EXPECT(s.txPragma.size() == 4))
+            {
+                BEAST_EXPECT(s.txPragma.at(0) == "PRAGMA page_size=4096;");
+                BEAST_EXPECT(
+                    s.txPragma.at(1) == "PRAGMA journal_size_limit=1582080;");
+                BEAST_EXPECT(
+                    s.txPragma.at(2) == "PRAGMA max_page_count=4294967294;");
+                BEAST_EXPECT(
+                    s.txPragma.at(3) == "PRAGMA mmap_size=17179869184;");
+            }
+        }
+        {
+            // Success: Valid values
+            Env env = [&]() {
+                auto p = test::jtx::envconfig();
+                {
+                    auto& section = p->section("sqlite");
+                    section.set("page_size", "512");
+                    section.set("journal_size_limit", "2582080");
+                }
+                return Env(*this, std::move(p));
+            }();
+            auto const s = setup_DatabaseCon(env.app().config());
+            if (BEAST_EXPECT(s.txPragma.size() == 4))
+            {
+                BEAST_EXPECT(s.txPragma.at(0) == "PRAGMA page_size=512;");
+                BEAST_EXPECT(
+                    s.txPragma.at(1) == "PRAGMA journal_size_limit=2582080;");
+                BEAST_EXPECT(
+                    s.txPragma.at(2) == "PRAGMA max_page_count=4294967294;");
+                BEAST_EXPECT(
+                    s.txPragma.at(3) == "PRAGMA mmap_size=17179869184;");
+            }
+        }
+        {
+            // Error: Invalid values
+            auto const expected =
+                "Invalid page_size. Must be between 512 and 65536.";
+            bool found = false;
+            auto p = test::jtx::envconfig();
+            {
+                auto& section = p->section("sqlite");
+                section.set("page_size", "256");
+            }
+            try
+            {
+                Env env(
+                    *this,
+                    std::move(p),
+                    std::make_unique<CheckMessageLogs>(expected, &found),
+                    beast::severities::kWarning);
+                fail();
+            }
+            catch (...)
+            {
+                BEAST_EXPECT(found);
+            }
+        }
+        {
+            // Error: Invalid values
+            auto const expected =
+                "Invalid page_size. Must be between 512 and 65536.";
+            bool found = false;
+            auto p = test::jtx::envconfig();
+            {
+                auto& section = p->section("sqlite");
+                section.set("page_size", "131072");
+            }
+            try
+            {
+                Env env(
+                    *this,
+                    std::move(p),
+                    std::make_unique<CheckMessageLogs>(expected, &found),
+                    beast::severities::kWarning);
+                fail();
+            }
+            catch (...)
+            {
+                BEAST_EXPECT(found);
+            }
+        }
+        {
+            // Error: Invalid values
+            auto const expected = "Invalid page_size. Must be a power of 2.";
+            bool found = false;
+            auto p = test::jtx::envconfig();
+            {
+                auto& section = p->section("sqlite");
+                section.set("page_size", "513");
+            }
+            try
+            {
+                Env env(
+                    *this,
+                    std::move(p),
+                    std::make_unique<CheckMessageLogs>(expected, &found),
+                    beast::severities::kWarning);
+                fail();
+            }
+            catch (...)
+            {
+                BEAST_EXPECT(found);
+            }
+        }
     }
 
     //--------------------------------------------------------------------------

src/test/rpc/BookChanges_test.cpp

@@ -0,0 +1,100 @@
+//------------------------------------------------------------------------------
+/*
+    This file is part of rippled: https://github.com/ripple/rippled
+    Copyright (c) 2024 Ripple Labs Inc.
+
+    Permission to use, copy, modify, and/or distribute this software for any
+    purpose  with  or without fee is hereby granted, provided that the above
+    copyright notice and this permission notice appear in all copies.
+
+    THE  SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+    WITH  REGARD  TO  THIS  SOFTWARE  INCLUDING  ALL  IMPLIED  WARRANTIES  OF
+    MERCHANTABILITY  AND  FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+    ANY  SPECIAL ,  DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+    WHATSOEVER  RESULTING  FROM  LOSS  OF USE, DATA OR PROFITS, WHETHER IN AN
+    ACTION  OF  CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+    OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+//==============================================================================
+
+#include <test/jtx.h>
+
+namespace ripple {
+namespace test {
+
+class BookChanges_test : public beast::unit_test::suite
+{
+public:
+    void
+    testConventionalLedgerInputStrings()
+    {
+        testcase("Specify well-known strings as ledger input");
+        jtx::Env env(*this);
+        Json::Value params, resp;
+
+        // As per convention in XRPL, ledgers can be specified with strings
+        // "closed", "validated" or "current"
+        params["ledger"] = "validated";
+        resp = env.rpc("json", "book_changes", to_string(params));
+        BEAST_EXPECT(!resp[jss::result].isMember(jss::error));
+        BEAST_EXPECT(resp[jss::result][jss::status] == "success");
+        BEAST_EXPECT(resp[jss::result][jss::validated] == true);
+
+        params["ledger"] = "current";
+        resp = env.rpc("json", "book_changes", to_string(params));
+        BEAST_EXPECT(!resp[jss::result].isMember(jss::error));
+        BEAST_EXPECT(resp[jss::result][jss::status] == "success");
+        BEAST_EXPECT(resp[jss::result][jss::validated] == false);
+
+        params["ledger"] = "closed";
+        resp = env.rpc("json", "book_changes", to_string(params));
+        BEAST_EXPECT(!resp[jss::result].isMember(jss::error));
+        BEAST_EXPECT(resp[jss::result][jss::status] == "success");
+
+        // In the unit-test framework, requesting for "closed" ledgers appears
+        // to yield "validated" ledgers. This is not new behavior. It is also
+        // observed in the unit tests for the LedgerHeader class.
+        BEAST_EXPECT(resp[jss::result][jss::validated] == true);
+
+        // non-conventional ledger input should throw an error
+        params["ledger"] = "non_conventional_ledger_input";
+        resp = env.rpc("json", "book_changes", to_string(params));
+        BEAST_EXPECT(resp[jss::result].isMember(jss::error));
+        BEAST_EXPECT(resp[jss::result][jss::status] != "success");
+    }
+
+    void
+    testLedgerInputDefaultBehavior()
+    {
+        testcase(
+            "If ledger_hash or ledger_index is not specified, the behavior "
+            "must default to the `current` ledger");
+        jtx::Env env(*this);
+
+        // As per convention in XRPL, ledgers can be specified with strings
+        // "closed", "validated" or "current"
+        Json::Value const resp =
+            env.rpc("json", "book_changes", to_string(Json::Value{}));
+        BEAST_EXPECT(!resp[jss::result].isMember(jss::error));
+        BEAST_EXPECT(resp[jss::result][jss::status] == "success");
+
+        // I dislike asserting the below statement, because its dependent on the
+        // unit-test framework BEAST_EXPECT(resp[jss::result][jss::ledger_index]
+        // == 3);
+    }
+
+    void
+    run() override
+    {
+        testConventionalLedgerInputStrings();
+        testLedgerInputDefaultBehavior();
+
+        // Note: Other aspects of the book_changes rpc are fertile grounds for
+        // unit-testing purposes. It can be included in future work
+    }
+};
+
+BEAST_DEFINE_TESTSUITE(BookChanges, app, ripple);
+
+}  // namespace test
+}  // namespace ripple