Consolidate definitions of fields, objects, transactions, and features

[thejohnfreeman]

Leverages a preprocessor technique to put the entire definitions of fields, objects, transactions, and features into one file for each type. When you want to add a field, object, transaction, or feature, you touch just one file. When you want to inspect a definition, you look up just one file.

[donovanhide]

This is a useful change and would make the manual process of updating third party libraries with new field types, transactions and ledger entries a little easier.
However, an even more useful change would be to abstract the definition files into easily machine parsable formats, maybe either JSON or CSV, so that any third party API libraries can choose to auto-generate their data structures and stream encoders and decoders from those files, thereby keeping up to date much more easily than using git blame to find recent changes to the large definition files.

[thejohnfreeman]

Yes, that would be great, but the preprocessor cannot parse JSON or CSV or any other format. It would require the introduction of an additional build step to generate these headers from that definition file. I would welcome that contribution from someone else, but it's out of scope for my current work. This was a drive-by change I made while working on single asset vault.

[donovanhide]

Thanks for the reply. I do think the pool of people who could wrangle the rippled build process to add a code generation stage is probably limited to one :-) If it's not something that appeals, fair enough.

[Bronek]

Few initial comments, also this: I wonder if this pattern is perhaps too permissive

#pragma push_macro("FOO")
#undef FOO
#pragma push_macro("BAR")
#undef BAR

#define FOO . . .;
#define BAR . . .;

#include <xrpl/protocol/. . .>

#undef FOO
#pragma pop_macro("FOO")
#undef BAR
#pragma pop_macro("BAR")

Could this be better (i.e. more robust) ?

#if defined(FOO) || defined(BAR)
#error "Macro FOO or BAR was found to be unexpectedly defined"
#endif

#define FOO . . .;
#define BAR . . . ;

#include <xrpl/protocol/. . .>

#undef FOO
#undef BAR

[Bronek]

For consistency, please follow the same style as in Feature.h (vertical spacing)

[donovanhide]

Would LEDGER_ENTRY be a more specific name for the macro? OBJECT seems very general and suggests it can be used elsewhere.

[Bronek]

I would put it in include/xrpl/protocol/detail . Also, how about name which clearly refers to a header it is used in, e.g. Feature_list.h or even better Feature.inl ?

[thejohnfreeman]

I much prefer the simple features.h naming.

[Bronek]

I totally do not mind .ipp, or even stick with .h if you insist, but then I would very much prefer if it was moved to detail subdir as not to make clutter. These files are not standalone, like all well-written header files should be, so having a different extension is a hint to the user that IDE errors are expected when viewing them

[Bronek]

never mind, I see they have been moved to detail, thanks !

[thejohnfreeman]

I moved them to detail/ subdirectory.

[mvadari]

This is a useful change and would make the manual process of updating third party libraries with new field types, transactions and ledger entries a little easier. However, an even more useful change would be to abstract the definition files into easily machine parsable formats, maybe either JSON or CSV, so that any third party API libraries can choose to auto-generate their data structures and stream encoders and decoders from those files, thereby keeping up to date much more easily than using git blame to find recent changes to the large definition files.

@donovanhide does the server_definitions RPC solve your problem? https://xrpl.org/docs/references/http-websocket-apis/public-api-methods/server-info-methods/server_definitions

[donovanhide]

@mvadari Thanks for pointing it out. It is interesting and I was unaware of it, but it is not the full answer. I need to know all the fields for each ledger entry and transaction type, but that seems to be missing from that response.

[tequdev]

I need to know all the fields for each ledger entry and transaction type, but that seems to be missing from that response.

Note that the internal structure of fields defined with CONSTRUCT_UNTYPED_SFIELD, such as the sfMemo field, for example, cannot be known (as before).

[donovanhide]

Just to be clear of my use case, it is updating the data structures here:

https://github.com/rubblelabs/ripple/blob/master/data/ledgerentry.go
https://github.com/rubblelabs/ripple/blob/master/data/transaction.go
https://github.com/rubblelabs/ripple/blob/master/data/format.go
https://github.com/rubblelabs/ripple/blob/master/data/flags.go

And the canonical source of truth for these data structures, encodings and flags are the source code files being changed in this PR. It would be much more efficient if my library and others could be derived from an easily machine readable file rather than a hard to diff C++ file. Seeing as this is the essence of the Ripple protocol, it doesn't seem like an unreasonable request :-)

[mvadari]

Just to be clear of my use case, it is updating the data structures here:

https://github.com/rubblelabs/ripple/blob/master/data/ledgerentry.go https://github.com/rubblelabs/ripple/blob/master/data/transaction.go https://github.com/rubblelabs/ripple/blob/master/data/format.go https://github.com/rubblelabs/ripple/blob/master/data/flags.go

And the canonical source of truth for these data structures, encodings and flags are the source code files being changed in this PR. It would be much more efficient if my library and others could be derived from an easily machine readable file rather than a hard to diff C++ file. Seeing as this is the essence of the Ripple protocol, it doesn't seem like an unreasonable request :-)

@donovanhide that's a fair statement, but I think it should be in an RPC rather than in the source code (perhaps added to server_definitions). Let's move this discussion to this issue I just opened on the topic.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.2%. Comparing base (bf4a7b6) to head (9099f71).

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5122     +/-   ##
=========================================
- Coverage     76.2%   76.2%   -0.0%     
=========================================
  Files          760     762      +2     
  Lines        61568   61568             
  Branches      8126    8126             
=========================================
- Hits         46898   46897      -1     
- Misses       14670   14671      +1     

Files with missing lines	Coverage Δ
include/xrpl/protocol/Feature.h	100.0% <ø> (ø)
include/xrpl/protocol/SField.h	100.0% <ø> (ø)
include/xrpl/protocol/detail/ledger_entries.m	100.0% <100.0%> (ø)
include/xrpl/protocol/detail/transactions.m	100.0% <100.0%> (ø)
src/libxrpl/protocol/Feature.cpp	94.6% <ø> (ø)
src/libxrpl/protocol/LedgerFormats.cpp	100.0% <ø> (ø)
src/libxrpl/protocol/SField.cpp	77.5% <ø> (ø)
src/libxrpl/protocol/TxFormats.cpp	100.0% <ø> (ø)

... and 4 files with indirect coverage changes

[thejohnfreeman]

Whenever it comes time to merge this branch with the changes in #5143, regardless of whichever lands in develop first, the conflict resolutions can be cherry picked from my vault branch.