feat: index numerical and date fields in Solr with appropriate types + more targeted search result highlighting

[vera]

What this PR does / why we need it:

Currently, all fields regardless of type are indexed in Solr as English text (text_en). With this PR, numerical and date fields are indexed in Solr with appropriate types:

Field type defined in TSV	Field type indexed in Solr
int	plong
float	pdouble
date	date_range (solr.DateRangeField)

I chose to index dates as DateRangeField because they can be used to represent dates to any precision, e.g. a day YYYY-MM-DD, a month YYYY-MM or a year YYYY. See: Date Formatting and Date Math :: Apache Solr Reference Guide

This matches the allowed formats in a date field as defined by Dataverse.

This means that range queries are now possible on numerical and date fields, e.g. exampleIntegerField:[25 TO 50] or exampleDateField:[2000-11-01 TO 2014-12-01].

Which issue(s) this PR closes:

This PR implements ranged queries as discussed in #370 (issue was already closed)

This issue is related to #8813 and IQSS/dataverse-frontend#278 (the range queries that are now possible lay the groundwork for a nicer search facet UI)

Special notes for your reviewer:

For testing, I've created a sample TSV containing all relevant fields here.

Suggestions on how to test this:

Regression Testing

• On a server running "develop", publish some datasets that use the affected fields mentioned in the release note (coverage.Depth, etc.). Be sure dates, integers and floats are all used by populating various different fields.
• Deploy this branch
• Test, test, test for any regressions

Feature testing

1. Load sample TSV and update + reload Solr schema as described in docs
2. In the UI:
   1. Activate metadata block
   2. Activate facets for all three fields
   3. Create dataset with values in all three fields
3. Run test range queries via the search bar, e.g. exampleIntegerField:[25 TO 50] or exampleDateField:[2000-11-01 TO 2014-12-01]
4. Check that facets are working correctly

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Facets still look the same as before. There is only a small change in the highlighting of search results, see my comment below

Is there a release notes update needed for this change?:

Yes, there should be an info text describing the new feature + instructions for how to activate the feature:

• the Solr schema.xml needs to be updated
• all datasets need to be reindexed

Additional documentation:

/

[coveralls]

coverage: 20.866% (-0.006%) from 20.872%
when pulling ed7e38e on vera:feat/solr-field-types
into 050064e on IQSS:develop.

[vera]

Additionally, I've set hl.requireFieldMatch to true:

If false, all query terms will be highlighted for each field to be highlighted (hl.fl) no matter what fields the parsed query refer to. If set to true, only query terms aligning with the field being highlighted will in turn be highlighted.

https://solr.apache.org/guide/solr/latest/query-guide/highlighting.html

Two reasons:

1. Querying solr with a date range query with activated highlighting using the default (unified) highlighter without requireFieldMatch triggers a 500 error in Solr (see my post on the Solr mailing list. My guess is that Solr is attempting to highlight the matched date range within fields in a nonsensical way which triggers the error)
2. I think this improves the highlighting of search results, previously a match of my search term is highlighted anywhere even if I limited my query to a specific field, e.g. here "replication" is also highlighted in the title even though I limited my search specifically to the description:

With this change, the highlighting is limited to specific fields if the query is:

[johannes-darms]

@qqmyers Would it make sense to include this feature in the next release? It's a rahter small adaption that improve the serach experience.

[pdurbin]

@johannes-darms it's a very cool feature that adds a lot of value, something I've wanted for years.

Let's see what @cmbz and @scolapasta think.

[qqmyers]

One question - is it possible to enter or have legacy values that don't fit the new types that would break indexing?

[cmbz]

2024/10/15: Added to sprint ready after conversation with @pdurbin

[vera]

@qqmyers good question. When I tried to enter to invalid data (non-integer in an integer field, non-float in a float field, or non-date in a date field), I got the following errors via the UI:

...and via the API:

{"status":"ERROR","message":"Validation Failed: Example integer field is not a valid integer. (Invalid value:edu.harvard.iq.dataverse.DatasetFieldValueValue[ id=null ]), Example floating point field is not a valid number. (Invalid value:edu.harvard.iq.dataverse.DatasetFieldValueValue[ id=null ]), Example date field is not a valid date. \"yyyy\" is a supported format. (Invalid value:edu.harvard.iq.dataverse.DatasetFieldValueValue[ id=null ]).java.util.stream.ReferencePipeline$3@6e9605fb"}

The validator code seems to be quite old, so I don't know if there could be any installations with legacy invalid values entered before it was added.

However, looking at the validator code, I found that date fields allow some formats which are not documented: YYYY followed by AD or BC, a "Bracket format" (not familiar with this), and datetime formats (yyyy-MM-dd'T'HH:mm:ss, yyyy-MM-dd'T'HH:mm:ss.SSS and yyyy-MM-dd HH:mm:ss).

When I add a dataset using one of those formats and try to index it, the indexing fails. The Dataverse log shows an error like dev_solr> org.apache.solr.common.SolrException: ERROR: [doc=dataset_2_draft] Error adding field 'exampleDateField'='2024-09-01 12:34:56' msg=Couldn't parse date because: Improperly formatted datetime: 2024-09-01 12:34:56 and the dataset is missing from the UI.

So, yes, there may be some installations with date field values using the above formats which would cause invisible datasets due to indexing errors. I am not sure how we should deal with this. Are those date formats intended to be officially fully supported/widely used?

If no, it might be OK to offer a workaround in the upgrade instructions like "Please ensure that all date fields containing legacy dates in formats other than YYYY, YYYY-MM, YYYY-MM-DD, or YYYY-MM-DDThh:mm:ssZ are not updated to the new date_range type in the Solr schema. Otherwise, datasets with these legacy dates will fail to index and disappear from your Dataverse page.".

If yes, this feature becomes a bit more complicated, because Solr does not support those date formats and we would need to work around that somehow.

[vera]

@pdurbin I've just added an API test for the range queries as you suggested.

[qqmyers]

I don't know what is required but I wouldn't be surprised if BC dates are something people want and want to have indexed. Perhaps @jggautier would know more about what legacy values exist and what's required.
W.r.t. validation - I'd also make sure that the API calls don't allow bad values - I know there was newer validation code just added there. W.r.t. the code, I'd definitely suggest doing checks for int, float, date values when indexing or somehow assure that a bad legacy value can't break the overall submission and we just drop that field rather than have the dataset not index. (I didn't see such a check but I might have missed it).

[vera]

I wouldn't be surprised if BC dates are something people want and want to have indexed.

It wouldn't be a problem in general. Solr does support BC dates, however in a different format than YYYYBC:

-0009 – The year 10 BC. A 0 in the year position is 0 AD, and is also considered 1 BC.

https://solr.apache.org/guide/solr/latest/indexing-guide/date-formatting-math.html

W.r.t. validation - I'd also make sure that the API calls don't allow bad values - I know there was newer validation code just added there.

Is the code doing checks for API-submitted datasets different from the code doing the UI checks? I assumed it was both the same code I linked above, since the error messages are the same.

I'd definitely suggest doing checks for int, float, date values when indexing or somehow assure that a bad legacy value can't break the overall submission and we just drop that field rather than have the dataset not index.

Yes, that would be nice.

[jggautier]

Hi all. I haven't been following this issue closely enough to contribute and won't have the time to catch up. But I agree with Jim and encourage folks to look into how others have and are using these fields. My dataset at https://doi.org/10.7910/DVN/2SA6SN might be helpful for seeing who's using the fields in different ways. And the list of contacts in our spreadsheet of Dataverse installations might help for contacting particular installations to learn more.

[pdurbin]

Here's the related issue about BC dates:

• Feature Request: BCE dates #10843

[vera]

I've just pushed a commit that implements the suggestion above (if encountering a bad legacy value in an int/float/date field, just drop that field, but index the rest of the dataset).

So, if a dataset contains a bad legacy value in an int/float/date field, this means that queries on that field will not yield that dataset, since the field hasn't been indexed. But for any other query, the dataset will still be found. (I've also added a test showing this)

I think this is a small limitation. And we could add BC support relatively easily in a future PR.

[pdurbin]

@vera I poked your PR today. I'm very excited about this feature. I started looking into how existing fields will be affected. Please take a look at this PR of suggestions:

• suggestions for #10887: Solr field types vera/dataverse#1

Also, it's expected that range queries do not produce highlights, right? I'm getting hits but no highlights:

[vera]

Thank you! Yes, I think that's expected. I also haven't seen any highlighting for range queries, and didn't find any indication that Solr supports it in the documentation. I've just sent a mail to the Solr user mailing list to make sure. I'll take a look at your suggestions now. :)

[vera]

@pdurbin Suggestions looked good, I have merged them!

[pdurbin]

API tests are passing. I'm approving this. @vera please note that I added "regression testing" to "how to test this". I played around with an empty database (a new installation) but I didn't test with existing data.

One last thing I'll mention is that while working on this I created a pull request at #11024. I think it would be get the doc changes into the guides. I used them to update the Solr schema.xml file.