Skip to content

hathitrust/hathifiles

Repository files navigation

Tests Coverage Status Ruby Style Guide

Getting Started

Developer Setup

git clone https://github.com/hathitrust/hathifiles
git submodule init
git submodule update
bin/setup_test.sh

Running Tests

docker compose run --rm hf bundle exec standardrb
docker compose run --rm hf bundle exec rspec

Hathifiles Generation

bundle exec ruby jobs/generate_hathifile.rb (upd|full)

Generates a metadata extract for public use, from the zephir _upd_ and _full_ files.

Hathifiles Listing

bundle exec ruby jobs/update_hathifile_listing.rb

Moves Hathifiles to the appropriate directory and generates the listing on the website.

Field Definitions

End user documentation is available on the Hathitrust website. The following is a more precise definition of the MARC extractions.

Column # Data element Field name in [header file] (https://www.hathitrust.org/filebrowser/download/269539) Description
1 Volume Identifier htid The "permanent" HathiTrust item identifier. Taken from the 974$u
2 Access access "allow" or "deny" indicating very generally whether or not users can view the item. N.B. it is United States biased. It is "allow" if 974$r is "pd" or "pdus", or if it starts with "world", "ic-world", "cc", or "und-world". All other values in 974$r get "deny".
3 Rights code rights 974$r
4 HathiTrust record number ht_bib_key HathiTrust's bib record number. Taken from the 001 field, without processing.
5 Enumeration/Chronology description 974$z or empty string
6 Source source In theory, it is the "code identifying the source of the bibliographic record." In practice, it is taken from the 974$b so may be unrelated to the bib record found in the catalog.
7 Source institution record number source_bib_num Local bib record number taken from the 035 sdr nums using the 974$c (collection) and a [prefix mapping] (https://github.com/cdlib/hathitrust-contrib-configs)
8 OCLC numbers oclc_num Extracted from the 035s with the regex: /((oco{0,1}lc)
9 ISBNs isbn 020a, stripped of whitespace, uniqd, and joined with ","
10 ISSNs issn 022a, stripped of whitespace, uniqd, and joined with ","
11 LCCNs lccn 010a, stripped of whitespace, uniqd, and joined with ","
12 Title title 245abcnp, stripped of whitespace, joined with ","
13 Publishing information imprint 260bc, stripped of whitespace, joined with ", ". If no 260bc is found, then we look for 264
14 Rights determination reason code rights_reason_code 974$q or ""
15 Date of last update rights_timestamp time field taken from the rights_current table of the rights database. Formatted as %Y-%m-%d %H:%M:%S
16 Government Document us_gov_doc_flag 1 if a US federal goverment document. 0 otherwise. Uses the third character of the pub place code and if character 28 in the 008 is "f". It incorporates [multiple exceptions] (https://github.com/hathitrust/hathifiles/blob/main/lib/us_fed_doc.rb#L7) that mean this is better considered a "Government Document that is also public domain".
17 Publication Date rights_date_used 974$y or, if empty, "9999"
18 Publication Place pub_place Three digit code for the place of publication taken from the 008 with some minor tweaks
19 Language lang 008[35-37] or " "
20 Bibliographic Format bib_fmt An extraction from the Leader, this is a reimplementation of https://github.com/mlibrary/traject_umich_format/blob/7d355a5be133dc86f8795954fdd2e01355758309/lib/traject/umich_format/bib_format.rb#L18
21 Collection Code collection_code 974$c
22 Content Provider Code content_provider_code The content_provider_cluster field taken from the ht_collections table using the collection_code
23 Responsible Entity Code responsible_entity_code The institution that took responsiblity for accessioning the content into HathiTrust. The responsible_entity field taken from the ht_collections table using the collection_code
24 Digitization Source digitization_agent_code The organization that digitized the content. 974$s
25 Access profile access_profile_code Indicates whether an item has view or download restrictions. The access_profile field from the rights_current table for this htid.
26 Author author 100$abcd, 110$abcd, and 111$acd stripped and uniqd, joined by ", "

Catalog Redirect: Generate hathifile history and compute record redirects

We want to generate redirects for catalog records that have been completely replaced by.

Basic usage: add_monthly_and_dump_redirects.rb

bundle exec ruby bin/add_monthly_and_dump_redirects.rb \
../archive/hathi_full_20211101.txt.gz

In general, the only script we really need is add_monthly_and_dump_redirects.rb. It does the following:

  • Find the most recent full hathfiles dump in /.../archive/hathi_full_YYYYMMDD.txt.gz
  • Figure out current/previous month (and thus filenames) from the found filename
  • Load up the data from history_file/#{yyyymm_prev}.ndj.gz
  • Add the data from the passed file
  • Dump the updated data to history_file/#{yyyymm_current}.ndj.gz
  • Compute the redirects (all of them, not just new ones) and dump them to redirects/redirects_#{yyyymm_current}.txt as two-column, tab-delimited lines of the form old_dead_record current_record

add_monthly_and_dump_redirects.rb can optionally take all those things as arguments; run with -h to see them.

Other scripts

bin/dump_redirects_from_history_file history_files/202111.ndj.gz my_redir_file.txt.gz dumps the redirects from an existing file.

bin/initial_load.rb is the script that was used to load all the monthlies to get everything up to date. It will only be useful if we need to rebuild everything.

Performance

Running under ruby 3.x it takes about 30-40mn.

Idempotence-ish

Because each history file is kept, it's easy to roll back to a given point and start from there. There's no database so no need to roll back any data or anything complex like that.

Using the underlying HathifileHistory code

$LOAD_PATH.unshift 'lib'
require 'hathifile_history'

hh = HathifileHistory.new_from_ndj('history_files/202110.ndj.gz')
hh.add_monthly("hathi_full_20211101.txt")
hh.dump_to_ndj('history_files/202111.ndj.gz')

# Eliminate any ids that are no longer used
hh.remove_missing_htids!

# ...or just get a list of them without deleting
# missing_ids = recs.missing_htids

# Compute and dump valid record redirect pairs

File.open('redirects/redirect_202111.txt', 'w:utf-8') do |out|
  hh.redirects.each_pair do |source, sink|
    out.puts "#{source}\t#{sink}"
  end
end

Generated files

redirects_YYYYMM.txt are tab-delimited files, two columns, each a zero-padded record id, old_dead_record current_record

YYYYMM.ndj.txt are json dumps of the ginormous data structure that holds all the history data (along with some extra fields to allow easy re-creation of the actual ruby classes upon load).

Data explanation and memory use

This whole project is just is simple(-ish) code to build up a history of

  • which HTIDs were added to which record IDs
  • and when was it added
  • and when was it last seen on this record in a hathifile
  • and when was the record last seen in a hathifile

When a file is loaded, it computes the year/month (YYYYMM) from the filename and notes which HTIDs are on which records, and which ids are seen at all. We end up with a big hash keyed on record id that contain data similar to this structure:

{
  rec_id: "000001046",
  most_recently_seen: 202111, # record appeared in Nov 2021 hathifile
    entries: {
    "mdp.39015070574192" => {
      appeared: 200808,
      last_seen_here: 202111 # was seen on this record Nov 2021
    }
  }
}

Because the queries we want to do can be pretty expensive in SQL-land, and because we have gobs of memory, the whole thing is stored in memory for processing, and later dumped to newline-delimited JSON (.ndj.gz) files for loading up again the next month.

How redirects are computed

We reduce the computation of redirects to say that record-A should redirect to record-B iff every record that has ever been on record-A is currently on record-B.

Things we do not redirect:

  • records whose component HTIDs have ended up on more than one record
  • records that current exist cannot be a source
  • records that no longer exist cannot be a target

To find the redirects:

  • Eliminate HTIDs that don't exist anymore. Otherwise, htid -> new_rec -> htid-dies could make it seem like htids got split over multiple records.

  • Build a hash of htid -> current_record by buzzing through all the htids and checking most_recent_appearance

  • For each record that was not seen in the most recent load (so, deleted records): *

  • Get a list of all the HTIDs that have ever moved

  • For each moved HTID

    • Figure out where it currently lives (record_current)
    • For every other record it's ever lived on record_past, see if record_current.htids.superset?(record_past.htids)
    • If so, set up a redirect from record_past to record_current