Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest process is overwriting corrected data #6

Open
caparker opened this issue Oct 26, 2022 · 0 comments
Open

Ingest process is overwriting corrected data #6

caparker opened this issue Oct 26, 2022 · 0 comments

Comments

@caparker
Copy link
Collaborator

This issue is specific to country because that is a value that we are correcting after data is ingested. This is what appears to be happening

  • Data comes in without a country code but with coordinates, realtime method either inserts new node or updates old node
  • The run_updates process than fixes the country code by using the coordinates
  • The next time that the data comes in we update it and we are back to a blank country code until the updates are run again

I propose one of the following

  • Fix issues with data before doing the comparison? Would be overly resource intensive for no good reason
  • Dont let the updater take a value and make it null? Though that might be desired in some cases
  • Fix the update process (so its not happening every time) and then come up with a better diff method that would ignore values that are later updated.

I advocate for the last one and if we cant get that to work we also add one of the more heavy handed approaches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant