Metadata needed for level of semantic validation for QA pairs

[jh111]

Kara is adding helping scale the number of SME-free test assets.

We need is one (or more) columns in testingAssetsfor50Github_2024-01-02_20240102evaluation.xlsx to represent the level of validation, and guidelines for values that go in that column.

Currently, most QA pairs hav validation SME or SmuRF. New QA pairs may have no validation yet (but are still useful for delta), or "came from Chat GPT".

In a related note, we have to decide if this xls is going to remain the one place for all QA pairs, or if we want multiple spreadsheets and/or a database.

[karafecho]

Here is a link to the G-sheet. (I think the one above may be broken.)

I added Column J, "Method of Generation", and included a dropdown menu with seletions of: Manual, SMuRF; Manual, SME; Automated, LLM; Other (please specify). I also added Column K, "Level of Validation", and included a dropdown menu with selections of: SMuRF; SME; No validation; Other/multiple (please specify). There may be a better way of capturing the desired information, so please feel free to delete the columns and suggest another approach.

Question for @jh111 : I was viewing ChatGPT as a method for (quickly) generating assets, but not necessarily validating them. I think that's what you mean, right? If not, then the approach I suggested will not work.

[jh111]

Your two column approach works well and I agree with ChatGPT generated, No Validation.

I supposed in the future we could have other types of validation. We could have ChatGPT vX confirmed, but I'm not sure how heavily I'd weigh that validation...