Finish/Redo experiment screen:

[IanEisenberg]

A few comments on this screen:

I think we want to change the text. Right now it makes it seem like if you don't like your result you can just change.

"You have completed the experiment. Click "Next Experiment" to keep your result, and progress to the next task. If you believe your ability to focus was significantly compromised by some external factor (e.g. someone started talking to you while you were doing the task) press "Redo Experiment" to be presented with the task again at a later time."

In line with that - "Next Experiment" should be the first button (or more prominent in some way). Redoing should be a rare thing.

Minor style points:
Would be great if this screen looked like the ad/consent/instructions.

Response and saving data

Another thing Russ mentioned was that we want to record how often/why they redo an experiment. Do we currently have any track of when a subject clicks "redo" and for what experiments?

Besides that, we'd also want a textbox such that if they click redo they must enter a reason why.

[vsoch]

Text and buttons are changed, please confirm.

[vsoch]

I do not think this idea of adding a feature to save information about a redo is a good idea - there are several concerns. First, where does it go? If you store it with the result object, it adds a new field that would be over-written if the user redid the experiment a second time. That's not a big deal, we could just save as JSON. A bigger issue is if it's stored with the result then deleting a result also means that you lose track of how often a user is redoing things, which could be important to know about a worker over time, across different batteries.

Adding this option to redo, and further, checking the reasons, also adds a manual intervention into the system. You will now need to manually check something every time that a user has chosen to redo an experiment. That's fine for 8 people, but what happens when you go up to 8000? I think we should either give the option to redo with substantial warning / detail (or some cost) OR just eliminate the option all together. If a user is not responsible or has some occurrence happen to not be able to complete a task to the best of his ability, this will be caught when the data quality is checked, and the experiment will not be counted.

[IanEisenberg]

I still like the idea of a redo button. In long batteries the chance of something coming up (phone call, bathroom, child, etc.) goes up. We don't have a pause button, but we could have a redo button. This isn't a perfect cure - we don't really want some people to have more practice on a task, but does alleviate this concern somewhat. If these were individual experiments, data quality checks/rejection would be fine, but within the context of a larger battery that may lead to unnecessary loss of data (and wasted money/time if it happens after the participant has completed many tasks).

Given that, I think we need the redo button. Is it not possible to store it in the battery object? As a dictionary for each experiment added when someone redoes an experiment? battery_redo['test_task'] +=1, etc.?

[vsoch]

Storing redo for a participant with the battery object doesn't make sense at all, how would you ever distinguish between people?

[IanEisenberg]

Ah, I see. So the battery object and experiment object are subject independent. They are related to the specific instances of that battery and that experiment template. Only the result object is subject specific, right?

[IanEisenberg]

What about the worker object?

[vsoch]

Yes, a result object is subject, experiment specific.

[IanEisenberg]

So would the worker object work? We have additional need for this "subject" level data besides the redo button. I've mentioned before our need for rejection based on failing multiple experiments (rather than just one). This would require some kind of tracker of "number of experiments who failed credit check". Might it make sense to add all of those to the worker object?

If we want to keep the worker object a pure amazon object then we may need a "subject" object.

[vsoch]

We have a worker object. But that would mean that a worker would be assessed / rejected across ALL batteries.

[IanEisenberg]

Ah, okay. So then it seems we'd need a subject/battery specific object. Not linked to a particular experiment. That seems like a useful thing to have regardless. We can imagine (in the future) setting particular subjects to 'conditions' within a battery which affects experiments or the sequence of experiments they see. That'd be a cool addition later on! For now, we just need it to record things like how many "redos", how often they fail a credit check, etc.

[IanEisenberg]

@rwblair For now, let's just update this page to let the worker know how many tasks they have completed out of how many.

"You have completed 3 of 61 tasks."

And when they get to certain markers (half way, whatever), some motivational phrase: e.g. "You are half way done!"

[IanEisenberg]

Tested. With a battery with 3 experiments a message came up saying "X experiments left out of 5". So the total amount of experiments in the battery isn't calculated correctly. I added more experiments and it still says 5, so it doesn't seem to be calculating total number of experiments at all.

When there are 0 experiments left (so pressing "Next Experiment" would end/submit the HIT) no message comes up. Instead of no message, can we put "You have completed all experiments in this HIT" in the same place where you have been putting the "X out of Y" message?. Workers were commenting that they felt surprised when it just ended.