PROTON-2792: [cpp] Segmentation fault in container::impl::run_timer_jobs

[zlomek]

PROTON-2792

[DreamPearl]

@astitcher @cliffjansen

[astitcher]

There is a clear thread safety violation in run_timer_jobs use of is_active_ without using the appropriate lock - in fact this is so obvious that I'm puzzled we haven't seen it in the TSAN or even static analyzer runs - maybe we just missed it!
Unfortunately this fix has changed the previous semantics of task cancellation: An important way to cancel a task is from a previous task callback. However this change makes that unreliable as it only checks for task cancellation before executing the entire batch of tasks that are runable at the point where run_timer_jobs is called. So if an earlier task in the batch cancels a later task in the batch then that cancellation would not be honored.
I'm looking at a fix that should work correctly now and I'll post it here when I'm done.

[astitcher]

My suggested fix is #422. @zlomek please check it out and see if it fixes your crash. I'd like to add some tests before merging this so that we make sure the crrect semantics are maintained with any further fixes and also preferably to exercise the race you found.

[zlomek]

Closing this in favour of #422.