backward_pass now clears nodes which will not be used

[Etragas]

Attempting to help issues mentioned in #219

This line removed any references to Array nodes which will not be revisited, since they have passed all their gradient information to their parents.

In the special case of the start_node, the gradients will not be lost, since setting outgrads[node] to None only deletes the reference, not the values. So as long as cur_outgrad holds on to the gradient reference all is fine.

On a test dataset, using a proprietary algorithm which can be roughly described as two RNN's feeding into eachother, I obtained the following gains in performance.

https://cloud.githubusercontent.com/assets/6620250/25814415/6a8f085a-33eb-11e7-9237-ce58fa09300a.png

In addition, I compared the numerical results of my change to the previous version, and they were identical.

[j-towns]

Your code is throwing this error on Python 3:

ERROR: Failure: TabError (inconsistent use of tabs and spaces in indentation (core.py, line 43))

should be easy to fix :)

[Etragas]

Haha, sorry about that.
Should be good now.

[j-towns]

This looks good to me, being pernickety maybe it would be slightly cleaner to use

del outgrads[node]

or

outgrads.pop(node)

to completely remove node from the dictionary.

This would mean no left over/unnecessary reference to node (edit: does a dict contain references to its keys?).

[Etragas]

I agree that it's cleaner, since it also drops the key as opposed to just setting it to None.

[mattjj]

Thanks for this improvement, @Etragas and @j-towns! This is open-source software collaboration at its best.

I marked @dougalm as a reviewer because I think he's also modifying this part of the code, but on reflection I see no reason not to merge this change immediately, so long as he doesn't accidentally clobber it.

Thanks again for this!

[mattjj]

For posterity: see this related comment.