Optimize destruct somewhat.

Avoid keeping track of if destroy() has been called for objects that
lack destroy().