While not a permanent fix, the decision was made to block TCP device registration. This affected less than 1% of our total registered devices. Devices that were configured for TCP must be reconfigured to use UDP. We are continuing to work with NetSapiens senior engineer staff to determine how a single errant device could crash the stack. FInally, we have instituted additional safeguards that will immediately move core dump files off server immediately to prevent a full storage again.
Update 11/26/19: Worked with NS engineering to isolate issues to TCP connections in SIP trunks only. SIP trunk TCP functionality was left disabled. TCP functionality for endpoints was restored and devices re-registered successfully. Systems have been stable sense. NS will continue to follow up regarding SIP trunk TCP.