How to troubleshoot the following error:
SQL Error  [HY000]: Leaf Error (127.0.0.1:3307): Compilation submission failed. If the error persists please restart MemSQL. This may have been due to an overusage of compilation memory or accidental process kill. If the issue continues after restart, please set the max_compilation_memory_mb flag.
Hitting error 2423 compilation submission failed errors typically means the node specified in the error message has a defunct memsqld process. A common scenario where you might have a defunct memsqld process would be after the memsqld process on the respective node from the error message invoked the Linux OOM Killer.
Check for invocation of the Linux OOM Killer
To confirm if the Linux OOM Killer killed the memsqld process, check the dmesg log for Out of memory events involving memsqld:
[Mon Apr 5 13:16:46 2021] Out of memory: Kill process 8657 (memsqld) score 899 or sacrifice child
[Mon Apr 5 13:16:46 2021] Killed process 8662 (memsqld), UID 992731, total-vm:284964kB, anon-rss:520kB, file-rss:4kB, shmem-rss:0kB
Restart node(s) with defunct memsqld process
To restore the memsqld process to a working state, restart any node(s) with defunct memsqld processes.
- Check the output of ps on the node in question to see the state of the memsqld process.
root 1234 0.0 0.0 0 0 ? Zs Mar21 0:01 [memsqld] <defunct>
- Issue a restart to the node with sdb-admin restart-node, documented here.
Prevent memsqld from invoking the Linux OOM Killer
To prevent memsqld from invoking the Linux OOM Killer, consider the following suggestions.
- Memory limits may be misconfigured on the SingleStore nodes themselves. If maximum_memory is set too high on any node, then SingleStore may be allowed to use enough memory to invoke the Linux OOM Killer. In this case, check the value of maximum_memory. In most cases, the sum of maximum_memory of nodes on any given host should not be configured to greater than 90% of physical host memory.
- Confirm all Linux configurations meet all of the recommended requirements.
- SingleStore is not designed to be co-located with other memory-intensive programs for a few reasons, one of them being the avoidance of the Linux OOM Killer. The Linux OOM Killer will usually kill the memsqld process because it is likely to be consuming the most amount of memory when the Linux OOM Killer is activated.
- Consider increasing max_compilation_memory_mb to a higher value than the default. max_compilation_memory_mb is the maximum amount of memory used to compile a query.