Why does a leaf take longer than an aggregator to startup? What is it doing?
Leaf nodes are where all the sharded data resides in SingleStore, so they generally have much more data than aggregator nodes that need to be loaded from snapshots and transaction log files when they startup. This is especially true for rowstore tables (for which the full table data must be instantiated in memory) and true for columnstore metadata and indexes.
Leaf startup times are expedited somewhat when there is a more recent snapshot available since instantiating from a snapshot binary is faster than replaying through a transaction log. In the ideal case, a snapshot was taken immediately before a node last shut down so that it contains the entirety of the node's data. In contrast, the corresponding transaction log is empty. This may not always be possible (e.g., if a node is restarting due to an unexpected host crash), but for planned restarts, maintenance windows, upgrades, etc., the leaf restart time can be optimized by running (on the Master Aggregator) SNAPSHOT DATABASE <database_name>; for all user databases before the shutdown, as described here.
Snapshots can also be executed via Singlestore Toolbox, as described here.