Just had recently a very positive experience with the applyInPandas spark function to calculate a group by with some customized function on big data fast.