memory optimizations for pyannote.audio.core.inference.Inference.aggregate()

memory optimizations for pyannote.audio.core.inference.Inference.aggregate() #99