Silicone Ice Tray SLUDGE

I used silicone ice cube trays to make clear ice cubes for cocktails. I did not realized the residue the silicone ice cube trays produced. You can clean the white residue with vinegar, but it is apparent that there is sludge and it will always reappear.