How are you resolving collisions currently? If you're using impulses, you might want to damp the resolving collision impulse by some factor < 1 and see if that helps things settle
Currently, I use line vs line to detect collisions, and find the shortest of the line normals to push out. I use how far the collision point is along both lines to weight the push outs to account for rotation. If I damp the push out vector (which also effects velocity, as I use motion verlet), objects sink into each other and it becomes more unstable
How are you resolving collisions currently? If you're using impulses, you might want to damp the resolving collision impulse by some factor < 1 and see if that helps things settle
Currently, I use line vs line to detect collisions, and find the shortest of the line normals to push out. I use how far the collision point is along both lines to weight the push outs to account for rotation. If I damp the push out vector (which also effects velocity, as I use motion verlet), objects sink into each other and it becomes more unstable