Managed to find the first problem with the collision. After rotating the colliders I didn’t take into account the region it checks. Since the player car is not square but rectangular it basically was not checking all tiles that it should. And probably checking too many tiles than it should.
So now it rotates the collision volume correctly. Making the Separating Axis Theorem check the correct tiles. And I immediately could see an improvement. Though still there is something going wrong. But it now only happens at the very edge. When two tiles meet. That may indicate that the collision region is still not correct though. That’s what I have to found out now.