The solver follows feature motion
If a car drives across the frame, features on the car do not describe the camera movement. They describe the car movement. Too many of those tracks can pull the solve away from the real background.
Mask the problem before Heavy Tracking
MotionMaster 3D can create SAM2 AI masks for objects like cars or people, add polygon masks for manual regions, and compose multiple layers into one mask sequence. Then Heavy Tracking can track the black background or the white object region depending on the goal.