Abstract: To explore the quantitative relationship between unevenness characteristics and contact stiffness, this study analyzed the effects of surface roughness (micro-scale) and form error (macro-scale) on contact stiffness from these two respective scales. First, based on the fractal model of rough surfaces, the correlation between surface topography parameters and surface roughness was clarified, and the contact stiffness per unit area of the joint surface under different surface roughness conditions was compared. Second, the finite element method was adopted to establish a simulated surface with joint surface form error. The influence of flatness magnitude on contact pressure was analyzed, and the influence mechanism of flatness magnitude on the contact stiffness of joint surfaces was summarized. The results show that: The enhancement effect of reducing surface roughness (i.e., optimizing the microscale surface quality) on the stiffness per unit area exhibits an applicable boundary. The form error regulates the contact stiffness in a “first increase and then decrease” pattern by changing the contact characteristics in the medium pressure range (10–50 MPa). Finally, the study comprehensively analyzed the influence of unevenness characteristics at both micro and macro scales on the consistency of contact stiffness, providing theoretical guidance for the design and processing of practical joint surfaces.