Abstract: In the gear grinding process, the selection of process parameters directly affects the surface quality of the processed gear and the spindle vibration. To obtain the optimal combination of process parameters for the best surface quality of the gear and the lowest spindle vibration, a grinding experiment was designed using the response surface methodology (RSM), with the grinding wheel linear speed, generating speed, and grinding depth as the optimization variables. A multi-objective optimization model was constructed with the goals of minimizing surface roughness, minimizing spindle vibration, and maximizing residual compressive stress. An optimization decision-making method was proposed, integrating the multi-objective whale optimization algorithm (MOWOA) with the analytic hierarchy process (AHP) and the technique for order preference by similarity to an ideal solution (TOPSIS). The most desirable solution was obtained after ranking. Experimental results show that the optimized process parameters can reduce the vibration of the machine tool spindle during gear grinding by 7.43%, lower the surface roughness of the ground gear teeth by 18.96%, and increase the residual compressive stress by 13.02%.