This paper addresses the integrated process planning and job-shop scheduling problem for mass customization in a reconfigurable manufacturing system. A bi-objective mixed-integer non-linear programming mathematical model for minimizing the total tardiness penalty of products and the total cost covering setup, machine reconfiguration as well as processing activities is built to formulate the problem. A Q-learning based reinforcement learning solution approach is presented to solve the formulated problem. Numerical experiments were carried out to validate the mathematical model and the solution approach. The computational results of the numerical examples show the great efficiency of the proposed solution approach in the aspect of computation time, compared with NSGA-II and the exhaustive search. The effectiveness of the problem-specific designed policies is also discussed.