Upcoming machine tools need to be extremely efficient systems to maintain the needed intellectual performance and stability. The spindle tool system is necessary to optimize which enhances the rigidity of the spindle and in turn produces the cutting stability with higher productivity. Prediction of the dynamic behavior at spindle tool tip is therefore an important criterion for assessing the machining stability of a machine tool at design stage. In this work, a realistic dynamic high-speed spindle /milling tool holder/ tool system model is elaborated on the basis of rotor dynamics predictions. The integrated spindle tool system is analyzed with the Timoshenko beam theory by including the effects of shear and rotary deformation effects. Using the frequency response at the tool tip the corresponding stability lobe diagrams are plotted for the vertical end mill system. Furthermore an optimization study is carried out at design stage for the bearing system and the rotor positions for maximizing the chatter vibration free cutting operation at the desired depth of cuts with precise rotational speeds. It is markedly found that the first mode of vibration had a large impact on the dynamic stability of the system. The parametric studies are conducted such as tool overhang and bearing span and the influence of these on the system dynamics are identified and the corresponding stability lobe diagrams are plotted. It is evidently found that the second mode of the frequency response has critically affected the bearing span and competing lobes are identified. These results are assisted to design a spindle bearing system at the desired machining conditions. A neural network based observer is designed based on the simulation results to predict optimum design parameter values.