Mathematics is one of the core subjects in the curriculum at national and international level as the direct implication on fostering various important qualities such as creativity, logical and abstract thinking, problem solving, originality and curiosity among the learners, by the subject Mathematics. However, it is quite disheartening to note that the low performance of Indian students at international assessment in Mathematics as reported by PISA 2009, Indian students ranked second last among 73 countries. Kumar & Karimi (2010) found that the anxiety in Mathematics has a negative effect with Mathematics performance and achievement though it has also been found that a degree of cognitive anxiety (worry and concern) may motivate students to try harder. Also, (Guin, 2018) observed that Phobia in Mathematics is not genetic but highly contagious. Every negative comment to the subject like it is hard, boring, abstract, useless etc. puts a negative image about Mathematics among the children. In the present study, the author has critically reasoned out the key factors leading to low performance of the students such as emphasis on rote memory than competency skills like critical, collaborative and creative thinking, connection the school knowledge to real life setting, rigidity in problem solving, shortage of qualified professional Mathematics teachers, more emphasis on completion of syllabus rather than meaningful learning of Mathematics concepts, inadequate Mathematics laboratories, improper teaching methods, system of assessment, student ’s understanding of conceptual and procedural knowledge in the main operations, rigidity in problem solving, etc. The present paper also highlighted the pedagogical factors, administration factors, institutional factors and evaluation factors in order to reason out the present status of mathematics learning in specific context to Indian schools. The paper has also been supported with the inferences drawn through an empirically conducted pilot study by the investigator on teachers and students with the purpose to identify the factors affecting the performance of students in Mathematics. The study has also included some concrete suggestions and recommendations to be employed at micro (Classroom level) and macro level (Administration/Institutional level) so as to improve performance of the learners in Mathematics.