Over the last several years, there has been a growing focus on the CL field in the context of machine learning and its goal to create models capable of learning new tasks step by step without loss of prior knowledge. Among these, catastrophic forgetting is especially challenging in real-world settings where the data experience changes over time. To this effect, what has become pivotal for models is mechanisms for memory update to enable the models to learn information as well as update what has been previously learned easily. This survey specifically investigates the memory update strategy in the continual learning setup wherein new categories and domains are continuously added in the text datasets including sentiment analysis, named entity recognition, text classification tasks etc. Moving on, three primary memory update strategies of memory replay, memory consolidation, and parameter isolation are discussed; this paper further addresses certain adaptations of the proposed methods for text-based applications. Memory replay means that part of previous data is stored to be replayed when new tasks are learned while memory consolidation strengthens only significant memories. Parameter isolation helps avoid masking previous tasks or overwriting the parameters when the machine learning algorithm is trained to accomplish new tasks. In this paper, we discuss the latest in these techniques and offer a thorough insight into their use in text datasets such as Amazon Reviews and Yelp Reviews. Further, we outline the primary drawbacks of existing solutions for memory updates such as capacity limitations, domain variation, and continually learning without having access to new task information. In addition, a summary table of literature review identifying the most relevant works within the field is offered. Lastly, we discuss the remaining issues and potential research directions where more focus and development should be given in CL for text data by noting the importance of efficient and adaptive update policies towards the memory.