TERA: Time Series Prediction through Transformer-Enabled Retrieval Augmentation

Author

Ruslan Mikhailov, Mohamed bin Zayed University of Artificial IntelligenceFollow

Date of Award

4-30-2024

Document Type

Thesis

Degree Name

Master of Science in Machine Learning

Department

Machine Learning

First Advisor

Dr. Mingming Gong

Second Advisor

Dr. Zhiqiang Shen

Abstract

Time series prediction plays a crucial role in many fields, including finance and healthcare, where forecasting future events with high accuracy using historical data can result in significant breakthroughs. More efficient techniques are required because traditional methods frequently need to improve when dealing with the complexity of real-world data. Transformer-Enabled Retrieval Augmentation (TERA), a new approach that uses retrieval-augmented mechanisms within transformer-based encoder-decoder architectures to improve predictive accuracy, is introduced in this thesis. The central hypothesis of this study is that continuous re-parametrization of the knowledge base can help models perform better in forecasting tasks. The study conducts a three-dimensional analysis to test this hypothesis: it compares retrieval-augmented methods with one another, assesses their performance on various datasets, and investigates their effectiveness in different transformer configurations. TERA showed a significant improvement in error reduction with an average of 44% decrease in error compared to 23% for MQRetNN. Though TERA has many advantages, it also has drawbacks. For example, training time increases significantly, and complexity increases quadratically with the increase of context size. Additionally, TERA’s performance is limited on datasets that exhibit strong stationarity. These difficulties point to significant directions for further study in enhancing the retrieval process and broadening the techniques to different neural network topologies. By setting new benchmarks for accuracy and dependability, the results of this thesis advance the field of time series prediction research and pave the way for the valuable implementation of retrieval-augmented techniques in various dynamic and heterogeneous settings. The source code of the project is available publicly on github: github.com/Dmmc123/time-wise-rat

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the M.Sc degree in Machine Learning

Advisors: Mingming Gong, Zhiqiang Shen

Online access available for MBZUAI patrons

Recommended Citation

R. Mikhailov, "TERA: Time Series Prediction through Transformer-Enabled Retrieval Augmentation,", Apr 2024.