Multimodal Models of Time Series and Text

Abstract

Time series are critical data which drive countless decisions in finance, healthcare, and science. However, multimodal NLP research has mostly focused on images and video. Here I enumerate barriers towards such models and describe my work towards mitigating them. I detail new multimodal NLP tasks for reasoning about time series, describe an LLM-powered agent that can answer questions about time series, and present methods for pretraining time series encoders. I also share work on using language models for code generation to assist scientists.