Modeling Irregular Time Series: From Neural ODEs to Graph Networks and Foundation Models

The irregular time series problem has a natural mathematical framing: if observations arrive at arbitrary times, why not model the latent state as a continuous trajectory? This is exactly what Neural ODE-based models do.

Latent ODE (Rubanova et al., NeurIPS 2019) was the breakthrough. It places a Neural ODE inside a variational autoencoder: an ODE-RNN encoder reads observations at their actual timestamps to produce a latent distribution, then a Neural ODE decoder evolves the latent state forward continuously. The trajectory is defined by a neural network parameterizing dz/dt, with numerical solvers (e.g., Runge-Kutta) performing integration. This was the first model to show that continuous latent dynamics could handle irregular sampling natively, without imputation or discretization.

Published concurrently, GRU-ODE-Bayes (De Brouwer et al., NeurIPS 2019) took a different architectural path. Instead of a VAE framework, it defined a continuous-time GRU where the hidden state evolves according to GRU-style ODEs between observations, and a separate Bayesian update network incorporates each new observation. This single-pass filtering design is more natural for online/streaming scenarios (healthcare monitoring, real-time tracking) where you update beliefs incrementally rather than encoding the entire sequence first.

Both models share a critical bottleneck: numerical ODE solvers are slow. Each forward pass requires multiple integration steps, and backpropagation through the solver (via adjoint methods) adds further cost. Two subsequent models attacked this directly.

CRU (Schirmer et al., ICML 2022) restricts the latent dynamics to linear stochastic differential equations, which admit closed-form solutions via the continuous-discrete Kalman filter. The payoff is exact integration (no solver), principled uncertainty quantification (the Kalman filter naturally propagates variance), and faster training. The trade-off is reduced expressivity in the dynamics — though the authors show this matters less than expected on standard benchmarks.

Neural Flows (Biloš et al., NeurIPS 2021) take the opposite approach: keep nonlinear dynamics but eliminate the solver by directly parameterizing the solution map z(t) rather than the derivative dz/dt. Given an initial condition z(t₀), the network outputs z(t) in a single forward pass. The key insight is defining mathematical constraints (identity at t₀, invertibility) that guarantee the output is a valid ODE solution. This gives O(1) cost per evaluation versus O(N) for numerical integration, while retaining the expressivity of nonlinear dynamics.

A parallel research thread asked: do we actually need differential equations to handle irregular time? Attention mechanisms offer an alternative — they can naturally weight observations by learned relevance, regardless of when those observations occurred.

mTAN (Shukla & Marlin, ICLR 2021) was the pivotal model here. It learns a continuous-time embedding of timestamps using parametric functions (sines/cosines at learnable frequencies), then uses multi-head attention where queries are desired reference time points, keys are actual observation times, and values are the measurements. This produces smooth interpolations of the irregular data into a fixed-length representation that can be fed into any downstream model. The key result: mTAN runs 10–100× faster than Neural ODE baselines while achieving comparable or better performance, because attention is parallelizable while ODE solvers are inherently sequential.

STraTS (Tipirneni & Reddy, ACM TKDD 2022) made a radical representational choice. Instead of treating a multivariate time series as a matrix (features × time steps) — which forces you to deal with missing cells — STraTS represents it as a set of observation triplets (time, variable-id, value). Each triplet becomes one token, processed by a standard Transformer encoder. This means the model never sees 'empty' cells; sparsity is handled by construction. Two technical innovations make this work: Continuous Value Embedding (CVE), a small neural network that maps scalar values (timestamps and measurements) to dense vectors without discretization, and a self-supervised forecasting pretext task for pre-training on unlabeled clinical data.

ContiFormer (Chen et al., NeurIPS 2023) unified the ODE and Transformer threads. It defines attention in continuous time: latent states evolve via ODEs between observations, and the attention mechanism operates over these continuous trajectories rather than discrete tokens. Both queries and keys are functions of time, not fixed vectors. The authors prove that several existing architectures (including designs resembling mTAN) are special cases of ContiFormer under specific function hypotheses. A reparameterization trick enables parallel computation of continuous-time attention scores, avoiding the sequential bottleneck that plagues standard Neural ODE approaches.

The evolution here is clear: mTAN showed attention could replace ODEs for temporal modeling; STraTS showed you could eliminate the time-series matrix representation entirely; ContiFormer showed you could have both ODE dynamics and attention in a single principled framework.

Sequence models (whether ODE-based or attention-based) process irregular time series as flat sequences of observations. But multivariate time series have relational structure: sensors are physically connected, lab tests share physiological pathways, and IoT devices form spatial networks. Graph neural networks encode this structure explicitly.

RAINDROP (Zhang et al., ICLR 2022) was the first major GNN approach for irregular multivariate time series. It represents each sample as a separate sensor graph where nodes are variables and edges encode learned inter-sensor dependencies. The key mechanism: when an observation arrives at one sensor (a 'raindrop'), it sends messages to neighboring sensors ('creating ripples'). This means the graph dynamically reflects which sensors have observations at any given time — sparsity becomes a structural property of the graph rather than a problem to solve. Temporal self-attention captures within-sensor dynamics, while the GNN message-passing captures cross-sensor dependencies.

GraFITi (Yalavarthi et al., AAAI 2024) made a bold reformulation. Instead of the usual 'inputs → hidden states → predictions' pipeline, GraFITi casts forecasting as edge weight prediction on a bipartite graph. One set of nodes represents observed time-variable pairs; the other represents target time-variable pairs. A GNN adapted from Graph Attention Networks learns embeddings and predicts edge weights that yield forecasted values. This sidesteps ODE solvers and imputation entirely — sparsity is a natural graph property — and runs up to 5× faster than ODE-based methods with 17% better accuracy.

T-PatchGNN (Zhang et al., ICML 2024) bridged the patching paradigm (popularized by PatchTST for regular time series) with graph neural networks. It divides each univariate irregular series into 'transformable patches' — segments covering a uniform time horizon but containing a variable number of observations. A Transformer handles intra-series (temporal) modeling within patches, while time-adaptive GNNs model inter-series (cross-variable) correlations through learned time-varying graph structures. The patching avoids the sequence-length explosion of processing every observation individually.

These three models share a conviction: irregular multivariate time series are inherently graph-structured, and making this structure explicit improves both accuracy and interpretability. They differ in what the graph represents (sensor networks vs. observation-target bipartite graphs vs. patch-level adaptive graphs) and how the graph is constructed (learned vs. task-derived).

The foundation model wave has reached time series, promising zero-shot generalization across domains. But irregular sampling poses a unique challenge: most foundation models assume regular grids.

MOMENT (Goswami et al., ICML 2024) is a family of open-source, pre-trained Transformer models for general-purpose time series analysis. Pre-trained on the 'Time Series Pile' — a large, diverse collection of public time series data — MOMENT handles five tasks: short/long-horizon forecasting, classification, anomaly detection, and imputation. It addresses time series-specific challenges (varying lengths, multi-channel inputs, distribution shifts) through systematic design choices. However, it was primarily designed for regularly sampled data and requires adaptation for irregular inputs.

Chronos (Ansari et al., Amazon, TMLR 2024) made the provocative choice to treat time series as language. It tokenizes real-valued observations into a fixed discrete vocabulary via scaling and quantization, then trains T5-family transformers (20M–710M parameters) with standard cross-entropy loss. Training data is augmented with synthetic time series from Gaussian processes. The result: strong zero-shot probabilistic forecasting on 42 benchmarks. Like MOMENT, Chronos assumes regular sampling but its language-model approach means irregular data can potentially be handled through special tokens or positional encodings.

MIRA (Li et al., Microsoft, 2025) is the first foundation model built specifically for irregular medical time series. Three architectural innovations address this: Continuous-Time Rotary Positional Encoding (CT-RoPE) handles variable time intervals (extending the standard RoPE used in LLMs to continuous time); a frequency-specific mixture-of-experts layer routes computation across latent frequency regimes; and a Continuous Dynamics Extrapolation Block based on Neural ODEs enables forecasting at arbitrary timestamps. Pre-trained on 454+ billion medical time points, MIRA achieves 7–10% error reductions over baselines.

The progression from MOMENT/Chronos to MIRA mirrors the broader field's evolution: general-purpose architectures work well for regular data, but irregular time series ultimately demand the continuous-time machinery that the specialized model community has been developing since 2019. MIRA's CT-RoPE and Neural ODE blocks are direct descendants of the ideas in Latent ODE and mTAN.

Every new model must answer: does it beat these simple approaches, and by how much?

GRU-D (Che et al., Scientific Reports 2018) extends the standard GRU with trainable decay mechanisms for both input values and hidden states. When a variable hasn't been observed for a while, its last value decays toward the empirical mean at a learned rate, and the hidden state similarly decays. Binary masking indicators and time-since-last-observation intervals are fed directly into the recurrence. Despite its simplicity, GRU-D remains one of the most competitive baselines for clinical time series — many architecturally complex models struggle to beat it convincingly on standard benchmarks like PhysioNet and MIMIC.

TCN (Bai et al., 2018) established that temporal convolutions could rival or beat recurrent networks for sequence modeling. Using causal dilated convolutions with residual connections, TCNs achieve exponentially large receptive fields while being fully parallelizable (unlike RNNs). However, TCN does not natively handle missing data — it assumes a complete, regularly sampled input. When used as a baseline for irregular time series, observations must first be placed on a regular grid (e.g., via forward-fill or mean imputation).

Mean-impute + Transformer is the simplest possible combination: fill missing values with per-feature training means, then apply a standard Transformer encoder. It decouples imputation from modeling and tests a key question: can a sufficiently powerful sequence model compensate for naive preprocessing? This baseline isolates the contribution of sophisticated missing-data handling — if a complex model can't beat mean-impute + Transformer, its architectural innovations aren't earning their keep.

These baselines serve different purposes. GRU-D is the strong specialized baseline that directly models missingness. TCN is the architecture baseline testing whether recurrence or attention is needed at all. Mean-impute + Transformer is the ablation baseline isolating the value of sophisticated imputation. Together, they establish the lower bound that any new model must clear.