publications

2026

1. ICML

   StretchTime: Adaptive Time Series Forecasting via Symplectic Attention

   Yubin Kim, Viresh Pati, Jevon Twitty, Vinh Pham, Shihao Yang, and Jiecheng Lu

   In International Conference on Machine Learning (ICML), 2026

   Abs arXiv Bib PDF

   Transformer architectures have established strong baselines in time series forecasting, yet they typically rely on positional encodings that assume uniform, index-based temporal progression. However, real-world systems, from shifting financial cycles to elastic biological rhythms, frequently exhibit "time-warped" dynamics where the effective flow of time decouples from the sampling index. In this work, we first formalize this misalignment and prove that rotary position embedding (RoPE) is mathematically incapable of representing non-affine temporal warping. To address this, we propose Symplectic Positional Embeddings (SyPE), a learnable encoding framework derived from Hamiltonian mechanics. SyPE strictly generalizes RoPE by extending the rotation group \mathrmSO(2) to the symplectic group \mathrmSp(2,\mathbbR), modulated by a novel input-dependent adaptive warp module. By allowing the attention mechanism to adaptively dilate or contract temporal coordinates end-to-end, our approach captures locally varying periodicities without requiring pre-defined warping functions. We implement this mechanism in StretchTime, a multivariate forecasting architecture that achieves state-of-the-art performance on standard benchmarks, demonstrating superior robustness on datasets exhibiting non-stationary temporal dynamics.

   @inproceedings{kim2026stretchtime,
     title = {StretchTime: Adaptive Time Series Forecasting via Symplectic Attention},
     author = {Kim, Yubin and Pati, Viresh and Twitty, Jevon and Pham, Vinh and Yang, Shihao and Lu, Jiecheng},
     booktitle = {International Conference on Machine Learning (ICML)},
     year = {2026},
   }

2. arXiv

   Beyond Similarity: Temporal Operator Attention for Time Series Analysis

   Jevon Twitty, Vinh Pham, Nitiwith Rotchanarak, Viresh Pati, Yubin Kim, Shihao Yang, and Jiecheng Lu

   arXiv preprint arXiv:2605.11287, 2026

   Abs arXiv Bib PDF

   A persistent paradox in time-series forecasting is that structurally simple MLP and linear models often outperform high-capacity Transformers. We argue that this gap arises from a mismatch in the sequence-modeling primitive: while many time-series dynamics are governed by global temporal operators (e.g., filtering and harmonic structure), standard attention forms each output as a convex combination of inputs. This restricts its ability to represent signed and oscillatory transformations that are fundamental to temporal signal processing. We formalize this limitation as a simplex-constrained mixing bottleneck in softmax attention, which becomes especially restrictive for operator-driven time-series tasks. To address this, we propose Temporal Operator Attention (TOA), a framework that augments attention with explicit, learnable sequence-space operators, enabling direct signed mixing across time while preserving input-dependent adaptivity. To make dense N \times N operators practical, we introduce Stochastic Operator Regularization, a high-variance dropout mechanism that stabilizes training and prevents trivial memorization. Across forecasting, anomaly detection, and classification benchmarks, TOA consistently improves performance when integrated into standard backbones such as PatchTST and iTransformer, with particularly strong gains in reconstruction-heavy tasks. These results suggest that explicit operator learning is a key ingredient for effective time-series modeling.

   @article{twitty2026toa,
     title = {Beyond Similarity: Temporal Operator Attention for Time Series Analysis},
     author = {Twitty, Jevon and Pham, Vinh and Rotchanarak, Nitiwith and Pati, Viresh and Kim, Yubin and Yang, Shihao and Lu, Jiecheng},
     journal = {arXiv preprint arXiv:2605.11287},
     year = {2026},
   }

3. arXiv

   CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting

   Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, and Jiecheng Lu

   arXiv preprint arXiv:2602.02729, 2026

   Abs arXiv Bib PDF Code

   This paper presents CAPS (Clock-weighted Aggregation with Prefix-products and Softmax), a structured attention mechanism for time series forecasting that decouples three distinct temporal structures: global trends, local shocks, and seasonal patterns. Standard softmax attention entangles these through global normalization, while recent recurrent models sacrifice long-term, order-independent selection for order-dependent causal structure. CAPS combines SO(2) rotations for phase alignment with three additive gating paths – Riemann softmax, prefix-product gates, and a Clock baseline – within a single attention layer. We introduce the Clock mechanism, a learned temporal weighting that modulates these paths through a shared notion of temporal importance. Experiments on long- and short-term forecasting benchmarks surpass vanilla softmax and linear attention mechanisms and demonstrate competitive performance against seven strong baselines with linear complexity.

   @article{pati2026caps,
     title = {CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting},
     author = {Pati, Viresh and Kim, Yubin and Pham, Vinh and Twitty, Jevon and Yang, Shihao and Lu, Jiecheng},
     journal = {arXiv preprint arXiv:2602.02729},
     year = {2026},
   }

2025

1. NeurIPS

   ZeroS: Zero-Sum Linear Attention for Efficient Transformers

   Jiecheng Lu, Xu Han, Yan Sun, Viresh Pati, Yubin Kim, Siddhartha Somani, and Shihao Yang

   In Advances in Neural Information Processing Systems (NeurIPS), 2025

   Spotlight Abs arXiv Bib PDF

   Selected as a Spotlight paper at NeurIPS 2025.

   Linear attention methods offer Transformers O(N) complexity but typically underperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only permits additive information blending, and uniform accumulated weight bias that dilutes attention in long contexts. We propose Zero-Sum Linear Attention (ZeroS), which addresses these limitations by removing the constant zero-order term 1/t and reweighting the remaining zero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single attention layer to perform contrastive operations. While maintaining O(N) complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, it matches or exceeds standard softmax attention across various sequence modeling benchmarks.

   @inproceedings{lu2025zeros,
     title = {ZeroS: Zero-Sum Linear Attention for Efficient Transformers},
     author = {Lu, Jiecheng and Han, Xu and Sun, Yan and Pati, Viresh and Kim, Yubin and Somani, Siddhartha and Yang, Shihao},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     year = {2025},
   }