The Problem With Single-View Time-Series Language Models

Time Series Language Models (TSLMs) promise to bring the reasoning power of large language models to temporal data. They can answer questions, forecast future behavior, and infer causal relationships from sequences of numbers. But a fundamental tension sits at the center of every TSLM design: what should the representation preserve, and what should it discard?

Existing TSLMs commit to a single representation. Some serialize raw numbers into text. Others learn dedicated embeddings or project features from temporal encoders. A third approach, popularized by recent vision-language models, renders each series as a line plot and feeds the image to a visual encoder. Each choice locks in a fixed inductive bias. A line plot keeps absolute amplitudes and axis scales intact, which helps when the task asks for exact values or comparisons across series. But it also preserves nuisance variation: a sensor recalibration that multiplies every reading by ten changes the plot dramatically while the underlying dynamics stay the same. Normalized representations factor out that variation, but they throw away the measurement frame that quantitative questions depend on.

The paper "Twice Upon a Time: Complementary Representations for Time-Series Reasoning" argues that this trade-off is false. Instead of picking one representation and accepting its blind spots, the authors propose giving the model two complementary views of the same signal. One view preserves the original coordinate frame. The other strips away affine amplitude changes to expose temporal structure. Together they cover what either misses alone.

Prior Work and the Representation Gap

Time-series representation is a long-standing research area. Classical methods transform sequences into frequency spectra, recurrence plots, or delay embeddings that reconstruct phase-space trajectories. Takens' embedding theorem showed that overlapping time-lagged windows can recover the geometry of the underlying dynamical system. More recently, Wang and Oates introduced Gramian Angular Fields and Markov Transition Fields to encode sequences as images. Naiman et al. demonstrated that delay embeddings and short-time Fourier transforms work as interfaces for diffusion-based time-series generation.

On the language-model side, early efforts focused on forecasting. PromptCast and LLMTime serialized numbers as text. Time-LLM and related work reprogrammed patches of time-series data to align with pretrained LLM representations. These approaches targeted numerical prediction, not general question answering.

The shift toward Time Series Language Models broadened the scope. ChatTS aligns learned time-series embeddings with an LLM through instruction tuning. OpenTSLM explores soft-prompt and cross-attention interfaces for multivariate signals. TimeOmni-1 represents sequences directly in the token space. TS-Reasoner aligns a pretrained time-series foundation model with an LLM. In parallel, benchmarks showed that general-purpose vision-language models can reason effectively over time series rendered as line plots.

Despite this diversity, every prior TSLM exposes the language model to a single primary representation. Table 1 in the paper makes this explicit: each method lists one representation type, one normalization strategy, and one way (or none) to provide scale information. The inductive bias is fixed before the model sees any downstream task. The gap is clear: no existing TSLM lets the model access both the measurement-preserving view and the structure-exposing view simultaneously.

Two Visual Views With Opposing Invariances

The core contribution is a dual-view architecture built on Qwen3-VL-8B. Each time series generates two visual representations that enter the model through separate encoding pipelines.

Line-Plot View: Preserving the Measurement Frame

The first view is a standard line plot rendered without amplitude normalization. Raw observations become a 2D image with the original value range on the vertical axis and axis labels visible. For multivariate inputs, each variate gets its own plot rather than being overlaid, so series with different scales keep their own measurement frames. This view retains absolute amplitudes, offsets, and the coordinate system exactly as observed. It is the representation a human analyst would sketch to read off values or compare magnitudes.

The Qwen3-VL native vision encoder (initialized from SigLIP-2) processes these plots. It merges spatial features over 2×2 patch groups and projects them to the language-model hidden dimension, producing 230 visual tokens per series. The encoder stays frozen throughout training.

Normalized Delay-Embedding View: Exposing Temporal Geometry

The second view constructs a delay embedding. For a series of length m, the method extracts W overlapping windows of length ℓ, where ℓ is a fixed fraction ρ of m bounded by minimum and maximum window lengths. The c-th window starts at index p_c = round(c × (m - ℓ) / (W - 1)). Each window is linearly interpolated to H values. The W resampled windows form columns of an H×W image. Crucially, this image is min-max normalized to [0,1].

The normalization means any positive affine transformation a·x + b (with a > 0) yields the identical delay map. Amplitude rescaling and offset shifts disappear. What remains is the geometry of temporal trajectories: periodic patterns appear as parallel diagonal stripes, trends as smooth gradients, local events as isolated streaks. This is the structure that Takens' theorem guarantees reflects the underlying dynamics.

A dedicated DINOv3 ViT-L/16 encoder processes these delay maps. Its patch features are similarly merged and projected through a two-layer module to produce 64 delay tokens per variate. To match the multi-level fusion interface, intermediate features after blocks 6, 12, and 18 are also extracted, merged, and projected by dedicated modules, yielding three additional feature maps aligned to the 64 tokens.

Dual-View Fusion: Tokens and Deep Feature Injection

Both token streams enter the language model as separate visual spans, each wrapped by vision_start and vision_end tokens. This is input-level fusion: the two views coexist in the same sequence and interact through the shared self-attention layers.

Beyond concatenation, the architecture adopts DeepStack to inject multi-level visual features into the first three language-model layers. For each stream, its three intermediate feature maps are added to the hidden states at the positions of that stream's own tokens, one feature map per layer. The plot and delay streams inject into the same layers but at disjoint token positions. This gives the language model access to visual features at multiple abstraction levels throughout early processing, not just at the input boundary.

Two-Stage Training: Alignment Then Reasoning

The model trains in two stages, following the paradigm established for multimodal LLMs.

Stage I aligns the new delay-embedding stream. The Qwen vision encoder and the language model remain frozen. LoRA adapters on DINOv3 and the projection modules (the final-token projector and the three multi-level projectors) train on captioning data. The corpus contains 55,696 examples from three sources: CaTS-Bench (real-world series with reference descriptions), 32,000 ChatTS-style synthetic captions sampled from the ChatTS generator's attribute space, and 7,701 structural captions generated from a TimeSeriesExam fork with deterministic descriptions of generative components. Sources are sampled with equal probability per epoch.

Stage II performs instruction tuning. The DINOv3 adapters are merged, the entire visual pathway freezes, and LoRA is applied to the language model. The training corpus spans 182,815 examples across complementary task families: ChatTS instruction-following and supervised fine-tuning sets (51,175 total), 20,640 TSExam perception multiple-choice questions, 11,000 numerical questions with exact computed answers, and a 100,000-example reasoning corpus covering numerical, deductive, inductive, abductive, causal, etiological, temporal-relation, forecasting, and event-prediction tasks. The reasoning corpus uses procedural rules, dynamical systems, and domain-specific simulators. Supervision is gold-by-construction: labels come from known generative parameters or exact computations, never from a teacher model. No benchmark evaluation data appears in training.

Experiments: Benchmarks, Baselines, and Ablations

The evaluation covers three benchmarks probing different capability levels.

TSRBench: The Primary Benchmark

TSRBench contains 4,125 multiple-choice questions across 15 tasks in four categories: Perception (pattern recognition, noise understanding, anomaly detection, causality), Reasoning (explanatory, counterfactual, abductive, temporal-relation, deductive, inductive), Prediction (time-series forecasting, event prediction), and Decision-Making (qualitative and quantitative decision making).

Baselines include general-purpose vision-language models (Qwen3-VL-8B, GPT-5 with text, vision, or both) and dedicated TSLMs (ChatTS-14B, TS-Reasoner-7B, TimeOmni-1-7B). The dual-view model achieves 53.2% overall accuracy. The plot-only variant under the same training reaches 45.2%. The delay-only variant reaches 40.5%. The strongest dedicated TSLM baseline, TimeOmni-1-7B, scores 36.7%. GPT-5 with text and vision scores 55.6%; with vision only, 52.4%.

The 8-percentage-point gain from adding the delay view to the plot view is concentrated beyond basic perception. Plot-only already scores 87.4% on Perception; dual-view scores 86.4%. But Reasoning rises from 35.8% to 40.1%, Prediction from 38.0% to 61.7%, and Decision-Making from 36.4% to 37.6%. The gain is not uniform across tasks, confirming that the complementary view helps selectively where temporal structure matters.

Two same-view controls test whether a second stream alone explains the improvement. Plot+Plot (both towers get line plots) scores 44.8%. Delay+Delay (both towers get delay embeddings) scores 42.6%. Neither reproduces the 53.2% of the complementary pair, despite matching architecture, token budget, and training procedure.

TSExam and ChatTS: Cross-Benchmark Generalization

TSExam contains 746 multiple-choice questions on pattern recognition, noise, anomalies, similarity, and causality. The dual-view model achieves 89.9% accuracy, leading all evaluated baselines.

ChatTS evaluates categorical and numerical alignment on univariate (Dataset A) and multivariate (Dataset B) series, plus reasoning. On Dataset A numerical alignment, dual-view reaches 80.5% versus 78.1% for plot-only. On Dataset B (multivariate), it reaches 86.8% versus 86.4%. Categorical alignment stays strong. The multivariate numerical gain is notable: the delay view helps most where multiple series interact and absolute scales differ.

Evaluation-Time Interventions: Proving Distinct Invariance Profiles

To verify that the two views induce different invariances, the authors apply controlled transformations at evaluation time on a 2,020-item subset of TSRBench where gold targets are invariant to the transformations. Affine changes (×0.1, ×10, +5σ, -5σ, z-score) leave delay-only accuracy nearly unchanged (clean 44.7%, shifts ≤0.8 points). Plot-only and dual-view remain sensitive, as expected since the plot stream encodes the measurement frame. Under noise and smoothing, all models degrade. Under sampling horizon changes (256 vs 2048 points), dual-view accuracy is virtually unchanged from clean performance.

This confirms the design intent: the views have distinct sensitivity profiles. The combined model retains access to absolute measurements through the plot stream while gaining the delay stream's invariance to affine amplitude changes. It does not become uniformly invariant; it becomes selectively informed.

Limitations and Trade-Offs

The authors acknowledge several limitations. The dual-view architecture increases inference cost: 294 visual tokens per series (230 plot + 64 delay) versus 230 for plot-only. The same-view controls show this token budget alone does not explain the gains, but compute scales with token count.

The choice of delay embedding as the secondary representation is not universally optimal. Appendix experiments replacing delay maps with short-time Fourier transform (STFT) representations show that STFT slightly outperforms delay on ChatTS numerical alignment and reasoning average, while delay substantially outperforms STFT on TSRBench. The preferred secondary representation depends on the target capability distribution. Future work could explore query-dependent view selection or more than two views.

The intervention analysis reveals that the combined model is not invariant to affine transformations. This is by design—the plot stream must preserve the measurement frame—but it means the model can still be misled by sensor recalibrations unless it learns to rely on the delay stream for structure-sensitive tasks. The paper does not demonstrate explicit gating or routing mechanisms; the language model must learn implicitly when to attend to which stream.

Training uses synthetic data heavily. While gold-by-construction supervision avoids teacher-model bias, the reasoning corpus aligns its task mixture and formats with TSRBench's evaluation interface. This raises questions about how well the approach generalizes to truly out-of-distribution reasoning tasks not reflected in the benchmark suite.

What This Means for Practitioners

For developers building time-series reasoning systems, the paper offers a concrete architectural pattern: render each series twice, once as a line plot and once as a normalized delay embedding, feed both to dedicated visual encoders, and fuse the tokens into a shared language model.

The implementation is accessible. The base model is Qwen3-VL-8B, an open-weight vision-language model. The delay-embedding encoder uses DINOv3 ViT-L/16, also open. The projection modules are small (two-layer MLPs). LoRA adapters keep fine-tuning efficient. The authors release code, model weights, and training data on Hugging Face and GitHub.

Key practical takeaways:

  • If your tasks require reading absolute values (threshold alerts, sensor calibration, cross-series magnitude comparison), keep the line-plot view. Do not normalize it away.
  • If your tasks involve pattern recognition, periodicity detection, anomaly shape matching, or forecasting dynamics, add the normalized delay view. It is invariant to gain and offset changes that commonly occur across deployments.
  • The multi-level DeepStack fusion matters. Injecting intermediate visual features into early LLM layers outperforms input-only concatenation because it lets language processing condition on visual abstractions progressively.
  • Training can proceed in two stages: align the new visual stream first with caption data, then instruction-tune the LLM with both streams active. This isolates the representation-learning problem from the reasoning-learning problem.
  • Expect the largest gains on reasoning, prediction, and decision-making tasks. Basic perception (reading a peak value, spotting a visible anomaly) already works well with line plots alone.

The approach also suggests a broader design principle for multimodal time-series models: instead of engineering a single "perfect" representation, compose representations with known, complementary invariances. A third view (spectral, recurrence, symbolic) could add further inductive biases for specific domains.

Closing Perspective

The paper reframes a design tension as a design opportunity. Time-series reasoning needs both the measurement frame and the temporal geometry. Forcing one representation to serve both roles creates a compromise. Giving the model both views, each processed by a specialized encoder, lets the language model allocate attention dynamically. The results on TSRBench, TSExam, and ChatTS show this works: a Qwen3-VL-8B backbone with complementary views outperforms larger dedicated TSLMs and narrows the gap to frontier proprietary models.

The deeper implication is that representation design for time-series language models should be treated as an explicit architectural choice with known invariance properties, not an afterthought. The dual-view principle—preserve the coordinate frame in one stream, factor it out in another—extends beyond the specific delay-embedding instantiation. Any pair of representations with provably complementary invariances can plug into the same architecture. That generality makes this work a reference point for the next generation of TSLMs.

Read the paper on arXiv