大家好，欢迎来到IT知识分享网。

Input Representation

MOMENT begins by taking in a univariate time-series \( T \) of shape \( R^{1 \times T} \), where \( T \) is the length of the time-series. Alongside, it uses a binary mask \( M \) of the same length, where elements marked as 0 represent unobserved timestamps, and those marked as 1 represent observed timestamps. This kind of masking is useful for dealing with missing data or for purposely hiding data during training as a form of regularization.

Preprocessing with Reversible Instance Normalization

Before the model processes the time-series, it applies a technique called “reversible instance normalization,” which was proposed by Kim et al. [2022]. This normalization is applied only to the observed parts of the time-series, helping to standardize the data while allowing for the original values to be restored, which is crucial for effective learning and generation.

Patching and Encoding

The time-series is then segmented into \( N \) disjoint patches, each of length \( P \). These patches are processed to convert them into embeddings:

• If all timestamps in a patch are observed, a trainable linear projection is used to map the patch into a \( D \)-dimensional embedding.
• If any timestamps in a patch are unobserved, a special pre-trained mask embedding denoted as \([MASK] \in R^{1 \times D}\) is used instead.

Transformer Model

The core of the MOMENT model is a transformer encoder, which processes the sequence of patch embeddings. The transformer maintains the \( D \)-dimensional shape of these embeddings throughout its operations. This part of the model includes modifications from Raffel et al. [2020], such as:

• Removal of additive bias from layer normalization (Ba et al. [2016]), placing it before the residual connections (He et al. [2016]).
• Use of a relative positional embedding scheme (Shaw et al. [2018]) to account for the positions of patches within the time-series.

Reconstruction Head

The output of the transformer encoder for each patch embedding is processed through a lightweight prediction head called the “reconstruction head.” The purpose of this head is to map the transformed patch embeddings back into the original or desired output dimensions, effectively reconstructing both observed and unobserved parts of the time-series patches.

Pre-training Strategy

During pre-training, patches are masked uniformly at random, and their embeddings are replaced with the \([MASK]\) embedding. The goal of this phase is to train the model to accurately predict the original data of masked patches, thereby learning robust patch embeddings.

Key Design Decisions

The architecture of MOMENT is heavily influenced by the need to handle patches of data efficiently, cope with missing data, and leverage the powerful capabilities of transformers for sequence modeling. The use of reversible instance normalization and the strategic placement of layer normalization and positional embeddings enhance the model’s ability to handle time-series data more effectively.

Overall, the MOMENT model represents a complex yet efficient approach to time-series analysis, balancing the need for accurate data reconstruction with the robustness required to handle real-world data irregularities and missing information.

The MOMENT model addresses several challenges associated with processing time-series data, particularly those related to varying characteristics such as length, number of channels, amplitude, and temporal resolution. Here’s an explanation of how MOMENT handles these aspects and its structural adaptations:

Handling Variable Length

Time-series data can vary significantly in length. To standardize input, MOMENT restricts its time-series input to a fixed length of \( T = 512 \). For time-series longer than this length, sub-sampling is used to reduce the data to the fixed length. For shorter sequences, zero-padding on the left is applied to reach the required length. This approach ensures consistency in data input size, which is crucial for maintaining the model’s architecture and performance.

Reducing Memory Footprint and Computational Complexity

Segmenting time-series into patches is a strategic choice in MOMENT’s design. By breaking down a time-series into smaller parts, MOMENT significantly reduces its memory usage and computational demands. This segmentation not only makes the model more efficient but also enables it to handle longer time-series linearly by adjusting the number of patches processed.

Handling Multi-variate Time-series

MOMENT can process multi-variate time-series by operating on each variable or channel independently along the batch dimension. This method aligns with findings from recent studies (Zhou et al. [2023], Nie et al. [2023]) which suggest that modeling each channel independently is an effective strategy for handling complex multivariate time-series data.

Modeling Time-series with Different Temporal Distributions

The use of reversible instance normalization allows MOMENT to effectively handle time-series with varying amplitudes and temporal distributions. This normalization adjusts the scale and centers the data, enabling the model to handle inputs with significantly different characteristics without needing to know the exact temporal resolution, which is often absent outside specific forecasting datasets.

Simplified Encoder Design

MOMENT employs a simple encoder design that closely follows the principles of transformers used in the language processing domain. This simplicity allows MOMENT to leverage scalable and efficient transformer implementations, such as gradient checkpointing and mixed precision training. The encoder maintains a small footprint while capturing high-level features necessary for time-series analysis.

Lightweight Prediction Head

Instead of using a large decoder, MOMENT utilizes a lightweight prediction head. This component is critical for enabling task-specific fine-tuning with a limited number of trainable parameters. By keeping this part of the model smaller and more focused, MOMENT ensures that the bulk of the parameters and the critical features learned by the encoder are preserved during fine-tuning phases.

Overall, these design decisions allow MOMENT to be a robust and adaptable model for handling diverse time-series data, balancing efficiency and complexity to meet the challenges posed by real-world applications.

训练：

The training method for the MOMENT model uses a pre-training strategy centered around Masked Time-series Modeling. This approach is particularly designed to prepare the model for handling incomplete or partially masked time-series data. Here’s a detailed breakdown of the training process:

1.Masked Time-series Modeling (Pre-training)

MOMENT utilizes a pre-training technique where a subset of the time-series patches is masked randomly by replacing their patch embeddings with a learnable mask embedding, denoted as [MASK]. This step simulates missing data scenarios and helps the model learn to infer missing information effectively. The corrupted time-series patches are processed through the transformer encoder, which aims to learn useful patch representations despite some data being masked. These learned representations are then used by a lightweight reconstruction head to reconstruct the original time-series. The pre-training objective focuses on minimizing the masked reconstruction error, specifically the Mean Squared Error (MSE) between the actual data and the predictions, averaged over the masked patches.

2.Pre-training Setup

The MOMENT model is pre-trained in three different sizes to accommodate various computational and application needs, similar to the tiered approach seen in models like T5 (Small, Base, and Large):

• Base Model: Utilizes a transformer with 12 layers, 12 attention heads, and a feed-forward network size of 3072, totaling approximately 125 million parameters.
• Small Model: Has 6 layers, 8 attention heads, and a feed-forward network size of 2048, with around 40 million parameters.
• Large Model: Features 24 layers, 16 attention heads, and a feed-forward network size of 4096, amounting to roughly 385 million parameters.

Each variant handles an input time-series of length \(T = 512\), segmented into \(N = 64\) disjoint patches each of length \(P = 8\). During pre-training, 30% of the patches are masked randomly.

3.Optimization and Training Regimen

The training employs the Adam optimizer with weight decay, using specific parameters such as \(\lambda = 0.05\), \(\beta_1 = 0.9\), and \(\beta_2 = 0.999\). Gradient clipping is set at 5.0 to prevent the exploding gradient problem, which is critical in training deep neural networks. MOMENT models are trained with large batch sizes (2048), utilizing a cosine learning rate schedule starting from \(1e-4\) and tapering to \(1e-5\). This approach helps in smoothing out the training process and adapting the learning rate over time for better convergence.

4.Efficiency and Precision Adjustments

To enhance training throughput and reduce memory consumption, gradient checkpointing is used. This technique involves storing certain intermediate outputs and recomputing others during the backward pass, which balances memory usage against computational overhead. Additionally, mixed precision training is employed, where most of the model uses bfloat-16 precision, except for numerically unstable operations like layer normalization, which use float-32 to maintain numerical stability.

5.Duration of Training

All variants of the MOMENT model are trained for a relatively short span of 2 epochs. This limited training period is likely sufficient due to the effectiveness of the pre-training strategy and the large amount of data processed in each epoch due to the large batch sizes.

Overall, this training methodology for the MOMENT model is designed to efficiently handle the complexities of time-series data by learning robust representations through a strategy that mimics real-world scenarios of incomplete data, preparing the model for diverse applications in time-series analysis.

Fine-Tune:

The fine-tuning method for the MOMENT model is tailored to optimize its performance across a variety of downstream time-series analysis tasks. This flexibility is crucial for adapting the model to specific applications beyond the general capabilities trained during pre-training. Here’s how the MOMENT model is fine-tuned for different tasks:

1.Overview of Downstream Tasks

MOMENT is designed to handle multiple practical time-series analysis tasks. These include:

• Long- and Short-Horizon Forecasting: These tasks require predicting future values of a time-series over long or short time intervals.
• Classification: Involves categorizing time-series into predefined classes based on their temporal patterns.
• Anomaly Detection: Identifying unusual patterns that do not conform to expected behavior.
• Imputation: Filling in missing values or correcting erroneous data within a time-series.

Each task demands specific modifications to the model’s output mechanism to handle the particularities of the task effectively.

2.Task-Specific Model Adaptations

For **forecasting tasks**, the standard reconstruction head used during pre-training is replaced with a **forecasting head**. This new head works by first flattening all the \( N \) \( D \)-dimensional patch embeddings into a single \( N \times D \) dimensional vector. This vector is then projected into a \( H \)-dimensional output, where \( H \) is the forecast horizon, using a linear projection layer. This adaptation specifically supports the generation of future time-series values.

For **classification, anomaly detection, and imputation tasks**, the original reconstruction head is retained. This head is adept at reconstructing or generating time-series data based on learned embeddings, which is essential for tasks that involve understanding the full context or structure of the data, such as detecting anomalies or filling missing values.

3.Fine-tuning Settings

MOMENT offers two primary fine-tuning approaches:

• End-to-End Fine-Tuning: Here, all parameters of the model are trainable, which allows the model to adjust all of its weights based on the specifics of the downstream task. This approach is likely more effective when significant adaptation of the model to new patterns or features is necessary.
• Linear Probing (MOMENTLP): In this approach, all parameters except for those in the task-specific head (reconstruction or forecasting) are frozen. Only the head is trained, which restricts learning to the final stage of the model. This method is generally faster and requires less computational resources, suitable for scenarios where the pre-trained model already performs well, and only minor adjustments are needed.

Additionally, for tasks such as anomaly detection and imputation, a zero-shot setting (MOMENT0) may be employed. In zero-shot learning, the model uses the learned capabilities from pre-training directly without any additional fine-tuning, leveraging the generalizability of the learned patch embeddings and the reconstruction head.

4.Application and Configurations

Detailed descriptions of each task’s specific configuration and how MOMENT is adapted are provided in an appendix in the model documentation. This typically includes specifics on how inputs are handled, any additional preprocessing needed for different tasks, and the exact architecture changes for the forecasting or reconstruction heads.

By employing these fine-tuning strategies, MOMENT can be effectively adapted to a wide range of time-series analysis tasks, making it a versatile tool in both academic research and practical applications. This flexibility allows MOMENT to not only leverage its strong pre-trained foundations but also tailor its capabilities to meet the specific needs of diverse time-series analysis challenges.