Sliding Window Technique

The Sliding Window Approach is a technique used to generate multiple Input-Target Pairs from a text dataset. A window of a fixed size (determined by the Context Size) slides over the text to create overlapping sequences of data.

Mechanism

1. Window: Selects a chunk of text of length max_length (or context size).
2. Slide: The window moves forward by a certain number of steps, defined by the Stride.
3. Input/Output: For each window position, the input x is the chunk, and the target y is the chunk shifted by one token.

Ref: sliding-window-algorithm by Logicmojo

Purpose

This approach allows the model to learn from every part of the dataset, effectively increasing the number of training examples available from a single text source.