LLM Training Data: The 8 Main Public Data Sources

The biggest challenge in training large language models (LLMs) isn’t their architecture—it’s finding high-quality, diverse, and unbiased data in a vast and noisy digital landscape. This is true whether you’re building an LLM from scratch or fine-tuning a pre-trained one, as you’ll need to use high-quality data compiled from multiple sources.

This article overviews LLM training, the need for public web data, and the major public data sources for highly-performant LLMs.

What is training data for LLM?

Training data for LLMs consists of vast collections of text, often terabytes in size, gathered from public sources like websites, online books, research papers, and code repositories. Raw text is commonly used as the main format, which is cleaned and then tokenized into smaller units, such as subwords, using various techniques like byte pair encoding. The processed data is then fed to the model's architecture to teach it to analyze and generate human-like text by identifying patterns, context, grammar, and meaning.

How to train an LLM?

Training LLM on custom data generally includes the following steps:

1. Collect and preprocess data

• Gather large-scale textual data from diverse sources to ensure a varied data distribution.

• Clean the data to remove irrelevant and incomplete information, duplicates, and inappropriate content. Then, you can normalize the text.

• Tokenize the text into words, subwords, or characters to create a manageable and efficient token set for large language model training.

2. Choose or create a model

• You may want to use a pre-trained model like GPT, BERT, or similar, which you can fine-tune with custom data. This will save you considerable time and costs.

• If you have specific objectives where pre-trained models aren’t sufficient, you can create an LLM architecture from scratch. Some popular tools include PyTorch and TensorFlow. This step requires high-performance hardware, time, and a serious budget.

3. Train the model

• Pre-training: If you’re starting from scratch, in this phase, the model learns general language patterns. The training objective often involves predicting words or tokens in a sequence, enabling the model to understand the context and structure of the language.

• Fine-tuning: Adjust the model to improve its performance on a specific task or domain. This may involve different fine-tuning approaches with human involvement, such as supervised learning and reinforcement learning.

4. Test, evaluate, and modify hyperparameters

• Throughout the training process, evaluate the model’s performance using metrics like perplexity, BLEU score, F1 score, accuracy, task-specific benchmarks, and other.

• Optimize various hyperparameters like learning rates, batch sizes, gradient clipping, warmup steps, and others to adjust the model’s performance.

Why LLMs need public web data

While synthetic data created by generative AI models can help train and fine-tune large language models, the so-called LLM hallucinations can also bring a significant downside to the end result. This phenomenon refers to scenarios where large language models generate clear and grammatically correct but false and misleading results. There are countless reasons for LLM hallucinations, but the limits that synthetic data introduces are one of the contributing factors.

When using synthetically produced data, large language models may inherit the same mistakes and biases found in the original data, potentially amplifying inaccuracies. Therefore, utilizing high-quality public web data for LLM training and fine-tuning is crucial. This allows models to navigate “the jungle” by improving the overall accuracy and knowledge base, offering benefits like:

• Extensive knowledge: when trained on a variety of topics, gathered throughout different web sources, LLM models can in turn produce more accurate and relevant responses across various domains.

• Diverse writing styles and perspectives: the broad range of sources, data types (i.e. conversational data), and formats (text, images, etc.), enables LLM models to capture diverse linguistic patterns, nuances, and styles, and enhance contextual understanding.

• Up-to-date information: public websites are interested in constant content updates in order to rank better on search engines and attract new audiences. Consequently, training large language models on such data ensures that models are aligned with the current and most relevant information.