What Is Common Crawl? Building and Preprocessing a Large-Scale Japanese Web Corpus

When planning for AI search optimization (LLMO, AIO, AISEO, GEO), one question often comes up: “Where is AI learning information about our company?” One known source is Common Crawl.

Common Crawl

Common Crawl is a U.S.-based nonprofit project that regularly and automatically collects web pages and provides them as a large public dataset that anyone can use. The collected data is also used to train generative AI.

This article explains how Common Crawl collection works, the difference between WARC and WET, the preprocessing flow for creating a Japanese corpus, and key points to watch when using the data. It also explains technical terms so you can understand what to ask your development team.

*This article is operated by Queue Inc. Updated: October 13, 2026

What Common Crawl collects and why it makes the data public

Common Crawl is a project that collects web pages and publishes archives of their state at the time of collection. Its goal is to make large-scale web data available to researchers and companies.

Opening access to web data for universities and small and midsize companies

Crawling the entire web on your own is expensive. With public data, universities and small and midsize companies can also begin research using large-scale material.

Common Crawl was founded in 2007 and has become established as foundational data for research and generative AI development.

It has been reported that Common Crawl was also used in AI development by companies such as OpenAI as a starting point for generative AI pre-training data. Wikipedia also describes its connection to the research competition “Norvig Web Data Science Award.”

Sources: Common Crawl official About page, Wikipedia, “Common Crawl”.

CCBot checks robots.txt and collects web pages

A crawler is a program that automatically visits web pages. Common Crawl’s crawler is called “CCBot,” and it collects pages in accordance with each site’s robots.txt.

robots.txt

robots.txt is a file that tells crawlers whether they are allowed to crawl a site. Collection results are grouped into “crawl archives” by time period. A snapshot is a record that preserves the state of something at a particular point in time.

According to the official FAQ, Common Crawl collects part of the web and usually does not preserve an entire site. CCBot also does not execute JavaScript. Keep in mind that the body text visible in a browser may not match the body text that remains in the collected HTML.

To check records for your own site, you can use the AI Pre-training Data Checker. For how to use it and interpret the results, see How to check Common Crawl preservation status.

Source: Common Crawl official FAQ. Whether a page is preserved does not prove that it was used for training in a specific model or cited in AI search.

What criteria should you use to evaluate Common Crawl data?

Before using it, check four things: coverage, freshness, quality, and usage terms. Do not judge by data volume alone. Check whether the information includes the languages and time periods that match your purpose.

 

Even with public data, check usage terms and copyright

The most commonly overlooked point is usage terms. The data itself can be downloaded for free by anyone, but you must follow the terms of use.

Even when Common Crawl makes data public, the copyright for each page remains with the original rights holder. For commercial use, it is essential to check the terms of use and go through internal legal review before using the data. The official terms of use also require users to respect applicable laws and the rights of third parties.

In November 2025, GIGAZINE reported that Common Crawl had collected billions of web pages since 2013. The fact that it includes paid pages was also raised as an issue. Make sure to separate the question of whether data is available from whether there are any problems with using it.

Sources: Common Crawl official Terms of Use, GIGAZINE report (November 5, 2025). Sources are the official website and publicly available information. Date checked: October 13, 2026.

What is the difference between WARC and WET, and which should you choose?

As a rule of thumb, use WARC if you prioritize quality and WET if you prioritize ease of use. The difference is whether you extract the body text yourself from collected data such as HTML, or start from already extracted text.

WARC lets you customize extraction, while WET is easier to start processing

WARC (Web ARChive)

WARC (Web ARChive) is raw data that includes HTTP responses and HTML. An HTTP response is what a web server returns when a page is requested. If you build a corpus from WARC, you extract the text yourself.

This gives you room to more accurately separate the body text from menus and ads. Extraction requires more computation, but WARC is better suited to building a high-quality corpus.

WET

WET is processed data that contains only extracted body text. It is lighter to process, but text such as navigation labels can be mixed in, so noise removal is still required.

 

Source: Common Crawl official data overview.

How should you preprocess Common Crawl to build a Japanese corpus?

A Japanese corpus is typically built in this order: extraction, filtering, deduplication, and ML-based selection.

Corpus

A corpus is a large collection of text data used to train AI. ML (machine learning) refers here to the use of models that select text.

Narrow down likely Japanese pages before extracting the body text

One way to reduce processing load is to split the work into stages. At the WARC stage, you can use a simple check to identify pages that appear to be Japanese, then fully extract only those targets. ABEJA structures its extraction process around this approach.

The Japanese corpus built for continued pre-training of Mixtral 8x7B is about 400B tokens (400 billion tokens) when counted with the Mixtral Tokenizer. Most of it is said to come from Common Crawl.

A token is the unit a model uses to process text, and a tokenizer is the mechanism that splits text into those units. Character count and token count are not the same.

1. Narrow down the target WARC records using a simple Japanese-language check.

2. Extract body text from the HTML.

3. Run language detection again and exclude non-Japanese text.

The Tokyo Tech Swallow project has also published an example of building a large-scale Japanese web corpus from Common Crawl.

Remove unnecessary text and duplicates with rules and machine learning

Unnecessary text is removed in two stages: rule-based and ML-based. In ABEJA’s case, the following criteria are used.

• Determining whether text looks like natural prose based on part-of-speech ratios, such as the proportion of verbs.

• Exclusion by URL domain.

• Exclusion of pages with many ellipses.

• Deduplication.

• Selection using quality scores from a machine learning model.

To process large numbers of files in parallel, ABEJA uses an execution environment that combines Google Cloud Pub/Sub and Cloud Run Jobs. Pub/Sub is a service for passing processing notifications, and Cloud Run Jobs is a service for running processing tasks.

Sources: ABEJA Tech Blog case study on building a Japanese corpus (May 7, 2024), research paper on the Swallow corpus (April 27, 2024).

How should you check corpus quality and usage risks?

There is still no fixed answer to “what makes data good for a large language model (LLM).” In LLM development, there is a trade-off: the stronger the filters, the more diversity is lost.

Test line breaks, broken tables, and genre bias in small steps

Rather than trying to solve everything at once, it is more practical to run small tests repeatedly. ABEJA lists the following as future challenges: text with many line breaks, broken tabular data, removal of unnecessary text, genre bias, and deduplication accuracy.

One way to check genre bias is to aggregate extraction results by domain and review the distribution. Then, you can reduce bias by thinning out fields that are extremely overrepresented. However, deciding which fields to keep and how much to keep requires validation based on the intended use.

If you want to understand the relationship between text units and citations, how chunks and citations work may also be helpful. Corpus extraction and filtering should be considered separately from how text is divided for reference at answer time.

For financial and similar information, compare the collection-time data with current primary sources

YMYL (Your Money or Your Life)

YMYL (Your Money or Your Life) refers to areas that can significantly affect people’s lives or finances. In finance and similar fields, archives may contain interest rates, fees, risk disclosures, and disclaimer terms from before they were revised.

A crawl archive is a snapshot from the time it was collected. When using or publishing data, assume that you need to compare it against current primary sources and review it internally.

How can understanding Common Crawl inform your AI search optimization?

Understanding Common Crawl shows that web data used for AI training also requires extraction, selection, and quality checks. In this article, LLMO (Large Language Model Optimization) refers to AI search optimization. Teams working on this in-house need to separately check the construction of training corpora and how they currently appear in AI-generated answers.

Distinguish pre-training data from search at answer time

RAG (Retrieval-Augmented Generation)

RAG (retrieval-augmented generation) is a mechanism that searches external information and generates answers based on that content. It is different from the pre-training process, where knowledge is incorporated into a model.

Even if a page is preserved in Common Crawl, that does not prove that a specific model trained on the page or that it will be cited or recommended in AI answers. Check preservation records separately from actual AI answers.

For a more detailed explanation of the relationship, see How RAG and AI search work. For the three goals of LLMO, the five criteria for deciding whether your company needs it, and the basic process, see the parent article, Definition of LLMO and how to decide whether to adopt it. You can also read a separate article on the difference between SEO and LLMO.

umoren.ai support scope and published results

umoren.ai, provided by Queue Inc., focuses on improvements based on what AI reads on the web and which information it chooses to use. An LLM engineering team with an understanding of RAG and embedding (a technology that converts text into vectors) supports AI search visibility audits, LLMO strategy design, content/structure improvements, and continuous analysis and improvement.

Published results include more than 100 companies supported in total (as of October 13, 2026), an average AI citation improvement rate of +460%, 98% customer satisfaction, and a 4.4x improvement in AI search traffic conversions. Client companies include CyberBuzz, KINUJO, Peach Aviation, Rena Robotics, and Ecoprocoat. These are published figures and do not guarantee AI recommendations or results.

umoren.ai is suitable for companies whose company name does not appear in AI search or that find competitors are the only ones being recommended. However, building the corpus itself for pre-training an LLM in-house is outside the scope of support. For details, please see umoren.ai service information.

Frequently asked questions about Common Crawl

After checking free use, the benefits for pre-training, and how to refuse collection, choose the next task that fits your purpose.

What is the biggest benefit of using it for LLM pre-training?

You can obtain large-scale web text without crawling it yourself. The ABEJA case introduced in this article also extracts and filters that material and processes it into a Japanese corpus.

How can you block CCBot from collecting your site?

The basic method is to specify the crawler “CCBot” in robots.txt. Blocking applies to future collection. Data collected in the past may remain in already published archives.

Source: Common Crawl official FAQ. You can also check the roles and crawl settings for each AI crawler in crawl requirements for sites cited by ChatGPT. Settings for CCBot and settings for search bots from other services are separate.

How can you check how AI sees your company?

Check both your Common Crawl preservation status and how your company is described, cited, and recommended in AI answers. umoren.ai’s “AI Search Current State Analysis Report (LLM Perception Analysis)” is sent by email in Excel format within 24 hours and can be a first step toward understanding how you appear in AI search. Costs vary depending on the scope of support, so confirm them when making a request.

Organize Common Crawl use cases and choose your next task

Common Crawl is a foundation of public web data that can also become training material for generative AI. When building a Japanese corpus, choose the format, proceed with extraction and selection, and check freshness, bias, and usage terms. In finance and similar fields, comparison with current primary sources is also essential.

If you are moving forward with LLMO for your own company, look at preservation records and actual AI answers separately. For the order of steps when handling this in-house, see how to approach LLMO measures. If you want to check how your company is perceived, you can organize your current situation through the AI Search Current State Analysis Report (LLM Perception Analysis).