Skip to content

release-8.5: clarify IMPORT INTO format detection by platform and version #23490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alastori wants to merge 2 commits into
base: release-8.5
Choose a base branch
from
+19 −1
Draft

release-8.5: clarify IMPORT INTO format detection by platform and version #23490

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
22b5006
docs: document IMPORT INTO format detection
alastori Aug 11, 2026
3f69993
docs: qualify format detection by platform and version
alastori Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion sql-statements/sql-statement-import-into.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,25 @@ In the `fileLocation` parameter, you can specify a single file, or use the `*` a

### Format

The `IMPORT INTO` statement supports three data file formats: `CSV`, `SQL`, and `PARQUET`. If not specified, the default format is `CSV`.
The `IMPORT INTO` statement supports three data file formats: `CSV`, `SQL`, and `PARQUET`. When you specify the `FORMAT` clause, TiDB uses that format regardless of the file extension.

<CustomContent platform="tidb">

Starting from v8.5.7, if you omit `FORMAT`, TiDB automatically detects the format from the `.csv`, `.sql`, or `.parquet` file extension. Detection is case-insensitive. For compressed files, TiDB ignores the `.gz`, `.gzip`, `.zstd`, `.zst`, or `.snappy` compression suffix before detecting the data file format. If the remaining file name has no extension or an unrecognized extension, TiDB treats the file as `CSV`.

In v8.5.6 and earlier versions, TiDB treats the file as `CSV` when you omit `FORMAT`.
Comment on lines +135 to +141

@coderabbitai coderabbitai Bot Aug 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- target document ---'
sed -n '100,165p' sql-statements/sql-statement-import-into.md

printf '%s\n' '--- related terms and versions ---'
rg -n -i 'TiDB Cloud|server-disk|local file path|file path|FORMAT|v8\.5\.7|automatic.*detect|extension' \
  sql-statements/sql-statement-import-into.md resources 2>/dev/null | head -200

printf '%s\n' '--- tracked workflow/review context ---'
git status --short
git diff --stat

Repository: pingcap/docs

Length of output: 22212

🌐 Web query:

TiDB v8.5.7 IMPORT INTO automatic format detection local server disk path supported suffix FORMAT bypass validator TiDB Cloud

💡 Result:

In TiDB v8.5, the IMPORT INTO statement is the recommended method for high-performance data ingestion [1][2]. Below are the details regarding your specific queries: Automatic Format Detection and File Suffixes TiDB does not have a general "automatic format detection" feature that infers the format of any arbitrary file based on its contents alone; instead, it relies on file extensions or explicit configuration [1][3]. For IMPORT INTO FROM FILE, the supported formats are CSV, SQL, and PARQUET [1]. Regarding compression, IMPORT INTO can automatically identify the compression format of a file based on its extension [1]. Supported extensions include: -.gz /.gzip (gzip) -.zstd /.zst (ZStd) -.snappy (snappy) If you are using Parquet files, TiDB Lightning and similar tools generally require files to have the.parquet extension to be correctly identified [3]. If files lack this, they must be renamed [3]. Local Server Disk Path IMPORT INTO supports importing files from the local server disk of the TiDB node executing the statement [1][4]. When specifying a local path, the path must be accessible to the TiDB server process. If you are using distributed import tasks, ensure the files are accessible to the relevant TiDB nodes or use shared storage. Bypass Validator There is no "bypass validator" command in the standard IMPORT INTO syntax. IMPORT INTO enforces schema validation to ensure data integrity [2]. If your data does not match the target schema or contains conflicts (such as duplicate primary keys), the import job will typically fail [2]. For conflict handling, it is expected that the target table is empty or that the schema is defined to avoid conflicts, as IMPORT INTO does not currently support complex conflict resolution strategies (like IGNORE or REPLACE) [2]. TiDB Cloud In TiDB Cloud, the import process often uses a UI-driven approach for importing from cloud storage (e.g., S3, GCS) [5][6]. When importing from a directory in TiDB Cloud, there is an option called "Use TiDB file naming conventions for automatic mapping," which helps map source files to the correct target tables based on their names (e.g., ${db_name}.${table_name}.csv) [5][6]. This is distinct from content-based format detection. Summary of Recommendations - Explicitly define the format (FORMAT='CSV', 'SQL', or 'PARQUET') in your IMPORT INTO statement rather than relying on auto-detection [1]. - Ensure file extensions (including compression suffixes) are correct, as these are used for identification [1][3]. - For data validation issues, clean your source data to align with the target schema prior to import, as there is no bypass validator function [2].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu

tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT

curl -fsSL \
  https://raw.githubusercontent.com/pingcap/tidb/v8.5.7/pkg/executor/importer/import.go \
  -o "$tmpdir/import.go"

printf '%s\n' '--- relevant importer symbols ---'
rg -n -C 8 'extension|suffix|FORMAT|format|wildcard|glob|file name|filepath.Ext|detect' \
  "$tmpdir/import.go" | head -260

printf '%s\n' '--- exact source lines around matches ---'
python3 - "$tmpdir/import.go" <<'PY'
from pathlib import Path
p = Path(__import__("sys").argv[1])
lines = p.read_text().splitlines()
need = set()
for i, line in enumerate(lines, 1):
    low = line.lower()
    if any(x in low for x in ("filepath.ext", "suffix", "wildcard", "format", "extension")):
        need.update(range(max(1, i-6), min(len(lines), i+10)+1))
for i in sorted(need):
    print(f"{i:5}: {lines[i-1]}")
PY

printf '%s\n' '--- cloud documentation references ---'
for url in \
  https://docs.pingcap.com/tidbcloud/sql-statement-import-into/ \
  https://docs.pingcap.com/tidb/stable/sql-statement-import-into/; do
  printf '\nURL: %s\n' "$url"
  curl -fsSL "$url" | \
    sed 's/<[^>]*>/ /g' | \
    tr -s '[:space:]' ' ' | \
    grep -o -i '.\{0,180\}\(format\|extension\|local file\|cloud\).\{0,260\}' | \
    head -20 || true
done

Repository: pingcap/docs

Length of output: 49386

🌐 Web query:

site:docs.pingcap.com/tidbcloud "If you omit FORMAT" "automatically detects" IMPORT INTO

💡 Result:

In the TiDB IMPORT INTO statement, omitting the FORMAT clause does not trigger an automatic detection mechanism for the file format. Instead, the system defaults the format to CSV [1]. The statement supports CSV, SQL, and PARQUET formats, but if you do not explicitly specify one, it will proceed assuming the input is in CSV format [1].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu

tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT

curl -fsSL \
  https://raw.githubusercontent.com/pingcap/tidb/v8.5.7/pkg/executor/importer/import.go \
  -o "$tmpdir/import.go"
curl -fsSL \
  https://raw.githubusercontent.com/pingcap/tidb/v8.5.7/pkg/executor/importer/import_test.go \
  -o "$tmpdir/import_test.go"

printf '%s\n' '--- validation and wildcard control flow ---'
sed -n '1095,1265p' "$tmpdir/import.go"

printf '%s\n' '--- format detection implementation ---'
sed -n '1268,1298p' "$tmpdir/import.go"

printf '%s\n' '--- relevant upstream tests ---'
rg -n -C 12 'supportedSuffixForServerDisk|server disk|detectAndUpdateFormat|parseFileType|arbitrary|wildcard|glob|DataFormatAuto' \
  "$tmpdir/import_test.go" "$tmpdir/import.go" | head -320

printf '%s\n' '--- document metadata and repository revision ---'
sed -n '1,35p' sql-statements/sql-statement-import-into.md
git rev-parse --show-toplevel
git branch --show-current
git rev-parse HEAD

Repository: pingcap/docs

Length of output: 34518

🏁 Script executed:

#!/bin/bash
set -eu

python3 - <<'PY'
from pathlib import PurePosixPath

supported = {
    ".csv", ".sql", ".parquet",
    ".gz", ".gzip", ".zstd", ".zst", ".snappy",
}
compression = {".gz", ".gzip", ".zstd", ".zst", ".snappy"}

def go_like_ext(path):
    # filepath.Ext returns the suffix beginning at the final dot in the final path element.
    name = path.rsplit("/", 1)[-1]
    i = name.rfind(".")
    return name[i:] if i >= 0 else ""

def parse_file_type(path):
    path = path.lower()
    ext = go_like_ext(path)
    if ext in compression:
        path = path[:-len(ext)]
        ext = go_like_ext(path)
    return {"sql": ".sql", "parquet": ".parquet"}.get(ext[1:], "csv")

cases = [
    "/var/lib/import/data.csv",
    "/var/lib/import/data.SQL",
    "/var/lib/import/data.parquet.gz",
    "/var/lib/import/data.csv.zst",
    "/var/lib/import/data",
    "/var/lib/import/data.tsv",
    "/var/lib/import/data.unknown.gz",
]
for path in cases:
    suffix = go_like_ext(path).lower()
    accepted = suffix in supported
    print(f"{path}: suffix={suffix!r}, local_path={'accepted' if accepted else 'rejected'}, auto_format={parse_file_type(path)!r}")
PY

Repository: pingcap/docs

Length of output: 724

🌐 Web query:

TiDB Cloud IMPORT INTO FORMAT automatic detection v8.5.7 file extension

💡 Result:

In TiDB v8.5.7, the IMPORT INTO statement supports automatic file format detection when the FORMAT clause is omitted [1]. This feature simplifies the import process for CSV, SQL, and Parquet files [1]. How Automatic Detection Works: 1. File Extension Inspection: When the FORMAT clause is not specified, TiDB inspects the file extension to determine the format [2][3]. The supported extensions are.csv,.sql, and.parquet [2][3]. 2. Handling Compression: If the file is compressed, TiDB first removes the compression suffix (e.g.,.gz,.gzip,.zstd,.zst, or.snappy) and then inspects the remaining file extension to identify the primary format (CSV, SQL, or Parquet) [2][3]. 3. Default Behavior: If a file does not have an extension, or if the extension does not match the recognized types, TiDB defaults to the CSV format [4][2][3]. This detection logic applies to single-file imports as well as wildcard imports. If multiple file types are present in the same path, it is recommended to separate them or run distinct IMPORT INTO statements, as TiDB may infer the format from a single representative file, potentially leading to errors for mismatched files [4].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu

tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT

printf '%s\n' '--- development documentation ---'
for url in \
  https://docs.pingcap.com/tidb/dev/sql-statement-import-into/ \
  https://raw.githubusercontent.com/pingcap/docs/master/sql-statements/sql-statement-import-into.md; do
  printf '\nURL: %s\n' "$url"
  curl -fsSL "$url" -o "$tmpdir/page"
  python3 - "$tmpdir/page" <<'PY'
from pathlib import Path
import re, sys
text = Path(sys.argv[1]).read_text(errors="replace")
text = re.sub(r"<[^>]+>", " ", text)
text = re.sub(r"\s+", " ", text)
for needle in ("Starting from v8.5.7", "automatically detects", "default format is CSV",
               "local file path", "wildcard paths", "arbitrary"):
    pos = text.lower().find(needle.lower())
    if pos >= 0:
        print(text[max(0, pos-220):pos+650])
PY
done

printf '%s\n' '--- cited TiDB pull request metadata and changed paths ---'
curl -fsSL https://api.github.com/repos/pingcap/tidb/pulls/59543 \
  | jq '{number,title,state,merged_at,base:.base.ref,head:.head.ref,body}'
curl -fsSL 'https://api.github.com/repos/pingcap/tidb/pulls/59543/files?per_page=100' \
  | jq -r '.[].filename' | head -100

printf '%s\n' '--- pull request diff excerpts mentioning format detection ---'
curl -fsSL https://github.com/pingcap/tidb/pull/59543.patch \
  | grep -n -C 5 -E 'format|suffix|wildcard|compression|detect' \
  | head -240

Repository: pingcap/docs

Length of output: 15859

🌐 Web query:

site:docs.pingcap.com/tidbcloud sql-statement-import-into v8.5.7 FORMAT CSV SQL PARQUET extension

💡 Result:

The IMPORT INTO statement is a TiDB SQL extension used to efficiently import data into empty tables [1][2]. As of TiDB v8.5.7, this statement supports importing data from files in CSV, SQL, and PARQUET formats [1]. Key details regarding the IMPORT INTO statement include: Functionality: It leverages the Physical Import Mode of TiDB Lightning to perform high-performance data ingestion directly within TiDB, eliminating the need to deploy separate infrastructure for the import process [1][2]. Supported Formats: The statement explicitly supports three file formats: CSV, SQL, and PARQUET [1]. When importing from files, the syntax allows specifying the format using the FORMAT clause (e.g., FORMAT 'CSV', FORMAT 'SQL', or FORMAT 'PARQUET') [1]. If the format is not explicitly specified, it defaults to CSV [1]. Usage: IMPORT INTO... FROM FILE: Used to import data from files located in external storage [1]. IMPORT INTO... FROM SELECT: Used to import the results of a SELECT query into a table [1]. Compatibility: This statement is a specific TiDB extension and is not part of standard MySQL syntax [3]. It is designed for bulk data operations such as migrations or data recovery and does not provide ACID transactional guarantees [2]. For further management, you can monitor the progress of these operations using the SHOW IMPORT JOBS statement [3] or cancel an ongoing job with CANCEL IMPORT JOB [4].

Citations:

Scope format detection by data source.

  • For TiDB Self-Managed server-disk paths, require a supported final suffix: .csv, .sql, .parquet, .gz, .gzip, .zstd, .zst, or .snappy. Matching is case-insensitive. Missing or unknown final suffixes are rejected, even when FORMAT is specified. Update the local-path paragraph at line 118.
  • For wildcard paths without FORMAT, state that TiDB selects an arbitrary matched file to determine the job format. Mixed-format matches can fail.
  • Keep the TiDB Cloud auto-detection text only if the target Cloud release supports it. The public TiDB Cloud reference documents omitted FORMAT as defaulting to CSV.

Sources: Path instructions, MCP tools


</CustomContent>

<CustomContent platform="tidb-cloud">

If you omit `FORMAT`, TiDB automatically detects the format from the `.csv`, `.sql`, or `.parquet` file extension. Detection is case-insensitive. For compressed files, TiDB ignores the `.gz`, `.gzip`, `.zstd`, `.zst`, or `.snappy` compression suffix before detecting the data file format. If the remaining file name has no extension or an unrecognized extension, TiDB treats the file as `CSV`.

</CustomContent>

> **Note:**
>
> For wildcard paths, make sure that all matched files use the same data file format. TiDB determines one format for the import job and applies it to every matched file. Files that do not use that format can cause the import to fail. Use separate `IMPORT INTO` statements for different formats.

@coderabbitai coderabbitai Bot Aug 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document which wildcard file controls automatic detection.

When FORMAT is omitted, TiDB selects an arbitrary matched file for format detection and applies that format to every matched file. The current note omits this rule and can imply per-file detection. (raw.githubusercontent.com)

Exact replacement for Line 153 
-> For wildcard paths, make sure that all matched files use the same data file format. TiDB determines one format for the import job and applies it to every matched file. Files that do not use that format can cause the import to fail. Use separate `IMPORT INTO` statements for different formats.
+> For wildcard paths, when `FORMAT` is omitted, TiDB detects the format from an arbitrary matched file and applies that format to every matched file. Make sure that all matched files use the same data file format. If any matched file uses another format, the import can fail during parsing. Use separate `IMPORT INTO` statements for different formats.

As per path instructions, “For every actionable issue, provide a GitHub committable suggestion block ... whenever the fix can be safely and completely applied to contiguous lines in the diff”; this comment includes an exact contiguous replacement.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
> For wildcard paths, make sure that all matched files use the same data file format. TiDB determines one format for the import job and applies it to every matched file. Files that do not use that format can cause the import to fail. Use separate `IMPORT INTO` statements for different formats.
> For wildcard paths, when `FORMAT` is omitted, TiDB detects the format from an arbitrary matched file and applies that format to every matched file. Make sure that all matched files use the same data file format. If any matched file uses another format, the import can fail during parsing. Use separate `IMPORT INTO` statements for different formats.

Sources: Path instructions, MCP tools


### WithOptions

Expand Down