[ywxt]

This PR implements the SIMD UTF-8 validation lookup algorithm proposed by Keiser and Lemire.

There have been several previous attempts to improve UTF-8 validation, but they raised various concerns:

• Test utf8 validaiton based on rust-isutf8 #68463: SIMD implementation using SSE3 intrinsics and targeting AVX2, while Rust's default x86 target feature level is SSE2. This PR instead targets AArch64, where NEON is enabled by default.

• Faster UTF-8 string validation #111367:

  1. It did not report the error position on validation failure. This PR falls back to the scalar path to preserve the existing error reporting behavior.

  2. It included only a small set of benchmarks and did not cover enough real-world workloads. See the benchmark results below.

• Rewrite libcore's UTF-8 validation for performance #107760 and Rewrite UTF-8 validation in shift-based DFA for 70%~135% performance increase on non-ASCII strings #136693:

  1. They used a large lookup table of about 1 KiB, which is less suitable for embedded environments. This PR adds only three 16-byte lookup tables, for a total of 48 bytes.

  2. They also lacked sufficient benchmark coverage.

The current implementation still has regressions for small inputs and early errors. I would appreciate suggestions on how to improve this.

r? @Amanieu

1. Overall

Category	Cases	NEON / main avg.	DFA / main avg.
Aligned in-tree benchmarks	269	+80.56%	+15.20%
Unaligned in-tree benchmarks	5,111	+77.85%	+18.68%
Aligned real-world text	264	+237.46%	+75.05%
Unaligned real-world text	5,016	+234.26%	+78.85%
End-to-end CBOR decoding	8	+0.48%	−1.01%
Overall	10,668	+155.35%	+48.26%

2. UTF-8 microbenchmarks

Aligned

Type	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
Valid ASCII	28	−1.12%	−9.48% to +0.36%	−4.05%	−44.13% to +34.38%
Valid Latin	28	+130.75%	−13.51% to +325.28%	+45.43%	−43.15% to +83.02%
Valid CJK	28	+127.82%	−5.96% to +311.02%	+51.64%	−44.07% to +103.98%
Valid emoji	28	+91.67%	−12.10% to +236.91%	+13.28%	−44.07% to +42.19%
Valid mixed	28	+163.79%	−5.96% to +383.80%	+65.68%	−44.13% to +100.72%
In-tree corpora (valid)	20	+173.52%	−9.09% to +476.49%	+46.41%	−24.63% to +113.00%
Errors in ASCII	29	+3.27%	−2.24% to +10.31%	−29.68%	−55.67% to +21.23%
Errors in non-ASCII	40	+11.99%	−12.03% to +50.93%	−19.49%	−55.89% to +64.46%
In-tree corpora (errors)	36	+91.18%	−0.81% to +446.68%	+1.53%	−56.04% to +109.93%
Truncated tail	4	−4.43%	−6.70% to −2.66%	−17.37%	−51.00% to +27.09%
All	269	+80.56%	−13.51% to +476.49%	+15.20%	−56.04% to +113.00%

Length	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
0	5	0.00%	−0.19% to +0.19%	−28.58%	−28.69% to −28.45%
1–8	20	−2.09%	−9.09% to +4.54%	−19.48%	−44.13% to +11.65%
9–31	5	−2.53%	−12.10% to +0.50%	+8.43%	−33.48% to +45.50%
32–64	42	+1.92%	−13.51% to +19.97%	−1.96%	−56.04% to +103.98%
65–128	16	+40.12%	−9.48% to +125.88%	+41.65%	−32.21% to +100.72%
129–256	27	+82.66%	−3.95% to +201.41%	+25.94%	−55.73% to +93.38%
257–512	15	+149.96%	−0.63% to +271.68%	+49.97%	−9.94% to +94.84%
513–1024	15	+198.34%	−1.14% to +328.32%	+52.93%	+9.67% to +95.54%
1025–2048	15	+231.99%	−0.85% to +364.42%	+55.40%	+20.38% to +95.27%
2049–4096	89	+74.11%	−12.03% to +446.68%	−0.08%	−55.87% to +109.93%
All	269	+80.56%	−13.51% to +476.49%	+15.20%	−56.04% to +113.00%

Unaligned

Type	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
Valid ASCII	532	−0.29%	−10.33% to +11.59%	+14.89%	−44.24% to +68.44%
Valid Latin	532	+121.99%	−21.98% to +346.51%	+45.48%	−44.19% to +102.53%
Valid CJK	532	+130.12%	−5.96% to +365.95%	+51.91%	−44.24% to +108.43%
Valid emoji	532	+82.04%	−15.61% to +236.04%	+13.70%	−44.19% to +53.11%
Valid mixed	532	+163.84%	−5.96% to +441.98%	+63.88%	−44.19% to +102.65%
In-tree corpora (valid)	380	+164.07%	−10.19% to +544.67%	+48.81%	−25.71% to +120.31%
Errors in ASCII	551	+2.01%	−5.81% to +20.04%	−18.69%	−59.28% to +39.24%
Errors in non-ASCII	760	+13.88%	−13.68% to +79.56%	−19.15%	−68.63% to +74.77%
In-tree corpora (errors)	684	+86.85%	−25.81% to +510.71%	+2.66%	−58.68% to +111.31%
Truncated tail	76	−3.80%	−6.02% to +4.20%	−13.99%	−54.42% to +38.97%
All	5,111	+77.85%	−25.81% to +544.67%	+18.68%	−68.63% to +120.31%

Length	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
0	95	0.00%	−0.19% to +0.19%	−28.75%	−29.23% to −28.35%
1–8	380	−0.62%	−9.21% to +4.86%	−18.91%	−44.24% to +22.57%
9–31	95	−2.26%	−12.14% to +1.63%	+7.13%	−36.77% to +62.20%
32–64	798	+1.80%	−21.98% to +46.21%	+1.94%	−59.96% to +108.43%
65–128	304	+38.65%	−10.33% to +127.49%	+48.51%	−32.80% to +102.53%
129–256	513	+80.97%	−25.81% to +216.09%	+30.74%	−58.68% to +100.07%
257–512	285	+149.13%	−6.23% to +302.93%	+54.47%	−18.60% to +102.59%
513–1024	285	+189.90%	−1.51% to +369.04%	+55.61%	+7.53% to +96.71%
1025–2048	285	+218.97%	−1.89% to +415.69%	+56.88%	+13.41% to +95.25%
2049–4096	1,691	+72.29%	−13.68% to +510.71%	+3.84%	−68.63% to +111.31%
All	5,111	+77.85%	−25.81% to +544.67%	+18.68%	−68.63% to +120.31%

3. Real-world web / book text

Aligned

Corpus	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
Chinese Wikipedia HTML	33	+36.39%	−43.86% to +391.31%	+5.73%	−52.80% to +113.82%
Chinese Wikipedia, collapsed whitespace	33	+38.41%	−43.57% to +387.74%	+5.85%	−50.13% to +102.19%
Chinese book	33	+174.77%	−43.57% to +753.46%	+82.42%	−50.64% to +194.92%
Greek book	33	+172.31%	−43.57% to +1,097.20%	+76.25%	−50.00% to +264.52%
Vietnamese book	33	+204.45%	−43.57% to +1,647.27%	+83.38%	−50.38% to +260.90%
Chinese, ASCII removed	33	+192.75%	−44.02% to +646.25%	+97.33%	−50.64% to +188.65%
Greek, ASCII removed	33	+143.70%	−43.86% to +423.46%	+60.86%	−53.49% to +105.46%
Vietnamese, ASCII removed	33	+181.79%	−43.57% to +1,512.11%	+88.11%	−50.00% to +518.56%

Length	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
0	8	0.00%	0.00% to 0.00%	−28.77%	−28.94% to −28.55%
1–8	32	−1.07%	−5.96% to +3.68%	−18.24%	−44.24% to +13.17%
9–31	8	+0.34%	−4.75% to +10.62%	+7.55%	−36.51% to +32.56%
32–64	32	+0.37%	−14.64% to +10.57%	+32.81%	−45.71% to +102.62%
65–128	24	+47.84%	−9.61% to +145.32%	+53.21%	−65.40% to +128.54%
129–256	24	+117.42%	−4.08% to +243.29%	+64.18%	−61.12% to +138.24%
257–512	24	+188.77%	−1.86% to +356.66%	+74.15%	−36.60% to +144.78%
513–1024	24	+258.09%	−1.02% to +452.93%	+85.55%	−4.45% to +160.26%
1025–2048	24	+314.88%	+47.83% to +513.49%	+92.10%	+28.29% to +160.24%
2049–4096	24	+398.97%	+103.96% to +567.70%	+110.06%	+44.01% to +166.02%
/>4096	40	+772.13%	+156.90% to +1,873.90%	+200.33%	+66.05% to +580.88%
All	264	+237.46%	−14.64% to +1,873.90%	+75.05%	−65.40% to +580.88%

Unaligned

Corpus	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
Greek book	33	+264.99%	−2.33% to +1,137.40%	+71.97%	−28.55% to +253.14%
Greek, ASCII removed	33	+153.61%	−14.64% to +319.40%	+49.78%	−28.94% to +95.34%
Vietnamese book	33	+400.87%	−5.96% to +1,873.90%	+90.66%	−44.24% to +246.86%
Vietnamese, ASCII removed	33	+377.32%	0.00% to +1,615.18%	+141.25%	−28.85% to +580.88%
Chinese book	33	+276.28%	−11.35% to +626.70%	+103.55%	−28.85% to +166.25%
Chinese, ASCII removed	33	+272.52%	−5.48% to +571.58%	+108.42%	−28.85% to +166.21%
Chinese Wikipedia, collapsed whitespace	33	+77.24%	−9.61% to +414.54%	+14.67%	−65.40% to +126.02%
Chinese Wikipedia HTML	33	+76.83%	−9.59% to +417.62%	+20.06%	−44.24% to +127.10%
All	264	+237.46%	−14.64% to +1,873.90%	+75.05%	−65.40% to +580.88%

Length	Cases	NEON / main avg.	NEON worst–best	DFA / main avg.	DFA worst–best
0	152	0.00%	−0.19% to +0.19%	−28.72%	−29.18% to −28.16%
1–8	608	−0.44%	−11.59% to +7.69%	−17.84%	−44.58% to +23.40%
9–31	152	+0.71%	−5.52% to +10.92%	+11.24%	−38.32% to +62.22%
32–64	608	+0.69%	−20.63% to +13.34%	+43.59%	−32.45% to +102.28%
65–128	456	+46.73%	−20.62% to +157.28%	+63.08%	−32.46% to +138.36%
129–256	456	+113.78%	−6.10% to +262.20%	+72.74%	−27.77% to +158.14%
257–512	456	+184.38%	−7.87% to +424.16%	+79.95%	−19.55% to +162.99%
513–1024	456	+250.59%	−2.85% to +525.80%	+86.83%	+8.83% to +176.10%
1025–2048	456	+307.01%	+34.60% to +581.54%	+91.78%	+33.15% to +168.49%
2049–4096	456	+392.57%	+87.33% to +640.57%	+110.09%	+37.40% to +166.32%
/>4096	760	+768.75%	+151.46% to +1,897.40%	+200.63%	+65.16% to +601.63%
All	5,016	+234.26%	−20.63% to +1,897.40%	+78.85%	−44.58% to +601.63%

4. CBOR

benchmarks: https://github.com/oxalica/shift-dfa-utf8/blob/d2b25627de764d1e857095bcf72136f921dd7d27/bench-de/benches/cbor.rs

File	Mode	main	NEON	DFA	NEON / main	DFA / main
trivial_helloworld	ignored	0.083 µs	0.084 µs	0.086 µs	−1.12%	−4.19%
trivial_helloworld	value	0.107 µs	0.107 µs	0.109 µs	−0.62%	−2.45%
twitter.json	ignored	1,778.301 µs	1,731.679 µs	1,775.472 µs	+2.69%	+0.12%
twitter.json	value	3,149.307 µs	3,086.004 µs	3,109.508 µs	+2.05%	+1.02%
citm_catalog.json	ignored	3,852.560 µs	3,838.158 µs	3,950.303 µs	+0.38%	−2.39%
citm_catalog.json	value	6,369.132 µs	6,365.560 µs	6,389.013 µs	+0.06%	−0.41%
canada.json	ignored	9,199.590 µs	9,197.941 µs	9,205.096 µs	+0.02%	−0.06%
canada.json	value	15,456.600 µs	15,398.894 µs	15,422.068 µs	+0.37%	+0.25%
Average					+0.48%	−1.01%

[rustbot]

⚠️ #[rustc_allow_const_fn_unstable] needs careful audit to avoid accidentally exposing unstable
implementation details on stable.

cc @rust-lang/wg-const-eval