Tokenize emoji as if they were valid identifiers by estebank · Pull Request #88781 · rust-lang/rust

estebank · 2021-09-09T15:05:48Z

In the lexer, consider emojis to be valid identifiers and reject
them later to avoid knock down parse errors.

Partially address #86102.

rust-highfive · 2021-09-09T15:05:52Z

r? @oli-obk

(rust-highfive has picked a reviewer for you, use r? to override)

ghost · 2021-09-09T17:42:00Z

cc @Manishearth

ghost · 2021-09-13T15:38:18Z

+LL | fn i_like_to_😅_a_lot() -> 👀 {
+   |    ^^^^^^^^^^^^^^^^^^
+
+error: identifiers cannot contain emoji: `ABig👩👩👧👧Family`


🤣 the family got deaggregated

ghost · 2021-09-13T15:38:55Z

@bors r+ rollup

ghost · 2021-09-13T15:38:56Z

📌 Commit 5979ed5 has been approved by oli-obk

ghost · 2021-09-13T19:19:55Z

Failed in #88913.

@bors: r-

ghost · 2021-09-13T20:30:48Z

+    /// Places where identifiers that contain invalid Unicode codepoints but that look like they
+    /// should be. Useful to avoid bad tokenization when encountering emoji. We group them to
+    /// provide a single error per unique incorrect identifier.
+    pub bad_unicode_identifiers: Lock<FxHashMap<Symbol, Vec<Span>>>,


Suggested change

pub bad_unicode_identifiers: Lock<FxHashMap<Symbol, Vec<Span>>>,

pub bad_unicode_identifiers: Lock<VecMap<Symbol, Vec<Span>>>,

Hash maps are great, until they aren't XD

Tried using it's API, but it really doesn't fit well with this case. I'm sorting the output before emitting the diagnostic instead.

ghost · 2021-09-14T11:40:33Z

@bors r=oli-obk

ghost · 2021-09-14T11:40:34Z

📌 Commit 30f9807 has been approved by oli-obk

ghost · 2021-11-24T21:07:42Z

Once rust-lang/cargo#10117 is pulled into this repo, we can merge this PR.

ghost · 2021-11-24T23:14:40Z

@estebank Feel free to update cargo in this PR (git submodule update --remote src/tools/cargo && git add src/tools/cargo) if you don't want to wait. Otherwise, I probably won't update it till next week. I think your change is the only thing not currently synced.

ghost · 2021-11-25T00:20:24Z

@bors r=oli-obk

ghost · 2021-11-25T00:20:25Z

📌 Commit d929164 has been approved by oli-obk

ghost · 2021-11-25T02:46:56Z

⌛ Testing commit d929164 with merge 9dc1d2de4eb6199a77eedd3b61e644919794b59c...

ghost · 2021-11-25T03:51:41Z

The job x86_64-msvc-cargo failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

test workspaces::ws_warn_unused ... ok

failures:

---- lto::complicated stdout ----
running `D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe build -v --release`
thread 'lto::complicated' panicked at '
test failed running `D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe build -v --release`
error: process exited with code 101 (expected 0)

--- stderr
--- stderr
    Updating `dummy-registry` index
---
The targets should have unique names.
Consider changing their names to be unique or compiling them separately.
This may become a hard error in the future; see <https://github.com/rust-lang/cargo/issues/6313>.
   Compiling dep-shared v0.0.1
   Compiling dep-build2 v0.0.1
   Compiling dep-proc-macro2 v0.0.1
   Compiling dep-normal2 v0.0.1
     Running `rustc --crate-name dep_shared D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-shared-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=815663fa4ca118de -C extra-filename=-815663fa4ca118de --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --cap-lints allow`
     Running `rustc --crate-name dep_build2 D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-build2-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=df572682f4638748 -C extra-filename=-df572682f4638748 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --cap-lints allow`
     Running `rustc --crate-name dep_proc_macro2 D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-proc-macro2-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=f829152b9eb79c53 -C extra-filename=-f829152b9eb79c53 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --cap-lints allow`
     Running `rustc --crate-name dep_normal2 D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-normal2-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C metadata=8cf56e81cc7e8007 -C extra-filename=-8cf56e81cc7e8007 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --cap-lints allow`
   Compiling dep-normal v0.0.1
     Running `rustc --crate-name dep_normal D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-normal-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C metadata=06ea447f8e785b1a -C extra-filename=-06ea447f8e785b1a --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_normal2=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_normal2-8cf56e81cc7e8007.rmeta --extern dep_shared=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_shared-815663fa4ca118de.rmeta --cap-lints allow`
   Compiling dep-build v0.0.1
     Running `rustc --crate-name dep_build D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-build-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=4b7343fd1f9e1e57 -C extra-filename=-4b7343fd1f9e1e57 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_build2=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_build2-df572682f4638748.rmeta --extern dep_shared=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_shared-815663fa4ca118de.rmeta --cap-lints allow`
   Compiling dep-proc-macro v0.0.1
     Running `rustc --crate-name dep_proc_macro D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\home\.cargo\registry\src\-45f4c251e71fd8a7\dep-proc-macro-0.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C embed-bitcode=no -C metadata=92b443923583e648 -C extra-filename=-92b443923583e648 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_proc_macro2=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_proc_macro2-f829152b9eb79c53.rlib --extern dep_shared=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_shared-815663fa4ca118de.rlib --extern proc_macro --cap-lints allow`
   Compiling test v0.0.0 (D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo)
     Running `rustc --crate-name build_script_build build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C metadata=1132ccd72abf6ae6 -C extra-filename=-1132ccd72abf6ae6 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\build\test-1132ccd72abf6ae6 -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_build=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_build-4b7343fd1f9e1e57.rlib`
     Running `D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\build\test-1132ccd72abf6ae6\build-script-build`
     Running `rustc --crate-name test src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type cdylib --crate-type staticlib --emit=dep-info,link -C opt-level=3 -C lto -C metadata=10d13534f609fec7 --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_normal=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_normal-06ea447f8e785b1a.rlib --extern dep_proc_macro=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\dep_proc_macro-92b443923583e648.dll`
     Running `rustc --crate-name test src\main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C metadata=f9bd7972e02c2e3b --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_normal=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_normal-06ea447f8e785b1a.rlib --extern dep_proc_macro=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\dep_proc_macro-92b443923583e648.dll`
error: linking with `link.exe` failed: exit code: 1201
  |
  = note: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\link.exe" "/NOLOGO" "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2-tools\\x86_64-pc-windows-msvc\\tmp\\cit\\t1300\\foo\\target\\release\\deps\\test.test.1ec84935-cgu.1.rcgu.o" "/LIBPATH:D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2-tools\\x86_64-pc-windows-msvc\\tmp\\cit\\t1300\\foo\\target\\release\\deps" "/LIBPATH:D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-2e46b64d42824fd5.rlib" "kernel32.lib" "ws2_32.lib" "bcrypt.lib" "advapi32.lib" "userenv.lib" "kernel32.lib" "msvcrt.lib" "/NXCOMPAT" "/LIBPATH:D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "/OUT:D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2-tools\\x86_64-pc-windows-msvc\\tmp\\cit\\t1300\\foo\\target\\release\\deps\\test.exe" "/OPT:REF,ICF" "/DEBUG"
  = note: LINK : fatal error LNK1201: error writing to program database 'D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\test.pdb'; check for insufficient disk space, invalid path, or insufficient privilege

error: could not compile `test` due to previous error

Caused by:
Caused by:
  process didn't exit successfully: `rustc --crate-name test src\main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C metadata=f9bd7972e02c2e3b --out-dir D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps -L dependency=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps --extern dep_normal=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\libdep_normal-06ea447f8e785b1a.rlib --extern dep_proc_macro=D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\tmp\cit\t1300\foo\target\release\deps\dep_proc_macro-92b443923583e648.dll` (exit code: 1)
error: build failed
', src\tools\cargo\tests\testsuite\lto.rs:217:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

ghost · 2021-11-25T03:51:59Z

💔 Test failed - checks-actions

ghost · 2021-11-25T04:14:13Z

@bors retry

LNK1201 #81890

ghost · 2021-11-25T08:16:17Z

⌛ Testing commit d929164 with merge 23a4366...

ghost · 2021-11-25T11:23:51Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 23a4366 to master...

ghost · 2021-11-25T14:23:36Z

Finished benchmarking commit (23a4366): comparison url.

Summary: This benchmark run did not return any relevant changes.

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

rust-highfive assigned oli-obk Sep 9, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 9, 2021

estebank mentioned this pull request Sep 9, 2021

Improve diagnostics for non XID characters in identifiers #86102

Closed

Artoria2e5 mentioned this pull request Sep 9, 2021

Rework emoji tokenizer to match Rust flaneur2020/pua-lang#56

Open

This comment has been minimized.

Sign in to view

Manishearth reviewed Sep 9, 2021

View reviewed changes

Comment thread compiler/rustc_interface/src/passes.rs Outdated

aDotInTheVoid reviewed Sep 9, 2021

View reviewed changes

Comment thread compiler/rustc_errors/src/emitter.rs Outdated

Manishearth changed the title ~~Tokenize emoji as if they were valid indentifiers~~ Tokenize emoji as if they were valid identifiers Sep 10, 2021

oli-obk reviewed Sep 13, 2021

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 13, 2021

GuillaumeGomez mentioned this pull request Sep 13, 2021

Rollup of 7 pull requests #88913

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 13, 2021

oli-obk reviewed Sep 13, 2021

View reviewed changes

estebank force-pushed the emoji-idents branch from 5979ed5 to 30f9807 Compare September 14, 2021 11:40

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 14, 2021

Manishearth mentioned this pull request Sep 14, 2021

Rollup of 8 pull requests #88939

Closed

Manishearth mentioned this pull request Sep 14, 2021

Rollup of 9 pull requests #88940

Closed

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Nov 24, 2021

estebank mentioned this pull request Nov 24, 2021

Change test to not trigger emoji error rust-lang/cargo#10117

Merged

estebank added the S-blocked Status: Blocked on something else such as an RFC or other implementation work. label Nov 24, 2021

Update cargo to fix tokenizer test affected by tokenizing emoji

d929164

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 25, 2021

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 25, 2021

bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 25, 2021

bors merged commit 23a4366 into rust-lang:master Nov 25, 2021

rustbot added this to the 1.58.0 milestone Nov 25, 2021

estebank deleted the emoji-idents branch November 9, 2023 05:14

fmease mentioned this pull request Apr 9, 2024

Rust 2015 and 2018 allow emoji in identifiers in "Unknown prefix" position #123696

Closed

	pub bad_unicode_identifiers: Lock<FxHashMap<Symbol, Vec<Span>>>,
	pub bad_unicode_identifiers: Lock<VecMap<Symbol, Vec<Span>>>,

Uh oh!

Uh oh!

Conversation

estebank commented Sep 9, 2021

Uh oh!

rust-highfive commented Sep 9, 2021

Uh oh!

This comment has been minimized.

ghost commented Sep 9, 2021

Uh oh!

Uh oh!

Uh oh!

ghost Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

ghost commented Sep 13, 2021

Uh oh!

ghost commented Sep 13, 2021

Uh oh!

ghost commented Sep 13, 2021

Uh oh!

ghost Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

ghost Sep 14, 2021

Choose a reason for hiding this comment

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

ghost commented Nov 24, 2021

Uh oh!

ghost commented Nov 24, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

ghost commented Nov 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants