Normalize all identifiers to NFC #2489
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tamaroning
force-pushed
the
ucnorm-parser
branch
3 times, most recently
from
a6273d2 to
6c8bf02
Compare
tamaroning
commented
Aug 6, 2023
Comment on lines
-207
to
-208
| // Input source wrapper thing. | ||
| class InputSource |
There was a problem hiding this comment.
This class has been moved to new file rust-input-source.h to avoid recursive #include
tamaroning
commented
Aug 6, 2023
Comment on lines
+34
to
+35
| static tl::optional<Utf8String> | ||
| make_utf8_string (const std::string &maybe_utf8) |
There was a problem hiding this comment.
This is a factory function for Utf8String.
It returns an optional type value. Returns non-null value if a give std::string is properly encoded as UTF-8.
tamaroning
commented
Aug 6, 2023
Comment on lines
+1
to
+6
| fn main() { | ||
| // U+304C | ||
| let が = (); | ||
| // U+304B + U+3099 | ||
| let _ = が; | ||
| } |
There was a problem hiding this comment.
This code compiles despite that these two identifiers has different byte string.
It means identifier normalization seems to work.
tamaroning
force-pushed
the
ucnorm-parser
branch
from
6c8bf02 to
35b67c3
Compare
This was referenced Aug 6, 2023
P-E-P
reviewed
Aug 8, 2023
gcc/rust/util/rust-unicode.h
Outdated
|
|
||
| return buf; | ||
| }; | ||
|
|
||
| // Returns UTF codepoints when string is valid as UTF-8, returns nullopt |
There was a problem hiding this comment.
Shouldn't this comment be updated ? Looks like the function signature has changed and does not reflect it's description.
gcc/rust/util/rust-unicode.cc
Outdated
| @@ -309,9 +318,10 @@ is_numeric (uint32_t codepoint) | |||
| namespace selftest { | |||
|
|
|||
| void | |||
| assert_normalize (std::vector<uint32_t> origin, std::vector<uint32_t> expected) | |||
| assert_normalize (std::vector<Rust::Codepoint> origin, | |||
| std::vector<Rust::Codepoint> expected) | |||
There was a problem hiding this comment.
Expected could be const. Also, we should probably take some references here.
gcc/rust/ChangeLog: * lex/rust-lex.cc (assert_source_content): Fix namespace specifier (test_buffer_input_source): Likewise. (test_file_input_source): Likewise. * lex/rust-lex.h: Move InputSource ... * lex/rust-input-source.h: ... to here. (New file) * lex/rust-token.cc (nfc_normalize_token_string): New function * lex/rust-token.h (nfc_normalize_token_string): New function * rust-lang.cc (run_rust_tests): Modify order of selftests. * rust-session-manager.cc (validate_crate_name): Modify interface of Utf8String. * util/rust-unicode.cc (lookup_cc): Modify codepoint_t typedef. (lookup_recomp): Likewise. (recursive_decomp_cano): Likewise. (decomp_cano): Likewise. (sort_cano): Likewise. (compose_hangul): Likewise. (assert_normalize): Likewise. (Utf8String::nfc_normalize): New function. * util/rust-unicode.h: Modify interface of Utf8String. gcc/testsuite/ChangeLog: * rust/compile/unicode_norm1.rs: New test. Signed-off-by: Raiki Tamura <[email protected]>
tamaroning
force-pushed
the
ucnorm-parser
branch
from
35b67c3 to
2fa4f4a
Compare
|
@P-E-P Thank you for your review. Fixed all. |
P-E-P
approved these changes
Aug 9, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #2287
depends on #2467Normalize all identifiers (tokens) to their NFC form.
Normalization must be done before any macro expansion.
See https://doc.rust-lang.org/reference/identifiers.html#normalization for details
Changelog