Enable string conversion in EUC-JP. #1296
Conversation
Background: EUC-JP is not supported by OSS CoreFoundation, while it is supported by macOS Foundation Framework. See swiftlang#1016 This commit resolves the issue by calling ICU API if necessary.
|
@swift-ci Please test |
| // Attempt an up-call into swift-corelibs-foundation, which can defer to the CoreFoundation implementation | ||
| _cfStringEncodingConvert(string: self, using: encoding.rawValue, allowLossyConversion: allowLossyConversion) ?? | ||
| // Or attempt an up-call into ICU via FoundationInternationalization | ||
| _icuStringEncodingConvert(string: self, using: encoding, allowLossyConversion: allowLossyConversion) |
There was a problem hiding this comment.
When do we fall back to the ICU one here? On Darwin this encoding is handled via CF, which itself calls into ICU.
There was a problem hiding this comment.
I guess that'll happen if import Foundation has not been done (and CF is present), but import FoundationInternationalization has. I wonder how we can further unify these paths; on Darwin too...
There was a problem hiding this comment.
I had a similar question - I believe _cfStringEncodingConvert will return nil if any of the following are the case:
- CF isn't loaded
- CF is loaded but it doesn't support the provided encoding
- CF is loaded and supports the encoding but the data is malformed
We want the up-call in 2, and I think Tony's question was about 1, but do we want the up call in case 3?
There was a problem hiding this comment.
I now agree that unexpected fallback should be avoided.
Although ICU can support other encodings, I should've made this change focus on EUC-JP.
| internal import _FoundationICU | ||
|
|
||
| private extension String.Encoding { | ||
| var _icuConverterName: String? { |
There was a problem hiding this comment.
Should we make sure this returns nil for any encoding which we never want to use ICU for? UTF8, ASCII, UTF16/32, etc.
There was a problem hiding this comment.
Certainly.
I'll let them return nil here or in ICU.StringConverter's init.
| } | ||
|
|
||
| extension ICU { | ||
| final class StringConverter: @unchecked Sendable { |
There was a problem hiding this comment.
nit: does this need the @unchecked? I would have assumed that since it's a final class with all immutable, Sendable properties that the compiler can validate this conformance
There was a problem hiding this comment.
Thank you.
That's a vestige of my first implementation where _converter was a bare pointer...
| } | ||
|
|
||
| extension ICU.StringConverter { | ||
| nonisolated(unsafe) static private var _converters: LockedState<[String.Encoding: ICU.StringConverter]> = .init(initialState: [:]) |
There was a problem hiding this comment.
I think this should just be a static private let (the immutable-ness affects the LockedState itself but not the contents IIRC, and this isn't unsafe because it's guarded by a lock)
There was a problem hiding this comment.
Indeed it should be.
This is also a vestige of my first impl, I guess.
| return nil | ||
| } | ||
| let data = Data( | ||
| bytesNoCopy: UnsafeMutableRawPointer(mutating: pointer), |
There was a problem hiding this comment.
Is it worth adding a comment here about this being "safe" because this data neither escapes this function nor is mutated? Since the pointer provided is not guaranteed to live longer than the function or be mutable, this could be a trap for a later addition that doesn't see those two constraints
There was a problem hiding this comment.
I agree it's better to have some comments. I'm going to add ones.
| // Attempt an up-call into swift-corelibs-foundation, which can defer to the CoreFoundation implementation | ||
| _cfStringEncodingConvert(string: self, using: encoding.rawValue, allowLossyConversion: allowLossyConversion) ?? | ||
| // Or attempt an up-call into ICU via FoundationInternationalization | ||
| _icuStringEncodingConvert(string: self, using: encoding, allowLossyConversion: allowLossyConversion) |
There was a problem hiding this comment.
I had a similar question - I believe _cfStringEncodingConvert will return nil if any of the following are the case:
- CF isn't loaded
- CF is loaded but it doesn't support the provided encoding
- CF is loaded and supports the encoding but the data is malformed
We want the up-call in 2, and I think Tony's question was about 1, but do we want the up call in case 3?
Background: EUC-JP is not supported by OSS CoreFoundation, while it is supported by macOS Foundation Framework.
See #1016
This PR resolves the issue by calling ICU API if necessary.