URL: Update IdnaTestV2 to UTS46 16.0.0 #48301
Conversation
To work correctly if the OS system default encoding is not utf-8
| @@ -92,13 +96,11 @@ def parse(lines, exclude_ipv4_like, exclude_std3, exclude_bidi): | |||
| continue | |||
|
|
|||
| if exclude_std3: | |||
| if re.search(r"\u2260|\u226E|\u226F|\<|\>|\$|,", to_unicode): | |||
| if re.search(r"\<|\>|\$|\?|,", to_unicode): | |||
There was a problem hiding this comment.
- I removed
\u2260,\u226Eand\u226F. because they are labeledvalidstarting with UTS46 15.1.0. - I added
?because IdnaTestV2.txt contains tests with it and it is the forbidden domain code point. - I am not sure if
$and,are still needed here?
|
Thank you for working on this, but I'm not sure this can be merged as-is. This results in failures for both Firefox and Safari, both thought to have compliant implementations. I'm not sure I fully trust the updated tests. Especially given that we still have some open issues against the URL Standard about sorting out some IDNA issues with UTS 46. Did you investigate the failures? |
|
I think it's normal that about 70 tests fail with Firefox and Safari. About the same number of tests failed when I tested with my UTS46 15.1.0 implementation. There are no failed tests on my UTS46 16.0.0 implementation. I think there may be similar results with browsers when they switch to UTS46 16.0.0 |
|
That's fair, I guess the problem is that I'm not entirely convinced that the UTS 46 update is fault-free given the response we've been given to some of the issues we've raised. |
Updates
IdnaTestV2-parser.pyfor version 16.0.0 of Unicode IDNA Compatibility Processing [UTS46] and fixes other issues:encoding="utf-8"parameter to file open calls to run correctly if the default system encoding is notutf-8--exclude_bidichanged to--exclude-bidi