Skip to content

gh-99064: Ignore the encoding declaration when parsing decoded text - #156734

Merged
serhiy-storchaka merged 3 commits into
python:mainfrom
serhiy-storchaka:gh-99064-parse-whole-encoding
Aug 31, 2026
Merged

gh-99064: Ignore the encoding declaration when parsing decoded text#156734
serhiy-storchaka merged 3 commits into
python:mainfrom
serhiy-storchaka:gh-99064-parse-whole-encoding

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Aug 31, 2026 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

ElementTree.parse() mis-decodes a text file in the C implementation: _parse_whole() encodes the read string as UTF-8, but leaves expat to honor the encoding declared in the document.

>>> doc = '<?xml version="1.0" encoding="ISO-8859-1"?><r a="äöü"/>'
>>> ET.parse(io.StringIO(doc)).getroot().get('a')                  # before
'äöü'
>>> ET.parse(io.StringIO(doc), ET.XMLParser()).getroot().get('a')
'äöü'
>>> ET.fromstring(doc).get('a')
'äöü'

XMLParser.feed() overrides the encoding for str data (bpo-16986), but _parse_whole() was never changed, so parse() without an explicit parser has been the odd one out. It now overrides it too, and a text source declaring a non-ASCII-compatible encoding (UTF-16, UTF-32) is no longer a ParseError.
@serhiy-storchaka
pythongh-99064: Ignore the encoding declaration when parsing decoded …
c0c802d 
…text

ElementTree.parse() with a text file mis-decoded the text in the C
implementation: _parse_whole() encoded it as UTF-8, but left expat to honor
the encoding declared in the document.  It now overrides the encoding, as
XMLParser.feed() already does for str data.
@bedevere-app bedevere-app Bot mentioned this pull request Aug 31, 2026
Closed
@serhiy-storchaka serhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Aug 31, 2026
vstinner
vstinner approved these changes Aug 31, 2026
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

_elementtree.XMLParser.feed() already calls EXPAT(st, SetEncoding)(self->parser, "utf-8"); if the input is an Unicode string, so doing the same in _elementtree.XMLParser._parse_whole() sounds correct to me.
Comment thread Lib/test/test_xml_etree.py Outdated
@serhiy-storchaka @vstinner
Apply suggestion from @vstinner
dfbcdb4 
Co-authored-by: Victor Stinner <vstinner@python.org>
@serhiy-storchaka
serhiy-storchaka enabled auto-merge (squash) August 31, 2026 20:42
@serhiy-storchaka
serhiy-storchaka merged commit c83013c into python:main Aug 31, 2026
54 checks passed
@miss-islington-app

miss-islington-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖
@miss-islington-app

miss-islington-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c83013c92dfdc77b87a523b736b76d5abb8ede2a 3.15
@miss-islington-app

miss-islington-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c83013c92dfdc77b87a523b736b76d5abb8ede2a 3.14
@miss-islington-app

miss-islington-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c83013c92dfdc77b87a523b736b76d5abb8ede2a 3.13
@bedevere-app

bedevere-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

GH-156750 is a backport of this pull request to the 3.15 branch.
@bedevere-app bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Aug 31, 2026
@bedevere-app

bedevere-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

GH-156751 is a backport of this pull request to the 3.14 branch.
@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Aug 31, 2026
@bedevere-app

bedevere-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

GH-156752 is a backport of this pull request to the 3.13 branch.
@bedevere-app bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Aug 31, 2026
serhiy-storchaka added a commit that referenced this pull request Aug 31, 2026
@serhiy-storchaka
[3.14] gh-99064: Ignore the encoding declaration when parsing decoded…
4e8bce4 
… text (GH-156734) (GH-156751)

ElementTree.parse() with a text file mis-decoded the text in the C
implementation: _parse_whole() encoded it as UTF-8, but left expat to honor
the encoding declared in the document.  It now overrides the encoding, as
XMLParser.feed() already does for str data.
(cherry picked from commit c83013c)
serhiy-storchaka added a commit that referenced this pull request Sep 1, 2026
@serhiy-storchaka
[3.13] gh-99064: Ignore the encoding declaration when parsing decoded…
16f4dee 
… text (GH-156734) (GH-156752)

ElementTree.parse() with a text file mis-decoded the text in the C
implementation: _parse_whole() encoded it as UTF-8, but left expat to honor
the encoding declared in the document.  It now overrides the encoding, as
XMLParser.feed() already does for str data.
(cherry picked from commit c83013c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@serhiy-storchaka @vstinner