XML refers to Extensible Markup Language, a subset of the Standard Generalized Markup Language, a markup language used to mark up electronic documents to make them structured. XML is designed to transfer and store data.

Python has three common ways of parsing XML: SAX (simple API for XML), DOM (Document Object Model), and ElementTree.

  • DOM approach: DOM, translated as Document Object Model, is a standard programming interface recommended by the W3C organization that parses XML data into a tree in memory, and manipulates XML by manipulating the tree.
  • SAX: SAX is an event-driven model for handling XML. It scans documents line by line, parsing them as it goes, and has great advantages for parsing large documents.
  • ElementTree approach: ElementTree has better performance compared to DOM, about the same performance as SAX, and an easy to use API.

Python has very many other parsing tools in addition to the built-in xml parser. Which tool is easier to use and has better performance? Let’s explore.

xml.dom.* module

xml.dom implements the DOM API developed by the W3C. xml.dom is a package you can use if you are used to using the DOM API. xml.dom parses XML data into a tree in memory and manipulates XML by manipulating the tree. a DOM parser reads the whole document at once when parsing an XML document, saving all the elements of the document in memory in You can then use the different functions provided by the DOM to read or modify the content and structure of the document, or you can write the modified content to the xml file.

Note: There are many modules in the xml.dom package, note the differences between them.

  • minidom is an extremely simplified implementation of the DOM API, much simpler than the full version of DOM, and the package is also much smaller.
  • The pulldom module provides a “pull parser”, the basic concept behind which refers to pulling events from the XML stream and then processing them.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# <?xml version="1.0" encoding="UTF-8"?>
#  <employees>
#   <employee>
#     <name>linux</name>
#     <age>30</age>
#   </employee>
#   <employee>
#     <name>windows</name>
#     <age>20</age>
#   </employee>
#  </employees>

from xml.dom import minidom
doc = minidom.parse("employees.xml")
root = doc.documentElement
employees = root.getElementsByTagName("employee")
for employee in employees:
    print (employee.nodeName)
    print (employee.toxml())
    nameNode = employee.getElementsByTagName("name")[0]
    print (nameNode.childNodes)
    print (nameNode.nodeName + ":" + nameNode.childNodes[0].nodeValue)
    ageNode = employee.getElementsByTagName("age")[0]
    print (ageNode.childNodes)
    print (ageNode.nodeName + ":" + ageNode.childNodes[0].nodeValue)
    for n in employee.childNodes:
        print (n)

xml.sax.* module

The xml.sax.* module is an implementation of the SAX API. This module sacrifices convenience for speed and memory footprint.SAX (simple API for XML), is based on event handling, when XML documents are read sequentially, each time an element is encountered the corresponding event handler function is triggered to handle it.

SAX features.

  • is an event-based API
  • operates at a lower level than the DOM
  • gives you more control than the DOM
  • is almost always more efficient than the DOM
  • but unfortunately, requires more work than the DOM

Parsing XML with Python requires import xml.sax and xml.sax.handler

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
# <?xml version="1.0"?>
# <collection shelf="New Arrivals">
#     <movie title="Enemy Behind">
#        <type>War, Thriller</type>
#        <format>DVD</format>
#        <year>2003</year>
#        <rating>PG</rating>
#        <stars>10</stars>
#        <description>Talk about a US-Japan war</description>
#     </movie>
#     <movie title="Transformers">
#        <type>Anime, Science Fiction</type>
#        <format>DVD</format>
#        <year>1989</year>
#        <rating>R</rating>
#        <stars>8</stars>
#        <description>A schientific fiction</description>
#     </movie>
#        <movie title="Trigun">
#        <type>Anime, Action</type>
#        <format>DVD</format>
#        <episodes>4</episodes>
#        <rating>PG</rating>
#        <stars>10</stars>
#        <description>Vash the Stampede!</description>
#     </movie>
#     <movie title="Ishtar">
#        <type>Comedy</type>
#        <format>VHS</format>
#        <rating>PG</rating>
#        <stars>2</stars>
#        <description>Viewable boredom</description>
#     </movie>
# </collection>

import xml.sax

class MovieHandler( xml.sax.ContentHandler):
   def __init__(self):
      self.CurrentData = ""
      self.type = ""
      self.format = ""
      self.year = ""
      self.rating = ""
      self.stars = ""
      self.description = ""

   # 元素开始事件处理
   def startElement(self, tag, attributes):
      self.CurrentData = tag
      if tag == "movie":
         print "*****Movie*****"
         title = attributes["title"]
         print "Title:", title

   # 元素结束事件处理
   def endElement(self, tag):
      if self.CurrentData == "type":
         print "Type:", self.type
      elif self.CurrentData == "format":
         print "Format:", self.format
      elif self.CurrentData == "year":
         print "Year:", self.year
      elif self.CurrentData == "rating":
         print "Rating:", self.rating
      elif self.CurrentData == "stars":
         print "Stars:", self.stars
      elif self.CurrentData == "description":
         print "Description:", self.description
      self.CurrentData = ""

   # 内容事件处理
   def characters(self, content):
      if self.CurrentData == "type":
         self.type = content
      elif self.CurrentData == "format":
         self.format = content
      elif self.CurrentData == "year":
         self.year = content
      elif self.CurrentData == "rating":
         self.rating = content
      elif self.CurrentData == "stars":
         self.stars = content
      elif self.CurrentData == "description":
         self.description = content

if ( __name__ == "__main__"):
   # 创建一个 XMLReader
   parser = xml.sax.make_parser()
   # turn off namepsaces
   parser.setFeature(xml.sax.handler.feature_namespaces, 0)
   # 重写 ContextHandler
   Handler = MovieHandler()
   parser.setContentHandler( Handler )
   parser.parse("movies.xml")

xml.parser.expat

xml.parser.expat provides a direct, underlying API interface to the expat parser written in C. The expat interface is similar to SAX in that it is also based on an event callback mechanism, but this interface is not standardized and only applies to the expat library.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import xml.parsers.expat

class ExParser(object):
    '''Parse roster xml'''
    def __init__(self, xml_raw):
        '''init parser and setup handlers'''
        self.parser = xml.parsers.expat.ParserCreate()

        #connect handlers
        self.parser.StartElementHandler = self.start_element
        self.parser.EndElementHandler = self.end_element
        self.parser.CharacterDataHandler = self.char_data
        self.parser.Parse(xml_raw)
        del(xml_raw)

    def start_element(self, name, attrs):
        '''Start xml element handler'''
        print('start:'+name)

    def end_element(self, name):
        '''End xml element handler'''
        print('end:'+name)

    def char_data(self, data):
        '''Char xml element handler'''
        print('data is '+data)

ElementTree

The xml.etree.ElementTree module provides a lightweight, Pythonic API along with an efficient C implementation, xml.etree.cElementTree. ET is faster and the API is more straightforward and easy to use than the DOM. Compared to SAX, the ET.iterparse function also provides on-demand parsing without reading the entire document in memory at once. ET’s performance is roughly similar to the SAX module, but it has a higher-level API that is easier for users to use.

There are two implementations of ElementTree in the Python standard library. One is a pure Python implementation such as xml.etree.ElementTree, and the other is the faster xml.etree.cElementTree. Remember: try to use the C implementation, because it’s faster and consumes less memory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# <?xml version="1.0"?>
# <doc>
#     <branch name="testing" hash="1cdf045c">
#         text,source
#     </branch>
#     <branch name="release01" hash="f200013e">
#         <sub-branch name="subrelease01">
#             xml,sgml
#         </sub-branch>
#     </branch>
#     <branch name="invalid">
#     </branch>
# </doc>

try:
    import xml.etree.cElementTree as ET
except ImportError:
    import xml.etree.ElementTree as ET

tree = ET.ElementTree(file='doc1.xml')
root = tree.getroot()
print root.tag, root.attrib
for child_of_root in root:
    print child_of_root.tag, child_of_root.attrib
for elem in tree.iter():
    print elem.tag, elem.attrib
for elem in tree.iter(tag='branch'):
    print elem.tag, elem.attrib
for elem in tree.iterfind('branch/sub-branch'):
    print elem.tag, elem.attrib
for elem in tree.iterfind('branch[@name="release01"]'):
    print elem.tag, elem.attrib

Element object

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class xml.etree.ElementTree.Element(tag, attrib={}, **extra)

  tagstring元素代表的数据种类
  textstring元素的内容
  tailstring元素的尾形
  attribdictionary元素的属性字典
  
  #针对属性的操作
  clear()清空元素的后代属性text和tail也设置为None
  get(key, default=None)获取key对应的属性值如该属性不存在则返回default值
  items()根据属性字典返回一个列表列表元素为(key, value)。
  keys()返回包含所有元素属性键的列表
  set(key, value)设置新的属性键与值

  #针对后代的操作
  append(subelement)添加直系子元素
  extend(subelements)增加一串元素对象作为子元素。#python2.7新特性
  find(match)寻找第一个匹配子元素匹配对象可以为tag或path
  findall(match)寻找所有匹配子元素匹配对象可以为tag或path
  findtext(match)寻找第一个匹配子元素返回其text值匹配对象可以为tag或path
  insert(index, element)在指定位置插入子元素
  iter(tag=None)生成遍历当前元素所有后代或者给定tag的后代的迭代器。#python2.7新特性
  iterfind(match)根据tag或path查找所有的后代
  itertext()遍历所有后代并返回text值
  remove(subelement)删除子元素

ElementTree object

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
class xml.etree.ElementTree.ElementTree(element=None, file=None)
  element如果给定则为新的ElementTree的根节点

  _setroot(element)用给定的element替换当前的根节点慎用
  
  # 以下方法与Element类中同名方法近似,区别在于它们指定以根节点作为操作对象。
  find(match)
  findall(match)
  findtext(match, default=None)
  getroot()获取根节点.
  iter(tag=None)
  iterfind(match)
  parse(source, parser=None)装载xml对象source可以为文件名或文件类型对象.
  write(file, encoding="us-ascii", xml_declaration=None, default_namespace=None,method="xml")

Module Method

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
xml.etree.ElementTree.Comment(text=None)
创建一个特别的element通过标准序列化使其代表了一个commentcomment可以为bytestring或unicode

xml.etree.ElementTree.dump(elem)
生成一个element tree通过sys.stdout输出elem可以是元素树或单个元素这个方法最好只用于debug

xml.etree.ElementTree.fromstring(text)
text是一个包含XML数据的字符串与XML()方法类似返回一个Element实例

xml.etree.ElementTree.fromstringlist(sequence, parser=None)
从字符串的序列对象中解析xml文档缺省parser为XMLParser返回Element实例

xml.etree.ElementTree.iselement(element)
检查是否是一个element对象

xml.etree.ElementTree.iterparse(source, events=None, parser=None)
将文件或包含xml数据的文件对象递增解析为element tree并且报告进度events是一个汇报列表如果忽略将只有end事件会汇报出来
注意iterparse()只会在看见开始标签的">"符号时才会抛出start事件因此届时属性是已经定义了但是text和tail属性在那时还没有定义同样子元素也没有定义因此他们可能不能被显示出来如果你想要完整的元素请查找end事件

xml.etree.ElementTree.parse(source, parser=None)
将一个文件或者字符串解析为element tree

xml.etree.ElementTree.ProcessingInstruction(target, text=None)
这个方法会创建一个特别的element该element被序列化为一个xml处理命令

xml.etree.ElementTree.register_namespace(prefix, uri)
注册命名空间前缀这个注册是全局有效任何已经给出的前缀或者命名空间uri的映射关系会被删除

xml.etree.ElementTree.SubElement(parent, tag, attrib={}, **extra)
子元素工厂创建一个Element实例并追加到已知的节点

xml.etree.ElementTree.tostring(element, encoding="us-ascii", method="xml")
生成一个字符串来表示表示xml的element包括所有子元素element是Element实例method为"xml","html","text"返回包含了xml数据的字符串

xml.etree.ElementTree.tostringlist(element, encoding="us-ascii", method="xml")
生成一个字符串来表示表示xml的element包括所有子元素element是Element实例method为"xml","html","text"返回包含了xml数据的字符串列表

xml.etree.ElementTree.XML(text, parser=None)
从一个字符串常量中解析出xml片段返回Element实例

xml.etree.ElementTree.XMLID(text, parser=None)
从字符串常量解析出xml片段同时返回一个字典用以映射element的id到其自身

xmltodict

xmltodict is a Python module that allows you to process XML as if you were processing JSON.

For an XML file like this.

1
2
3
4
5
6
7
8
9
<mydocument has="an attribute">
  <and>
    <many>elements</many>
    <many>more elements</many>
  </and>
  <plus a="complex">
    element as well
  </plus>
</mydocument>

It can be loaded into a Python dictionary at

1
2
3
4
import xmltodict

with open('path/to/file.xml') as fd:
    obj = xmltodict.parse(fd.read())

You can access elements, attributes and values.

1
2
3
4
doc['mydocument']['@has'] # == u'an attribute'
doc['mydocument']['and']['many'] # == [u'elements', u'more elements']
doc['mydocument']['plus']['@a'] # == u'complex'
doc['mydocument']['plus']['#text'] # == u'element as well'

xmltodict also has an unparse function that allows you to convert back to XML. this function has a streaming mode suitable for handling files that cannot be put into memory, and it also supports namespaces.

dicttoxml is a tool for converting dictionaries to xml, for those interested.

untangle

The untangle library maps an XML document to a Python object that contains the node and attribute information of the original document in its structure.

Example.

1
2
3
4
<?xml version="1.0"?>
<root>
    <child name="child1">
</root>

It can be loaded as follows.

1
2
3
import untangle

obj = untangle.parse('path/to/file.xml')

Then you can get the child element name like this.

1
obj.root.child['name']

untangle also supports loading XML from strings or URLs.

Other Tools