boost_endian/doc/index.html

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns="http://www.w3.org/TR/REC-html40">

<head>
<meta name="GENERATOR" content="Microsoft FrontPage 5.0">
<meta name="ProgId" content="FrontPage.Editor.Document">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">

<title>Boost Endian Library</title>
<link rel="stylesheet" type="text/css" href="../../../doc/src/minimal.css">
</head>

<body>

<table border="0" cellpadding="5" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="710">
  <tr>
    <td width="277">
<a href="../../../index.html">
<img src="../../../boost.png" alt="boost.png (6897 bytes)" align="middle" border="0" width="277" height="86"></a></td>
    <td width="413" align="middle">
    <font size="7">Endian Library</font></td>
  </tr>
</table>

<table border="0" cellpadding="5" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" bgcolor="#D7EEFF" width="100%">
  <tr>
    <td><b><a href="../../../index.htm">Boost Home</a>&nbsp;&nbsp;&nbsp;&nbsp;
    <a href="index.html">Endian Home</a>&nbsp;&nbsp;&nbsp;&nbsp;
    <a href="conversion.html">Conversion Functions</a>&nbsp;&nbsp;&nbsp;&nbsp;
    <a href="types.html">Endian Types</a>&nbsp;&nbsp;&nbsp;&nbsp; Tutorial</b></td>
  </tr>
</table>
<table border="1" cellpadding="5" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" align="right">
  <tr>
    <td width="100%" bgcolor="#D7EEFF" align="center">
      <i><b>Contents</b></i></td>
  </tr>
  <tr>
    <td width="100%" bgcolor="#E8F5FF">
      <a href="#Abstract">Abstract</a><br>
      <a href="#Introduction-to-endianness">Introduction to endianness</a><br>
      <a href="#Introduction">Introduction to the Boost.Endian library</a><br>
      <a href="#Acknowledgements">Acknowledgements</a></td>
  </tr>
  <tr>
    <td width="100%" bgcolor="#D7EEFF" align="center">
      <b><i>Headers</i></b></td>
  </tr>
  <tr>
    <td width="100%" bgcolor="#E8F5FF">
      <a href="../../../boost/endian/conversion.hpp">&lt;boost/endian/conversion.hpp&gt;</a><br>
      <a href="../../../boost/endian/types.hpp">&lt;boost/endian/types.hpp&gt;</a></td>
  </tr>
</table>
<h2><a name="Abstract">Abstract</a></h2>

<p>Boost.Endian provides two facilities to manipulate the byte ordering of integers.</p>
<ul>
  <li>The primary use case is binary I/O of integers for portable exchange with 
  other systems, via either file or network transmission.<br>
&nbsp;</li>
  <li>A secondary use case is minimizing storage size via integers of sizes 
  and/or alignments not supported by the built-in types. Integers 1, 2, 3, 4, 5, 
  6, 7, and 8 bytes in length are supported.<br>
&nbsp;</li>
  <li>Two distinct approaches to byte ordering are provided. Each approach has a 
  long history of successful use, and each approach has use cases where it is 
  superior to the other approach.</li>
</ul>

<h2><a name="Introduction-to-endianness">Introduction to endianness</a></h2>

<p>Consider the following code:</p>

<blockquote>
  <pre>int16_t i = 0x0102;
FILE * file = fopen(&quot;test.bin&quot;, &quot;wb&quot;);   // MUST BE BINARY
fwrite(&amp;i, sizeof(int16_t), 1, file);
fclose(file);</pre>
</blockquote>
<p>On OS X, Linux, or Windows systems with an Intel CPU, a hex dump 
of the &quot;test.bin&quot; output file produces:</p>
<blockquote>
  <p><code>0201</code></p>
</blockquote>
<p>On OS X systems with a PowerPC CPU, or Solaris systems with a SPARC CPU, a hex dump of the &quot;test.bin&quot; 
output file produces:</p>
<blockquote>
  <p><code>0102</code></p>
</blockquote>
<p>What's happening here is that Intel CPU's order the bytes of an integer with 
the least-significant byte first, while SPARC CPU's place the most-significant 
byte first. Some CPU's, such as the PowerPC, allow the operating system to 
choose which ordering applies.</p>
<p><a name="definition"></a>Most-significant-byte-first ordering is traditionally called &quot;big endian&quot; 
ordering and the least-significant-byte-first is traditionally called 
&quot;little-endian&quot; ordering. The names are derived from
<a href="http://en.wikipedia.org/wiki/Jonathan_Swift" title="Jonathan Swift">
Jonathan Swift</a>'s satirical novel <i>
<a href="http://en.wikipedia.org/wiki/Gulliver's_Travels" title="Gulliver's Travels">
Gulliver’s Travels</a></i>, where rival kingdoms opened their soft-boiled eggs 
at different ends.</p>
<p>See the Wikipedia's
<a href="http://en.wikipedia.org/wiki/Endianness">Endianness</a> article for an 
extensive discussion of endianness.</p>
<p>Except for reading a core dump on little-endian systems, most programmers can 
ignore endianness. But when exchanging binary integers and binary floating point 
values between computer systems with differing endianness, whether by physical file transfer or over a network, programmers have to deal with endianness 
in their code. </p>
<h2><a name="Introduction">Introduction</a> to the Boost.Endian library</h2>

<p>The Boost.Endian library provides two facilities for dealing with endianness.</p>

<p>The library provides two approaches to dealing with integer endianness:</p>

<blockquote>

<p><b><a href="conversion.html">Endian conversions</a> -</b> The application 
uses the built-in integer and floating point types, and calls the provided 
conversion functions to convert byte ordering as needed. Both mutating and 
non-mutating conversions are supplied, and each comes in unconditional and 
conditional variants. Type <code>long double</code> is not currently supported.</p>

<p><b><a href="types.html">Endian types</a> -</b> The application uses the provided endian types 
which mimic the 
built-in integer types. For example, <code>big32_t</code> or <code>little64_t</code>. Types with lengths of 
1-8 bytes are supported, rather than just&nbsp; 2, 4, and 8 bytes. There are no alignment 
requirements. Floating point types are not currently supported.</p>

</blockquote>

<p>Boost Endian is a header-only library.</p>

<h2><a name="Choosing">Choosing</a> between endian types and endian 
conversion functions</h2>

<p>Which approach is best for dealing with endianness depends on  
application concerns.</p>

<table border="1" cellpadding="5" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111">
  <tr>
    <th colspan="2">Needs that favor one approach over the other</th>
  </tr>
  <tr>
    <th width="50%"><b><a href="types.html">Endian types</a></b></th>
    <th><b><a href="conversion.html">Endian conversion functions</a></b></th>
  </tr>
  <tr>
    <td valign="top">
    <ul>
      <li>A need to simplify program logic and eliminate logic 
      errors. Since the endian types mimic the built-in types, there is no need to reason about the current endianness of variables 
      and that can simplify program logic and eliminate logic errors.<br>
&nbsp;</li>
      <li>A need to use unusual integer sizes (i.e. 3, 5, 
      6, or 7 bytes) to reduce internal and external space usage and 
      save I/O time.<br>
&nbsp;</li>
      <li>A need to use unaligned variables. Endian types can eliminate padding bytes in 
      structures, reducing internal and external space usage and saving I/O 
      time. They can deals with structures defined like this:</li>
    </ul>
    <blockquote>
      <p><code>struct S {<br>
      &nbsp; uint16_t a;<br>
      &nbsp; uint32_t b;<br>
      } __attribute__ ((packed));</code></p>
    </blockquote>
    </td>
    <td valign="top">
    <ul>
      <li>A need to leverage knowledge of developers who have been using C byte 
      swapping 
      functions for years.<br>
&nbsp;</li>
      <li>A need to save CPU time when a variable is used many times 
      relative to its I/O.<br>
&nbsp;</li>
      <li>A need to pass structures to third-party libraries expecting a 
      specific structure format.<br>
&nbsp;</li>
    </ul>
    </td>
  </tr>
</table>

<h2>Timing tests</h2>
<p>These tests were run against release builds on a circa 2012 4-core little endian X64 Intel Core i5-3570K 
CPU @ 3.40GHz under Windows 7.</p>

<p>See <a href="../test/speed_test.cpp">speed_test.cpp</a>, 
<a href="../test/speed_test_functions.hpp">speed_test_functions.hpp</a>, 
<a href="../test/speed_test_functions.cpp">speed_test_functions.cpp</a>, and 
<a href="../build/Jamfile.v2">Jamfile.v2</a> for the actual code and build. The timed functions are in a separate 
compilation unit to prevent being optimized away.</p>

<p>Because the timings are anomalous, particularly for those high-lighted below 
in yellow, the generated code from the GNU compiler was studied in detail. <b>
Exactly the same code is being generated for by-value conversion functions, 
in-place conversion functions, and the endian types. Exactly the same code is 
being generated whether intrinsics are used or not for 32 and 64-bit tests.</b> 
(For GCC 4.7, there are no 16-bit intrinsics, so they are emulated by using 
32-bit intrinsics.)</p>

<table border="1" cellpadding="5" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111">
  <tr>
    <td bgcolor="#D7EEFF">
    <p align="center"><b>Conclusions</b></p>
    <p>The decision to use endian types or endian conversion functions should be 
    made based on application use cases, not assumptions about generated code 
    efficiency. Modern optimizers generate the same code for either approach, 
    and whether or not intrinsics are available.&nbsp; </td>
  </tr>
</table>

<table border="1" cellpadding="5" cellspacing="0"style="border-collapse: collapse" bordercolor="#111111">
<tr><td colspan="6" align="center"><b>GNU g++ version 4.7.0</b></td></tr>
<tr><td colspan="6" align="center"><b> Iterations: 1,000,000,000, Intrinsics: __builtin_bswap16, etc.</b></td></tr>
<tr><td><b>Test Case</b></td>
<td align="center"><b>int<br>arg</b></td>
<td align="center"><b>int<br>value(arg)</b></td>
<td align="center"><b>int<br>in place(arg)</b></td>
<td align="center"><b>Endian<br>arg</b></td>
</tr>
<tr><td>16-bit aligned big endian</td><td align="right" bgcolor="#FFFFCC">2.71 s</td>
  <td align="right">2.42 s</td><td align="right">2.42 s</td><td align="right">2.68 s</td></tr>
<tr><td>16-bit aligned little endian</td><td align="right">2.42 s</td>
  <td align="right">2.40 s</td><td align="right">2.68 s</td><td align="right">2.45 s</td></tr>
<tr><td>32-bit aligned big endian</td><td align="right">2.68 s</td>
  <td align="right">2.70 s</td><td align="right">2.70 s</td><td align="right">2.68 s</td></tr>
<tr><td>32-bit aligned little endian</td><td align="right">2.68 s</td>
  <td align="right">2.68 s</td><td align="right">2.65 s</td><td align="right">2.68 s</td></tr>
<tr><td>64-bit aligned big endian</td><td align="right" bgcolor="#FFFFCC">2.96 s</td>
  <td align="right" bgcolor="#FFFFCC">2.95 s</td>
  <td align="right" bgcolor="#FFFFCC">2.95 s</td>
  <td align="right" bgcolor="#FFFFCC">2.95 s</td></tr>
<tr><td>64-bit aligned little endian</td><td align="right">2.42 s</td>
  <td align="right">2.40 s</td><td align="right">2.70 s</td><td align="right">2.42 s</td></tr>
<tr><td colspan="6" align="center"><b> Iterations: 1,000,000,000, Intrinsics: no byte swap intrinsics</b></td></tr>
<tr><td><b>Test Case</b></td>
<td align="center"><b>int<br>arg</b></td>
<td align="center"><b>int<br>value(arg)</b></td>
<td align="center"><b>int<br>in place(arg)</b></td>
<td align="center"><b>Endian<br>arg</b></td>
</tr>
<tr><td>16-bit aligned big endian</td><td align="right" bgcolor="#FFFFCC">2.71 s</td>
  <td align="right">2.42 s</td><td align="right">2.42 s</td><td align="right">2.68 s</td></tr>
<tr><td>16-bit aligned little endian</td><td align="right">2.42 s</td>
  <td align="right">2.40 s</td><td align="right">2.68 s</td><td align="right">2.42 s</td></tr>
<tr><td>32-bit aligned big endian</td><td align="right">2.68 s</td>
  <td align="right">2.70 s</td><td align="right">2.67 s</td><td align="right">2.70 s</td></tr>
<tr><td>32-bit aligned little endian</td><td align="right">2.68 s</td>
  <td align="right">2.67 s</td><td align="right">2.70 s</td><td align="right">2.67 s</td></tr>
<tr><td>64-bit aligned big endian</td><td align="right" bgcolor="#FFFFCC">2.96 s</td>
  <td align="right" bgcolor="#FFFFCC">2.95 s</td>
  <td align="right" bgcolor="#FFFFCC">2.95 s</td>
  <td align="right" bgcolor="#FFFFCC">2.93 s</td></tr>
<tr><td>64-bit aligned little endian</td><td align="right">2.42 s</td>
  <td align="right">2.42 s</td><td align="right">2.67 s</td><td align="right">2.40 s</td></tr>

</table>
<p></p>

<table border="1" cellpadding="5" cellspacing="0"style="border-collapse: collapse" bordercolor="#111111">
<tr><td colspan="6" align="center"><b>Microsoft Visual C++ version 11.0</b></td></tr>
<tr><td colspan="6" align="center"><b> Iterations: 1,000,000,000, Intrinsics: cstdlib _byteswap_ushort, etc.</b></td></tr>
<tr><td><b>Test Case</b></td>
<td align="center"><b>int<br>arg</b></td>
<td align="center"><b>int<br>value(arg)</b></td>
<td align="center"><b>int<br>in place(arg)</b></td>
<td align="center"><b>Endian<br>arg</b></td>
</tr>
<tr><td>16-bit aligned big endian</td><td align="right">1.90 s</td>
  <td align="right">1.87 s</td><td align="right">1.89 s</td><td align="right">1.87 s</td></tr>
<tr><td>16-bit aligned little endian</td><td align="right">1.89 s</td>
  <td align="right">1.87 s</td><td align="right">1.89 s</td><td align="right">1.87 s</td></tr>
<tr><td>32-bit aligned big endian</td><td align="right">1.89 s</td>
  <td align="right">1.87 s</td><td align="right">1.89 s</td><td align="right">1.87 s</td></tr>
<tr><td>32-bit aligned little endian</td><td align="right">1.89 s</td>
  <td align="right">1.87 s</td><td align="right">1.87 s</td><td align="right">1.89 s</td></tr>
<tr><td>64-bit aligned big endian</td><td align="right">1.87 s</td>
  <td align="right">1.89 s</td><td align="right">1.87 s</td><td align="right">1.89 s</td></tr>
<tr><td>64-bit aligned little endian</td><td align="right">1.87 s</td>
  <td align="right">1.87 s</td><td align="right">1.87 s</td><td align="right">1.89 s</td></tr>
<tr><td colspan="6" align="center"><b> Iterations: 1,000,000,000, Intrinsics: no byte swap intrinsics</b></td></tr>
<tr><td><b>Test Case</b></td>
<td align="center"><b>int<br>arg</b></td>
<td align="center"><b>int<br>value(arg)</b></td>
<td align="center"><b>int<br>in place(arg)</b></td>
<td align="center"><b>Endian<br>arg</b></td>
</tr>
<tr><td>16-bit aligned big endian</td><td align="right">1.90 s</td>
  <td align="right">1.89 s</td><td align="right">1.87 s</td><td align="right">1.87 s</td></tr>
<tr><td>16-bit aligned little endian</td><td align="right">1.89 s</td>
  <td align="right">1.87 s</td><td align="right">1.89 s</td><td align="right">1.87 s</td></tr>
<tr><td>32-bit aligned big endian</td><td align="right">1.89 s</td>
  <td align="right">1.87 s</td><td align="right">1.87 s</td><td align="right">1.89 s</td></tr>
<tr><td>32-bit aligned little endian</td><td align="right">1.87 s</td>
  <td align="right">1.89 s</td><td align="right">1.87 s</td><td align="right">1.89 s</td></tr>
<tr><td>64-bit aligned big endian</td><td align="right" bgcolor="#FFFFCC">2.32 s</td>
  <td align="right" bgcolor="#FFFFCC">2.46 s</td>
  <td align="right" bgcolor="#FFFFCC">2.45 s</td>
  <td align="right" bgcolor="#FFFFCC">2.34 s</td></tr>
<tr><td>64-bit aligned little endian</td><td align="right">1.87 s</td>
  <td align="right">1.87 s</td><td align="right">1.89 s</td><td align="right">1.87 s</td></tr>

</table>


<h2>Overall <a name="FAQ">FAQ</a></h2>
<p><b>Why bother with endianness?</b></p>
<blockquote>
<p>Binary data portability is the primary use case.</p>
</blockquote>
<p><b>Does endianness have any uses outside of portable binary file or network 
I/O formats?</b> </p>
<blockquote>
<p>Using the unaligned integer types to save internal or external 
memory space is a minor secondary use case.</p>
</blockquote>
<p><b>Why bother with binary I/O? Why not just use C++ Standard Library stream 
inserters and extractors?</b></p>
<blockquote>
<p>Binary arithmetic data is smaller and therefore I/O is faster and file sizes 
are smaller. Transfer between systems is less expensive. Standard interchange 
formats often specify binary arithmetic data.</p>
<p>Furthermore, binary arithmetic data is of fixed size, and so fixed-size disk 
records are possible without padding, easing sorting and allowing direct access. 
Disadvantages, such as the inability to use text utilities on the resulting 
files, limit usefulness to applications where the binary I/O advantages are 
paramount.</p>
</blockquote>

<p><b>Why is only big, little, and native endianness supported?</b></p>
<blockquote>
<p>These are the only endian schemes that have any practical value today. PDP-11 
and the other middle endian approaches are interesting historical curiosities 
but have no relevance to C++ developers.</p>
</blockquote>

<h2><a name="Acknowledgements">Acknowledgements</a></h2>
<p>Comments and suggestions were 
received from
Adder, Benaka Moorthi,
Christopher Kohlhoff,
Cliff Green,Daniel James, Gennaro Proto,
Giovanni Piero Deretta, Gordon Woodhull, dizzy, Hartmut Kaiser, Jeff Flinn,
John Filo, John Maddock,
Kim Barrett,
Marsh Ray,
Martin Bonner, Mathias Gaunard, Matias Capeletto,
Neil Mayhew, Paul Bristow, Phil Endecott, Pyry Jahkola, Rene Rivera,
Robert Stewart, Roland Schwarz, Scott McMurray, 
Sebastian Redl, 
Tim Blechmann, Tim Moore, tymofey, Tomas Puverle, Vincente Botet, Yuval Ronen 
and Vitaly Budovski,.</p>
<hr>
<p>Last revised:
<!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 May, 2013<!--webbot bot="Timestamp" endspan i-checksum="13986" --></p>
<p>© Copyright Beman Dawes, 2011, 2013</p>
<p>Distributed under the Boost Software License, Version 1.0. See
<a href="http://www.boost.org/LICENSE_1_0.txt">www.boost.org/ LICENSE_1_0.txt</a></p>

<p>&nbsp;</p>

</body>

</html>