Boost Endian Library

Endian Library

Boost Home Endian Home Conversion Functions Endian Types Tutorial

Contents

Abstract
Introduction to endianness
Introduction to the Boost.Endian library
Acknowledgements

Headers

Abstract

Boost.Endian provides two facilities to manipulate the byte ordering of integers.

The primary use case is binary I/O of integers for portable exchange with other systems, via either file or network transmission.
A secondary use case is minimizing storage size via integers of sizes and/or alignments not supported by the built-in types. Integers 1, 2, 3, 4, 5, 6, 7, and 8 bytes in length are supported.
Two distinct approaches to byte ordering are provided. Each approach has a long history of successful use, and each approach has use cases where it is superior to the other approach.

Introduction to endianness

Consider the following code:

int16_t i = 0x0102;
FILE * file = fopen("test.bin", "wb");   // MUST BE BINARY
fwrite(&i, sizeof(int16_t), 1, file);
fclose(file);

On OS X, Linux, or Windows systems with an Intel CPU, a hex dump of the "test.bin" output file produces:

0201

On OS X systems with a PowerPC CPU, or Solaris systems with a SPARC CPU, a hex dump of the "test.bin" output file produces:

0102

What's happening here is that Intel CPU's order the bytes of an integer with the least-significant byte first, while SPARC CPU's place the most-significant byte first. Some CPU's, such as the PowerPC, allow the operating system to choose which ordering applies.

Most-significant-byte-first ordering is traditionally called "big endian" ordering and the least-significant-byte-first is traditionally called "little-endian" ordering. The names are derived from Jonathan Swift's satirical novel Gulliver’s Travels, where rival kingdoms opened their soft-boiled eggs at different ends.

See the Wikipedia's Endianness article for an extensive discussion of endianness.

Except for reading a core dump on little-endian systems, most programmers can ignore endianness. But when exchanging binary integers and binary floating point values between computer systems with differing endianness, whether by physical file transfer or over a network, programmers have to deal with endianness in their code.

Introduction to the Boost.Endian library

The Boost.Endian library provides two facilities for dealing with endianness.

The library provides two approaches to dealing with integer endianness:

Endian conversions - The application uses the built-in integer and floating point types, and calls the provided conversion functions to convert byte ordering as needed. Both mutating and non-mutating conversions are supplied, and each comes in unconditional and conditional variants. Type long double is not currently supported.

Endian types - The application uses the provided endian types which mimic the built-in integer types. For example, big32_t or little64_t. Types with lengths of 1-8 bytes are supported, rather than just 2, 4, and 8 bytes. There are no alignment requirements. Floating point types are not currently supported.

Boost Endian is a header-only library.

Choosing between endian types and endian conversion functions

Which approach is best for dealing with endianness depends on application concerns.

Needs that favor one approach over the other

Needs that favor one approach over the other
Endian types	Endian conversion functions
A need to simplify program logic and eliminate logic errors. Since the endian types mimic the built-in types, there is no need to reason about the current endianness of variables and that can simplify program logic and eliminate logic errors. A need to use unusual integer sizes (i.e. 3, 5, 6, or 7 bytes) to reduce internal and external space usage and save I/O time. A need to use unaligned variables. Endian types can eliminate padding bytes in structures, reducing internal and external space usage and saving I/O time. They can deals with structures defined like this: `struct S { uint16_t a; uint32_t b; } __attribute__ ((packed));`	A need to leverage knowledge of developers who have been using C byte swapping functions for years. A need to save CPU time when a variable is used many times relative to its I/O. A need to pass structures to third-party libraries expecting a specific structure format.

Endian types

Endian conversion functions

A need to simplify program logic and eliminate logic errors. Since the endian types mimic the built-in types, there is no need to reason about the current endianness of variables and that can simplify program logic and eliminate logic errors.
A need to use unusual integer sizes (i.e. 3, 5, 6, or 7 bytes) to reduce internal and external space usage and save I/O time.
A need to use unaligned variables. Endian types can eliminate padding bytes in structures, reducing internal and external space usage and saving I/O time. They can deals with structures defined like this:

struct S { uint16_t a; uint32_t b; } __attribute__ ((packed));

A need to leverage knowledge of developers who have been using C byte swapping functions for years.
A need to save CPU time when a variable is used many times relative to its I/O.
A need to pass structures to third-party libraries expecting a specific structure format.

Timing tests

These tests were run against release builds on a circa 2012 4-core little endian X64 Intel Core i5-3570K CPU @ 3.40GHz under Windows 7.

See speed_test.cpp, speed_test_functions.hpp, speed_test_functions.cpp, and Jamfile.v2 for the actual code and build. The timed functions are in a separate compilation unit to prevent being optimized away.

Because the timings are anomalous, particularly for those high-lighted below in yellow, the generated code from the GNU compiler was studied in detail. Exactly the same code is being generated for by-value conversion functions, in-place conversion functions, and the endian types. Exactly the same code is being generated whether intrinsics are used or not for 32 and 64-bit tests. (For GCC 4.7, there are no 16-bit intrinsics, so they are emulated by using 32-bit intrinsics.)

Conclusions

The decision to use endian types or endian conversion functions should be made based on application use cases, not assumptions about generated code efficiency. Modern optimizers generate the same code for either approach, and whether or not intrinsics are available.

GNU g++ version 4.7.0
Iterations: 1,000,000,000, Intrinsics: __builtin_bswap16, etc.
Test Case	int arg	int value(arg)	int in place(arg)	Endian arg
16-bit aligned big endian	2.71 s	2.42 s	2.42 s	2.68 s
16-bit aligned little endian	2.42 s	2.40 s	2.68 s	2.45 s
32-bit aligned big endian	2.68 s	2.70 s	2.70 s	2.68 s
32-bit aligned little endian	2.68 s	2.68 s	2.65 s	2.68 s
64-bit aligned big endian	2.96 s	2.95 s	2.95 s	2.95 s
64-bit aligned little endian	2.42 s	2.40 s	2.70 s	2.42 s
Iterations: 1,000,000,000, Intrinsics: no byte swap intrinsics
Test Case	int arg	int value(arg)	int in place(arg)	Endian arg
16-bit aligned big endian	2.71 s	2.42 s	2.42 s	2.68 s
16-bit aligned little endian	2.42 s	2.40 s	2.68 s	2.42 s
32-bit aligned big endian	2.68 s	2.70 s	2.67 s	2.70 s
32-bit aligned little endian	2.68 s	2.67 s	2.70 s	2.67 s
64-bit aligned big endian	2.96 s	2.95 s	2.95 s	2.93 s
64-bit aligned little endian	2.42 s	2.42 s	2.67 s	2.40 s

Microsoft Visual C++ version 11.0
Iterations: 1,000,000,000, Intrinsics: cstdlib _byteswap_ushort, etc.
Test Case	int arg	int value(arg)	int in place(arg)	Endian arg
16-bit aligned big endian	1.90 s	1.87 s	1.89 s	1.87 s
16-bit aligned little endian	1.89 s	1.87 s	1.89 s	1.87 s
32-bit aligned big endian	1.89 s	1.87 s	1.89 s	1.87 s
32-bit aligned little endian	1.89 s	1.87 s	1.87 s	1.89 s
64-bit aligned big endian	1.87 s	1.89 s	1.87 s	1.89 s
64-bit aligned little endian	1.87 s	1.87 s	1.87 s	1.89 s
Iterations: 1,000,000,000, Intrinsics: no byte swap intrinsics
Test Case	int arg	int value(arg)	int in place(arg)	Endian arg
16-bit aligned big endian	1.90 s	1.89 s	1.87 s	1.87 s
16-bit aligned little endian	1.89 s	1.87 s	1.89 s	1.87 s
32-bit aligned big endian	1.89 s	1.87 s	1.87 s	1.89 s
32-bit aligned little endian	1.87 s	1.89 s	1.87 s	1.89 s
64-bit aligned big endian	2.32 s	2.46 s	2.45 s	2.34 s
64-bit aligned little endian	1.87 s	1.87 s	1.89 s	1.87 s

Overall FAQ

Why bother with endianness?

Binary data portability is the primary use case.

Does endianness have any uses outside of portable binary file or network I/O formats?

Using the unaligned integer types to save internal or external memory space is a minor secondary use case.

Why bother with binary I/O? Why not just use C++ Standard Library stream inserters and extractors?

Binary arithmetic data is smaller and therefore I/O is faster and file sizes are smaller. Transfer between systems is less expensive. Standard interchange formats often specify binary arithmetic data.

Furthermore, binary arithmetic data is of fixed size, and so fixed-size disk records are possible without padding, easing sorting and allowing direct access. Disadvantages, such as the inability to use text utilities on the resulting files, limit usefulness to applications where the binary I/O advantages are paramount.

Why is only big, little, and native endianness supported?

These are the only endian schemes that have any practical value today. PDP-11 and the other middle endian approaches are interesting historical curiosities but have no relevance to C++ developers.

Acknowledgements

Comments and suggestions were received from Adder, Benaka Moorthi, Christopher Kohlhoff, Cliff Green,Daniel James, Gennaro Proto, Giovanni Piero Deretta, Gordon Woodhull, dizzy, Hartmut Kaiser, Jeff Flinn, John Filo, John Maddock, Kim Barrett, Marsh Ray, Martin Bonner, Mathias Gaunard, Matias Capeletto, Neil Mayhew, Paul Bristow, Phil Endecott, Pyry Jahkola, Rene Rivera, Robert Stewart, Roland Schwarz, Scott McMurray, Sebastian Redl, Tim Blechmann, Tim Moore, tymofey, Tomas Puverle, Vincente Botet, Yuval Ronen and Vitaly Budovski,.

Last revised: 25 May, 2013

Distributed under the Boost Software License, Version 1.0. See www.boost.org/ LICENSE_1_0.txt