Improved Iterator Categories and Requirements

Introduction

Examples of useful iterators that do not ``fit''

Because of the mixing of iterator traversal and dereference return type, many useful iterators can not be appropriately categorized. For example, vector<bool>::iterator is almost a random access iterator, but the return type is not bool& (see issue 96 and Herb Sutter's paper J16/99-0008 = WG21 N1185). Therefore, the iterators only meet the requirements of input iterator and output iterator. This is so nonintuitive that at least one implementation erroneously assigns random_access_iterator_tag as its iterator_category. Also, vector<bool> is not the only example of useful iterators that do not return true references: there is the often cited example of disk-based collections.

Another example is a counting iterator, an iterator the returns a sequence of integers when incremented and dereferenced (see boost::counting_iterator). There are two ways to implement this iterator, 1) make the reference type be a true reference (a reference to an integer data member of the counting iterator) or 2) make the reference type be the same as the value_type. Option 1) runs into the problems discussed in Issue 198, the reference will not be valid after the iterator is destroyed. Option 2) is therefore a better choice, but then we have a counting iterator that cannot be a random access iterator.

Yet another example is a transform iterator, an iterator adaptor that applies a unary function object to the dereference value of the wrapped iterator (see boost::transform_iterator). For unary functions such as std::times the return type of operator* clearly needs to be the result_type of the function object, which is typically not a reference. However, with the current iterator requirements, if you wrap int* with a transform iterator, you do not get a random access iterator as expected, but an input iterator.

A fourth example is found in the vertex and edge iterators of the Boost Graph Library. These iterators return vertex and edge descriptors, which are lightweight handles created on-the-fly. They must be returned by-value. As a result, their current standard iterator category is std::input_iterator_tag, which means that, strictly speaking, you could not use these iterators with algorithms like std::min_element(). As a temporary solution, we introduced the concept Multi-Pass Input Iterator to describe the vertex and edge descriptors, but as the design notes for concept suggest, a better solution is needed.

In short, there are many useful iterators that do not fit into the current standard iterator categories. As a result, the following bad things happen:

• Iterators are often miss-categorized.
• Algorithm requirements are more strict than necessary, because they can not separate out the need for random-access from the need for a true reference return type.

Proposal for new iterator categories and requirements

• Readable Iterator
• Writable Iterator
• Swappable Iterator
• Constant Lvalue Iterator
• Mutable Lvalue Iterator

• Forward Traversal Iterator
• Bidirectional Traversal Iterator
• Random Access Traversal Iterator

The current Input Iterator and Output Iterator requirements will continue to be used as is. Note that Input Iterator implies Readable Iterator and Output Iterator implies Writable Iterator.

Note: we considered defining a Single-Pass Iterator, which could be combined with Readable or Writable Iterator to replace the Input and Output Iterator requirements. We rejected this idea because there are several differences between Input and Output Iterators that make it hard to merge them: Input Iterator requires Equality Comparable while Output Iterator does not and Input Iterator requires Assignable while Output Iterator does not.

New category tags and traits classes

The new iterator categories will require new tag classes.

namespace std {

  // Returns Category Tags
  struct readable_iterator_tag { };
  struct writable_iterator_tag { };
  struct swappable_iterator_tag { };
  struct mutable_lvalue_iterator_tag : virtual public writable_iterator_tag,
    virtual public readable_iterator_tag { };
  struct constant_lvalue_iterator_tag : public readable_iterator_tag { };

  // Traversal Category Tags
  struct input_traversal_tag { };
  struct output_traversal_tag { };
  struct forward_traversal_tag { };
  struct bidirectional_traversal_tag : public forward_traversal_tag { };
  struct random_access_traversal_tag : public bidirectional_traversal_tag { };

}

Access to the return and traversal tags will be through the following two traits classes, which have a member typedef named type that provides the tag type. We explain the definitions of these classes later.

  template <typename Iterator>
  struct return_category; // contains: typedef ... type;

  template <typename Iterator>
  struct traversal_category; // contains: typedef ... type;

We want it to be convenient for programmers to create iterators that satisfy both the old and new iterator requirements. Therefore the following class is provided as a way to create tags for use as the old iterator_category typedef within iterator_traits.

namespace std {
  template <class ReturnTag, class TraversalTag>
  struct iterator_tag : cvt_iterator_category<ReturnTag, TraversalTag>::type
  {
    typedef ReturnTag returns;
    typedef TraversalTag traversal;
  };

The cvt_iterator_category template computes the appropriate old iterator category based on the return and traversal category.

namespace std {
  template <class RC, class TC>
  struct cvt_iterator_category
  {
    // Pseudo-code, <= means inherits or same type
    if (RC <= constant_lvalue_iterator_tag || RC <= mutable_lvalue_iterator_tag) {
      if (TC <= random_access_traversal_tag)
        typedef random_access_iterator_tag type;
      else if (TC <= bidirectional_traversal_tag)
        typedef bidirectional_iterator_tag type;
      else if (TC <= forward_traversal_tag)
        typedef forward_iterator_tag type;
      else
        error;
   } else if (RC <= readable_iterator_tag && RC <= input_traversal_tag)
     typedef input_iterator_tag type;
   else if (RC <= writable_iterator_tag && output_traversal_tag)
     typedef output_iterator_tag type;
   else
     error;
  };
}

The following is an example of a new iterator class using the iterator_tag class to create its iterator_category member typedef.

struct my_iterator {
  typedef std::iterator_tag<std::readable_iterator_tag, 
    std::random_access_traversal_tag> iterator_category;
  ...
};

  template <typename Iterator>
  class return_category
  {
    // Pseudo-code
    typedef iterator_traits<Iterator>::iterator_category tag;
    typedef iterator_traits<Iterator>::value_type T;
  public:
    if (exists(tag::returns)) // must be a new iterator
      typedef tag::returns type;
    else if (tag <= forward_iterator_tag) {
      if (is-const(T))
        typedef constant_lvalue_iterator_tag type;
      else
        typedef mutable_lvalue_iterator_tag type;
    } else if (tag <= input_iterator_tag)
      typedef readable_iterator_tag type;
    else if (tag <= output_iterator_tag)
      typedef writable_iterator_tag type;
    else
      error;
  };

  template <typename T>
  struct return_category<T*>
  {
    // Pseudo-code
    if (is-const(T))
      typedef boost::constant_lvalue_iterator_tag type;
    else
      typedef boost::mutable_lvalue_iterator_tag type;
  };

  template <typename Iterator>
  class traversal_category
  {
    typedef iterator_traits<Iterator>::iterator_category tag;
  public:
    // Pseudo-code
    if (exists(tag::traversal)) // must be a new iterator
      typedef tag::traversal type;
    else if (tag <= random_access_iterator_tag)
      typedef random_access_traversal_tag type;
    else if (tag <= bidirectional_iterator_tag)
      typedef bidirectional_traversal_tag type;
    else if (tag <= is_forward_iterator_tag)
      typedef forward_traversal_tag type;
    else if (tag <= input_iterator_tag)
      typedef input_traversal_tag type;
    else if (tag <= out_iterator_tag)
      typedef output_traversal_tag type;
    else
      error;
  };

  template <typename T>
  struct traversal_category<T*>
  {
    typedef random_access_traversal_tag type;
  };

Impact on the Standard Algorithms

Many of the standard algorithms place more requirements than necessary on their iterator parameters due to the coarseness of the current iterator categories. By using the new iterator categories a better fit can be achieved, thereby increasing the reusability of the algorithms. These changes will not affect user-code, though they will require changes by standard implementers: dispatching should be based on the new categories, and in places return values may need to be handled more carefully. In particular, uses of std::swap() will need to be replaced with std::iter_swap(), and std::iter_swap() will need to call std::swap().

The New Iterator Requirements

Notation

Readable Iterator

Associated Types

Refinement of

Valid expressions

Writable Iterator

Definitions

Associated Types

Refinement of

Valid expressions

Swappable Iterator

Note: the requirements for Swappable Iterator are dependent on the issues surrounding std::swap() being resolved. Here we assume that the issue will be resolved by allowing the overload of std::swap() for user-defined types.

Note: Readable Iterator and Writable Iterator combined implies Swappable Iterator because of the fully templated std::swap(). However, Swappable Iterator does not imply Readable Iterator nor Writable Iterator.

Associated Types

Valid expressions

Name	Expression	Return type
Iterator Swap	std::iter_swap(x, y)	void
Dereference and Swap	std::swap(*x, *y)	void

Constant Lvalue Iterator

Refinement of

Associated Types

Mutable Lvalue Iterator

Refinement of

Associated Types

Forward Traversal Iterator

Refinement of

Associated types

Valid expressions

Bidirectional Traversal Iterator

Refinement of

Associated types

Valid expressions

Random Access Traversal Iterator

Refinement of

Associated types

Valid expressions