This will save us toLatin1() conversations in CppTools (which already
holds UTF-8 encoded QByteArrays) and thus loss of information (see
QTCREATORBUG-7356). It also gives us support for non-latin1 identifiers.
API-wise the following functions are added to Token. In follow-up
patches these will become handy in combination with QStrings.
utf16chars() - aequivalent of bytes()
utf16charsBegin() - aequivalent of bytesBegin()
utf16charsEnd() - aequivalent of bytesEnd()
Next steps:
* Adapt functions from TranslationUnit. They should work with utf16
chars in order to calculate lines and columns correctly also for
UTF-8 multi-byte code points.
* Adapt the higher level clients:
* Cpp{Tools,Editor} should expect UTF-8 encoded Literals.
* Cpp{Tools,Editor}: When dealing with identifiers on the
QString/QTextDocument layer, code points
represendet by two QChars need to be respected, too.
* Ensure Macro::offsets() and Document::MacroUse::{begin,end}() report
offsets usable in CppEditor/CppTools.
Addresses QTCREATORBUG-7356.
Change-Id: I0791b5236be8215d24fb8e38a1f7cb0d279454c0
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>
The necessary data can be retrieved by the resulting Token.
Change-Id: I79afb23183c156240c690beff30bb11dfe943e61
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>
Also fix false positive line continuation on blank line
e.g.
"foo \
bar"
Change-Id: Ic6d345a4b578c955411d119b8438c8dc5065c072
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>
... by assuming we operate on NUL-terminated data, which is
(in theory) guaranteed by (non-raw) QByteArray which we have.
Change-Id: I855d01ea0dee5328ec737fbabee1086d7a28aa5a
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>
Although they are now supported by the lexer
and parser, it is worth to remind that we still
need to address an issue concerning the highlight
of multiline literals (which with the advent of
the new raw strings will become more common).
Task-number: QTCREATORBUG-6722
Change-Id: I137337a9ac0152a1f8b9faded0b960c6fe3dd38a
Reviewed-by: Roberto Raggi <roberto.raggi@nokia.com>
This makes things slightly more efficient. But it will be more
significant when we introduce R"rawliterals" since we would avoid
an even further lookahead for cases like u8R"string".
Change-Id: Id4bad8b917752d23daf2f4989330434979cf602f
Reviewed-by: Roberto Raggi <roberto.raggi@nokia.com>
Reviewed-by: hjk <qthjk@ovi.com>
Those are the types char16_t and char32_t along with the new
char/string literals u'', U'', u"", u8"", and U"".
This is particularly important for the use of QStringLiteral
since in some platforms it relies on expansion such as above.
Note: The string literals quickfixes still need some tunning.
Task-number: QTCREATORBUG-7449
Change-Id: Iebcfea15677dc8e0ebb6143def89a5477e1be7d4
Reviewed-by: hjk <qthjk@ovi.com>
they are lying. nokia has no copyright on this code. and the double
license in a single file looks weird. that's why we moved it to
3rdparty/, so it is clear it is not nokia's.
Approved-by: legal
