Add utf7 decoding for Gmail labels

.gitignore

@@ -0,0 +1,3 @@
+.idea
+.vscode
+node_modules

CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+Changes made in this fork. Versions up to and including 0.8.19 come from
+upstream [mscdex/node-imap](https://github.com/mscdex/node-imap) and are not
+listed here.
+
+## 0.8.24
+
+Merged upstream [mscdex/node-imap](https://github.com/mscdex/node-imap) up to
+`9918f08`, which is the last commit there (February 2022), and followed up on
+the two gaps it left behind.
+
+- Fixed TLS connections not sending SNI at all. `tls.connect()` sends the
+  `server_name` extension only when `servername` is set explicitly — node never
+  derives it from `host`, which is used just to dial the connection and to check
+  the certificate identity. Servers that serve several domains from one address
+  (Gmail, Exchange Online, anything behind a load balancer) therefore answered
+  with a default certificate that does not match the requested host, surfacing as
+  `Error: self signed certificate` / `DEPTH_ZERO_SELF_SIGNED_CERT` and forcing
+  callers to pass `tlsOptions.servername` by hand.
+
+  Imported from upstream `9918f08` for `connect()` (implicit TLS) and extended to
+  `_starttls()`, which builds its own `tlsOptions` and stayed uncovered upstream.
+  In both paths the assignment happens before the caller's `tlsOptions` are
+  copied over, so an explicit `tlsOptions.servername` still wins.
+
+  The default is only derived from `host` when `host` is a name.
+  [RFC 6066 §3](https://www.rfc-editor.org/rfc/rfc6066#section-3) does not permit
+  IP addresses in `server_name`, and node warns about them (`DEP0123`, "This will
+  be ignored in a future version"), so defaulting one in would only trade a
+  missing extension for a deprecated one. An explicit `tlsOptions.servername` is
+  still passed through unchanged even when it is an IP — that call belongs to the
+  caller, not to this library.
+
+  References:
+  - [mscdex/node-imap#724](https://github.com/mscdex/node-imap/issues/724) —
+    "Servername option is mandatory with gmail and Openssl 1.1.1", reporting
+    `Error: self signed certificate` against `imap.gmail.com` and the manual
+    `servername` workaround.
+  - [mscdex/node-imap#866](https://github.com/mscdex/node-imap/issues/866) — the
+    same `DEPTH_ZERO_SELF_SIGNED_CERT` symptom, misattributed to a MITM in the
+    upstream discussion.
+  - [RFC 6066 §3](https://www.rfc-editor.org/rfc/rfc6066#section-3) — defines
+    `server_name` and restricts it to host names, which is why the connection
+    host must be a name and not an IP for SNI to be meaningful.
+
+- Removed the dead `require('readable-stream')` fallback in `Parser.js`. Upstream
+  `7dbc664` dropped `readable-stream` from the dependencies but left the
+  `require('stream').Readable || require('readable-stream').Readable` fallback in
+  place. At runtime the `||` short-circuits, but bundlers (webpack, esbuild, ncc)
+  resolve `require()` statically and fail on the missing module.
+
+- Replaced the deprecated `Buffer` constructor with `Buffer.from()` /
+  `Buffer.allocUnsafe()` and the `'binary'` encoding alias with `'latin1'`
+  (upstream `7dbc664`). Both `allocUnsafe()` call sites overwrite the whole
+  buffer with `copy()` before reading it. The same commit dropped the
+  `readable-stream` dependency and raised `engines.node` to `>=10.0.0`.
+
+- Added regression tests that read the TLS handshake off the wire and assert the
+  client really announces the configured host, independent of the TLS version and
+  without certificate fixtures. They cover both the implicit TLS and the STARTTLS
+  path, and pin the IP behaviour down as well: no default for an IP host, but an
+  explicitly configured IP `servername` is still sent.
+
+## 0.8.23
+
+- Fixed `_login()` aborting with `Logging in is disabled on this server` whenever
+  the server advertised `LOGINDISABLED`, even when an XOAUTH/XOAUTH2 token was
+  configured. Per RFC 3501, `LOGINDISABLED` only forbids the `LOGIN` command, so
+  the check now runs in the `LOGIN` branch only and the XOAUTH/XOAUTH2
+  `AUTHENTICATE` branches are attempted first. This unblocks Exchange Online
+  mailboxes, which advertise `LOGINDISABLED` together with `AUTH=XOAUTH2` once
+  basic auth is disabled.
+
+  References:
+  - [RFC 3501 §6.2.3](https://www.rfc-editor.org/rfc/rfc3501#section-6.2.3) —
+    "A client implementation MUST NOT send a LOGIN command if the LOGINDISABLED
+    capability is advertised." The restriction is scoped to `LOGIN`; nothing in
+    it applies to `AUTHENTICATE`.
+  - [Deprecation of Basic authentication in Exchange Online](https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online)
+    — Microsoft "removed the ability to use Basic authentication in Exchange
+    Online for […] POP, IMAP […]" and it "is now disabled in all tenants",
+    with no way for admins or support to re-enable it. Such mailboxes answer
+    `CAPABILITY` with `LOGINDISABLED` and without `AUTH=PLAIN`, leaving
+    `AUTH=XOAUTH2` as the only usable mechanism.
+  - [Authenticate an IMAP, POP, or SMTP connection using OAuth](https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth)
+    — the XOAUTH2 flow Exchange Online expects instead.
+  - [mscdex/node-imap#685](https://github.com/mscdex/node-imap/issues/685) —
+    the same symptom reported upstream against Microsoft Exchange IMAP4.
+
+## 0.8.22
+
+- Fixed `TypeError: list[i].toLowerCase is not a function` in `parseFetch()` when
+  a server emitted an integer-shaped token in a key position. Numeric values
+  (`RFC822.SIZE`, `MODSEQ`, `UIDNEXT`, …) are still parsed as numbers.
+
+## 0.8.21
+
+- Published the fork as the scoped npm package `@integromat/imap`.
+
+## 0.8.20
+
+- Added UTF-7 decoding of Gmail labels (`X-GM-LABELS`).

lib/Connection.js

@@ -1,5 +1,6 @@
 var tls = require('tls'),
     Socket = require('net').Socket,
+    isIP = require('net').isIP,
     EventEmitter = require('events').EventEmitter,
     inherits = require('util').inherits,
     inspect = require('util').inspect,
@@ -117,10 +118,18 @@ Connection.prototype.connect = function() {
 
   if (config.tls) {
     tlsOptions = {};
-    // servername must be set to prevent issues with some imap server and openssl 1.1.1
-    tlsOptions.servername = config.host;
+    if (!isIP(config.host)) {
+      // tls.connect() sends SNI only when servername is set -- it never derives it
+      // from host. Without it, servers that answer for several domains on one
+      // address hand back a default certificate that does not match the requested
+      // host. See mscdex/node-imap#724.
+      //
+      // Note: RFC 6066 does not permit IP addresses in SNI, so only default it for host
+      // names -- an explicit tlsOptions.servername below still wins either way.
+      tlsOptions.servername = config.host;
+    }
     tlsOptions.host = config.host;
-    // Host name may be overridden the tlsOptions
+    // Host name may be overridden by the tlsOptions
     for (var k in config.tlsOptions)
       tlsOptions[k] = config.tlsOptions[k];
     tlsOptions.socket = socket;
@@ -1451,7 +1460,7 @@ Connection.prototype._resUntagged = function(info) {
                 if (toget[i] === 'X-GM-LABELS') {
                   var labels = info.text[keys[j]];
                   for (var k = 0, lenk = labels.length; k < lenk; ++k)
-                    labels[k] = (''+labels[k]).replace(RE_ESCAPE, '\\');
+                    labels[k] = utf7.decode((''+labels[k]).replace(RE_ESCAPE, '\\'));
                 }
                 key = FETCH_ATTR_MAP[toget[i]];
                 if (!key)
@@ -1654,12 +1663,6 @@ Connection.prototype._login = function() {
           return;
       }
 
-      if (self.serverSupports('LOGINDISABLED')) {
-        err = new Error('Logging in is disabled on this server');
-        err.source = 'authentication';
-        return reentry(err);
-      }
-
       var cmd;
       if (self.serverSupports('AUTH=XOAUTH') && self._config.xoauth) {
         self._caps = undefined;
@@ -1675,6 +1678,18 @@ Connection.prototype._login = function() {
           cmd += ' ' + escape(self._config.xoauth2);
         self._enqueue(cmd, checkCaps);
       } else if (self._config.user && self._config.password) {
+        if (self.serverSupports('LOGINDISABLED')) {
+          // LOGINDISABLED (RFC 3501) only forbids the LOGIN command -- SASL
+          // mechanisms advertised as AUTH=* stay available. Servers that disable
+          // basic auth (e.g. Exchange Online) advertise LOGINDISABLED together
+          // with AUTH=XOAUTH2, so this check must not run before the AUTHENTICATE
+          // branches above.
+
+          err = new Error('Logging in is disabled on this server');
+          err.source = 'authentication';
+          return reentry(err);
+        }
+
         self._caps = undefined;
         self._enqueue('LOGIN "' + escape(self._config.user) + '" "'
                       + escape(self._config.password) + '"', checkCaps);
@@ -1702,8 +1717,11 @@ Connection.prototype._starttls = function() {
 
     var tlsOptions = {};
 
+    if (!isIP(this._config.host)) {
+      tlsOptions.servername = this._config.host;
+    }
     tlsOptions.host = this._config.host;
-    // Host name may be overridden the tlsOptions
+    // Host name may be overridden by the tlsOptions
     for (var k in this._config.tlsOptions)
       tlsOptions[k] = this._config.tlsOptions[k];
     tlsOptions.socket = self._sock;

lib/Parser.js

@@ -1,6 +1,5 @@
 var EventEmitter = require('events').EventEmitter,
-    ReadableStream = require('stream').Readable
-                     || require('readable-stream').Readable,
+    ReadableStream = require('stream').Readable,
     inherits = require('util').inherits,
     inspect = require('util').inspect;
 
@@ -428,7 +427,12 @@ function parseFetch(text, literals, seqno) {
   var list = parseExpr(text, literals)[0], attrs = {}, m, body;
   // list is [KEY1, VAL1, KEY2, VAL2, .... KEYn, VALn]
   for (var i = 0, len = list.length, key, val; i < len; i += 2) {
-    key = list[i].toLowerCase();
+    // Coerce the key token to a string before lowercasing. Some servers send
+    // integer-shaped atoms at key positions in malformed FETCH responses;
+    // convStr returns a JS number for those, and Number has no .toLowerCase.
+    // Coercing here keeps value-position numeric atoms (RFC822.SIZE, MODSEQ,
+    // etc.) untouched.
+    key = String(list[i]).toLowerCase();
     val = list[i + 1];
     if (key === 'envelope')
       val = parseFetchEnvelope(val);

package.json

@@ -1,6 +1,6 @@
-{ "name": "imap",
-  "version": "0.8.19",
-  "author": "Brian White <mscdex@mscdex.net>",
+{ "name": "@integromat/imap",
+  "version": "0.8.24",
+  "author": "Brian White <mscdex@mscdex.net>, Petr Malimanek, Oleksandr Hushcha, Martin Zaloudek",
   "description": "An IMAP module for node.js that makes communicating with IMAP servers easy",
   "main": "./lib/Connection",
   "dependencies": {
@@ -12,5 +12,5 @@
   "engines": { "node": ">=10.0.0" },
   "keywords": [ "imap", "mail", "email", "reader", "client" ],
   "licenses": [ { "type": "MIT", "url": "http://github.com/mscdex/node-imap/raw/master/LICENSE" } ],
-  "repository": { "type": "git", "url": "http://github.com/mscdex/node-imap.git" }
+  "repository": { "type": "git", "url": "https://github.com/integromat/node-imap.git" }
 }

test/clienthello.js

@@ -0,0 +1,66 @@
+// Minimal TLS ClientHello reader used by the SNI tests. Working on the raw
+// handshake bytes keeps the tests independent of the TLS/OpenSSL version the
+// running node happens to be linked against: the server_name extension is sent
+// in the clear by every TLS version, so a plain TCP server can observe it.
+//
+// Returns the requested host name, or null when the buffer holds no readable
+// server_name extension -- including when it is truncated or malformed, which a
+// reader of network data has to survive without throwing.
+
+// Bounds-checked reads. Both yield -1 instead of reading past the end, so a
+// short buffer ends the walk instead of raising RangeError.
+function u8(buf, p) {
+  return (p >= 0 && p < buf.length) ? buf[p] : -1;
+}
+
+function u16(buf, p) {
+  return (p >= 0 && p + 2 <= buf.length) ? buf.readUInt16BE(p) : -1;
+}
+
+function parseSNI(buf) {
+  // TLSPlaintext: type(1) version(2) length(2), then Handshake: type(1) len(3)
+  if (buf.length < 45 || buf[0] !== 0x16 || buf[5] !== 0x01)
+    return null;
+
+  let p = 5 + 4 + 2 + 32; // handshake header + client_version + random
+
+  const sessionId = u8(buf, p); // legacy_session_id
+  if (sessionId < 0)
+    return null;
+  p += 1 + sessionId;
+
+  const ciphers = u16(buf, p); // cipher_suites
+  if (ciphers < 0)
+    return null;
+  p += 2 + ciphers;
+
+  const compression = u8(buf, p); // legacy_compression_methods
+  if (compression < 0)
+    return null;
+  p += 1 + compression;
+
+  const extensions = u16(buf, p);
+  if (extensions < 0)
+    return null;
+  const end = Math.min(p + 2 + extensions, buf.length);
+  p += 2;
+
+  while (p + 4 <= end) {
+    const type = buf.readUInt16BE(p);
+    const len = buf.readUInt16BE(p + 2);
+    if (type === 0x0000) { // server_name
+      // ServerNameList: list length(2), then name_type(1) + length(2) + name
+      const q = p + 4 + 2;
+      if (u8(buf, q) !== 0x00) // not a host_name entry, or out of bounds
+        return null;
+      const nlen = u16(buf, q + 1);
+      if (nlen < 0 || q + 3 + nlen > buf.length)
+        return null;
+      return buf.toString('ascii', q + 3, q + 3 + nlen);
+    }
+    p += 4 + len;
+  }
+  return null;
+}
+
+module.exports = { parseSNI };