Merge branch 'develop'

Author: webzwo0i

.github/stale.yml

@@ -12,6 +12,7 @@ exemptLabels:
   - Black hole bug
   - Special case Bug
   - Upstream bug
+  - Feature Request
 # Label to use when marking an issue as stale
 staleLabel: wontfix
 # Comment to post when marking an issue as stale. Set to `false` to disable

.github/workflows/backend-tests.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        node: [10, 12, 14, 15]
+        node: [12, 14, 16]
 
     steps:
     - name: Checkout repository
@@ -50,7 +50,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        node: [10, 12, 14, 15]
+        node: [12, 14, 16]
 
     steps:
     - name: Checkout repository

.github/workflows/frontend-admin-tests.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        node: [10, 12, 14, 15]
+        node: [12, 14, 16]
 
     steps:
     - name: Generate Sauce Labs strings

.github/workflows/major-version-git-pull-update.yml renamed to .github/workflows/upgrade-from-latest-release.yml

@@ -1,4 +1,4 @@
-name: "In-place git pull from master"
+name: "Upgrade from latest release"
 
 # any branch is useful for testing before a PR is submitted
 on: [push, pull_request]
@@ -16,10 +16,10 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        node: [10, 12, 14, 15]
+        node: [12, 14, 16]
 
     steps:
-    - name: Checkout master repository
+    - name: Check out latest release
       uses: actions/checkout@v2
       with:
         ref: master
@@ -60,10 +60,18 @@ jobs:
     - name: Run the backend tests
       run: cd src && npm test
 
-    - name: Git fetch
-      run: git fetch
+    # Because actions/checkout@v2 is called with "ref: master" and without
+    # "fetch-depth: 0", the local clone does not have the ${GITHUB_SHA} commit.
+    # Fetch ${GITHUB_REF} to get the ${GITHUB_SHA} commit. Note that a plain
+    # "git fetch" only fetches "normal" references (refs/heads/* and
+    # refs/tags/*), and for pull requests none of the normal references include
+    # ${GITHUB_SHA}, so we have to explicitly tell Git to fetch ${GITHUB_REF}.
+    - name: Fetch the new Git commits
+      run: git fetch --depth=1 origin "${GITHUB_REF}"
 
-    - name: Checkout this branch over master
+    - name: Upgrade to the new Git revision
+      # For pull requests, ${GITHUB_SHA} is the automatically generated merge
+      # commit that merges the PR's source branch to its destination branch.
       run: git checkout "${GITHUB_SHA}"
 
     - name: Install all dependencies and symlink for ep_etherpad-lite

CHANGELOG.md

@@ -1,9 +1,68 @@
+# 1.8.14
+
+### Security fixes
+
+* Fixed a persistent XSS vulnerability in the Chat component. In case you can't
+  update to 1.8.14 directly, we strongly recommend to cherry-pick
+  a7968115581e20ef47a533e030f59f830486bdfa. Thanks to sonarsource for the
+  professional disclosure.
+
+### Compatibility changes
+
+* Node.js v12.13.0 or later is now required.
+* The `favicon` setting is now interpreted as a pathname to a favicon file, not
+  a URL. Please see the documentation comment in `settings.json.template`.
+* The undocumented `faviconPad` and `faviconTimeslider` settings have been
+  removed.
+* MySQL/MariaDB now uses connection pooling, which means you will see up to 10
+  connections to the MySQL/MariaDB server (by default) instead of 1. This might
+  cause Etherpad to crash with a "ER_CON_COUNT_ERROR: Too many connections"
+  error if your server is configured with a low connection limit.
+* Changes to environment variable substitution in `settings.json` (see the
+  documentation comments in `settings.json.template` for details):
+  * An environment variable set to the string "null" now becomes `null` instead
+    of the string "null". Similarly, if the environment variable is unset and
+    the default value is "null" (e.g., `"${UNSET_VAR:null}"`), the value now
+    becomes `null` instead of the string "null". It is no longer possible to
+    produce the string "null" via environment variable substitution.
+  * An environment variable set to the string "undefined" now causes the setting
+    to be removed instead of set to the string "undefined". Similarly, if the
+    environment variable is unset and the default value is "undefined" (e.g.,
+    `"${UNSET_VAR:undefined}"`), the setting is now removed instead of set to
+    the string "undefined". It is no longer possible to produce the string
+    "undefined" via environment variable substitution.
+  * Support for unset variables without a default value is now deprecated.
+    Please change all instances of `"${FOO}"` in your `settings.json` to
+    `${FOO:null}` to keep the current behavior.
+  * The `DB_*` variable substitutions in `settings.json.docker` that previously
+    defaulted to `null` now default to "undefined".
+* Calling `next` without argument when using `Changeset.opIterator` does always
+  return a new Op. See b9753dcc7156d8471a5aa5b6c9b85af47f630aa8 for details.
+
+### Notable enhancements and fixes
+
+* MySQL/MariaDB now uses connection pooling, which should improve stability and
+  reduce latency.
+* Bulk database writes are now retried individually on write failure.
+* Minify: Avoid crash due to unhandled Promise rejection if stat fails.
+* padIds are now included in /socket.io query string, e.g.
+  `https://video.etherpad.com/socket.io/?padId=AWESOME&EIO=3&transport=websocket&t=...&sid=...`.
+  This is useful for directing pads to separate socket.io nodes.
+* <script> elements added via aceInitInnerdocbodyHead hook are now executed.
+* Fix read only pad access with authentication.
+* Await more db writes.
+* Disabled wtfnode dump by default.
+* Send `USER_NEWINFO` messages on reconnect.
+* Fixed loading in a hidden iframe.
+* Fixed a race condition with composition. (Thanks @ingoncalves for an exceptionally
+  detailed analysis and @rhansen for the fix.)
+
 # 1.8.13
 
 ### Notable fixes
 
 * Fixed a bug in the safeRun.sh script (#4935)
-* Don't create sessions on some static resources (#4921)
+* Add more endpoints that do not need authentication/authorization (#4921)
 * Fixed issue with non-opening device keyboard on smartphones (#4929)
 * Add version string to iframe_editor.css to prevent stale cache entry (#4964)
 

Dockerfile

@@ -55,24 +55,38 @@ RUN groupadd --system ${EP_GID:+--gid "${EP_GID}" --non-unique} etherpad && \
 ARG EP_DIR=/opt/etherpad-lite
 RUN mkdir -p "${EP_DIR}" && chown etherpad:etherpad "${EP_DIR}"
 
-# install abiword for DOC/PDF/ODT export
-RUN [ -z "${INSTALL_ABIWORD}" ] || (apt update && apt -y install abiword && apt clean && rm -rf /var/lib/apt/lists/*)
-
-# install libreoffice for DOC/PDF/ODT export
-# the mkdir is needed for configuration of openjdk-11-jre-headless, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
-RUN [ -z "${INSTALL_SOFFICE}" ] || (apt update && mkdir -p /usr/share/man/man1 && apt -y install libreoffice && apt clean && rm -rf /var/lib/apt/lists/*)
+# the mkdir is needed for configuration of openjdk-11-jre-headless, see
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
+RUN export DEBIAN_FRONTEND=noninteractive; \
+    mkdir -p /usr/share/man/man1 && \
+    apt-get -qq update && \
+    apt-get -qq --no-install-recommends install \
+        ca-certificates \
+        git \
+        ${INSTALL_ABIWORD:+abiword} \
+        ${INSTALL_SOFFICE:+libreoffice} \
+        && \
+    apt-get -qq clean && \
+    rm -rf /var/lib/apt/lists/*
 
 USER etherpad
 
 WORKDIR "${EP_DIR}"
 
 COPY --chown=etherpad:etherpad ./ ./
 
-# install node dependencies for Etherpad
-RUN src/bin/installDeps.sh && \
-	rm -rf ~/.npm/_cacache
-
-RUN [ -z "${ETHERPAD_PLUGINS}" ] || npm install ${ETHERPAD_PLUGINS}
+# Plugins must be installed before installing Etherpad's dependencies, otherwise
+# npm will try to hoist common dependencies by removing them from
+# src/node_modules and installing them in the top-level node_modules. As of
+# v6.14.10, npm's hoist logic appears to be buggy, because it sometimes removes
+# dependencies from src/node_modules but fails to add them to the top-level
+# node_modules. Even if npm correctly hoists the dependencies, the hoisting
+# seems to confuse tools such as `npm outdated`, `npm update`, and some ESLint
+# rules.
+RUN { [ -z "${ETHERPAD_PLUGINS}" ] || \
+      npm install --no-save ${ETHERPAD_PLUGINS}; } && \
+    src/bin/installDeps.sh && \
+    rm -rf ~/.npm
 
 # Copy the configuration file.
 COPY --chown=etherpad:etherpad ./settings.json.docker "${EP_DIR}"/settings.json

README.md

@@ -32,7 +32,7 @@ Etherpad is extremely flexible providing you the means to modify it to solve wha
 # Installation
 
 ## Requirements
-- `nodejs` >= **10.17.0**.
+- [Node.js](https://nodejs.org/) >= **12.13.0**.
 
 ## GNU/Linux and other UNIX-like systems
 
@@ -46,7 +46,8 @@ src/bin/run.sh
 ```
 
 ### Manual install
-You'll need git and [node.js](https://nodejs.org) installed (minimum required Node version: **10.17.0**).
+
+You'll need Git and [Node.js](https://nodejs.org/) installed.
 
 **As any user (we recommend creating a separate user called etherpad):**
 

doc/api/changeset_library.md

@@ -148,4 +148,9 @@ This is an atext. An atext has two parts: text and attribs. The text is just the
 
 The attribs are again a bunch of operators like .ops in the changeset was. But these operators are only + operators. They describe which part of the text has which attributes
 
-For more information see /doc/easysync/easysync-notes.txt in the source.
+## Resources / further reading
+
+Detailed information about the changesets & Easysync protocol:
+
+* Easysync Protocol - [/doc/easysync/easysync-notes.pdf](https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-notes.pdf)
+* Etherpad and EasySync Technical Manual - [/doc/easysync/easysync-full-description.pdf](https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-full-description.pdf)

doc/api/hooks_client-side.md

@@ -294,6 +294,11 @@ Things in context:
 
 This hook is called on the client side whenever a chat message is received from
 the server. It can be used to create different notifications for chat messages.
+Hoook functions can modify the `author`, `authorName`, `duration`, `sticky`,
+`text`, and `timeStr` context properties to change how the message is processed.
+The `text` and `timeStr` properties may contain HTML, but plugins should be
+careful to sanitize any added user input to avoid introducing an XSS
+vulnerability.
 
 ## collectContentPre
 

doc/api/hooks_server-side.md

@@ -156,11 +156,13 @@ Called from: src/node/db/SecurityManager.js
 
 Things in context:
 
-1. padID - the pad the user wants to access
+1. padID - the real ID (never the read-only ID) of the pad the user wants to
+   access
 2. token - the token of the author
 3. sessionCookie - the session the use has
 
-This hook gets called when the access to the concrete pad is being checked. Return `false` to deny access.
+This hook gets called when the access to the concrete pad is being checked.
+Return `false` to deny access.
 
 ## padCreate
 Called from: src/node/db/Pad.js
@@ -615,14 +617,14 @@ is sent to the client. Plugins can use this hook to manipulate the
 configuration. (Example: Add a tracking ID for an external analytics tool that
 is used client-side.)
 
-The clientVars function must return a Promise that resolves to an object (or
-null/undefined) whose properties will be merged into `context.clientVars`.
-Returning `callback(value)` will return a Promise that is resolved to `value`.
-
-You can modify `context.clientVars` to change the values sent to the client, but
-beware: async functions from other clientVars plugins might also be reading or
-manipulating the same `context.clientVars` object. For this reason it is
-recommended you return an object rather than modify `context.clientVars`.
+You can manipulate `clientVars` in two different ways:
+* Return an object. The object will be merged into `clientVars` via
+  `Object.assign()`, so any keys that already exist in `clientVars` will be
+  overwritten by the values in the returned object.
+* Modify `context.clientVars`. Beware: Other plugins might also be reading or
+  manipulating the same `context.clientVars` object. To avoid race conditions,
+  you are encouraged to return an object rather than modify
+  `context.clientVars`.
 
 If needed, you can access the user's account information (if authenticated) via
 `context.socket.client.request.session.user`.
@@ -643,8 +645,6 @@ exports.clientVars = (hookName, context, callback) => {
 };
 ```
 
-This can be accessed on the client-side using `clientVars.currentYear`.
-
 ## getLineHTMLForExport
 Called from: src/node/utils/ExportHtml.js