v1.18.7

this fixes a DOM-Based XSS in the recent-uploads page:

it was possible to execute arbitrary javascript by
tricking someone into visiting `/?ru&filter=</script>`

huge thanks to @Ju0x for finding and reporting this!

.github/ISSUE_TEMPLATE/bug_report.md

@@ -8,33 +8,42 @@ assignees: '9001'
 ---
 
 NOTE:
+**please use english, or include an english translation.** aside from that,  
 all of the below are optional, consider them as inspiration, delete and rewrite at will, thx md
 
 
-**Describe the bug**
+### Describe the bug
 a description of what the bug is
 
-**To Reproduce**
+### To Reproduce
 List of steps to reproduce the issue, or, if it's hard to reproduce, then at least a detailed explanation of what you did to run into it
 
-**Expected behavior**
+### Expected behavior
 a description of what you expected to happen
 
-**Screenshots**
+### Screenshots
 if applicable, add screenshots to help explain your problem, such as the kickass crashpage :^)
 
-**Server details**
-if the issue is possibly on the server-side, then mention some of the following:
-* server OS / version: 
-* python version: 
-* copyparty arguments: 
-* filesystem (`lsblk -f` on linux): 
+### Server details (if you are using docker/podman)
+remove the ones that are not relevant:
+* **server OS / version:** 
+* **how you're running copyparty:** (docker/podman/something-else)
+* **docker image:** (variant, version, and arch if you know)
+* **copyparty arguments and/or config-file:** 
 
-**Client details**
+### Server details (if you're NOT using docker/podman)
+remove the ones that are not relevant:
+* **server OS / version:** 
+* **what copyparty did you grab:** (sfx/exe/pip/arch/...)
+* **how you're running it:** (in a terminal, as a systemd-service, ...)
+* run copyparty with `--version` and grab the last 3 lines (they start with `copyparty`, `CPython`, `sqlite`) and paste them below this line:
+* **copyparty arguments and/or config-file:** 
+
+### Client details
 if the issue is possibly on the client-side, then mention some of the following:
 * the device type and model: 
 * OS version: 
 * browser version: 
 
-**Additional context**
+### Additional context
 any other context about the problem here

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,6 +7,8 @@ assignees: '9001'
 
 ---
 
+NOTE:
+**please use english, or include an english translation.** aside from that,  
 all of the below are optional, consider them as inspiration, delete and rewrite at will
 
 **is your feature request related to a problem? Please describe.**

CONTRIBUTING.md

@@ -1,8 +1,21 @@
-* do something cool
+* **found a bug?** [create an issue!](https://github.com/9001/copyparty/issues) or let me know in the [discord](https://discord.gg/25J8CdTT6G) :>
+* **fixed a bug?** create a PR or post a patch! big thx in advance :>
+* **have a cool idea?** let's discuss it! anywhere's fine, you choose.
 
-really tho, send a PR or an issue or whatever, all appreciated, anything goes, just behave aight 👍👍
+but please:
+
+
+
+# do not use AI / LMM when writing code
+
+copyparty is 100% organic, free-range, human-written software!
+
+> ⚠ you are now entering a no-copilot zone
+
+the *only* place where LMM/AI *may* be accepted is for [localization](https://github.com/9001/copyparty/tree/hovudstraum/docs/rice#translations) if you are fluent and have confirmed that the translation is accurate.
+
+sorry for the harsh tone, but this is important to me 🙏
 
-but to be more specific,
 
 
 # contribution ideas
@@ -28,6 +41,8 @@ aside from documentation and ideas, some other things that would be cool to have
 
 * **translations** -- the copyparty web-UI has translations for english and norwegian at the top of [browser.js](https://github.com/9001/copyparty/blob/hovudstraum/copyparty/web/browser.js); if you'd like to add a translation for another language then that'd be welcome! and if that language has a grammar that doesn't fit into the way the strings are assembled, then we'll fix that as we go :>
 
+  * but please note that support for [RTL (Right-to-Left) languages](https://en.wikipedia.org/wiki/Right-to-left_script) is currently not planned, since the javascript is a bit too jank for that
+
 * **UI ideas** -- at some point I was thinking of rewriting the UI in react/preact/something-not-vanilla-javascript, but I'll admit the comfiness of not having any build stage combined with raw performance has kinda convinced me otherwise :p but I'd be very open to ideas on how the UI could be improved, or be more intuitive.
 
 * **docker improvements** -- I don't really know what I'm doing when it comes to containers, so I'm sure there's a *huge* room for improvement here, mainly regarding how you're supposed to use the container with kubernetes / docker-compose / any of the other popular ways to do things. At some point I swear I'll start learning about docker so I can pick up clach04's [docker-compose draft](https://github.com/9001/copyparty/issues/38) and learn how that stuff ticks, unless someone beats me to it!

bin/README.md

@@ -78,3 +78,6 @@ cd /mnt/nas/music/.hist
 # [`prisonparty.sh`](prisonparty.sh)
 * run copyparty in a chroot, preventing any accidental file access
 * creates bindmounts for /bin, /lib, and so on, see `sysdirs=`
+
+# [`bubbleparty.sh`](bubbleparty.sh)
+* run copyparty in an isolated process, preventing any accidental file access and more

bin/bubbleparty.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+# usage: ./bubbleparty.sh ./copyparty-sfx.py ....
+bwrap \
+  --unshare-all \
+  --ro-bind /usr /usr \
+  --ro-bind /bin /bin \
+  --ro-bind /lib /lib \
+  --ro-bind /etc/resolv.conf /etc/resolv.conf \
+  --dev-bind /dev /dev \
+  --dir /tmp \
+  --dir /var \
+  --bind $(pwd) $(pwd) \
+  --share-net \
+  --die-with-parent \
+  --file 11 /etc/passwd \
+  --file 12 /etc/group \
+  "$@" \
+  11< <(getent passwd $(id -u) 65534) \
+  12< <(getent group $(id -g) 65534) 

bin/handlers/README.md

@@ -20,6 +20,8 @@ each plugin must define a `main()` which takes 3 arguments;
 
 ## on404
 
+* [redirect.py](redirect.py) sends an HTTP 301 or 302, redirecting the client to another page/file
+* [randpic.py](randpic.py) redirects `/foo/bar/randpic.jpg` to a random pic in `/foo/bar/`
 * [sorry.py](answer.py) replies with a custom message instead of the usual 404
 * [nooo.py](nooo.py) replies with an endless noooooooooooooo
 * [never404.py](never404.py) 100% guarantee that 404 will never be a thing again as it automatically creates dummy files whenever necessary

bin/handlers/randpic.py

@@ -0,0 +1,35 @@
+import os
+import random
+from urllib.parse import quote
+
+
+# assuming /foo/bar/ is a valid URL but /foo/bar/randpic.png does not exist,
+# hijack the 404 with a redirect to a random pic in that folder
+#
+# thx to lia & kipu for the idea
+
+
+def main(cli, vn, rem):
+    req_fn = rem.split("/")[-1]
+    if not cli.can_read or not req_fn.startswith("randpic"):
+        return
+
+    req_abspath = vn.canonical(rem)
+    req_ap_dir = os.path.dirname(req_abspath)
+    files_in_dir = os.listdir(req_ap_dir)
+
+    if "." in req_fn:
+        file_ext = "." + req_fn.split(".")[-1]
+        files_in_dir = [x for x in files_in_dir if x.lower().endswith(file_ext)]
+
+    if not files_in_dir:
+        return
+
+    selected_file = random.choice(files_in_dir)
+
+    req_url = "/".join([vn.vpath, rem]).strip("/")
+    req_dir = req_url.rsplit("/", 1)[0]
+    new_url = "/".join([req_dir, quote(selected_file)]).strip("/")
+
+    cli.reply(b"redirecting...", 302, headers={"Location": "/" + new_url})
+    return "true"

bin/handlers/redirect.py

@@ -0,0 +1,52 @@
+# if someone hits a 404, redirect them to another location
+
+
+def send_http_302_temporary_redirect(cli, new_path):
+    """
+    replies with an HTTP 302, which is a temporary redirect;
+    "new_path" can be any of the following:
+      - "http://a.com/" would redirect to another website,
+      - "/foo/bar" would redirect to /foo/bar on the same server;
+          note the leading '/' in the location which is important
+    """
+    cli.reply(b"redirecting...", 302, headers={"Location": new_path})
+
+
+def send_http_301_permanent_redirect(cli, new_path):
+    """
+    replies with an HTTP 301, which is a permanent redirect;
+    otherwise identical to send_http_302_temporary_redirect
+    """
+    cli.reply(b"redirecting...", 301, headers={"Location": new_path})
+
+
+def send_errorpage_with_redirect_link(cli, new_path):
+    """
+    replies with a website explaining that the page has moved;
+    "new_path" must be an absolute location on the same server
+    but without a leading '/', so for example "foo/bar"
+    would redirect to "/foo/bar"
+    """
+    cli.redirect(new_path, click=False, msg="this page has moved")
+
+
+def main(cli, vn, rem):
+    """
+    this is the function that gets called by copyparty;
+    note that vn.vpath and cli.vpath does not have a leading '/'
+    so we're adding the slash in the debug messages below
+    """
+    print(f"this client just hit a 404: {cli.ip}")
+    print(f"they were accessing this volume: /{vn.vpath}")
+    print(f"and the original request-path (straight from the URL) was /{cli.vpath}")
+    print(f"...which resolves to the following filesystem path: {vn.canonical(rem)}")
+
+    new_path = "/foo/bar/"
+    print(f"will now redirect the client to {new_path}")
+
+    # uncomment one of these:
+    send_http_302_temporary_redirect(cli, new_path)
+    #send_http_301_permanent_redirect(cli, new_path)
+    #send_errorpage_with_redirect_link(cli, new_path)
+
+    return "true"

bin/hooks/README.md

@@ -14,6 +14,8 @@ run copyparty with `--help-hooks` for usage details / hook type explanations (xm
 * [discord-announce.py](discord-announce.py) announces new uploads on discord using webhooks ([example](https://user-images.githubusercontent.com/241032/215304439-1c1cb3c8-ec6f-4c17-9f27-81f969b1811a.png))
 * [reject-mimetype.py](reject-mimetype.py) rejects uploads unless the mimetype is acceptable
 * [into-the-cache-it-goes.py](into-the-cache-it-goes.py) avoids bugs in caching proxies by immediately downloading each file that is uploaded
+* [podcast-normalizer.py](podcast-normalizer.py) creates a second file with dynamic-range-compression whenever an audio file is uploaded
+  * good example of the `idx` [hook effect](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#hook-effects) to tell copyparty about additional files to scan/index
 
 
 # upload batches
@@ -25,9 +27,11 @@ these are `--xiu` hooks; unlike `xbu` and `xau` (which get executed on every sin
 # before upload
 * [reject-extension.py](reject-extension.py) rejects uploads if they match a list of file extensions
 * [reloc-by-ext.py](reloc-by-ext.py) redirects an upload to another destination based on the file extension
+  * good example of the `reloc` [hook effect](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#hook-effects)
 
 
 # on message
 * [wget.py](wget.py) lets you download files by POSTing URLs to copyparty
 * [qbittorrent-magnet.py](qbittorrent-magnet.py) starts downloading a torrent if you post a magnet url
+* [usb-eject.py](usb-eject.py) adds web-UI buttons to safe-remove usb flashdrives shared through copyparty
 * [msg-log.py](msg-log.py) is a guestbook; logs messages to a doc in the same folder

bin/hooks/podcast-normalizer.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+
+import json
+import os
+import sys
+import subprocess as sp
+
+
+_ = r"""
+sends all uploaded audio files through an aggressive
+dynamic-range-compressor to even out the volume levels
+
+dependencies:
+    ffmpeg
+
+being an xau hook, this gets eXecuted After Upload completion
+    but before copyparty has started hashing/indexing the file, so
+    we'll create a second normalized copy in a subfolder and tell
+    copyparty to hash/index that additional file as well
+
+example usage as global config:
+    -e2d -e2t --xau j,c1,bin/hooks/podcast-normalizer.py
+
+parameters explained,
+    e2d/e2t = enable database and metadata indexing
+    xau = execute after upload
+    j   = this hook needs upload information as json (not just the filename)
+    c1  = this hook returns json on stdout, so tell copyparty to read that
+
+example usage as a volflag (per-volume config):
+    -v srv/inc/pods:inc/pods:r:rw,ed:c,xau=j,c1,bin/hooks/podcast-normalizer.py
+                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    (share fs-path srv/inc/pods at URL /inc/pods,
+     readable by all, read-write for user ed,
+     running this xau (exec-after-upload) plugin for all uploaded files)
+
+example usage as a volflag in a copyparty config file:
+    [/inc/pods]
+      srv/inc/pods
+      accs:
+        r: *
+        rw: ed
+      flags:
+        e2d  # enables file indexing
+        e2t  # metadata tags too
+        xau: j,c1,bin/hooks/podcast-normalizer.py
+
+"""
+
+########################################################################
+### CONFIG
+
+# filetypes to process; ignores everything else
+EXTS = "mp3 flac ogg oga opus m4a aac wav wma"
+
+# the name of the subdir to put the normalized files in
+SUBDIR = "normalized"
+
+########################################################################
+
+
+# try to enable support for crazy filenames
+try:
+    from copyparty.util import fsenc
+except:
+
+    def fsenc(p):
+        return p.encode("utf-8")
+
+
+def main():
+    # read info from copyparty
+    inf = json.loads(sys.argv[1])
+    vpath = inf["vp"]
+    abspath = inf["ap"]
+
+    # check if the file-extension is on the to-be-processed list
+    ext = abspath.lower().split(".")[-1]
+    if ext not in EXTS.split():
+        return
+
+    # jump into the folder where the file was uploaded
+    # and create the subfolder to place the normalized copy inside
+    dirpath, filename = os.path.split(abspath)
+    os.chdir(fsenc(dirpath))
+    os.makedirs(SUBDIR, exist_ok=True)
+
+    # the input and output filenames to give ffmpeg
+    fname_in = fsenc(f"./{filename}")
+    fname_out = fsenc(f"{SUBDIR}/{filename}.opus")
+
+    # fmt: off
+    # create and run the ffmpeg command
+    cmd = [
+        b"ffmpeg",
+        b"-nostdin",
+        b"-hide_banner",
+        b"-i", fname_in,
+        b"-af", b"dynaudnorm=f=100:g=9",  # the normalizer config
+        b"-c:a", b"libopus",
+        b"-b:a", b"128k",
+        fname_out,
+    ]
+    # fmt: on
+    sp.check_output(cmd)
+
+    # and finally, tell copyparty about the new file
+    # so it appears in the database and rss-feed:
+    vpath = f"{SUBDIR}/{filename}.opus"
+    print(json.dumps({"idx": {"vp": [vpath]}}))
+
+    # (it's fine to give it a relative path like that; it gets
+    #  resolved relative to the folder the file was uploaded into)
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except Exception as ex:
+        print("podcast-normalizer failed; %r" % (ex,))

bin/hooks/reloc-by-ext.py

@@ -71,6 +71,9 @@ def main():
     ## selecting it inside the print at the end:
     ##
 
+    # move all uploads to one specific folder
+    into_junk = {"vp": "/junk"}
+
     # create a subfolder named after the filetype and move it into there
     into_subfolder = {"vp": ext}
 
@@ -92,8 +95,8 @@ def main():
         by_category = {}  # no action
 
     # now choose the default effect to apply; can be any of these:
-    # into_subfolder  into_toplevel  into_sibling  by_category
-    effect = {"vp": "/junk"}
+    # into_junk  into_subfolder  into_toplevel  into_sibling  by_category
+    effect = into_sibling
 
     ##
     ## but we can keep going, adding more speicifc rules

bin/hooks/usb-eject.js

@@ -0,0 +1,62 @@
+// see usb-eject.py for usage
+
+function usbclick() {
+    var o = QS('#treeul a[dst="/usb/"]') || QS('#treepar a[dst="/usb/"]');
+    if (o)
+        o.click();
+}
+
+function eject_cb() {
+    var t = ('' + this.responseText).trim();
+    if (t.indexOf('can be safely unplugged') < 0 && t.indexOf('Device can be removed') < 0)
+        return toast.err(30, 'usb eject failed:\n\n' + t);
+
+    toast.ok(5, esc(t.replace(/ - /g, '\n\n')).trim());
+    usbclick(); setTimeout(usbclick, 10);
+};
+
+function add_eject_2(a) {
+    var aw = a.getAttribute('href').split(/\//g);
+    if (aw.length != 4 || aw[3])
+        return;
+
+    var v = aw[2],
+        k = 'umount_' + v;
+
+    for (var b = 0; b < 9; b++) {
+        var o = ebi(k);
+        if (!o)
+            break;
+        o.parentNode.removeChild(o);
+    }
+
+    a.appendChild(mknod('span', k, '⏏'), a);
+    o = ebi(k);
+    o.style.cssText = 'position:absolute; right:1em; margin-top:-.2em; font-size:1.3em';
+    o.onclick = function (e) {
+        ev(e);
+        var xhr = new XHR();
+        xhr.open('POST', get_evpath(), true);
+        xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded;charset=UTF-8');
+        xhr.send('msg=' + uricom_enc(':usb-eject:' + v + ':'));
+        xhr.onload = xhr.onerror = eject_cb;
+        toast.inf(10, "ejecting " + v + "...");
+    };
+};
+
+function add_eject() {
+    var o = QSA('#treeul a[href^="/usb/"]') || QSA('#treepar a[href^="/usb/"]');
+    for (var a = o.length - 1; a > 0; a--)
+        add_eject_2(o[a]);
+};
+
+(function() {
+    var f0 = treectl.rendertree;
+    treectl.rendertree = function (res, ts, top0, dst, rst) {
+        var ret = f0(res, ts, top0, dst, rst);
+        add_eject();
+        return ret;
+    };
+})();
+
+setTimeout(add_eject, 50);

bin/hooks/usb-eject.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+
+import os
+import stat
+import subprocess as sp
+import sys
+from urllib.parse import unquote_to_bytes as unquote
+
+
+"""
+if you've found yourself using copyparty to serve flashdrives on a LAN
+and your only wish is that the web-UI had a button to unmount / safely
+remove those flashdrives, then boy howdy are you in the right place :D
+
+put usb-eject.js in the webroot (or somewhere else http-accessible)
+then run copyparty with these args:
+
+   -v /run/media/egon:/usb:A:c,hist=/tmp/junk
+   --xm=c1,bin/hooks/usb-eject.py
+   --js-browser=/usb-eject.js
+
+which does the following respectively,
+
+  * share all of /run/media/egon as /usb with admin for everyone
+     and put the histpath somewhere it won't cause trouble
+  * run the usb-eject hook with stdout redirect to the web-ui
+  * add the complementary usb-eject.js to the browser
+
+"""
+
+
+MOUNT_BASE = b"/run/media/egon/"
+
+
+def main():
+    try:
+        label = sys.argv[1].split(":usb-eject:")[1].split(":")[0]
+        mp = MOUNT_BASE + unquote(label)
+        # print("ejecting [%s]... " % (mp,), end="")
+        mp = os.path.abspath(os.path.realpath(mp))
+        st = os.lstat(mp)
+        if not stat.S_ISDIR(st.st_mode) or not mp.startswith(MOUNT_BASE):
+            raise Exception("not a regular directory")
+
+        # if you're running copyparty as root (thx for the faith)
+        # you'll need something like this to make dbus talkative
+        cmd = b"sudo -u egon DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus gio mount -e"
+
+        # but if copyparty and the ui-session is running
+        # as the same user (good) then this is plenty
+        cmd = b"gio mount -e"
+
+        cmd = cmd.split(b" ") + [mp]
+        ret = sp.check_output(cmd).decode("utf-8", "replace")
+        print(ret.strip() or (label + " can be safely unplugged"))
+
+    except Exception as ex:
+        print("unmount failed: %r" % (ex,))
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/README.md

@@ -31,6 +31,9 @@ plugins in this section should only be used with appropriate precautions:
 * [very-bad-idea.py](./very-bad-idea.py) combined with [meadup.js](https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/meadup.js) converts copyparty into a janky yet extremely flexible chromecast clone
   * also adds a virtual keyboard by @steinuil to the basic-upload tab for comfy couch crowd control
   * anything uploaded through the [android app](https://github.com/9001/party-up) (files or links) are executed on the server, meaning anyone can infect your PC with malware... so protect this with a password and keep it on a LAN!
+  * [kamelåså](https://github.com/steinuil/kameloso) is a much better (and MUCH safer) alternative to this plugin
+    * powered by [chicken-curry-banana-pineapple-peanut pizza](https://a.ocv.me/pub/g/i/2025/01/298437ce-8351-4c8c-861c-fa131d217999.jpg?cache) so you know it's good
+    * and, unlike this plugin, kamelåså even has windows support (nice)
 
 
 # dependencies

bin/mtag/cksum.py

@@ -2,11 +2,15 @@
 
 import sys
 import json
-import zlib
 import struct
 import base64
 import hashlib
 
+try:
+    from zlib_ng import zlib_ng as zlib
+except:
+    import zlib
+
 try:
     from copyparty.util import fsenc
 except:

bin/mtag/install-deps.sh

@@ -22,6 +22,8 @@ set -e
 #   modifies the keyfinder python lib to load the .so in ~/pe
 
 
+export FORCE_COLOR=1
+
 linux=1
 
 win=
@@ -186,12 +188,15 @@ install_keyfinder() {
 		echo "so not found at $sop"
 		exit 1
 	}
-	
+
+	x=${-//[^x]/}; set -x; cat /etc/alpine-release
 	# rm -rf /Users/ed/Library/Python/3.9/lib/python/site-packages/*keyfinder*
 	CFLAGS="-I$h/pe/keyfinder/include -I/opt/local/include -I/usr/include/ffmpeg" \
+	CXXFLAGS="-I$h/pe/keyfinder/include -I/opt/local/include -I/usr/include/ffmpeg" \
 	LDFLAGS="-L$h/pe/keyfinder/lib -L$h/pe/keyfinder/lib64 -L/opt/local/lib" \
-	PKG_CONFIG_PATH=/c/msys64/mingw64/lib/pkgconfig \
+	PKG_CONFIG_PATH="/c/msys64/mingw64/lib/pkgconfig:$h/pe/keyfinder/lib/pkgconfig" \
 	$pybin -m pip install --user keyfinder
+	[ "$x" ] || set +x
 
 	pypath="$($pybin -c 'import keyfinder; print(keyfinder.__file__)')"
 	for pyso in "${pypath%/*}"/*.so; do

bin/mtag/very-bad-idea.py

@@ -6,6 +6,11 @@ WARNING -- DANGEROUS PLUGIN --
   running this plugin, they can execute malware on your machine
   so please keep this on a LAN and protect it with a password
 
+here is a MUCH BETTER ALTERNATIVE (which also works on Windows):
+  https://github.com/steinuil/kameloso
+
+----------------------------------------------------------------------
+
 use copyparty as a chromecast replacement:
   * post a URL and it will open in the default browser
   * upload a file and it will open in the default application

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "2.7"
-S_BUILD_DT = "2024-12-06"
+S_VERSION = "2.11"
+S_BUILD_DT = "2025-05-18"
 
 """
 u2c.py: upload to copyparty
@@ -234,6 +234,10 @@ CLEN = "Content-Length"
 
 web = None  # type: HCli
 
+links = []  # type: list[str]
+linkmtx = threading.Lock()
+linkfile = None
+
 
 class File(object):
     """an up2k upload task; represents a single file"""
@@ -761,6 +765,29 @@ def get_hashlist(file, pcb, mth):
             file.kchunks[k] = [v1, v2]
 
 
+def printlink(ar, purl, name, fk):
+    if not name:
+        url = purl  # srch
+    else:
+        name = quotep(name.encode("utf-8", WTF8)).decode("utf-8")
+        if fk:
+            url = "%s%s?k=%s" % (purl, name, fk)
+        else:
+            url = "%s%s" % (purl, name)
+
+    url = "%s/%s" % (ar.burl, url.lstrip("/"))
+
+    with linkmtx:
+        if ar.u:
+            links.append(url)
+        if ar.ud:
+            print(url)
+        if linkfile:
+            zs = "%s\n" % (url,)
+            zb = zs.encode("utf-8", "replace")
+            linkfile.write(zb)
+
+
 def handshake(ar, file, search):
     # type: (argparse.Namespace, File, bool) -> tuple[list[str], bool]
     """
@@ -780,7 +807,9 @@ def handshake(ar, file, search):
     else:
         if ar.touch:
             req["umod"] = True
-        if ar.ow:
+        if ar.owo:
+            req["replace"] = "mt"
+        elif ar.ow:
             req["replace"] = True
 
     file.recheck = False
@@ -832,12 +861,17 @@ def handshake(ar, file, search):
         raise Exception(txt)
 
     if search:
+        if ar.uon and r["hits"]:
+            printlink(ar, r["hits"][0]["rp"], "", "")
         return r["hits"], False
 
     file.url = quotep(r["purl"].encode("utf-8", WTF8)).decode("utf-8")
     file.name = r["name"]
     file.wark = r["wark"]
 
+    if ar.uon and not r["hash"]:
+        printlink(ar, file.url, r["name"], r.get("fk"))
+
     return r["hash"], r["sprs"]
 
 
@@ -1103,7 +1137,7 @@ class Ctl(object):
             nleft = self.nfiles - self.up_f
             tail = "\033[K\033[u" if VT100 and not self.ar.ns else "\r"
 
-            t = "%s eta @ %s/s, %s, %d# left\033[K" % (self.eta, spd, sleft, nleft)
+            t = "%s eta @ %s/s, %s, %d# left" % (self.eta, spd, sleft, nleft)
             if not self.hash_b:
                 t = " now hashing..."
             eprint(txt + "\033]0;{0}\033\\\r{0}{1}".format(t, tail))
@@ -1249,13 +1283,13 @@ class Ctl(object):
                         for n, zsii in enumerate(file.cids)
                     ]
                     print("chs: %s\n%s" % (vp, "\n".join(zsl)))
-                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.kchunks]
+                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.cids]
                 zb = hashlib.sha512("\n".join(zsl).encode("utf-8")).digest()[:33]
                 wark = ub64enc(zb).decode("utf-8")
                 if self.ar.jw:
                     print("%s  %s" % (wark, vp))
                 else:
-                    zd = datetime.datetime.fromtimestamp(file.lmod, UTC)
+                    zd = datetime.datetime.fromtimestamp(max(0, file.lmod), UTC)
                     dt = "%04d-%02d-%02d %02d:%02d:%02d" % (
                         zd.year,
                         zd.month,
@@ -1472,7 +1506,7 @@ class APF(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFor
 
 
 def main():
-    global web
+    global web, linkfile
 
     time.strptime("19970815", "%Y%m%d")  # python#7980
     "".encode("idna")  # python#29288
@@ -1506,9 +1540,15 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
     ap.add_argument("--touch", action="store_true", help="if last-modified timestamps differ, push local to server (need write+delete perms)")
     ap.add_argument("--ow", action="store_true", help="overwrite existing files instead of autorenaming")
+    ap.add_argument("--owo", action="store_true", help="overwrite existing files if server-file is older")
     ap.add_argument("--spd", action="store_true", help="print speeds for each file")
     ap.add_argument("--version", action="store_true", help="show version and exit")
 
+    ap = app.add_argument_group("print links")
+    ap.add_argument("-u", action="store_true", help="print list of download-links after all uploads finished")
+    ap.add_argument("-ud", action="store_true", help="print download-link after each upload finishes")
+    ap.add_argument("-uf", type=unicode, metavar="PATH", help="print list of download-links to file")
+
     ap = app.add_argument_group("compatibility")
     ap.add_argument("--cls", action="store_true", help="clear screen before start")
     ap.add_argument("--rh", type=int, metavar="TRIES", default=0, help="resolve server hostname before upload (good for buggy networks, but TLS certs will break)")
@@ -1594,6 +1634,10 @@ source file/folder selection uses rsync syntax, meaning that:
     ar.x = "|".join(ar.x or [])
 
     setattr(ar, "wlist", ar.url == "-")
+    setattr(ar, "uon", ar.u or ar.ud or ar.uf)
+
+    if ar.uf:
+        linkfile = open(ar.uf, "wb")
 
     for k in "dl dr drd wlist".split():
         errs = []
@@ -1656,6 +1700,12 @@ source file/folder selection uses rsync syntax, meaning that:
         ar.z = True
         ctl = Ctl(ar, ctl.stats)
 
+    if links:
+        print()
+        print("\n".join(links))
+    if linkfile:
+        linkfile.close()
+
     if ctl.errs:
         print("WARNING: %d errors" % (ctl.errs))
 

bin/zmq-recv.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+
+import sys
+import zmq
+
+"""
+zmq-recv.py: demo zmq receiver
+2025-01-22, v1.0, ed <irc.rizon.net>, MIT-Licensed
+https://github.com/9001/copyparty/blob/hovudstraum/bin/zmq-recv.py
+
+basic zmq-server to receive events from copyparty; try one of
+the below and then "send a message to serverlog" in the web-ui:
+
+1) dumb fire-and-forget to any and all listeners;
+run this script with "sub" and run copyparty with this:
+  --xm zmq:pub:tcp://*:5556
+
+2) one lucky listener gets the message, blocks if no listeners:
+run this script with "pull" and run copyparty with this:
+  --xm t3,zmq:push:tcp://*:5557
+
+3) blocking syn/ack mode, client must ack each message;
+run this script with "rep" and run copyparty with this:
+  --xm t3,zmq:req:tcp://localhost:5555
+
+note: to conditionally block uploads based on message contents,
+use rep_server to answer with "return 1" and run copyparty with
+  --xau t3,c,zmq:req:tcp://localhost:5555
+"""
+
+
+ctx = zmq.Context()
+
+
+def sub_server():
+    # PUB/SUB allows any number of servers/clients, and
+    # messages are fire-and-forget
+    sck = ctx.socket(zmq.SUB)
+    sck.connect("tcp://localhost:5556")
+    sck.setsockopt_string(zmq.SUBSCRIBE, "")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def pull_server():
+    # PUSH/PULL allows any number of servers/clients, and
+    # each message is sent to a exactly one PULL client
+    sck = ctx.socket(zmq.PULL)
+    sck.connect("tcp://localhost:5557")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def rep_server():
+    # REP/REQ is a server/client pair where each message must be
+    # acked by the other before another message can be sent, so
+    # copyparty will do a blocking-wait for the ack
+    sck = ctx.socket(zmq.REP)
+    sck.bind("tcp://*:5555")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+        reply = b"thx"
+        # reply = b"return 1"  # non-zero to block an upload
+        sck.send(reply)
+
+
+mode = sys.argv[1].lower() if len(sys.argv) > 1 else ""
+
+if mode == "sub":
+    sub_server()
+elif mode == "pull":
+    pull_server()
+elif mode == "rep":
+    rep_server()
+else:
+    print("specify mode as first argument:  SUB | PULL | REP")

contrib/README.md

@@ -12,14 +12,19 @@
 * assumes the webserver and copyparty is running on the same server/IP
 * modify `10.13.1.1` as necessary if you wish to support browsers without javascript
 
-### [`sharex.sxcu`](sharex.sxcu)
-* sharex config file to upload screenshots and grab the URL
+### [`sharex.sxcu`](sharex.sxcu) - Windows screenshot uploader
+* [sharex](https://getsharex.com/) config file to upload screenshots and grab the URL
 * `RequestURL`: full URL to the target folder
 * `pw`: password (remove the `pw` line if anon-write)
 * the `act:bput` thing is optional since copyparty v1.9.29
 * using an older sharex version, maybe sharex v12.1.1 for example? dw fam i got your back 👉😎👉 [`sharex12.sxcu`](sharex12.sxcu)
 
-### [`flameshot.sh`](flameshot.sh)
+### [`ishare.iscu`](ishare.iscu) - MacOS screenshot uploader
+* [ishare](https://isharemac.app/) config file to upload screenshots and grab the URL
+* `RequestURL`: full URL to the target folder
+* `pw`: password (remove the `pw` line if anon-write)
+
+### [`flameshot.sh`](flameshot.sh) - Linux screenshot uploader
 * takes a screenshot with [flameshot](https://flameshot.org/) on Linux, uploads it, and writes the URL to clipboard
 
 ### [`send-to-cpp.contextlet.json`](send-to-cpp.contextlet.json)
@@ -45,6 +50,9 @@
 * give a 3rd argument to install it to your copyparty config
 * systemd service at [`systemd/cfssl.service`](systemd/cfssl.service)
 
+### [`zfs-tune.py`](zfs-tune.py)
+* optimizes databases for optimal performance when stored on a zfs filesystem; also see [openzfs docs](https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#database-workloads) and specifically the SQLite subsection
+
 # OS integration
 init-scripts to start copyparty as a service
 * [`systemd/copyparty.service`](systemd/copyparty.service) runs the sfx normally
@@ -53,5 +61,10 @@ init-scripts to start copyparty as a service
 * [`openrc/copyparty`](openrc/copyparty)
 
 # Reverse-proxy
-copyparty has basic support for running behind another webserver
-* [`nginx/copyparty.conf`](nginx/copyparty.conf)
+copyparty supports running behind another webserver
+* [`apache/copyparty.conf`](apache/copyparty.conf)
+* [`haproxy/copyparty.conf`](haproxy/copyparty.conf)
+* [`lighttpd/subdomain.conf`](lighttpd/subdomain.conf)
+* [`lighttpd/subpath.conf`](lighttpd/subpath.conf)
+* [`nginx/copyparty.conf`](nginx/copyparty.conf) -- recommended
+* [`traefik/copyparty.yaml`](traefik/copyparty.yaml)

contrib/apache/copyparty.conf

@@ -1,14 +1,29 @@
-# when running copyparty behind a reverse proxy,
-# the following arguments are recommended:
+# if you would like to use unix-sockets (recommended),
+# you must run copyparty with one of the following:
 #
-#   -i 127.0.0.1    only accept connections from nginx
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
 #
 # if you are doing location-based proxying (such as `/stuff` below)
 # you must run copyparty with --rp-loc=stuff
 #
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 
+
 LoadModule proxy_module modules/mod_proxy.so
-ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"
-# do not specify ProxyPassReverse
+
 RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
+# NOTE: do not specify ProxyPassReverse
+
+
+##
+## then, enable one of the below:
+
+# use subdomain proxying to unix-socket (best)
+ProxyPass "/" "unix:///dev/shm/party.sock|http://whatever/"
+
+# use subdomain proxying to 127.0.0.1 (slower)
+#ProxyPass "/" "http://127.0.0.1:3923/"
+
+# use subpath proxying to 127.0.0.1 (slow and maybe buggy)
+#ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"

contrib/haproxy/copyparty.conf

@@ -0,0 +1,24 @@
+# this config is essentially two separate examples;
+#
+#   foo1 connects to copyparty using tcp, and
+#   foo2 uses unix-sockets for 27% higher performance
+#
+# to use foo2 you must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+
+defaults
+  mode http
+  option forwardfor
+  timeout connect 1s
+  timeout client 610s
+  timeout server 610s
+
+listen foo1
+  bind *:8081
+  server srv1 127.0.0.1:3923 maxconn 512
+
+listen foo2
+  bind *:8082
+  server srv1 /dev/shm/party.sock maxconn 512

contrib/ishare.iscu

@@ -0,0 +1,10 @@
+{
+  "Name": "copyparty",
+  "RequestURL": "http://127.0.0.1:3923/screenshots/",
+  "Headers": {
+    "pw": "PUT_YOUR_PASSWORD_HERE_MY_DUDE",
+    "accept": "json"
+  },
+  "FileFormName": "f",
+  "ResponseURL": "{{fileurl}}"
+}

contrib/lighttpd/subdomain.conf

@@ -0,0 +1,24 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subdomain.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+proxy.forwarded = ( "for" => 1, "proto" => 1 )
+proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+# if you really need to use tcp instead of unix-sockets, do this instead:
+#proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )

contrib/lighttpd/subpath.conf

@@ -0,0 +1,31 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subpath.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# also since this example proxies a subpath instead of the
+# recommended subdomain-proxying, we must also specify this:
+#
+#   --rp-loc files
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+$HTTP["url"] =~ "^/files" {
+    proxy.forwarded = ( "for" => 1, "proto" => 1 )
+    proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+    # if you really need to use tcp instead of unix-sockets, do this instead:
+    #proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )
+}

contrib/nginx/copyparty.conf

@@ -2,19 +2,38 @@
 # not accept more consecutive clients than what copyparty is able to;
 # nginx default is 512  (worker_processes 1, worker_connections 512)
 #
+# ======================================================================
+#
+# to reverse-proxy a specific path/subpath/location below a domain
+# (rather than a complete subdomain), for example "/qw/er", you must
+# run copyparty with --rp-loc /qw/as and also change the following:
+# 	location / {
+# 		proxy_pass http://cpp_tcp;
+# to this:
+# 	location /qw/er/ {
+# 		proxy_pass http://cpp_tcp/qw/er/;
+#
+# ======================================================================
+#
 # rarely, in some extreme usecases, it can be good to add -j0
 # (40'000 requests per second, or 20gbps upload/download in parallel)
 # but this is usually counterproductive and slightly buggy
 #
+# ======================================================================
+#
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 #
-# if you are behind cloudflare (or another protection service),
+# ======================================================================
+#
+# if you are behind cloudflare (or another CDN/WAF/protection service),
 # remember to reject all connections which are not coming from your
 # protection service -- for cloudflare in particular, you can
 # generate the list of permitted IP ranges like so:
 #   (curl -s https://www.cloudflare.com/ips-v{4,6} | sed 's/^/allow /; s/$/;/'; echo; echo "deny all;") > /etc/nginx/cloudflare-only.conf
 #
 # and then enable it below by uncomenting the cloudflare-only.conf line
+#
+# ======================================================================
 
 
 upstream cpp_tcp {
@@ -36,9 +55,9 @@ upstream cpp_uds {
 	# but there must be at least one unix-group which both
 	# nginx and copyparty is a member of; if that group is
 	# "www" then run copyparty with the following args:
-	# -i unix:770:www:/tmp/party.sock
+	# -i unix:770:www:/dev/shm/party.sock
 
-	server unix:/tmp/party.sock fail_timeout=1s;
+	server unix:/dev/shm/party.sock fail_timeout=1s;
 	keepalive 1;
 }
 
@@ -61,14 +80,18 @@ server {
 		client_max_body_size 0;
 		proxy_buffering off;
 		proxy_request_buffering off;
+		# improve download speed from 600 to 1500 MiB/s
+		proxy_buffers 32 8k;
+		proxy_buffer_size 16k;
+		proxy_busy_buffers_size 24k;
 
+		proxy_set_header   Connection        "Keep-Alive";
 		proxy_set_header   Host              $host;
 		proxy_set_header   X-Real-IP         $remote_addr;
-		proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
-		# NOTE: with cloudflare you want this instead:
-		#proxy_set_header   X-Forwarded-For   $http_cf_connecting_ip;
 		proxy_set_header   X-Forwarded-Proto $scheme;
-		proxy_set_header   Connection        "Keep-Alive";
+		proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
+		# NOTE: with cloudflare you want this X-Forwarded-For instead:
+		#proxy_set_header   X-Forwarded-For   $http_cf_connecting_ip;
 	}
 }
 

contrib/nixos/modules/copyparty.nix

@@ -1,21 +1,26 @@
-{ config, pkgs, lib, ... }:
-
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
 with lib;
-
 let
-  mkKeyValue = key: value:
+  mkKeyValue =
+    key: value:
     if value == true then
-    # sets with a true boolean value are coerced to just the key name
+      # sets with a true boolean value are coerced to just the key name
       key
     else if value == false then
-    # or omitted completely when false
+      # or omitted completely when false
       ""
     else
       (generators.mkKeyValueDefault { inherit mkValueString; } ": " key value);
 
   mkAttrsString = value: (generators.toKeyValue { inherit mkKeyValue; } value);
 
-  mkValueString = value:
+  mkValueString =
+    value:
     if isList value then
       (concatStringsSep ", " (map mkValueString value))
     else if isAttrs value then
@@ -49,13 +54,14 @@ let
     ${concatStringsSep "\n" (mapAttrsToList mkVolume cfg.volumes)}
   '';
 
-  name = "copyparty";
   cfg = config.services.copyparty;
-  configFile = pkgs.writeText "${name}.conf" configStr;
-  runtimeConfigPath = "/run/${name}/${name}.conf";
-  home = "/var/lib/${name}";
-  defaultShareDir = "${home}/data";
-in {
+  configFile = pkgs.writeText "copyparty.conf" configStr;
+  runtimeConfigPath = "/run/copyparty/copyparty.conf";
+  externalCacheDir = "/var/cache/copyparty";
+  externalStateDir = "/var/lib/copyparty";
+  defaultShareDir = "${externalStateDir}/data";
+in
+{
   options.services.copyparty = {
     enable = mkEnableOption "web-based file manager";
 
@@ -68,6 +74,35 @@ in {
       '';
     };
 
+    mkHashWrapper = mkOption {
+      type = types.bool;
+      default = true;
+      description = ''
+        Make a shell script wrapper called 'copyparty-hash' with all options set here,
+        that launches the hashing cli.
+      '';
+    };
+
+    user = mkOption {
+      type = types.str;
+      default = "copyparty";
+      description = ''
+        The user that copyparty will run under.
+
+        If changed from default, you are responsible for making sure the user exists.
+      '';
+    };
+
+    group = mkOption {
+      type = types.str;
+      default = "copyparty";
+      description = ''
+        The group that copyparty will run under.
+
+        If changed from default, you are responsible for making sure the user exists.
+      '';
+    };
+
     openFilesLimit = mkOption {
       default = 4096;
       type = types.either types.int types.str;
@@ -79,33 +114,41 @@ in {
       description = ''
         Global settings to apply.
         Directly maps to values in the [global] section of the copyparty config.
+        Cannot set "c" or "hist", those are set by this module.
         See `${getExe cfg.package} --help` for more details.
       '';
       default = {
         i = "127.0.0.1";
         no-reload = true;
+        hist = externalCacheDir;
       };
       example = literalExpression ''
         {
           i = "0.0.0.0";
           no-reload = true;
+          hist = ${externalCacheDir};
         }
       '';
     };
 
     accounts = mkOption {
-      type = types.attrsOf (types.submodule ({ ... }: {
-        options = {
-          passwordFile = mkOption {
-            type = types.str;
-            description = ''
-              Runtime file path to a file containing the user password.
-              Must be readable by the copyparty user.
-            '';
-            example = "/run/keys/copyparty/ed";
-          };
-        };
-      }));
+      type = types.attrsOf (
+        types.submodule (
+          { ... }:
+          {
+            options = {
+              passwordFile = mkOption {
+                type = types.str;
+                description = ''
+                  Runtime file path to a file containing the user password.
+                  Must be readable by the copyparty user.
+                '';
+                example = "/run/keys/copyparty/ed";
+              };
+            };
+          }
+        )
+      );
       description = ''
         A set of copyparty accounts to create.
       '';
@@ -118,74 +161,81 @@ in {
     };
 
     volumes = mkOption {
-      type = types.attrsOf (types.submodule ({ ... }: {
-        options = {
-          path = mkOption {
-            type = types.str;
-            description = ''
-              Path of a directory to share.
-            '';
-          };
-          access = mkOption {
-            type = types.attrs;
-            description = ''
-              Attribute list of permissions and the users to apply them to.
-
-              The key must be a string containing any combination of allowed permission:
-                "r" (read):   list folder contents, download files
-                "w" (write):  upload files; need "r" to see the uploads
-                "m" (move):   move files and folders; need "w" at destination
-                "d" (delete): permanently delete files and folders
-                "g" (get):    download files, but cannot see folder contents
-                "G" (upget):  "get", but can see filekeys of their own uploads
-                "h" (html):   "get", but folders return their index.html
-                "a" (admin):  can see uploader IPs, config-reload
-
-              For example: "rwmd"
-
-              The value must be one of:
-                an account name, defined in `accounts`
-                a list of account names
-                "*", which means "any account"
-            '';
-            example = literalExpression ''
-              {
-                # wG = write-upget = see your own uploads only
-                wG = "*";
-                # read-write-modify-delete for users "ed" and "k"
-                rwmd = ["ed" "k"];
+      type = types.attrsOf (
+        types.submodule (
+          { ... }:
+          {
+            options = {
+              path = mkOption {
+                type = types.path;
+                description = ''
+                  Path of a directory to share.
+                '';
               };
-            '';
-          };
-          flags = mkOption {
-            type = types.attrs;
-            description = ''
-              Attribute list of volume flags to apply.
-              See `${getExe cfg.package} --help-flags` for more details.
-            '';
-            example = literalExpression ''
-              {
-                # "fk" enables filekeys (necessary for upget permission) (4 chars long)
-                fk = 4;
-                # scan for new files every 60sec
-                scan = 60;
-                # volflag "e2d" enables the uploads database
-                e2d = true;
-                # "d2t" disables multimedia parsers (in case the uploads are malicious)
-                d2t = true;
-                # skips hashing file contents if path matches *.iso
-                nohash = "\.iso$";
+              access = mkOption {
+                type = types.attrs;
+                description = ''
+                  Attribute list of permissions and the users to apply them to.
+
+                  The key must be a string containing any combination of allowed permission:
+                    "r" (read):   list folder contents, download files
+                    "w" (write):  upload files; need "r" to see the uploads
+                    "m" (move):   move files and folders; need "w" at destination
+                    "d" (delete): permanently delete files and folders
+                    "g" (get):    download files, but cannot see folder contents
+                    "G" (upget):  "get", but can see filekeys of their own uploads
+                    "h" (html):   "get", but folders return their index.html
+                    "a" (admin):  can see uploader IPs, config-reload
+
+                  For example: "rwmd"
+
+                  The value must be one of:
+                    an account name, defined in `accounts`
+                    a list of account names
+                    "*", which means "any account"
+                '';
+                example = literalExpression ''
+                  {
+                    # wG = write-upget = see your own uploads only
+                    wG = "*";
+                    # read-write-modify-delete for users "ed" and "k"
+                    rwmd = ["ed" "k"];
+                  };
+                '';
               };
-            '';
-            default = { };
-          };
-        };
-      }));
+              flags = mkOption {
+                type = types.attrs;
+                description = ''
+                  Attribute list of volume flags to apply.
+                  See `${getExe cfg.package} --help-flags` for more details.
+                '';
+                example = literalExpression ''
+                  {
+                    # "fk" enables filekeys (necessary for upget permission) (4 chars long)
+                    fk = 4;
+                    # scan for new files every 60sec
+                    scan = 60;
+                    # volflag "e2d" enables the uploads database
+                    e2d = true;
+                    # "d2t" disables multimedia parsers (in case the uploads are malicious)
+                    d2t = true;
+                    # skips hashing file contents if path matches *.iso
+                    nohash = "\.iso$";
+                  };
+                '';
+                default = { };
+              };
+            };
+          }
+        )
+      );
       description = "A set of copyparty volumes to create";
       default = {
         "/" = {
           path = defaultShareDir;
-          access = { r = "*"; };
+          access = {
+            r = "*";
+          };
         };
       };
       example = literalExpression ''
@@ -204,80 +254,122 @@ in {
     };
   };
 
-  config = mkIf cfg.enable {
-    systemd.services.copyparty = {
-      description = "http file sharing hub";
-      wantedBy = [ "multi-user.target" ];
+  config = mkIf cfg.enable (
+    let
+      command = "${getExe cfg.package} -c ${runtimeConfigPath}";
+    in
+    {
+      systemd.services.copyparty = {
+        description = "http file sharing hub";
+        wantedBy = [ "multi-user.target" ];
 
-      environment = {
-        PYTHONUNBUFFERED = "true";
-        XDG_CONFIG_HOME = "${home}/.config";
+        environment = {
+          PYTHONUNBUFFERED = "true";
+          XDG_CONFIG_HOME = externalStateDir;
+        };
+
+        preStart =
+          let
+            replaceSecretCommand =
+              name: attrs:
+              "${getExe pkgs.replace-secret} '${passwordPlaceholder name}' '${attrs.passwordFile}' ${runtimeConfigPath}";
+          in
+          ''
+            set -euo pipefail
+            install -m 600 ${configFile} ${runtimeConfigPath}
+            ${concatStringsSep "\n" (mapAttrsToList replaceSecretCommand cfg.accounts)}
+          '';
+
+        serviceConfig = {
+          Type = "simple";
+          ExecStart = command;
+          # Hardening options
+          User = cfg.user;
+          Group = cfg.group;
+          RuntimeDirectory = [ "copyparty" ];
+          RuntimeDirectoryMode = "0700";
+          StateDirectory = [ "copyparty" ];
+          StateDirectoryMode = "0700";
+          CacheDirectory = lib.mkIf (cfg.settings ? hist) [ "copyparty" ];
+          CacheDirectoryMode = lib.mkIf (cfg.settings ? hist) "0700";
+          WorkingDirectory = externalStateDir;
+          BindReadOnlyPaths = [
+            "/nix/store"
+            "-/etc/resolv.conf"
+            "-/etc/nsswitch.conf"
+            "-/etc/group"
+            "-/etc/hosts"
+            "-/etc/localtime"
+          ] ++ (mapAttrsToList (k: v: "-${v.passwordFile}") cfg.accounts);
+          BindPaths =
+            (if cfg.settings ? hist then [ cfg.settings.hist ] else [ ])
+            ++ [ externalStateDir ]
+            ++ (mapAttrsToList (k: v: v.path) cfg.volumes);
+          # ProtectSystem = "strict";
+          # Note that unlike what 'ro' implies,
+          # this actually makes it impossible to read anything in the root FS,
+          # except for things explicitly mounted via `RuntimeDirectory`, `StateDirectory`, `CacheDirectory`, and `BindReadOnlyPaths`.
+          # This is because TemporaryFileSystem creates a *new* *empty* filesystem for the process, so only bindmounts are visible.
+          TemporaryFileSystem = "/:ro";
+          PrivateTmp = true;
+          PrivateDevices = true;
+          ProtectKernelTunables = true;
+          ProtectControlGroups = true;
+          RestrictSUIDSGID = true;
+          PrivateMounts = true;
+          ProtectKernelModules = true;
+          ProtectKernelLogs = true;
+          ProtectHostname = true;
+          ProtectClock = true;
+          ProtectProc = "invisible";
+          ProcSubset = "pid";
+          RestrictNamespaces = true;
+          RemoveIPC = true;
+          UMask = "0077";
+          LimitNOFILE = cfg.openFilesLimit;
+          NoNewPrivileges = true;
+          LockPersonality = true;
+          RestrictRealtime = true;
+          MemoryDenyWriteExecute = true;
+        };
       };
 
-      preStart = let
-        replaceSecretCommand = name: attrs:
-          "${getExe pkgs.replace-secret} '${
-            passwordPlaceholder name
-          }' '${attrs.passwordFile}' ${runtimeConfigPath}";
-      in ''
-        set -euo pipefail
-        install -m 600 ${configFile} ${runtimeConfigPath}
-        ${concatStringsSep "\n"
-        (mapAttrsToList replaceSecretCommand cfg.accounts)}
-      '';
+      # ensure volumes exist:
+      systemd.tmpfiles.settings."copyparty" = (
+        lib.attrsets.mapAttrs' (
+          name: value:
+          lib.attrsets.nameValuePair (value.path) {
+            d = {
+              #: in front of things means it wont change it if the directory already exists.
+              group = ":${cfg.group}";
+              user = ":${cfg.user}";
+              mode = ":755";
+            };
+          }
+        ) cfg.volumes
+      );
 
-      serviceConfig = {
-        Type = "simple";
-        ExecStart = "${getExe cfg.package} -c ${runtimeConfigPath}";
-
-        # Hardening options
-        User = "copyparty";
-        Group = "copyparty";
-        RuntimeDirectory = name;
-        RuntimeDirectoryMode = "0700";
-        StateDirectory = [ name "${name}/data" "${name}/.config" ];
-        StateDirectoryMode = "0700";
-        WorkingDirectory = home;
-        TemporaryFileSystem = "/:ro";
-        BindReadOnlyPaths = [
-          "/nix/store"
-          "-/etc/resolv.conf"
-          "-/etc/nsswitch.conf"
-          "-/etc/hosts"
-          "-/etc/localtime"
-        ] ++ (mapAttrsToList (k: v: "-${v.passwordFile}") cfg.accounts);
-        BindPaths = [ home ] ++ (mapAttrsToList (k: v: v.path) cfg.volumes);
-        # Would re-mount paths ignored by temporary root
-        #ProtectSystem = "strict";
-        ProtectHome = true;
-        PrivateTmp = true;
-        PrivateDevices = true;
-        ProtectKernelTunables = true;
-        ProtectControlGroups = true;
-        RestrictSUIDSGID = true;
-        PrivateMounts = true;
-        ProtectKernelModules = true;
-        ProtectKernelLogs = true;
-        ProtectHostname = true;
-        ProtectClock = true;
-        ProtectProc = "invisible";
-        ProcSubset = "pid";
-        RestrictNamespaces = true;
-        RemoveIPC = true;
-        UMask = "0077";
-        LimitNOFILE = cfg.openFilesLimit;
-        NoNewPrivileges = true;
-        LockPersonality = true;
-        RestrictRealtime = true;
+      users.groups.copyparty = lib.mkIf (cfg.user == "copyparty" && cfg.group == "copyparty") { };
+      users.users.copyparty = lib.mkIf (cfg.user == "copyparty" && cfg.group == "copyparty") {
+        description = "Service user for copyparty";
+        group = "copyparty";
+        home = externalStateDir;
+        isSystemUser = true;
       };
-    };
+      environment.systemPackages = lib.mkIf cfg.mkHashWrapper [
+        (pkgs.writeShellScriptBin "copyparty-hash" ''
+          set -a  # automatically export variables
+          # set same environment variables as the systemd service
+          ${lib.pipe config.systemd.services.copyparty.environment [
+            (lib.filterAttrs (n: v: v != null && n != "PATH"))
+            (lib.mapAttrs (_: v: "${v}"))
+            (lib.toShellVars)
+          ]}
+          PATH=${config.systemd.services.copyparty.environment.PATH}:$PATH
 
-    users.groups.copyparty = { };
-    users.users.copyparty = {
-      description = "Service user for copyparty";
-      group = "copyparty";
-      home = home;
-      isSystemUser = true;
-    };
-  };
+          exec ${command} --ah-cli
+        '')
+      ];
+    }
+  );
 }

contrib/package/arch/PKGBUILD

@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.16.3"
+pkgver="1.18.6"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -16,12 +16,13 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
             "libkeyfinder-git: detection of musical keys" 
             "qm-vamp-plugins: BPM detection" 
             "python-pyopenssl: ftps functionality" 
-            "python-argon2_cffi: hashed passwords in config" 
+            "python-pyzmq: send zeromq messages from event-hooks" 
+            "python-argon2-cffi: hashed passwords in config" 
             "python-impacket-git: smb support (bad idea)"
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("82e79e4b27661b23986b0eba7bd38c42d5f8a2d2c278b755134ac08588a7f27e")
+sha256sums=("80762d91ac88815e73d0ca2806c6391dcf8ccd521bc402cc312349f3bc8e8b28")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/default.nix

@@ -1,38 +1,67 @@
-{ lib, stdenv, makeWrapper, fetchurl, utillinux, python, jinja2, impacket, pyftpdlib, pyopenssl, argon2-cffi, pillow, pyvips, ffmpeg, mutagen,
+{
+  lib,
+  stdenv,
+  makeWrapper,
+  fetchurl,
+  util-linux,
+  python,
+  jinja2,
+  impacket,
+  pyopenssl,
+  cfssl,
+  argon2-cffi,
+  pillow,
+  pyvips,
+  pyzmq,
+  ffmpeg,
+  mutagen,
 
-# use argon2id-hashed passwords in config files (sha2 is always available)
-withHashedPasswords ? true,
+  # use argon2id-hashed passwords in config files (sha2 is always available)
+  withHashedPasswords ? true,
 
-# generate TLS certificates on startup (pointless when reverse-proxied)
-withCertgen ? false,
+  # generate TLS certificates on startup (pointless when reverse-proxied)
+  withCertgen ? false,
 
-# create thumbnails with Pillow; faster than FFmpeg / MediaProcessing
-withThumbnails ? true,
+  # create thumbnails with Pillow; faster than FFmpeg / MediaProcessing
+  withThumbnails ? true,
 
-# create thumbnails with PyVIPS; even faster, uses more memory
-# -- can be combined with Pillow to support more filetypes
-withFastThumbnails ? false,
+  # create thumbnails with PyVIPS; even faster, uses more memory
+  # -- can be combined with Pillow to support more filetypes
+  withFastThumbnails ? false,
 
-# enable FFmpeg; thumbnails for most filetypes (also video and audio), extract audio metadata, transcode audio to opus
-# -- possibly dangerous if you allow anonymous uploads, since FFmpeg has a huge attack surface
-# -- can be combined with Thumbnails and/or FastThumbnails, since FFmpeg is slower than both
-withMediaProcessing ? true,
+  # enable FFmpeg; thumbnails for most filetypes (also video and audio), extract audio metadata, transcode audio to opus
+  # -- possibly dangerous if you allow anonymous uploads, since FFmpeg has a huge attack surface
+  # -- can be combined with Thumbnails and/or FastThumbnails, since FFmpeg is slower than both
+  withMediaProcessing ? true,
 
-# if MediaProcessing is not enabled, you probably want this instead (less accurate, but much safer and faster)
-withBasicAudioMetadata ? false,
+  # if MediaProcessing is not enabled, you probably want this instead (less accurate, but much safer and faster)
+  withBasicAudioMetadata ? false,
 
-# enable FTPS support in the FTP server
-withFTPS ? false,
+  # send ZeroMQ messages from event-hooks
+  withZeroMQ ? true,
 
-# samba/cifs server; dangerous and buggy, enable if you really need it
-withSMB ? false,
+  # enable FTPS support in the FTP server
+  withFTPS ? false,
+
+  # samba/cifs server; dangerous and buggy, enable if you really need it
+  withSMB ? false,
+
+  # extra packages to add to the PATH
+  extraPackages ? [ ],
+
+  # function that accepts a python packageset and returns a list of packages to
+  # be added to the python venv. useful for scripts and such that require
+  # additional dependencies
+  extraPythonPackages ? (_p: [ ]),
 
 }:
 
 let
   pinData = lib.importJSON ./pin.json;
-  pyEnv = python.withPackages (ps:
-    with ps; [
+  pyEnv = python.withPackages (
+    ps:
+    with ps;
+    [
       jinja2
     ]
     ++ lib.optional withSMB impacket
@@ -43,21 +72,37 @@ let
     ++ lib.optional withMediaProcessing ffmpeg
     ++ lib.optional withBasicAudioMetadata mutagen
     ++ lib.optional withHashedPasswords argon2-cffi
-    );
-in stdenv.mkDerivation {
+    ++ lib.optional withZeroMQ pyzmq
+    ++ (extraPythonPackages ps)
+  );
+
+  runtimeDeps = ([ util-linux ] ++ extraPackages ++ lib.optional withMediaProcessing ffmpeg);
+in
+stdenv.mkDerivation {
   pname = "copyparty";
-  version = pinData.version;
+  inherit (pinData) version;
   src = fetchurl {
-    url = pinData.url;
-    hash = pinData.hash;
+    inherit (pinData) url hash;
   };
-  buildInputs = [ makeWrapper ];
+  nativeBuildInputs = [ makeWrapper ];
   dontUnpack = true;
-  dontBuild = true;
   installPhase = ''
     install -Dm755 $src $out/share/copyparty-sfx.py
     makeWrapper ${pyEnv.interpreter} $out/bin/copyparty \
-      --set PATH '${lib.makeBinPath ([ utillinux ] ++ lib.optional withMediaProcessing ffmpeg)}:$PATH' \
-      --add-flags "$out/share/copyparty-sfx.py"
+      --prefix PATH : ${lib.makeBinPath runtimeDeps} \
+      --add-flag $out/share/copyparty-sfx.py
   '';
+  meta = {
+    description = "Turn almost any device into a file server";
+    longDescription = ''
+      Portable file server with accelerated resumable uploads, dedup, WebDAV,
+      FTP, TFTP, zeroconf, media indexer, thumbnails++ all in one file, no deps
+    '';
+    homepage = "https://github.com/9001/copyparty";
+    changelog = "https://github.com/9001/copyparty/releases/tag/v${pinData.version}";
+    license = lib.licenses.mit;
+    inherit (python.meta) platforms;
+    mainProgram = "copyparty";
+    sourceProvenance = [ lib.sourceTypes.binaryBytecode ];
+  };
 }

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.16.3/copyparty-sfx.py",
-    "version": "1.16.3",
-    "hash": "sha256-O1yVIUJ/zy1kde4CHxeBEex2gT+HWUJ28nhHgonW6Qg="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.18.6/copyparty-sfx.py",
+    "version": "1.18.6",
+    "hash": "sha256-No89mzKHHZZH19ws9dqfvQO0pnZw7jKDMGhNa4LOFlY="
 }

contrib/package/nix/partyfuse/default.nix

@@ -0,0 +1,26 @@
+{
+  stdenvNoCC,
+  copyparty,
+  python3,
+  makeBinaryWrapper,
+}:
+let
+  python = python3.withPackages (p: [ p.fusepy ]);
+in
+stdenvNoCC.mkDerivation {
+  pname = "partyfuse";
+  inherit (copyparty) version meta;
+  src = ../../../..;
+
+  nativeBuildInputs = [ makeBinaryWrapper ];
+
+  installPhase = ''
+    runHook preInstall
+
+    install -Dm444 bin/partyfuse.py -t $out/share/copyparty
+    makeWrapper ${python.interpreter} $out/bin/partyfuse \
+      --add-flag $out/share/copyparty/partyfuse.py
+
+    runHook postInstall
+  '';
+}

contrib/package/nix/u2c/default.nix

@@ -0,0 +1,24 @@
+{
+  stdenvNoCC,
+  copyparty,
+  python312,
+  makeBinaryWrapper,
+}:
+stdenvNoCC.mkDerivation {
+  pname = "u2c";
+  inherit (copyparty) version meta;
+  src = ../../../..;
+
+  nativeBuildInputs = [ makeBinaryWrapper ];
+
+  installPhase = ''
+    runHook preInstall
+
+    install -Dm444 bin/u2c.py -t $out/share/copyparty
+    mkdir $out/bin
+    makeWrapper ${python312.interpreter} $out/bin/u2c \
+      --add-flag $out/share/copyparty/u2c.py
+
+    runHook postInstall
+  '';
+}

contrib/plugins/README.md

@@ -15,6 +15,7 @@ save one of these as `.epilogue.html` inside a folder to customize it:
 point `--js-browser` to one of these by URL:
 
 * [`minimal-up2k.js`](minimal-up2k.js) is similar to the above `minimal-up2k.html` except it applies globally to all write-only folders
+* [`quickmove.js`](quickmove.js) adds a hotkey to move selected files into a subfolder
 * [`up2k-hooks.js`](up2k-hooks.js) lets you specify a ruleset for files to skip uploading
   * [`up2k-hook-ytid.js`](up2k-hook-ytid.js) is a more specific example checking youtube-IDs against some API
 

contrib/plugins/graft-thumbs.js

@@ -0,0 +1,117 @@
+// USAGE:
+//   place this file somewhere in the webroot and then
+//   python3 -m copyparty --js-browser /.res/graft-thumbs.js
+//
+// DESCRIPTION:
+//   this is a gridview plugin which, for each file in a folder,
+//   looks for another file with the same filename (but with a
+//   different file extension)
+//
+//   if one of those files is an image and the other is not,
+//   then this plugin assumes the image is a "sidecar thumbnail"
+//   for the other file, and it will graft the image thumbnail
+//   onto the non-image file (for example an mp3)
+//
+//   optional feature 1, default-enabled:
+//   the image-file is then hidden from the directory listing
+//
+//   optional feature 2, default-enabled:
+//   when clicking the audio file, the image will also open
+
+
+(function() {
+
+	// `graft_thumbs` assumes the gridview has just been rendered;
+	// it looks for sidecars, and transplants those thumbnails onto
+	// the other file with the same basename (filename sans extension)
+
+	var graft_thumbs = function () {
+		if (!thegrid.en)
+			return;  // not in grid mode
+
+		var files = msel.getall(),
+			pairs = {};
+
+		console.log(files);
+
+		for (var a = 0; a < files.length; a++) {
+			var file = files[a],
+				is_pic = /\.(jpe?g|png|gif|webp)$/i.exec(file.vp),
+				is_audio = re_au_all.exec(file.vp),
+				basename = file.vp.replace(/\.[^\.]+$/, ""),
+				entry = pairs[basename];
+
+			if (!entry)
+				// first time seeing this basename; create a new entry in pairs
+				entry = pairs[basename] = {};
+
+			if (is_pic)
+				entry.thumb = file;
+			else if (is_audio)
+				entry.audio = file;
+		}
+
+		var basenames = Object.keys(pairs);
+		for (var a = 0; a < basenames.length; a++)
+			(function(a) {
+				var pair = pairs[basenames[a]];
+
+				if (!pair.thumb || !pair.audio)
+					return;  // not a matching pair of files
+
+				var img_thumb = QS('#ggrid a[ref="' + pair.thumb.id + '"] img[onload]'),
+					img_audio = QS('#ggrid a[ref="' + pair.audio.id + '"] img[onload]');
+
+				if (!img_thumb || !img_audio)
+					return;  // something's wrong... let's bail
+
+				// alright, graft the thumb...
+				img_audio.src = img_thumb.src;
+
+				// ...and hide the sidecar
+				img_thumb.closest('a').style.display = 'none';
+
+				// ...and add another onclick-handler to the audio,
+				// so it also opens the pic while playing the song
+				img_audio.addEventListener('click', function() {
+					img_thumb.click();
+					return false;  // let it bubble to the next listener
+				});
+
+			})(a);
+	};
+
+	// ...and then the trick! near the end of loadgrid,
+	// thegrid.bagit is called to initialize the baguettebox
+	// (image/video gallery); this is the perfect function to
+	// "hook" (hijack) so we can run our code :^)
+
+	// need to grab a backup of the original function first,
+	var orig_func = thegrid.bagit;
+
+	// and then replace it with our own:
+	thegrid.bagit = function (isrc) {
+
+		if (isrc !== '#ggrid')
+			// we only want to modify the grid, so
+			// let the original function handle this one
+			return orig_func(isrc);
+
+		graft_thumbs();
+
+		// when changing directories, the grid is
+		// rendered before msel returns the correct
+		// filenames, so schedule another run:
+		setTimeout(graft_thumbs, 1);
+
+		// and finally, call the original thegrid.bagit function
+		return orig_func(isrc);
+	};
+
+	if (ls0) {
+		// the server included an initial listing json (ls0),
+		// so the grid has already been rendered without our hook
+		graft_thumbs();
+	}
+
+})();

contrib/plugins/minimal-up2k.js

@@ -12,6 +12,23 @@ almost the same as minimal-up2k.html except this one...:
 
  -- looks slightly better
 
+
+========================
+== USAGE INSTRUCTIONS ==
+
+1. create a volume which anyone can read from (if you haven't already)
+2. copy this file into that volume, so anyone can download it
+3. enable the plugin by telling the webbrowser to load this file;
+    assuming the URL to the public volume is /res/, and
+    assuming you're using config-files, then add this to your config:
+
+    [global]
+      js-browser: /res/minimal-up2k.js
+
+alternatively, if you're not using config-files, then
+add the following commandline argument instead:
+  --js-browser=/res/minimal-up2k.js
+
 */
 
 var u2min = `

contrib/plugins/quickmove.js

@@ -0,0 +1,140 @@
+"use strict";
+
+
+// USAGE:
+//   place this file somewhere in the webroot, 
+//   for example in a folder named ".res" to hide it, and then
+//   python3 copyparty-sfx.py -v .::A --js-browser /.res/quickmove.js
+//
+// DESCRIPTION:
+//   the command above launches copyparty with one single volume;
+//   ".::A" = current folder as webroot, and everyone has Admin
+//
+//   the plugin adds hotkey "W" which moves all selected files
+//   into a subfolder named "foobar" inside the current folder
+
+
+(function() {
+
+    var action_to_perform = ask_for_confirmation_and_then_move;
+    // this decides what the new hotkey should do;
+    //  ask_for_confirmation_and_then_move  =  show a yes/no box,
+    //  move_selected_files  =  just move the files immediately
+
+    var move_destination = "foobar";
+    // this is the target folder to move files to;
+    // by default it is a subfolder of the current folder,
+    // but it can also be an absolute path like "/foo/bar"
+
+    // ===
+    // ===   END OF CONFIG
+    // ===
+
+    var main_hotkey_handler,  // copyparty's original hotkey handler
+        plugin_enabler,  // timer to engage this plugin when safe
+        files_to_move;  // list of files to move
+
+    function ask_for_confirmation_and_then_move() {
+        var num_files = msel.getsel().length,
+            msg = "move the selected " + num_files + " files?";
+
+        if (!num_files)
+            return toast.warn(2, 'no files were selected to be moved');
+
+        modal.confirm(msg, move_selected_files, null);
+    }
+
+    function move_selected_files() {
+        var selection = msel.getsel();
+
+        if (!selection.length)
+            return toast.warn(2, 'no files were selected to be moved');
+
+        if (thegrid.bbox) {
+            // close image/video viewer
+            thegrid.bbox = null;
+            baguetteBox.destroy();
+        }
+
+        files_to_move = [];
+        for (var a = 0; a < selection.length; a++)
+            files_to_move.push(selection[a].vp);
+
+        move_next_file();
+    }
+
+    function move_next_file() {
+        var num_files = files_to_move.length,
+            filepath = files_to_move.pop(),
+            filename = vsplit(filepath)[1];
+
+        toast.inf(10, "moving " + num_files + " files...\n\n" + filename);
+
+        var dst = move_destination;
+
+        if (!dst.endsWith('/'))
+            // must have a trailing slash, so add it
+            dst += '/';
+
+        if (!dst.startsWith('/'))
+            // destination is a relative path, so prefix current folder path
+            dst = get_evpath() + dst;
+
+        // and finally append the filename
+        dst += '/' + filename;
+
+        // prepare the move-request to be sent
+        var xhr = new XHR();
+        xhr.onload = xhr.onerror = function() {
+            if (this.status !== 201)
+                return toast.err(30, 'move failed: ' + esc(this.responseText));
+
+            if (files_to_move.length)
+                return move_next_file();  // still more files to go
+
+            toast.ok(1, 'move OK');
+            treectl.goto(); // reload the folder contents
+        };
+        xhr.open('POST', filepath + '?move=' + dst);
+        xhr.send();
+    }
+
+    function our_hotkey_handler(e) {
+        // bail if either ALT, CTRL, or SHIFT is pressed
+        if (e.altKey || e.shiftKey || e.isComposing || ctrl(e))
+            return main_hotkey_handler(e);  // let copyparty handle this keystroke
+
+        var key_name = (e.code || e.key) + '',
+    		ae = document.activeElement,
+		    aet = ae && ae != document.body ? ae.nodeName.toLowerCase() : '';
+
+        // check the current aet (active element type),
+        // only continue if one of the following currently has input focus:
+        //   nothing | link | button | table-row | table-cell | div | text
+        if (aet && !/^(a|button|tr|td|div|pre)$/.test(aet))
+            return main_hotkey_handler(e);  // let copyparty handle this keystroke
+
+        if (key_name == 'KeyW') {
+            // okay, this one's for us... do the thing
+            action_to_perform();
+            return ev(e);
+        }
+
+        return main_hotkey_handler(e);  // let copyparty handle this keystroke
+    }
+
+    function enable_plugin() {
+        if (!window.hotkeys_attached)
+            return console.log('quickmove is waiting for the page to finish loading');
+
+        clearInterval(plugin_enabler);
+        main_hotkey_handler = document.onkeydown;
+        document.onkeydown = our_hotkey_handler;
+        console.log('quickmove is now enabled');
+    }
+
+    // copyparty doesn't enable its hotkeys until the page
+    // has finished loading, so we'll wait for that too
+    plugin_enabler = setInterval(enable_plugin, 100);
+
+})();

contrib/traefik/copyparty.yaml

@@ -0,0 +1,25 @@
+# ./traefik --configFile=copyparty.yaml
+
+entryPoints:
+  web:
+    address: :8080
+    transport:
+      # don't disconnect during big uploads
+      respondingTimeouts:
+        readTimeout: "0s"
+log:
+  level: DEBUG
+providers:
+  file:
+    # WARNING: must be same filename as current file
+    filename: "copyparty.yaml"
+http:
+  services:
+    service-cpp:
+      loadBalancer:
+        servers:
+          - url: "http://127.0.0.1:3923/"
+  routers:
+    my-router:
+      rule: "PathPrefix(`/`)"
+      service: service-cpp

contrib/zfs-tune.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+
+import os
+import sqlite3
+import sys
+import traceback
+
+
+"""
+when the up2k-database is stored on a zfs volume, this may give
+slightly higher performance (actual gains not measured yet)
+
+NOTE: must be applied in combination with the related advice in the openzfs documentation;
+https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#database-workloads
+and see specifically the SQLite subsection
+
+it is assumed that all databases are stored in a single location,
+for example with `--hist /var/store/hists`
+
+three alternatives for running this script:
+
+1. copy it into /var/store/hists and run "python3 zfs-tune.py s"
+    (s = modify all databases below folder containing script)
+
+2. cd into /var/store/hists and run "python3 ~/zfs-tune.py w"
+    (w = modify all databases below current working directory)
+
+3. python3 ~/zfs-tune.py /var/store/hists
+
+if you use docker, run copyparty with `--hist /cfg/hists`, copy this script into /cfg, and run this:
+podman run --rm -it --entrypoint /usr/bin/python3 ghcr.io/9001/copyparty-ac /cfg/zfs-tune.py s
+
+"""
+
+
+PAGESIZE = 65536
+
+
+# borrowed from copyparty; short efficient stacktrace for errors
+def min_ex(max_lines: int = 8, reverse: bool = False) -> str:
+    et, ev, tb = sys.exc_info()
+    stb = traceback.extract_tb(tb) if tb else traceback.extract_stack()[:-1]
+    fmt = "%s:%d <%s>: %s"
+    ex = [fmt % (fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in stb]
+    if et or ev or tb:
+        ex.append("[%s] %s" % (et.__name__ if et else "(anonymous)", ev))
+    return "\n".join(ex[-max_lines:][:: -1 if reverse else 1])
+
+
+def set_pagesize(db_path):
+    try:
+        # check current page_size
+        with sqlite3.connect(db_path) as db:
+            v = db.execute("pragma page_size").fetchone()[0]
+            if v == PAGESIZE:
+                print(" `-- OK")
+                return
+
+        # https://www.sqlite.org/pragma.html#pragma_page_size
+        #  `- disable wal; set pagesize; vacuum
+        #      (copyparty will reenable wal if necessary)
+
+        with sqlite3.connect(db_path) as db:
+            db.execute("pragma journal_mode=delete")
+            db.commit()
+
+        with sqlite3.connect(db_path) as db:
+            db.execute(f"pragma page_size = {PAGESIZE}")
+            db.execute("vacuum")
+
+        print(" `-- new pagesize OK")
+
+    except Exception:
+        err = min_ex().replace("\n", "\n -- ")
+        print(f"FAILED: {db_path}\n -- {err}")
+
+
+def main():
+    top = os.path.dirname(os.path.abspath(__file__))
+    cwd = os.path.abspath(os.getcwd())
+    try:
+        x = sys.argv[1]
+    except:
+        print(f"""
+this script takes one mandatory argument:
+specify 's' to start recursing from folder containing this script file ({top})
+specify 'w' to start recursing from the current working directory ({cwd})
+specify a path to start recursing from there
+""")
+        sys.exit(1)
+
+    if x.lower() == "w":
+        top = cwd
+    elif x.lower() != "s":
+        top = x
+
+    for dirpath, dirs, files in os.walk(top):
+        for fname in files:
+            if not fname.endswith(".db"):
+                continue
+            db_path = os.path.join(dirpath, fname)
+            print(db_path)
+            set_pagesize(db_path)
+
+
+if __name__ == "__main__":
+    main()

copyparty/__init__.py

@@ -80,6 +80,7 @@ web/deps/prismd.css
 web/deps/scp.woff2
 web/deps/sha512.ac.js
 web/deps/sha512.hw.js
+web/idp.html
 web/iiam.gif
 web/md.css
 web/md.html
@@ -91,6 +92,9 @@ web/mde.html
 web/mde.js
 web/msg.css
 web/msg.html
+web/rups.css
+web/rups.html
+web/rups.js
 web/shares.css
 web/shares.html
 web/shares.js

copyparty/__main__.py

@@ -40,6 +40,7 @@ from .cfg import flagcats, onedash
 from .svchub import SvcHub
 from .util import (
     APPLESAN_TXT,
+    BAD_BOTS,
     DEF_EXP,
     DEF_MTE,
     DEF_MTH,
@@ -52,17 +53,20 @@ from .util import (
     PYFTPD_VER,
     RAM_AVAIL,
     RAM_TOTAL,
+    RE_ANSI,
     SQLITE_VER,
     UNPLICATIONS,
+    URL_BUG,
+    URL_PRJ,
     Daemon,
     align_tab,
-    ansi_re,
     b64enc,
     dedent,
     has_resource,
     load_resource,
     min_ex,
     pybin,
+    read_utf8,
     termsize,
     wrap,
 )
@@ -163,7 +167,7 @@ def lprint(*a: Any, **ka: Any) -> None:
     txt: str = " ".join(unicode(x) for x in a) + eol
     printed.append(txt)
     if not VT100:
-        txt = ansi_re.sub("", txt)
+        txt = RE_ANSI.sub("", txt)
 
     print(txt, end="", **ka)
 
@@ -224,7 +228,23 @@ def init_E(EE: EnvParams) -> None:
     if E.mod.endswith("__init__"):
         E.mod = os.path.dirname(E.mod)
 
-    if sys.platform == "win32":
+    try:
+        p = os.environ.get("XDG_CONFIG_HOME")
+        if not p:
+            raise Exception()
+        if p.startswith("~"):
+            p = os.path.expanduser(p)
+        p = os.path.abspath(os.path.realpath(p))
+        p = os.path.join(p, "copyparty")
+        if not os.path.isdir(p):
+            os.mkdir(p)
+        os.listdir(p)
+    except:
+        p = ""
+
+    if p:
+        E.cfg = p
+    elif sys.platform == "win32":
         bdir = os.environ.get("APPDATA") or os.environ.get("TEMP") or "."
         E.cfg = os.path.normpath(bdir + "/copyparty")
     elif sys.platform == "darwin":
@@ -253,8 +273,7 @@ def get_srvname(verbose) -> str:
     if verbose:
         lprint("using hostname from {}\n".format(fp))
     try:
-        with open(fp, "rb") as f:
-            ret = f.read().decode("utf-8", "replace").strip()
+        return read_utf8(None, fp, True).strip()
     except:
         ret = ""
         namelen = 5
@@ -263,47 +282,18 @@ def get_srvname(verbose) -> str:
             ret = re.sub("[234567=]", "", ret)[:namelen]
         with open(fp, "wb") as f:
             f.write(ret.encode("utf-8") + b"\n")
-
-    return ret
+        return ret
 
 
-def get_fk_salt() -> str:
-    fp = os.path.join(E.cfg, "fk-salt.txt")
+def get_salt(name: str, nbytes: int) -> str:
+    fp = os.path.join(E.cfg, "%s-salt.txt" % (name,))
     try:
-        with open(fp, "rb") as f:
-            ret = f.read().strip()
+        return read_utf8(None, fp, True).strip()
     except:
-        ret = b64enc(os.urandom(18))
+        ret = b64enc(os.urandom(nbytes))
         with open(fp, "wb") as f:
             f.write(ret + b"\n")
-
-    return ret.decode("utf-8")
-
-
-def get_dk_salt() -> str:
-    fp = os.path.join(E.cfg, "dk-salt.txt")
-    try:
-        with open(fp, "rb") as f:
-            ret = f.read().strip()
-    except:
-        ret = b64enc(os.urandom(30))
-        with open(fp, "wb") as f:
-            f.write(ret + b"\n")
-
-    return ret.decode("utf-8")
-
-
-def get_ah_salt() -> str:
-    fp = os.path.join(E.cfg, "ah-salt.txt")
-    try:
-        with open(fp, "rb") as f:
-            ret = f.read().strip()
-    except:
-        ret = b64enc(os.urandom(18))
-        with open(fp, "wb") as f:
-            f.write(ret + b"\n")
-
-    return ret.decode("utf-8")
+        return ret.decode("utf-8")
 
 
 def ensure_locale() -> None:
@@ -332,17 +322,16 @@ def ensure_webdeps() -> None:
     if has_resource(E, "web/deps/mini-fa.woff"):
         return
 
-    warn(
-        """could not find webdeps;
+    t = """could not find webdeps;
   if you are running the sfx, or exe, or pypi package, or docker image,
   then this is a bug! Please let me know so I can fix it, thanks :-)
-  https://github.com/9001/copyparty/issues/new?labels=bug&template=bug_report.md
+  %s
 
   however, if you are a dev, or running copyparty from source, and you want
   full client functionality, you will need to build or obtain the webdeps:
-  https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#building
+  %s/blob/hovudstraum/docs/devnotes.md#building
     """
-    )
+    warn(t % (URL_BUG, URL_PRJ))
 
 
 def configure_ssl_ver(al: argparse.Namespace) -> None:
@@ -558,14 +547,15 @@ def get_sects():
             when running behind a reverse-proxy, it's recommended to
             use unix-sockets for improved performance and security;
 
-            \033[32m-i unix:770:www:\033[33m/tmp/a.sock\033[0m listens on \033[33m/tmp/a.sock\033[0m with
-            permissions \033[33m0770\033[0m; only accessible to members of the \033[33mwww\033[0m
-            group. This is the best approach. Alternatively,
+            \033[32m-i unix:770:www:\033[33m/dev/shm/party.sock\033[0m listens on
+            \033[33m/dev/shm/party.sock\033[0m with permissions \033[33m0770\033[0m;
+            only accessible to members of the \033[33mwww\033[0m group.
+            This is the best approach. Alternatively,
 
-            \033[32m-i unix:777:\033[33m/tmp/a.sock\033[0m sets perms \033[33m0777\033[0m so anyone can
-            access it; bad unless it's inside a restricted folder
+            \033[32m-i unix:777:\033[33m/dev/shm/party.sock\033[0m sets perms \033[33m0777\033[0m so anyone
+            can access it; bad unless it's inside a restricted folder
 
-            \033[32m-i unix:\033[33m/tmp/a.sock\033[0m keeps umask-defined permissions
+            \033[32m-i unix:\033[33m/dev/shm/party.sock\033[0m keeps umask-defined permission
             (usually \033[33m0600\033[0m) and the same user/group as copyparty
 
             \033[33m-p\033[0m (tcp ports) is ignored for unix sockets
@@ -739,6 +729,10 @@ def get_sects():
               the \033[33m,,\033[35m stops copyparty from reading the rest as flags and
               the \033[33m--\033[35m stops notify-send from reading the message as args
               and the alert will be "hey" followed by the messagetext
+
+             \033[36m--xau zmq:pub:tcp://*:5556\033[35m announces uploads on zeromq;
+             \033[36m--xau t3,zmq:push:tcp://*:5557\033[35m also works, and you can
+             \033[36m--xau t3,j,zmq:req:tcp://localhost:5555\033[35m too for example
             \033[0m
             each hook is executed once for each event, except for \033[36mxiu\033[0m
             which builds up a backlog of uploads, running the hook just once
@@ -770,11 +764,22 @@ def get_sects():
             values for --urlform:
               \033[36mstash\033[35m dumps the data to file and returns length + checksum
               \033[36msave,get\033[35m dumps to file and returns the page like a GET
-              \033[36mprint,get\033[35m prints the data in the log and returns GET
-              (leave out the ",get" to return an error instead)\033[0m
+              \033[36mprint    \033[35m prints the data to log and returns an error
+              \033[36mprint,xm \033[35m prints the data to log and returns --xm output
+              \033[36mprint,get\033[35m prints the data to log and returns GET\033[0m
 
-            note that the \033[35m--xm\033[0m hook will only run if \033[35m--urlform\033[0m
-              is either \033[36mprint\033[0m or the default \033[36mprint,get\033[0m
+            note that the \033[35m--xm\033[0m hook will only run if \033[35m--urlform\033[0m is
+              either \033[36mprint\033[0m or \033[36mprint,get\033[0m or the default \033[36mprint,xm\033[0m
+
+            if an \033[35m--xm\033[0m hook returns text, then
+              the response code will be HTTP 202;
+              http/get responses will be HTTP 200
+
+            if there are multiple \033[35m--xm\033[0m hooks defined, then
+              the first hook that produced output is returned
+
+            if there are no \033[35m--xm\033[0m hooks defined, then the default
+              \033[36mprint,xm\033[0m behaves like \033[36mprint,get\033[0m (returning html)
             """
             ),
         ],
@@ -859,6 +864,43 @@ def get_sects():
             """
             ),
         ],
+        [
+            "chmod",
+            "file/folder permissions",
+            dedent(
+                """
+            global-option \033[33m--chmod-f\033[0m and volflag \033[33mchmod_f\033[0m specifies the unix-permission to use when creating a new file
+
+            similarly, \033[33m--chmod-d\033[0m and \033[33mchmod_d\033[0m sets the directory/folder perm
+
+            the value is a three-digit octal number such as \033[32m755\033[0m, \033[32m750\033[0m, \033[32m644\033[0m, etc.
+
+            first digit = "User"; permission for the unix-user
+            second digit = "Group"; permission for the unix-group
+            third digit = "Other"; permission for all other users/groups
+
+            for files:
+            \033[32m0\033[0m = \033[35m---\033[0m = no access
+            \033[32m1\033[0m = \033[35m--x\033[0m = can execute the file as a program
+            \033[32m2\033[0m = \033[35m-w-\033[0m = can write
+            \033[32m3\033[0m = \033[35m-wx\033[0m = can write and execute
+            \033[32m4\033[0m = \033[35mr--\033[0m = can read
+            \033[32m5\033[0m = \033[35mr-x\033[0m = can read and execute
+            \033[32m6\033[0m = \033[35mrw-\033[0m = can read and write
+            \033[32m7\033[0m = \033[35mrwx\033[0m = can read, write, execute
+
+            for directories/folders:
+            \033[32m0\033[0m = \033[35m---\033[0m = no access
+            \033[32m1\033[0m = \033[35m--x\033[0m = can read files in folder but not list contents
+            \033[32m2\033[0m = \033[35m-w-\033[0m = n/a
+            \033[32m3\033[0m = \033[35m-wx\033[0m = can create files but not list
+            \033[32m4\033[0m = \033[35mr--\033[0m = can list, but not read/write
+            \033[32m5\033[0m = \033[35mr-x\033[0m = can list and read files
+            \033[32m6\033[0m = \033[35mrw-\033[0m = n/a
+            \033[32m7\033[0m = \033[35mrwx\033[0m = can read, write, list
+            """
+            ),
+        ],
         [
             "pwhash",
             "password hashing",
@@ -955,11 +997,12 @@ def add_general(ap, nc, srvname):
     ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m], see --help-accounts")
     ap2.add_argument("--grp", metavar="G:N,N", type=u, action="append", help="add group, \033[33mNAME\033[0m:\033[33mUSER1\033[0m,\033[33mUSER2\033[0m,\033[33m...\033[0m; example [\033[32madmins:ed,foo,bar\033[0m]")
     ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files (volflag=dots)")
-    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,xm", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
     ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="server terminal title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
     ap2.add_argument("--name", metavar="TXT", type=u, default=srvname, help="server name (displayed topleft in browser and in mDNS)")
     ap2.add_argument("--mime", metavar="EXT=MIME", type=u, action="append", help="map file \033[33mEXT\033[0mension to \033[33mMIME\033[0mtype, for example [\033[32mjpg=image/jpeg\033[0m]")
     ap2.add_argument("--mimes", action="store_true", help="list default mimetype mapping and exit")
+    ap2.add_argument("--rmagic", action="store_true", help="do expensive analysis to improve accuracy of returned mimetypes; will make file-downloads, rss, and webdav slower (volflag=rmagic)")
     ap2.add_argument("--license", action="store_true", help="show licenses and exit")
     ap2.add_argument("--version", action="store_true", help="show versions and exit")
 
@@ -999,16 +1042,24 @@ def add_upload(ap):
     ap2 = ap.add_argument_group('upload options')
     ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads, hiding them from clients unless \033[33m-ed\033[0m")
     ap2.add_argument("--plain-ip", action="store_true", help="when avoiding filename collisions by appending the uploader's ip to the filename: append the plaintext ip instead of salting and hashing the ip")
+    ap2.add_argument("--put-name", metavar="TXT", type=u, default="put-{now.6f}-{cip}.bin", help="filename for nameless uploads (when uploader doesn't provide a name); default is [\033[32mput-UNIXTIME-IP.bin\033[0m] (the \033[32m.6f\033[0m means six decimal places) (volflag=put_name)")
+    ap2.add_argument("--put-ck", metavar="ALG", type=u, default="sha512", help="default checksum-hasher for PUT/WebDAV uploads: no / md5 / sha1 / sha256 / sha512 / b2 / blake2 / b2s / blake2s (volflag=put_ck)")
+    ap2.add_argument("--bup-ck", metavar="ALG", type=u, default="sha512", help="default checksum-hasher for bup/basic-uploader: no / md5 / sha1 / sha256 / sha512 / b2 / blake2 / b2s / blake2s (volflag=bup_ck)")
     ap2.add_argument("--unpost", metavar="SEC", type=int, default=3600*12, help="grace period where uploads can be deleted by the uploader, even without delete permissions; 0=disabled, default=12h")
     ap2.add_argument("--u2abort", metavar="NUM", type=int, default=1, help="clients can abort incomplete uploads by using the unpost tab (requires \033[33m-e2d\033[0m). [\033[32m0\033[0m] = never allowed (disable feature), [\033[32m1\033[0m] = allow if client has the same IP as the upload AND is using the same account, [\033[32m2\033[0m] = just check the IP, [\033[32m3\033[0m] = just check account-name (volflag=u2abort)")
     ap2.add_argument("--blank-wt", metavar="SEC", type=int, default=300, help="file write grace period (any client can write to a blank file last-modified more recently than \033[33mSEC\033[0m seconds ago)")
     ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without \033[33m-e2d\033[0m; roughly 1 MiB RAM per 600")
     ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (bad idea to enable this on windows and/or cow filesystems)")
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
+    ap2.add_argument("--chmod-f", metavar="UGO", type=u, default="", help="unix file permissions to use when creating files; default is probably 644 (OS-decided), see --help-chmod. Examples: [\033[32m644\033[0m] = owner-RW + all-R, [\033[32m755\033[0m] = owner-RWX + all-RX, [\033[32m777\033[0m] = full-yolo (volflag=chmod_f)")
+    ap2.add_argument("--chmod-d", metavar="UGO", type=u, default="755", help="unix file permissions to use when creating directories; see --help-chmod. Examples: [\033[32m755\033[0m] = owner-RW + all-R, [\033[32m777\033[0m] = full-yolo (volflag=chmod_d)")
+    ap2.add_argument("--uid", metavar="N", type=int, default=-1, help="unix user-id to chown new files/folders to; default = -1 = do-not-change (volflag=uid)")
+    ap2.add_argument("--gid", metavar="N", type=int, default=-1, help="unix group-id to chown new files/folders to; default = -1 = do-not-change (volflag=gid)")
     ap2.add_argument("--dedup", action="store_true", help="enable symlink-based upload deduplication (volflag=dedup)")
     ap2.add_argument("--safe-dedup", metavar="N", type=int, default=50, help="how careful to be when deduplicating files; [\033[32m1\033[0m] = just verify the filesize, [\033[32m50\033[0m] = verify file contents have not been altered (volflag=safededup)")
     ap2.add_argument("--hardlink", action="store_true", help="enable hardlink-based dedup; will fallback on symlinks when that is impossible (across filesystems) (volflag=hardlink)")
     ap2.add_argument("--hardlink-only", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=hardlinkonly)")
+    ap2.add_argument("--reflink", action="store_true", help="enable reflink-based dedup; will fallback on full copies when that is impossible (non-CoW filesystem) (volflag=reflink)")
     ap2.add_argument("--no-dupe", action="store_true", help="reject duplicate files during upload; only matches within the same volume (volflag=nodupe)")
     ap2.add_argument("--no-clone", action="store_true", help="do not use existing data on disk to satisfy dupe uploads; reduces server HDD reads in exchange for much more network load (volflag=noclone)")
     ap2.add_argument("--no-snap", action="store_true", help="disable snapshots -- forget unfinished uploads on shutdown; don't create .hist/up2k.snap files -- abandoned/interrupted uploads must be cleaned up manually")
@@ -1021,8 +1072,10 @@ def add_upload(ap):
     ap2.add_argument("--df", metavar="GiB", type=u, default="0", help="ensure \033[33mGiB\033[0m free disk space by rejecting upload requests; assumes gigabytes unless a unit suffix is given: [\033[32m256m\033[0m], [\033[32m4\033[0m], [\033[32m2T\033[0m] (volflag=df)")
     ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="windows-only: minimum size of incoming uploads through up2k before they are made into sparse files")
     ap2.add_argument("--turbo", metavar="LVL", type=int, default=0, help="configure turbo-mode in up2k client; [\033[32m-1\033[0m] = forbidden/always-off, [\033[32m0\033[0m] = default-off and warn if enabled, [\033[32m1\033[0m] = default-off, [\033[32m2\033[0m] = on, [\033[32m3\033[0m] = on and disable datecheck")
-    ap2.add_argument("--u2j", metavar="JOBS", type=int, default=2, help="web-client: number of file chunks to upload in parallel; 1 or 2 is good for low-latency (same-country) connections, 4-8 for android clients, 16 for cross-atlantic (max=64)")
+    ap2.add_argument("--nosubtle", metavar="N", type=int, default=0, help="when to use a wasm-hasher instead of the browser's builtin; faster on chrome, but buggy in older chrome versions. [\033[32m0\033[0m] = only when necessary (non-https), [\033[32m1\033[0m] = always (all browsers), [\033[32m2\033[0m] = always on chrome/firefox, [\033[32m3\033[0m] = always on chrome, [\033[32mN\033[0m] = chrome-version N and newer (recommendation: 137)")
+    ap2.add_argument("--u2j", metavar="JOBS", type=int, default=2, help="web-client: number of file chunks to upload in parallel; 1 or 2 is good when latency is low (same-country), 2~4 for android-clients, 2~6 for cross-atlantic. Max is 6 in most browsers. Big values increase network-speed but may reduce HDD-speed")
     ap2.add_argument("--u2sz", metavar="N,N,N", type=u, default="1,64,96", help="web-client: default upload chunksize (MiB); sets \033[33mmin,default,max\033[0m in the settings gui. Each HTTP POST will aim for \033[33mdefault\033[0m, and never exceed \033[33mmax\033[0m. Cloudflare max is 96. Big values are good for cross-atlantic but may increase HDD fragmentation on some FS. Disable this optimization with [\033[32m1,1,1\033[0m]")
+    ap2.add_argument("--u2ow", metavar="NUM", type=int, default=0, help="web-client: default setting for when to replace/overwrite existing files; [\033[32m0\033[0m]=never, [\033[32m1\033[0m]=if-client-newer, [\033[32m2\033[0m]=always (volflag=u2ow)")
     ap2.add_argument("--u2sort", metavar="TXT", type=u, default="s", help="upload order; [\033[32ms\033[0m]=smallest-first, [\033[32mn\033[0m]=alphabetical, [\033[32mfs\033[0m]=force-s, [\033[32mfn\033[0m]=force-n -- alphabetical is a bit slower on fiber/LAN but makes it easier to eyeball if everything went fine")
     ap2.add_argument("--write-uplog", action="store_true", help="write POST reports to textfiles in working-directory")
 
@@ -1039,8 +1092,10 @@ def add_network(ap):
     ap2.add_argument("--rp-loc", metavar="PATH", type=u, default="", help="if reverse-proxying on a location instead of a dedicated domain/subdomain, provide the base location here; example: [\033[32m/foo/bar\033[0m]")
     if ANYWIN:
         ap2.add_argument("--reuseaddr", action="store_true", help="set reuseaddr on listening sockets on windows; allows rapid restart of copyparty at the expense of being able to accidentally start multiple instances")
-    else:
+    elif not MACOS:
         ap2.add_argument("--freebind", action="store_true", help="allow listening on IPs which do not yet exist, for example if the network interfaces haven't finished going up. Only makes sense for IPs other than '0.0.0.0', '127.0.0.1', '::', and '::1'. May require running as root (unless net.ipv6.ip_nonlocal_bind)")
+    ap2.add_argument("--wr-h-eps", metavar="PATH", type=u, default="", help="write list of listening-on ip:port to textfile at \033[33mPATH\033[0m when http-servers have started")
+    ap2.add_argument("--wr-h-aon", metavar="PATH", type=u, default="", help="write list of accessible-on ip:port to textfile at \033[33mPATH\033[0m when http-servers have started")
     ap2.add_argument("--s-thead", metavar="SEC", type=int, default=120, help="socket timeout (read request header)")
     ap2.add_argument("--s-tbody", metavar="SEC", type=float, default=128.0, help="socket timeout (read/write request/response bodies). Use 60 on fast servers (default is extremely safe). Disable with 0 if reverse-proxied for a 2%% speed boost")
     ap2.add_argument("--s-rd-sz", metavar="B", type=int, default=256*1024, help="socket read size in bytes (indirectly affects filesystem writes; recommendation: keep equal-to or lower-than \033[33m--iobuf\033[0m)")
@@ -1081,12 +1136,16 @@ def add_cert(ap, cert_path):
 
 
 def add_auth(ap):
+    idp_db = os.path.join(E.cfg, "idp.db")
     ses_db = os.path.join(E.cfg, "sessions.db")
     ap2 = ap.add_argument_group('IdP / identity provider / user authentication options')
-    ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks and assume the request-header \033[33mHN\033[0m contains the username of the requesting user (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
+    ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks if the request-header \033[33mHN\033[0m contains a username to associate the request with (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
     ap2.add_argument("--idp-h-grp", metavar="HN", type=u, default="", help="assume the request-header \033[33mHN\033[0m contains the groupname of the requesting user; can be referenced in config files for group-based access control")
     ap2.add_argument("--idp-h-key", metavar="HN", type=u, default="", help="optional but recommended safeguard; your reverse-proxy will insert a secret header named \033[33mHN\033[0m into all requests, and the other IdP headers will be ignored if this header is not present")
     ap2.add_argument("--idp-gsep", metavar="RE", type=u, default="|:;+,", help="if there are multiple groups in \033[33m--idp-h-grp\033[0m, they are separated by one of the characters in \033[33mRE\033[0m")
+    ap2.add_argument("--idp-db", metavar="PATH", type=u, default=idp_db, help="where to store the known IdP users/groups (if you run multiple copyparty instances, make sure they use different DBs)")
+    ap2.add_argument("--idp-store", metavar="N", type=int, default=1, help="how to use \033[33m--idp-db\033[0m; [\033[32m0\033[0m] = entirely disable, [\033[32m1\033[0m] = write-only (effectively disabled), [\033[32m2\033[0m] = remember users, [\033[32m3\033[0m] = remember users and groups.\nNOTE: Will remember and restore the IdP-volumes of all users for all eternity if set to 2 or 3, even when user is deleted from your IdP")
+    ap2.add_argument("--idp-adm", metavar="U,U", type=u, default="", help="comma-separated list of users allowed to use /?idp (the cache management UI)")
     ap2.add_argument("--no-bauth", action="store_true", help="disable basic-authentication support; do not accept passwords from the 'Authenticate' header at all. NOTE: This breaks support for the android app")
     ap2.add_argument("--bauth-last", action="store_true", help="keeps basic-authentication enabled, but only as a last-resort; if a cookie is also provided then the cookie wins")
     ap2.add_argument("--ses-db", metavar="PATH", type=u, default=ses_db, help="where to store the sessions database (if you run multiple copyparty instances, make sure they use different DBs)")
@@ -1167,6 +1226,7 @@ def add_webdav(ap):
     ap2.add_argument("--dav-mac", action="store_true", help="disable apple-garbage filter -- allow macos to create junk files (._* and .DS_Store, .Spotlight-*, .fseventsd, .Trashes, .AppleDouble, __MACOS)")
     ap2.add_argument("--dav-rt", action="store_true", help="show symlink-destination's lastmodified instead of the link itself; always enabled for recursive listings (volflag=davrt)")
     ap2.add_argument("--dav-auth", action="store_true", help="force auth for all folders (required by davfs2 when only some folders are world-readable) (volflag=davauth)")
+    ap2.add_argument("--dav-ua1", metavar="PTN", type=u, default=r" kioworker/", help="regex of tricky user-agents which expect 401 from GET requests; disable with [\033[32mno\033[0m] or blank")
 
 
 def add_tftp(ap):
@@ -1232,7 +1292,9 @@ def add_stats(ap):
 def add_yolo(ap):
     ap2 = ap.add_argument_group('yolo options')
     ap2.add_argument("--allow-csrf", action="store_true", help="disable csrf protections; let other domains/sites impersonate you through cross-site requests")
+    ap2.add_argument("--cookie-lax", action="store_true", help="allow cookies from other domains (if you follow a link from another website into your server, you will arrive logged-in); this reduces protection against CSRF")
     ap2.add_argument("--getmod", action="store_true", help="permit ?move=[...] and ?delete as GET")
+    ap2.add_argument("--wo-up-readme", action="store_true", help="allow users with write-only access to upload logues and readmes without adding the _wo_ filename prefix (volflag=wo_up_readme)")
 
 
 def add_optouts(ap):
@@ -1247,12 +1309,18 @@ def add_optouts(ap):
     ap2.add_argument("-nih", action="store_true", help="no info hostname -- don't show in UI")
     ap2.add_argument("-nid", action="store_true", help="no info disk-usage -- don't show in UI")
     ap2.add_argument("-nb", action="store_true", help="no powered-by-copyparty branding in UI")
-    ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
+    ap2.add_argument("--zipmaxn", metavar="N", type=u, default="0", help="reject download-as-zip if more than \033[33mN\033[0m files in total; optionally takes a unit suffix: [\033[32m256\033[0m], [\033[32m9K\033[0m], [\033[32m4G\033[0m] (volflag=zipmaxn)")
+    ap2.add_argument("--zipmaxs", metavar="SZ", type=u, default="0", help="reject download-as-zip if total download size exceeds \033[33mSZ\033[0m bytes; optionally takes a unit suffix: [\033[32m256M\033[0m], [\033[32m4G\033[0m], [\033[32m2T\033[0m] (volflag=zipmaxs)")
+    ap2.add_argument("--zipmaxt", metavar="TXT", type=u, default="", help="custom errormessage when download size exceeds max (volflag=zipmaxt)")
+    ap2.add_argument("--zipmaxu", action="store_true", help="authenticated users bypass the zip size limit (volflag=zipmaxu)")
+    ap2.add_argument("--zip-who", metavar="LVL", type=int, default=3, help="who can download as zip/tar? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=authenticated-with-read-access, [\033[32m3\033[0m]=everyone-with-read-access (volflag=zip_who)\n\033[1;31mWARNING:\033[0m if a nested volume has a more restrictive value than a parent volume, then this will be \033[33mignored\033[0m if the download is initiated from the parent, more lenient volume")
+    ap2.add_argument("--ua-nozip", metavar="PTN", type=u, default=BAD_BOTS, help="regex of user-agents to reject from download-as-zip/tar; disable with [\033[32mno\033[0m] or blank")
+    ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar; same as \033[33m--zip-who=0\033[0m")
     ap2.add_argument("--no-tarcmp", action="store_true", help="disable download as compressed tar (?tar=gz, ?tar=bz2, ?tar=xz, ?tar=gz:9, ...)")
     ap2.add_argument("--no-lifetime", action="store_true", help="do not allow clients (or server config) to schedule an upload to be deleted after a given time")
-    ap2.add_argument("--no-up-list", action="store_true", help="don't show list of incoming files in controlpanel")
     ap2.add_argument("--no-pipe", action="store_true", help="disable race-the-beam (lockstep download of files which are currently being uploaded) (volflag=nopipe)")
-    ap2.add_argument("--no-db-ip", action="store_true", help="do not write uploader IPs into the database")
+    ap2.add_argument("--no-tail", action="store_true", help="disable streaming a growing files with ?tail (volflag=notail)")
+    ap2.add_argument("--no-db-ip", action="store_true", help="do not write uploader-IP into the database; will also disable unpost, you may want \033[32m--forget-ip\033[0m instead (volflag=no_db_ip)")
 
 
 def add_safety(ap):
@@ -1272,6 +1340,7 @@ def add_safety(ap):
     ap2.add_argument("--no-robots", action="store_true", help="adds http and html headers asking search engines to not index anything (volflag=norobots)")
     ap2.add_argument("--logout", metavar="H", type=float, default=8086.0, help="logout clients after \033[33mH\033[0m hours of inactivity; [\033[32m0.0028\033[0m]=10sec, [\033[32m0.1\033[0m]=6min, [\033[32m24\033[0m]=day, [\033[32m168\033[0m]=week, [\033[32m720\033[0m]=month, [\033[32m8760\033[0m]=year)")
     ap2.add_argument("--ban-pw", metavar="N,W,B", type=u, default="9,60,1440", help="more than \033[33mN\033[0m wrong passwords in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; disable with [\033[32mno\033[0m]")
+    ap2.add_argument("--ban-pwc", metavar="N,W,B", type=u, default="5,60,1440", help="more than \033[33mN\033[0m password-changes in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; disable with [\033[32mno\033[0m]")
     ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="50,60,1440", help="hitting more than \033[33mN\033[0m 404's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; only affects users who cannot see directory listings because their access is either g/G/h")
     ap2.add_argument("--ban-403", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 403's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; [\033[32m1440\033[0m]=day, [\033[32m10080\033[0m]=week, [\033[32m43200\033[0m]=month")
     ap2.add_argument("--ban-422", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 422's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (invalid requests, attempted exploits ++)")
@@ -1294,6 +1363,9 @@ def add_salt(ap, fk_salt, dk_salt, ah_salt):
     ap2.add_argument("--fk-salt", metavar="SALT", type=u, default=fk_salt, help="per-file accesskey salt; used to generate unpredictable URLs for hidden files")
     ap2.add_argument("--dk-salt", metavar="SALT", type=u, default=dk_salt, help="per-directory accesskey salt; used to generate unpredictable URLs to share folders with users who only have the 'get' permission")
     ap2.add_argument("--warksalt", metavar="SALT", type=u, default="hunter2", help="up2k file-hash salt; serves no purpose, no reason to change this (but delete all databases if you do)")
+    ap2.add_argument("--show-ah-salt", action="store_true", help="on startup, print the effective value of \033[33m--ah-salt\033[0m (the autogenerated value in $XDG_CONFIG_HOME unless otherwise specified)")
+    ap2.add_argument("--show-fk-salt", action="store_true", help="on startup, print the effective value of \033[33m--fk-salt\033[0m (the autogenerated value in $XDG_CONFIG_HOME unless otherwise specified)")
+    ap2.add_argument("--show-dk-salt", action="store_true", help="on startup, print the effective value of \033[33m--dk-salt\033[0m (the autogenerated value in $XDG_CONFIG_HOME unless otherwise specified)")
 
 
 def add_shutdown(ap):
@@ -1326,12 +1398,16 @@ def add_admin(ap):
     ap2.add_argument("--no-reload", action="store_true", help="disable ?reload=cfg (reload users/volumes/volflags from config file)")
     ap2.add_argument("--no-rescan", action="store_true", help="disable ?scan (volume reindexing)")
     ap2.add_argument("--no-stack", action="store_true", help="disable ?stack (list all stacks)")
+    ap2.add_argument("--no-ups-page", action="store_true", help="disable ?ru (list of recent uploads)")
+    ap2.add_argument("--no-up-list", action="store_true", help="don't show list of incoming files in controlpanel")
     ap2.add_argument("--dl-list", metavar="LVL", type=int, default=2, help="who can see active downloads in the controlpanel? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=everyone")
+    ap2.add_argument("--ups-who", metavar="LVL", type=int, default=2, help="who can see recent uploads on the ?ru page? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=everyone (volflag=ups_who)")
+    ap2.add_argument("--ups-when", action="store_true", help="let everyone see upload timestamps on the ?ru page, not just admins")
 
 
 def add_thumbnail(ap):
     th_ram = (RAM_AVAIL or RAM_TOTAL or 9) * 0.6
-    th_ram = int(max(min(th_ram, 6), 1) * 10) / 10
+    th_ram = int(max(min(th_ram, 6), 0.3) * 10) / 10
     ap2 = ap.add_argument_group('thumbnail options')
     ap2.add_argument("--no-thumb", action="store_true", help="disable all thumbnails (volflag=dthumb)")
     ap2.add_argument("--no-vthumb", action="store_true", help="disable video thumbnails (volflag=dvthumb)")
@@ -1358,7 +1434,8 @@ def add_thumbnail(ap):
     ap2.add_argument("--th-r-vips", metavar="T,T", type=u, default="avif,exr,fit,fits,fts,gif,hdr,heic,jp2,jpeg,jpg,jpx,jxl,nii,pfm,pgm,png,ppm,svg,tif,tiff,webp", help="image formats to decode using pyvips")
     ap2.add_argument("--th-r-ffi", metavar="T,T", type=u, default="apng,avif,avifs,bmp,cbz,dds,dib,fit,fits,fts,gif,hdr,heic,heics,heif,heifs,icns,ico,jp2,jpeg,jpg,jpx,jxl,pbm,pcx,pfm,pgm,png,pnm,ppm,psd,qoi,sgi,tga,tif,tiff,webp,xbm,xpm", help="image formats to decode using ffmpeg")
     ap2.add_argument("--th-r-ffv", metavar="T,T", type=u, default="3gp,asf,av1,avc,avi,flv,h264,h265,hevc,m4v,mjpeg,mjpg,mkv,mov,mp4,mpeg,mpeg2,mpegts,mpg,mpg2,mts,nut,ogm,ogv,rm,ts,vob,webm,wmv", help="video formats to decode using ffmpeg")
-    ap2.add_argument("--th-r-ffa", metavar="T,T", type=u, default="aac,ac3,aif,aiff,alac,alaw,amr,apac,ape,au,bonk,dfpwm,dts,flac,gsm,ilbc,it,itgz,itxz,itz,m4a,mdgz,mdxz,mdz,mo3,mod,mp2,mp3,mpc,mptm,mt2,mulaw,ogg,okt,opus,ra,s3m,s3gz,s3xz,s3z,tak,tta,ulaw,wav,wma,wv,xm,xmgz,xmxz,xmz,xpk", help="audio formats to decode using ffmpeg")
+    ap2.add_argument("--th-r-ffa", metavar="T,T", type=u, default="aac,ac3,aif,aiff,alac,alaw,amr,apac,ape,au,bonk,dfpwm,dts,flac,gsm,ilbc,it,itgz,itxz,itz,m4a,mdgz,mdxz,mdz,mo3,mod,mp2,mp3,mpc,mptm,mt2,mulaw,oga,ogg,okt,opus,ra,s3m,s3gz,s3xz,s3z,tak,tta,ulaw,wav,wma,wv,xm,xmgz,xmxz,xmz,xpk", help="audio formats to decode using ffmpeg")
+    ap2.add_argument("--th-spec-cnv", metavar="T", type=u, default="it,itgz,itxz,itz,mdgz,mdxz,mdz,mo3,mod,s3m,s3gz,s3xz,s3z,xm,xmgz,xmxz,xmz,xpk", help="audio formats which provoke https://trac.ffmpeg.org/ticket/10797 (huge ram usage for s3xmodit spectrograms)")
     ap2.add_argument("--au-unpk", metavar="E=F.C", type=u, default="mdz=mod.zip, mdgz=mod.gz, mdxz=mod.xz, s3z=s3m.zip, s3gz=s3m.gz, s3xz=s3m.xz, xmz=xm.zip, xmgz=xm.gz, xmxz=xm.xz, itz=it.zip, itgz=it.gz, itxz=it.xz, cbz=jpg.cbz", help="audio/image formats to decompress before passing to ffmpeg")
 
 
@@ -1366,14 +1443,26 @@ def add_transcoding(ap):
     ap2 = ap.add_argument_group('transcoding options')
     ap2.add_argument("--q-opus", metavar="KBPS", type=int, default=128, help="target bitrate for transcoding to opus; set 0 to disable")
     ap2.add_argument("--q-mp3", metavar="QUALITY", type=u, default="q2", help="target quality for transcoding to mp3, for example [\033[32m192k\033[0m] (CBR) or [\033[32mq0\033[0m] (CQ/CRF, q0=maxquality, q9=smallest); set 0 to disable")
+    ap2.add_argument("--no-caf", action="store_true", help="disable transcoding to caf-opus (affects iOS v12~v17), will use mp3 instead")
+    ap2.add_argument("--no-owa", action="store_true", help="disable transcoding to webm-opus (iOS v18 and later), will use mp3 instead")
     ap2.add_argument("--no-acode", action="store_true", help="disable audio transcoding")
     ap2.add_argument("--no-bacode", action="store_true", help="disable batch audio transcoding by folder download (zip/tar)")
     ap2.add_argument("--ac-maxage", metavar="SEC", type=int, default=86400, help="delete cached transcode output after \033[33mSEC\033[0m seconds")
 
 
+def add_tail(ap):
+    ap2 = ap.add_argument_group('tailing options (realtime streaming of a growing file)')
+    ap2.add_argument("--tail-who", metavar="LVL", type=int, default=2, help="who can tail? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=authenticated-with-read-access, [\033[32m3\033[0m]=everyone-with-read-access (volflag=tail_who)")
+    ap2.add_argument("--tail-cmax", metavar="N", type=int, default=64, help="do not allow starting a new tail if more than \033[33mN\033[0m active downloads")
+    ap2.add_argument("--tail-tmax", metavar="SEC", type=float, default=0, help="terminate connection after \033[33mSEC\033[0m seconds; [\033[32m0\033[0m]=never (volflag=tail_tmax)")
+    ap2.add_argument("--tail-rate", metavar="SEC", type=float, default=0.2, help="check for new data every \033[33mSEC\033[0m seconds (volflag=tail_rate)")
+    ap2.add_argument("--tail-ka", metavar="SEC", type=float, default=3.0, help="send a zerobyte if connection is idle for \033[33mSEC\033[0m seconds to prevent disconnect")
+    ap2.add_argument("--tail-fd", metavar="SEC", type=float, default=1.0, help="check if file was replaced (new fd) if idle for \033[33mSEC\033[0m seconds (volflag=tail_fd)")
+
+
 def add_rss(ap):
     ap2 = ap.add_argument_group('RSS options')
-    ap2.add_argument("--rss", action="store_true", help="enable RSS output (experimental)")
+    ap2.add_argument("--rss", action="store_true", help="enable RSS output (experimental) (volflag=rss)")
     ap2.add_argument("--rss-nf", metavar="HITS", type=int, default=250, help="default number of files to return (url-param 'nf')")
     ap2.add_argument("--rss-fext", metavar="E,E", type=u, default="", help="default list of file extensions to include (url-param 'fext'); blank=all")
     ap2.add_argument("--rss-sort", metavar="ORD", type=u, default="m", help="default sort order (url-param 'sort'); [\033[32mm\033[0m]=last-modified [\033[32mu\033[0m]=upload-time [\033[32mn\033[0m]=filename [\033[32ms\033[0m]=filesize; Uppercase=oldest-first. Note that upload-time is 0 for non-uploaded files")
@@ -1389,6 +1478,7 @@ def add_db_general(ap, hcores):
     ap2.add_argument("-e2vu", action="store_true", help="on hash mismatch: update the database with the new hash")
     ap2.add_argument("-e2vp", action="store_true", help="on hash mismatch: panic and quit copyparty")
     ap2.add_argument("--hist", metavar="PATH", type=u, default="", help="where to store volume data (db, thumbs); default is a folder named \".hist\" inside each volume (volflag=hist)")
+    ap2.add_argument("--dbpath", metavar="PATH", type=u, default="", help="override where the volume databases are to be placed; default is the same as \033[33m--hist\033[0m (volflag=dbpath)")
     ap2.add_argument("--no-hash", metavar="PTN", type=u, default="", help="regex: disable hashing of matching absolute-filesystem-paths during e2ds folder scans (volflag=nohash)")
     ap2.add_argument("--no-idx", metavar="PTN", type=u, default=noidx, help="regex: disable indexing of matching absolute-filesystem-paths during e2ds folder scans (volflag=noidx)")
     ap2.add_argument("--no-dirsz", action="store_true", help="do not show total recursive size of folders in listings, show inode size instead; slightly faster (volflag=nodirsz)")
@@ -1396,6 +1486,7 @@ def add_db_general(ap, hcores):
     ap2.add_argument("--no-dhash", action="store_true", help="disable rescan acceleration; do full database integrity check -- makes the db ~5%% smaller and bootup/rescans 3~10x slower")
     ap2.add_argument("--re-dhash", action="store_true", help="force a cache rebuild on startup; enable this once if it gets out of sync (should never be necessary)")
     ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice -- only useful for offloading uploads to a cloud service or something (volflag=noforget)")
+    ap2.add_argument("--forget-ip", metavar="MIN", type=int, default=0, help="remove uploader-IP from database (and make unpost impossible) \033[33mMIN\033[0m minutes after upload, for GDPR reasons. Default [\033[32m0\033[0m] is never-forget. [\033[32m1440\033[0m]=day, [\033[32m10080\033[0m]=week, [\033[32m43200\033[0m]=month. (volflag=forget_ip)")
     ap2.add_argument("--dbd", metavar="PROFILE", default="wal", help="database durability profile; sets the tradeoff between robustness and speed, see \033[33m--help-dbd\033[0m (volflag=dbd)")
     ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (probably buggy, not recommended) (volflag=xlink)")
     ap2.add_argument("--hash-mt", metavar="CORES", type=int, default=hcores, help="num cpu cores to use for file hashing; set 0 or 1 for single-core hashing")
@@ -1426,11 +1517,13 @@ def add_db_metadata(ap):
 
 def add_txt(ap):
     ap2 = ap.add_argument_group('textfile options')
+    ap2.add_argument("--md-hist", metavar="TXT", type=u, default="s", help="where to store old version of markdown files; [\033[32ms\033[0m]=subfolder, [\033[32mv\033[0m]=volume-histpath, [\033[32mn\033[0m]=nope/disabled (volflag=md_hist)")
     ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="the textfile editor will check for serverside changes every \033[33mSEC\033[0m seconds")
     ap2.add_argument("-emp", action="store_true", help="enable markdown plugins -- neat but dangerous, big XSS risk")
     ap2.add_argument("--exp", action="store_true", help="enable textfile expansion -- replace {{self.ip}} and such; see \033[33m--help-exp\033[0m (volflag=exp)")
     ap2.add_argument("--exp-md", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in markdown files; add/remove stuff on the default list with +hdr_foo or /vf.scan (volflag=exp_md)")
     ap2.add_argument("--exp-lg", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in prologue/epilogue files (volflag=exp_lg)")
+    ap2.add_argument("--ua-nodoc", metavar="PTN", type=u, default=BAD_BOTS, help="regex of user-agents to reject from viewing documents through ?doc=[...]; disable with [\033[32mno\033[0m] or blank")
 
 
 def add_og(ap):
@@ -1462,9 +1555,13 @@ def add_ui(ap, retry):
     ap2.add_argument("--sort", metavar="C,C,C", type=u, default="href", help="default sort order, comma-separated column IDs (see header tooltips), prefix with '-' for descending. Examples: \033[32mhref -href ext sz ts tags/Album tags/.tn\033[0m (volflag=sort)")
     ap2.add_argument("--nsort", action="store_true", help="default-enable natural sort of filenames with leading numbers (volflag=nsort)")
     ap2.add_argument("--hsortn", metavar="N", type=int, default=2, help="number of sorting rules to include in media URLs by default (volflag=hsortn)")
-    ap2.add_argument("--unlist", metavar="REGEX", type=u, default="", help="don't show files matching \033[33mREGEX\033[0m in file list. Purely cosmetic! Does not affect API calls, just the browser. Example: [\033[32m\\.(js|css)$\033[0m] (volflag=unlist)")
+    ap2.add_argument("--see-dots", action="store_true", help="default-enable seeing dotfiles; only takes effect if user has the necessary permissions")
+    ap2.add_argument("--qdel", metavar="LVL", type=int, default=2, help="number of confirmations to show when deleting files (2/1/0)")
+    ap2.add_argument("--unlist", metavar="REGEX", type=u, default="", help="don't show files/folders matching \033[33mREGEX\033[0m in file list. WARNING: Purely cosmetic! Does not affect API calls, just the browser. Example: [\033[32m\\.(js|css)$\033[0m] (volflag=unlist)")
     ap2.add_argument("--favico", metavar="TXT", type=u, default="c 000 none" if retry else "🎉 000 none", help="\033[33mfavicon-text\033[0m [ \033[33mforeground\033[0m [ \033[33mbackground\033[0m ] ], set blank to disable")
+    ap2.add_argument("--ext-th", metavar="E=VP", type=u, action="append", help="use thumbnail-image \033[33mVP\033[0m for file-extension \033[33mE\033[0m, example: [\033[32mexe=/.res/exe.png\033[0m] (volflag=ext_th)")
     ap2.add_argument("--mpmc", metavar="URL", type=u, default="", help="change the mediaplayer-toggle mouse cursor; URL to a folder with {2..5}.png inside (or disable with [\033[32m.\033[0m])")
+    ap2.add_argument("--spinner", metavar="TXT", type=u, default="🌲", help="\033[33memoji\033[0m or \033[33memoji,css\033[0m Example: [\033[32m🥖,padding:0\033[0m]")
     ap2.add_argument("--css-browser", metavar="L", type=u, default="", help="URL to additional CSS to include in the filebrowser html")
     ap2.add_argument("--js-browser", metavar="L", type=u, default="", help="URL to additional JS to include in the filebrowser html")
     ap2.add_argument("--js-other", metavar="L", type=u, default="", help="URL to additional JS to include in all other pages")
@@ -1474,14 +1571,17 @@ def add_ui(ap, retry):
     ap2.add_argument("--txt-max", metavar="KiB", type=int, default=64, help="max size of embedded textfiles on ?doc= (anything bigger will be lazy-loaded by JS)")
     ap2.add_argument("--doctitle", metavar="TXT", type=u, default="copyparty @ --name", help="title / service-name to show in html documents")
     ap2.add_argument("--bname", metavar="TXT", type=u, default="--name", help="server name (displayed in filebrowser document title)")
-    ap2.add_argument("--pb-url", metavar="URL", type=u, default="https://github.com/9001/copyparty", help="powered-by link; disable with \033[33m-np\033[0m")
+    ap2.add_argument("--pb-url", metavar="URL", type=u, default=URL_PRJ, help="powered-by link; disable with \033[33m-nb\033[0m")
     ap2.add_argument("--ver", action="store_true", help="show version on the control panel (incompatible with \033[33m-nb\033[0m)")
     ap2.add_argument("--k304", metavar="NUM", type=int, default=0, help="configure the option to enable/disable k304 on the controlpanel (workaround for buggy reverse-proxies); [\033[32m0\033[0m] = hidden and default-off, [\033[32m1\033[0m] = visible and default-off, [\033[32m2\033[0m] = visible and default-on")
     ap2.add_argument("--no304", metavar="NUM", type=int, default=0, help="configure the option to enable/disable no304 on the controlpanel (workaround for buggy caching in browsers); [\033[32m0\033[0m] = hidden and default-off, [\033[32m1\033[0m] = visible and default-off, [\033[32m2\033[0m] = visible and default-on")
-    ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#attr-sandbox")
-    ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for prologue/epilogue docs  (volflag=lg_sbf)")
+    ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to allow in the iframe 'sandbox' attribute for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#sandbox")
+    ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to allow in the iframe 'sandbox' attribute for prologue/epilogue docs (volflag=lg_sbf)")
+    ap2.add_argument("--md-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for README.md docs, for example [\033[32mfullscreen\033[0m] (volflag=md_sba)")
+    ap2.add_argument("--lg-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for prologue/epilogue docs (volflag=lg_sba); see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes")
     ap2.add_argument("--no-sb-md", action="store_true", help="don't sandbox README/PREADME.md documents (volflags: no_sb_md | sb_md)")
     ap2.add_argument("--no-sb-lg", action="store_true", help="don't sandbox prologue/epilogue docs (volflags: no_sb_lg | sb_lg); enables non-js support")
+    ap2.add_argument("--have-unlistc", action="store_true", help=argparse.SUPPRESS)
 
 
 def add_debug(ap):
@@ -1505,6 +1605,7 @@ def add_debug(ap):
     ap2.add_argument("--bf-nc", metavar="NUM", type=int, default=200, help="bak-flips: stop if there's more than \033[33mNUM\033[0m files at \033[33m--kf-dir\033[0m already; default: 6.3 GiB max (200*32M)")
     ap2.add_argument("--bf-dir", metavar="PATH", type=u, default="bf", help="bak-flips: store corrupted chunks at \033[33mPATH\033[0m; default: folder named 'bf' wherever copyparty was started")
     ap2.add_argument("--bf-log", metavar="PATH", type=u, default="", help="bak-flips: log corruption info to a textfile at \033[33mPATH\033[0m")
+    ap2.add_argument("--no-cfg-cmt-warn", action="store_true", help=argparse.SUPPRESS)
 
 
 # fmt: on
@@ -1522,9 +1623,9 @@ def run_argparse(
 
     cert_path = os.path.join(E.cfg, "cert.pem")
 
-    fk_salt = get_fk_salt()
-    dk_salt = get_dk_salt()
-    ah_salt = get_ah_salt()
+    fk_salt = get_salt("fk", 18)
+    dk_salt = get_salt("dk", 30)
+    ah_salt = get_salt("ah", 18)
 
     # alpine peaks at 5 threads for some reason,
     # all others scale past that (but try to avoid SMT),
@@ -1566,6 +1667,7 @@ def run_argparse(
     add_hooks(ap)
     add_stats(ap)
     add_txt(ap)
+    add_tail(ap)
     add_og(ap)
     add_ui(ap, retry)
     add_admin(ap)
@@ -1737,7 +1839,7 @@ def main(argv: Optional[list[str]] = None) -> None:
         except:
             lprint("\nfailed to disable quick-edit-mode:\n" + min_ex() + "\n")
 
-    if al.ansi:
+    if not al.ansi:
         al.wintitle = ""
 
     # propagate implications

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 16, 4)
-CODENAME = "COPYparty"
-BUILD_DT = (2024, 12, 7)
+VERSION = (1, 18, 7)
+CODENAME = "logtail"
+BUILD_DT = (2025, 7, 30)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/bos/bos.py

@@ -9,8 +9,11 @@ from . import path as path
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Optional
 
-_ = (path,)
-__all__ = ["path"]
+MKD_755 = {"chmod_d": 0o755}
+MKD_700 = {"chmod_d": 0o700}
+
+_ = (path, MKD_755, MKD_700)
+__all__ = ["path", "MKD_755", "MKD_700"]
 
 # grep -hRiE '(^|[^a-zA-Z_\.-])os\.' . | gsed -r 's/ /\n/g;s/\(/(\n/g' | grep -hRiE '(^|[^a-zA-Z_\.-])os\.' | sort | uniq -c
 # printf 'os\.(%s)' "$(grep ^def bos/__init__.py | gsed -r 's/^def //;s/\(.*//' | tr '\n' '|' | gsed -r 's/.$//')"
@@ -20,19 +23,39 @@ def chmod(p: str, mode: int) -> None:
     return os.chmod(fsenc(p), mode)
 
 
+def chown(p: str, uid: int, gid: int) -> None:
+    return os.chown(fsenc(p), uid, gid)
+
+
 def listdir(p: str = ".") -> list[str]:
     return [fsdec(x) for x in os.listdir(fsenc(p))]
 
 
-def makedirs(name: str, mode: int = 0o755, exist_ok: bool = True) -> bool:
+def makedirs(name: str, vf: dict[str, Any] = MKD_755, exist_ok: bool = True) -> bool:
+    # os.makedirs does 777 for all but leaf; this does mode on all
+    todo = []
     bname = fsenc(name)
-    try:
-        os.makedirs(bname, mode)
-        return True
-    except:
-        if not exist_ok or not os.path.isdir(bname):
-            raise
+    while bname:
+        if os.path.isdir(bname):
+            break
+        todo.append(bname)
+        bname = os.path.dirname(bname)
+    if not todo:
+        if not exist_ok:
+            os.mkdir(bname)  # to throw
         return False
+    mode = vf["chmod_d"]
+    chown = "chown" in vf
+    for zb in todo[::-1]:
+        try:
+            os.mkdir(zb, mode)
+            if chown:
+                os.chown(zb, vf["uid"], vf["gid"])
+        except:
+            if os.path.isdir(zb):
+                continue
+            raise
+    return True
 
 
 def mkdir(p: str, mode: int = 0o755) -> None:

copyparty/cert.py

@@ -1,13 +1,11 @@
 import calendar
 import errno
-import filecmp
 import json
 import os
-import shutil
 import time
 
 from .__init__ import ANYWIN
-from .util import Netdev, load_resource, runcmd, wrename, wunlink
+from .util import Netdev, atomic_move, load_resource, runcmd, wunlink
 
 HAVE_CFSSL = not os.environ.get("PRTY_NO_CFSSL")
 
@@ -122,7 +120,7 @@ def _gen_ca(log: "RootLogger", args):
         wunlink(nlog, bname + ".key", VF)
     except:
         pass
-    wrename(nlog, bname + "-key.pem", bname + ".key", VF)
+    atomic_move(nlog, bname + "-key.pem", bname + ".key", VF)
     wunlink(nlog, bname + ".csr", VF)
 
     log("cert", "new ca OK", 2)
@@ -215,7 +213,7 @@ def _gen_srv(log: "RootLogger", args, netdevs: dict[str, Netdev]):
         wunlink(nlog, bname + ".key", VF)
     except:
         pass
-    wrename(nlog, bname + "-key.pem", bname + ".key", VF)
+    atomic_move(nlog, bname + "-key.pem", bname + ".key", VF)
     wunlink(nlog, bname + ".csr", VF)
 
     with open(os.path.join(args.crt_dir, "ca.pem"), "rb") as f:

copyparty/cfg.py

@@ -5,6 +5,9 @@ from __future__ import print_function, unicode_literals
 zs = "a c e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vp e2vu ed emp i j lo mcr mte mth mtm mtp nb nc nid nih nth nw p q s ss sss v z zv"
 onedash = set(zs.split())
 
+# verify that all volflags are documented here:
+# grep volflag= __main__.py | sed -r 's/.*volflag=//;s/\).*//' | sort | uniq | while IFS= read -r x; do grep -E "\"$x(=[^ \"]+)?\": \"" cfg.py || printf '%s\n' "$x"; done
+
 
 def vf_bmap() -> dict[str, str]:
     """argv-to-volflag: simple bools"""
@@ -19,6 +22,7 @@ def vf_bmap() -> dict[str, str]:
         "no_forget": "noforget",
         "no_pipe": "nopipe",
         "no_robots": "norobots",
+        "no_tail": "notail",
         "no_thumb": "dthumb",
         "no_vthumb": "dvthumb",
         "no_athumb": "dathumb",
@@ -40,6 +44,7 @@ def vf_bmap() -> dict[str, str]:
         "gsel",
         "hardlink",
         "magic",
+        "no_db_ip",
         "no_sb_md",
         "no_sb_lg",
         "nsort",
@@ -47,10 +52,14 @@ def vf_bmap() -> dict[str, str]:
         "og_no_head",
         "og_s_title",
         "rand",
+        "reflink",
+        "rmagic",
         "rss",
+        "wo_up_readme",
         "xdev",
         "xlink",
         "xvol",
+        "zipmaxu",
     ):
         ret[k] = k
     return ret
@@ -69,12 +78,20 @@ def vf_vmap() -> dict[str, str]:
         "th_x3": "th3x",
     }
     for k in (
+        "bup_ck",
+        "chmod_d",
+        "chmod_f",
         "dbd",
+        "forget_ip",
         "hsortn",
         "html_head",
         "lg_sbf",
         "md_sbf",
+        "lg_sba",
+        "md_sba",
+        "md_hist",
         "nrand",
+        "u2ow",
         "og_desc",
         "og_site",
         "og_th",
@@ -84,13 +101,26 @@ def vf_vmap() -> dict[str, str]:
         "og_title_i",
         "og_tpl",
         "og_ua",
+        "put_ck",
+        "put_name",
         "mv_retry",
         "rm_retry",
         "sort",
+        "tail_fd",
+        "tail_rate",
+        "tail_tmax",
+        "tail_who",
         "tcolor",
         "unlist",
         "u2abort",
         "u2ts",
+        "uid",
+        "gid",
+        "ups_who",
+        "zip_who",
+        "zipmaxn",
+        "zipmaxs",
+        "zipmaxt",
     ):
         ret[k] = k
     return ret
@@ -102,6 +132,7 @@ def vf_cmap() -> dict[str, str]:
     for k in (
         "exp_lg",
         "exp_md",
+        "ext_th",
         "mte",
         "mth",
         "mtp",
@@ -140,14 +171,24 @@ flagcats = {
         "dedup": "enable symlink-based file deduplication",
         "hardlink": "enable hardlink-based file deduplication,\nwith fallback on symlinks when that is impossible",
         "hardlinkonly": "dedup with hardlink only, never symlink;\nmake a full copy if hardlink is impossible",
+        "reflink": "enable reflink-based file deduplication,\nwith fallback on full copy when that is impossible",
         "safededup": "verify on-disk data before using it for dedup",
         "noclone": "take dupe data from clients, even if available on HDD",
         "nodupe": "rejects existing files (instead of linking/cloning them)",
+        "chmod_d=755": "unix-permission for new dirs/folders",
+        "chmod_f=644": "unix-permission for new files",
+        "uid=573": "change owner of new files/folders to unix-user 573",
+        "gid=999": "change owner of new files/folders to unix-group 999",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
+        "nosparse": "deny use of sparse files, mainly for slow storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
         "nosub": "forces all uploads into the top folder of the vfs",
         "magic": "enables filetype detection for nameless uploads",
-        "gz": "allows server-side gzip of uploads with ?gz (also c,xz)",
+        "put_name": "fallback filename for nameless uploads",
+        "put_ck": "default checksum-hasher for PUT/WebDAV uploads",
+        "bup_ck": "default checksum-hasher for bup/basic uploads",
+        "gz": "allows server-side gzip compression of uploads with ?gz",
+        "xz": "allows server-side lzma compression of uploads with ?xz",
         "pk": "forces server-side compression, optional arg: xz,9",
     },
     "upload rules": {
@@ -156,8 +197,10 @@ flagcats = {
         "vmaxb=1g": "total volume size max 1 GiB (suffixes: b, k, m, g, t)",
         "vmaxn=4k": "max 4096 files in volume (suffixes: b, k, m, g, t)",
         "medialinks": "return medialinks for non-up2k uploads (not hotlinks)",
+        "wo_up_readme": "write-only users can upload logues without getting renamed",
         "rand": "force randomized filenames, 9 chars long by default",
         "nrand=N": "randomized filenames are N chars long",
+        "u2ow=N": "overwrite existing files? 0=no 1=if-older 2=always",
         "u2ts=fc": "[f]orce [c]lient-last-modified or [u]pload-time",
         "u2abort=1": "allow aborting unfinished uploads? 0=no 1=strict 2=ip-chk 3=acct-chk",
         "sz=1k-3m": "allow filesizes between 1 KiB and 3MiB",
@@ -174,17 +217,24 @@ flagcats = {
         "e2dsa": "scans all folders for new files on startup; also sets -e2d",
         "e2t": "enable multimedia indexing; makes it possible to search for tags",
         "e2ts": "scan existing files for tags on startup; also sets -e2t",
-        "e2tsa": "delete all metadata from DB (full rescan); also sets -e2ts",
+        "e2tsr": "delete all metadata from DB (full rescan); also sets -e2ts",
         "d2ts": "disables metadata collection for existing files",
+        "e2v": "verify integrity on startup by hashing files and comparing to db",
+        "e2vu": "when e2v fails, update the db (assume on-disk files are good)",
+        "e2vp": "when e2v fails, panic and quit copyparty",
         "d2ds": "disables onboot indexing, overrides -e2ds*",
         "d2t": "disables metadata collection, overrides -e2t*",
         "d2v": "disables file verification, overrides -e2v*",
         "d2d": "disables all database stuff, overrides -e2*",
         "hist=/tmp/cdb": "puts thumbnails and indexes at that location",
+        "dbpath=/tmp/cdb": "puts indexes at that location",
+        "landmark=foo": "disable db if file foo doesn't exist",
         "scan=60": "scan for new files every 60sec, same as --re-maxage",
         "nohash=\\.iso$": "skips hashing file contents if path matches *.iso",
         "noidx=\\.iso$": "fully ignores the contents at paths matching *.iso",
         "noforget": "don't forget files when deleted from disk",
+        "forget_ip=43200": "forget uploader-IP after 30 days (GDPR)",
+        "no_db_ip": "never store uploader-IP in the db; disables unpost",
         "fat32": "avoid excessive reindexing on android sdcardfs",
         "dbd=[acid|swal|wal|yolo]": "database speed-durability tradeoff",
         "xlink": "cross-volume dupe detection / linking (dangerous)",
@@ -195,6 +245,8 @@ flagcats = {
         "srch_excl": "exclude search results with URL matching this regex",
     },
     'database, audio tags\n"mte", "mth", "mtp", "mtm" all work the same as -mte, -mth, ...': {
+        "mte=artist,title": "media-tags to index/display",
+        "mth=fmt,res,ac": "media-tags to hide by default",
         "mtp=.bpm=f,audio-bpm.py": 'uses the "audio-bpm.py" program to\ngenerate ".bpm" tags from uploads (f = overwrite tags)',
         "mtp=ahash,vhash=media-hash.py": "collects two tags at once",
     },
@@ -208,6 +260,7 @@ flagcats = {
         "crop": "center-cropping (y/n/fy/fn)",
         "th3x": "3x resolution (y/n/fy/fn)",
         "convt": "conversion timeout in seconds",
+        "ext_th=s=/b.png": "use /b.png as thumbnail for file-extension s",
     },
     "handlers\n(better explained in --help-handlers)": {
         "on404=PY": "handle 404s by executing PY file",
@@ -230,22 +283,69 @@ flagcats = {
         "grid": "show grid/thumbnails by default",
         "gsel": "select files in grid by ctrl-click",
         "sort": "default sort order",
+        "nsort": "natural-sort of leading digits in filenames",
+        "hsortn": "number of sort-rules to add to media URLs",
         "unlist": "dont list files matching REGEX",
         "html_head=TXT": "includes TXT in the <head>, or @PATH for file at PATH",
+        "tcolor=#fc0": "theme color (a hint for webbrowsers, discord, etc.)",
+        "nodirsz": "don't show total folder size",
         "robots": "allows indexing by search engines (default)",
         "norobots": "kindly asks search engines to leave",
+        "unlistcr": "don't list read-access in controlpanel",
+        "unlistcw": "don't list write-access in controlpanel",
         "no_sb_md": "disable js sandbox for markdown files",
         "no_sb_lg": "disable js sandbox for prologue/epilogue",
         "sb_md": "enable js sandbox for markdown files (default)",
         "sb_lg": "enable js sandbox for prologue/epilogue (default)",
         "md_sbf": "list of markdown-sandbox safeguards to disable",
         "lg_sbf": "list of *logue-sandbox safeguards to disable",
+        "md_sba": "value of iframe allow-prop for markdown-sandbox",
+        "lg_sba": "value of iframe allow-prop for *logue-sandbox",
         "nohtml": "return html and markdown as text/html",
     },
+    "opengraph (discord embeds)": {
+        "og": "enable OG (disables hotlinking)",
+        "og_site": "sitename; defaults to --name, disable with '-'",
+        "og_desc": "description text for all files; disable with '-'",
+        "og_th=jf": "thumbnail format; j / jf / jf3 / w / w3 / ...",
+        "og_title_a": "audio title format; default: {{ artist }} - {{ title }}",
+        "og_title_v": "video title format; default: {{ title }}",
+        "og_title_i": "image title format; default: {{ title }}",
+        "og_title=foo": "fallback title if there's nothing in the db",
+        "og_s_title": "force default title; do not read from tags",
+        "og_tpl": "custom html; see --og-tpl in --help",
+        "og_no_head": "you want to add tags manually with og_tpl",
+        "og_ua": "if defined: only send OG html if useragent matches this regex",
+    },
+    "textfiles": {
+        "md_hist": "where to put markdown backups; s=subfolder, v=volHist, n=nope",
+        "exp": "enable textfile expansion; see --help-exp",
+        "exp_md": "placeholders to expand in markdown files; see --help",
+        "exp_lg": "placeholders to expand in prologue/epilogue; see --help",
+    },
+    "tailing": {
+        "notail": "disable ?tail (download a growing file continuously)",
+        "tail_fd=1": "check if file was replaced (new fd) every 1 sec",
+        "tail_rate=0.2": "check for new data every 0.2 sec",
+        "tail_tmax=30": "kill connection after 30 sec",
+        "tail_who=2": "restrict ?tail access (1=admins,2=authed,3=everyone)",
+    },
     "others": {
         "dots": "allow all users with read-access to\nenable the option to show dotfiles in listings",
         "fk=8": 'generates per-file accesskeys,\nwhich are then required at the "g" permission;\nkeys are invalidated if filesize or inode changes',
         "fka=8": 'generates slightly weaker per-file accesskeys,\nwhich are then required at the "g" permission;\nnot affected by filesize or inode numbers',
+        "dk=8": 'generates per-directory accesskeys,\nwhich are then required at the "g" permission;\nkeys are invalidated if filesize or inode changes',
+        "dks": "per-directory accesskeys allow browsing into subdirs",
+        "dky": 'allow seeing files (not folders) inside a specific folder\nwith "g" perm, and does not require a valid dirkey to do so',
+        "rss": "allow '?rss' URL suffix (experimental)",
+        "rmagic": "expensive analysis for mimetype accuracy",
+        "ups_who=2": "restrict viewing the list of recent uploads",
+        "zip_who=2": "restrict access to download-as-zip/tar",
+        "zipmaxn=9k": "reject download-as-zip if more than 9000 files",
+        "zipmaxs=2g": "reject download-as-zip if size over 2 GiB",
+        "zipmaxt=no": "reply with 'no' if download-as-zip exceeds max",
+        "zipmaxu": "zip-size-limit does not apply to authenticated users",
+        "nopipe": "disable race-the-beam (download unfinished uploads)",
         "mv_retry": "ms-windows: timeout for renaming busy files",
         "rm_retry": "ms-windows: timeout for deleting busy files",
         "davauth": "ask webdav clients to login for all folders",
@@ -255,3 +355,10 @@ flagcats = {
 
 
 flagdescs = {k.split("=")[0]: v for tab in flagcats.values() for k, v in tab.items()}
+
+
+if True:  # so it gets removed in release-builds
+    for fun in [vf_bmap, vf_cmap, vf_vmap]:
+        for k in fun().values():
+            if k not in flagdescs:
+                raise Exception("undocumented volflag: " + k)

copyparty/dxml.py

@@ -1,3 +1,6 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
 import importlib
 import sys
 import xml.etree.ElementTree as ET
@@ -8,6 +11,10 @@ if True:  # pylint: disable=using-constant-test
     from typing import Any, Optional
 
 
+class BadXML(Exception):
+    pass
+
+
 def get_ET() -> ET.XMLParser:
     pn = "xml.etree.ElementTree"
     cn = "_elementtree"
@@ -34,7 +41,7 @@ def get_ET() -> ET.XMLParser:
 XMLParser: ET.XMLParser = get_ET()
 
 
-class DXMLParser(XMLParser):  # type: ignore
+class _DXMLParser(XMLParser):  # type: ignore
     def __init__(self) -> None:
         tb = ET.TreeBuilder()
         super(DXMLParser, self).__init__(target=tb)
@@ -49,8 +56,12 @@ class DXMLParser(XMLParser):  # type: ignore
         raise BadXML("{}, {}".format(a, ka))
 
 
-class BadXML(Exception):
-    pass
+class _NG(XMLParser):  # type: ignore
+    def __int__(self) -> None:
+        raise BadXML("dxml selftest failed")
+
+
+DXMLParser = _DXMLParser
 
 
 def parse_xml(txt: str) -> ET.Element:
@@ -59,6 +70,40 @@ def parse_xml(txt: str) -> ET.Element:
     return parser.close()  # type: ignore
 
 
+def selftest() -> bool:
+    qbe = r"""<!DOCTYPE d [
+<!ENTITY a "nice_bakuretsu">
+]>
+<root>&a;&a;&a;</root>"""
+
+    emb = r"""<!DOCTYPE d [
+<!ENTITY a SYSTEM "file:///etc/hostname">
+]>
+<root>&a;</root>"""
+
+    # future-proofing; there's never been any known vulns
+    # regarding DTDs and ET.XMLParser, but might as well
+    # block them since webdav-clients don't use them
+    dtd = r"""<!DOCTYPE d SYSTEM "a.dtd">
+<root>a</root>"""
+
+    for txt in (qbe, emb, dtd):
+        try:
+            parse_xml(txt)
+            t = "WARNING: dxml selftest failed:\n%s\n"
+            print(t % (txt,), file=sys.stderr)
+            return False
+        except BadXML:
+            pass
+
+    return True
+
+
+DXML_OK = selftest()
+if not DXML_OK:
+    DXMLParser = _NG
+
+
 def mktnod(name: str, text: str) -> ET.Element:
     el = ET.Element(name)
     el.text = text

copyparty/fsutil.py

@@ -42,14 +42,14 @@ class Fstab(object):
             self.cache = {}
 
         fs = "ext4"
-        msg = "failed to determine filesystem at [{}]; assuming {}\n{}"
+        msg = "failed to determine filesystem at %r; assuming %s\n%s"
 
         if ANYWIN:
             fs = "vfat"
             try:
                 path = self._winpath(path)
             except:
-                self.log(msg.format(path, fs, min_ex()), 3)
+                self.log(msg % (path, fs, min_ex()), 3)
                 return fs
 
         path = undot(path)
@@ -61,11 +61,11 @@ class Fstab(object):
         try:
             fs = self.get_w32(path) if ANYWIN else self.get_unix(path)
         except:
-            self.log(msg.format(path, fs, min_ex()), 3)
+            self.log(msg % (path, fs, min_ex()), 3)
 
         fs = fs.lower()
         self.cache[path] = fs
-        self.log("found {} at {}".format(fs, path))
+        self.log("found %s at %r" % (fs, path))
         return fs
 
     def _winpath(self, path: str) -> str:
@@ -78,7 +78,7 @@ class Fstab(object):
             return vid
 
     def build_fallback(self) -> None:
-        self.tab = VFS(self.log_func, "idk", "/", AXS(), {})
+        self.tab = VFS(self.log_func, "idk", "/", "/", AXS(), {})
         self.trusted = False
 
     def build_tab(self) -> None:
@@ -111,9 +111,10 @@ class Fstab(object):
 
         tab1.sort(key=lambda x: (len(x[0]), x[0]))
         path1, fs1 = tab1[0]
-        tab = VFS(self.log_func, fs1, path1, AXS(), {})
+        tab = VFS(self.log_func, fs1, path1, path1, AXS(), {})
         for path, fs in tab1[1:]:
-            tab.add(fs, path.lstrip("/"))
+            zs = path.lstrip("/")
+            tab.add(fs, zs, zs)
 
         self.tab = tab
         self.srctab = srctab
@@ -130,9 +131,10 @@ class Fstab(object):
         if not self.trusted:
             # no mtab access; have to build as we go
             if "/" in rem:
-                self.tab.add("idk", os.path.join(vn.vpath, rem.split("/")[0]))
+                zs = os.path.join(vn.vpath, rem.split("/")[0])
+                self.tab.add("idk", zs, zs)
             if rem:
-                self.tab.add(nval, path)
+                self.tab.add(nval, path, path)
             else:
                 vn.realpath = nval
 

copyparty/ftpd.py

@@ -19,6 +19,7 @@ from .__init__ import PY2, TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
 from .util import (
+    FN_EMB,
     VF_CAREFUL,
     Daemon,
     ODict,
@@ -30,6 +31,7 @@ from .util import (
     relchk,
     runhook,
     sanitize_fn,
+    set_fperms,
     vjoin,
     wunlink,
 )
@@ -170,6 +172,16 @@ class FtpFs(AbstractedFS):
             fn = sanitize_fn(fn or "", "")
             vpath = vjoin(rd, fn)
             vfs, rem = self.hub.asrv.vfs.get(vpath, self.uname, r, w, m, d)
+            if (
+                w
+                and fn.lower() in FN_EMB
+                and self.h.uname not in vfs.axs.uread
+                and "wo_up_readme" not in vfs.flags
+            ):
+                fn = "_wo_" + fn
+                vpath = vjoin(rd, fn)
+                vfs, rem = self.hub.asrv.vfs.get(vpath, self.uname, r, w, m, d)
+
             if not vfs.realpath:
                 t = "No filesystem mounted at [{}]"
                 raise FSE(t.format(vpath))
@@ -218,7 +230,7 @@ class FtpFs(AbstractedFS):
         r = "r" in mode
         w = "w" in mode or "a" in mode or "+" in mode
 
-        ap = self.rv2a(filename, r, w)[0]
+        ap, vfs, _ = self.rv2a(filename, r, w)
         self.validpath(ap)
         if w:
             try:
@@ -250,7 +262,11 @@ class FtpFs(AbstractedFS):
 
             wunlink(self.log, ap, VF_CAREFUL)
 
-        return open(fsenc(ap), mode, self.args.iobuf)
+        ret = open(fsenc(ap), mode, self.args.iobuf)
+        if w and "fperms" in vfs.flags:
+            set_fperms(ret, vfs.flags)
+
+        return ret
 
     def chdir(self, path: str) -> None:
         nwd = join(self.cwd, path)
@@ -281,8 +297,8 @@ class FtpFs(AbstractedFS):
         ) = avfs.can_access("", self.h.uname)
 
     def mkdir(self, path: str) -> None:
-        ap = self.rv2a(path, w=True)[0]
-        bos.makedirs(ap)  # filezilla expects this
+        ap, vfs, _ = self.rv2a(path, w=True)
+        bos.makedirs(ap, vf=vfs.flags)  # filezilla expects this
 
     def listdir(self, path: str) -> list[str]:
         vpath = join(self.cwd, path)

copyparty/httpconn.py

@@ -224,3 +224,6 @@ class HttpConn(object):
             if self.u2idx:
                 self.hsrv.put_u2idx(str(self.addr), self.u2idx)
                 self.u2idx = None
+
+            if self.rproxy:
+                self.set_rproxy()

copyparty/httpsrv.py

@@ -123,6 +123,7 @@ class HttpSrv(object):
         self.nm = NetMap([], [])
         self.ssdp: Optional["SSDPr"] = None
         self.gpwd = Garda(self.args.ban_pw)
+        self.gpwc = Garda(self.args.ban_pwc)
         self.g404 = Garda(self.args.ban_404)
         self.g403 = Garda(self.args.ban_403)
         self.g422 = Garda(self.args.ban_422, False)
@@ -172,15 +173,17 @@ class HttpSrv(object):
         env = jinja2.Environment()
         env.loader = jinja2.FunctionLoader(lambda f: load_jinja2_resource(self.E, f))
         jn = [
-            "splash",
-            "shares",
-            "svcs",
             "browser",
             "browser2",
-            "msg",
+            "cf",
+            "idp",
             "md",
             "mde",
-            "cf",
+            "msg",
+            "rups",
+            "shares",
+            "splash",
+            "svcs",
         ]
         self.j2 = {x: env.get_template(x + ".html") for x in jn}
         self.prism = has_resource(self.E, "web/deps/prism.js.gz")
@@ -194,10 +197,6 @@ class HttpSrv(object):
         self.xff_nm = build_netmap(self.args.xff_src)
         self.xff_lan = build_netmap("lan")
 
-        self.ptn_cc = re.compile(r"[\x00-\x1f]")
-        self.ptn_hsafe = re.compile(r"[\x00-\x1f<>\"'&]")
-        self.uparam_cc_ok = set("doc move tree".split())
-
         self.mallow = "GET HEAD POST PUT DELETE OPTIONS".split()
         if not self.args.no_dav:
             zs = "PROPFIND PROPPATCH LOCK UNLOCK MKCOL COPY MOVE"
@@ -316,6 +315,8 @@ class HttpSrv(object):
 
         Daemon(self.broker.say, "sig-hsrv-up1", ("cb_httpsrv_up",))
 
+        saddr = ("", 0)  # fwd-decl for `except TypeError as ex:`
+
         while not self.stopping:
             if self.args.log_conn:
                 self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="90")
@@ -397,6 +398,19 @@ class HttpSrv(object):
                 self.log(self.name, "accept({}): {}".format(fno, ex), c=6)
                 time.sleep(0.02)
                 continue
+            except TypeError as ex:
+                # on macOS, accept() may return a None saddr if blocked by LittleSnitch;
+                # unicode(saddr[0]) ==> TypeError: 'NoneType' object is not subscriptable
+                if tcp and not saddr:
+                    t = "accept(%s): failed to accept connection from client due to firewall or network issue"
+                    self.log(self.name, t % (fno,), c=3)
+                    try:
+                        sck.close()  # type: ignore
+                    except:
+                        pass
+                    time.sleep(0.02)
+                    continue
+                raise
 
             if self.args.log_conn:
                 t = "|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(

copyparty/ico.py

@@ -94,10 +94,21 @@ class Ico(object):
 <?xml version="1.0" encoding="UTF-8"?>
 <svg version="1.1" viewBox="0 0 100 {}" xmlns="http://www.w3.org/2000/svg"><g>
 <rect width="100%" height="100%" fill="#{}" />
-<text x="50%" y="50%" dominant-baseline="middle" text-anchor="middle" xml:space="preserve"
+<text x="50%" y="{}" dominant-baseline="middle" text-anchor="middle" xml:space="preserve"
   fill="#{}" font-family="monospace" font-size="14px" style="letter-spacing:.5px">{}</text>
 </g></svg>
 """
-        svg = svg.format(h, c[:6], c[6:], html_escape(ext, True))
+
+        txt = html_escape(ext, True)
+        if "\n" in txt:
+            lines = txt.split("\n")
+            n = len(lines)
+            y = "20%" if n == 2 else "10%" if n == 3 else "0"
+            zs = '<tspan x="50%%" dy="1.2em">%s</tspan>'
+            txt = "".join([zs % (x,) for x in lines])
+        else:
+            y = "50%"
+
+        svg = svg.format(h, c[:6], y, c[6:], txt)
 
         return "image/svg+xml", svg.encode("utf-8")

copyparty/metrics.py

@@ -18,7 +18,7 @@ class Metrics(object):
 
     def tx(self, cli: "HttpCli") -> bool:
         if not cli.avol:
-            raise Pebkac(403, "not allowed for user " + cli.uname)
+            raise Pebkac(403, "'stats' not allowed for user " + cli.uname)
 
         args = cli.args
         if not args.stats:

copyparty/mtag.py

@@ -18,6 +18,7 @@ from .util import (
     REKOBO_LKEY,
     VF_CAREFUL,
     fsenc,
+    gzip,
     min_ex,
     pybin,
     retchk,
@@ -138,8 +139,6 @@ def au_unpk(
         fd, ret = tempfile.mkstemp("." + au)
 
         if pk == "gz":
-            import gzip
-
             fi = gzip.GzipFile(abspath, mode="rb")
 
         elif pk == "xz":
@@ -167,12 +166,13 @@ def au_unpk(
             znil = [x for x in znil if "cover" in x[0]] or znil
             znil = [x for x in znil if CBZ_01.search(x[0])] or znil
             t = "cbz: %d files, %d hits" % (nf, len(znil))
+            using = sorted(znil)[0][1].filename
             if znil:
-                t += ", using " + znil[0][1].filename
+                t += ", using " + using
             log(t)
             if not znil:
                 raise Exception("no images inside cbz")
-            fi = zf.open(znil[0][1])
+            fi = zf.open(using)
 
         else:
             raise Exception("unknown compression %s" % (pk,))
@@ -194,7 +194,7 @@ def au_unpk(
 
     except Exception as ex:
         if ret:
-            t = "failed to decompress audio file [%s]: %r"
+            t = "failed to decompress audio file %r: %r"
             log(t % (abspath, ex))
             wunlink(log, ret, vn.flags if vn else VF_CAREFUL)
 
@@ -582,7 +582,7 @@ class MTag(object):
                 raise Exception()
         except Exception as ex:
             if self.args.mtag_v:
-                self.log("mutagen-err [{}] @ [{}]".format(ex, abspath), "90")
+                self.log("mutagen-err [%s] @ %r" % (ex, abspath), "90")
 
             return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 
@@ -699,8 +699,8 @@ class MTag(object):
                             ret[tag] = zj[tag]
             except:
                 if self.args.mtag_v:
-                    t = "mtag error: tagname {}, parser {}, file {} => {}"
-                    self.log(t.format(tagname, parser.bin, abspath, min_ex()))
+                    t = "mtag error: tagname %r, parser %r, file %r => %r"
+                    self.log(t % (tagname, parser.bin, abspath, min_ex()), 6)
 
         if ap != abspath:
             wunlink(self.log, ap, VF_CAREFUL)

copyparty/multicast.py

@@ -163,6 +163,7 @@ class MCast(object):
             sck.settimeout(None)
             sck.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
             try:
+                # safe for this purpose; https://lwn.net/Articles/853637/
                 sck.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
             except:
                 pass

copyparty/pwhash.py

@@ -15,7 +15,7 @@ try:
         raise Exception()
 
     HAVE_ARGON2 = True
-    from argon2 import __version__ as argon2ver
+    from argon2 import exceptions as argon2ex
 except:
     HAVE_ARGON2 = False
 

copyparty/smbd.py

@@ -263,7 +263,7 @@ class SMB(object):
                 time.time(),
                 "",
             ):
-                yeet("blocked by xbu server config: " + vpath)
+                yeet("blocked by xbu server config: %r" % (vpath,))
 
         ret = bos.open(ap, flags, *a, mode=chmod, **ka)
         if wr:
@@ -320,7 +320,7 @@ class SMB(object):
 
         self.hub.up2k.handle_mv(uname, "1.7.6.2", vp1, vp2)
         try:
-            bos.makedirs(ap2)
+            bos.makedirs(ap2, vf=vfs2.flags)
         except:
             pass
 
@@ -334,7 +334,7 @@ class SMB(object):
             t = "blocked mkdir (no-write-acc %s): /%s @%s"
             yeet(t % (vfs.axs.uwrite, vpath, uname))
 
-        return bos.mkdir(ap)
+        return bos.mkdir(ap, vfs.flags["chmod_d"])
 
     def _stat(self, vpath: str, *a: Any, **ka: Any) -> os.stat_result:
         try:

copyparty/sutil.py

@@ -17,6 +17,9 @@ if True:  # pylint: disable=using-constant-test
     from .util import NamedLogger
 
 
+TAR_NO_OPUS = set("aac|m4a|mp3|oga|ogg|opus|wma".split("|"))
+
+
 class StreamArc(object):
     def __init__(
         self,
@@ -82,9 +85,7 @@ def enthumb(
 ) -> dict[str, Any]:
     rem = f["vp"]
     ext = rem.rsplit(".", 1)[-1].lower()
-    if (fmt == "mp3" and ext == "mp3") or (
-        fmt == "opus" and ext in "aac|m4a|mp3|ogg|opus|wma".split("|")
-    ):
+    if (fmt == "mp3" and ext == "mp3") or (fmt == "opus" and ext in TAR_NO_OPUS):
         raise Exception()
 
     vp = vjoin(vtop, rem.split("/", 1)[1])
@@ -110,7 +111,7 @@ def errdesc(
     report = ["copyparty failed to add the following files to the archive:", ""]
 
     for fn, err in errors:
-        report.extend([" file: {}".format(fn), "error: {}".format(err), ""])
+        report.extend([" file: %r" % (fn,), "error: %s" % (err,), ""])
 
     btxt = "\r\n".join(report).encode("utf-8", "replace")
     btxt = vol_san(list(vfs.all_vols.values()), btxt)

copyparty/svchub.py

@@ -3,7 +3,6 @@ from __future__ import print_function, unicode_literals
 
 import argparse
 import errno
-import gzip
 import logging
 import os
 import re
@@ -28,6 +27,7 @@ if True:  # pylint: disable=using-constant-test
 
 from .__init__ import ANYWIN, EXE, MACOS, PY2, TYPE_CHECKING, E, EnvParams, unicode
 from .authsrv import BAD_CFG, AuthSrv
+from .bos import bos
 from .cert import ensure_cert
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, HAVE_MUTAGEN
 from .pwhash import HAVE_ARGON2
@@ -50,6 +50,9 @@ from .util import (
     FFMPEG_URL,
     HAVE_PSUTIL,
     HAVE_SQLITE3,
+    HAVE_ZMQ,
+    RE_ANSI,
+    URL_BUG,
     UTC,
     VERSIONS,
     Daemon,
@@ -58,9 +61,11 @@ from .util import (
     HMaccas,
     ODict,
     alltrace,
-    ansi_re,
     build_netmap,
+    expat_ver,
+    gzip,
     load_ipu,
+    lock_file,
     min_ex,
     mp,
     odfusion,
@@ -70,6 +75,9 @@ from .util import (
     ub64enc,
 )
 
+if HAVE_SQLITE3:
+    import sqlite3
+
 if TYPE_CHECKING:
     try:
         from .mdns import MDNS
@@ -81,6 +89,11 @@ if PY2:
     range = xrange  # type: ignore
 
 
+VER_IDP_DB = 1
+VER_SESSION_DB = 1
+VER_SHARES_DB = 2
+
+
 class SvcHub(object):
     """
     Hosts all services which cannot be parallelized due to reliance on monolithic resources.
@@ -155,6 +168,7 @@ class SvcHub(object):
         # for non-http clients (ftp, tftp)
         self.bans: dict[str, int] = {}
         self.gpwd = Garda(self.args.ban_pw)
+        self.gpwc = Garda(self.args.ban_pwc)
         self.g404 = Garda(self.args.ban_404)
         self.g403 = Garda(self.args.ban_403)
         self.g422 = Garda(self.args.ban_422, False)
@@ -183,8 +197,14 @@ class SvcHub(object):
 
         if not args.use_fpool and args.j != 1:
             args.no_fpool = True
-            t = "multithreading enabled with -j {}, so disabling fpool -- this can reduce upload performance on some filesystems"
-            self.log("root", t.format(args.j))
+            t = "multithreading enabled with -j {}, so disabling fpool -- this can reduce upload performance on some filesystems, and make some antivirus-softwares "
+            c = 0
+            if ANYWIN:
+                t += "(especially Microsoft Defender) stress your CPU and HDD severely during big uploads"
+                c = 3
+            else:
+                t += "consume more resources (CPU/HDD) than normal"
+            self.log("root", t.format(args.j), c)
 
         if not args.no_fpool and args.j != 1:
             t = "WARNING: ignoring --use-fpool because multithreading (-j{}) is enabled"
@@ -236,8 +256,20 @@ class SvcHub(object):
             setattr(args, "ipu_iu", iu)
             setattr(args, "ipu_nm", nm)
 
+        for zs in "ah_salt fk_salt dk_salt".split():
+            if getattr(args, "show_%s" % (zs,)):
+                self.log("root", "effective %s is %s" % (zs, getattr(args, zs)))
+
+        if args.ah_cli or args.ah_gen:
+            args.idp_store = 0
+            args.no_ses = True
+            args.shr = ""
+
+        if args.idp_store and args.idp_h_usr:
+            self.setup_db("idp")
+
         if not self.args.no_ses:
-            self.setup_session_db()
+            self.setup_db("ses")
 
         args.shr1 = ""
         if args.shr:
@@ -396,33 +428,91 @@ class SvcHub(object):
             except:
                 pass
 
-    def setup_session_db(self) -> None:
+    def _db_onfail_ses(self) -> None:
+        self.args.no_ses = True
+
+    def _db_onfail_idp(self) -> None:
+        self.args.idp_store = 0
+
+    def setup_db(self, which: str) -> None:
+        """
+        the "non-mission-critical" databases; if something looks broken then just nuke it
+        """
+        if which == "ses":
+            native_ver = VER_SESSION_DB
+            db_path = self.args.ses_db
+            desc = "sessions-db"
+            pathopt = "ses-db"
+            sanchk_q = "select count(*) from us"
+            createfun = self._create_session_db
+            failfun = self._db_onfail_ses
+        elif which == "idp":
+            native_ver = VER_IDP_DB
+            db_path = self.args.idp_db
+            desc = "idp-db"
+            pathopt = "idp-db"
+            sanchk_q = "select count(*) from us"
+            createfun = self._create_idp_db
+            failfun = self._db_onfail_idp
+        else:
+            raise Exception("unknown cachetype")
+
+        if not db_path.endswith(".db"):
+            zs = "config option --%s (the %s) was configured to [%s] which is invalid; must be a filepath ending with .db"
+            self.log("root", zs % (pathopt, desc, db_path), 1)
+            raise Exception(BAD_CFG)
+
         if not HAVE_SQLITE3:
-            self.args.no_ses = True
-            t = "WARNING: sqlite3 not available; disabling sessions, will use plaintext passwords in cookies"
-            self.log("root", t, 3)
+            failfun()
+            if which == "ses":
+                zs = "disabling sessions, will use plaintext passwords in cookies"
+            elif which == "idp":
+                zs = "disabling idp-db, will be unable to remember IdP-volumes after a restart"
+            self.log("root", "WARNING: sqlite3 not available; %s" % (zs,), 3)
             return
 
-        import sqlite3
+        assert sqlite3  # type: ignore  # !rm
 
-        create = True
-        db_path = self.args.ses_db
-        self.log("root", "opening sessions-db %s" % (db_path,))
-        for n in range(2):
+        db_lock = db_path + ".lock"
+        try:
+            create = not os.path.getsize(db_path)
+        except:
+            create = True
+        zs = "creating new" if create else "opening"
+        self.log("root", "%s %s %s" % (zs, desc, db_path))
+
+        for tries in range(2):
+            sver = 0
             try:
                 db = sqlite3.connect(db_path)
                 cur = db.cursor()
                 try:
-                    cur.execute("select count(*) from us").fetchone()
-                    create = False
-                    break
+                    zs = "select v from kv where k='sver'"
+                    sver = cur.execute(zs).fetchall()[0][0]
+                    if sver > native_ver:
+                        zs = "this version of copyparty only understands %s v%d and older; the db is v%d"
+                        raise Exception(zs % (desc, native_ver, sver))
+
+                    cur.execute(sanchk_q).fetchone()
                 except:
-                    pass
+                    if sver:
+                        raise
+                    sver = createfun(cur)
+
+                err = self._verify_db(
+                    cur, which, pathopt, db_path, desc, sver, native_ver
+                )
+                if err:
+                    tries = 99
+                    self.args.no_ses = True
+                    self.log("root", err, 3)
+                break
+
             except Exception as ex:
-                if n:
+                if tries or sver > native_ver:
                     raise
-                t = "sessions-db corrupt; deleting and recreating: %r"
-                self.log("root", t % (ex,), 3)
+                t = "%s is unusable; deleting and recreating: %r"
+                self.log("root", t % (desc, ex), 3)
                 try:
                     cur.close()  # type: ignore
                 except:
@@ -431,8 +521,13 @@ class SvcHub(object):
                     db.close()  # type: ignore
                 except:
                     pass
+                try:
+                    os.unlink(db_lock)
+                except:
+                    pass
                 os.unlink(db_path)
 
+    def _create_session_db(self, cur: "sqlite3.Cursor") -> int:
         sch = [
             r"create table kv (k text, v int)",
             r"create table us (un text, si text, t0 int)",
@@ -442,17 +537,74 @@ class SvcHub(object):
             r"create index us_t0 on us(t0)",
             r"insert into kv values ('sver', 1)",
         ]
+        for cmd in sch:
+            cur.execute(cmd)
+        self.log("root", "created new sessions-db")
+        return 1
 
-        assert db  # type: ignore  # !rm
-        assert cur  # type: ignore  # !rm
-        if create:
-            for cmd in sch:
-                cur.execute(cmd)
-            self.log("root", "created new sessions-db")
-            db.commit()
+    def _create_idp_db(self, cur: "sqlite3.Cursor") -> int:
+        sch = [
+            r"create table kv (k text, v int)",
+            r"create table us (un text, gs text)",
+            # username, groups
+            r"create index us_un on us(un)",
+            r"insert into kv values ('sver', 1)",
+        ]
+        for cmd in sch:
+            cur.execute(cmd)
+        self.log("root", "created new idp-db")
+        return 1
 
+    def _verify_db(
+        self,
+        cur: "sqlite3.Cursor",
+        which: str,
+        pathopt: str,
+        db_path: str,
+        desc: str,
+        sver: int,
+        native_ver: int,
+    ) -> str:
+        # ensure writable (maybe owned by other user)
+        db = cur.connection
+
+        try:
+            zil = cur.execute("select v from kv where k='pid'").fetchall()
+            if len(zil) > 1:
+                raise Exception()
+            owner = zil[0][0]
+        except:
+            owner = 0
+
+        if which == "ses":
+            cons = "Will now disable sessions and instead use plaintext passwords in cookies."
+        elif which == "idp":
+            cons = "Each IdP-volume will not become available until its associated user sends their first request."
+        else:
+            raise Exception()
+
+        if not lock_file(db_path + ".lock"):
+            t = "the %s [%s] is already in use by another copyparty instance (pid:%d). This is not supported; please provide another database with --%s or give this copyparty-instance its entirely separate config-folder by setting another path in the XDG_CONFIG_HOME env-var. You can also disable this safeguard by setting env-var PRTY_NO_DB_LOCK=1. %s"
+            return t % (desc, db_path, owner, pathopt, cons)
+
+        vars = (("pid", os.getpid()), ("ts", int(time.time() * 1000)))
+        if owner:
+            # wear-estimate: 2 cells; offsets 0x10, 0x50, 0x19720
+            for k, v in vars:
+                cur.execute("update kv set v=? where k=?", (v, k))
+        else:
+            # wear-estimate: 3~4 cells; offsets 0x10, 0x50, 0x19180, 0x19710, 0x36000, 0x360b0, 0x36b90
+            for k, v in vars:
+                cur.execute("insert into kv values(?, ?)", (k, v))
+
+        if sver < native_ver:
+            cur.execute("delete from kv where k='sver'")
+            cur.execute("insert into kv values('sver',?)", (native_ver,))
+
+        db.commit()
         cur.close()
         db.close()
+        return ""
 
     def setup_share_db(self) -> None:
         al = self.args
@@ -461,7 +613,7 @@ class SvcHub(object):
             al.shr = ""
             return
 
-        import sqlite3
+        assert sqlite3  # type: ignore  # !rm
 
         al.shr = al.shr.strip("/")
         if "/" in al.shr or not al.shr:
@@ -472,34 +624,48 @@ class SvcHub(object):
         al.shr = "/%s/" % (al.shr,)
         al.shr1 = al.shr[1:]
 
-        create = True
-        modified = False
+        # policy:
+        # the shares-db is important, so panic if something is wrong
+
         db_path = self.args.shr_db
-        self.log("root", "opening shares-db %s" % (db_path,))
-        for n in range(2):
-            try:
-                db = sqlite3.connect(db_path)
-                cur = db.cursor()
-                try:
-                    cur.execute("select count(*) from sh").fetchone()
-                    create = False
-                    break
-                except:
-                    pass
-            except Exception as ex:
-                if n:
-                    raise
-                t = "shares-db corrupt; deleting and recreating: %r"
-                self.log("root", t % (ex,), 3)
-                try:
-                    cur.close()  # type: ignore
-                except:
-                    pass
-                try:
-                    db.close()  # type: ignore
-                except:
-                    pass
-                os.unlink(db_path)
+        db_lock = db_path + ".lock"
+        try:
+            create = not os.path.getsize(db_path)
+        except:
+            create = True
+        zs = "creating new" if create else "opening"
+        self.log("root", "%s shares-db %s" % (zs, db_path))
+
+        sver = 0
+        try:
+            db = sqlite3.connect(db_path)
+            cur = db.cursor()
+            if not create:
+                zs = "select v from kv where k='sver'"
+                sver = cur.execute(zs).fetchall()[0][0]
+                if sver > VER_SHARES_DB:
+                    zs = "this version of copyparty only understands shares-db v%d and older; the db is v%d"
+                    raise Exception(zs % (VER_SHARES_DB, sver))
+
+                cur.execute("select count(*) from sh").fetchone()
+        except Exception as ex:
+            t = "could not open shares-db; will now panic...\nthe following database must be repaired or deleted before you can launch copyparty:\n%s\n\nERROR: %s\n\nadditional details:\n%s\n"
+            self.log("root", t % (db_path, ex, min_ex()), 1)
+            raise
+
+        try:
+            zil = cur.execute("select v from kv where k='pid'").fetchall()
+            if len(zil) > 1:
+                raise Exception()
+            owner = zil[0][0]
+        except:
+            owner = 0
+
+        if not lock_file(db_lock):
+            t = "the shares-db [%s] is already in use by another copyparty instance (pid:%d). This is not supported; please provide another database with --shr-db or give this copyparty-instance its entirely separate config-folder by setting another path in the XDG_CONFIG_HOME env-var. You can also disable this safeguard by setting env-var PRTY_NO_DB_LOCK=1. Will now panic."
+            t = t % (db_path, owner)
+            self.log("root", t, 1)
+            raise Exception(t)
 
         sch1 = [
             r"create table kv (k text, v int)",
@@ -511,34 +677,37 @@ class SvcHub(object):
             r"create index sf_k on sf(k)",
             r"create index sh_k on sh(k)",
             r"create index sh_t1 on sh(t1)",
+            r"insert into kv values ('sver', 2)",
         ]
 
         assert db  # type: ignore  # !rm
         assert cur  # type: ignore  # !rm
-        if create:
-            dver = 2
-            modified = True
+        if not sver:
+            sver = VER_SHARES_DB
             for cmd in sch1 + sch2:
                 cur.execute(cmd)
             self.log("root", "created new shares-db")
-        else:
-            (dver,) = cur.execute("select v from kv where k = 'sver'").fetchall()[0]
 
-        if dver == 1:
-            modified = True
+        if sver == 1:
             for cmd in sch2:
                 cur.execute(cmd)
             cur.execute("update sh set st = 0")
             self.log("root", "shares-db schema upgrade ok")
 
-        if modified:
-            for cmd in [
-                r"delete from kv where k = 'sver'",
-                r"insert into kv values ('sver', %d)" % (2,),
-            ]:
-                cur.execute(cmd)
-            db.commit()
+        if sver < VER_SHARES_DB:
+            cur.execute("delete from kv where k='sver'")
+            cur.execute("insert into kv values('sver',?)", (VER_SHARES_DB,))
 
+        vars = (("pid", os.getpid()), ("ts", int(time.time() * 1000)))
+        if owner:
+            # wear-estimate: same as sessions-db
+            for k, v in vars:
+                cur.execute("update kv set v=? where k=?", (v, k))
+        else:
+            for k, v in vars:
+                cur.execute("insert into kv values(?, ?)", (k, v))
+
+        db.commit()
         cur.close()
         db.close()
 
@@ -639,6 +808,7 @@ class SvcHub(object):
             (HAVE_FFPROBE, "ffprobe", t_ff + ", read audio/media tags"),
             (HAVE_MUTAGEN, "mutagen", "read audio tags (ffprobe is better but slower)"),
             (HAVE_ARGON2, "argon2", "secure password hashing (advanced users only)"),
+            (HAVE_ZMQ, "pyzmq", "send zeromq messages from event-hooks"),
             (HAVE_HEIF, "pillow-heif", "read .heif images with pillow (rarely useful)"),
             (HAVE_AVIF, "pillow-avif", "read .avif images with pillow (rarely useful)"),
         ]
@@ -675,10 +845,11 @@ class SvcHub(object):
                 t += ", "
             t += "\033[0mNG: \033[35m" + sng
 
-        t += "\033[0m, see --deps"
-        self.log("dependencies", t, 6)
+        t += "\033[0m, see --deps (this is fine btw)"
+        self.log("optional-dependencies", t, 6)
 
     def _check_env(self) -> None:
+        al = self.args
         try:
             files = os.listdir(E.cfg)
         except:
@@ -695,6 +866,30 @@ class SvcHub(object):
             if self.args.bauth_last:
                 self.log("root", "WARNING: ignoring --bauth-last due to --no-bauth", 3)
 
+        have_tcp = False
+        for zs in al.i:
+            if not zs.startswith("unix:"):
+                have_tcp = True
+        if not have_tcp:
+            zb = False
+            zs = "z zm zm4 zm6 zmv zmvv zs zsv zv"
+            for zs in zs.split():
+                if getattr(al, zs, False):
+                    setattr(al, zs, False)
+                    zb = True
+            if zb:
+                t = "only listening on unix-sockets; cannot enable zeroconf/mdns/ssdp as requested"
+                self.log("root", t, 3)
+
+        if not self.args.no_dav:
+            from .dxml import DXML_OK
+
+            if not DXML_OK:
+                if not self.args.no_dav:
+                    self.args.no_dav = True
+                    t = "WARNING:\nDisabling WebDAV support because dxml selftest failed. Please report this bug;\n%s\n...and include the following information in the bug-report:\n%s | expat %s\n"
+                    self.log("root", t % (URL_BUG, VERSIONS, expat_ver()), 1)
+
     def _process_config(self) -> bool:
         al = self.args
 
@@ -750,13 +945,20 @@ class SvcHub(object):
             vl = [os.path.expandvars(os.path.expanduser(x)) for x in vl]
             setattr(al, k, vl)
 
-        for k in "lo hist ssl_log".split(" "):
+        for k in "lo hist dbpath ssl_log".split(" "):
             vs = getattr(al, k)
             if vs:
                 vs = os.path.expandvars(os.path.expanduser(vs))
                 setattr(al, k, vs)
 
-        for k in "sus_urls nonsus_urls".split(" "):
+        for k in "idp_adm".split(" "):
+            vs = getattr(al, k)
+            vsa = [x.strip() for x in vs.split(",")]
+            vsa = [x.lower() for x in vsa if x]
+            setattr(al, k + "_set", set(vsa))
+
+        zs = "dav_ua1 sus_urls nonsus_urls ua_nodoc ua_nozip"
+        for k in zs.split(" "):
             vs = getattr(al, k)
             if not vs or vs == "no":
                 setattr(al, k, None)
@@ -918,7 +1120,7 @@ class SvcHub(object):
 
         fn = sel_fn
         try:
-            os.makedirs(os.path.dirname(fn))
+            bos.makedirs(os.path.dirname(fn))
         except:
             pass
 
@@ -935,6 +1137,9 @@ class SvcHub(object):
 
             lh = codecs.open(fn, "w", encoding="utf-8", errors="replace")
 
+        if getattr(self.args, "free_umask", False):
+            os.fchmod(lh.fileno(), 0o644)
+
         argv = [pybin] + self.argv
         if hasattr(shlex, "quote"):
             argv = [shlex.quote(x) for x in argv]
@@ -1204,9 +1409,9 @@ class SvcHub(object):
             if self.no_ansi:
                 fmt = "%s %-21s %s\n"
                 if "\033" in msg:
-                    msg = ansi_re.sub("", msg)
+                    msg = RE_ANSI.sub("", msg)
                 if "\033" in src:
-                    src = ansi_re.sub("", src)
+                    src = RE_ANSI.sub("", src)
             elif c:
                 if isinstance(c, int):
                     msg = "\033[3%sm%s\033[0m" % (c, msg)
@@ -1247,7 +1452,7 @@ class SvcHub(object):
                 raise
 
     def check_mp_support(self) -> str:
-        if MACOS:
+        if MACOS and not os.environ.get("PRTY_FORCE_MP"):
             return "multiprocessing is wonky on mac osx;"
         elif sys.version_info < (3, 3):
             return "need python 3.3 or newer for multiprocessing;"
@@ -1267,7 +1472,7 @@ class SvcHub(object):
             return False
 
         try:
-            if mp.cpu_count() <= 1:
+            if mp.cpu_count() <= 1 and not os.environ.get("PRTY_FORCE_MP"):
                 raise Exception()
         except:
             self.log("svchub", "only one CPU detected; multiprocessing disabled")

copyparty/szip.py

@@ -4,12 +4,11 @@ from __future__ import print_function, unicode_literals
 import calendar
 import stat
 import time
-import zlib
 
 from .authsrv import AuthSrv
 from .bos import bos
 from .sutil import StreamArc, errdesc
-from .util import min_ex, sanitize_fn, spack, sunpack, yieldfile
+from .util import min_ex, sanitize_fn, spack, sunpack, yieldfile, zlib
 
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Generator, Optional
@@ -55,6 +54,7 @@ def gen_fdesc(sz: int, crc32: int, z64: bool) -> bytes:
 
 def gen_hdr(
     h_pos: Optional[int],
+    z64: bool,
     fn: str,
     sz: int,
     lastmod: int,
@@ -71,7 +71,6 @@ def gen_hdr(
     # appnote 4.5 / zip 3.0 (2008) / unzip 6.0 (2009) says to add z64
     # extinfo for values which exceed H, but that becomes an off-by-one
     # (can't tell if it was clamped or exactly maxval), make it obvious
-    z64 = sz >= 0xFFFFFFFF
     z64v = [sz, sz] if z64 else []
     if h_pos and h_pos >= 0xFFFFFFFF:
         # central, also consider ptr to original header
@@ -245,6 +244,7 @@ class StreamZip(StreamArc):
 
         sz = st.st_size
         ts = st.st_mtime
+        h_pos = self.pos
 
         crc = 0
         if self.pre_crc:
@@ -253,8 +253,12 @@ class StreamZip(StreamArc):
 
             crc &= 0xFFFFFFFF
 
-        h_pos = self.pos
-        buf = gen_hdr(None, name, sz, ts, self.utf8, crc, self.pre_crc)
+        # some unzip-programs expect a 64bit data-descriptor
+        # even if the only 32bit-exceeding value is the offset,
+        # so force that by placeholdering the filesize too
+        z64 = h_pos >= 0xFFFFFFFF or sz >= 0xFFFFFFFF
+
+        buf = gen_hdr(None, z64, name, sz, ts, self.utf8, crc, self.pre_crc)
         yield self._ct(buf)
 
         for buf in yieldfile(src, self.args.iobuf):
@@ -267,8 +271,6 @@ class StreamZip(StreamArc):
 
         self.items.append((name, sz, ts, crc, h_pos))
 
-        z64 = sz >= 4 * 1024 * 1024 * 1024
-
         if z64 or not self.pre_crc:
             buf = gen_fdesc(sz, crc, z64)
             yield self._ct(buf)
@@ -307,7 +309,8 @@ class StreamZip(StreamArc):
 
             cdir_pos = self.pos
             for name, sz, ts, crc, h_pos in self.items:
-                buf = gen_hdr(h_pos, name, sz, ts, self.utf8, crc, self.pre_crc)
+                z64 = h_pos >= 0xFFFFFFFF or sz >= 0xFFFFFFFF
+                buf = gen_hdr(h_pos, z64, name, sz, ts, self.utf8, crc, self.pre_crc)
                 mbuf += self._ct(buf)
                 if len(mbuf) >= 16384:
                     yield mbuf

copyparty/tcpsrv.py

@@ -151,9 +151,15 @@ class TcpSrv(object):
             if just_ll or self.args.ll:
                 ll_ok.add(ip.split("/")[0])
 
+        listening_on = []
+        for ip, ports in sorted(ok.items()):
+            for port in sorted(ports):
+                listening_on.append("%s %s" % (ip, port))
+
         qr1: dict[str, list[int]] = {}
         qr2: dict[str, list[int]] = {}
         msgs = []
+        accessible_on = []
         title_tab: dict[str, dict[str, int]] = {}
         title_vars = [x[1:] for x in self.args.wintitle.split(" ") if x.startswith("$")]
         t = "available @ {}://{}:{}/  (\033[33m{}\033[0m)"
@@ -169,6 +175,10 @@ class TcpSrv(object):
                 ):
                     continue
 
+                zs = "%s %s" % (ip, port)
+                if zs not in accessible_on:
+                    accessible_on.append(zs)
+
                 proto = " http"
                 if self.args.http_only:
                     pass
@@ -219,6 +229,14 @@ class TcpSrv(object):
         else:
             print("\n", end="")
 
+        for fn, ls in (
+            (self.args.wr_h_eps, listening_on),
+            (self.args.wr_h_aon, accessible_on),
+        ):
+            if fn:
+                with open(fn, "wb") as f:
+                    f.write(("\n".join(ls)).encode("utf-8"))
+
         if self.args.qr or self.args.qrs:
             self.qr = self._qr(qr1, qr2)
 
@@ -264,7 +282,7 @@ class TcpSrv(object):
         except:
             pass  # will create another ipv4 socket instead
 
-        if not ANYWIN and self.args.freebind:
+        if getattr(self.args, "freebind", False):
             srv.setsockopt(socket.SOL_IP, socket.IP_FREEBIND, 1)
 
         try:
@@ -402,17 +420,17 @@ class TcpSrv(object):
             if not netdevs:
                 continue
 
-            added = "nothing"
-            removed = "nothing"
+            add = []
+            rem = []
             for k, v in netdevs.items():
                 if k not in self.netdevs:
-                    added = "{} = {}".format(k, v)
+                    add.append("\n\033[32m  added %s = %s" % (k, v))
             for k, v in self.netdevs.items():
                 if k not in netdevs:
-                    removed = "{} = {}".format(k, v)
+                    rem.append("\n\033[33mremoved %s = %s" % (k, v))
 
-            t = "network change detected:\n  added {}\033[0;33m\nremoved {}"
-            self.log("tcpsrv", t.format(added, removed), 3)
+            t = "network change detected:%s%s"
+            self.log("tcpsrv", t % ("".join(add), "".join(rem)), 3)
             self.netdevs = netdevs
             self._distribute_netdevs()
 
@@ -548,7 +566,7 @@ class TcpSrv(object):
         ip = None
         ips = list(t1) + list(t2)
         qri = self.args.qri
-        if self.args.zm and not qri:
+        if self.args.zm and not qri and ips:
             name = self.args.name + ".local"
             t1[name] = next(v for v in (t1 or t2).values())
             ips = [name] + ips

copyparty/tftpd.py

@@ -36,7 +36,20 @@ from partftpy.TftpShared import TftpException
 from .__init__ import EXE, PY2, TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
-from .util import UTC, BytesIO, Daemon, ODict, exclude_dotfiles, min_ex, runhook, undot
+from .util import (
+    FN_EMB,
+    UTC,
+    BytesIO,
+    Daemon,
+    ODict,
+    exclude_dotfiles,
+    min_ex,
+    runhook,
+    set_fperms,
+    undot,
+    vjoin,
+    vsplit,
+)
 
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Union
@@ -244,16 +257,25 @@ class Tftpd(object):
         for srv in srvs:
             srv.stop()
 
-    def _v2a(self, caller: str, vpath: str, perms: list, *a: Any) -> tuple[VFS, str]:
+    def _v2a(
+        self, caller: str, vpath: str, perms: list, *a: Any
+    ) -> tuple[VFS, str, str]:
         vpath = vpath.replace("\\", "/").lstrip("/")
         if not perms:
             perms = [True, True]
 
         debug('%s("%s", %s) %s\033[K\033[0m', caller, vpath, str(a), perms)
         vfs, rem = self.asrv.vfs.get(vpath, "*", *perms)
+        if perms[1] and "*" not in vfs.axs.uread and "wo_up_readme" not in vfs.flags:
+            zs, fn = vsplit(vpath)
+            if fn.lower() in FN_EMB:
+                vpath = vjoin(zs, "_wo_" + fn)
+                vfs, rem = self.asrv.vfs.get(vpath, "*", *perms)
+
         if not vfs.realpath:
             raise Exception("unmapped vfs")
-        return vfs, vfs.canonical(rem)
+
+        return vfs, vpath, vfs.canonical(rem)
 
     def _ls(self, vpath: str, raddress: str, rport: int, force=False) -> Any:
         # generate file listing if vpath is dir.txt and return as file object
@@ -263,6 +285,7 @@ class Tftpd(object):
             if not ptn or not ptn.match(fn.lower()):
                 return None
 
+        tsdt = datetime.fromtimestamp
         vn, rem = self.asrv.vfs.get(vpath, "*", True, False)
         fsroot, vfs_ls, vfs_virt = vn.ls(
             rem,
@@ -275,7 +298,7 @@ class Tftpd(object):
         dirs1 = [(v.st_mtime, v.st_size, k + "/") for k, v in vfs_ls if k in dnames]
         fils1 = [(v.st_mtime, v.st_size, k) for k, v in vfs_ls if k not in dnames]
         real1 = dirs1 + fils1
-        realt = [(datetime.fromtimestamp(mt, UTC), sz, fn) for mt, sz, fn in real1]
+        realt = [(tsdt(max(0, mt), UTC), sz, fn) for mt, sz, fn in real1]
         reals = [
             (
                 "%04d-%02d-%02d %02d:%02d:%02d"
@@ -331,7 +354,7 @@ class Tftpd(object):
         else:
             raise Exception("bad mode %s" % (mode,))
 
-        vfs, ap = self._v2a("open", vpath, [rd, wr])
+        vfs, vpath, ap = self._v2a("open", vpath, [rd, wr])
         if wr:
             if "*" not in vfs.axs.uwrite:
                 yeet("blocked write; folder not world-writable: /%s" % (vpath,))
@@ -357,7 +380,7 @@ class Tftpd(object):
                 time.time(),
                 "",
             ):
-                yeet("blocked by xbu server config: " + vpath)
+                yeet("blocked by xbu server config: %r" % (vpath,))
 
         if not self.args.tftp_nols and bos.path.isdir(ap):
             return self._ls(vpath, "", 0, True)
@@ -365,18 +388,24 @@ class Tftpd(object):
         if not a:
             a = (self.args.iobuf,)
 
-        return open(ap, mode, *a, **ka)
+        ret = open(ap, mode, *a, **ka)
+        if wr and "fperms" in vfs.flags:
+            set_fperms(ret, vfs.flags)
+
+        return ret
 
     def _mkdir(self, vpath: str, *a) -> None:
-        vfs, ap = self._v2a("mkdir", vpath, [])
+        vfs, _, ap = self._v2a("mkdir", vpath, [False, True])
         if "*" not in vfs.axs.uwrite:
             yeet("blocked mkdir; folder not world-writable: /%s" % (vpath,))
 
-        return bos.mkdir(ap)
+        bos.mkdir(ap, vfs.flags["chmod_d"])
+        if "chown" in vfs.flags:
+            bos.chown(ap, vfs.flags["uid"], vfs.flags["gid"])
 
     def _unlink(self, vpath: str) -> None:
         # return bos.unlink(self._v2a("stat", vpath, *a)[1])
-        vfs, ap = self._v2a("delete", vpath, [True, False, False, True])
+        vfs, _, ap = self._v2a("delete", vpath, [True, False, False, True])
 
         try:
             inf = bos.stat(ap)
@@ -400,7 +429,7 @@ class Tftpd(object):
 
     def _p_exists(self, vpath: str) -> bool:
         try:
-            ap = self._v2a("p.exists", vpath, [False, False])[1]
+            ap = self._v2a("p.exists", vpath, [False, False])[2]
             bos.stat(ap)
             return True
         except:
@@ -408,7 +437,7 @@ class Tftpd(object):
 
     def _p_isdir(self, vpath: str) -> bool:
         try:
-            st = bos.stat(self._v2a("p.isdir", vpath, [False, False])[1])
+            st = bos.stat(self._v2a("p.isdir", vpath, [False, False])[2])
             ret = stat.S_ISDIR(st.st_mode)
             return ret
         except:

copyparty/th_cli.py

@@ -1,13 +1,15 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
+import errno
 import os
+import stat
 
 from .__init__ import TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
-from .th_srv import HAVE_WEBP, thumb_path
-from .util import Cooldown
+from .th_srv import EXTS_AC, HAVE_WEBP, thumb_path
+from .util import Cooldown, Pebkac
 
 if True:  # pylint: disable=using-constant-test
     from typing import Optional, Union
@@ -16,6 +18,9 @@ if TYPE_CHECKING:
     from .httpsrv import HttpSrv
 
 
+IOERROR = "reading the file was denied by the server os; either due to filesystem permissions, selinux, apparmor, or similar:\n%r"
+
+
 class ThumbCli(object):
     def __init__(self, hsrv: "HttpSrv") -> None:
         self.broker = hsrv.broker
@@ -57,13 +62,17 @@ class ThumbCli(object):
         if is_vid and "dvthumb" in dbv.flags:
             return None
 
-        want_opus = fmt in ("opus", "caf", "mp3")
+        want_opus = fmt in EXTS_AC
         is_au = ext in self.fmt_ffa
         is_vau = want_opus and ext in self.fmt_ffv
         if is_au or is_vau:
             if want_opus:
                 if self.args.no_acode:
                     return None
+                elif fmt == "caf" and self.args.no_caf:
+                    fmt = "mp3"
+                elif fmt == "owa" and self.args.no_owa:
+                    fmt = "mp3"
             else:
                 if "dathumb" in dbv.flags:
                     return None
@@ -109,18 +118,18 @@ class ThumbCli(object):
             fmt = sfmt
 
         elif fmt[:1] == "p" and not is_au and not is_vid:
-            t = "cannot thumbnail [%s]: png only allowed for waveforms"
-            self.log(t % (rem), 6)
+            t = "cannot thumbnail %r: png only allowed for waveforms"
+            self.log(t % (rem,), 6)
             return None
 
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
         tpaths = [tpath]
-        if fmt == "w":
+        if fmt[:1] == "w":
             # also check for jpg (maybe webp is unavailable)
             tpaths.append(tpath.rsplit(".", 1)[0] + ".jpg")
 
@@ -153,8 +162,22 @@ class ThumbCli(object):
         if abort:
             return None
 
-        if not bos.path.getsize(os.path.join(ptop, rem)):
-            return None
+        ap = os.path.join(ptop, rem)
+        try:
+            st = bos.stat(ap)
+            if not st.st_size or not stat.S_ISREG(st.st_mode):
+                return None
+
+            with open(ap, "rb", 4) as f:
+                if not f.read(4):
+                    raise Exception()
+        except OSError as ex:
+            if ex.errno == errno.ENOENT:
+                raise Pebkac(404)
+            else:
+                raise Pebkac(500, IOERROR % (ex,))
+        except Exception as ex:
+            raise Pebkac(500, IOERROR % (ex,))
 
         x = self.broker.ask("thumbsrv.get", ptop, rem, mtime, fmt)
         return x.get()  # type: ignore

copyparty/th_srv.py

@@ -4,8 +4,10 @@ from __future__ import print_function, unicode_literals
 import hashlib
 import logging
 import os
+import re
 import shutil
 import subprocess as sp
+import tempfile
 import threading
 import time
 
@@ -18,21 +20,22 @@ from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, au_unpk, ffprobe
 from .util import BytesIO  # type: ignore
 from .util import (
     FFMPEG_URL,
+    VF_CAREFUL,
     Cooldown,
     Daemon,
     afsenc,
+    atomic_move,
     fsenc,
     min_ex,
     runcmd,
     statdir,
     ub64enc,
     vsplit,
-    wrename,
     wunlink,
 )
 
 if True:  # pylint: disable=using-constant-test
-    from typing import Optional, Union
+    from typing import Any, Optional, Union
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -46,6 +49,13 @@ HAVE_HEIF = False
 HAVE_AVIF = False
 HAVE_WEBP = False
 
+EXTS_TH = set(["jpg", "webp", "png"])
+EXTS_AC = set(["opus", "owa", "caf", "mp3"])
+EXTS_SPEC_SAFE = set("aif aiff flac mp3 opus wav".split())
+
+PTN_TS = re.compile("^-?[0-9a-f]{8,10}$")
+
+
 try:
     if os.environ.get("PRTY_NO_PIL"):
         raise Exception()
@@ -86,6 +96,10 @@ try:
         if os.environ.get("PRTY_NO_PIL_AVIF"):
             raise Exception()
 
+        if ".avif" in Image.registered_extensions():
+            HAVE_AVIF = True
+            raise Exception()
+
         import pillow_avif  # noqa: F401  # pylint: disable=unused-import
 
         HAVE_AVIF = True
@@ -139,7 +153,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
     h = hashlib.sha512(afsenc(fn)).digest()
     fn = ub64enc(h).decode("ascii")[:24]
 
-    if fmt in ("opus", "caf", "mp3"):
+    if fmt in EXTS_AC:
         cat = "ac"
     else:
         fc = fmt[:1]
@@ -160,12 +174,15 @@ class ThumbSrv(object):
 
         self.mutex = threading.Lock()
         self.busy: dict[str, list[threading.Condition]] = {}
+        self.untemp: dict[str, list[str]] = {}
         self.ram: dict[str, float] = {}
         self.memcond = threading.Condition(self.mutex)
         self.stopping = False
         self.rm_nullthumbs = True  # forget failed conversions on startup
         self.nthr = max(1, self.args.th_mt)
 
+        self.exts_spec_unsafe = set(self.args.th_spec_cnv.split(","))
+
         self.q: Queue[Optional[tuple[str, str, str, VFS]]] = Queue(self.nthr * 4)
         for n in range(self.nthr):
             Daemon(self.worker, "thumb-{}-{}".format(n, self.nthr))
@@ -239,7 +256,7 @@ class ThumbSrv(object):
     def get(self, ptop: str, rem: str, mtime: float, fmt: str) -> Optional[str]:
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
@@ -249,10 +266,11 @@ class ThumbSrv(object):
         with self.mutex:
             try:
                 self.busy[tpath].append(cond)
-                self.log("joined waiting room for %s" % (tpath,))
+                self.log("joined waiting room for %r" % (tpath,))
             except:
                 thdir = os.path.dirname(tpath)
-                bos.makedirs(os.path.join(thdir, "w"))
+                chmod = bos.MKD_700 if self.args.free_umask else bos.MKD_755
+                bos.makedirs(os.path.join(thdir, "w"), vf=chmod)
 
                 inf_path = os.path.join(thdir, "dir.txt")
                 if not bos.path.exists(inf_path):
@@ -266,11 +284,11 @@ class ThumbSrv(object):
             allvols = list(self.asrv.vfs.all_vols.values())
             vn = next((x for x in allvols if x.realpath == ptop), None)
             if not vn:
-                self.log("ptop [{}] not in {}".format(ptop, allvols), 3)
-                vn = self.asrv.vfs.all_aps[0][1]
+                self.log("ptop %r not in %s" % (ptop, allvols), 3)
+                vn = self.asrv.vfs.all_aps[0][1][0]
 
             self.q.put((abspath, tpath, fmt, vn))
-            self.log("conv {} :{} \033[0m{}".format(tpath, fmt, abspath), c=6)
+            self.log("conv %r :%s \033[0m%r" % (tpath, fmt, abspath), 6)
 
         while not self.stopping:
             with self.mutex:
@@ -334,9 +352,10 @@ class ThumbSrv(object):
                 ap_unpk = abspath
 
             if not bos.path.exists(tpath):
-                want_mp3 = tpath.endswith(".mp3")
-                want_opus = tpath.endswith(".opus") or tpath.endswith(".caf")
-                want_png = tpath.endswith(".png")
+                tex = tpath.rsplit(".", 1)[-1]
+                want_mp3 = tex == "mp3"
+                want_opus = tex in ("opus", "owa", "caf")
+                want_png = tex == "png"
                 want_au = want_mp3 or want_opus
                 for lib in self.args.th_dec:
                     can_au = lib == "ff" and (
@@ -375,14 +394,18 @@ class ThumbSrv(object):
                     fun(ap_unpk, ttpath, fmt, vn)
                     break
                 except Exception as ex:
-                    msg = "{} could not create thumbnail of {}\n{}"
-                    msg = msg.format(fun.__name__, abspath, min_ex())
+                    msg = "%s could not create thumbnail of %r\n%s"
+                    msg = msg % (fun.__name__, abspath, min_ex())
                     c: Union[str, int] = 1 if "<Signals.SIG" in msg else "90"
                     self.log(msg, c)
                     if getattr(ex, "returncode", 0) != 321:
                         if fun == funs[-1]:
-                            with open(ttpath, "wb") as _:
-                                pass
+                            try:
+                                with open(ttpath, "wb") as _:
+                                    pass
+                            except Exception as ex:
+                                t = "failed to create the file [%s]: %r"
+                                self.log(t % (ttpath, ex), 3)
                     else:
                         # ffmpeg may spawn empty files on windows
                         try:
@@ -394,14 +417,25 @@ class ThumbSrv(object):
                 wunlink(self.log, ap_unpk, vn.flags)
 
             try:
-                wrename(self.log, ttpath, tpath, vn.flags)
-            except:
+                atomic_move(self.log, ttpath, tpath, vn.flags)
+            except Exception as ex:
+                if not os.path.exists(tpath):
+                    t = "failed to move  [%s]  to  [%s]:  %r"
+                    self.log(t % (ttpath, tpath, ex), 3)
                 pass
 
+            untemp = []
             with self.mutex:
                 subs = self.busy[tpath]
                 del self.busy[tpath]
                 self.ram.pop(ttpath, None)
+                untemp = self.untemp.pop(ttpath, None) or []
+
+            for ap in untemp:
+                try:
+                    wunlink(self.log, ap, VF_CAREFUL)
+                except:
+                    pass
 
             for x in subs:
                 with x:
@@ -648,22 +682,50 @@ class ThumbSrv(object):
                 except:
                     pass
             else:
-                wrename(self.log, wtpath, tpath, vn.flags)
+                atomic_move(self.log, wtpath, tpath, vn.flags)
 
     def conv_spec(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in ret:
             raise Exception("not audio")
 
+        fext = abspath.split(".")[-1].lower()
+
         # https://trac.ffmpeg.org/ticket/10797
         # expect 1 GiB every 600 seconds when duration is tricky;
         # simple filetypes are generally safer so let's special-case those
-        safe = ("flac", "wav", "aif", "aiff", "opus")
-        coeff = 1800 if abspath.split(".")[-1].lower() in safe else 600
-        dur = ret[".dur"][1] if ".dur" in ret else 300
+        coeff = 1800 if fext in EXTS_SPEC_SAFE else 600
+        dur = ret[".dur"][1] if ".dur" in ret else 900
         need = 0.2 + dur / coeff
         self.wait4ram(need, tpath)
 
+        infile = abspath
+        if dur >= 900 or fext in self.exts_spec_unsafe:
+            with tempfile.NamedTemporaryFile(suffix=".spec.flac", delete=False) as f:
+                f.write(b"h")
+                infile = f.name
+                try:
+                    self.untemp[tpath].append(infile)
+                except:
+                    self.untemp[tpath] = [infile]
+
+            # fmt: off
+            cmd = [
+                b"ffmpeg",
+                b"-nostdin",
+                b"-v", b"error",
+                b"-hide_banner",
+                b"-i", fsenc(abspath),
+                b"-map", b"0:a:0",
+                b"-ac", b"1",
+                b"-ar", b"48000",
+                b"-sample_fmt", b"s16",
+                b"-t", b"900",
+                b"-y", fsenc(infile),
+            ]
+            # fmt: on
+            self._run_ff(cmd, vn)
+
         fc = "[0:a:0]aresample=48000{},showspectrumpic=s="
         if "3" in fmt:
             fc += "1280x1024,crop=1420:1056:70:48[o]"
@@ -683,7 +745,7 @@ class ThumbSrv(object):
             b"-nostdin",
             b"-v", b"error",
             b"-hide_banner",
-            b"-i", fsenc(abspath),
+            b"-i", fsenc(infile),
             b"-filter_complex", fc.encode("utf-8"),
             b"-map", b"[o]",
             b"-frames:v", b"1",
@@ -754,47 +816,102 @@ class ThumbSrv(object):
         if "ac" not in tags:
             raise Exception("not audio")
 
+        sq = "%dk" % (self.args.q_opus,)
+        bq = sq.encode("ascii")
+        if tags["ac"][1] == "opus":
+            enc = "-c:a copy"
+        else:
+            enc = "-c:a libopus -b:a " + sq
+
+        fun = self._conv_caf if fmt == "caf" else self._conv_owa
+
+        fun(abspath, tpath, tags, rawtags, enc, bq, vn)
+
+    def _conv_owa(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        if tpath.endswith(".owa"):
+            container = b"webm"
+            tagset = [b"-map_metadata", b"-1"]
+        else:
+            container = b"opus"
+            tagset = self.big_tags(rawtags)
+
+        self.log("conv2 %s [%s]" % (container, enc), 6)
+        benc = enc.encode("ascii").split(b" ")
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+        ] + tagset + [
+            b"-map", b"0:a:0",
+        ] + benc + [
+            b"-f", container,
+            fsenc(tpath)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, oom=300)
+
+    def _conv_caf(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        tmp_opus = tpath + ".opus"
+        try:
+            wunlink(self.log, tmp_opus, vn.flags)
+        except:
+            pass
+
         try:
             dur = tags[".dur"][1]
         except:
             dur = 0
 
-        src_opus = abspath.lower().endswith(".opus") or tags["ac"][1] == "opus"
-        want_caf = tpath.endswith(".caf")
-        tmp_opus = tpath
-        if want_caf:
-            tmp_opus = tpath + ".opus"
-            try:
-                wunlink(self.log, tmp_opus, vn.flags)
-            except:
-                pass
+        self.log("conv2 caf-tmp [%s]" % (enc,), 6)
+        benc = enc.encode("ascii").split(b" ")
 
-        caf_src = abspath if src_opus else tmp_opus
-        bq = ("%dk" % (self.args.q_opus,)).encode("ascii")
-
-        if not want_caf or not src_opus:
-            # fmt: off
-            cmd = [
-                b"ffmpeg",
-                b"-nostdin",
-                b"-v", b"error",
-                b"-hide_banner",
-                b"-i", fsenc(abspath),
-            ] + self.big_tags(rawtags) + [
-                b"-map", b"0:a:0",
-                b"-c:a", b"libopus",
-                b"-b:a", bq,
-                fsenc(tmp_opus)
-            ]
-            # fmt: on
-            self._run_ff(cmd, vn, oom=300)
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-map_metadata", b"-1",
+            b"-map", b"0:a:0",
+        ] + benc + [
+            b"-f", b"opus",
+            fsenc(tmp_opus)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, oom=300)
 
         # iOS fails to play some "insufficiently complex" files
         # (average file shorter than 8 seconds), so of course we
         # fix that by mixing in some inaudible pink noise :^)
         # 6.3 sec seems like the cutoff so lets do 7, and
         # 7 sec of psyqui-musou.opus @ 3:50 is 174 KiB
-        if want_caf and (dur < 20 or bos.path.getsize(caf_src) < 256 * 1024):
+        sz = bos.path.getsize(tmp_opus)
+        if dur < 20 or sz < 256 * 1024:
+            zs = bq.decode("ascii")
+            self.log("conv2 caf-transcode; dur=%d sz=%d q=%s" % (dur, sz, zs), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
@@ -813,15 +930,16 @@ class ThumbSrv(object):
             # fmt: on
             self._run_ff(cmd, vn, oom=300)
 
-        elif want_caf:
+        else:
             # simple remux should be safe
+            self.log("conv2 caf-remux; dur=%d sz=%d" % (dur, sz), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
                 b"-nostdin",
                 b"-v", b"error",
                 b"-hide_banner",
-                b"-i", fsenc(abspath if src_opus else tmp_opus),
+                b"-i", fsenc(tmp_opus),
                 b"-map_metadata", b"-1",
                 b"-map", b"0:a:0",
                 b"-c:a", b"copy",
@@ -831,11 +949,10 @@ class ThumbSrv(object):
             # fmt: on
             self._run_ff(cmd, vn, oom=300)
 
-        if tmp_opus != tpath:
-            try:
-                wunlink(self.log, tmp_opus, vn.flags)
-            except:
-                pass
+        try:
+            wunlink(self.log, tmp_opus, vn.flags)
+        except:
+            pass
 
     def big_tags(self, raw_tags: dict[str, list[str]]) -> list[bytes]:
         ret = []
@@ -891,7 +1008,7 @@ class ThumbSrv(object):
 
     def _clean(self, cat: str, thumbpath: str) -> int:
         # self.log("cln {}".format(thumbpath))
-        exts = ["jpg", "webp", "png"] if cat == "th" else ["opus", "caf", "mp3"]
+        exts = EXTS_TH if cat == "th" else EXTS_AC
         maxage = getattr(self.args, cat + "_maxage")
         now = time.time()
         prev_b64 = None
@@ -932,6 +1049,8 @@ class ThumbSrv(object):
             # thumb file
             try:
                 b64, ts, ext = f.split(".")
+                if len(ts) > 8 and PTN_TS.match(ts):
+                    ts = "yeahokay"
                 if len(b64) != 24 or len(ts) != 8 or ext not in exts:
                     raise Exception()
             except:

copyparty/u2idx.py

@@ -134,9 +134,9 @@ class U2idx(object):
         assert sqlite3  # type: ignore  # !rm
 
         ptop = vn.realpath
-        histpath = self.asrv.vfs.histtab.get(ptop)
+        histpath = self.asrv.vfs.dbpaths.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no dbpath for %r" % (ptop,))
             return None
 
         db_path = os.path.join(histpath, "up2k.db")
@@ -151,7 +151,7 @@ class U2idx(object):
                 db = sqlite3.connect(uri, timeout=2, uri=True, check_same_thread=False)
                 cur = db.cursor()
                 cur.execute('pragma table_info("up")').fetchone()
-                self.log("ro: {}".format(db_path))
+                self.log("ro: %r" % (db_path,))
             except:
                 self.log("could not open read-only: {}\n{}".format(uri, min_ex()))
                 # may not fail until the pragma so unset it
@@ -161,7 +161,7 @@ class U2idx(object):
             # on windows, this steals the write-lock from up2k.deferred_init --
             # seen on win 10.0.17763.2686, py 3.10.4, sqlite 3.37.2
             cur = sqlite3.connect(db_path, timeout=2, check_same_thread=False).cursor()
-            self.log("opened {}".format(db_path))
+            self.log("opened %r" % (db_path,))
 
         self.cur[ptop] = cur
         return cur

copyparty/util.py

@@ -31,6 +31,17 @@ from collections import Counter
 from ipaddress import IPv4Address, IPv4Network, IPv6Address, IPv6Network
 from queue import Queue
 
+try:
+    from zlib_ng import gzip_ng as gzip
+    from zlib_ng import zlib_ng as zlib
+
+    sys.modules["gzip"] = gzip
+    # sys.modules["zlib"] = zlib
+    # `- somehow makes tarfile 3% slower with default malloc, and barely faster with mimalloc
+except:
+    import gzip
+    import zlib
+
 from .__init__ import (
     ANYWIN,
     EXE,
@@ -94,6 +105,7 @@ def _ens(want: str) -> tuple[int, ...]:
 # WSAENOTSOCK - no longer a socket
 # EUNATCH - can't assign requested address (wifi down)
 E_SCK = _ens("ENOTCONN EUNATCH EBADF WSAENOTSOCK WSAECONNRESET")
+E_SCK_WR = _ens("EPIPE ESHUTDOWN EBADFD")
 E_ADDR_NOT_AVAIL = _ens("EADDRNOTAVAIL WSAEADDRNOTAVAIL")
 E_ADDR_IN_USE = _ens("EADDRINUSE WSAEADDRINUSE")
 E_ACCESS = _ens("EACCES WSAEACCES")
@@ -103,8 +115,14 @@ IP6ALL = "0:0:0:0:0:0:0:0"
 
 
 try:
-    import ctypes
     import fcntl
+
+    HAVE_FCNTL = True
+except:
+    HAVE_FCNTL = False
+
+try:
+    import ctypes
     import termios
 except:
     pass
@@ -120,6 +138,13 @@ try:
 except:
     HAVE_SQLITE3 = False
 
+try:
+    import importlib.util
+
+    HAVE_ZMQ = bool(importlib.util.find_spec("zmq"))
+except:
+    HAVE_ZMQ = False
+
 try:
     if os.environ.get("PRTY_NO_PSUTIL"):
         raise Exception()
@@ -129,6 +154,16 @@ try:
 except:
     HAVE_PSUTIL = False
 
+try:
+    if os.environ.get("PRTY_NO_MAGIC") or (
+        ANYWIN and not os.environ.get("PRTY_FORCE_MAGIC")
+    ):
+        raise Exception()
+
+    import magic
+except:
+    pass
+
 if True:  # pylint: disable=using-constant-test
     import types
     from collections.abc import Callable, Iterable
@@ -151,8 +186,6 @@ if True:  # pylint: disable=using-constant-test
 
 
 if TYPE_CHECKING:
-    import magic
-
     from .authsrv import VFS
     from .broker_util import BrokerCli
     from .up2k import Up2k
@@ -210,7 +243,18 @@ except:
     BITNESS = struct.calcsize("P") * 8
 
 
-ansi_re = re.compile("\033\\[[^mK]*[mK]")
+RE_ANSI = re.compile("\033\\[[^mK]*[mK]")
+RE_HTML_SH = re.compile(r"[<>&$?`\"';]")
+RE_CTYPE = re.compile(r"^content-type: *([^; ]+)", re.IGNORECASE)
+RE_CDISP = re.compile(r"^content-disposition: *([^; ]+)", re.IGNORECASE)
+RE_CDISP_FIELD = re.compile(
+    r'^content-disposition:(?: *|.*; *)name="([^"]+)"', re.IGNORECASE
+)
+RE_CDISP_FILE = re.compile(
+    r'^content-disposition:(?: *|.*; *)filename="(.*)"', re.IGNORECASE
+)
+RE_MEMTOTAL = re.compile("^MemTotal:.* kB")
+RE_MEMAVAIL = re.compile("^MemAvailable:.* kB")
 
 
 BOS_SEP = ("%s" % (os.sep,)).encode("ascii")
@@ -227,11 +271,19 @@ SYMTIME = PY36 and os.utime in os.supports_follow_symlinks
 
 META_NOBOTS = '<meta name="robots" content="noindex, nofollow">\n'
 
+# smart enough to understand javascript while also ignoring rel="nofollow"
+BAD_BOTS = r"Barkrowler|bingbot|BLEXBot|Googlebot|GoogleOther|GPTBot|PetalBot|SeekportBot|SemrushBot|YandexBot"
+
 FFMPEG_URL = "https://www.gyan.dev/ffmpeg/builds/ffmpeg-git-full.7z"
 
+URL_PRJ = "https://github.com/9001/copyparty"
+
+URL_BUG = URL_PRJ + "/issues/new?labels=bug&template=bug_report.md"
+
 HTTPCODE = {
     200: "OK",
     201: "Created",
+    202: "Accepted",
     204: "No Content",
     206: "Partial Content",
     207: "Multi-Status",
@@ -319,6 +371,7 @@ DAV_ALLPROPS = set(DAV_ALLPROP_L)
 
 MIMES = {
     "opus": "audio/ogg; codecs=opus",
+    "owa": "audio/webm; codecs=opus",
 }
 
 
@@ -435,18 +488,22 @@ UNHUMANIZE_UNITS = {
 
 VF_CAREFUL = {"mv_re_t": 5, "rm_re_t": 5, "mv_re_r": 0.1, "rm_re_r": 0.1}
 
+FN_EMB = set([".prologue.html", ".epilogue.html", "readme.md", "preadme.md"])
+
 
 def read_ram() -> tuple[float, float]:
+    # NOTE: apparently no need to consider /sys/fs/cgroup/memory.max
+    #  (cgroups2) since the limit is synced to /proc/meminfo
     a = b = 0
     try:
         with open("/proc/meminfo", "rb", 0x10000) as f:
             zsl = f.read(0x10000).decode("ascii", "replace").split("\n")
 
-        p = re.compile("^MemTotal:.* kB")
+        p = RE_MEMTOTAL
         zs = next((x for x in zsl if p.match(x)))
         a = int((int(zs.split()[1]) / 0x100000) * 100) / 100
 
-        p = re.compile("^MemAvailable:.* kB")
+        p = RE_MEMAVAIL
         zs = next((x for x in zsl if p.match(x)))
         b = int((int(zs.split()[1]) / 0x100000) * 100) / 100
     except:
@@ -491,6 +548,15 @@ def py_desc() -> str:
     )
 
 
+def expat_ver() -> str:
+    try:
+        import pyexpat
+
+        return ".".join([str(x) for x in pyexpat.version_info])
+    except:
+        return "?"
+
+
 def _sqlite_ver() -> str:
     assert sqlite3  # type: ignore  # !rm
     try:
@@ -572,6 +638,38 @@ except Exception as ex:
         print("using fallback base64 codec due to %r" % (ex,))
 
 
+class NotUTF8(Exception):
+    pass
+
+
+def read_utf8(log: Optional["NamedLogger"], ap: Union[str, bytes], strict: bool) -> str:
+    with open(ap, "rb") as f:
+        buf = f.read()
+
+    try:
+        return buf.decode("utf-8", "strict")
+    except UnicodeDecodeError as ex:
+        eo = ex.start
+        eb = buf[eo : eo + 1]
+
+    if not strict:
+        t = "WARNING: The file [%s] is not using the UTF-8 character encoding; some characters in the file will be skipped/ignored. The first unreadable character was byte %r at offset %d. Please convert this file to UTF-8 by opening the file in your text-editor and saving it as UTF-8."
+        t = t % (ap, eb, eo)
+        if log:
+            log(t, 3)
+        else:
+            print(t)
+        return buf.decode("utf-8", "replace")
+
+    t = "ERROR: The file [%s] is not using the UTF-8 character encoding, and cannot be loaded. The first unreadable character was byte %r at offset %d. Please convert this file to UTF-8 by opening the file in your text-editor and saving it as UTF-8."
+    t = t % (ap, eb, eo)
+    if log:
+        log(t, 3)
+    else:
+        print(t)
+    raise NotUTF8(t)
+
+
 class Daemon(threading.Thread):
     def __init__(
         self,
@@ -1024,7 +1122,7 @@ class ProgressPrinter(threading.Thread):
             now = time.time()
             if msg and now - tp > 10:
                 tp = now
-                self.log("progress: %s" % (msg,), 6)
+                self.log("progress: %r" % (msg,), 6)
 
             if no_stdout:
                 continue
@@ -1178,8 +1276,6 @@ class Magician(object):
         self.magic: Optional["magic.Magic"] = None
 
     def ext(self, fpath: str) -> str:
-        import magic
-
         try:
             if self.bad_magic:
                 raise Exception()
@@ -1397,8 +1493,6 @@ def stackmon(fp: str, ival: float, suffix: str) -> None:
         buf = st.encode("utf-8", "replace")
 
         if fp.endswith(".gz"):
-            import gzip
-
             # 2459b 2304b 2241b 2202b 2194b 2191b lv3..8
             # 0.06s 0.08s 0.11s 0.13s 0.16s 0.19s
             buf = gzip.compress(buf, compresslevel=6)
@@ -1478,6 +1572,12 @@ def vol_san(vols: list["VFS"], txt: bytes) -> bytes:
         txt = txt.replace(bap.replace(b"\\", b"\\\\"), bvp)
         txt = txt.replace(bhp.replace(b"\\", b"\\\\"), bvph)
 
+        if vol.histpath != vol.dbpath:
+            bdp = vol.dbpath.encode("utf-8")
+            bdph = b"$db(/" + bvp + b")"
+            txt = txt.replace(bdp, bdph)
+            txt = txt.replace(bdp.replace(b"\\", b"\\\\"), bdph)
+
     if txt != txt0:
         txt += b"\r\nNOTE: filepaths sanitized; see serverlog for correct values"
 
@@ -1498,6 +1598,8 @@ def ren_open(fname: str, *args: Any, **kwargs: Any) -> tuple[typing.IO[Any], str
     fun = kwargs.pop("fun", open)
     fdir = kwargs.pop("fdir", None)
     suffix = kwargs.pop("suffix", None)
+    vf = kwargs.pop("vf", None)
+    fperms = vf and "fperms" in vf
 
     if fname == os.devnull:
         return fun(fname, *args, **kwargs), fname
@@ -1541,6 +1643,11 @@ def ren_open(fname: str, *args: Any, **kwargs: Any) -> tuple[typing.IO[Any], str
                 fp2 = os.path.join(fdir, fp2)
                 with open(fsenc(fp2), "wb") as f2:
                     f2.write(orig_name.encode("utf-8"))
+                    if fperms:
+                        set_fperms(f2, vf)
+
+            if fperms:
+                set_fperms(f, vf)
 
             return f, fname
 
@@ -1602,14 +1709,10 @@ class MultipartParser(object):
         self.args = args
         self.headers = http_headers
 
-        self.re_ctype = re.compile(r"^content-type: *([^; ]+)", re.IGNORECASE)
-        self.re_cdisp = re.compile(r"^content-disposition: *([^; ]+)", re.IGNORECASE)
-        self.re_cdisp_field = re.compile(
-            r'^content-disposition:(?: *|.*; *)name="([^"]+)"', re.IGNORECASE
-        )
-        self.re_cdisp_file = re.compile(
-            r'^content-disposition:(?: *|.*; *)filename="(.*)"', re.IGNORECASE
-        )
+        self.re_ctype = RE_CTYPE
+        self.re_cdisp = RE_CDISP
+        self.re_cdisp_field = RE_CDISP_FIELD
+        self.re_cdisp_file = RE_CDISP_FILE
 
         self.boundary = b""
         self.gen: Optional[
@@ -1626,7 +1729,7 @@ class MultipartParser(object):
         (only the fallback non-js uploader relies on these filenames)
         """
         for ln in read_header(self.sr, 2, 2592000):
-            self.log(ln)
+            self.log(repr(ln))
 
             m = self.re_ctype.match(ln)
             if m:
@@ -1881,7 +1984,7 @@ def rand_name(fdir: str, fn: str, rnd: int) -> str:
     return fn
 
 
-def gen_filekey(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str:
+def _gen_filekey(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str:
     if alg == 1:
         zs = "%s %s %s %s" % (salt, fspath, fsize, inode)
     else:
@@ -1891,6 +1994,13 @@ def gen_filekey(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str
     return ub64enc(hashlib.sha512(zb).digest()).decode("ascii")
 
 
+def _gen_filekey_w(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str:
+    return _gen_filekey(alg, salt, fspath.replace("/", "\\"), fsize, inode)
+
+
+gen_filekey = _gen_filekey_w if ANYWIN else _gen_filekey
+
+
 def gen_filekey_dbg(
     alg: int,
     salt: str,
@@ -1917,11 +2027,11 @@ def gen_filekey_dbg(
             if p2 != fspath:
                 raise Exception()
         except:
-            t = "maybe wrong abspath for filekey;\norig: {}\nreal: {}"
-            log(t.format(fspath, p2), 1)
+            t = "maybe wrong abspath for filekey;\norig: %r\nreal: %r"
+            log(t % (fspath, p2), 1)
 
-        t = "fk({}) salt({}) size({}) inode({}) fspath({}) at({})"
-        log(t.format(ret[:8], salt, fsize, inode, fspath, ctx), 5)
+        t = "fk(%s) salt(%s) size(%d) inode(%d) fspath(%r) at(%s)"
+        log(t % (ret[:8], salt, fsize, inode, fspath, ctx), 5)
 
     return ret
 
@@ -1937,15 +2047,25 @@ def formatdate(ts: Optional[float] = None) -> str:
     return RFC2822 % (WKDAYS[wd], d, MONTHS[mo - 1], y, h, mi, s)
 
 
-def gencookie(k: str, v: str, r: str, tls: bool, dur: int = 0, txt: str = "") -> str:
+def gencookie(
+    k: str, v: str, r: str, lax: bool, tls: bool, dur: int = 0, txt: str = ""
+) -> str:
     v = v.replace("%", "%25").replace(";", "%3B")
     if dur:
         exp = formatdate(time.time() + dur)
     else:
         exp = "Fri, 15 Aug 1997 01:00:00 GMT"
 
-    t = "%s=%s; Path=/%s; Expires=%s%s%s; SameSite=Lax"
-    return t % (k, v, r, exp, "; Secure" if tls else "", txt)
+    t = "%s=%s; Path=/%s; Expires=%s%s%s; SameSite=%s"
+    return t % (
+        k,
+        v,
+        r,
+        exp,
+        "; Secure" if tls else "",
+        txt,
+        "Lax" if lax else "Strict",
+    )
 
 
 def humansize(sz: float, terse: bool = False) -> str:
@@ -2134,6 +2254,16 @@ def find_prefix(ips: list[str], cidrs: list[str]) -> list[str]:
     return ret
 
 
+def html_sh_esc(s: str) -> str:
+    s = re.sub(RE_HTML_SH, "_", s).replace(" ", "%20")
+    s = s.replace("\r", "_").replace("\n", "_")
+    return s
+
+
+def json_hesc(s: str) -> str:
+    return s.replace("<", "\\u003c").replace(">", "\\u003e").replace("&", "\\u0026")
+
+
 def html_escape(s: str, quot: bool = False, crlf: bool = False) -> str:
     """html.escape but also newlines"""
     s = s.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
@@ -2277,7 +2407,7 @@ def log_reloc(
     rem: str,
 ) -> None:
     nap, nvp, nfn, (nvn, nrem) = pm
-    t = "reloc %s:\nold ap [%s]\nnew ap [%s\033[36m/%s\033[0m]\nold vp [%s]\nnew vp [%s\033[36m/%s\033[0m]\nold fn [%s]\nnew fn [%s]\nold vfs [%s]\nnew vfs [%s]\nold rem [%s]\nnew rem [%s]"
+    t = "reloc %s:\nold ap %r\nnew ap %r\033[36m/%r\033[0m\nold vp %r\nnew vp %r\033[36m/%r\033[0m\nold fn %r\nnew fn %r\nold vfs %r\nnew vfs %r\nold rem %r\nnew rem %r"
     log(t % (re, ap, nap, nfn, vp, nvp, nfn, fn, nfn, vn.vpath, nvn.vpath, rem, nrem))
 
 
@@ -2314,11 +2444,11 @@ def pathmod(
 
         # try to map abspath to vpath
         np = np.replace("/", os.sep)
-        for vn_ap, vn in vfs.all_aps:
+        for vn_ap, vns in vfs.all_aps:
             if not np.startswith(vn_ap):
                 continue
             zs = np[len(vn_ap) :].replace(os.sep, "/")
-            nvp = vjoin(vn.vpath, zs)
+            nvp = vjoin(vns[0].vpath, zs)
             break
 
     if nvp == "\n":
@@ -2448,11 +2578,19 @@ def lsof(log: "NamedLogger", abspath: str) -> None:
     try:
         rc, so, se = runcmd([b"lsof", b"-R", fsenc(abspath)], timeout=45)
         zs = (so.strip() + "\n" + se.strip()).strip()
-        log("lsof {} = {}\n{}".format(abspath, rc, zs), 3)
+        log("lsof %r = %s\n%s" % (abspath, rc, zs), 3)
     except:
         log("lsof failed; " + min_ex(), 3)
 
 
+def set_fperms(f: Union[typing.BinaryIO, typing.IO[Any]], vf: dict[str, Any]) -> None:
+    fno = f.fileno()
+    if "chmod_f" in vf:
+        os.fchmod(fno, vf["chmod_f"])
+    if "chown" in vf:
+        os.fchown(fno, vf["uid"], vf["gid"])
+
+
 def _fs_mvrm(
     log: "NamedLogger", src: str, dst: str, atomic: bool, flags: dict[str, Any]
 ) -> bool:
@@ -2484,29 +2622,34 @@ def _fs_mvrm(
     for attempt in range(90210):
         try:
             if ino and os.stat(bsrc).st_ino != ino:
-                t = "src inode changed; aborting %s %s"
+                t = "src inode changed; aborting %s %r"
                 log(t % (act, src), 1)
                 return False
             if (dst and not atomic) and os.path.exists(bdst):
-                t = "something appeared at dst; aborting rename [%s] ==> [%s]"
+                t = "something appeared at dst; aborting rename %r ==> %r"
                 log(t % (src, dst), 1)
                 return False
             osfun(*args)
             if attempt:
                 now = time.time()
-                t = "%sd in %.2f sec, attempt %d: %s"
+                t = "%sd in %.2f sec, attempt %d: %r"
                 log(t % (act, now - t0, attempt + 1, src))
             return True
         except OSError as ex:
             now = time.time()
             if ex.errno == errno.ENOENT:
                 return False
+            if not attempt and ex.errno == errno.EXDEV:
+                t = "using copy+delete (%s)\n  %s\n  %s"
+                log(t % (ex.strerror, src, dst))
+                osfun = shutil.move
+                continue
             if now - t0 > maxtime or attempt == 90209:
                 raise
             if not attempt:
                 if not PY2:
                     ino = os.stat(bsrc).st_ino
-                t = "%s failed (err.%d); retrying for %d sec: [%s]"
+                t = "%s failed (err.%d); retrying for %d sec: %r"
                 log(t % (act, ex.errno, maxtime + 0.99, src))
 
         time.sleep(chill)
@@ -2525,15 +2668,18 @@ def atomic_move(log: "NamedLogger", src: str, dst: str, flags: dict[str, Any]) -
     elif flags.get("mv_re_t"):
         _fs_mvrm(log, src, dst, True, flags)
     else:
-        os.replace(bsrc, bdst)
-
-
-def wrename(log: "NamedLogger", src: str, dst: str, flags: dict[str, Any]) -> bool:
-    if not flags.get("mv_re_t"):
-        os.rename(fsenc(src), fsenc(dst))
-        return True
-
-    return _fs_mvrm(log, src, dst, False, flags)
+        try:
+            os.replace(bsrc, bdst)
+        except OSError as ex:
+            if ex.errno != errno.EXDEV:
+                raise
+            t = "using copy+delete (%s);\n  %s\n  %s"
+            log(t % (ex.strerror, src, dst))
+            try:
+                os.unlink(bdst)
+            except:
+                pass
+            shutil.move(bsrc, bdst)
 
 
 def wunlink(log: "NamedLogger", abspath: str, flags: dict[str, Any]) -> bool:
@@ -2544,7 +2690,7 @@ def wunlink(log: "NamedLogger", abspath: str, flags: dict[str, Any]) -> bool:
     return _fs_mvrm(log, abspath, "", False, flags)
 
 
-def get_df(abspath: str, prune: bool) -> tuple[Optional[int], Optional[int], str]:
+def get_df(abspath: str, prune: bool) -> tuple[int, int, str]:
     try:
         ap = fsenc(abspath)
         while prune and not os.path.isdir(ap) and BOS_SEP in ap:
@@ -2555,17 +2701,22 @@ def get_df(abspath: str, prune: bool) -> tuple[Optional[int], Optional[int], str
             assert ctypes  # type: ignore  # !rm
             abspath = fsdec(ap)
             bfree = ctypes.c_ulonglong(0)
+            btotal = ctypes.c_ulonglong(0)
+            bavail = ctypes.c_ulonglong(0)
             ctypes.windll.kernel32.GetDiskFreeSpaceExW(  # type: ignore
-                ctypes.c_wchar_p(abspath), None, None, ctypes.pointer(bfree)
+                ctypes.c_wchar_p(abspath),
+                ctypes.pointer(bavail),
+                ctypes.pointer(btotal),
+                ctypes.pointer(bfree),
             )
-            return (bfree.value, None, "")
+            return (bavail.value, btotal.value, "")
         else:
             sv = os.statvfs(ap)
-            free = sv.f_frsize * sv.f_bfree
+            free = sv.f_frsize * sv.f_bavail
             total = sv.f_frsize * sv.f_blocks
             return (free, total, "")
     except Exception as ex:
-        return (None, None, repr(ex))
+        return (0, 0, repr(ex))
 
 
 if not ANYWIN and not MACOS:
@@ -3062,11 +3213,13 @@ def unescape_cookie(orig: str) -> str:
     return "".join(ret)
 
 
-def guess_mime(url: str, fallback: str = "application/octet-stream") -> str:
+def guess_mime(
+    url: str, path: str = "", fallback: str = "application/octet-stream"
+) -> str:
     try:
         ext = url.rsplit(".", 1)[1].lower()
     except:
-        return fallback
+        ext = ""
 
     ret = MIMES.get(ext)
 
@@ -3074,6 +3227,16 @@ def guess_mime(url: str, fallback: str = "application/octet-stream") -> str:
         x = mimetypes.guess_type(url)
         ret = "application/{}".format(x[1]) if x[1] else x[0]
 
+    if not ret and path:
+        try:
+            with open(fsenc(path), "rb", 0) as f:
+                ret = magic.from_buffer(f.read(4096), mime=True)
+                if ret.startswith("text/htm"):
+                    # avoid serving up HTML content unless there was actually a .html extension
+                    ret = "text/plain"
+        except Exception as ex:
+            pass
+
     if not ret:
         ret = fallback
 
@@ -3386,6 +3549,7 @@ def _parsehook(
 
 def runihook(
     log: Optional["NamedLogger"],
+    verbose: bool,
     cmd: str,
     vol: "VFS",
     ups: list[tuple[str, int, int, str, str, str, int]],
@@ -3415,6 +3579,17 @@ def runihook(
     else:
         sp_ka["sin"] = b"\n".join(fsenc(x) for x in aps)
 
+    if acmd[0].startswith("zmq:"):
+        try:
+            msg = sp_ka["sin"].decode("utf-8", "replace")
+            _zmq_hook(log, verbose, "xiu", acmd[0][4:].lower(), msg, wait, sp_ka)
+            if verbose and log:
+                log("hook(xiu) %r OK" % (cmd,), 6)
+        except Exception as ex:
+            if log:
+                log("zeromq failed: %r" % (ex,))
+        return True
+
     t0 = time.time()
     if fork:
         Daemon(runcmd, cmd, bcmd, ka=sp_ka)
@@ -3424,15 +3599,126 @@ def runihook(
             retchk(rc, bcmd, err, log, 5)
             return False
 
-    wait -= time.time() - t0
-    if wait > 0:
-        time.sleep(wait)
+    if wait:
+        wait -= time.time() - t0
+        if wait > 0:
+            time.sleep(wait)
 
     return True
 
 
+ZMQ = {}
+ZMQ_DESC = {
+    "pub": "fire-and-forget to all/any connected SUB-clients",
+    "push": "fire-and-forget to one of the connected PULL-clients",
+    "req": "send messages to a REP-server and blocking-wait for ack",
+}
+
+
+def _zmq_hook(
+    log: Optional["NamedLogger"],
+    verbose: bool,
+    src: str,
+    cmd: str,
+    msg: str,
+    wait: float,
+    sp_ka: dict[str, Any],
+) -> tuple[int, str]:
+    import zmq
+
+    try:
+        mtx = ZMQ["mtx"]
+    except:
+        ZMQ["mtx"] = threading.Lock()
+        time.sleep(0.1)
+        mtx = ZMQ["mtx"]
+
+    ret = ""
+    nret = 0
+    t0 = time.time()
+    if verbose and log:
+        log("hook(%s) %r entering zmq-main-lock" % (src, cmd), 6)
+
+    with mtx:
+        try:
+            mode, sck, mtx = ZMQ[cmd]
+        except:
+            mode, uri = cmd.split(":", 1)
+            try:
+                desc = ZMQ_DESC[mode]
+                if log:
+                    t = "libzmq(%s) pyzmq(%s) init(%s); %s"
+                    log(t % (zmq.zmq_version(), zmq.__version__, cmd, desc))
+            except:
+                raise Exception("the only supported ZMQ modes are REQ PUB PUSH")
+
+            try:
+                ctx = ZMQ["ctx"]
+            except:
+                ctx = ZMQ["ctx"] = zmq.Context()
+
+            timeout = sp_ka["timeout"]
+
+            if mode == "pub":
+                sck = ctx.socket(zmq.PUB)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.bind(uri)
+                time.sleep(1)  # give clients time to connect; avoids losing first msg
+            elif mode == "push":
+                sck = ctx.socket(zmq.PUSH)
+                if timeout:
+                    sck.SNDTIMEO = int(timeout * 1000)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.bind(uri)
+            elif mode == "req":
+                sck = ctx.socket(zmq.REQ)
+                if timeout:
+                    sck.RCVTIMEO = int(timeout * 1000)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.connect(uri)
+            else:
+                raise Exception()
+
+            mtx = threading.Lock()
+            ZMQ[cmd] = (mode, sck, mtx)
+
+    if verbose and log:
+        log("hook(%s) %r entering socket-lock" % (src, cmd), 6)
+
+    with mtx:
+        if verbose and log:
+            log("hook(%s) %r sending |%d|" % (src, cmd, len(msg)), 6)
+
+        sck.send_string(msg)  # PUSH can safely timeout here
+
+        if mode == "req":
+            if verbose and log:
+                log("hook(%s) %r awaiting ack from req" % (src, cmd), 6)
+            try:
+                ret = sck.recv().decode("utf-8", "replace")
+                if ret.startswith("return "):
+                    m = re.search("^return ([0-9]+)", ret[:12])
+                    if m:
+                        nret = int(m.group(1))
+            except:
+                sck.close()
+                del ZMQ[cmd]  # bad state; must reset
+                raise Exception("ack timeout; zmq socket killed")
+
+    if ret and log:
+        log("hook(%s) %r ACK: %r" % (src, cmd, ret), 6)
+
+    if wait:
+        wait -= time.time() - t0
+        if wait > 0:
+            time.sleep(wait)
+
+    return nret, ret
+
+
 def _runhook(
     log: Optional["NamedLogger"],
+    verbose: bool,
     src: str,
     cmd: str,
     ap: str,
@@ -3473,6 +3759,12 @@ def _runhook(
     else:
         arg = txt or ap
 
+    if acmd[0].startswith("zmq:"):
+        zi, zs = _zmq_hook(log, verbose, src, acmd[0][4:].lower(), arg, wait, sp_ka)
+        if zi:
+            raise Exception("zmq says %d" % (zi,))
+        return {"rc": 0, "stdout": zs}
+
     acmd += [arg]
     if acmd[0].endswith(".py"):
         acmd = [pybin] + acmd
@@ -3501,9 +3793,10 @@ def _runhook(
             except:
                 ret = {"rc": rc, "stdout": v}
 
-    wait -= time.time() - t0
-    if wait > 0:
-        time.sleep(wait)
+    if wait:
+        wait -= time.time() - t0
+        if wait > 0:
+            time.sleep(wait)
 
     return ret
 
@@ -3527,15 +3820,16 @@ def runhook(
 ) -> dict[str, Any]:
     assert broker or up2k  # !rm
     args = (broker or up2k).args
+    verbose = args.hook_v
     vp = vp.replace("\\", "/")
     ret = {"rc": 0}
     for cmd in cmds:
         try:
             hr = _runhook(
-                log, src, cmd, ap, vp, host, uname, perms, mt, sz, ip, at, txt
+                log, verbose, src, cmd, ap, vp, host, uname, perms, mt, sz, ip, at, txt
             )
-            if log and args.hook_v:
-                log("hook(%s) [%s] => \033[32m%s" % (src, cmd, hr), 6)
+            if verbose and log:
+                log("hook(%s) %r => \033[32m%s" % (src, cmd, hr), 6)
             if not hr:
                 return {}
             for k, v in hr.items():
@@ -3550,6 +3844,8 @@ def runhook(
                 elif k in ret:
                     if k == "rc" and v:
                         ret[k] = v
+                    elif k == "stdout" and v and not ret[k]:
+                        ret[k] = v
                 else:
                     ret[k] = v
         except Exception as ex:
@@ -3733,8 +4029,75 @@ def hidedir(dp) -> None:
             pass
 
 
+_flocks = {}
+
+
+def _lock_file_noop(ap: str) -> bool:
+    return True
+
+
+def _lock_file_ioctl(ap: str) -> bool:
+    assert fcntl  # type: ignore  # !rm
+    try:
+        fd = _flocks.pop(ap)
+        os.close(fd)
+    except:
+        pass
+
+    fd = os.open(ap, os.O_RDWR | os.O_CREAT, 438)
+    # NOTE: the fcntl.lockf identifier is (pid,node);
+    #  the lock will be dropped if os.close(os.open(ap))
+    #  is performed anywhere else in this thread
+
+    try:
+        fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        _flocks[ap] = fd
+        return True
+    except Exception as ex:
+        eno = getattr(ex, "errno", -1)
+        try:
+            os.close(fd)
+        except:
+            pass
+        if eno in (errno.EAGAIN, errno.EACCES):
+            return False
+        print("WARNING: unexpected errno %d from fcntl.lockf; %r" % (eno, ex))
+        return True
+
+
+def _lock_file_windows(ap: str) -> bool:
+    try:
+        import msvcrt
+
+        try:
+            fd = _flocks.pop(ap)
+            os.close(fd)
+        except:
+            pass
+
+        fd = os.open(ap, os.O_RDWR | os.O_CREAT, 438)
+        msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
+        return True
+    except Exception as ex:
+        eno = getattr(ex, "errno", -1)
+        if eno == errno.EACCES:
+            return False
+        print("WARNING: unexpected errno %d from msvcrt.locking; %r" % (eno, ex))
+        return True
+
+
+if os.environ.get("PRTY_NO_DB_LOCK"):
+    lock_file = _lock_file_noop
+elif ANYWIN:
+    lock_file = _lock_file_windows
+elif HAVE_FCNTL:
+    lock_file = _lock_file_ioctl
+else:
+    lock_file = _lock_file_noop
+
+
 try:
-    if sys.version_info < (3, 10):
+    if sys.version_info < (3, 10) or os.environ.get("PRTY_NO_IMPRESO"):
         # py3.8 doesn't have .files
         # py3.9 has broken .is_file
         raise ImportError()
@@ -3834,7 +4197,14 @@ def load_resource(E: EnvParams, name: str, mode="rb") -> IO[bytes]:
                 stream = codecs.getreader(enc)(stream)
             return stream
 
-    return open(os.path.join(E.mod, name), mode, encoding=enc)
+    ap = os.path.join(E.mod, name)
+
+    if PY2:
+        import codecs
+
+        return codecs.open(ap, "r", encoding=enc)  # type: ignore
+
+    return open(ap, mode, encoding=enc)
 
 
 class Pebkac(Exception):
@@ -3866,9 +4236,22 @@ class WrongPostKey(Pebkac):
         self.datagen = datagen
 
 
-_: Any = (mp, BytesIO, quote, unquote, SQLITE_VER, JINJA_VER, PYFTPD_VER, PARTFTPY_VER)
+_: Any = (
+    gzip,
+    mp,
+    zlib,
+    BytesIO,
+    quote,
+    unquote,
+    SQLITE_VER,
+    JINJA_VER,
+    PYFTPD_VER,
+    PARTFTPY_VER,
+)
 __all__ = [
+    "gzip",
     "mp",
+    "zlib",
     "BytesIO",
     "quote",
     "unquote",

copyparty/web/baguettebox.js

@@ -592,9 +592,7 @@ window.baguetteBox = (function () {
             preloadPrev(currentIndex);
         });
 
-        clmod(ebi('bbox-btns'), 'off');
-        clmod(btnPrev, 'off');
-        clmod(btnNext, 'off');
+        show_buttons(0);
 
         updateOffset();
         overlay.style.display = 'block';
@@ -633,6 +631,9 @@ window.baguetteBox = (function () {
         catch (ex) { }
         isFullscreen = false;
 
+        if (toast.tag == 'bb-ded')
+            toast.hide();
+
         if (dtor || overlay.style.display === 'none')
             return;
 
@@ -668,6 +669,7 @@ window.baguetteBox = (function () {
             if (v == keep)
                 continue;
 
+            unbind(v, 'error', lerr);
             v.src = '';
             v.load();
 
@@ -695,6 +697,28 @@ window.baguetteBox = (function () {
         }
     }
 
+    function lerr() {
+        var t;
+        try {
+            t = this.getAttribute('src');
+            t = uricom_dec(t.split('/').pop().split('?')[0]);
+        }
+        catch (ex) { }
+
+        t = 'Failed to open ' + (t?t:'file');
+        console.log('bb-ded', t);
+        t += '\n\nEither the file is corrupt, or your browser does not understand the file format or codec';
+
+        try {
+            t += "\n\nerr#" + this.error.code + ", " + this.error.message;
+        }
+        catch (ex) { }
+ 
+        this.ded = esc(t);
+        if (this === vidimg())
+            toast.err(20, this.ded, 'bb-ded');
+    }
+
     function loadImage(index, callback) {
         var imageContainer = imagesElements[index];
         var galleryItem = currentGallery[index];
@@ -739,7 +763,8 @@ window.baguetteBox = (function () {
         var image = mknod(is_vid ? 'video' : 'img');
         clmod(imageContainer, 'vid', is_vid);
 
-        image.addEventListener(is_vid ? 'loadedmetadata' : 'load', function () {
+        bind(image, 'error', lerr);
+        bind(image, is_vid ? 'loadedmetadata' : 'load', function () {
             // Remove loader element
             qsr('#baguette-img-' + index + ' .bbox-spinner');
             if (!options.async && callback)
@@ -749,6 +774,8 @@ window.baguetteBox = (function () {
         if (is_vid) {
             image.volume = clamp(fcfg_get('vol', dvol / 100), 0, 1);
             image.setAttribute('controls', 'controls');
+            image.setAttribute('playsinline', '1');
+            // ios ignores poster
             image.onended = vidEnd;
             image.onplay = function () { show_buttons(1); };
             image.onpause = function () { show_buttons(); };
@@ -816,6 +843,12 @@ window.baguetteBox = (function () {
         });
         updateOffset();
 
+        var im = vidimg();
+        if (im && im.ded)
+            toast.err(20, im.ded, 'bb-ded');
+        else if (toast.tag == 'bb-ded')
+            toast.hide();
+
         if (options.animation == 'none')
             unvid(vid());
         else

copyparty/web/browser.css

@@ -4,6 +4,8 @@
 	--grid-sz: 10em;
 	--grid-ln: 3;
 	--nav-sz: 16em;
+	--sbw: 0.5em;
+	--sbh: 0.5em;
 
 	--fg: #ccc;
 	--fg-max: #fff;
@@ -1151,17 +1153,17 @@ html.y #widget.open {
 	background: #fff;
 	background: var(--bg-u3);
 }
-#wfs, #wfm, #wzip, #wnp {
+#wfs, #wfm, #wzip, #wnp, #wm3u {
 	display: none;
 }
-#wfs, #wzip, #wnp {
+#wfs, #wzip, #wnp, #wm3u {
 	margin-right: .2em;
 	padding-right: .2em;
 	border: 1px solid var(--bg-u5);
 	border-width: 0 .1em 0 0;
 }
-#wfm.act+#wzip,
-#wfm.act+#wzip+#wnp {
+#wfm.act+#wzip1+#wzip,
+#wfm.act+#wzip1+#wzip+#wnp {
 	margin-left: .2em;
 	padding-left: .2em;
 	border-left-width: .1em;
@@ -1175,14 +1177,18 @@ html.y #widget.open {
 	line-height: 1em;
 }
 #wtoggle.sel #wzip,
+#wtoggle.m3u #wm3u,
 #wtoggle.np #wnp {
 	display: inline-block;
 }
+#wtoggle.sel #wzip1,
 #wtoggle.sel.np #wnp {
 	display: none;
 }
 #wfm a,
 #wnp a,
+#wm3u a,
+#zip1,
 #wzip a {
 	font-size: .5em;
 	padding: 0 .3em;
@@ -1190,6 +1196,13 @@ html.y #widget.open {
 	position: relative;
 	display: inline-block;
 }
+#zip1 {
+	font-size: .38em;
+}
+#wm3u a {
+	margin: -.2em .1em;
+	font-size: .45em;
+}
 #wfs {
 	font-size: .36em;
 	text-align: right;
@@ -1198,13 +1211,22 @@ html.y #widget.open {
 	border-width: 0 .25em 0 0;
 }
 #wfm span,
+#wm3u span,
+#zip1 span,
 #wnp span {
 	font-size: .6em;
 	display: block;
 }
+#zip1 span {
+	font-size: .9em;
+}
 #wnp span {
 	font-size: .7em;
 }
+#wm3u span {
+	font-size: .77em;
+	padding-top: .2em;
+}
 #wfm a:not(.en) {
 	opacity: .3;
 	color: var(--fm-off);
@@ -1538,8 +1560,8 @@ html {
 	z-index: 1;
 	position: fixed;
 	background: var(--tree-bg);
-	left: -.98em;
-	width: calc(var(--nav-sz) - 0.5em);
+	left: -.96em;
+	width: calc(.3em + var(--nav-sz) - var(--sbw));
 	border-bottom: 1px solid var(--bg-u5);
 	overflow: hidden;
 }
@@ -1695,15 +1717,24 @@ html.y #tree.nowrap .ntree a+a:hover {
 	line-height: 0;
 }
 .dumb_loader_thing {
-	display: inline-block;
+	display: block;
 	margin: 1em .3em 1em 1em;
 	padding: 0 1.2em 0 0;
 	font-size: 4em;
+	min-width: 1em;
+	min-height: 1em;
 	opacity: 0;
 	animation: 1s linear .15s infinite forwards spin, .2s ease .15s 1 forwards fadein;
-	position: absolute;
+	position: fixed;
+	top: .3em;
 	z-index: 9;
 }
+#dlt_t {
+	left: 0;
+}
+#dlt_f {
+	right: .5em;
+}
 #files .cfg {
 	display: none;
 	font-size: 2em;
@@ -1796,10 +1827,11 @@ html.y #tree.nowrap .ntree a+a:hover {
 	line-height: 2.3em;
 	margin-bottom: 1.5em;
 }
+#hdoc,
 #ghead {
 	position: sticky;
 	top: -.3em;
-	z-index: 1;
+	z-index: 2;
 }
 .ghead .btn {
 	position: relative;
@@ -1809,6 +1841,13 @@ html.y #tree.nowrap .ntree a+a:hover {
 	white-space: pre;
 	padding-left: .3em;
 }
+#tailbtns {
+	display: none;
+}
+#taildoc.on+#tailbtns {
+	display: inherit;
+	display: unset;
+}
 #op_unpost {
 	padding: 1em;
 }
@@ -1905,6 +1944,9 @@ html.y #tree.nowrap .ntree a+a:hover {
 	padding: 1em 0 1em 0;
 	border-radius: .3em;
 }
+#doc.wrap {
+	white-space: pre-wrap;
+}
 html.y #doc {
 	box-shadow: 0 0 .3em var(--bg-u5);
 	background: #f7f7f7;
@@ -1993,6 +2035,9 @@ a.btn,
 	font-family: 'scp', monospace, monospace;
 	font-family: var(--font-mono), 'scp', monospace, monospace;
 }
+#hkhelp b {
+	text-shadow: 1px 0 0 var(--fg), -1px 0 0 var(--fg), 0 -1px 0 var(--fg);
+}
 html.noscroll,
 html.noscroll .sbar {
 	scrollbar-width: none;
@@ -2154,18 +2199,25 @@ html.y #bbox-overlay figcaption a {
 	top: calc(50% - 30px);
 	width: 44px;
 	height: 60px;
+	transition: background-color .3s ease, color .3s ease, left .3s ease, right .3s ease;
+}
+#bbox-btns button {
+	transition: background-color .3s ease, color .3s ease;
+}
+#bbox-btns {
+	transition: top .3s ease;
 }
 .bbox-btn {
 	position: fixed;
 }
-.bbox-btn,
-#bbox-btns {
-	opacity: 1;
-	animation: opacity .2s infinite ease-in-out;
+#bbox-next.off {
+	right: -2.6em;
+}
+#bbox-prev.off {
+	left: -2.6em;
 }
-.bbox-btn.off,
 #bbox-btns.off {
-	opacity: 0;
+	top: -2.2em;
 }
 #bbox-overlay button {
 	cursor: pointer;
@@ -2176,8 +2228,6 @@ html.y #bbox-overlay figcaption a {
 	border-radius: 15%;
 	background: rgba(50, 50, 50, 0.5);
 	color: rgba(255,255,255,0.7);
-	transition: background-color .3s ease;
-	transition: color .3s ease;
 	font-size: 1.4em;
 	line-height: 1.4em;
 	vertical-align: top;
@@ -3026,7 +3076,8 @@ html.b .ntree a {
 	padding: .6em .2em;
 }
 html.b #treepar {
-	margin-left: .62em;
+	margin-left: .63em;
+	width: calc(.1em + var(--nav-sz) - var(--sbw));
 	border-bottom: .2em solid var(--f-h-b1);
 }
 html.b #wrap {
@@ -3198,7 +3249,7 @@ html.d #treepar {
 
   	#ggrid>a>span {
 		text-align: center;
-		padding: 0.2em;
+		padding: .2em .2em .15em .2em;
 	}
 }
 
@@ -3221,4 +3272,9 @@ html.d #treepar {
 	.dropdesc>div>div {
 		transition: none;
 	}
+	#bbox-next,
+	#bbox-prev,
+	#bbox-btns {
+		transition: background-color .3s ease, color .3s ease;
+	}
 }

copyparty/web/browser.html

@@ -109,8 +109,8 @@
 {%- for f in files %}
 <tr><td>{{ f.lead }}</td><td><a href="{{ f.href }}">{{ f.name|e }}</a></td><td>{{ f.sz }}</td>
 {%- if f.tags is defined %}
-	{%- for k in taglist %}<td>{{ f.tags[k] }}</td>{%- endfor %}
-{%- endif %}<td>{{ f.ext }}</td><td>{{ f.dt }}</td></tr>
+	{%- for k in taglist %}<td>{{ f.tags[k]|e }}</td>{%- endfor %}
+{%- endif %}<td>{{ f.ext|e }}</td><td>{{ f.dt }}</td></tr>
 {%- endfor %}
 
 		</tbody>
@@ -124,14 +124,12 @@
 
 </div>
 
-	{%- if srv_info %}
 	<div id="srv_info"><span>{{ srv_info }}</span></div>
-	{%- endif %}
 
 	<div id="widget"></div>
 
 	<script>
-		var SR = {{ r|tojson }},
+		var SR = "{{ r }}",
 			CGV1 = {{ cgv1 }},
 			CGV = {{ cgv|tojson }},
 			TS = "{{ ts }}",

copyparty/web/idp.html

@@ -0,0 +1,55 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+	<meta charset="utf-8">
+	<title>{{ s_doctitle }}</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
+	<meta name="theme-color" content="#{{ tcolor }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/shares.css?_={{ ts }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
+{{ html_head }}
+</head>
+
+<body>
+	<div id="wrap">
+		<a href="{{ r }}/?idp">refresh</a>
+		<a href="{{ r }}/?h">control-panel</a>
+
+        <table id="tab"><thead><tr>
+            <th>forget</th>
+            <th>user</th>
+            <th>groups</th>
+        </tr></thead><tbody>
+        {% for un, gn in rows %}
+        <tr>
+            <td><a href="{{ r }}/?idp=rm={{ un|e }}">forget</a></td>
+            <td>{{ un|e }}</td>
+            <td>{{ gn|e }}</td>
+        </tr>
+        {% endfor %}
+        </tbody></table>
+        {% if not rows %}
+        (there are no IdP users in the cache)
+        {% endif %}
+    </div>
+	<a href="#" id="repl">π</a>
+	<script>
+
+var SR="{{ r }}",
+	lang="{{ lang }}",
+	dfavico="{{ favico }}";
+
+var STG = window.localStorage;
+document.documentElement.className = (STG && STG.cpp_thm) || "{{ this.args.theme }}";
+
+</script>
+<script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>
+{%- if js %}
+<script src="{{ js }}_={{ ts }}"></script>
+{%- endif %}
+</body>
+</html>
+

copyparty/web/md.html

@@ -128,9 +128,9 @@ write markdown (most html is 🙆 too)
 
 	<script>
 
-var SR = {{ r|tojson }},
+var SR = "{{ r }}",
 	last_modified = {{ lastmod }},
-	have_emp = {{ have_emp|tojson }},
+	have_emp = {{ "true" if have_emp else "false" }},
 	dfavico = "{{ favico }}";
 
 var md_opt = {

copyparty/web/md.js

@@ -23,8 +23,7 @@ var dbg = function () { };
 
 // dodge browser issues
 (function () {
-    var ua = navigator.userAgent;
-    if (ua.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(ua)) {
+    if (UA.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(UA)) {
         // necessary on ff-68.7 at least
         var s = mknod('style');
         s.innerHTML = '@page { margin: .5in .6in .8in .6in; }';

copyparty/web/md2.js

@@ -450,7 +450,7 @@ function savechk_cb() {
 
 // firefox bug: initial selection offset isn't cleared properly through js
 var ff_clearsel = (function () {
-    if (navigator.userAgent.indexOf(') Gecko/') === -1)
+    if (UA.indexOf(') Gecko/') === -1)
         return function () { }
 
     return function () {
@@ -1078,26 +1078,28 @@ action_stack = (function () {
         var p1 = from.length,
             p2 = to.length;
 
-        while (p1-- > 0 && p2-- > 0)
+        while (p1 --> 0 && p2 --> 0)
             if (from[p1] != to[p2])
                 break;
 
-        if (car > ++p1) {
+        if (car > ++p1)
             car = p1;
-        }
 
         var txt = from.substring(car, p1)
         return {
             car: car,
-            cdr: ++p2,
+            cdr: p2 + (car && 1),
             txt: txt,
             cpos: cpos
         };
     }
 
     var undiff = function (from, change) {
+        var t1 = from.substring(0, change.car),
+            t2 = from.substring(change.cdr);
+
         return {
-            txt: from.substring(0, change.car) + change.txt + from.substring(change.cdr),
+            txt: t1 + change.txt + t2,
             cpos: change.cpos
         };
     }

copyparty/web/mde.html

@@ -26,9 +26,9 @@
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR = "{{ r }}",
 	last_modified = {{ lastmod }},
-	have_emp = {{ have_emp|tojson }},
+	have_emp = {{ "true" if have_emp else "false" }},
 	dfavico = "{{ favico }}";
 
 var md_opt = {

copyparty/web/rups.css

@@ -0,0 +1,107 @@
+html {
+	color: #333;
+	background: #f7f7f7;
+	font-family: sans-serif;
+	font-family: var(--font-main), sans-serif;
+	touch-action: manipulation;
+}
+#wrap {
+	margin: 2em auto;
+	padding: 0 1em 3em 1em;
+	line-height: 2.3em;
+}
+a {
+	color: #047;
+	background: #fff;
+	text-decoration: none;
+	border-bottom: 1px solid #8ab;
+	border-radius: .2em;
+	padding: .2em .6em;
+	margin: 0 .3em;
+}
+#wrap td a {
+	margin: 0;
+	line-height: 1em;
+	display: inline-block;
+	white-space: initial;
+	font-family: var(--font-main), sans-serif;
+}
+#repl {
+	border: none;
+	background: none;
+	color: inherit;
+	padding: 0;
+	position: fixed;
+	bottom: .25em;
+	left: .2em;
+}
+#wrap table {
+	border-collapse: collapse;
+	position: relative;
+	margin-top: 2em;
+}
+#wrap th {
+	top: -1px;
+	position: sticky;
+	background: #f7f7f7;
+}
+#wrap td {
+	font-family: var(--font-mono), monospace, monospace;
+	white-space: pre; /*date*/
+	overflow: hidden; /*ipv6*/
+}
+#wrap th:first-child,
+#wrap td:first-child {
+	text-align: right;
+}
+#wrap td,
+#wrap th {
+	text-align: left;
+	padding: .3em .6em;
+	max-width: 30vw;
+}
+#wrap tr:hover td {
+	background: #ddd;
+	box-shadow: 0 -1px 0 rgba(128, 128, 128, 0.5) inset;
+}
+#wrap th:first-child,
+#wrap td:first-child {
+	border-radius: .5em 0 0 .5em;
+}
+#wrap th:last-child,
+#wrap td:last-child {
+	border-radius: 0 .5em .5em 0;
+}
+
+
+
+html.z {
+	background: #222;
+	color: #ccc;
+}
+html.bz {
+	background: #11121d;
+	color: #bbd;
+}
+html.z a {
+	color: #fff;
+	background: #057;
+	border-color: #37a;
+}
+html.z input[type=text] {
+	color: #ddd;
+	background: #223;
+	border: none;
+	border-bottom: 1px solid #fc5;
+	border-radius: .2em;
+	padding: .2em .3em;
+}
+html.z #wrap th {
+	background: #222;
+}
+html.bz #wrap th {
+	background: #223;
+}
+html.z #wrap tr:hover td {
+	background: #000;
+}

copyparty/web/rups.html

@@ -0,0 +1,50 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+	<meta charset="utf-8">
+	<title>{{ s_doctitle }}</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
+	<meta name="theme-color" content="#{{ tcolor }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/rups.css?_={{ ts }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
+{{ html_head }}
+</head>
+
+<body>
+	<div id="wrap">
+        <a href="#" id="re">refresh</a>
+        <a href="{{ r }}/?h">control-panel</a>
+        &nbsp; Filter: <input type="text" id="filter" size="20" placeholder="documents/passwords" />
+        &nbsp; <span id="hits"></span>
+        <table id="tab"><thead><tr>
+            <th>size</th>
+            <th>who</th>
+            <th>when</th>
+            <th>age</th>
+            <th>dir</th>
+            <th>file</th>
+        </tr></thead><tbody id="tb"></tbody></table>
+    </div>
+	<a href="#" id="repl">π</a>
+	<script>
+
+var SR="{{ r }}",
+	lang="{{ lang }}",
+	dfavico="{{ favico }}";
+
+var STG = window.localStorage;
+document.documentElement.className = (STG && STG.cpp_thm) || "{{ this.args.theme }}";
+
+</script>
+<script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>
+<script>var V={{ v }};</script>
+<script src="{{ r }}/.cpr/rups.js?_={{ ts }}"></script>
+{%- if js %}
+<script src="{{ js }}_={{ ts }}"></script>
+{%- endif %}
+</body>
+</html>
+

copyparty/web/rups.js

@@ -0,0 +1,66 @@
+function render() {
+    var ups = V.ups, now = V.now, html = [];
+    ebi('filter').value = V.filter;
+    ebi('hits').innerHTML = 'showing ' + ups.length + ' files';
+
+    for (var a = 0; a < ups.length; a++) {
+        var f = ups[a],
+            vsp = vsplit(f.vp.split('?')[0]),
+            dn = esc(uricom_dec(vsp[0])),
+            fn = esc(uricom_dec(vsp[1])),
+            at = f.at,
+            td = now - f.at,
+            ts = !at ? '(?)' : unix2iso(at),
+            sa = !at ? '(?)' : td > 60 ? shumantime(td) : (td + 's'),
+            sz = ('' + f.sz).replace(/\B(?=(\d{3})+(?!\d))/g, " ");
+
+        html.push('<tr><td>' + sz +
+            '</td><td>' + f.ip +
+            '</td><td>' + ts +
+            '</td><td>' + sa +
+            '</td><td><a href="' + vsp[0] + '">' + dn +
+            '</a></td><td><a href="' + f.vp + '">' + fn +
+            '</a></td></tr>');
+    }
+    if (!ups.length) {
+        var t = V.filter ? ' matching the filter' : '';
+        html = ['<tr><td colspan="6">there are no uploads' + t + '</td></tr>'];
+    }
+    ebi('tb').innerHTML = html.join('');
+}
+render();
+
+var ti;
+function ask(e) {
+    ev(e);
+    clearTimeout(ti);
+    ebi('hits').innerHTML = 'Loading...';
+
+    var xhr = new XHR(),
+        filter = unsmart(ebi('filter').value);
+
+    hist_replace(get_evpath().split('?')[0] + '?ru&filter=' + uricom_enc(filter));
+
+    xhr.onload = xhr.onerror = function () {
+        try {
+            V = JSON.parse(this.responseText)
+        }
+        catch (ex) {
+            ebi('tb').innerHTML = '<tr><td colspan="6">failed to decode server response as json: <pre>' + esc(this.responseText) + '</pre></td></tr>';
+            return;
+        }
+        render();
+    };
+    xhr.open('GET', SR + '/?ru&j&filter=' + uricom_enc(filter), true);
+    xhr.send();
+}
+ebi('re').onclick = ask;
+ebi('filter').oninput = function () {
+    clearTimeout(ti);
+    ti = setTimeout(ask, 500);
+    ebi('hits').innerHTML = '...';
+};
+ebi('filter').onkeydown = function (e) {
+    if (('' + e.key).endsWith('Enter'))
+        ask();
+};

copyparty/web/shares.css

@@ -44,9 +44,10 @@ a {
 	bottom: .25em;
 	left: .2em;
 }
-table {
+#wrap table {
 	border-collapse: collapse;
-    position: relative;
+	position: relative;
+	margin-top: 2em;
 }
 th {
 	top: -1px;
@@ -62,6 +63,14 @@ th {
 #wrap td+td+td+td+td+td+td+td {
 	font-family: var(--font-mono), monospace, monospace;
 }
+#wrap th:first-child,
+#wrap td:first-child {
+	border-radius: .5em 0 0 .5em;
+}
+#wrap th:last-child,
+#wrap td:last-child {
+	border-radius: 0 .5em .5em 0;
+}
 
 
 
@@ -81,3 +90,6 @@ html.bz {
 	color: #bbd;
 	background: #11121d;
 }
+html.bz th {
+	background: #223;
+}

copyparty/web/shares.html

@@ -6,6 +6,7 @@
 	<title>{{ s_doctitle }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/shares.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
@@ -14,8 +15,8 @@
 
 <body>
 	<div id="wrap">
-		<a id="a" href="{{ r }}/?shares" class="af">refresh</a>
-		<a id="a" href="{{ r }}/?h" class="af">control-panel</a>
+		<a href="{{ r }}/?shares">refresh</a>
+		<a href="{{ r }}/?h">control-panel</a>
 
         <span>axs = perms (read,write,move,delet)</span>
         <span>nf = numFiles (0=dir)</span>
@@ -58,9 +59,11 @@
         {% if not rows %}
         (you don't have any active shares btw)
         {% endif %}
+    </div>
+	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
     shr="{{ shr }}",
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";

copyparty/web/shares.js

@@ -1,9 +1,11 @@
+var SRS = SR.trimEnd('/') + '/';
+
 var t = QSA('a[k]');
 for (var a = 0; a < t.length; a++)
     t[a].onclick = rm;
 
 function rm() {
-    var u = SR + shr + uricom_enc(this.getAttribute('k')) + '?eshare=rm',
+    var u = SRS + '?eshare=rm&skey=' + uricom_enc(this.getAttribute('k')),
         xhr = new XHR();
 
     xhr.open('POST', u, true);
@@ -13,7 +15,7 @@ function rm() {
 
 function bump() {
     var k = this.closest('tr').getElementsByTagName('a')[2].getAttribute('k'),
-        u = SR + shr + uricom_enc(k) + '?eshare=' + this.value,
+        u = SRS + '?skey=' + uricom_enc(k) + '&eshare=' + this.value,
         xhr = new XHR();
 
     xhr.open('POST', u, true);

copyparty/web/splash.html

@@ -122,7 +122,7 @@
 				<input type="hidden" id="la" name="act" value="login" />
 				<input type="password" id="lp" name="cppwd" placeholder=" password" />
 				<input type="hidden" name="uhash" id="uhash" value="x" />
-				<input type="submit" id="ls" value="Login" />
+				<input type="submit" id="ls" value="login" />
 				{% if chpw %}
 				<a id="x" href="#">change password</a>
 				{% endif %}
@@ -135,6 +135,10 @@
 
 		<h1 id="cc">other stuff:</h1>
 		<ul>
+			{%- if this.uname in this.args.idp_adm_set %}
+			<li><a id="ag" href="{{ r }}/?idp">view idp cache</a></li>
+			{% endif %}
+
 			{%- if this.uname != '*' and this.args.shr %}
 			<li><a id="y" href="{{ r }}/?shares">edit shares</a></li>
 			{% endif %}
@@ -157,6 +161,7 @@
 			<blockquote id="ad">enabling no304 will disable all caching; try this if k304 wasn't enough. This will waste a huge amount of network traffic!</blockquote></li>
 			{% endif %}
 
+			<li><a id="af" href="{{ r }}/?ru">show recent uploads</a></li>
 			<li><a id="k" href="{{ r }}/?reset" class="r" onclick="localStorage.clear();return true">reset client settings</a></li>
 		</ul>
 
@@ -167,7 +172,7 @@
 	{%- endif %}
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";
 

copyparty/web/splash.js

@@ -38,6 +38,8 @@ var Ls = {
 		"ac1": "skru på no304",
 		"ad1": "no304 stopper all bruk av cache. Hvis ikke k304 var nok, prøv denne. Vil mangedoble dataforbruk!",
 		"ae1": "utgående:",
+		"af1": "vis nylig opplastede filer",
+		"ag1": "vis kjente IdP-brukere",
 	},
 	"eng": {
 		"d2": "shows the state of all active threads",
@@ -88,7 +90,51 @@ var Ls = {
 		"ac1": "开启 k304",
 		"ad1": "启用 no304 将禁用所有缓存；如果 k304 不够，可以尝试此选项。这将消耗大量的网络流量！", //m
 		"ae1": "正在下载：", //m
-	}
+		"af1": "显示最近上传的文件", //m
+		"ag1": "查看已知 IdP 用户", //m
+	},
+
+	"deu": {
+		"a1": "Neu laden",
+		"b1": "Tach, wie geht's? &nbsp; <small>(Du bist nicht angemeldet)</small>",
+		"c1": "Abmelden",
+		"d1": "Zustand",  // TLNote: "d2" is the tooltip for this button
+		"d2": "Zeigt den Zustand aller aktiven Threads",
+		"e1": "Config neu laden",
+		"e2": "Konfigurationsdatei neu laden (Accounts/Volumes/VolFlags)$Nund scannt alle e2ds-Volumes$N$NBeachte: Jegliche Änderung an globalen Einstellungen$Nbenötigt einen Neustart zum Anwenden",
+		"f1": "Du kannst lesen:",
+		"g1": "Du kannst hochladen nach:",
+		"cc1": "Andere Dinge:",
+		"h1": "k304 deaktivieren",  // TLNote: "j1" explains what k304 is
+		"i1": "k304 aktivieren",
+		"j1": "k304 trennt die Clientverbindung bei jedem HTTP 304, was Bugs mit problematischen Proxies vorbeugen kann (z.B. nicht ladenden Seiten), macht Dinge aber generell langsamer",
+		"k1": "Client-Einstellungen zurücksetzen",
+		"l1": "Melde dich an für mehr:",
+		"m1": "Willkommen zurück,",  // TLNote: "welcome back, USERNAME"
+		"n1": "404 Nicht gefunden &nbsp;┐( ´ -`)┌",
+		"o1": 'or maybe you don\'t have access -- try a password or <a href="' + SR + '/?h">go home</a>',
+		"p1": "403 Verboten &nbsp;~┻━┻",
+		"q1": 'Benutze ein Passwort oder <a href="' + SR + '/?h">gehe zur Homepage</a>',
+		"r1": "Gehe zur Homepage",
+		".s1": "Neu scannen",
+		"t1": "Aktion",  // TLNote: this is the header above the "rescan" buttons
+		"u2": "time since the last server write$N( upload / rename / ... )$N$N17d = 17 days$N1h23 = 1 hour 23 minutes$N4m56 = 4 minutes 56 seconds",
+		"v1": "Verbinden",
+		"v2": "Benutze diesen Server als lokale Festplatte",
+		"w1": "Zu HTTPS wechseln",
+		"x1": "Passwort ändern",
+		"y1": "Shares bearbeiten",  // TLNote: shows the list of folders that the user has decided to share
+		"z1": "Share entsperren:",  // TLNote: the password prompt to see a hidden share
+		"ta1": "Trage zuerst dein Passwort ein",
+		"ta2": "Wiederhole dein Passwort zur Bestätigung:",
+		"ta3": "Da stimmt etwas nicht; probier's nochmal",
+		"aa1": "Eingehende Dateien:",
+		"ab1": "no304 deaktivieren",
+		"ac1": "no304 aktivieren",
+		"ad1": "Das Aktivieren von no304 deaktiviert jegliche Form von Caching; probier dies, wenn k304 nicht genug war. Dies verschwendet eine grosse Menge Netzwerk-Traffic!",
+		"ae1": "Aktive Downloads:",
+        "af1": "Zeige neue Uploads",
+	},
 };
 
 if (window.langmod)

copyparty/web/svcs.html

@@ -9,7 +9,7 @@
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/splash.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
-	<style>ul{padding-left:1.3em}li{margin:.4em 0}</style>
+	<style>ul{padding-left:1.3em}li{margin:.4em 0}.txa{float:right;margin:0 0 0 1em}</style>
 {{ html_head }}
 </head>
 
@@ -31,15 +31,22 @@
             <br />
             <span class="os win lin mac">placeholders:</span>
             <span class="os win">
-                {% if accs %}<code><b>{{ pw }}</b></code>=password, {% endif %}<code><b>W:</b></code>=mountpoint
+                {% if accs %}<code><b id="pw0">{{ pw }}</b></code>=password, {% endif %}<code><b>W:</b></code>=mountpoint
             </span>
             <span class="os lin mac">
-                {% if accs %}<code><b>{{ pw }}</b></code>=password, {% endif %}<code><b>mp</b></code>=mountpoint
+                {% if accs %}<code><b id="pw0">{{ pw }}</b></code>=password, {% endif %}<code><b>mp</b></code>=mountpoint
             </span>
+            {% if accs %}<a href="#" id="setpw">use real password</a>{% endif %}
         </p>
 
 
 
+        {% if args.idp_h_usr %}
+        <p style="line-height:2em"><b>WARNING:</b> this server is using IdP-based authentication, so this stuff may not work as advertised. Depending on server config, these commands can probably only be used to access areas which don't require authentication, unless you auth using any non-IdP accounts defined in the copyparty config. Please see <a href="https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients">the IdP docs</a></p>
+        {% endif %}
+
+
+
         {% if not args.no_dav %}
         <h1>WebDAV</h1>
 
@@ -53,7 +60,6 @@
                 {% if s %}
                 <li>running <code>rclone mount</code> on LAN (or just dont have valid certificates)? add <code>--no-check-certificate</code></li>
                 {% endif %}
-                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
                 <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
             </ul>
 
@@ -95,6 +101,7 @@
                 gio mount -a dav{{ s }}://{{ ep }}/{{ rvp }}
                 {%- endif %}
             </pre>
+            <p>on KDE Dolphin, use <code>webdav{{ s }}://{{ ep }}/{{ rvp }}</code></p>
         </div>
 
         <div class="os mac">
@@ -137,7 +144,6 @@
                 {% if args.ftps %}
                 <li>running on LAN (or just dont have valid certificates)? add <code>no_check_certificate=true</code> to the config command</li>
                 {% endif %}
-                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
                 <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
             </ul>
             <p>if you want to use the native FTP client in windows instead (please dont), press <code>win+R</code> and run this command:</p>
@@ -231,11 +237,77 @@
 
 
 
+        <div class="os win">
+            <h1>ShareX</h1>
+
+            <p>to upload screenshots using ShareX <a href="https://getsharex.com/">v15+</a>, save this as <code>copyparty.sxcu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.sxcu">
+                { "Version": "15.0.0", "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>", {% endif %}"accept": "url"
+                },
+                "DestinationType": "ImageUploader, TextUploader, FileUploader",
+                "Body": "MultipartFormData", "URL": "{response}",
+                "RequestMethod": "POST", "FileFormName": "f" }
+            </pre>
+
+            <p>for ShareX <a href="https://github.com/ShareX/ShareX/releases/tag/v12.1.1">v12</a> specifically, save this as <code>copyparty.sxcu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.sxcu">
+                { "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>", {% endif %}"accept": "url"
+                },
+                "DestinationType": "ImageUploader, TextUploader, FileUploader",
+                "FileFormName": "f" }
+            </pre>
+        </div>
+
+
+
+        <div class="os mac">
+            <h1>ishare</h1>
+
+            <p>to upload screenshots using <a href="https://isharemac.app/">ishare</a>, save this as <code>copyparty.iscu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.iscu">
+                { "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>",{% endif %}
+                    "accept": "json"
+                },
+                "ResponseURL": "{{ '{{fileurl}}' }}",
+                "FileFormName": "f" }
+            </pre>
+        </div>
+
+
+
+        <div class="os lin">
+            <h1>flameshot</h1>
+
+            <p>to upload screenshots using <a href="https://flameshot.org/">flameshot</a>, save this as <code>flameshot.sh</code> and run it:</p>
+
+            <pre class="dl" name="flameshot.sh">
+                #!/bin/bash
+                pw="<b>{{ pw }}</b>"
+                url="http{{ s }}://{{ ep }}/{{ rvp }}"
+                filename="$(date +%Y-%m%d-%H%M%S).png"
+                flameshot gui -s -r | curl -sT- "$url$filename?want=url&pw=$pw" | xsel -ib
+            </pre>
+        </div>
+
+
+
     </div>
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
     lang="{{ lang }}",
 	dfavico="{{ favico }}";
 

copyparty/web/svcs.js

@@ -1,11 +1,3 @@
-function QSA(x) {
-    return document.querySelectorAll(x);
-}
-var LINUX = /Linux/.test(navigator.userAgent),
-    MACOS = /[^a-z]mac ?os/i.test(navigator.userAgent),
-    WINDOWS = /Windows/.test(navigator.userAgent);
-
-
 var oa = QSA('pre');
 for (var a = 0; a < oa.length; a++) {
     var html = oa[a].innerHTML,
@@ -15,6 +7,21 @@ for (var a = 0; a < oa.length; a++) {
     oa[a].innerHTML = html.replace(rd, '$1').replace(/[ \r\n]+$/, '').replace(/\r?\n/g, '<br />');
 }
 
+function add_dls() {
+    oa = QSA('pre.dl');
+    for (var a = 0; a < oa.length; a++) {
+        var an = 'ta' + a,
+            o = ebi(an) || mknod('a', an, 'download');
+
+        oa[a].setAttribute('id', 'tx' + a);
+        oa[a].parentNode.insertBefore(o, oa[a]);
+        o.setAttribute('download', oa[a].getAttribute('name'));
+        o.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(oa[a].innerText));
+        clmod(o, 'txa', 1);
+    }
+}
+add_dls();
+
 
 oa = QSA('.ossel a');
 for (var a = 0; a < oa.length; a++)
@@ -40,3 +47,23 @@ function setos(os) {
 }
 
 setos(WINDOWS ? 'win' : LINUX ? 'lin' : MACOS ? 'mac' : 'idk');
+
+
+function setpw(e) {
+    ev(e);
+    modal.prompt('password:', '', function (v) {
+        if (!v)
+            return;
+
+        var pw0 = ebi('pw0').innerHTML,
+            oa = QSA('b');
+    
+        for (var a = 0; a < oa.length; a++)
+            if (oa[a].innerHTML == pw0)
+                oa[a].textContent = v;
+
+        add_dls();
+    });
+}
+if (ebi('setpw'))
+    ebi('setpw').onclick = setpw;

copyparty/web/ui.css

@@ -381,6 +381,9 @@ html.y .btn:focus {
 	box-shadow: 0 .1em .2em #037 inset;
 	outline: #037 solid .1em;
 }
+input, button {
+	font-family: var(--font-main), sans-serif;
+}
 input[type="submit"] {
 	cursor: pointer;
 }

copyparty/web/up2k.js

@@ -1,6 +1,18 @@
 "use strict";
 
 
+(function () {
+    var x = sread('nosubtle');
+    if (x === '0' || x === '1')
+        nosubtle = parseInt(x);
+    if ((nosubtle > 1 && !CHROME && !FIREFOX) ||
+        (nosubtle > 2 && !CHROME) ||
+        (CHROME && nosubtle > VCHROME) ||
+        !WebAssembly)
+        nosubtle = 0;
+})();
+
+
 function goto_up2k() {
     if (up2k === false)
         return goto('bup');
@@ -23,7 +35,7 @@ var up2k = null,
     m = 'will use ' + sha_js + ' instead of native sha512 due to';
 
 try {
-    if (sread('nosubtle') || window.nosubtle)
+    if (nosubtle)
         throw 'chickenbit';
     var cf = crypto.subtle || crypto.webkitSubtle;
     cf.digest('SHA-512', new Uint8Array(1)).then(
@@ -695,8 +707,9 @@ function Donut(uc, st) {
         }
 
         if (++r.tc >= 10) {
+            var s = r.eta === null ? 'paused' : r.eta > 60 ? shumantime(r.eta) : (r.eta + 's');
             wintitle("{0}%, {1}, #{2}, ".format(
-                f2f(v * 100 / t, 1), shumantime(r.eta), st.files.length - st.nfile.upload), true);
+                f2f(v * 100 / t, 1), s, st.files.length - st.nfile.upload), true);
             r.tc = 0;
         }
 
@@ -824,7 +837,7 @@ function up2k_init(subtle) {
             }
             qsr('#u2depmsg');
             var o = mknod('div', 'u2depmsg');
-            o.innerHTML = m;
+            o.innerHTML = nosubtle ? '' : m;
             ebi('u2foot').appendChild(o);
         }
         loading_deps = true;
@@ -880,10 +893,30 @@ function up2k_init(subtle) {
     bcfg_bind(uc, 'turbo', 'u2turbo', turbolvl > 1, draw_turbo);
     bcfg_bind(uc, 'datechk', 'u2tdate', turbolvl < 3, null);
     bcfg_bind(uc, 'az', 'u2sort', u2sort.indexOf('n') + 1, set_u2sort);
-    bcfg_bind(uc, 'hashw', 'hashw', !!WebAssembly && (!subtle || !CHROME || MOBILE || VCHROME >= 107), set_hashw);
+    bcfg_bind(uc, 'hashw', 'hashw', !!WebAssembly && !(CHROME && MOBILE) && (!subtle || !CHROME || VCHROME > 136), set_hashw);
+    bcfg_bind(uc, 'hwasm', 'nosubtle', nosubtle, set_nosubtle);
     bcfg_bind(uc, 'upnag', 'upnag', false, set_upnag);
     bcfg_bind(uc, 'upsfx', 'upsfx', false, set_upsfx);
 
+    uc.ow = parseInt(sread('u2ow', ['0', '1', '2']) || u2ow);
+    uc.owt = ['🛡️', '🕒', '♻️'];
+    function set_ow() {
+        QS('label[for="u2ow"]').innerHTML = uc.owt[uc.ow];
+        ebi('u2ow').checked  = true; //cosmetic
+    }
+    ebi('u2ow').onclick = function (e) {
+        ev(e);
+        if (++uc.ow > 2)
+            uc.ow = 0;
+        swrite('u2ow', uc.ow);
+        set_ow();
+        if (uc.ow && !has(perms, 'delete'))
+            toast.warn(10, L.u_enoow, 'noow');
+        else if (toast.tag == 'noow')
+            toast.hide();
+    };
+    set_ow();
+
     var st = {
         "files": [],
         "nfile": {
@@ -968,7 +1001,7 @@ function up2k_init(subtle) {
             ud = function () { ebi('dir' + fdom_ctr).click(); };
 
         // too buggy on chrome <= 72
-        var m = / Chrome\/([0-9]+)\./.exec(navigator.userAgent);
+        var m = / Chrome\/([0-9]+)\./.exec(UA);
         if (m && parseInt(m[1]) < 73)
             return uf();
 
@@ -1299,7 +1332,7 @@ function up2k_init(subtle) {
         if (bad_files.length) {
             var msg = L.u_badf.format(bad_files.length, ntot);
             for (var a = 0, aa = Math.min(20, bad_files.length); a < aa; a++)
-                msg += '-- ' + bad_files[a][1] + '\n';
+                msg += '-- ' + esc(bad_files[a][1]) + '\n';
 
             msg += L.u_just1;
             return modal.alert(msg, function () {
@@ -1311,7 +1344,7 @@ function up2k_init(subtle) {
         if (nil_files.length) {
             var msg = L.u_blankf.format(nil_files.length, ntot);
             for (var a = 0, aa = Math.min(20, nil_files.length); a < aa; a++)
-                msg += '-- ' + nil_files[a][1] + '\n';
+                msg += '-- ' + esc(nil_files[a][1]) + '\n';
 
             msg += L.u_just1;
             return modal.confirm(msg, function () {
@@ -1323,10 +1356,68 @@ function up2k_init(subtle) {
             });
         }
 
+        var fps = new Set(), pdp = '';
+        for (var a = 0; a < good_files.length; a++) {
+            var fp = good_files[a][1],
+                dp = vsplit(fp)[0];
+            fps.add(fp);
+            if (pdp != dp) {
+                pdp = dp;
+                dp = dp.slice(0, -1);
+                while (dp) {
+                    fps.add(dp);
+                    dp = vsplit(dp)[0].slice(0, -1);
+                }
+            }
+        }
+
+        var junk = [], rmi = [];
+        for (var a = 0; a < good_files.length; a++) {
+            var fn = good_files[a][1];
+            if (fn.indexOf("/.") < 0 && fn.indexOf("/__MACOS") < 0)
+                continue;
+
+            if (/\/__MACOS|\/\.(DS_Store|AppleDouble|LSOverride|DocumentRevisions-|fseventsd|Spotlight-V[0-9]|TemporaryItems|Trashes|VolumeIcon\.icns|com\.apple\.timemachine\.donotpresent|AppleDB|AppleDesktop|apdisk)/.exec(fn)) {
+                junk.push(good_files[a]);
+                rmi.push(a);
+                continue;
+            }
+
+            if (fn.indexOf("/._") + 1 &&
+                fps.has(fn.replace("/._", "/")) &&
+                fn.split("/").pop().startsWith("._") &&
+                !has(rmi, a)
+            ) {
+                junk.push(good_files[a]);
+                rmi.push(a);
+            }
+        }
+
+        if (!junk.length)
+            return gotallfiles2(good_files);
+
+        junk.sort();
+        rmi.sort(function (a, b) { return a - b; });
+
+        var msg = L.u_applef.format(junk.length, good_files.length);
+        for (var a = 0, aa = Math.min(1000, junk.length); a < aa; a++)
+            msg += '-- ' + esc(junk[a][1]) + '\n';
+
+        return modal.confirm(msg, function () {
+            for (var a = rmi.length - 1; a >= 0; a--)
+                good_files.splice(rmi[a], 1);
+
+            start_actx();
+            gotallfiles2(good_files);
+        }, function () {
+            start_actx();
+            gotallfiles2(good_files);
+        });
+    }
+
+    function gotallfiles2(good_files) {
         good_files.sort(function (a, b) {
-            a = a[1];
-            b = b[1];
-            return a < b ? -1 : a > b ? 1 : 0;
+            return a[1] < b[1] ? -1 : 1;
         });
 
         var msg = [];
@@ -1337,7 +1428,7 @@ function up2k_init(subtle) {
         if (FIREFOX && good_files.length > 3000)
             msg.push(L.u_ff_many + "\n\n");
 
-        msg.push(L.u_asku.format(good_files.length, esc(get_vpath())) + '<ul>');
+        msg.push(L.u_asku.format(good_files.length, esc(uricom_dec(get_evpath()))) + '<ul>');
         for (var a = 0, aa = Math.min(20, good_files.length); a < aa; a++)
             msg.push('<li>' + esc(good_files[a][1]) + '</li>');
 
@@ -1359,7 +1450,22 @@ function up2k_init(subtle) {
             draw_each = good_files.length < 50;
 
         if (WebAssembly && !hws.length) {
-            for (var a = 0; a < Math.min(navigator.hardwareConcurrency || 4, 16); a++)
+            var nw = Math.min(navigator.hardwareConcurrency || 4, 16);
+
+            if (CHROME) {
+                // chrome-bug 383568268 // #124
+                nw = Math.max(1, (nw > 4 ? 4 : (nw - 1)));
+                if (VCHROME < 137)
+                nw = (subtle && !MOBILE && nw > 2) ? 2 : nw;
+            }
+
+            var x = sread('u2hashers') || window.u2hashers;
+            if (x) {
+                console.log('u2hashers is overriding default-value ' + nw);
+                nw = parseInt(x);
+            }
+
+            for (var a = 0; a < nw; a++)
                 hws.push(new Worker(SR + '/.cpr/w.hash.js?_=' + TS));
 
             if (!subtle)
@@ -1371,9 +1477,7 @@ function up2k_init(subtle) {
 
         if (!uc.az)
             good_files.sort(function (a, b) {
-                a = a[0].size;
-                b = b[0].size;
-                return a < b ? -1 : a > b ? 1 : 0;
+                return a[0].size - b[0].size;
             });
 
         for (var a = 0; a < good_files.length; a++) {
@@ -1381,7 +1485,7 @@ function up2k_init(subtle) {
                 name = good_files[a][1],
                 fdir = evpath,
                 now = Date.now(),
-                lmod = uc.u2ts ? (fobj.lastModified || now) : 0,
+                lmod = (uc.u2ts && fobj.lastModified) || 0,
                 ofs = name.lastIndexOf('/') + 1;
 
             if (ofs) {
@@ -1963,38 +2067,90 @@ function up2k_init(subtle) {
             nchunk = 0,
             chunksize = get_chunksize(t.size),
             nchunks = Math.ceil(t.size / chunksize),
+            csz_mib = chunksize / 1048576,
+            tread = t.t_hashing,
+            cache_buf = null,
+            cache_car = 0,
+            cache_cdr = 0,
+            hashers = 0,
             hashtab = {};
 
+        // resolving subtle.digest w/o worker takes 1sec on blur if the actx hack breaks
+        var use_workers = hws.length && !hws_ng && uc.hashw && (nchunks > 1 || document.visibilityState == 'hidden'),
+            hash_par = (!subtle && !use_workers) ? 0 : csz_mib < 48 ? 2 : csz_mib < 96 ? 1 : 0;
+
         pvis.setab(t.n, nchunks);
         pvis.move(t.n, 'bz');
 
-        if (hws.length && !hws_ng && uc.hashw && (nchunks > 1 || document.visibilityState == 'hidden'))
-            // resolving subtle.digest w/o worker takes 1sec on blur if the actx hack breaks
+        if (use_workers)
             return wexec_hash(t, chunksize, nchunks);
 
         var segm_next = function () {
             if (nchunk >= nchunks || bpend)
                 return false;
 
-            var reader = new FileReader(),
-                nch = nchunk++,
+            var nch = nchunk++,
                 car = nch * chunksize,
                 cdr = Math.min(chunksize + car, t.size);
 
             st.bytes.hashed += cdr - car;
             st.etac.h++;
 
-            var orz = function (e) {
-                bpend--;
-                segm_next();
-                hash_calc(nch, e.target.result);
+            if (MOBILE && CHROME && st.slow_io === null && nch == 1 && cdr - car >= 1024 * 512) {
+                var spd = Math.floor((cdr - car) / (Date.now() + 1 - tread));
+                st.slow_io = spd < 40 * 1024;
+                console.log('spd {0}, slow: {1}'.format(spd, st.slow_io));
             }
+
+            if (cdr <= cache_cdr && car >= cache_car) {
+                try {
+                    var ofs = car - cache_car,
+                        ofs2 = ofs + (cdr - car),
+                        buf = cache_buf.subarray(ofs, ofs2);
+
+                    hash_calc(nch, buf);
+                }
+                catch (ex) {
+                    vis_exh(ex + '', 'up2k.js', '', '', ex);
+                }
+                return;
+            }
+
+            var reader = new FileReader(),
+                fr_cdr = cdr;
+
+            if (st.slow_io) {
+                var step = cdr - car,
+                    tgt = 48 * 1048576;
+
+                while (step && fr_cdr - car < tgt)
+                    fr_cdr += step;
+                if (fr_cdr - car > tgt && fr_cdr > cdr)
+                    fr_cdr -= step;
+                if (fr_cdr > t.size)
+                    fr_cdr = t.size;
+            }
+
+            var orz = function (e) {
+                bpend = 0;
+                var buf = e.target.result;
+                if (fr_cdr > cdr) {
+                    cache_buf = new Uint8Array(buf);
+                    cache_car = car;
+                    cache_cdr = fr_cdr;
+                    buf = cache_buf.subarray(0, cdr - car);
+                }
+                if (hashers < hash_par)
+                    segm_next();
+
+                hash_calc(nch, buf);
+            };
             reader.onload = function (e) {
                 try { orz(e); } catch (ex) { vis_exh(ex + '', 'up2k.js', '', '', ex); }
             };
             reader.onerror = function () {
-                var err = reader.error + '';
-                var handled = false;
+                var err = esc('' + reader.error),
+                    handled = false;
 
                 if (err.indexOf('NotReadableError') !== -1 || // win10-chrome defender
                     err.indexOf('NotFoundError') !== -1  // macos-firefox permissions
@@ -2015,17 +2171,20 @@ function up2k_init(subtle) {
 
                 toast.err(0, 'y o u   b r o k e    i t\nfile: ' + esc(t.name + '') + '\nerror: ' + err);
             };
-            bpend++;
-            reader.readAsArrayBuffer(t.fobj.slice(car, cdr));
+            bpend = 1;
+            tread = Date.now();
+            reader.readAsArrayBuffer(t.fobj.slice(car, fr_cdr));
 
             return true;
         };
 
         var hash_calc = function (nch, buf) {
+            hashers++;
             var orz = function (hashbuf) {
                 var hslice = new Uint8Array(hashbuf).subarray(0, 33),
                     b64str = buf2b64(hslice);
 
+                hashers--;
                 hashtab[nch] = b64str;
                 t.hash.push(nch);
                 pvis.hashed(t);
@@ -2074,6 +2233,7 @@ function up2k_init(subtle) {
             reading = 0,
             max_readers = 1,
             opt_readers = 2,
+            failed = false,
             free = [],
             busy = {},
             nbusy = 0,
@@ -2123,6 +2283,14 @@ function up2k_init(subtle) {
                 tasker();
         }
 
+        function go_fail() {
+            failed = true;
+            if (nbusy)
+                return;
+            apop(st.busy.hash, t);
+            st.bytes.finished += t.size;
+        }
+
         function onmsg(d) {
             d = d.data;
             var k = d[0];
@@ -2137,6 +2305,12 @@ function up2k_init(subtle) {
                 return vis_exh(d[1], 'up2k.js', '', '', d[1]);
 
             if (k == "fail") {
+                var nchunk = d[1];
+                free.push(busy[nchunk]);
+                delete busy[nchunk];
+                nbusy--;
+                reading--;
+
                 pvis.seth(t.n, 1, d[1]);
                 pvis.seth(t.n, 2, d[2]);
                 console.log(d[1], d[2]);
@@ -2144,9 +2318,7 @@ function up2k_init(subtle) {
                     got_oserr();
 
                 pvis.move(t.n, 'ng');
-                apop(st.busy.hash, t);
-                st.bytes.finished += t.size;
-                return;
+                return go_fail();
             }
 
             if (k == "ferr")
@@ -2179,6 +2351,9 @@ function up2k_init(subtle) {
                 t.hash.push(nchunk);
                 pvis.hashed(t);
 
+                if (failed)
+                    return go_fail();
+
                 if (t.hash.length < nchunks)
                     return nbusy < opt_readers && go_next();
 
@@ -2215,7 +2390,7 @@ function up2k_init(subtle) {
         xhr.onerror = xhr.ontimeout = function () {
             console.log('head onerror, retrying', t.name, t);
             if (!toast.visible)
-                toast.warn(9.98, L.u_enethd + "\n\nfile: " + t.name, t);
+                toast.warn(9.98, L.u_enethd + "\n\nfile: " + esc(t.name), t);
 
             apop(st.busy.head, t);
             st.todo.head.unshift(t);
@@ -2256,8 +2431,8 @@ function up2k_init(subtle) {
             try { orz(e); } catch (ex) { vis_exh(ex + '', 'up2k.js', '', '', ex); }
         };
 
-        xhr.timeout = 34000;
         xhr.open('HEAD', t.purl + uricom_enc(t.name), true);
+        xhr.timeout = 34000;
         xhr.send();
     }
 
@@ -2290,7 +2465,7 @@ function up2k_init(subtle) {
                 return console.log('zombie handshake onerror', t.name, t);
 
             if (!toast.visible)
-                toast.warn(9.98, L.u_eneths + "\n\nfile: " + t.name, t);
+                toast.warn(9.98, L.u_eneths + "\n\nfile: " + esc(t.name), t);
 
             console.log('handshake onerror, retrying', t.name, t);
             apop(st.busy.handshake, t);
@@ -2395,7 +2570,7 @@ function up2k_init(subtle) {
                     var idx = t.hash.indexOf(missing[a]);
                     if (idx < 0)
                         return modal.alert('wtf negative index for hash "{0}" in task:\n{1}'.format(
-                            missing[a], JSON.stringify(t)));
+                            missing[a], esc(JSON.stringify(t))));
 
                     t.postlist.push(idx);
                     cbd[idx] = 0;
@@ -2549,7 +2724,7 @@ function up2k_init(subtle) {
                     return toast.err(0, L.u_ehsdf + "\n\n" + rsp.replace(/.*; /, ''));
 
                 err = t.t_uploading ? L.u_ehsfin : t.srch ? L.u_ehssrch : L.u_ehsinit;
-                xhrchk(xhr, err + "\n\nfile: " + t.name + "\n\nerror ", "404, target folder not found", "warn", t);
+                xhrchk(xhr, err + "\n\nfile: " + esc(t.name) + "\n\nerror ", "404, target folder not found", "warn", t);
             }
         }
         xhr.onload = function (e) {
@@ -2570,6 +2745,13 @@ function up2k_init(subtle) {
         else if (t.umod)
             req.umod = true;
 
+        if (!t.srch) {
+            if (uc.ow == 1)
+                req.replace = 'mt';
+            if (uc.ow == 2)
+                req.replace = true;
+        }
+
         xhr.open('POST', t.purl, true);
         xhr.responseType = 'text';
         xhr.timeout = 42000 + (t.srch || t.t_uploaded ? 0 :
@@ -2699,7 +2881,7 @@ function up2k_init(subtle) {
                     toast.inf(10, L.u_cbusy);
             }
             else {
-                xhrchk(xhr, L.u_cuerr2.format(snpart, Math.ceil(t.size / chunksize), t.name), "404, target folder not found (???)", "warn", t);
+                xhrchk(xhr, L.u_cuerr2.format(snpart, Math.ceil(t.size / chunksize), esc(t.name)), "404, target folder not found (???)", "warn", t);
                 chill(t);
             }
             orz2(xhr);
@@ -2729,7 +2911,8 @@ function up2k_init(subtle) {
 
                 st.bytes.inflight += db;
                 xhr.bsent = nb;
-                xhr.timeout = 64000 + Date.now() - xhr.t0;
+                if (!IE)
+                    xhr.timeout = 64000 + Date.now() - xhr.t0;
                 pvis.prog(t, pcar, nb);
             };
             xhr.onload = function (xev) {
@@ -2743,7 +2926,7 @@ function up2k_init(subtle) {
                 xhr.bsent = 0;
 
                 if (!toast.visible)
-                    toast.warn(9.98, L.u_cuerr.format(snpart, Math.ceil(t.size / chunksize), t.name), t);
+                    toast.warn(9.98, L.u_cuerr.format(snpart, Math.ceil(t.size / chunksize), esc(t.name)), t);
 
                 t.nojoin = t.nojoin || t.postlist.length;  // maybe rproxy postsize limit
                 console.log('chunkpit onerror,', t.name, t);
@@ -2777,7 +2960,7 @@ function up2k_init(subtle) {
 
             xhr.bsent = 0;
             xhr.t0 = Date.now();
-            xhr.timeout = 42000;
+            xhr.timeout = 1000 * (IE ? 1234 : 42);
             xhr.responseType = 'text';
             xhr.send(t.fobj.slice(car, cdr));
         }
@@ -3035,7 +3218,7 @@ function up2k_init(subtle) {
                 new_state = false;
                 fixed = true;
             }
-            if (new_state === undefined)
+            if (new_state === undefined && preferred === undefined)
                 new_state = can_write ? false : have_up2k_idx ? true : undefined;
         }
 
@@ -3123,6 +3306,12 @@ function up2k_init(subtle) {
         }
     }
 
+    function set_nosubtle(v) {
+        if (!WebAssembly)
+            return toast.err(10, L.u_nowork);
+        modal.confirm(L.lang_set, location.reload.bind(location), null);
+    }
+
     function set_upnag(en) {
         function nopenag() {
             bcfg_set('upnag', uc.upnag = false);

copyparty/web/util.js

@@ -29,14 +29,17 @@ var wah = '',
     HTTPS = ('' + location).indexOf('https:') === 0,
     TOUCH = 'ontouchstart' in window,
     MOBILE = TOUCH,
-    CHROME = !!window.chrome,
+    CHROME = !!window.chrome,  // safari=false
     VCHROME = CHROME ? 1 : 0,
-    IE = /Trident\//.test(navigator.userAgent),
-    FIREFOX = ('netscape' in window) && / rv:/.test(navigator.userAgent),
-    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(navigator.userAgent),
-    LINUX = /Linux/.test(navigator.userAgent),
-    MACOS = /[^a-z]mac ?os/i.test(navigator.userAgent),
-    WINDOWS = /Windows/.test(navigator.userAgent);
+    UA = '' + navigator.userAgent,
+    IE = !!document.documentMode,
+    FIREFOX = ('netscape' in window) && / rv:/.test(UA),
+    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(UA),
+    LINUX = /Linux/.test(UA),
+    MACOS = /Macintosh/.test(UA),
+    WINDOWS = /Windows/.test(UA),
+    APPLE = IPHONE || MACOS,
+    APPLEM = TOUCH && APPLE;
 
 if (!window.WebAssembly || !WebAssembly.Memory)
     window.WebAssembly = false;
@@ -66,7 +69,7 @@ try {
 
     CHROME = navigator.userAgentData.brands.find(function (d) { return d.brand == 'Chromium' });
     if (CHROME)
-        VCHROME = CHROME.version;
+        VCHROME = parseInt(CHROME.version);
     else
         VCHROME = 0;
 
@@ -180,7 +183,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
     if (url.indexOf(' > eval') + 1 && !evalex_fatal)
         return;  // md timer
 
-    if (IE && url.indexOf('prism.js') + 1)
+    if (url.indexOf('prism.js') + 1)
         return;
 
     if (url.indexOf('easymde.js') + 1)
@@ -196,7 +199,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
         '<p style="font-size:1.3em;margin:0;line-height:2em">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a> / <a href="?b=u">basic</a></p>',
         '<p style="color:#fff">please send me a screenshot arigathanks gozaimuch: <a href="<ghi>" target="_blank">new github issue</a></p>',
         '<p class="b">' + esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(msg).replace(/\n/g, '<br />') + '</p>',
-        '<p><b>UA:</b> ' + esc(navigator.userAgent + '')
+        '<p><b>UA:</b> ' + esc(UA)
     ];
 
     try {
@@ -361,7 +364,8 @@ if (!Element.prototype.matches)
         Element.prototype.mozMatchesSelector ||
         Element.prototype.webkitMatchesSelector;
 
-if (!Element.prototype.closest)
+var CLOSEST = !!Element.prototype.closest;
+if (!CLOSEST)
     Element.prototype.closest = function (s) {
         var el = this;
         do {
@@ -379,8 +383,10 @@ if (!String.prototype.format)
         });
     };
 
+var have_URL = false;
 try {
     new URL('/a/', 'https://a.com/');
+    have_URL = true;
 }
 catch (ex) {
     console.log('ie11 shim URL()');
@@ -431,7 +437,7 @@ function import_js(url, cb, ecb) {
 
 
 function unsmart(txt) {
-    return !IPHONE ? txt : (txt.
+    return !APPLEM ? txt : (txt.
         replace(/[\u2014]/g, "--").
         replace(/[\u2022]/g, "*").
         replace(/[\u2018\u2019]/g, "'").
@@ -458,6 +464,13 @@ function namesan(txt, win, fslash) {
 }
 
 
+var NATSORT, ENATSORT;
+try {
+    NATSORT = new Intl.Collator([], {numeric: true});
+}
+catch (ex) { }
+
+
 var crctab = (function () {
     var c, tab = [];
     for (var n = 0; n < 256; n++) {
@@ -611,6 +624,33 @@ function showsort(tab) {
         }
     }
 }
+function st_cmp_num(a, b) {
+    a = a[0];
+    b = b[0];
+    return (
+        a === null ? -1 :
+        b === null ? 1 :
+        (a - b)
+    );
+}
+function st_cmp_nat(a, b) {
+    a = a[0];
+    b = b[0];
+    return (
+        a === null ? -1 :
+        b === null ? 1 :
+        NATSORT.compare(a, b)
+    );
+}
+function st_cmp_gen(a, b) {
+    a = a[0];
+    b = b[0];
+    return (
+        a === null ? -1 :
+        b === null ? 1 :
+        a.localeCompare(b)
+    );
+}
 function sortTable(table, col, cb) {
     var tb = table.tBodies[0],
         th = table.tHead.rows[0].cells,
@@ -656,19 +696,17 @@ function sortTable(table, col, cb) {
         }
         vl.push([v, a]);
     }
-    vl.sort(function (a, b) {
-        a = a[0];
-        b = b[0];
-        if (a === null)
-            return -1;
-        if (b === null)
-            return 1;
 
-        if (stype == 'int') {
-            return reverse * (a - b);
-        }
-        return reverse * (a.localeCompare(b));
-    });
+    if (stype == 'int')
+        vl.sort(st_cmp_num);
+    else if (ENATSORT)
+        vl.sort(st_cmp_nat);
+    else
+        vl.sort(st_cmp_gen);
+
+    if (reverse < 0)
+        vl.reverse();
+
     if (sread('dir1st') !== '0') {
         var r1 = [], r2 = [];
         for (var i = 0; i < tr.length; i++) {
@@ -696,6 +734,16 @@ function makeSortable(table, cb) {
 }
 
 
+function assert_vp(path) {
+    if (path.indexOf('//') + 1)
+        throw 'nonlocal1: ' + path;
+
+    var o = window.location.origin;
+    if (have_URL && (new URL(path, o)).origin != o)
+        throw 'nonlocal2: ' + path;
+}
+
+
 function linksplit(rp, id) {
     var ret = [],
         apath = '/',
@@ -854,11 +902,6 @@ function get_evpath() {
 }
 
 
-function get_vpath() {
-    return uricom_dec(get_evpath());
-}
-
-
 function noq_href(el) {
     return el.getAttribute('href').split('?')[0];
 }
@@ -886,8 +929,11 @@ if (window.Number && Number.isFinite)
 
 function f2f(val, nd) {
     // 10.toFixed(1) returns 10.00 for certain values of 10
-    if (!isNum(val))
-        val = 999;
+    if (!isNum(val)) {
+        val = parseFloat(val);
+        if (!isNum(val))
+            val = 999;
+    }
     val = (val * Math.pow(10, nd)).toFixed(0).split('.')[0];
     return nd ? (val.slice(0, -nd) || '0') + '.' + val.slice(-nd) : val;
 }
@@ -1195,7 +1241,7 @@ function dl_file(url) {
 function cliptxt(txt, ok) {
     var fb = function () {
         console.log('clip-fb');
-        var o = mknod('input');
+        var o = mknod('textarea');
         o.value = txt;
         document.body.appendChild(o);
         o.focus();
@@ -1205,6 +1251,8 @@ function cliptxt(txt, ok) {
         ok();
     };
     try {
+        if (!window.isSecureContext)
+            throw 1;
         navigator.clipboard.writeText(txt).then(ok, fb);
     }
     catch (ex) { fb(); }
@@ -1353,7 +1401,7 @@ var tt = (function () {
     };
 
     r.getmsg = function (el) {
-        if (IPHONE && QS('body.bbox-open'))
+        if (APPLEM && QS('body.bbox-open'))
             return;
 
         var cfg = sread('tooltips');

copyparty/web/w.hash.js

@@ -4,6 +4,16 @@
 function hex2u8(txt) {
     return new Uint8Array(txt.match(/.{2}/g).map(function (b) { return parseInt(b, 16); }));
 }
+function esc(txt) {
+    return txt.replace(/[&"<>]/g, function (c) {
+        return {
+            '&': '&amp;',
+            '"': '&quot;',
+            '<': '&lt;',
+            '>': '&gt;'
+        }[c];
+    });
+}
 
 
 var subtle = null;
@@ -19,6 +29,8 @@ catch (ex) {
 }
 function load_fb() {
     subtle = null;
+    if (self.hashwasm)
+        return;
     importScripts('deps/sha512.hw.js');
     console.log('using fallback hasher');
 }
@@ -64,7 +76,7 @@ onmessage = (d) => {
     };
     reader.onerror = function () {
         busy = false;
-        var err = reader.error + '';
+        var err = esc('' + reader.error);
 
         if (err.indexOf('NotReadableError') !== -1 || // win10-chrome defender
             err.indexOf('NotFoundError') !== -1  // macos-firefox permissions

docs/README.md

@@ -25,6 +25,9 @@
 ## [`changelog.md`](changelog.md)
 * occasionally grabbed from github release notes
 
+## [`synology-dsm.md`](synology-dsm.md)
+* running copyparty on a synology nas
+
 ## [`devnotes.md`](devnotes.md)
 * technical stuff
 

docs/changelog.md

@@ -1,3 +1,812 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0727-2305  `v1.18.5`  SECURITY: fix XSS in media tags
+
+## ⚠️ ATTN: this release fixes an XSS vulnerability
+
+[GHSA-9q4r-x2hj-jmvr](https://github.com/9001/copyparty/security/advisories/GHSA-9q4r-x2hj-jmvr), exploitable in two different ways, could let an attacker execute arbitrary javascript on other users:
+* either: tricking someone into clicking a malicious URL to load and execute javascript
+* or: uploading a malicious audio file to the server, affecting any successive visitors
+
+so, with new and curious eyes on the project, we are starting off with a bang. Huge thanks to @altperfect for finding and reporting this earlier today.
+
+## recent important news
+
+* [v1.18.5 (2025-07-28)](https://github.com/9001/copyparty/releases/tag/v1.18.5) fixed XSS in display of media tags
+* [v1.15.0 (2024-09-08)](https://github.com/9001/copyparty/releases/tag/v1.15.0) changed upload deduplication to be default-disabled
+* [v1.14.3 (2024-08-30)](https://github.com/9001/copyparty/releases/tag/v1.14.3) fixed a bug that was introduced in v1.13.8 (2024-08-13); this bug could lead to **data loss** -- see the v1.14.3 release-notes for details
+
+## 🧪 new features
+
+* #214 option to stop playback after one song, and/or at end of folder 6bb27e60
+
+## 🩹 bugfixes
+
+* GHSA-9q4r-x2hj-jmvr 895880ae 
+* block external m3u files 2228f81f
+* #202 the connect-page could show IP-address when it should have used hostnames/domains b0dec83a
+* scrolling locked after tailing a file and closing it creatively d197e754
+
+## 🔧 other changes
+
+* #189 the `SameSite` cookie parameter now defaults to `Strict`, increasing CSRF protection ca6d0b8d
+  * new option `--cookie-lax` reverts to previous value `Lax`
+* docker: add FTPS support b4199847
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0725-1841  `v1.18.4`  Landmarks
+
+## 🧪 new features
+
+* #182 [Landmarks](https://github.com/9001/copyparty#database-location) edba7fff
+  * detects that a storage backend is glitching out and disengage the up2k-database as a precaution
+* #183 quickdelete 21a96bcf 
+  * new togglebutton `qdel` in the UI which reduces the number of deletion confirmations by one
+  * global-option `--qdel=0` which can bring it all the way to zero (good luck)
+
+## 🩹 bugfixes
+
+* fix unpost in recently created shares 2d322dd4
+* fix filekeys on windows df6d4df4
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0721-2307  `v1.18.3`  drop the umask
+
+## 🧪 new features
+
+* #181 the default chmod (unix-permissions) of new files and folders can now be changed 9921c43e
+  * `--chmod-d` or volflag `chmod_d` sets directory permissions; default is 755
+  * `--chmod-f` or volflag `chmod_f` sets file permissions; default is usually 644 (OS-defined)
+  * see `--help-chmod` which explains the numbers
+
+## 🩹 bugfixes
+
+* #179 couldn't combine `--shr` (shares) and `--xvol` (symlink-guard) 0f0f8d90
+* #180 gallery buttons could still be clicked when faded-out 8c32b0e7
+* rss-feeds were slightly busted when combined with rp-loc (location-based proxying) 56d3bcf5
+* music-playback within search-results no longer jumps into the next folder at end-of-list 9bc4c5d2
+* video-playback on iOS now behaves like on all other platforms 78605d9a
+  * (it would force-switch into fullscreen because that's their default)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0707-1419  `v1.18.2`  idp-vol persistence
+
+## 🧪 new features
+
+* IdP-volumes can optionally be persisted across restarts d162502c
+  * there is a UI to manage the cached users/groups 4f264a0a
+    * only available to users listed in the new option `--idp-adm`
+* api for manually rescanning several volumes at once 42c199e7
+  * `/some/path/?scan` does that one volume like before
+  * `/any/path/?scan=/vol1,/another/vol2` rescans `/vol1` and `/another/vol2`
+* volflag to hide volume from listing in controlpanel fd7c71d6
+
+## 🩹 bugfixes
+
+* macos: fix confusing crash when blocked by [Little Snitch](https://www.obdev.at/products/littlesnitch/) bf11b2a4
+* unpost could break in some hairy reverseproxy setups 1b2d3985
+* copyparty32.exe: fix segfault on win7 c9fafb20
+* ui: fix navpane overlapping the scrollbar (still a bit jank but eh) 7ef6fd13
+* usb-eject: support all volume names ed908b98
+* docker: ensure clean slate deb6711b
+* fix up2k on ie11 d2714434
+
+## 🔧 other changes
+
+* update buildscript for keyfinder to support llvm 65c4e035
+* #175 add `python-magic` into the `iv` and `dj` docker flavors (thx @Morganamilo) 77274e9d
+* properly killed the experimental docker flavors to avoid confusion 8306e3d9
+* copyparty.exe: updated pillow 299cff3f f6be3905
+  * avif support was removed to save 2 MiB
+
+## 🌠 fun facts
+
+* this release was slightly delayed due to a [norwegian traffic jam](https://a.ocv.me/pub/g/2025/07/PXL_20250706_143558381.jpg)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0622-0020  `v1.18.0`  Logtail
+
+## 🧪 new features
+
+* textfile-viewer can now livestream logfiles (and other growing files) 17fa4906 77df17d1 a1c7a095 6ecf4fdc
+  * see [readme](https://github.com/9001/copyparty/#textfile-viewer) and the [live demo](https://a.ocv.me/pub/demo/logtail/)
+* IdP-volumes: extend syntax for excluding certain users/groups 2e53f797
+  * the commit-message explains it well enough
+* new option `--see-dots` to show dotfiles in the web-ui by default c599e2aa
+* #171 automatic mimetype detection for files without extensions (thx @Morganamilo!) ec05f8cc 9dd5dec0
+  * default-disabled since it has a performance impact on webdav
+    * there are plans to fix this by using the db instead
+* #170 improve custom filetype icons
+  * be less strict; if a thumbnail is set for `.gz` files, use it for `.tar.gz` too c75b0c25
+  * improve config docs fa5845ff
+
+## 🩹 bugfixes
+
+* cosmetic: get rid of some noise along the bottom of some cards in the gridview 8cae7a71
+* cosmetic: satisfy a new syntax warning in cpython-3.14 5ac38648
+
+## 🔧 other changes
+
+* properly document how to [build from source](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#build-from-scratch) / build from scratch f61511d8
+* update deps
+  * copyparty.exe: python 3.13 1eff87c3
+  * webdeps: dompurify 7eca90cc
+
+## 🌠 fun facts
+
+* this release was cooked up in a [swedish forest cabin](https://a.ocv.me/pub/g/nerd-stuff/forestparty.jpg)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0527-1939  `v1.17.2`  pushing chrome to the limits (and then some)
+
+## 🧪 new features
+
+* not this time
+
+## 🩹 bugfixes
+
+* up2k: improve file-hashing speed on recent versions of google chrome e3e51fb8
+  * speed increased from 319 to 513 MiB/s by default (but older chrome versions did 748...)
+  * read the commit message for the full story, but basically chrome has gotten gradually slower over the past couple versions (starting from v133) and this makes it slightly less bad again
+  * hashing speed can be further improved from `0.5` to `1.1` GiB/s by enabling the `[wasm]` option in the `[⚙️] settings` tab
+    * this option can be made default-enabled with `--nosubtle 137` but beware that this increases the chances of running into browser-bugs (foreshadowing...)
+* up2k: fix errorhandler for browser-bugs (oom and such) 49c71247
+  * because [chrome-bug 383568268](https://issues.chromium.org/issues/383568268) is about to make a [surprise return?!](https://issues.chromium.org/issues/383568268#comment14)
+* #168 fix uploading into shares if path-based proxying is used 9cb93ae1
+* #165 unconditionally heed `--rp-loc` 84f5f417
+  * the config-option for [path-based proxying](https://github.com/9001/copyparty/#reverse-proxy) was ignored if the reverse-proxy was untrusted; this was confusing and not strictly necessary
+
+## 🔧 other changes
+
+* #166 the nixos module was improved once more (thx @msfjarvis!) 48470f6b 60fb1207
+* added usage instructions to [minimal-up2k.js](https://github.com/9001/copyparty/tree/hovudstraum/contrib/plugins#example-browser-js), the up2k-ui [simplifier](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png) 1d308eeb
+* docker: improve feedback if config is bad or missing 28b63e58
+
+## 🌠 fun facts
+
+* this release was tested using an [unreliable rdp connection](https://a.ocv.me/pub/g/nerd-stuff/PXL_20250526_021207825.jpg) through two ssh-jumphosts to a qemu win10 vm back home from the bergen-oslo night train wifi
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0518-2234  `v1.17.1`  as seen on archlinux
+
+## 🧪 new features
+
+* new toolbar button to zip/tar the currently open folder 256dad8c
+* new options to specify the default checksum algorithm for PUT/bup/WebDAV uploads 0de09860
+* #164 new option `--put-name` to specify the filename of nameless uploads 5dcd88a6
+  * the default is still `put-TIMESTAMP-IPADDRESS.bin`
+
+## 🩹 bugfixes
+
+* #162 password-protected shares was incompatible with password-hashing c3ef3fdc
+* #161 m3u playlist creation was only possible over https 94352f27
+* when relocating/redirecting an upload from an xbu hook (execute-before-upload), could miss an already existing file at the destination and create another copy 0a9a8077
+* some edgecases when moving files between filesystems f425ff51
+* improve tagscan-resume after a server restart (primarily for dupes) 41fa6b25
+* support prehistoric timestamps in fat16 vhd-drives on windows 261236e3
+
+## 🔧 other changes
+
+* #159 the nixos module was improved (thx @gabevenberg and @chinponya!) d1bca1f5
+* an archlinux maintainer adopted the aur package; copyparty is now [officially in arch](https://archlinux.org/packages/extra/any/copyparty/) b9ba783c
+* #162 add KDE Dolphin instructions to the conect-page d4a8071d
+* audioplayer now knows that `.oga` means `.ogg`
+
+## 🌠 fun facts
+
+* this release contains code [pair-programmed during an anime rave](https://a.ocv.me/pub/g/nerd-stuff/PXL_20250503_222654610.jpg)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0426-2149  `v1.17.0`  mixtape.m3u
+
+## 🧪 new features
+
+* [m3u playlists](https://github.com/9001/copyparty/#playlists) 897f9d32 ad200f2b 4195762d fff45552
+  * create and play m3u / m3u8 files
+
+## 🩹 bugfixes
+
+* improve support for ie11 (yes, internet explorer 11) 3090c748 95157d02
+* now possible to launch the password-hasher cli while another instance is running dbfc899d
+  * in preparation of #157 / #159
+
+## 🔧 other changes
+
+* make better decisions when running in a VM with less than 1 GiB RAM dc3b7a27
+
+## 🌠 fun facts
+
+* this release contains code written [less than 1masl](https://a.ocv.me/pub/g/nerd-stuff/PXL_20250425_170037812.jpg) and was gonna be named [hash again](https://www.youtube.com/watch?v=twUFbqyul_M) since it was originally just the password-hasher fix, but then kipun suggested adding playlist support (thx kipun)
+* [donations](https://github.com/9001/) are now also possible through github -- good alternative to paypal (y)
+  * and thanks a lot for the support (and kind words therein) so far, appreciate it :>
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0420-1836  `v1.16.21`  unzip-compat
+
+a couple guys have been asking if I accept donations -- thanks a lot!! added a few options on [my github page](https://github.com/9001/) :>
+
+## 🧪 new features
+
+* #156 add button to loop/repeat music 71c55659
+
+## 🩹 bugfixes
+
+* #155 download-as-zip: increase compatibility with the unix `unzip` command db33d68d
+  * this unfortunately reduces support for huge zipfiles on old software (WinXP and such)
+  * and makes it less safe to stream zips into unzippers, so use tar.gz instead
+  * and is perhaps not even a copyparty bug; see commit-message for the full story
+
+## 🔧 other changes
+
+* show warning on Ctrl-A in lazy-loaded folders 5b3a5fe7
+* docker: hide keepalive pings from logs d5a9bd80
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0413-2151  `v1.16.20`  all sorted
+
+## 🧪 new features
+
+* when enabled, natural-sort will now also apply to tags, not just filenames 7b2bd6da
+
+## 🩹 bugfixes
+
+* some sorting-related stuff 7b2bd6da
+  * folders with non-ascii names would sort incorrectly in the navpane/sidebar
+  * natural-sort didn't apply correctly after changing the sort order
+* workaround [ffmpeg-bug 10797](https://trac.ffmpeg.org/ticket/10797) 98dcaee2
+  * reduces ram usage from 1534 to 230 MiB when generating spectrograms of s3xmodit songs (amiga chiptunes)
+* disable mdns if only listening on uds (unix-sockets) ffc16109 361aebf8
+
+## 🔧 other changes
+
+* hotkey CTRL-A will now select all files in gridview 233075ae
+  * and it toggles (just like in list-view) so try pressing it again
+* copyparty.exe: upgrade to pillow v11.2.1 c7aa1a35
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0408-2132  `v1.16.19`  GHOST
+
+did you know that every song named `GHOST` is a banger? it's true! [ghost](https://www.youtube.com/watch?v=NoUAwC4yiAw) // [ghost](https://www.youtube.com/watch?v=IKKar5SS29E) // [ghost](https://www.youtube.com/watch?v=tFSFlgm_tsw)
+
+## 🧪 new features
+
+* option to store markdown backups out-of-volume fc883418
+  * the default is still a subfolder named `.hist` next to the markdown file
+  * `--md-hist v` puts them in the volume's hist-folder instead
+  * `--md-hist n` disables markdown-backups entirely
+* #149 option to store the volume sqlite databases at a custom locations outside the hist-folder e1b9ac63
+  * new option `--dbpath` works like `--hist` but it only moves the database file, not the thumbnails
+  * they can be combined, in which case `--hist` is applied to thumbnails, `--dbpath` to the db
+  * useful when you're squeezing every last drop of performance out of your filesystem (see the issue)
+* actively prevent sharing certain databases (sessions/shares) between multiple copyparty instances acfaacbd
+  * an errormessage was added to explain some different alternatives for doing this safely
+    * for example by setting `XDG_CONFIG_HOME` which now works on all platforms b17ccc38
+
+## 🩹 bugfixes
+
+* #151 mkdir did not work in locations outside the volume root (via symlinks) 2b50fc20
+* improve the ui feedback when trying to play an audio file which failed to transcode f9954bc4
+  * also helps with server-filesystem issues, including image-thumbs
+
+## 🔧 other changes
+
+* #152 custom fonts are also applied to textboxes and buttons (thx @thaddeuskkr) d450f615
+* be more careful with the shares-db 8e0364ef
+* be less careful with the sessions-db 8e0364ef
+* update deps c0becc64
+  * web: dompurify
+  * copyparty.exe: python 3.12.10
+* rephrase `-j0` warning on windows to also mention that Microsoft Defender will freak out c0becc64
+* #149 add [a script](https://github.com/9001/copyparty/tree/hovudstraum/contrib#zfs-tunepy) to optimize the sqlite databases for storage on zfs 4f397b9b
+* block `GoogleOther` (another recalcitrant bot) from zip-downloads c2034f7b
+* rephrase `-j0` warning on windows to also mention that Microsoft Defender will freak out c0becc64
+* update [contributing.md](https://github.com/9001/copyparty/blob/hovudstraum/CONTRIBUTING.md) with a section regarding LLM/AI-written code cec3bee0
+* the [helptext](https://ocv.me/copyparty/helptext.html) will also be uploaded to each github release from now on, [permalink](https://github.com/9001/copyparty/releases/latest/download/helptext.html)
+* add review from ixbt forums b383c08c
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0323-2216  `v1.16.18`  zlib-ng
+
+## 🧪 new features
+
+* prefer zlib-ng when available 57a56073
+  * download-as-tar-gz becomes 2.5x faster
+  * default-enabled in docker-images
+  * not enabled in copyparty.exe yet; coming in a future python version
+* docker: add mimalloc (optional, default-disabled) de2c9788
+  * gives twice the speed, and twice the ram usage
+
+## 🩹 bugfixes
+
+* small up2k glitch 3c90cec0
+
+## 🔧 other changes
+
+* rename logues/readmes when uploaded with write-only access 2525d594
+  * since they are used as helptext when viewing the page
+* try to block google and other bad bots from `?doc` and `?zip` 99f63adf
+  * apparently `rel="nofollow"` means nothing these days
+
+### the docker images for this release were built from e1dea7ef
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0316-2002  `v1.16.17`  boot2party
+
+## NEW: make it a bootable usb flashdrive
+
+get the party going anywhere, anytime, no OS required! [download flashdrive image](https://a.ocv.me/pub/stuff/edcd001/enterprise-edition/) or watch the [low-effort demo video](https://a.ocv.me/pub/stuff/edcd001/enterprise-edition/hub-demo-hq.webm) which eventually gets to the copyparty part after showing off a bunch of other stuff on there
+
+* there is [source code](https://github.com/9001/asm/tree/hovudstraum/p/hub) and [build instructions](https://github.com/9001/asm/tree/hovudstraum/p/hub/sm/how2build) too
+* please don't take this too seriously
+
+## 🧪 new features
+
+* option to specify max-size for download-as-zip/tar 494179bd 0a33336d
+  * either the total download size (`--zipmaxs 500M`), and/or max number of files (`--zipmaxn 9k`)
+  * applies to all uesrs by default; can also ignore limits for authorized users (`--zipmaxu`)
+  * errormessage can be customized with `--zipmaxt "winter is coming... but this download isn't"`
+* [appledoubles](https://a.ocv.me/pub/stuff/?doc=appledoubles-and-friends.txt) are detected and skipped when uploading with the browser-UI 78208405
+* IdP-volumes can be filtered by group 9c2c4237
+  * `[/users/${u}]` in a config-file creates the volume for all users like before
+  * `[/users/${u%+canwrite}]` only if the user is in the `canwrite` group
+  * `[/users/${u%-admins}]` only if the user is NOT in the `admins` group
+
+## 🩹 bugfixes
+
+* when moving a folder with symlinks, don't expand them into full files 5ab09769
+  * absolute symlinks are moved as-is; relative symlinks are rewritten so they still point to the same file when possible (if both source and destination are indexed in the db)
+  * the previous behavior was good for un-deduplicating files after changing the server-settings, but was too inconvenient for all other usecases
+* #146 fix downloading from shares when `-j0` enabled 8417098c
+* only show the download-as-zip link when the user is actually allowed to 14bb2999
+* the suggestions in the serverlog regarding how to fix incorrect X-Forwarded-For settings would be incorrect if the reverse-proxy used IPv6 to communicate with copyparty 16462ee5
+* set nofollow on `?doc` links so crawlers don't download binary files as text 6a2644fe
+
+## 🔧 other changes
+
+* #147 IdP: fix the warning about dangerous misconfigurations to be more accurate 29a17ae2
+* #143 print a warning on incorrect character-encoding in textfiles (config-files, logues, readmes etc.) 25974d66
+* copyparty.exe: update to jinja 3.1.6 (copyparty was *not affected* by the jinja-3.1.5 vuln)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0228-1846  `v1.16.16`  lemon melon cookie
+
+<img src="https://github.com/9001/copyparty/raw/hovudstraum/docs/logo.svg" width="250" align="right"/>
+
+webdev is [like a lemon](https://youtu.be/HPURbfKb7to) sometimes
+
+* read-only demo server at https://a.ocv.me/pub/demo/
+* [docker image](https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker) ╱ [similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) ╱ [client testbed](https://cd.ocv.me/b/)
+
+there is a [discord server](https://discord.gg/25J8CdTT6G) with an `@everyone` in case of future important updates, such as [vulnerabilities](https://github.com/9001/copyparty/security) (most recently 2025-02-25)
+
+## recent important news
+
+* [v1.16.15 (2025-02-25)](https://github.com/9001/copyparty/releases/tag/v1.16.15) fixed low-severity xss when uploading maliciously-named files
+* [v1.15.0 (2024-09-08)](https://github.com/9001/copyparty/releases/tag/v1.15.0) changed upload deduplication to be default-disabled
+* [v1.14.3 (2024-08-30)](https://github.com/9001/copyparty/releases/tag/v1.14.3) fixed a bug that was introduced in v1.13.8 (2024-08-13); this bug could lead to **data loss** -- see the v1.14.3 release-notes for details
+
+## 🧪 new features
+
+* #142 workaround android-chrome timestamp bug 5e12abbb
+  * all files were uploaded with last-modified year 1601 in specific recent versions of chrome
+  * https://issues.chromium.org/issues/393149335 has the actual fix; will be out soon
+
+## 🩹 bugfixes
+
+* add helptext for volflags `dk`, `dks`, `dky` 65a7706f
+* fix false-positive warning when disabling a global option per-volume by unsetting the volflag
+
+## 🔧 other changes
+
+* #140 nixos: @daimond113 fixed a warning in the nixpkg (thx!) e0fe2b97
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0225-0017  `v1.16.15`  fix low-severity vuln
+
+<img src="https://github.com/9001/copyparty/raw/hovudstraum/docs/logo.svg" width="250" align="right"/>
+
+* read-only demo server at https://a.ocv.me/pub/demo/
+* [docker image](https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker) ╱ [similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) ╱ [client testbed](https://cd.ocv.me/b/)
+
+## ⚠️ this fixes a minor vulnerability; CVE-score `3.6`/`10`
+
+[GHSA-m2jw-cj8v-937r](https://github.com/9001/copyparty/security/advisories/GHSA-m2jw-cj8v-937r) aka [CVE-2025-27145](https://www.cve.org/CVERecord?id=CVE-2025-27145) could let an attacker run arbitrary javascript by tricking an authenticated user into uploading files with malicious filenames
+
+* ...but it required some clever social engineering, and is **not likely** to be a cause for concern... ah, better safe than sorry
+
+there is a [discord server](https://discord.gg/25J8CdTT6G) with an `@everyone` in case of future important updates, such as [vulnerabilities](https://github.com/9001/copyparty/security) (most recently 2025-02-25)
+
+## recent important news
+
+* [v1.15.0 (2024-09-08)](https://github.com/9001/copyparty/releases/tag/v1.15.0) changed upload deduplication to be default-disabled
+* [v1.14.3 (2024-08-30)](https://github.com/9001/copyparty/releases/tag/v1.14.3) fixed a bug that was introduced in v1.13.8 (2024-08-13); this bug could lead to **data loss** -- see the v1.14.3 release-notes for details
+
+## 🧪 new features
+
+* nothing this time
+
+## 🩹 bugfixes
+
+* fix [GHSA-m2jw-cj8v-937r](https://github.com/9001/copyparty/security/advisories/GHSA-m2jw-cj8v-937r) / [CVE-2025-27145](https://www.cve.org/CVERecord?id=CVE-2025-27145) in 438ea6cc
+  * when trying to upload an empty files by dragging it into the browser, the filename would be rendered as HTML, allowing javascript injection if the filename was malicious
+  * issue discovered and reported by @JayPatel48 (thx!)
+* related issues in errorhandling of uploads 499ae1c7 36866f1d
+  * these all had the same consequences as the GHSA above, but a network outage was necessary to trigger them
+    * which would probably have the lucky side-effect of blocking the javascript download, nice
+* paranoid fixing of probably-not-even-issues 3adbb2ff
+* fix some markdown / texteditor bugs 407531bc
+  * only indicate file-versions for markdown files in listings, since it's tricky to edit non-textfiles otherwise
+  * CTRL-C followed by CTRL-V and CTRL-Z in a single-line file would make a character fall off
+  * ensure safety of extensions
+
+## 🔧 other changes
+
+* readme:
+  * mention support for running the server on risc-v 6d102fc8
+  * mention that the [sony psp](https://github.com/user-attachments/assets/9d21f020-1110-4652-abeb-6fc09c533d4f) can browse and upload 598a29a7
+
+----
+
+# 💾 what to download?
+| download link | is it good? | description |
+| -- | -- | -- |
+| **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py)** | ✅ the best 👍 | runs anywhere! only needs python |
+| [a docker image](https://github.com/9001/copyparty/blob/hovudstraum/scripts/docker/README.md) | it's ok | good if you prefer docker 🐋 |
+| [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) |  ⚠️ [acceptable](https://github.com/9001/copyparty#copypartyexe) | for [win8](https://user-images.githubusercontent.com/241032/221445946-1e328e56-8c5b-44a9-8b9f-dee84d942535.png) or later; built-in thumbnailer |
+| [u2c.exe](https://github.com/9001/copyparty/releases/download/v1.16.14/u2c.exe) | ⚠️ acceptable | [CLI uploader](https://github.com/9001/copyparty/blob/hovudstraum/bin/u2c.py) as a win7+ exe ([video](https://a.ocv.me/pub/demo/pics-vids/u2cli.webm)) |
+| [copyparty.pyz](https://github.com/9001/copyparty/releases/latest/download/copyparty.pyz) | ⚠️ acceptable | similar to the regular sfx, [mostly worse](https://github.com/9001/copyparty#zipapp) |
+| [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) | ⛔️ [dangerous](https://github.com/9001/copyparty#copypartyexe) | for [win7](https://user-images.githubusercontent.com/241032/221445944-ae85d1f4-d351-4837-b130-82cab57d6cca.png) -- never expose to the internet! |
+| [cpp-winpe64.exe](https://github.com/9001/copyparty/releases/download/v1.16.5/copyparty-winpe64.exe) | ⛔️ dangerous | runs on [64bit WinPE](https://user-images.githubusercontent.com/241032/205454984-e6b550df-3c49-486d-9267-1614078dd0dd.png), otherwise useless |
+
+* except for [u2c.exe](https://github.com/9001/copyparty/releases/download/v1.16.14/u2c.exe), all of the options above are mostly equivalent
+* the zip and tar.gz files below are just source code
+* python packages are available at [PyPI](https://pypi.org/project/copyparty/#files)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0219-2309  `v1.16.14`  overwrite by upload
+
+## 🧪 new features
+
+* #139 overwrite existing files by uploading over them e9f78ea7
+  * default-disabled; a new togglebutton in the upload-UI configures it
+  * can optionally compare last-modified-time and only overwrite older files
+* [GDPR compliance](https://github.com/9001/copyparty#GDPR-compliance) (maybe/probably) 4be0d426
+
+## 🩹 bugfixes
+
+* some cosmetic volflag stuff, all harmless b190e676
+  * disabling a volflag `foo` with `-foo` shows a warning that `-foo` was not a recognized volflag, but it still does the right thing
+  * some volflags give the *"unrecognized volflag, will ignore"* warning, but not to worry, they still work just fine:
+    * `xz` to allow serverside xz-compression of uploaded files 
+* the option to customize the loader-spinner would glitch out during the initial page load 7d7d5d6c
+
+## 🔧 other changes
+
+* [randpic.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/handlers/randpic.py), new 404-handler example, returns a random pic from a folder 60d5f271
+* readme: [howto permanent cloudflare tunnel](https://github.com/9001/copyparty#permanent-cloudflare-tunnel) for easy hosting from home 2beb2acc
+* [synology-dsm](https://github.com/9001/copyparty/blob/hovudstraum/docs/synology-dsm.md): mention how to update the docker image 56ce5919
+* spinner improvements 6858cb06
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0213-2057  `v1.16.13`  configure with confidence
+
+## 🧪 new features
+
+* make the config-parser more helpful regarding volflags a255db70
+  * if an unrecognized volflag is specified, print a warning instead of silently ignoring it
+  * understand volflag-names with Uppercase and/or kebab-case (dashes), and not just snake_case (underscores)
+  * improve `--help-flags` to mention and explain all available flags
+* #136 WebDAV: support COPY 62ee7f69
+  * also support overwrite of existing target files (default-enabled according to the spec)
+    * the user must have the delete-permission to actually replace files
+* option to specify custom icons for certain file extensions 7e4702cf
+  * see `--ext-th` mentioned briefly in the [thumbnails section](https://github.com/9001/copyparty/#thumbnails)
+* option to replace the loading-spinner animation 685f0869
+  * including how to [make it exceptionally normal-looking](https://github.com/9001/copyparty/tree/hovudstraum/docs/rice#boring-loader-spinner)
+
+## 🩹 bugfixes
+
+* #136 WebDAV fixes 62ee7f69
+  * COPY/MOVE/MKCOL: challenge clients to provide the password as necessary
+    * most clients only need this in PROPFIND, but KDE-Dolphin is more picky
+  * MOVE: support `webdav://` Destination prefix as used by Dolphin, probably others
+* #136 WebDAV: improve support for KDE-Dolphin as client 9d769027
+  * it masquerades as a graphical browser yet still expects 401, so special-case it with a useragent scan
+
+## 🔧 other changes
+
+* Docker-only: quick hacky fix for the [musl CVE](https://www.openwall.com/lists/musl/2025/02/13/1) until the official fix is out 4d6626b0
+  * the docker images will be rebuilt when `musl-1.2.5-r9.apk` is released, in 6~24h or so
+  * until then, there is no support for reading korean XML files when running in docker
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0209-2331  `v1.16.12`  RTT
+
+## 🧪 new features
+
+* show rtt (network latency to server, including request processing time) in the top status text d27f1104 
+  * and log the client-reported RTT to serverlog 20ddeb6e
+* remember file selection when changing folders c7db08ed
+  * good for when you accidentally navigate elsewhere
+* option to restrict download-as-zip/tar to admins-only c87af9e8
+* #135 add [bubbleparty](https://github.com/9001/copyparty/blob/hovudstraum/bin/README.md#bubblepartysh), thx @coderofsalvation! 3582a100
+  * runs copyparty in a [sandbox](https://github.com/containers/bubblewrap), making it harder to gain unintended access through bugs in python or copyparty
+  * better alternative to [prisonparty](https://github.com/9001/copyparty/tree/hovudstraum/bin#prisonpartysh), more similar to [the sandboxing in the nixos package](https://github.com/9001/copyparty/blob/7dda77dcb/contrib/nixos/modules/copyparty.nix#L232-L272)
+* new plugin: [quickmove](https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/quickmove.js) 46f9e9ef
+  * adds hotkey `W` to quickly move selected files into a subfolder
+* #133 new plugin: [graft-thumbs.js](https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/graft-thumbs.js) 6c202eff
+  * in folders with foobar.mp3 and foobar.png, can copy the thumbnail from the png to the jpg (and then hide the png)
+* handlers: add [http-redirect example](https://github.com/9001/copyparty/blob/hovudstraum/bin/handlers/redirect.py) 22cbd2db
+* add [ping.html](https://github.com/9001/copyparty/blob/hovudstraum/srv/ping.html) 7de9d15a 910797cc
+
+## 🩹 bugfixes
+
+* improve iPad detection so they get opus instead of mp3 12dcea4f
+
+## 🔧 other changes
+
+* safeguard against accidental config loss cd71b505
+  * while no copyparty servers have ended up in this unfortunate situation yet (afaik), be proactive and borrow some experience from other docker-based services
+* readme: improve config examples 32e90859
+* improve serverlog entries regarding 403s b020fd4a
+* #132 mention fuse permissions in readme d9d2a092
+* traefik-example: fix disconnect during big uploads 6a9ffe7e
+* try to show an appropriate warning for media that the browser doesn't support playing 4ef35263
+  * was an attempt at detecting iphones failing to play high-color-precision webm files, but safari doesn't seem to realize itself that playback has failed, ah well 
+* copyparty.exe: update to python 3.12.9
+* update deps: dompurify 3.2.4
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0127-0140  `v1.16.11`  fix no-acode
+
+## 🧪 new features
+
+* u2c (commandline uploader): print download-links for uploaded files 1fe30363
+  * `-u` prints a list after all uploads finished
+  * `-ud` print during upload, after each file
+  * `-uf a.txt` writes them to `a.txt`
+
+## 🩹 bugfixes
+
+* [previous ver](https://github.com/9001/copyparty/releases/tag/v1.16.10) broke `--no-acode` (disable audio transcoding) by showing javascript errors 54a7256c
+  * reported on discord (thx)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0125-1809  `v1.16.10`  iOS9 is fine too
+
+## 🧪 new features
+
+* support audio playback on *really old* apple devices c9eba39e
+  * will now transcode to mp3 when necessary, since iOS didn't support opus-in-caf before iOS 11
+* support audio playback on *future* apple devices 28c9de3f 95390b65
+  * iOS 17.5 introduced support for opus-in-weba (like webp just audio instead) and, unlike caf, this intentionally supports vbr-opus (awesome)
+  * ...but the current code in iOS is too buggy, so this new format is default-disabled and we'll stick to caf for now fff38f48
+* ZeroMQ event-hooks can reject uploads 3a5c1d9f
+  * see [the example zmq listener](https://github.com/9001/copyparty/blob/1dace720/bin/zmq-recv.py#L26-L28)
+* chat with ZeroMQ event-hooks from javascript cdd3b67a
+  * replies from ZMQ REP servers are included in the msg-to-log responses
+  * which makes [this joke](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/usb-eject.py) possible f38c7543
+
+## 🩹 bugfixes
+
+* nope
+
+## 🔧 other changes
+
+* option to restrict the recent-uploads listing to admins-only b8b5214f
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0122-2326  `v1.16.9`  ZeroMQ says hello
+
+## 🧪 new features
+
+* event-hooks can send zeromq / zmq / 0mq messages; see [readme](https://github.com/9001/copyparty#zeromq) or `--help-hooks` for examples d9db1534
+* new volflags to specify the [allow-tag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes) of the markdown/logue sandbox, to allow fullscreen and such (see `--help-flags`) 6a0aaaf0
+* new volflag `nosparse` for possibly-better performance in very rare and specific scenarios 917380dd
+  * only enable this if you're uploading to s3 or something like that, and do plenty of benchmarking to make sure that it actually improved performance instead of making it worse
+
+## 🩹 bugfixes
+
+* restrict max-length of filekeys to 72 characters e0cac6fd
+* the hash-calculator mode of the commandline uploader produced incorrect whole-file hashes 4c04798a
+  * each chunk (`--chs`) was okay, but the final sum was not
+
+## 🔧 other changes
+
+* selftest the xml-parser on startup with malicious xml b2e8bf6e
+  * just in case a future python-version suddenly makes it unsafe somehow
+* disable some features if a dangerously misconfigured reverseproxy is detected 3f84b0a0
+* the download-as-zip feature now defaults to utf8 filenames 1231ce19
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0111-1611  `v1.16.8`  android boost
+
+## 🧪 new features
+
+* 10x faster file hashing in android-chrome ec507889
+  * on a recent pixel, speed went from 13 to 139 MiB/s
+  * android's sandboxing makes small reads expensive, so do bigger reads instead
+    * so the browser-tab will use more RAM on android now, maybe around 200 MiB
+    * this only affects chrome-based browsers on android, not firefox
+* PUT/multipart uploads: request-header `Accept: json` makes it return json instead of html, just like `?j` ce0e5be4
+* add config examples for [ishare](https://isharemac.app/), a MacOS screenshot utility inspired by ShareX 0c0d6b2b
+  * also includes a bug-workaround for [ishare#107](https://github.com/castdrian/ishare/issues/107) - copyparty will now include a toplevel json property `fileurl` in the response if exactly one file was uploaded
+  * the [connect-page](https://a.ocv.me/?hc) generates an appropriate `copyparty.iscu` for ishare; [it looks like this](https://github.com/user-attachments/assets/820730ad-2319-4912-8eb2-733755a4cf54)
+
+## 🩹 bugfixes
+
+* fix a potential upload deadlock when...
+  * ...the database (`-e2d`) is **not** enabled for any volume, and...
+  * ...either the shares feature, or user-changeable passwords, is enabled 9e542cf8
+* when loading the partial-uploads registry on startup, a cosmetic desync could occur 467acb47
+
+## 🔧 other changes
+
+* remove some deprecated properties in partial-upload metadata aa2a8fa2
+  * v1.15.7 is now the oldest version which still has any chance of reading a modern up2k.snap
+* #129 added howto: [using webdav when copyparty is behind IdP](https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients) -- thanks @wuast94 !
+* added howto: [install copyparty on a synology nas](https://github.com/9001/copyparty/blob/hovudstraum/docs/synology-dsm.md) 21f93042
+* more examples in the connect-page: 278258ee fb139697
+  * config-file for sharex on windows
+  * config-file for ishare on macos
+  * script for flameshot on linux
+* #75 add recommendation to use the [kamelåså project](https://github.com/steinuil/kameloso) instead of copyparty's [very-bad-idea.py](https://github.com/9001/copyparty/tree/hovudstraum/bin/mtag#dangerous-plugins) 9f84dc42
+* more reverse-proxy examples (haproxy, lighttpd, traefik, caddy) and improved nginx performance ac0a2da3
+  * readme has a [performance comparison](https://github.com/9001/copyparty?tab=readme-ov-file#reverse-proxy-performance) -- `haproxy > caddy > traefik > nginx > apache > lighttpd`
+* copyparty.exe: updated pillow 244e952f
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1223-0005  `v1.16.7`  an idp fix for xmas
+
+# ☃️🎄 **there is still time** 🎅🎁
+
+❄️❄️❄️ please [enjoy some appropriate music](https://a.ocv.me/pub/demo/music/.bonus/#af-55d4554d) -- you'll probably like this more than the idp thing honestly ❄️❄️❄️
+
+## 🧪 new features
+
+* more improvements to the recent-uploads feature 87598dcd
+  * move html rendering to clientside
+    * any changes to the filter-text applies in real-time
+    * loads 50% faster, reduces server-load by 30%
+    * inhibits search engines from indexing it
+
+## 🩹 bugfixes
+
+* using idp without e2d could mess with uploads dd6e9ea7
+* u2c (commandline uploader): fix window title 946a8c5b
+* mDNS/SSDP: fix incorrect log colors when multiple primary IPs are lost 552897ab
+
+## 🔧 other changes
+
+* ui: make it more obvious that the volume-control is a volume-control 7f044372
+* copyparty.exe: update deps (jinja2, markupsafe, pyinstaller) c0dacbc4
+* improve safety of custom plugins 988a7223
+  * if you've made your own plugins which expect certain values (host-header, filekeys) to be html-safe, then you'll want to upgrade
+  * also fixes rss-feed xml if password contains special characters
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1219-0037  `v1.16.6`  merry \x58mas
+
+# ☃️🎄 **it is time** 🎅🎁
+
+❄️❄️❄️ please [enjoy some appropriate music](https://a.ocv.me/pub/demo/music/.bonus/#af-55d4554d) (trust me on this one, you won't regret it) ❄️❄️❄️
+
+## 🧪 new features
+
+* [list of recent uploads](https://a.ocv.me/?ru) eaa4b04a
+  * new button in the controlpanel; can be disabled with `--no-ups-page`
+  * only users with the dot-permission can see dotfiles
+  * only admins can see uploader-ip and upload-times
+    * enable `--ups-when` to let all users see upload-times
+* #125 log decoded request-URLs 73f7249c
+  * non-ascii filenames would make the accesslog a wall of `%E5%B9%BB%E6%83%B3%E9%83%B7` so print [the decoded URL](https://github.com/user-attachments/assets/9d411183-30f3-4cb2-a880-84cf18011183) in addition to the original one, which is left as-is for debugging purposes
+
+## 🩹 bugfixes
+
+* #126 improve dotfile handling 4c4e48ba
+  * was impossible to delete a folder which contained hidden files if the user did not have the permission to see hidden files
+  * would also affect moving, renaming, copying folders, in which case the dotfiles would not be carried over to the new location
+  * now, dotfiles are always deleted, and always moved/copied into a new destination, on the condition that this is safe -- if the user has the dotfile permission in the target loocation but not in the source location, the dotfiles will be left behind to avoid accidentally making then browsable
+* ux: cosmetic eta/idle-timer fixes 01a3eb29
+
+## 🔧 other changes
+
+* warn on ambiguous comments in config files da5ad2ab
+* avoid writing mojibake to the log 3051b131
+  * use `\x`-encoding for unprintable text
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1211-2236  `v1.16.5`  4chrome
+
+## 🧪 new features
+
+* #124 add workaround for a chrome bug (crash during upload) 24ce46b3
+  * chrome and chromium-based browsers could OOM
+  * https://issues.chromium.org/issues/383568268
+
+* #122 "hybrid IdP", regular users can still auth while [IdP](https://github.com/9001/copyparty#identity-providers) is enabled 64501fd7
+  * previously, enabling IdP would entirely disable password-based login
+  * now, password-auth is attempted for requests without a valid IdP header
+
+## 🩹 bugfixes
+
+* the terminal window title would only change if `--no-ansi` was specified, which is exactly the opposite of what it should be (and now is) doing db3c0b09
+
+## 🔧 other changes
+
+* mDNS: better log messages when several IPs are added/removed a49bf81f
+* webdeps: update dompurify 06868606
+
+----
+
+this release includes a build of [copyparty-winpe64.exe](https://github.com/9001/copyparty/releases/download/v1.16.5/copyparty-winpe64.exe) since the last one was [almost a year ago](https://github.com/9001/copyparty/releases/tag/v1.10.1)
+
+* winpe64.exe is only for *very* specific usecases, you almost definitely *do not* want to download it, please just grab the regular [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) instead (works on all 64bit machines running win8 or newer)
+
+* the only difference between winpe64.exe and [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is that winpe64.exe works in the win7x64 PE (rescue-env), which makes it *almost* entirely useless, and every bit as dangerous to use as copyparty32.exe
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1207-0024  `v1.16.4`  ux is hard
+
+## 🧪 new features
+
+* improve the upload ui so it explains how to abort an unfinished upload when someone uploads to the wrong folder by accident be6afe2d
+  * also reduces serverload slightly when cloning an incoming file to multiple destinations
+* u2c (commandline uploader): windows improvements 91637800
+  * now supports globbing (filename wildcards) on windows
+  * progressbar in the windows taskbar (requires conemu or the "new windows terminal")
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2024-1204-0003  `v1.16.3`  120%
 

docs/chunksizes.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+
+# there's far better ways to do this but its 4am and i dont wanna think
+
+# just pypy it my dude
+
+import math
+
+def humansize(sz, terse=False):
+    for unit in ["B", "KiB", "MiB", "GiB", "TiB"]:
+        if sz < 1024:
+            break
+
+        sz /= 1024.0
+
+    ret = " ".join([str(sz)[:4].rstrip("."), unit])
+
+    if not terse:
+        return ret
+
+    return ret.replace("iB", "").replace(" ", "")
+
+
+def up2k_chunksize(filesize):
+    chunksize = 1024 * 1024
+    stepsize = 512 * 1024
+    while True:
+        for mul in [1, 2]:
+            nchunks = math.ceil(filesize * 1.0 / chunksize)
+            if nchunks <= 256 or (chunksize >= 32 * 1024 * 1024 and nchunks <= 4096):
+                return chunksize
+
+            chunksize += stepsize
+            stepsize *= mul
+
+
+def main():
+    prev = 1048576
+    n = n0 = 524288
+    while True:
+        csz = up2k_chunksize(n)
+        if csz > prev:
+            print(f"| {n-n0:>18_} | {humansize(n-n0):>8} | {prev:>13_} | {humansize(prev):>8} |".replace("_", " "))
+            prev = csz
+        n += n0
+
+
+main()

docs/devnotes.md

@@ -6,6 +6,7 @@
     * [up2k](#up2k) - quick outline of the up2k protocol
         * [why not tus](#why-not-tus) - I didn't know about [tus](https://tus.io/)
         * [why chunk-hashes](#why-chunk-hashes) - a single sha512 would be better, right?
+        * [list of chunk-sizes](#list-of-chunk-sizes) - specific chunksizes are enforced
 * [hashed passwords](#hashed-passwords) - regarding the curious decisions
 * [http api](#http-api)
     * [read](#read)
@@ -21,6 +22,7 @@
     * [dev env setup](#dev-env-setup)
     * [just the sfx](#just-the-sfx)
     * [build from release tarball](#build-from-release-tarball) - uses the included prebuilt webdeps
+    * [build from scratch](#build-from-scratch) - how the sausage is made
     * [complete release](#complete-release)
 * [debugging](#debugging)
     * [music playback halting on phones](#music-playback-halting-on-phones) - mostly fine on android
@@ -95,6 +97,44 @@ hashwasm would solve the streaming issue but reduces hashing speed for sha512 (x
 
 * blake2 might be a better choice since xxh is non-cryptographic, but that gets ~15 MiB/s on slower androids
 
+### list of chunk-sizes
+
+specific chunksizes are enforced  depending on total filesize
+
+each pair of filesize/chunksize is the largest filesize which will use its listed chunksize; a 512 MiB file will use chunksize 2 MiB, but if the file is one byte larger than 512 MiB then it becomes 3 MiB
+
+for the purpose of performance (or dodging arbitrary proxy limitations), it is possible to upload combined and/or partial chunks using stitching and/or subchunks respectively
+
+| filesize           | filesize | chunksize     | chunksz |
+| -----------------: | -------: | ------------: | ------: |
+|        268 435 456 |  256 MiB |     1 048 576 | 1.0 MiB |
+|        402 653 184 |  384 MiB |     1 572 864 | 1.5 MiB |
+|        536 870 912 |  512 MiB |     2 097 152 | 2.0 MiB |
+|        805 306 368 |  768 MiB |     3 145 728 | 3.0 MiB |
+|      1 073 741 824 |  1.0 GiB |     4 194 304 | 4.0 MiB |
+|      1 610 612 736 |  1.5 GiB |     6 291 456 | 6.0 MiB |
+|      2 147 483 648 |  2.0 GiB |     8 388 608 | 8.0 MiB |
+|      3 221 225 472 |  3.0 GiB |    12 582 912 |  12 MiB |
+|      4 294 967 296 |  4.0 GiB |    16 777 216 |  16 MiB |
+|      6 442 450 944 |  6.0 GiB |    25 165 824 |  24 MiB |
+|    137 438 953 472 |  128 GiB |    33 554 432 |  32 MiB |
+|    206 158 430 208 |  192 GiB |    50 331 648 |  48 MiB |
+|    274 877 906 944 |  256 GiB |    67 108 864 |  64 MiB |
+|    412 316 860 416 |  384 GiB |   100 663 296 |  96 MiB |
+|    549 755 813 888 |  512 GiB |   134 217 728 | 128 MiB |
+|    824 633 720 832 |  768 GiB |   201 326 592 | 192 MiB |
+|  1 099 511 627 776 |  1.0 TiB |   268 435 456 | 256 MiB |
+|  1 649 267 441 664 |  1.5 TiB |   402 653 184 | 384 MiB |
+|  2 199 023 255 552 |  2.0 TiB |   536 870 912 | 512 MiB |
+|  3 298 534 883 328 |  3.0 TiB |   805 306 368 | 768 MiB |
+|  4 398 046 511 104 |  4.0 TiB | 1 073 741 824 | 1.0 GiB |
+|  6 597 069 766 656 |  6.0 TiB | 1 610 612 736 | 1.5 GiB |
+|  8 796 093 022 208 |  8.0 TiB | 2 147 483 648 | 2.0 GiB |
+| 13 194 139 533 312 | 12.0 TiB | 3 221 225 472 | 3.0 GiB |
+| 17 592 186 044 416 | 16.0 TiB | 4 294 967 296 | 4.0 GiB |
+| 26 388 279 066 624 | 24.0 TiB | 6 442 450 944 | 6.0 GiB |
+| 35 184 372 088 832 | 32.0 TiB | 8 589 934 592 | 8.0 GiB |
+
 
 # hashed passwords
 
@@ -133,8 +173,8 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | GET | `?tar=xz:9` | ...as an xz-level-9 gnu-tar file |
 | GET | `?tar=pax` | ...as a pax-tar file |
 | GET | `?tar=pax,xz` | ...as an xz-level-1 pax-tar file |
-| GET | `?zip=utf-8` | ...as a zip file |
-| GET | `?zip` | ...as a WinXP-compatible zip file |
+| GET | `?zip` | ...as a zip file |
+| GET | `?zip=dos` | ...as a WinXP-compatible zip file |
 | GET | `?zip=crc` | ...as an MSDOS-compatible zip file |
 | GET | `?tar&w` | pregenerate webp thumbnails |
 | GET | `?tar&j` | pregenerate jpg thumbnails |
@@ -143,11 +183,17 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | GET | `?dls` | show active downloads (do this as admin) |
 | GET | `?ups` | show recent uploads from your IP |
 | GET | `?ups&filter=f` | ...where URL contains `f` |
+| GET | `?ru` | show all recent uploads |
+| GET | `?ru&filter=f` | ...where URL contains `f` |
+| GET | `?ru&j` | ...as json |
 | GET | `?mime=foo` | specify return mimetype `foo` |
 | GET | `?v` | render markdown file at URL |
 | GET | `?v` | open image/video/audio in mediaplayer |
 | GET | `?txt` | get file at URL as plaintext |
 | GET | `?txt=iso-8859-1` | ...with specific charset |
+| GET | `?tail` | continuously stream a growing file |
+| GET | `?tail=1024` | ...starting from byte 1024 |
+| GET | `?tail=-128` | ...starting 128 bytes from the end |
 | GET | `?th` | get image/video at URL as thumbnail |
 | GET | `?th=opus` | convert audio file to 128kbps opus |
 | GET | `?th=caf` | ...in the iOS-proprietary container |
@@ -170,6 +216,7 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | method | params | body | result |
 |--|--|--|--|
 | PUT | | (binary data) | upload into file at URL |
+| PUT | `?j` | (binary data) | ...and reply with json |
 | PUT | `?ck` | (binary data) | upload without checksum gen (faster) |
 | PUT | `?ck=md5` | (binary data) | return md5 instead of sha512 |
 | PUT | `?gz` | (binary data) | compress with gzip and write into file at URL |
@@ -194,6 +241,7 @@ upload modifiers:
 | http-header | url-param | effect |
 |--|--|--|
 | `Accept: url` | `want=url` | return just the file URL |
+| `Accept: json` | `want=json` | return upload info as json; same as `?j` |
 | `Rand: 4` | `rand=4` | generate random filename with 4 characters |
 | `Life: 30` | `life=30` | delete file after 30 seconds |
 | `CK: no` | `ck` | disable serverside checksum (maybe faster) |
@@ -213,6 +261,7 @@ upload modifiers:
 |--|--|--|
 | GET | `?reload=cfg` | reload config files and rescan volumes |
 | GET | `?scan` | initiate a rescan of the volume which provides URL |
+| GET | `?scan=/a,/b` | initiate a rescan of volumes `/a` and `/b` |
 | GET | `?stack` | show a stacktrace of all threads |
 
 ## general
@@ -237,8 +286,11 @@ on writing your own [hooks](../README.md#event-hooks)
 hooks can cause intentional side-effects,  such as redirecting an upload into another location, or creating+indexing additional files, or deleting existing files, by returning json on stdout
 
 * `reloc` can redirect uploads before/after uploading has finished, based on filename, extension, file contents, uploader ip/name etc.
+  * example: [reloc-by-ext](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/reloc-by-ext.py)
 * `idx` informs copyparty about a new file to index as a consequence of this upload
+  * example: [podcast-normalizer.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/podcast-normalizer.py)
 * `del` tells copyparty to delete an unrelated file by vpath
+  * example: (　´・ω・) nyoro~n
 
 for these to take effect, the hook must be defined with the `c1` flag; see example [reloc-by-ext](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/reloc-by-ext.py)
 
@@ -291,18 +343,19 @@ for the `re`pack to work, first run one of the sfx'es once to unpack it
 
 you need python 3.9 or newer due to type hints
 
-the rest is mostly optional; if you need a working env for vscode or similar
+setting up a venv with the below packages is only necessary if you want it for vscode or similar
 
 ```sh
 python3 -m venv .venv
 . .venv/bin/activate
 pip install jinja2 strip_hints  # MANDATORY
 pip install argon2-cffi  # password hashing
+pip install pyzmq  # send 0mq from hooks
 pip install mutagen  # audio metadata
 pip install pyftpdlib  # ftp server
 pip install partftpy  # tftp server
 pip install impacket  # smb server -- disable Windows Defender if you REALLY need this on windows
-pip install Pillow pyheif-pillow-opener pillow-avif-plugin  # thumbnails
+pip install Pillow pyheif-pillow-opener  # thumbnails
 pip install pyvips  # faster thumbnails
 pip install psutil  # better cleanup of stuck metadata parsers on windows 
 pip install black==21.12b0 click==8.0.2 bandit pylint flake8 isort mypy  # vscode tooling
@@ -344,6 +397,39 @@ python3 setup.py install --skip-build --prefix=/usr --root=$HOME/pe/copyparty
 ```
 
 
+## build from scratch
+
+how the sausage is made:
+
+to get started, first `cd` into the `scripts` folder
+
+* the first step is the webdeps; they end up in `../copyparty/web/deps/` for example `../copyparty/web/deps/marked.js.gz` -- if you need to build the webdeps, run `make -C deps-docker`
+  * this needs rootless podman and the `podman-docker` compat-layer to pretend it's docker, although it *should* be possible to use rootful/rootless docker too
+  * if you don't have rootless podman/docker then `sudo make -C deps-docker` is fine too
+  * alternatively, you can entirely skip building the webdeps and instead extract the compiled webdeps from the latest github release with `./make-sfx.sh fast dl-wd`
+
+* next, build `copyparty-sfx.py` by running `./make-sfx.sh gz fast`
+  * this is a dependency for most of the remaining steps, since they take the sfx as input
+  * removing `fast` makes it compress better
+  * removing `gz` too compresses even better, but startup gets slower
+
+* if you want to build the `.pyz` standalone "binary", now run `./make-pyz.sh`
+
+* if you want to build a pypi package, now run `./make-pypi-release.sh d`
+
+* if you want to build a docker-image, you have two options:
+  * if you want to use podman to build all docker-images for all supported architectures, now run `(cd docker; ./make.sh hclean; ./make.sh hclean pull img)`
+  * if you want to use docker to build all docker-images for your native architecture, now run `sudo make -C docker`
+  * if you want to do something else, please take a look at `docker/make.sh` or `docker/Makefile` for inspiration
+
+* if you want to build the windows exe, first grab some snacks and a beer, [you'll need it](https://github.com/9001/copyparty/tree/hovudstraum/scripts/pyinstaller)
+
+the complete list of buildtime dependencies to do a build from scratch is as follows:
+
+* on ubuntu-server, install podman or [docker](https://get.docker.com/), and then `sudo apt install make zip bzip2`
+  * because ubuntu is specifically what someone asked about :-p
+
+
 ## complete release
 
 also builds the sfx so skip the sfx section above

docs/examples/docker/basic-docker-compose/docker-compose.yml

@@ -1,4 +1,3 @@
-version: '3'
 services:
 
   copyparty:
@@ -11,9 +10,17 @@ services:
       - ./:/cfg:z
       - /path/to/your/fileshare/top/folder:/w:z
 
+    environment:
+      LD_PRELOAD: /usr/lib/libmimalloc-secure.so.NOPE
+      # enable mimalloc by replacing "NOPE" with "2" for a nice speed-boost (will use twice as much ram)
+
+      PYTHONUNBUFFERED: 1
+      # ensures log-messages are not delayed (but can reduce speed a tiny bit)
+
     stop_grace_period: 15s  # thumbnailer is allowed to continue finishing up for 10s after the shutdown signal
     healthcheck:
-      test: ["CMD-SHELL", "wget --spider -q 127.0.0.1:3923/?reset"]
+      # hide it from logs with "/._" so it matches the default --lf-url filter 
+      test: ["CMD-SHELL", "wget --spider -q 127.0.0.1:3923/?reset=/._"]
       interval: 1m
       timeout: 2s
       retries: 5

docs/examples/docker/idp-authelia-traefik/docker-compose.yml

@@ -23,6 +23,12 @@ services:
       - 'traefik.http.routers.copyparty.tls=true'
       - 'traefik.http.routers.copyparty.middlewares=authelia@docker'
     stop_grace_period: 15s  # thumbnailer is allowed to continue finishing up for 10s after the shutdown signal
+    environment:
+      LD_PRELOAD: /usr/lib/libmimalloc-secure.so.NOPE
+      # enable mimalloc by replacing "NOPE" with "2" for a nice speed-boost (will use twice as much ram)
+
+      PYTHONUNBUFFERED: 1
+      # ensures log-messages are not delayed (but can reduce speed a tiny bit)
 
   authelia:
     image: authelia/authelia:v4.38.0-beta3  # the config files in the authelia folder use the new syntax

docs/examples/docker/idp-authentik-traefik/docker-compose.yml

@@ -22,13 +22,13 @@ services:
       - 'traefik.http.routers.fs.rule=Host(`fs.example.com`)'
       - 'traefik.http.routers.fs.entrypoints=http'
       #- 'traefik.http.routers.fs.middlewares=authelia@docker'  # TODO: ???
-    healthcheck:
-      test: ["CMD-SHELL", "wget --spider -q 127.0.0.1:3923/?reset"]
-      interval: 1m
-      timeout: 2s
-      retries: 5
-      start_period: 15s
     stop_grace_period: 15s  # thumbnailer is allowed to continue finishing up for 10s after the shutdown signal
+    environment:
+      LD_PRELOAD: /usr/lib/libmimalloc-secure.so.NOPE
+      # enable mimalloc by replacing "NOPE" with "2" for a nice speed-boost (will use twice as much ram)
+
+      PYTHONUNBUFFERED: 1
+      # ensures log-messages are not delayed (but can reduce speed a tiny bit)
 
   traefik:
     image: traefik:v2.11

docs/examples/docker/idp/copyparty.conf

@@ -13,6 +13,8 @@
 # because that is the data-volume in the docker containers,
 # because a deployment like this (with an IdP) is more commonly
 # seen in containerized environments -- but this is not required
+#
+# the example group "su" (super-user) is the admins group
 
 
 [global]
@@ -78,6 +80,18 @@
     rwmda: @${g}, @su  # read-write-move-delete-admin for that group + the "su" group
 
 
+[/sus/${u%+su}]  # users which ARE members of group "su" gets /sus/username
+  /w/tank1/${u}  # which will be "tank1/username" in the docker data volume
+  accs:
+    rwmda: ${u}  # read-write-move-delete-admin for that username
+
+
+[/m8s/${u%-su}]  # users which are NOT members of group "su" gets /m8s/username
+  /w/tank2/${u}  # which will be "tank2/username" in the docker data volume
+  accs:
+    rwmda: ${u}  # read-write-move-delete-admin for that username
+
+
 # and create some strategic volumes to prevent anyone from gaining
 # unintended access to priv folders if the users/groups db is lost
 [/u]
@@ -88,3 +102,14 @@
   /w/lounge
   accs:
     rwmda: @su
+[/sus]
+  /w/tank1
+[/m8s]
+  /w/tank2
+
+
+# some other things you can do:
+#  [/demo/${u%-su,%-fds}]   # users which are NOT members of "su" or "fds"
+#  [/demo/${u%+su,%+fds}]   # users which ARE members of BOTH "su" and "fds"
+#  [/demo/${g%-su}]         # all groups except su
+#  [/demo/${g%-su,%-fds}]   # all groups except su and fds

docs/idp.md

@@ -9,9 +9,9 @@ in the copyparty `[global]` config, specify which headers to read client info fr
 
 # important notes
 
-## IdP volumes are forgotten on shutdown
+## by default, IdP volumes are forgotten on shutdown
 
-IdP volumes, meaning dynamically-created volumes, meaning volumes that contain `${u}` or `${g}` in their URL, will be forgotten during a server restart and then "revived" when the volume's owner sends their first request after the restart
+IdP volumes, meaning dynamically-created volumes, meaning volumes that contain `${u}` or `${g}` in their URL, will (by default) be forgotten during a server restart and then "revived" when the volume's owner sends their first request after the restart
 
 until each IdP volume is revived, it will inherit the permissions of its parent volume (if any)
 
@@ -19,4 +19,36 @@ this means that, if an IdP volume is located inside a folder that is readable by
 
 and likewise -- if the IdP volume is inside a folder that is only accessible by certain users, but the IdP volume is configured to allow access from unauthenticated users, then the contents of the volume will NOT be accessible until it is revived
 
-until this limitation is fixed (if ever), it is recommended to place IdP volumes inside an appropriate parent volume, so they can inherit acceptable permissions until their revival; see the "strategic volumes" at the bottom of [./examples/docker/idp/copyparty.conf](./examples/docker/idp/copyparty.conf)
+it is recommended to place IdP volumes inside an appropriate parent volume, so they can inherit acceptable permissions until their revival; see the "strategic volumes" at the bottom of [./examples/docker/idp/copyparty.conf](./examples/docker/idp/copyparty.conf)
+
+## but you can enable IdP volume persistence
+
+global-option `idp-store` can enable user/group persistence across restarts;
+
+* `idp-store: 1` (default) will log users into a database, but not actually "remember" them (the knowledge is ignored)
+* `idp-store: 2` remembers usernames only
+* `idp-store: 3` remembers usernames and their groups
+
+the reason why this is default-disabled, is because you may expect copyparty to forget about a user when you delete them from the IdP-server; this will not be the case any longer, you will need to click `view idp cache` in the controlpanel and manually remove the users you want gone
+
+
+## Connecting webdav clients
+
+If you use only idp and want to connect via rclone you have to adapt a few things.
+The following steps are for Authelia, but should be easy adaptable to other IdPs and clients. There may be better/smarter ways to do this, but this is a known solution.
+
+1. Add a rule for your domain and set it to one factor
+```
+  rules:
+    - domain: 'sub.domain.tld'
+      policy: one_factor
+```
+2. After you created your rclone config find its location with `rclone config file` and add the headers option to it, change the string to `username:password` base64 encoded. Make sure to set the right url location, otherwise you will get a 401 from copyparty.
+```
+[servername-dav]
+type = webdav
+url = https://sub.domain.tld/u/user/priv/
+vendor = owncloud
+pacer_min_sleep = 0.01ms
+headers = Proxy-Authorization,basic base64encodedstring==
+```