v1.16.10

also add transcoding from opus to mp3 if client wants mp3,
overriding the feature detection for opus support

README.md

@@ -80,6 +80,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [metadata from audio files](#metadata-from-audio-files) - set `-e2t` to index tags on upload
     * [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags
     * [event hooks](#event-hooks) - trigger a program on uploads, renames etc ([examples](./bin/hooks/))
+        * [zeromq](#zeromq) - event-hooks can send zeromq messages
         * [upload events](#upload-events) - the older, more powerful approach ([examples](./bin/mtag/))
     * [handlers](#handlers) - redefine behavior with plugins ([examples](./bin/handlers/))
     * [ip auth](#ip-auth) - autologin based on IP range (CIDR)
@@ -92,6 +93,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [listen on port 80 and 443](#listen-on-port-80-and-443) - become a *real* webserver
     * [reverse-proxy](#reverse-proxy) - running copyparty next to other websites
         * [real-ip](#real-ip) - teaching copyparty how to see client IPs
+        * [reverse-proxy performance](#reverse-proxy-performance)
     * [prometheus](#prometheus) - metrics/stats can be enabled
     * [other extremely specific features](#other-extremely-specific-features) - you'll never find a use for these
         * [custom mimetypes](#custom-mimetypes) - change the association of a file extension
@@ -140,6 +142,7 @@ just run **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/
 * or if you cannot install python, you can use [copyparty.exe](#copypartyexe) instead
 * or install [on arch](#arch-package) ╱ [on NixOS](#nixos-module) ╱ [through nix](#nix-package)
 * or if you are on android, [install copyparty in termux](#install-on-android)
+* or maybe you have a [synology nas / dsm](./docs/synology-dsm.md)
 * or if your computer is messed up and nothing else works, [try the pyz](#zipapp)
 * or if you prefer to [use docker](./scripts/docker/) 🐋 you can do that too
   * docker has all deps built-in, so skip this step:
@@ -350,10 +353,19 @@ same order here too
 * iPhones: the volume control doesn't work because [apple doesn't want it to](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html#//apple_ref/doc/uid/TP40009523-CH5-SW11)
   * `AudioContext` will probably never be a viable workaround as apple introduces new issues faster than they fix current ones
 
+* iPhones: music volume goes on a rollercoaster during song changes
+  * nothing I can do about it because `AudioContext` is still broken in safari
+
 * iPhones: the preload feature (in the media-player-options tab) can cause a tiny audio glitch 20sec before the end of each song, but disabling it may cause worse iOS bugs to appear instead
   * just a hunch, but disabling preloading may cause playback to stop entirely, or possibly mess with bluetooth speakers
   * tried to add a tooltip regarding this but looks like apple broke my tooltips
 
+* iPhones: preloaded awo files make safari log MEDIA_ERR_NETWORK errors as playback starts, but the song plays just fine so eh whatever
+  * awo, opus-weba, is apple's new take on opus support, replacing opus-caf which was technically limited to cbr opus
+
+* iPhones: preloading another awo file may cause playback to stop
+  * can be somewhat mitigated with `mp.au.play()` in `mp.onpreload` but that can hit a race condition in safari that starts playing the same audio object twice in parallel...
+
 * Windows: folders cannot be accessed if the name ends with `.`
   * python or windows bug
 
@@ -613,8 +625,8 @@ select which type of archive you want in the `[⚙️] config` tab:
 | `pax` | `?tar=pax` | pax-format tar, futureproof, not as fast |
 | `tgz` | `?tar=gz` | gzip compressed gnu-tar (slow), for `curl \| tar -xvz` |
 | `txz` | `?tar=xz` | gnu-tar with xz / lzma compression (v.slow) |
-| `zip` | `?zip=utf8` | works everywhere, glitchy filenames on win7 and older |
+| `zip` | `?zip` | works everywhere, glitchy filenames on win7 and older |
-| `zip_dos` | `?zip` | traditional cp437 (no unicode) to fix glitchy filenames |
+| `zip_dos` | `?zip=dos` | traditional cp437 (no unicode) to fix glitchy filenames |
 | `zip_crc` | `?zip=crc` | cp437 with crc32 computed early for truly ancient software |
 
 * gzip default level is `3` (0=fast, 9=best), change with `?tar=gz:9`
@@ -622,7 +634,7 @@ select which type of archive you want in the `[⚙️] config` tab:
 * bz2 default level is `2` (1=fast, 9=best), change with `?tar=bz2:9`
 * hidden files ([dotfiles](#dotfiles)) are excluded unless account is allowed to list them
   * `up2k.db` and `dir.txt` is always excluded
-* bsdtar supports streaming unzipping: `curl foo?zip=utf8 | bsdtar -xv`
+* bsdtar supports streaming unzipping: `curl foo?zip | bsdtar -xv`
   * good, because copyparty's zip is faster than tar on small files
 * `zip_crc` will take longer to download since the server has to read each file twice
   * this is only to support MS-DOS PKZIP v2.04g (october 1993) and older
@@ -646,7 +658,7 @@ dragdrop is the recommended way, but you may also:
 
 * select some files (not folders) in your file explorer and press CTRL-V inside the browser window
 * use the [command-line uploader](https://github.com/9001/copyparty/tree/hovudstraum/bin#u2cpy)
-* upload using [curl or sharex](#client-examples)
+* upload using [curl, sharex, ishare, ...](#client-examples)
 
 when uploading files through dragdrop or CTRL-V, this initiates an upload using `up2k`; there are two browser-based uploaders available:
 * `[🎈] bup`, the basic uploader, supports almost every browser since netscape 4.0
@@ -885,6 +897,8 @@ will show uploader IP and upload-time if the visitor has the admin permission
 
 * global-option `--ups-when` makes upload-time visible to all users, and not just admins
 
+* global-option `--ups-who` (volflag `ups_who`) specifies who gets access (0=nobody, 1=admins, 2=everyone), default=2
+
 note that the [🧯 unpost](#unpost) feature is better suited for viewing *your own* recent uploads, as it includes the option to undo/delete them
 
 
@@ -924,6 +938,11 @@ open the `[🎺]` media-player-settings tab to configure it,
   * `[aac]` converts `aac` and `m4a` files into opus (if supported by browser) or mp3
   * `[oth]` converts all other known formats into opus (if supported by browser) or mp3
     * `aac|ac3|aif|aiff|alac|alaw|amr|ape|au|dfpwm|dts|flac|gsm|it|m4a|mo3|mod|mp2|mp3|mpc|mptm|mt2|mulaw|ogg|okt|opus|ra|s3m|tak|tta|ulaw|wav|wma|wv|xm|xpk`
+* "transcode to":
+  * `[opus]` produces an `opus` whenever transcoding is necessary (the best choice on Android and PCs)
+  * `[awo]` is `opus` in a `weba` file, good for iPhones (iOS 17.5 and newer) but Apple is still fixing some state-confusion bugs as of iOS 18.2.1
+  * `[caf]` is `opus` in a `caf` file, good for iPhones (iOS 11 through 17), technically unsupported by Apple but works for the mos tpart
+  * `[mp3]` -- the myth, the legend, the undying master of mediocre sound quality that definitely works everywhere
 * "tint" reduces the contrast of the playback bar
 
 
@@ -1104,6 +1123,8 @@ on macos, connect from finder:
 
 in order to grant full write-access to webdav clients, the volflag `daw` must be set and the account must also have delete-access (otherwise the client won't be allowed to replace the contents of existing files, which is how webdav works)
 
+> note: if you have enabled [IdP authentication](#identity-providers) then that may cause issues for some/most webdav clients; see [the webdav section in the IdP docs](https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients)
+
 
 ### connecting to webdav from windows
 
@@ -1454,6 +1475,23 @@ there's a bunch of flags and stuff, see `--help-hooks`
 if you want to write your own hooks, see [devnotes](./docs/devnotes.md#event-hooks)
 
 
+### zeromq
+
+event-hooks can send zeromq messages  instead of running programs
+
+to send a 0mq message every time a file is uploaded,
+
+* `--xau zmq:pub:tcp://*:5556` sends a PUB to any/all connected SUB clients
+* `--xau t3,zmq:push:tcp://*:5557` sends a PUSH to exactly one connected PULL client
+* `--xau t3,j,zmq:req:tcp://localhost:5555` sends a REQ to the connected REP client
+
+the PUSH and REQ examples have `t3` (timeout after 3 seconds) because they block if there's no clients to talk to
+
+* the REQ example does `t3,j` to send extended upload-info as json instead of just the filesystem-path
+
+see [zmq-recv.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/zmq-recv.py) if you need something to receive the messages with
+
+
 ### upload events
 
 the older, more powerful approach ([examples](./bin/mtag/)):
@@ -1541,12 +1579,16 @@ connecting to an aws s3 bucket and similar
 
 there is no built-in support for this, but you can use FUSE-software such as [rclone](https://rclone.org/) / [geesefs](https://github.com/yandex-cloud/geesefs) / [JuiceFS](https://juicefs.com/en/) to first mount your cloud storage as a local disk, and then let copyparty use (a folder in) that disk as a volume
 
-you may experience poor upload performance this way, but that can sometimes be fixed by specifying the volflag `sparse` to force the use of sparse files; this has improved the upload speeds from `1.5 MiB/s` to over `80 MiB/s` in one case, but note that you are also more likely to discover funny bugs in your FUSE software this way, so buckle up
+you will probably get decent speeds with the default config, however most likely restricted to using one TCP connection per file, so the upload-client won't be able to send multiple chunks in parallel
+
+> before [v1.13.5](https://github.com/9001/copyparty/releases/tag/v1.13.5) it was recommended to use the volflag `sparse` to force-allow multiple chunks in parallel; this would improve the upload-speed from `1.5 MiB/s` to over `80 MiB/s` at the risk of provoking latent bugs in S3 or JuiceFS. But v1.13.5 added chunk-stitching, so this is now probably much less important. On the contrary, `nosparse` *may* now increase performance in some cases. Please try all three options (default, `sparse`, `nosparse`) as the optimal choice depends on your network conditions and software stack (both the FUSE-driver and cloud-server)
 
 someone has also tested geesefs in combination with [gocryptfs](https://nuetzlich.net/gocryptfs/) with surprisingly good results, getting 60 MiB/s upload speeds on a gbit line, but JuiceFS won with 80 MiB/s using its built-in encryption
 
 you may improve performance by specifying larger values for `--iobuf` / `--s-rd-sz` / `--s-wr-sz`
 
+> if you've experimented with this and made interesting observations, please share your findings so we can add a section with specific recommendations :-)
+
 
 ## hiding from google
 
@@ -1669,10 +1711,16 @@ some reverse proxies (such as [Caddy](https://caddyserver.com/)) can automatical
 
 for improved security (and a 10% performance boost) consider listening on a unix-socket with `-i unix:770:www:/tmp/party.sock` (permission `770` means only members of group `www` can access it)
 
-example webserver configs:
+example webserver / reverse-proxy configs:
 
-* [nginx config](contrib/nginx/copyparty.conf) -- entire domain/subdomain
+* [apache config](contrib/apache/copyparty.conf)
-* [apache2 config](contrib/apache/copyparty.conf) -- location-based
+* caddy uds: `caddy reverse-proxy --from :8080 --to unix///dev/shm/party.sock`
+* caddy tcp: `caddy reverse-proxy --from :8081 --to http://127.0.0.1:3923`
+* [haproxy config](contrib/haproxy/copyparty.conf)
+* [lighttpd subdomain](contrib/lighttpd/subdomain.conf) -- entire domain/subdomain
+* [lighttpd subpath](contrib/lighttpd/subpath.conf) -- location-based (not optimal, but in case you need it)
+* [nginx config](contrib/nginx/copyparty.conf) -- recommended
+* [traefik config](contrib/traefik/copyparty.yaml)
 
 
 ### real-ip
@@ -1684,6 +1732,38 @@ if you (and maybe everybody else) keep getting a message that says `thank you fo
 for most common setups, there should be a helpful message in the server-log explaining what to do, but see [docs/xff.md](docs/xff.md) if you want to learn more, including a quick hack to **just make it work** (which is **not** recommended, but hey...)
 
 
+### reverse-proxy performance
+
+most reverse-proxies support connecting to copyparty either using uds/unix-sockets (`/dev/shm/party.sock`, faster/recommended) or using tcp (`127.0.0.1`)
+
+with copyparty listening on a uds / unix-socket / unix-domain-socket and the reverse-proxy connecting to that:
+
+| index.html   | upload      | download    | software |
+| ------------ | ----------- | ----------- | -------- |
+| 28'900 req/s | 6'900 MiB/s | 7'400 MiB/s | no-proxy |
+| 18'750 req/s | 3'500 MiB/s | 2'370 MiB/s | haproxy |
+|  9'900 req/s | 3'750 MiB/s | 2'200 MiB/s | caddy |
+| 18'700 req/s | 2'200 MiB/s | 1'570 MiB/s | nginx |
+|  9'700 req/s | 1'750 MiB/s | 1'830 MiB/s | apache |
+|  9'900 req/s | 1'300 MiB/s | 1'470 MiB/s | lighttpd |
+
+when connecting the reverse-proxy to `127.0.0.1` instead (the basic and/or old-fasioned way), speeds are a bit worse:
+
+| index.html   | upload      | download    | software |
+| ------------ | ----------- | ----------- | -------- |
+| 21'200 req/s | 5'700 MiB/s | 6'700 MiB/s | no-proxy |
+| 14'500 req/s | 1'700 MiB/s | 2'170 MiB/s | haproxy |
+| 11'100 req/s | 2'750 MiB/s | 2'000 MiB/s | traefik |
+|  8'400 req/s | 2'300 MiB/s | 1'950 MiB/s | caddy |
+| 13'400 req/s | 1'100 MiB/s | 1'480 MiB/s | nginx |
+|  8'400 req/s | 1'000 MiB/s | 1'000 MiB/s | apache |
+|  6'500 req/s | 1'270 MiB/s | 1'500 MiB/s | lighttpd |
+
+in summary, `haproxy > caddy > traefik > nginx > apache > lighttpd`, and use uds when possible (traefik does not support it yet)
+
+* if these results are bullshit because my config exampels are bad, please submit corrections!
+
+
 ## prometheus
 
 metrics/stats can be enabled  at URL `/.cpr/metrics` for grafana / prometheus / etc (openmetrics 1.0.0)
@@ -1997,7 +2077,8 @@ interact with copyparty using non-browser clients
   * can be downloaded from copyparty: controlpanel -> connect -> [partyfuse.py](http://127.0.0.1:3923/.cpr/a/partyfuse.py)
   * [rclone](https://rclone.org/) as client can give ~5x performance, see [./docs/rclone.md](docs/rclone.md)
 
-* sharex (screenshot utility): see [./contrib/sharex.sxcu](contrib/#sharexsxcu)
+* sharex (screenshot utility): see [./contrib/sharex.sxcu](./contrib/#sharexsxcu)
+  * and for screenshots on macos, see [./contrib/ishare.iscu](./contrib/#ishareiscu)
   * and for screenshots on linux, see [./contrib/flameshot.sh](./contrib/flameshot.sh)
 
 * contextlet (web browser integration); see [contrib contextlet](contrib/#send-to-cppcontextletjson)
@@ -2261,13 +2342,13 @@ mandatory deps:
 
 install these to enable bonus features
 
-enable hashed passwords in config: `argon2-cffi`
+enable [hashed passwords](#password-hashing) in config: `argon2-cffi`
 
-enable ftp-server:
+enable [ftp-server](#ftp-server):
 * for just plaintext FTP, `pyftpdlib` (is built into the SFX)
 * with TLS encryption, `pyftpdlib pyopenssl`
 
-enable music tags:
+enable [music tags](#metadata-from-audio-files):
 * either `mutagen` (fast, pure-python, skips a few tags, makes copyparty GPL? idk)
 * or `ffprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users)
 
@@ -2278,8 +2359,9 @@ enable [thumbnails](#thumbnails) of...
 * **AVIF pictures:** `pyvips` or `ffmpeg` or `pillow-avif-plugin`
 * **JPEG XL pictures:** `pyvips` or `ffmpeg`
 
-enable [smb](#smb-server) support (**not** recommended):
+enable sending [zeromq messages](#zeromq) from event-hooks: `pyzmq`
-* `impacket==0.12.0`
+
+enable [smb](#smb-server) support (**not** recommended): `impacket==0.12.0`
 
 `pyvips` gives higher quality thumbnails than `Pillow` and is 320% faster, using 270% more ram: `sudo apt install libvips42 && python3 -m pip install --user -U pyvips`
 

bin/hooks/README.md

@@ -30,4 +30,5 @@ these are `--xiu` hooks; unlike `xbu` and `xau` (which get executed on every sin
 # on message
 * [wget.py](wget.py) lets you download files by POSTing URLs to copyparty
 * [qbittorrent-magnet.py](qbittorrent-magnet.py) starts downloading a torrent if you post a magnet url
+* [usb-eject.py](usb-eject.py) adds web-UI buttons to safe-remove usb flashdrives shared through copyparty
 * [msg-log.py](msg-log.py) is a guestbook; logs messages to a doc in the same folder

bin/hooks/usb-eject.js

@@ -0,0 +1,54 @@
+// see usb-eject.py for usage
+
+function usbclick() {
+    QS('#treeul a[href="/usb/"]').click();
+}
+
+function eject_cb() {
+    var t = this.responseText;
+    if (t.indexOf('can be safely unplugged') < 0 && t.indexOf('Device can be removed') < 0)
+        return toast.err(30, 'usb eject failed:\n\n' + t);
+
+    toast.ok(5, esc(t.replace(/ - /g, '\n\n')));
+    usbclick(); setTimeout(usbclick, 10);
+};
+
+function add_eject_2(a) {
+    var aw = a.getAttribute('href').split(/\//g);
+    if (aw.length != 4 || aw[3])
+        return;
+
+    var v = aw[2],
+        k = 'umount_' + v;
+
+    qsr('#' + k);
+    a.appendChild(mknod('span', k, '⏏'), a);
+
+    var o = ebi(k);
+    o.style.cssText = 'position:absolute; right:1em; margin-top:-.2em; font-size:1.3em';
+    o.onclick = function (e) {
+        ev(e);
+        var xhr = new XHR();
+        xhr.open('POST', get_evpath(), true);
+        xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded;charset=UTF-8');
+        xhr.send('msg=' + uricom_enc(':usb-eject:' + v + ':'));
+        xhr.onload = xhr.onerror = eject_cb;
+        toast.inf(10, "ejecting " + v + "...");
+    };
+};
+
+function add_eject() {
+    for (var a of QSA('#treeul a[href^="/usb/"]'))
+        add_eject_2(a);
+};
+
+(function() {
+    var f0 = treectl.rendertree;
+    treectl.rendertree = function (res, ts, top0, dst, rst) {
+        var ret = f0(res, ts, top0, dst, rst);
+        add_eject();
+        return ret;
+    };
+})();
+
+setTimeout(add_eject, 50);

bin/hooks/usb-eject.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import stat
+import subprocess as sp
+import sys
+
+
+"""
+if you've found yourself using copyparty to serve flashdrives on a LAN
+and your only wish is that the web-UI had a button to unmount / safely
+remove those flashdrives, then boy howdy are you in the right place :D
+
+put usb-eject.js in the webroot (or somewhere else http-accessible)
+then run copyparty with these args:
+
+   -v /run/media/ed:/usb:A:c,hist=/tmp/junk
+   --xm=c1,bin/hooks/usb-eject.py
+   --js-browser=/usb-eject.js
+
+which does the following respectively,
+
+  * share all of /run/media/ed as /usb with admin for everyone
+     and put the histpath somewhere it won't cause trouble
+  * run the usb-eject hook with stdout redirect to the web-ui
+  * add the complementary usb-eject.js to the browser
+
+"""
+
+
+def main():
+    try:
+        label = sys.argv[1].split(":usb-eject:")[1].split(":")[0]
+        mp = "/run/media/ed/" + label
+        # print("ejecting [%s]... " % (mp,), end="")
+        mp = os.path.abspath(os.path.realpath(mp.encode("utf-8")))
+        st = os.lstat(mp)
+        if not stat.S_ISDIR(st.st_mode):
+            raise Exception("not a regular directory")
+
+        cmd = [b"gio", b"mount", b"-e", mp]
+        print(sp.check_output(cmd).decode("utf-8", "replace").strip())
+
+    except Exception as ex:
+        print("unmount failed: %r" % (ex,))
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/README.md

@@ -31,6 +31,9 @@ plugins in this section should only be used with appropriate precautions:
 * [very-bad-idea.py](./very-bad-idea.py) combined with [meadup.js](https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/meadup.js) converts copyparty into a janky yet extremely flexible chromecast clone
   * also adds a virtual keyboard by @steinuil to the basic-upload tab for comfy couch crowd control
   * anything uploaded through the [android app](https://github.com/9001/party-up) (files or links) are executed on the server, meaning anyone can infect your PC with malware... so protect this with a password and keep it on a LAN!
+  * [kamelåså](https://github.com/steinuil/kameloso) is a much better (and MUCH safer) alternative to this plugin
+    * powered by [chicken-curry-banana-pineapple-peanut pizza](https://a.ocv.me/pub/g/i/2025/01/298437ce-8351-4c8c-861c-fa131d217999.jpg?cache) so you know it's good
+    * and, unlike this plugin, kamelåså even has windows support (nice)
 
 
 # dependencies

bin/mtag/very-bad-idea.py

@@ -6,6 +6,11 @@ WARNING -- DANGEROUS PLUGIN --
   running this plugin, they can execute malware on your machine
   so please keep this on a LAN and protect it with a password
 
+here is a MUCH BETTER ALTERNATIVE (which also works on Windows):
+  https://github.com/steinuil/kameloso
+
+----------------------------------------------------------------------
+
 use copyparty as a chromecast replacement:
   * post a URL and it will open in the default browser
   * upload a file and it will open in the default application

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "2.7"
+S_VERSION = "2.8"
-S_BUILD_DT = "2024-12-06"
+S_BUILD_DT = "2025-01-21"
 
 """
 u2c.py: upload to copyparty
@@ -1103,7 +1103,7 @@ class Ctl(object):
             nleft = self.nfiles - self.up_f
             tail = "\033[K\033[u" if VT100 and not self.ar.ns else "\r"
 
-            t = "%s eta @ %s/s, %s, %d# left\033[K" % (self.eta, spd, sleft, nleft)
+            t = "%s eta @ %s/s, %s, %d# left" % (self.eta, spd, sleft, nleft)
             if not self.hash_b:
                 t = " now hashing..."
             eprint(txt + "\033]0;{0}\033\\\r{0}{1}".format(t, tail))
@@ -1249,7 +1249,7 @@ class Ctl(object):
                         for n, zsii in enumerate(file.cids)
                     ]
                     print("chs: %s\n%s" % (vp, "\n".join(zsl)))
-                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.kchunks]
+                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.cids]
                 zb = hashlib.sha512("\n".join(zsl).encode("utf-8")).digest()[:33]
                 wark = ub64enc(zb).decode("utf-8")
                 if self.ar.jw:

bin/zmq-recv.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+
+import sys
+import zmq
+
+"""
+zmq-recv.py: demo zmq receiver
+2025-01-22, v1.0, ed <irc.rizon.net>, MIT-Licensed
+https://github.com/9001/copyparty/blob/hovudstraum/bin/zmq-recv.py
+
+basic zmq-server to receive events from copyparty; try one of
+the below and then "send a message to serverlog" in the web-ui:
+
+1) dumb fire-and-forget to any and all listeners;
+run this script with "sub" and run copyparty with this:
+  --xm zmq:pub:tcp://*:5556
+
+2) one lucky listener gets the message, blocks if no listeners:
+run this script with "pull" and run copyparty with this:
+  --xm t3,zmq:push:tcp://*:5557
+
+3) blocking syn/ack mode, client must ack each message;
+run this script with "rep" and run copyparty with this:
+  --xm t3,zmq:req:tcp://localhost:5555
+
+note: to conditionally block uploads based on message contents,
+use rep_server to answer with "return 1" and run copyparty with
+  --xau t3,c,zmq:req:tcp://localhost:5555
+"""
+
+
+ctx = zmq.Context()
+
+
+def sub_server():
+    # PUB/SUB allows any number of servers/clients, and
+    # messages are fire-and-forget
+    sck = ctx.socket(zmq.SUB)
+    sck.connect("tcp://localhost:5556")
+    sck.setsockopt_string(zmq.SUBSCRIBE, "")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def pull_server():
+    # PUSH/PULL allows any number of servers/clients, and
+    # each message is sent to a exactly one PULL client
+    sck = ctx.socket(zmq.PULL)
+    sck.connect("tcp://localhost:5557")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def rep_server():
+    # REP/REQ is a server/client pair where each message must be
+    # acked by the other before another message can be sent, so
+    # copyparty will do a blocking-wait for the ack
+    sck = ctx.socket(zmq.REP)
+    sck.bind("tcp://*:5555")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+        reply = b"thx"
+        # reply = b"return 1"  # non-zero to block an upload
+        sck.send(reply)
+
+
+mode = sys.argv[1].lower() if len(sys.argv) > 1 else ""
+
+if mode == "sub":
+    sub_server()
+elif mode == "pull":
+    pull_server()
+elif mode == "rep":
+    rep_server()
+else:
+    print("specify mode as first argument:  SUB | PULL | REP")

contrib/README.md

@@ -12,14 +12,19 @@
 * assumes the webserver and copyparty is running on the same server/IP
 * modify `10.13.1.1` as necessary if you wish to support browsers without javascript
 
-### [`sharex.sxcu`](sharex.sxcu)
+### [`sharex.sxcu`](sharex.sxcu) - Windows screenshot uploader
-* sharex config file to upload screenshots and grab the URL
+* [sharex](https://getsharex.com/) config file to upload screenshots and grab the URL
 * `RequestURL`: full URL to the target folder
 * `pw`: password (remove the `pw` line if anon-write)
 * the `act:bput` thing is optional since copyparty v1.9.29
 * using an older sharex version, maybe sharex v12.1.1 for example? dw fam i got your back 👉😎👉 [`sharex12.sxcu`](sharex12.sxcu)
 
-### [`flameshot.sh`](flameshot.sh)
+### [`ishare.iscu`](ishare.iscu) - MacOS screenshot uploader
+* [ishare](https://isharemac.app/) config file to upload screenshots and grab the URL
+* `RequestURL`: full URL to the target folder
+* `pw`: password (remove the `pw` line if anon-write)
+
+### [`flameshot.sh`](flameshot.sh) - Linux screenshot uploader
 * takes a screenshot with [flameshot](https://flameshot.org/) on Linux, uploads it, and writes the URL to clipboard
 
 ### [`send-to-cpp.contextlet.json`](send-to-cpp.contextlet.json)
@@ -53,5 +58,10 @@ init-scripts to start copyparty as a service
 * [`openrc/copyparty`](openrc/copyparty)
 
 # Reverse-proxy
-copyparty has basic support for running behind another webserver
+copyparty supports running behind another webserver
-* [`nginx/copyparty.conf`](nginx/copyparty.conf)
+* [`apache/copyparty.conf`](apache/copyparty.conf)
+* [`haproxy/copyparty.conf`](haproxy/copyparty.conf)
+* [`lighttpd/subdomain.conf`](lighttpd/subdomain.conf)
+* [`lighttpd/subpath.conf`](lighttpd/subpath.conf)
+* [`nginx/copyparty.conf`](nginx/copyparty.conf) -- recommended
+* [`traefik/copyparty.yaml`](traefik/copyparty.yaml)

contrib/apache/copyparty.conf

@@ -1,14 +1,29 @@
-# when running copyparty behind a reverse proxy,
+# if you would like to use unix-sockets (recommended),
-# the following arguments are recommended:
+# you must run copyparty with one of the following:
 #
-#   -i 127.0.0.1    only accept connections from nginx
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
 #
 # if you are doing location-based proxying (such as `/stuff` below)
 # you must run copyparty with --rp-loc=stuff
 #
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 
+
 LoadModule proxy_module modules/mod_proxy.so
-ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"
+
-# do not specify ProxyPassReverse
 RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
+# NOTE: do not specify ProxyPassReverse
+
+
+##
+## then, enable one of the below:
+
+# use subdomain proxying to unix-socket (best)
+ProxyPass "/" "unix:///dev/shm/party.sock|http://whatever/"
+
+# use subdomain proxying to 127.0.0.1 (slower)
+#ProxyPass "/" "http://127.0.0.1:3923/"
+
+# use subpath proxying to 127.0.0.1 (slow and maybe buggy)
+#ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"

contrib/haproxy/copyparty.conf

@@ -0,0 +1,24 @@
+# this config is essentially two separate examples;
+#
+#   foo1 connects to copyparty using tcp, and
+#   foo2 uses unix-sockets for 27% higher performance
+#
+# to use foo2 you must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+
+defaults
+  mode http
+  option forwardfor
+  timeout connect 1s
+  timeout client 610s
+  timeout server 610s
+
+listen foo1
+  bind *:8081
+  server srv1 127.0.0.1:3923 maxconn 512
+
+listen foo2
+  bind *:8082
+  server srv1 /dev/shm/party.sock maxconn 512

contrib/ishare.iscu

@@ -0,0 +1,10 @@
+{
+  "Name": "copyparty",
+  "RequestURL": "http://127.0.0.1:3923/screenshots/",
+  "Headers": {
+    "pw": "PUT_YOUR_PASSWORD_HERE_MY_DUDE",
+    "accept": "json"
+  },
+  "FileFormName": "f",
+  "ResponseURL": "{{fileurl}}"
+}

contrib/lighttpd/subdomain.conf

@@ -0,0 +1,24 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subdomain.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+proxy.forwarded = ( "for" => 1, "proto" => 1 )
+proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+# if you really need to use tcp instead of unix-sockets, do this instead:
+#proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )

contrib/lighttpd/subpath.conf

@@ -0,0 +1,31 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subpath.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# also since this example proxies a subpath instead of the
+# recommended subdomain-proxying, we must also specify this:
+#
+#   --rp-loc files
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+$HTTP["url"] =~ "^/files" {
+    proxy.forwarded = ( "for" => 1, "proto" => 1 )
+    proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+    # if you really need to use tcp instead of unix-sockets, do this instead:
+    #proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )
+}

contrib/nginx/copyparty.conf

@@ -36,9 +36,9 @@ upstream cpp_uds {
 	# but there must be at least one unix-group which both
 	# nginx and copyparty is a member of; if that group is
 	# "www" then run copyparty with the following args:
-	# -i unix:770:www:/tmp/party.sock
+	# -i unix:770:www:/dev/shm/party.sock
 
-	server unix:/tmp/party.sock fail_timeout=1s;
+	server unix:/dev/shm/party.sock fail_timeout=1s;
 	keepalive 1;
 }
 
@@ -61,6 +61,10 @@ server {
 		client_max_body_size 0;
 		proxy_buffering off;
 		proxy_request_buffering off;
+		# improve download speed from 600 to 1500 MiB/s
+		proxy_buffers 32 8k;
+		proxy_buffer_size 16k;
+		proxy_busy_buffers_size 24k;
 
 		proxy_set_header   Host              $host;
 		proxy_set_header   X-Real-IP         $remote_addr;

contrib/package/arch/PKGBUILD

@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.16.5"
+pkgver="1.16.9"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -16,12 +16,13 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
             "libkeyfinder-git: detection of musical keys" 
             "qm-vamp-plugins: BPM detection" 
             "python-pyopenssl: ftps functionality" 
-            "python-argon2_cffi: hashed passwords in config" 
+            "python-pyzmq: send zeromq messages from event-hooks" 
+            "python-argon2-cffi: hashed passwords in config" 
             "python-impacket-git: smb support (bad idea)"
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("2830086bd872aaa5174c2ca73ba395439e85c883d85438263bd89521c5f37d9c")
+sha256sums=("3e8f3c24c699aa41e0d51db6d781e453979c77abc34c919063b5bddd64d27bb0")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/default.nix

@@ -1,4 +1,4 @@
-{ lib, stdenv, makeWrapper, fetchurl, utillinux, python, jinja2, impacket, pyftpdlib, pyopenssl, argon2-cffi, pillow, pyvips, ffmpeg, mutagen,
+{ lib, stdenv, makeWrapper, fetchurl, utillinux, python, jinja2, impacket, pyftpdlib, pyopenssl, argon2-cffi, pillow, pyvips, pyzmq, ffmpeg, mutagen,
 
 # use argon2id-hashed passwords in config files (sha2 is always available)
 withHashedPasswords ? true,
@@ -21,6 +21,9 @@ withMediaProcessing ? true,
 # if MediaProcessing is not enabled, you probably want this instead (less accurate, but much safer and faster)
 withBasicAudioMetadata ? false,
 
+# send ZeroMQ messages from event-hooks
+withZeroMQ ? true,
+
 # enable FTPS support in the FTP server
 withFTPS ? false,
 
@@ -43,6 +46,7 @@ let
     ++ lib.optional withMediaProcessing ffmpeg
     ++ lib.optional withBasicAudioMetadata mutagen
     ++ lib.optional withHashedPasswords argon2-cffi
+    ++ lib.optional withZeroMQ pyzmq
     );
 in stdenv.mkDerivation {
   pname = "copyparty";

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.16.5/copyparty-sfx.py",
+    "url": "https://github.com/9001/copyparty/releases/download/v1.16.9/copyparty-sfx.py",
-    "version": "1.16.5",
+    "version": "1.16.9",
-    "hash": "sha256-rfZ76ujA6bLXYW52qP2pK8gdwDdED91mLOF2gzquG8E="
+    "hash": "sha256-456L3IHzf8ups3L9pTBZJMQjML8AlsQI66HZohDyEIA="
 }

contrib/traefik/copyparty.yaml

@@ -0,0 +1,12 @@
+# ./traefik --experimental.fastproxy=true --entrypoints.web.address=:8080 --providers.file.filename=copyparty.yaml
+
+http:
+  services:
+    service-cpp:
+      loadBalancer:
+        servers:
+          - url: "http://127.0.0.1:3923/"
+  routers:
+    my-router:
+      rule: "PathPrefix(`/`)"
+      service: service-cpp

copyparty/__main__.py

@@ -54,6 +54,8 @@ from .util import (
     RAM_TOTAL,
     SQLITE_VER,
     UNPLICATIONS,
+    URL_BUG,
+    URL_PRJ,
     Daemon,
     align_tab,
     ansi_re,
@@ -332,17 +334,16 @@ def ensure_webdeps() -> None:
     if has_resource(E, "web/deps/mini-fa.woff"):
         return
 
-    warn(
+    t = """could not find webdeps;
-        """could not find webdeps;
   if you are running the sfx, or exe, or pypi package, or docker image,
   then this is a bug! Please let me know so I can fix it, thanks :-)
-  https://github.com/9001/copyparty/issues/new?labels=bug&template=bug_report.md
+  %s
 
   however, if you are a dev, or running copyparty from source, and you want
   full client functionality, you will need to build or obtain the webdeps:
-  https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#building
+  %s/blob/hovudstraum/docs/devnotes.md#building
     """
-    )
+    warn(t % (URL_BUG, URL_PRJ))
 
 
 def configure_ssl_ver(al: argparse.Namespace) -> None:
@@ -739,6 +740,10 @@ def get_sects():
               the \033[33m,,\033[35m stops copyparty from reading the rest as flags and
               the \033[33m--\033[35m stops notify-send from reading the message as args
               and the alert will be "hey" followed by the messagetext
+
+             \033[36m--xau zmq:pub:tcp://*:5556\033[35m announces uploads on zeromq;
+             \033[36m--xau t3,zmq:push:tcp://*:5557\033[35m also works, and you can
+             \033[36m--xau t3,j,zmq:req:tcp://localhost:5555\033[35m too for example
             \033[0m
             each hook is executed once for each event, except for \033[36mxiu\033[0m
             which builds up a backlog of uploads, running the hook just once
@@ -770,11 +775,22 @@ def get_sects():
             values for --urlform:
               \033[36mstash\033[35m dumps the data to file and returns length + checksum
               \033[36msave,get\033[35m dumps to file and returns the page like a GET
-              \033[36mprint,get\033[35m prints the data in the log and returns GET
+              \033[36mprint    \033[35m prints the data to log and returns an error
-              (leave out the ",get" to return an error instead)\033[0m
+              \033[36mprint,xm \033[35m prints the data to log and returns --xm output
+              \033[36mprint,get\033[35m prints the data to log and returns GET\033[0m
 
-            note that the \033[35m--xm\033[0m hook will only run if \033[35m--urlform\033[0m
+            note that the \033[35m--xm\033[0m hook will only run if \033[35m--urlform\033[0m is
-              is either \033[36mprint\033[0m or the default \033[36mprint,get\033[0m
+              either \033[36mprint\033[0m or \033[36mprint,get\033[0m or the default \033[36mprint,xm\033[0m
+
+            if an \033[35m--xm\033[0m hook returns text, then
+              the response code will be HTTP 202;
+              http/get responses will be HTTP 200
+
+            if there are multiple \033[35m--xm\033[0m hooks defined, then
+              the first hook that produced output is returned
+
+            if there are no \033[35m--xm\033[0m hooks defined, then the default
+              \033[36mprint,xm\033[0m behaves like \033[36mprint,get\033[0m (returning html)
             """
             ),
         ],
@@ -955,7 +971,7 @@ def add_general(ap, nc, srvname):
     ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m], see --help-accounts")
     ap2.add_argument("--grp", metavar="G:N,N", type=u, action="append", help="add group, \033[33mNAME\033[0m:\033[33mUSER1\033[0m,\033[33mUSER2\033[0m,\033[33m...\033[0m; example [\033[32madmins:ed,foo,bar\033[0m]")
     ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files (volflag=dots)")
-    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,xm", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
     ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="server terminal title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
     ap2.add_argument("--name", metavar="TXT", type=u, default=srvname, help="server name (displayed topleft in browser and in mDNS)")
     ap2.add_argument("--mime", metavar="EXT=MIME", type=u, action="append", help="map file \033[33mEXT\033[0mension to \033[33mMIME\033[0mtype, for example [\033[32mjpg=image/jpeg\033[0m]")
@@ -1328,6 +1344,7 @@ def add_admin(ap):
     ap2.add_argument("--no-ups-page", action="store_true", help="disable ?ru (list of recent uploads)")
     ap2.add_argument("--no-up-list", action="store_true", help="don't show list of incoming files in controlpanel")
     ap2.add_argument("--dl-list", metavar="LVL", type=int, default=2, help="who can see active downloads in the controlpanel? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=everyone")
+    ap2.add_argument("--ups-who", metavar="LVL", type=int, default=2, help="who can see recent uploads on the ?ru page? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=everyone (volflag=ups_who)")
     ap2.add_argument("--ups-when", action="store_true", help="let everyone see upload timestamps on the ?ru page, not just admins")
 
 
@@ -1368,6 +1385,8 @@ def add_transcoding(ap):
     ap2 = ap.add_argument_group('transcoding options')
     ap2.add_argument("--q-opus", metavar="KBPS", type=int, default=128, help="target bitrate for transcoding to opus; set 0 to disable")
     ap2.add_argument("--q-mp3", metavar="QUALITY", type=u, default="q2", help="target quality for transcoding to mp3, for example [\033[32m192k\033[0m] (CBR) or [\033[32mq0\033[0m] (CQ/CRF, q0=maxquality, q9=smallest); set 0 to disable")
+    ap2.add_argument("--no-caf", action="store_true", help="disable transcoding to caf-opus (affects iOS v12~v17), will use mp3 instead")
+    ap2.add_argument("--no-owa", action="store_true", help="disable transcoding to webm-opus (iOS v18 and later), will use mp3 instead")
     ap2.add_argument("--no-acode", action="store_true", help="disable audio transcoding")
     ap2.add_argument("--no-bacode", action="store_true", help="disable batch audio transcoding by folder download (zip/tar)")
     ap2.add_argument("--ac-maxage", metavar="SEC", type=int, default=86400, help="delete cached transcode output after \033[33mSEC\033[0m seconds")
@@ -1476,12 +1495,14 @@ def add_ui(ap, retry):
     ap2.add_argument("--txt-max", metavar="KiB", type=int, default=64, help="max size of embedded textfiles on ?doc= (anything bigger will be lazy-loaded by JS)")
     ap2.add_argument("--doctitle", metavar="TXT", type=u, default="copyparty @ --name", help="title / service-name to show in html documents")
     ap2.add_argument("--bname", metavar="TXT", type=u, default="--name", help="server name (displayed in filebrowser document title)")
-    ap2.add_argument("--pb-url", metavar="URL", type=u, default="https://github.com/9001/copyparty", help="powered-by link; disable with \033[33m-np\033[0m")
+    ap2.add_argument("--pb-url", metavar="URL", type=u, default=URL_PRJ, help="powered-by link; disable with \033[33m-np\033[0m")
     ap2.add_argument("--ver", action="store_true", help="show version on the control panel (incompatible with \033[33m-nb\033[0m)")
     ap2.add_argument("--k304", metavar="NUM", type=int, default=0, help="configure the option to enable/disable k304 on the controlpanel (workaround for buggy reverse-proxies); [\033[32m0\033[0m] = hidden and default-off, [\033[32m1\033[0m] = visible and default-off, [\033[32m2\033[0m] = visible and default-on")
     ap2.add_argument("--no304", metavar="NUM", type=int, default=0, help="configure the option to enable/disable no304 on the controlpanel (workaround for buggy caching in browsers); [\033[32m0\033[0m] = hidden and default-off, [\033[32m1\033[0m] = visible and default-off, [\033[32m2\033[0m] = visible and default-on")
-    ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#attr-sandbox")
+    ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to allow in the iframe 'sandbox' attribute for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#sandbox")
-    ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for prologue/epilogue docs  (volflag=lg_sbf)")
+    ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to allow in the iframe 'sandbox' attribute for prologue/epilogue docs (volflag=lg_sbf)")
+    ap2.add_argument("--md-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for README.md docs, for example [\033[32mfullscreen\033[0m] (volflag=md_sba)")
+    ap2.add_argument("--lg-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for prologue/epilogue docs (volflag=lg_sba); see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes")
     ap2.add_argument("--no-sb-md", action="store_true", help="don't sandbox README/PREADME.md documents (volflags: no_sb_md | sb_md)")
     ap2.add_argument("--no-sb-lg", action="store_true", help="don't sandbox prologue/epilogue docs (volflags: no_sb_lg | sb_lg); enables non-js support")
 

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 16, 6)
+VERSION = (1, 16, 10)
 CODENAME = "COPYparty"
-BUILD_DT = (2024, 12, 19)
+BUILD_DT = (2025, 1, 25)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -1832,7 +1832,11 @@ class AuthSrv(object):
             if fka and not fk:
                 fk = fka
             if fk:
-                vol.flags["fk"] = int(fk) if fk is not True else 8
+                fk = 8 if fk is True else int(fk)
+                if fk > 72:
+                    t = "max filekey-length is 72; volume /%s specified %d (anything higher than 16 is pointless btw)"
+                    raise Exception(t % (vol.vpath, fk))
+                vol.flags["fk"] = fk
                 have_fk = True
 
             dk = vol.flags.get("dk")
@@ -2181,11 +2185,11 @@ class AuthSrv(object):
             if not self.args.no_voldump:
                 self.log(t)
 
-            if have_e2d:
+            if have_e2d or self.args.idp_h_usr:
                 t = self.chk_sqlite_threadsafe()
                 if t:
                     self.log("\n\033[{}\033[0m\n".format(t))
-
+            if have_e2d:
                 if not have_e2t:
                     t = "hint: enable multimedia indexing (artist/title/...) with argument -e2ts"
                     self.log(t, 6)
@@ -2339,6 +2343,7 @@ class AuthSrv(object):
                 "frand": bool(vf.get("rand")),
                 "lifetime": vf.get("lifetime") or 0,
                 "unlist": vf.get("unlist") or "",
+                "sb_lg": "" if "no_sb_lg" in vf else (vf.get("lg_sbf") or "y"),
             }
             js_htm = {
                 "s_name": self.args.bname,
@@ -2351,6 +2356,8 @@ class AuthSrv(object):
                 "have_unpost": int(self.args.unpost),
                 "have_emp": self.args.emp,
                 "sb_md": "" if "no_sb_md" in vf else (vf.get("md_sbf") or "y"),
+                "sba_md": vf.get("md_sba") or "",
+                "sba_lg": vf.get("lg_sba") or "",
                 "txt_ext": self.args.textfiles.replace(",", " "),
                 "def_hcols": list(vf.get("mth") or []),
                 "unlist0": vf.get("unlist") or "",

copyparty/cfg.py

@@ -74,6 +74,8 @@ def vf_vmap() -> dict[str, str]:
         "html_head",
         "lg_sbf",
         "md_sbf",
+        "lg_sba",
+        "md_sba",
         "nrand",
         "og_desc",
         "og_site",
@@ -91,6 +93,7 @@ def vf_vmap() -> dict[str, str]:
         "unlist",
         "u2abort",
         "u2ts",
+        "ups_who",
     ):
         ret[k] = k
     return ret
@@ -144,6 +147,7 @@ flagcats = {
         "noclone": "take dupe data from clients, even if available on HDD",
         "nodupe": "rejects existing files (instead of linking/cloning them)",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
+        "nosparse": "deny use of sparse files, mainly for slow storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
         "nosub": "forces all uploads into the top folder of the vfs",
         "magic": "enables filetype detection for nameless uploads",
@@ -240,6 +244,8 @@ flagcats = {
         "sb_lg": "enable js sandbox for prologue/epilogue (default)",
         "md_sbf": "list of markdown-sandbox safeguards to disable",
         "lg_sbf": "list of *logue-sandbox safeguards to disable",
+        "md_sba": "value of iframe allow-prop for markdown-sandbox",
+        "lg_sba": "value of iframe allow-prop for *logue-sandbox",
         "nohtml": "return html and markdown as text/html",
     },
     "others": {

copyparty/dxml.py

@@ -1,3 +1,6 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
 import importlib
 import sys
 import xml.etree.ElementTree as ET
@@ -8,6 +11,10 @@ if True:  # pylint: disable=using-constant-test
     from typing import Any, Optional
 
 
+class BadXML(Exception):
+    pass
+
+
 def get_ET() -> ET.XMLParser:
     pn = "xml.etree.ElementTree"
     cn = "_elementtree"
@@ -34,7 +41,7 @@ def get_ET() -> ET.XMLParser:
 XMLParser: ET.XMLParser = get_ET()
 
 
-class DXMLParser(XMLParser):  # type: ignore
+class _DXMLParser(XMLParser):  # type: ignore
     def __init__(self) -> None:
         tb = ET.TreeBuilder()
         super(DXMLParser, self).__init__(target=tb)
@@ -49,8 +56,12 @@ class DXMLParser(XMLParser):  # type: ignore
         raise BadXML("{}, {}".format(a, ka))
 
 
-class BadXML(Exception):
+class _NG(XMLParser):  # type: ignore
-    pass
+    def __int__(self) -> None:
+        raise BadXML("dxml selftest failed")
+
+
+DXMLParser = _DXMLParser
 
 
 def parse_xml(txt: str) -> ET.Element:
@@ -59,6 +70,40 @@ def parse_xml(txt: str) -> ET.Element:
     return parser.close()  # type: ignore
 
 
+def selftest() -> bool:
+    qbe = r"""<!DOCTYPE d [
+<!ENTITY a "nice_bakuretsu">
+]>
+<root>&a;&a;&a;</root>"""
+
+    emb = r"""<!DOCTYPE d [
+<!ENTITY a SYSTEM "file:///etc/hostname">
+]>
+<root>&a;</root>"""
+
+    # future-proofing; there's never been any known vulns
+    # regarding DTDs and ET.XMLParser, but might as well
+    # block them since webdav-clients don't use them
+    dtd = r"""<!DOCTYPE d SYSTEM "a.dtd">
+<root>a</root>"""
+
+    for txt in (qbe, emb, dtd):
+        try:
+            parse_xml(txt)
+            t = "WARNING: dxml selftest failed:\n%s\n"
+            print(t % (txt,), file=sys.stderr)
+            return False
+        except BadXML:
+            pass
+
+    return True
+
+
+DXML_OK = selftest()
+if not DXML_OK:
+    DXMLParser = _NG
+
+
 def mktnod(name: str, text: str) -> ET.Element:
     el = ET.Element(name)
     el.text = text

copyparty/httpcli.py

@@ -132,6 +132,8 @@ NO_CACHE = {"Cache-Control": "no-cache"}
 
 ALL_COOKIES = "k304 no304 js idxh dots cppwd cppws".split()
 
+BADXFF = " due to dangerous misconfiguration (the http-header specified by --xff-hdr was received from an untrusted reverse-proxy)"
+
 H_CONN_KEEPALIVE = "Connection: Keep-Alive"
 H_CONN_CLOSE = "Connection: Close"
 
@@ -145,6 +147,14 @@ A_FILE = os.stat_result(
     (0o644, -1, -1, 1, 1000, 1000, 8, 0x39230101, 0x39230101, 0x39230101)
 )
 
+RE_CC = re.compile(r"[\x00-\x1f]")  # search always faster
+RE_HSAFE = re.compile(r"[\x00-\x1f<>\"'&]")  # search always much faster
+RE_HOST = re.compile(r"[^][0-9a-zA-Z.:_-]")  # search faster <=17ch
+RE_MHOST = re.compile(r"^[][0-9a-zA-Z.:_-]+$")  # match faster >=18ch
+RE_K = re.compile(r"[^0-9a-zA-Z_-]")  # search faster <=17ch
+
+UPARAM_CC_OK = set("doc move tree".split())
+
 
 class HttpCli(object):
     """
@@ -154,6 +164,8 @@ class HttpCli(object):
     def __init__(self, conn: "HttpConn") -> None:
         assert conn.sr  # !rm
 
+        empty_stringlist: list[str] = []
+
         self.t0 = time.time()
         self.conn = conn
         self.u2mutex = conn.u2mutex  # mypy404
@@ -199,9 +211,7 @@ class HttpCli(object):
         self.trailing_slash = True
         self.uname = " "
         self.pw = " "
-        self.rvol = [" "]
+        self.rvol = self.wvol = self.avol = empty_stringlist
-        self.wvol = [" "]
-        self.avol = [" "]
         self.do_log = True
         self.can_read = False
         self.can_write = False
@@ -382,6 +392,7 @@ class HttpCli(object):
                     ) + "0.0/16"
                     zs2 = ' or "--xff-src=lan"' if self.conn.xff_lan.map(pip) else ""
                     self.log(t % (self.args.xff_hdr, pip, cli_ip, zso, zs, zs2), 3)
+                    self.bad_xff = True
                 else:
                     self.ip = cli_ip
                     self.is_vproxied = bool(self.args.R)
@@ -389,6 +400,15 @@ class HttpCli(object):
                     self.host = self.headers.get("x-forwarded-host") or self.host
                     trusted_xff = True
 
+        m = RE_HOST.search(self.host)
+        if m and self.host != self.args.name:
+            zs = self.host
+            t = "malicious user; illegal Host header; req(%r) host(%r) => %r"
+            self.log(t % (self.req, zs, zs[m.span()[0] :]), 1)
+            self.cbonk(self.conn.hsrv.gmal, zs, "bad_host", "illegal Host header")
+            self.terse_reply(b"illegal Host header", 400)
+            return False
+
         if self.is_banned():
             return False
 
@@ -434,6 +454,16 @@ class HttpCli(object):
                 self.loud_reply(t, status=400)
                 return False
 
+        ptn_cc = RE_CC
+        m = ptn_cc.search(self.req)
+        if m:
+            zs = self.req
+            t = "malicious user; Cc in req0 %r => %r"
+            self.log(t % (zs, zs[m.span()[0] :]), 1)
+            self.cbonk(self.conn.hsrv.gmal, zs, "cc_r0", "Cc in req0")
+            self.terse_reply(b"", 500)
+            return False
+
         # split req into vpath + uparam
         uparam = {}
         if "?" not in self.req:
@@ -446,8 +476,8 @@ class HttpCli(object):
             self.trailing_slash = vpath.endswith("/")
             vpath = undot(vpath)
 
-            ptn = self.conn.hsrv.ptn_cc
+            re_k = RE_K
-            k_safe = self.conn.hsrv.uparam_cc_ok
+            k_safe = UPARAM_CC_OK
             for k in arglist.split("&"):
                 if "=" in k:
                     k, zs = k.split("=", 1)
@@ -457,6 +487,14 @@ class HttpCli(object):
                 else:
                     sv = ""
 
+                m = re_k.search(k)
+                if m:
+                    t = "malicious user; bad char in query key; req(%r) qk(%r) => %r"
+                    self.log(t % (self.req, k, k[m.span()[0] :]), 1)
+                    self.cbonk(self.conn.hsrv.gmal, self.req, "bc_q", "illegal qkey")
+                    self.terse_reply(b"", 500)
+                    return False
+
                 k = k.lower()
                 uparam[k] = sv
 
@@ -464,17 +502,26 @@ class HttpCli(object):
                     continue
 
                 zs = "%s=%s" % (k, sv)
-                m = ptn.search(zs)
+                m = ptn_cc.search(zs)
                 if not m:
                     continue
 
-                hit = zs[m.span()[0] :]
+                t = "malicious user; Cc in query; req(%r) qp(%r) => %r"
-                t = "malicious user; Cc in query [{}] => [{!r}]"
+                self.log(t % (self.req, zs, zs[m.span()[0] :]), 1)
-                self.log(t.format(self.req, hit), 1)
                 self.cbonk(self.conn.hsrv.gmal, self.req, "cc_q", "Cc in query")
                 self.terse_reply(b"", 500)
                 return False
 
+            if "k" in uparam:
+                m = re_k.search(uparam["k"])
+                if m:
+                    zs = uparam["k"]
+                    t = "malicious user; illegal filekey; req(%r) k(%r) => %r"
+                    self.log(t % (self.req, zs, zs[m.span()[0] :]), 1)
+                    self.cbonk(self.conn.hsrv.gmal, zs, "bad_k", "illegal filekey")
+                    self.terse_reply(b"illegal filekey", 400)
+                    return False
+
         if self.is_vproxied:
             if vpath.startswith(self.args.R):
                 vpath = vpath[len(self.args.R) + 1 :]
@@ -518,7 +565,7 @@ class HttpCli(object):
             return self.tx_qr()
 
         if relchk(self.vpath) and (self.vpath != "*" or self.mode != "OPTIONS"):
-            self.log("invalid relpath %r" % ("/" + self.vpath,))
+            self.log("illegal relpath; req(%r) => %r" % (self.req, "/" + self.vpath))
             self.cbonk(self.conn.hsrv.gmal, self.req, "bad_vp", "invalid relpaths")
             return self.tx_404() and self.keepalive
 
@@ -871,12 +918,12 @@ class HttpCli(object):
         for k, zs in list(self.out_headers.items()) + self.out_headerlist:
             response.append("%s: %s" % (k, zs))
 
+        ptn_cc = RE_CC
         for zs in response:
-            m = self.conn.hsrv.ptn_cc.search(zs)
+            m = ptn_cc.search(zs)
             if m:
-                hit = zs[m.span()[0] :]
+                t = "malicious user; Cc in out-hdr; req(%r) hdr(%r) => %r"
-                t = "malicious user; Cc in out-hdr {!r} => [{!r}]"
+                self.log(t % (self.req, zs, zs[m.span()[0] :]), 1)
-                self.log(t.format(zs, hit), 1)
                 self.cbonk(self.conn.hsrv.gmal, zs, "cc_hdr", "Cc in out-hdr")
                 raise Pebkac(999)
 
@@ -1002,7 +1049,7 @@ class HttpCli(object):
         if not kv:
             return ""
 
-        r = ["%s=%s" % (k, quotep(zs)) if zs else k for k, zs in kv.items()]
+        r = ["%s=%s" % (quotep(k), quotep(zs)) if zs else k for k, zs in kv.items()]
         return "?" + "&amp;".join(r)
 
     def ourlq(self) -> str:
@@ -1154,8 +1201,8 @@ class HttpCli(object):
                     return self.tx_res(res_path)
 
             if res_path != undot(res_path):
-                t = "malicious user; attempted path traversal %r => %r"
+                t = "malicious user; attempted path traversal; req(%r) vp(%r) => %r"
-                self.log(t % ("/" + self.vpath, res_path), 1)
+                self.log(t % (self.req, "/" + self.vpath, res_path), 1)
                 self.cbonk(self.conn.hsrv.gmal, self.req, "trav", "path traversal")
 
             self.tx_404()
@@ -1298,8 +1345,8 @@ class HttpCli(object):
 
         pw = self.ouparam.get("pw")
         if pw:
-            q_pw = "?pw=%s" % (pw,)
+            q_pw = "?pw=%s" % (html_escape(pw, True, True),)
-            a_pw = "&pw=%s" % (pw,)
+            a_pw = "&pw=%s" % (html_escape(pw, True, True),)
             for i in hits:
                 i["rp"] += a_pw if "?" in i["rp"] else q_pw
         else:
@@ -1663,7 +1710,7 @@ class HttpCli(object):
 
         token = str(uuid.uuid4())
 
-        if not lk.find(r"./{DAV:}depth"):
+        if lk.find(r"./{DAV:}depth") is None:
             depth = self.headers.get("depth", "infinity")
             lk.append(mktnod("D:depth", depth))
 
@@ -1852,8 +1899,11 @@ class HttpCli(object):
             if "stash" in opt:
                 return self.handle_stash(False)
 
+            xm = []
+            xm_rsp = {}
+
             if "save" in opt:
-                post_sz, _, _, _, path, _ = self.dump_to_file(False)
+                post_sz, _, _, _, _, path, _ = self.dump_to_file(False)
                 self.log("urlform: %d bytes, %r" % (post_sz, path))
             elif "print" in opt:
                 reader, _ = self.get_body_reader()
@@ -1874,7 +1924,7 @@ class HttpCli(object):
                             plain = plain[4:]
                             xm = self.vn.flags.get("xm")
                             if xm:
-                                runhook(
+                                xm_rsp = runhook(
                                     self.log,
                                     self.conn.hsrv.broker,
                                     None,
@@ -1898,6 +1948,13 @@ class HttpCli(object):
                     except Exception as ex:
                         self.log(repr(ex))
 
+            if "xm" in opt:
+                if xm:
+                    self.loud_reply(xm_rsp.get("stdout") or "", status=202)
+                    return True
+                else:
+                    return self.handle_get()
+
             if "get" in opt:
                 return self.handle_get()
 
@@ -1934,11 +1991,11 @@ class HttpCli(object):
         else:
             return read_socket(self.sr, bufsz, remains), remains
 
-    def dump_to_file(self, is_put: bool) -> tuple[int, str, str, int, str, str]:
+    def dump_to_file(self, is_put: bool) -> tuple[int, str, str, str, int, str, str]:
-        # post_sz, sha_hex, sha_b64, remains, path, url
+        # post_sz, halg, sha_hex, sha_b64, remains, path, url
         reader, remains = self.get_body_reader()
         vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
-        rnd, _, lifetime, xbu, xau = self.upload_flags(vfs)
+        rnd, lifetime, xbu, xau = self.upload_flags(vfs)
         lim = vfs.get_dbv(rem)[0].lim
         fdir = vfs.canonical(rem)
         if lim:
@@ -2088,12 +2145,14 @@ class HttpCli(object):
                 # small toctou, but better than clobbering a hardlink
                 wunlink(self.log, path, vfs.flags)
 
+        halg = "sha512"
         hasher = None
         copier = hashcopy
         if "ck" in self.ouparam or "ck" in self.headers:
-            zs = self.ouparam.get("ck") or self.headers.get("ck") or ""
+            halg = zs = self.ouparam.get("ck") or self.headers.get("ck") or ""
             if not zs or zs == "no":
                 copier = justcopy
+                halg = ""
             elif zs == "md5":
                 hasher = hashlib.md5(**USED4SEC)
             elif zs == "sha1":
@@ -2127,7 +2186,7 @@ class HttpCli(object):
                 raise
 
         if self.args.nw:
-            return post_sz, sha_hex, sha_b64, remains, path, ""
+            return post_sz, halg, sha_hex, sha_b64, remains, path, ""
 
         at = mt = time.time() - lifetime
         cli_mt = self.headers.get("x-oc-mtime")
@@ -2238,19 +2297,30 @@ class HttpCli(object):
             self.args.RS + vpath + vsuf,
         )
 
-        return post_sz, sha_hex, sha_b64, remains, path, url
+        return post_sz, halg, sha_hex, sha_b64, remains, path, url
 
     def handle_stash(self, is_put: bool) -> bool:
-        post_sz, sha_hex, sha_b64, remains, path, url = self.dump_to_file(is_put)
+        post_sz, halg, sha_hex, sha_b64, remains, path, url = self.dump_to_file(is_put)
         spd = self._spd(post_sz)
         t = "%s wrote %d/%d bytes to %r  # %s"
         self.log(t % (spd, post_sz, remains, path, sha_b64[:28]))  # 21
 
-        ac = self.uparam.get(
+        mime = "text/plain; charset=utf-8"
-            "want", self.headers.get("accept", "").lower().split(";")[-1]
+        ac = self.uparam.get("want") or self.headers.get("accept") or ""
-        )
+        if ac:
+            ac = ac.split(";", 1)[0].lower()
+            if ac == "application/json":
+                ac = "json"
         if ac == "url":
             t = url
+        elif ac == "json" or "j" in self.uparam:
+            jmsg = {"fileurl": url, "filesz": post_sz}
+            if halg:
+                jmsg[halg] = sha_hex[:56]
+                jmsg["sha_b64"] = sha_b64
+
+            mime = "application/json"
+            t = json.dumps(jmsg, indent=2, sort_keys=True)
         else:
             t = "{}\n{}\n{}\n{}\n".format(post_sz, sha_b64, sha_hex[:56], url)
 
@@ -2260,7 +2330,7 @@ class HttpCli(object):
             h["X-OC-MTime"] = "accepted"
             t = ""  # some webdav clients expect/prefer this
 
-        self.reply(t.encode("utf-8"), 201, headers=h)
+        self.reply(t.encode("utf-8", "replace"), 201, mime=mime, headers=h)
         return True
 
     def bakflip(
@@ -2939,7 +3009,7 @@ class HttpCli(object):
         self.redirect(vpath, "?edit")
         return True
 
-    def upload_flags(self, vfs: VFS) -> tuple[int, bool, int, list[str], list[str]]:
+    def upload_flags(self, vfs: VFS) -> tuple[int, int, list[str], list[str]]:
         if self.args.nw:
             rnd = 0
         else:
@@ -2947,10 +3017,6 @@ class HttpCli(object):
             if vfs.flags.get("rand"):  # force-enable
                 rnd = max(rnd, vfs.flags["nrand"])
 
-        ac = self.uparam.get(
-            "want", self.headers.get("accept", "").lower().split(";")[-1]
-        )
-        want_url = ac == "url"
         zs = self.uparam.get("life", self.headers.get("life", ""))
         if zs:
             vlife = vfs.flags.get("lifetime") or 0
@@ -2960,7 +3026,6 @@ class HttpCli(object):
 
         return (
             rnd,
-            want_url,
             lifetime,
             vfs.flags.get("xbu") or [],
             vfs.flags.get("xau") or [],
@@ -3013,7 +3078,14 @@ class HttpCli(object):
             if not nullwrite:
                 bos.makedirs(fdir_base)
 
-        rnd, want_url, lifetime, xbu, xau = self.upload_flags(vfs)
+        rnd, lifetime, xbu, xau = self.upload_flags(vfs)
+        zs = self.uparam.get("want") or self.headers.get("accept") or ""
+        if zs:
+            zs = zs.split(";", 1)[0].lower()
+            if zs == "application/json":
+                zs = "json"
+        want_url = zs == "url"
+        want_json = zs == "json" or "j" in self.uparam
 
         files: list[tuple[int, str, str, str, str, str]] = []
         # sz, sha_hex, sha_b64, p_file, fname, abspath
@@ -3335,7 +3407,9 @@ class HttpCli(object):
                 msg += "\n" + errmsg
 
             self.reply(msg.encode("utf-8", "replace"), status=sc)
-        elif "j" in self.uparam:
+        elif want_json:
+            if len(jmsg["files"]) == 1:
+                jmsg["fileurl"] = jmsg["files"][0]["url"]
             jtxt = json.dumps(jmsg, indent=2, sort_keys=True).encode("utf-8", "replace")
             self.reply(jtxt, mime="application/json", status=sc)
         else:
@@ -3654,6 +3728,7 @@ class HttpCli(object):
         return logues, readmes
 
     def _expand(self, txt: str, phs: list[str]) -> str:
+        ptn_hsafe = RE_HSAFE
         for ph in phs:
             if ph.startswith("hdr."):
                 sv = str(self.headers.get(ph[4:], ""))
@@ -3671,7 +3746,7 @@ class HttpCli(object):
                 self.log("unknown placeholder in server config: [%s]" % (ph,), 3)
                 continue
 
-            sv = self.conn.hsrv.ptn_hsafe.sub("_", sv)
+            sv = ptn_hsafe.sub("_", sv)
             txt = txt.replace("{{%s}}" % (ph,), sv)
 
         return txt
@@ -4292,7 +4367,7 @@ class HttpCli(object):
             self.log,
             self.asrv,
             fgen,
-            utf8="utf" in uarg,
+            utf8="utf" in uarg or not uarg,
             pre_crc="crc" in uarg,
             cmp=uarg if cancmp or uarg == "pax" else "",
         )
@@ -4506,12 +4581,12 @@ class HttpCli(object):
             else self.conn.hsrv.nm.map(self.ip) or host
         )
         # safer than html_escape/quotep since this avoids both XSS and shell-stuff
-        pw = re.sub(r"[<>&$?`\"']", "_", self.pw or "pw")
+        pw = re.sub(r"[<>&$?`\"']", "_", self.pw or "hunter2")
         vp = re.sub(r"[<>&$?`\"']", "_", self.uparam["hc"] or "").lstrip("/")
         pw = pw.replace(" ", "%20")
         vp = vp.replace(" ", "%20")
         if pw in self.asrv.sesa:
-            pw = "pwd"
+            pw = "hunter2"
 
         html = self.j2s(
             "svcs",
@@ -4941,8 +5016,16 @@ class HttpCli(object):
             and (self.uname in vol.axs.uread or self.uname in vol.axs.upget)
         }
 
+        bad_xff = hasattr(self, "bad_xff")
+        if bad_xff:
+            allvols = []
+            t = "will not return list of recent uploads" + BADXFF
+            self.log(t, 1)
+            if self.avol:
+                raise Pebkac(500, t)
+
         x = self.conn.hsrv.broker.ask(
-            "up2k.get_unfinished_by_user", self.uname, self.ip
+            "up2k.get_unfinished_by_user", self.uname, "" if bad_xff else self.ip
         )
         uret = x.get()
 
@@ -4986,11 +5069,11 @@ class HttpCli(object):
                     ret.sort(key=lambda x: x["at"], reverse=True)  # type: ignore
                     ret = ret[:2000]
 
+        ret.sort(key=lambda x: x["at"], reverse=True)  # type: ignore
+
         if len(ret) > 2000:
             ret = ret[:2000]
 
-        ret.sort(key=lambda x: x["at"], reverse=True)  # type: ignore
-
         for rv in ret:
             rv["vp"] = quotep(rv["vp"])
             nfk = rv.pop("nfk")
@@ -5069,6 +5152,12 @@ class HttpCli(object):
             adm = "*" in vol.axs.uadmin or self.uname in vol.axs.uadmin
             dots = "*" in vol.axs.udot or self.uname in vol.axs.udot
 
+            lvl = int(vol.flags["ups_who"])
+            if not lvl:
+                continue
+            elif lvl == 1 and not adm:
+                continue
+
             n = 1000
             q = "select sz, rd, fn, ip, at from up where at>0 order by at desc"
             for sz, rd, fn, ip, at in cur.execute(q):
@@ -5079,10 +5168,6 @@ class HttpCli(object):
                 if not dots and "/." in vp:
                     continue
 
-                n -= 1
-                if not n:
-                    break
-
                 rv = {
                     "vp": vp,
                     "sz": sz,
@@ -5100,13 +5185,17 @@ class HttpCli(object):
                     ret.sort(key=lambda x: x["at"], reverse=True)  # type: ignore
                     ret = ret[:1000]
 
-        if len(ret) > 1000:
+                n -= 1
-            ret = ret[:1000]
+                if not n:
+                    break
 
         ret.sort(key=lambda x: x["at"], reverse=True)  # type: ignore
 
+        if len(ret) > 1000:
+            ret = ret[:1000]
+
         for rv in ret:
-            rv["evp"] = quotep(rv["vp"])
+            rv["vp"] = quotep(rv["vp"])
             nfk = rv.pop("nfk")
             if not nfk:
                 continue
@@ -5139,15 +5228,16 @@ class HttpCli(object):
             for v in ret:
                 v["vp"] = self.args.SR + v["vp"]
 
-        self.log("%s #%d %.2fsec" % (lm, len(ret), time.time() - t0))
+        now = time.time()
+        self.log("%s #%d %.2fsec" % (lm, len(ret), now - t0))
 
+        ret2 = {"now": int(now), "filter": sfilt, "ups": ret}
+        jtxt = json.dumps(ret2, separators=(",\n", ": "))
         if "j" in self.ouparam:
-            jtxt = json.dumps(ret, separators=(",\n", ": "))
             self.reply(jtxt.encode("utf-8", "replace"), mime="application/json")
             return True
 
-        rows = [[x["vp"], x["evp"], x["sz"], x["ip"], x["at"]] for x in ret]
+        html = self.j2s("rups", this=self, v=jtxt)
-        html = self.j2s("rups", this=self, rows=rows, filt=sfilt, now=int(time.time()))
         self.reply(html.encode("utf-8"), status=200)
         return True
 
@@ -5336,12 +5426,16 @@ class HttpCli(object):
         if self.args.no_del:
             raise Pebkac(403, "the delete feature is disabled in server config")
 
+        unpost = "unpost" in self.uparam
+        if unpost and hasattr(self, "bad_xff"):
+            self.log("unpost was denied" + BADXFF, 1)
+            raise Pebkac(403, "the delete feature is disabled in server config")
+
         if not req:
             req = [self.vpath]
         elif self.is_vproxied:
             req = [x[len(self.args.SR) :] for x in req]
 
-        unpost = "unpost" in self.uparam
         nlim = int(self.uparam.get("lim") or 0)
         lim = [nlim, nlim] if nlim else []
 
@@ -5741,7 +5835,7 @@ class HttpCli(object):
             "taglist": [],
             "have_tags_idx": int(e2t),
             "have_b_u": (self.can_write and self.uparam.get("b") == "u"),
-            "sb_lg": "" if "no_sb_lg" in vf else (vf.get("lg_sbf") or "y"),
+            "sb_lg": vn.js_ls["sb_lg"],
             "url_suf": url_suf,
             "title": html_escape("%s %s" % (self.args.bname, self.vpath), crlf=True),
             "srv_info": srv_infot,

copyparty/httpsrv.py

@@ -195,10 +195,6 @@ class HttpSrv(object):
         self.xff_nm = build_netmap(self.args.xff_src)
         self.xff_lan = build_netmap("lan")
 
-        self.ptn_cc = re.compile(r"[\x00-\x1f]")
-        self.ptn_hsafe = re.compile(r"[\x00-\x1f<>\"'&]")
-        self.uparam_cc_ok = set("doc move tree".split())
-
         self.mallow = "GET HEAD POST PUT DELETE OPTIONS".split()
         if not self.args.no_dav:
             zs = "PROPFIND PROPPATCH LOCK UNLOCK MKCOL COPY MOVE"

copyparty/svchub.py

@@ -50,6 +50,8 @@ from .util import (
     FFMPEG_URL,
     HAVE_PSUTIL,
     HAVE_SQLITE3,
+    HAVE_ZMQ,
+    URL_BUG,
     UTC,
     VERSIONS,
     Daemon,
@@ -60,6 +62,7 @@ from .util import (
     alltrace,
     ansi_re,
     build_netmap,
+    expat_ver,
     load_ipu,
     min_ex,
     mp,
@@ -639,6 +642,7 @@ class SvcHub(object):
             (HAVE_FFPROBE, "ffprobe", t_ff + ", read audio/media tags"),
             (HAVE_MUTAGEN, "mutagen", "read audio tags (ffprobe is better but slower)"),
             (HAVE_ARGON2, "argon2", "secure password hashing (advanced users only)"),
+            (HAVE_ZMQ, "pyzmq", "send zeromq messages from event-hooks"),
             (HAVE_HEIF, "pillow-heif", "read .heif images with pillow (rarely useful)"),
             (HAVE_AVIF, "pillow-avif", "read .avif images with pillow (rarely useful)"),
         ]
@@ -695,6 +699,15 @@ class SvcHub(object):
             if self.args.bauth_last:
                 self.log("root", "WARNING: ignoring --bauth-last due to --no-bauth", 3)
 
+        if not self.args.no_dav:
+            from .dxml import DXML_OK
+
+            if not DXML_OK:
+                if not self.args.no_dav:
+                    self.args.no_dav = True
+                    t = "WARNING:\nDisabling WebDAV support because dxml selftest failed. Please report this bug;\n%s\n...and include the following information in the bug-report:\n%s | expat %s\n"
+                    self.log("root", t % (URL_BUG, VERSIONS, expat_ver()), 1)
+
     def _process_config(self) -> bool:
         al = self.args
 

copyparty/tcpsrv.py

@@ -406,12 +406,12 @@ class TcpSrv(object):
             rem = []
             for k, v in netdevs.items():
                 if k not in self.netdevs:
-                    add.append("\n  added %s = %s" % (k, v))
+                    add.append("\n\033[32m  added %s = %s" % (k, v))
             for k, v in self.netdevs.items():
                 if k not in netdevs:
-                    rem.append("\nremoved %s = %s" % (k, v))
+                    rem.append("\n\033[33mremoved %s = %s" % (k, v))
 
-            t = "network change detected:\033[32m%s\033[33m%s"
+            t = "network change detected:%s%s"
             self.log("tcpsrv", t % ("".join(add), "".join(rem)), 3)
             self.netdevs = netdevs
             self._distribute_netdevs()

copyparty/th_cli.py

@@ -6,7 +6,7 @@ import os
 from .__init__ import TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
-from .th_srv import HAVE_WEBP, thumb_path
+from .th_srv import EXTS_AC, HAVE_WEBP, thumb_path
 from .util import Cooldown
 
 if True:  # pylint: disable=using-constant-test
@@ -57,13 +57,17 @@ class ThumbCli(object):
         if is_vid and "dvthumb" in dbv.flags:
             return None
 
-        want_opus = fmt in ("opus", "caf", "mp3")
+        want_opus = fmt in EXTS_AC
         is_au = ext in self.fmt_ffa
         is_vau = want_opus and ext in self.fmt_ffv
         if is_au or is_vau:
             if want_opus:
                 if self.args.no_acode:
                     return None
+                elif fmt == "caf" and self.args.no_caf:
+                    fmt = "mp3"
+                elif fmt == "owa" and self.args.no_owa:
+                    fmt = "mp3"
             else:
                 if "dathumb" in dbv.flags:
                     return None

copyparty/th_srv.py

@@ -32,7 +32,7 @@ from .util import (
 )
 
 if True:  # pylint: disable=using-constant-test
-    from typing import Optional, Union
+    from typing import Any, Optional, Union
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -46,6 +46,9 @@ HAVE_HEIF = False
 HAVE_AVIF = False
 HAVE_WEBP = False
 
+EXTS_TH = set(["jpg", "webp", "png"])
+EXTS_AC = set(["opus", "owa", "caf", "mp3"])
+
 try:
     if os.environ.get("PRTY_NO_PIL"):
         raise Exception()
@@ -139,7 +142,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
     h = hashlib.sha512(afsenc(fn)).digest()
     fn = ub64enc(h).decode("ascii")[:24]
 
-    if fmt in ("opus", "caf", "mp3"):
+    if fmt in EXTS_AC:
         cat = "ac"
     else:
         fc = fmt[:1]
@@ -334,9 +337,10 @@ class ThumbSrv(object):
                 ap_unpk = abspath
 
             if not bos.path.exists(tpath):
-                want_mp3 = tpath.endswith(".mp3")
+                tex = tpath.rsplit(".", 1)[-1]
-                want_opus = tpath.endswith(".opus") or tpath.endswith(".caf")
+                want_mp3 = tex == "mp3"
-                want_png = tpath.endswith(".png")
+                want_opus = tex in ("opus", "owa", "caf")
+                want_png = tex == "png"
                 want_au = want_mp3 or want_opus
                 for lib in self.args.th_dec:
                     can_au = lib == "ff" and (
@@ -754,25 +758,37 @@ class ThumbSrv(object):
         if "ac" not in tags:
             raise Exception("not audio")
 
-        try:
+        sq = "%dk" % (self.args.q_opus,)
-            dur = tags[".dur"][1]
+        bq = sq.encode("ascii")
-        except:
+        if tags["ac"][1] == "opus":
-            dur = 0
+            enc = "-c:a copy"
+        else:
+            enc = "-c:a libopus -b:a " + sq
 
-        src_opus = abspath.lower().endswith(".opus") or tags["ac"][1] == "opus"
+        fun = self._conv_caf if fmt == "caf" else self._conv_owa
-        want_caf = tpath.endswith(".caf")
-        tmp_opus = tpath
-        if want_caf:
-            tmp_opus = tpath + ".opus"
-            try:
-                wunlink(self.log, tmp_opus, vn.flags)
-            except:
-                pass
 
-        caf_src = abspath if src_opus else tmp_opus
+        fun(abspath, tpath, tags, rawtags, enc, bq, vn)
-        bq = ("%dk" % (self.args.q_opus,)).encode("ascii")
+
+    def _conv_owa(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        if tpath.endswith(".owa"):
+            container = b"webm"
+            tagset = [b"-map_metadata", b"-1"]
+        else:
+            container = b"opus"
+            tagset = self.big_tags(rawtags)
+
+        self.log("conv2 %s [%s]" % (container, enc), 6)
+        benc = enc.encode("ascii").split(b" ")
 
-        if not want_caf or not src_opus:
         # fmt: off
         cmd = [
             b"ffmpeg",
@@ -780,10 +796,50 @@ class ThumbSrv(object):
             b"-v", b"error",
             b"-hide_banner",
             b"-i", fsenc(abspath),
-            ] + self.big_tags(rawtags) + [
+        ] + tagset + [
             b"-map", b"0:a:0",
-                b"-c:a", b"libopus",
+        ] + benc + [
-                b"-b:a", bq,
+            b"-f", container,
+            fsenc(tpath)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, oom=300)
+
+    def _conv_caf(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        tmp_opus = tpath + ".opus"
+        try:
+            wunlink(self.log, tmp_opus, vn.flags)
+        except:
+            pass
+
+        try:
+            dur = tags[".dur"][1]
+        except:
+            dur = 0
+
+        self.log("conv2 caf-tmp [%s]" % (enc,), 6)
+        benc = enc.encode("ascii").split(b" ")
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-map_metadata", b"-1",
+            b"-map", b"0:a:0",
+        ] + benc + [
+            b"-f", b"opus",
             fsenc(tmp_opus)
         ]
         # fmt: on
@@ -794,7 +850,10 @@ class ThumbSrv(object):
         # fix that by mixing in some inaudible pink noise :^)
         # 6.3 sec seems like the cutoff so lets do 7, and
         # 7 sec of psyqui-musou.opus @ 3:50 is 174 KiB
-        if want_caf and (dur < 20 or bos.path.getsize(caf_src) < 256 * 1024):
+        sz = bos.path.getsize(tmp_opus)
+        if dur < 20 or sz < 256 * 1024:
+            zs = bq.decode("ascii")
+            self.log("conv2 caf-transcode; dur=%d sz=%d q=%s" % (dur, sz, zs), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
@@ -813,15 +872,16 @@ class ThumbSrv(object):
             # fmt: on
             self._run_ff(cmd, vn, oom=300)
 
-        elif want_caf:
+        else:
             # simple remux should be safe
+            self.log("conv2 caf-remux; dur=%d sz=%d" % (dur, sz), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
                 b"-nostdin",
                 b"-v", b"error",
                 b"-hide_banner",
-                b"-i", fsenc(abspath if src_opus else tmp_opus),
+                b"-i", fsenc(tmp_opus),
                 b"-map_metadata", b"-1",
                 b"-map", b"0:a:0",
                 b"-c:a", b"copy",
@@ -831,7 +891,6 @@ class ThumbSrv(object):
             # fmt: on
             self._run_ff(cmd, vn, oom=300)
 
-        if tmp_opus != tpath:
         try:
             wunlink(self.log, tmp_opus, vn.flags)
         except:
@@ -891,7 +950,7 @@ class ThumbSrv(object):
 
     def _clean(self, cat: str, thumbpath: str) -> int:
         # self.log("cln {}".format(thumbpath))
-        exts = ["jpg", "webp", "png"] if cat == "th" else ["opus", "caf", "mp3"]
+        exts = EXTS_TH if cat == "th" else EXTS_AC
         maxage = getattr(self.args, cat + "_maxage")
         now = time.time()
         prev_b64 = None

copyparty/up2k.py

@@ -795,7 +795,7 @@ class Up2k(object):
                 continue
 
             self.log("xiu: %d# %r" % (len(wrfs), cmd))
-            runihook(self.log, cmd, vol, ups)
+            runihook(self.log, self.args.hook_v, cmd, vol, ups)
 
     def _vis_job_progress(self, job: dict[str, Any]) -> str:
         perc = 100 - (len(job["need"]) * 100.0 / (len(job["hash"]) or 1))
@@ -856,9 +856,9 @@ class Up2k(object):
         self.iacct = self.asrv.iacct
         self.grps = self.asrv.grps
 
+        have_e2d = self.args.idp_h_usr or self.args.chpw or self.args.shr
         vols = list(all_vols.values())
         t0 = time.time()
-        have_e2d = False
 
         if self.no_expr_idx:
             modified = False
@@ -1119,6 +1119,7 @@ class Up2k(object):
         reg = {}
         drp = None
         emptylist = []
+        dotpart = "." if self.args.dotpart else ""
         snap = os.path.join(histpath, "up2k.snap")
         if bos.path.exists(snap):
             with gzip.GzipFile(snap, "rb") as f:
@@ -1131,6 +1132,8 @@ class Up2k(object):
             except:
                 pass
 
+            reg = reg2  # diff-golf
+
             if reg2 and "dwrk" not in reg2[next(iter(reg2))]:
                 for job in reg2.values():
                     job["dwrk"] = job["wark"]
@@ -1138,7 +1141,8 @@ class Up2k(object):
             rm = []
             for k, job in reg2.items():
                 job["ptop"] = ptop
-                if "done" in job:
+                is_done = "done" in job
+                if is_done:
                     job["need"] = job["hash"] = emptylist
                 else:
                     if "need" not in job:
@@ -1146,10 +1150,13 @@ class Up2k(object):
                     if "hash" not in job:
                         job["hash"] = []
 
+                if is_done:
                     fp = djoin(ptop, job["prel"], job["name"])
+                else:
+                    fp = djoin(ptop, job["prel"], dotpart + job["name"] + ".PARTIAL")
+
                 if bos.path.exists(fp):
-                    reg[k] = job
+                    if is_done:
-                    if "done" in job:
                         continue
                     job["poke"] = time.time()
                     job["busy"] = {}
@@ -1157,11 +1164,18 @@ class Up2k(object):
                     self.log("ign deleted file in snap: %r" % (fp,))
                     if not n4g:
                         rm.append(k)
-                        continue
 
             for x in rm:
                 del reg2[x]
 
+            # optimize pre-1.15.4 entries
+            if next((x for x in reg.values() if "done" in x and "poke" in x), None):
+                zsl = "host tnam busy sprs poke t0c".split()
+                for job in reg.values():
+                    if "done" in job:
+                        for k in zsl:
+                            job.pop(k, None)
+
             if drp is None:
                 drp = [k for k, v in reg.items() if not v["need"]]
             else:
@@ -3004,7 +3018,7 @@ class Up2k(object):
                 if wark in reg:
                     del reg[wark]
                 job["hash"] = job["need"] = []
-                job["done"] = True
+                job["done"] = 1
                 job["busy"] = {}
 
             if lost:
@@ -4891,7 +4905,8 @@ class Up2k(object):
             except:
                 pass
 
-        xbu = self.flags[job["ptop"]].get("xbu")
+        vf = self.flags[job["ptop"]]
+        xbu = vf.get("xbu")
         ap_chk = djoin(pdir, job["name"])
         vp_chk = djoin(job["vtop"], job["prel"], job["name"])
         if xbu:
@@ -4921,7 +4936,7 @@ class Up2k(object):
                 if x:
                     zvfs = vfs
                     pdir, _, job["name"], (vfs, rem) = x
-                    job["vcfg"] = vfs.flags
+                    job["vcfg"] = vf = vfs.flags
                     job["ptop"] = vfs.realpath
                     job["vtop"] = vfs.vpath
                     job["prel"] = rem
@@ -4971,8 +4986,13 @@ class Up2k(object):
                 fs = self.fstab.get(pdir)
                 if fs == "ok":
                     pass
-                elif "sparse" in self.flags[job["ptop"]]:
+                elif "nosparse" in vf:
-                    t = "volflag 'sparse' is forcing use of sparse files for uploads to [%s]"
+                    t = "volflag 'nosparse' is preventing creation of sparse files for uploads to [%s]"
+                    self.log(t % (job["ptop"],))
+                    relabel = True
+                    sprs = False
+                elif "sparse" in vf:
+                    t = "volflag 'sparse' is forcing creation of sparse files for uploads to [%s]"
                     self.log(t % (job["ptop"],))
                     relabel = True
                 else:

copyparty/util.py

@@ -120,6 +120,13 @@ try:
 except:
     HAVE_SQLITE3 = False
 
+try:
+    import importlib.util
+
+    HAVE_ZMQ = bool(importlib.util.find_spec("zmq"))
+except:
+    HAVE_ZMQ = False
+
 try:
     if os.environ.get("PRTY_NO_PSUTIL"):
         raise Exception()
@@ -229,9 +236,14 @@ META_NOBOTS = '<meta name="robots" content="noindex, nofollow">\n'
 
 FFMPEG_URL = "https://www.gyan.dev/ffmpeg/builds/ffmpeg-git-full.7z"
 
+URL_PRJ = "https://github.com/9001/copyparty"
+
+URL_BUG = URL_PRJ + "/issues/new?labels=bug&template=bug_report.md"
+
 HTTPCODE = {
     200: "OK",
     201: "Created",
+    202: "Accepted",
     204: "No Content",
     206: "Partial Content",
     207: "Multi-Status",
@@ -319,6 +331,7 @@ DAV_ALLPROPS = set(DAV_ALLPROP_L)
 
 MIMES = {
     "opus": "audio/ogg; codecs=opus",
+    "owa": "audio/webm; codecs=opus",
 }
 
 
@@ -491,6 +504,15 @@ def py_desc() -> str:
     )
 
 
+def expat_ver() -> str:
+    try:
+        import pyexpat
+
+        return ".".join([str(x) for x in pyexpat.version_info])
+    except:
+        return "?"
+
+
 def _sqlite_ver() -> str:
     assert sqlite3  # type: ignore  # !rm
     try:
@@ -3386,6 +3408,7 @@ def _parsehook(
 
 def runihook(
     log: Optional["NamedLogger"],
+    verbose: bool,
     cmd: str,
     vol: "VFS",
     ups: list[tuple[str, int, int, str, str, str, int]],
@@ -3415,6 +3438,17 @@ def runihook(
     else:
         sp_ka["sin"] = b"\n".join(fsenc(x) for x in aps)
 
+    if acmd[0].startswith("zmq:"):
+        try:
+            msg = sp_ka["sin"].decode("utf-8", "replace")
+            _zmq_hook(log, verbose, "xiu", acmd[0][4:].lower(), msg, wait, sp_ka)
+            if verbose and log:
+                log("hook(xiu) %r OK" % (cmd,), 6)
+        except Exception as ex:
+            if log:
+                log("zeromq failed: %r" % (ex,))
+        return True
+
     t0 = time.time()
     if fork:
         Daemon(runcmd, cmd, bcmd, ka=sp_ka)
@@ -3424,6 +3458,7 @@ def runihook(
             retchk(rc, bcmd, err, log, 5)
             return False
 
+    if wait:
         wait -= time.time() - t0
         if wait > 0:
             time.sleep(wait)
@@ -3431,8 +3466,118 @@ def runihook(
     return True
 
 
+ZMQ = {}
+ZMQ_DESC = {
+    "pub": "fire-and-forget to all/any connected SUB-clients",
+    "push": "fire-and-forget to one of the connected PULL-clients",
+    "req": "send messages to a REP-server and blocking-wait for ack",
+}
+
+
+def _zmq_hook(
+    log: Optional["NamedLogger"],
+    verbose: bool,
+    src: str,
+    cmd: str,
+    msg: str,
+    wait: float,
+    sp_ka: dict[str, Any],
+) -> tuple[int, str]:
+    import zmq
+
+    try:
+        mtx = ZMQ["mtx"]
+    except:
+        ZMQ["mtx"] = threading.Lock()
+        time.sleep(0.1)
+        mtx = ZMQ["mtx"]
+
+    ret = ""
+    nret = 0
+    t0 = time.time()
+    if verbose and log:
+        log("hook(%s) %r entering zmq-main-lock" % (src, cmd), 6)
+
+    with mtx:
+        try:
+            mode, sck, mtx = ZMQ[cmd]
+        except:
+            mode, uri = cmd.split(":", 1)
+            try:
+                desc = ZMQ_DESC[mode]
+                if log:
+                    t = "libzmq(%s) pyzmq(%s) init(%s); %s"
+                    log(t % (zmq.zmq_version(), zmq.__version__, cmd, desc))
+            except:
+                raise Exception("the only supported ZMQ modes are REQ PUB PUSH")
+
+            try:
+                ctx = ZMQ["ctx"]
+            except:
+                ctx = ZMQ["ctx"] = zmq.Context()
+
+            timeout = sp_ka["timeout"]
+
+            if mode == "pub":
+                sck = ctx.socket(zmq.PUB)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.bind(uri)
+                time.sleep(1)  # give clients time to connect; avoids losing first msg
+            elif mode == "push":
+                sck = ctx.socket(zmq.PUSH)
+                if timeout:
+                    sck.SNDTIMEO = int(timeout * 1000)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.bind(uri)
+            elif mode == "req":
+                sck = ctx.socket(zmq.REQ)
+                if timeout:
+                    sck.RCVTIMEO = int(timeout * 1000)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.connect(uri)
+            else:
+                raise Exception()
+
+            mtx = threading.Lock()
+            ZMQ[cmd] = (mode, sck, mtx)
+
+    if verbose and log:
+        log("hook(%s) %r entering socket-lock" % (src, cmd), 6)
+
+    with mtx:
+        if verbose and log:
+            log("hook(%s) %r sending |%d|" % (src, cmd, len(msg)), 6)
+
+        sck.send_string(msg)  # PUSH can safely timeout here
+
+        if mode == "req":
+            if verbose and log:
+                log("hook(%s) %r awaiting ack from req" % (src, cmd), 6)
+            try:
+                ret = sck.recv().decode("utf-8", "replace")
+                if ret.startswith("return "):
+                    m = re.search("^return ([0-9]+)", ret[:12])
+                    if m:
+                        nret = int(m.group(1))
+            except:
+                sck.close()
+                del ZMQ[cmd]  # bad state; must reset
+                raise Exception("ack timeout; zmq socket killed")
+
+    if ret and log:
+        log("hook(%s) %r ACK: %r" % (src, cmd, ret), 6)
+
+    if wait:
+        wait -= time.time() - t0
+        if wait > 0:
+            time.sleep(wait)
+
+    return nret, ret
+
+
 def _runhook(
     log: Optional["NamedLogger"],
+    verbose: bool,
     src: str,
     cmd: str,
     ap: str,
@@ -3473,6 +3618,12 @@ def _runhook(
     else:
         arg = txt or ap
 
+    if acmd[0].startswith("zmq:"):
+        zi, zs = _zmq_hook(log, verbose, src, acmd[0][4:].lower(), arg, wait, sp_ka)
+        if zi:
+            raise Exception("zmq says %d" % (zi,))
+        return {"rc": 0, "stdout": zs}
+
     acmd += [arg]
     if acmd[0].endswith(".py"):
         acmd = [pybin] + acmd
@@ -3501,6 +3652,7 @@ def _runhook(
             except:
                 ret = {"rc": rc, "stdout": v}
 
+    if wait:
         wait -= time.time() - t0
         if wait > 0:
             time.sleep(wait)
@@ -3527,14 +3679,15 @@ def runhook(
 ) -> dict[str, Any]:
     assert broker or up2k  # !rm
     args = (broker or up2k).args
+    verbose = args.hook_v
     vp = vp.replace("\\", "/")
     ret = {"rc": 0}
     for cmd in cmds:
         try:
             hr = _runhook(
-                log, src, cmd, ap, vp, host, uname, perms, mt, sz, ip, at, txt
+                log, verbose, src, cmd, ap, vp, host, uname, perms, mt, sz, ip, at, txt
             )
-            if log and args.hook_v:
+            if verbose and log:
                 log("hook(%s) %r => \033[32m%s" % (src, cmd, hr), 6)
             if not hr:
                 return {}
@@ -3550,6 +3703,8 @@ def runhook(
                 elif k in ret:
                     if k == "rc" and v:
                         ret[k] = v
+                    elif k == "stdout" and v and not ret[k]:
+                        ret[k] = v
                 else:
                     ret[k] = v
         except Exception as ex:

copyparty/web/browser.html

@@ -131,7 +131,7 @@
 	<div id="widget"></div>
 
 	<script>
-		var SR = {{ r|tojson }},
+		var SR = "{{ r }}",
 			CGV1 = {{ cgv1 }},
 			CGV = {{ cgv|tojson }},
 			TS = "{{ ts }}",

copyparty/web/browser.js

@@ -241,7 +241,7 @@ var Ls = {
 		"cut_nag": "OS notification when upload completes$N(only if the browser or tab is not active)",
 		"cut_sfx": "audible alert when upload completes$N(only if the browser or tab is not active)",
 
-		"cut_mt": "use multithreading to accelerate file hashing$N$Nthis uses web-workers and requires$Nmore RAM (up to 512 MiB extra)$N$N30% faster https, 4.5x faster http,$Nand 5.3x faster on android phones\">mt",
+		"cut_mt": "use multithreading to accelerate file hashing$N$Nthis uses web-workers and requires$Nmore RAM (up to 512 MiB extra)$N$Nmakes https 30% faster, http 4.5x faster\">mt",
 
 		"cft_text": "favicon text (blank and refresh to disable)",
 		"cft_fg": "foreground color",
@@ -263,6 +263,7 @@ var Ls = {
 		"ml_pmode": "at end of folder...",
 		"ml_btns": "cmds",
 		"ml_tcode": "transcode",
+		"ml_tcode2": "transcode to",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer",
 		"ml_drc": "dynamic range compressor",
@@ -286,6 +287,14 @@ var Ls = {
 		"mt_cflac": "convert flac / wav to opus\">flac",
 		"mt_caac": "convert aac / m4a to opus\">aac",
 		"mt_coth": "convert all others (not mp3) to opus\">oth",
+		"mt_c2opus": "best choice for desktops, laptops, android\">opus",
+		"mt_c2owa": "opus-weba, for iOS 17.5 and newer\">owa",
+		"mt_c2caf": "opus-caf, for iOS 11 through 17\">caf",
+		"mt_c2mp3": "use this on very old devices\">mp3",
+		"mt_c2ok": "nice, good choice",
+		"mt_c2nd": "that's not the recommended output format for your device, but that's fine",
+		"mt_c2ng": "your device does not seem to support this output format, but let's try anyways",
+		"mt_xowa": "there are bugs in iOS preventing background playback using this format; please use caf or mp3 instead",
 		"mt_tint": "background level (0-100) on the seekbar$Nto make buffering less distracting",
 		"mt_eq": "enables the equalizer and gain control;$N$Nboost <code>0</code> = standard 100% volume (unmodified)$N$Nwidth <code>1  </code> = standard stereo (unmodified)$Nwidth <code>0.5</code> = 50% left-right crossfeed$Nwidth <code>0  </code> = mono$N$Nboost <code>-0.8</code> & width <code>10</code> = vocal removal :^)$N$Nenabling the equalizer makes gapless albums fully gapless, so leave it on with all the values at zero (except width = 1) if you care about that",
 		"mt_drc": "enables the dynamic range compressor (volume flattener / brickwaller); will also enable EQ to balance the spaghetti, so set all EQ fields except for 'width' to 0 if you don't want it$N$Nlowers the volume of audio above THRESHOLD dB; for every RATIO dB past THRESHOLD there is 1 dB of output, so default values of tresh -24 and ratio 12 means it should never get louder than -22 dB and it is safe to increase the equalizer boost to 0.8, or even 1.8 with ATK 0 and a huge RLS like 90 (only works in firefox; RLS is max 1 in other browsers)$N$N(see wikipedia, they explain it much better)",
@@ -830,7 +839,7 @@ var Ls = {
 		"cut_nag": "meldingsvarsel når opplastning er ferdig$N(kun on nettleserfanen ikke er synlig)",
 		"cut_sfx": "lydvarsel når opplastning er ferdig$N(kun on nettleserfanen ikke er synlig)",
 
-		"cut_mt": "raskere befaring ved å bruke hele CPU'en$N$Ndenne funksjonen anvender web-workers$Nog krever mer RAM (opptil 512 MiB ekstra)$N$N30% raskere https, 4.5x raskere http,$Nog 5.3x raskere på android-telefoner\">mt",
+		"cut_mt": "raskere befaring ved å bruke hele CPU'en$N$Ndenne funksjonen anvender web-workers$Nog krever mer RAM (opptil 512 MiB ekstra)$N$Ngjør https 30% raskere, http 4.5x raskere\">mt",
 
 		"cft_text": "ikontekst (blank ut og last siden på nytt for å deaktivere)",
 		"cft_fg": "farge",
@@ -852,6 +861,7 @@ var Ls = {
 		"ml_pmode": "ved enden av mappen",
 		"ml_btns": "knapper",
 		"ml_tcode": "konvertering",
+		"ml_tcode2": "konverter til",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer (tonejustering)",
 		"ml_drc": "compressor (volum-utjevning)",
@@ -875,6 +885,14 @@ var Ls = {
 		"mt_cflac": "konverter flac / wav-filer til opus\">flac",
 		"mt_caac": "konverter aac / m4a-filer til to opus\">aac",
 		"mt_coth": "konverter alt annet (men ikke mp3) til opus\">andre",
+		"mt_c2opus": "det beste valget for alle PCer og Android\">opus",
+		"mt_c2owa": "opus-weba, for iOS 17.5 og nyere\">owa",
+		"mt_c2caf": "opus-caf, for iOS 11 tilogmed 17\">caf",
+		"mt_c2mp3": "bra valg for steinalder-utstyr (slår aldri feil)\">mp3",
+		"mt_c2ok": "bra valg!",
+		"mt_c2nd": "ikke det foretrukne valget for din enhet, men funker sikkert greit",
+		"mt_c2ng": "ser virkelig ikke ut som enheten din takler dette formatet... men ok, vi prøver",
+		"mt_xowa": "iOS har fortsatt problemer med avspilling av owa-musikk i bakgrunnen. Bruk caf eller mp3 istedenfor",
 		"mt_tint": "nivå av bakgrunnsfarge på søkestripa (0-100),$Ngjør oppdateringer mindre distraherende",
 		"mt_eq": "aktiver tonekontroll og forsterker;$N$Nboost <code>0</code> = normal volumskala$N$Nwidth <code>1  </code> = normal stereo$Nwidth <code>0.5</code> = 50% blanding venstre-høyre$Nwidth <code>0  </code> = mono$N$Nboost <code>-0.8</code> & width <code>10</code> = instrumental :^)$N$Nreduserer også dødtid imellom sangfiler",
 		"mt_drc": "aktiver volum-utjevning (dynamic range compressor); vil også aktivere tonejustering, så sett alle EQ-feltene bortsett fra 'width' til 0 hvis du ikke vil ha noe EQ$N$Nfilteret vil dempe volumet på alt som er høyere enn TRESH dB; for hver RATIO dB over grensen er det 1dB som treffer høyttalerne, så standardverdiene tresh -24 og ratio 12 skal bety at volumet ikke går høyere enn -22 dB, slik at man trygt kan øke boost-verdien i equalizer'n til rundt 0.8, eller 1.8 kombinert med ATK 0 og RLS 90 (bare mulig i firefox; andre nettlesere tar ikke høyere RLS enn 1)$N$Nwikipedia forklarer dette mye bedre forresten",
@@ -1419,7 +1437,7 @@ var Ls = {
 		"cut_nag": "上传完成时的操作系统通知$N（仅当浏览器或标签页不活跃时）",
 		"cut_sfx": "上传完成时的声音警报$N（仅当浏览器或标签页不活跃时）",
 
-		"cut_mt": "使用多线程加速文件哈希$N$N这使用 Web Worker 并且需要更多内存（额外最多 512 MiB）$N$N比https快30%，http快4.5倍，比Android 手机快5.3倍\">mt",
+		"cut_mt": "使用多线程加速文件哈希$N$N这使用 Web Worker 并且需要更多内存（额外最多 512 MiB）$N$N这使得 https 快 30%，http 快 4.5 倍\">mt",
 
 		"cft_text": "网站图标文本（为空并刷新以禁用）",
 		"cft_fg": "前景色",
@@ -1441,6 +1459,7 @@ var Ls = {
 		"ml_pmode": "在文件夹末尾时...",
 		"ml_btns": "命令",
 		"ml_tcode": "转码",
+		"ml_tcode2": "转换为", //m
 		"ml_tint": "透明度",
 		"ml_eq": "音频均衡器",
 		"ml_drc": "动态范围压缩器",
@@ -1464,6 +1483,14 @@ var Ls = {
 		"mt_cflac": "将 flac / wav 转换为 opus\">flac",
 		"mt_caac": "将 aac / m4a 转换为 opus\">aac",
 		"mt_coth": "将所有其他（不是 mp3）转换为 opus\">oth",
+		"mt_c2opus": "适合桌面电脑、笔记本电脑和安卓设备的最佳选择\">opus", //m
+		"mt_c2owa": "opus-weba（适用于 iOS 17.5 及更新版本）\">owa", //m
+		"mt_c2caf": "opus-caf（适用于 iOS 11 到 iOS 17）\">caf", //m
+		"mt_c2mp3": "适用于非常旧的设备\">mp3", //m
+		"mt_c2ok": "不错的选择！", //m
+		"mt_c2nd": "这不是您的设备推荐的输出格式，但应该没问题。", //m
+		"mt_c2ng": "您的设备似乎不支持此输出格式，不过我们还是试试看吧。", //m
+		"mt_xowa": "iOS 系统仍存在无法后台播放 owa 音乐的错误，请改用 caf 或 mp3 格式。", //m
 		"mt_tint": "在进度条上设置背景级别（0-100）",
 		"mt_eq": "启用均衡器和增益控制；$N$Nboost <code>0</code> = 标准 100% 音量（默认）$N$Nwidth <code>1  </code> = 标准立体声（默认）$Nwidth <code>0.5</code> = 50% 左右交叉反馈$Nwidth <code>0  </code> = 单声道$N$Nboost <code>-0.8</code> & width <code>10</code> = 人声移除 )$N$N启用均衡器使无缝专辑完全无缝，所以如果你在乎这一点，请保持启用，所有值设为零（除了宽度 = 1）",
 		"mt_drc": "启用动态范围压缩器（音量平滑器 / 限幅器）；还会启用均衡器以平衡音频，因此如果你不想要它，请将均衡器字段除了 '宽度' 外的所有字段设置为 0$N$N降低 THRESHOLD dB 以上的音频的音量；每超过 THRESHOLD dB 的 RATIO 会有 1 dB 输出，所以默认值 tresh -24 和 ratio 12 意味着它的音量不应超过 -22 dB，可以安全地将均衡器增益提高到 0.8，甚至在 ATK 0 和 RLS 如 90 的情况下提高到 1.8（仅在 Firefox 中有效；其他浏览器中 RLS 最大为 1）$N$N（见维基百科，他们解释得更好）",
@@ -2269,6 +2296,12 @@ var mpl = (function () {
 			'<a href="#" id="ac_flac" class="tgl btn" tt="' + L.mt_cflac + '</a>' +
 			'<a href="#" id="ac_aac" class="tgl btn" tt="' + L.mt_caac + '</a>' +
 			'<a href="#" id="ac_oth" class="tgl btn" tt="' + L.mt_coth + '</a>' +
+			'</div></div>' +
+			'<div><h3>' + L.ml_tcode2 + '</h3><div>' +
+			'<a href="#" id="ac2opus" class="tgl btn" tt="' + L.mt_c2opus + '</a>' +
+			'<a href="#" id="ac2owa" class="tgl btn" tt="' + L.mt_c2owa + '</a>' +
+			'<a href="#" id="ac2caf" class="tgl btn" tt="' + L.mt_c2caf + '</a>' +
+			'<a href="#" id="ac2mp3" class="tgl btn" tt="' + L.mt_c2mp3 + '</a>' +
 			'</div></div>'
 		) : '') +
 
@@ -2376,7 +2409,7 @@ var mpl = (function () {
 			c = r.ac_flac;
 		else if (/\.(aac|m4a)$/i.exec(cs))
 			c = r.ac_aac;
-		else if (/\.(ogg|opus)$/i.exec(cs) && !can_ogg)
+		else if (/\.(ogg|opus)$/i.exec(cs) && (!can_ogg || mpl.ac2 == 'mp3'))
 			c = true;
 		else if (re_au_native.exec(cs))
 			c = false;
@@ -2384,7 +2417,49 @@ var mpl = (function () {
 		if (!c)
 			return url;
 
-		return addq(url, 'th=' + (can_ogg ? 'opus' : (IPHONE || MACOS) ? 'caf' : 'mp3'));
+		return addq(url, 'th=' + r.ac2);
+	};
+
+	r.set_ac2 = function () {
+		r.init_ac2(this.getAttribute('id').split('ac2')[1]);
+	};
+
+	r.init_ac2 = function (v) {
+		var dv = can_ogg ? 'opus' : can_caf ? 'caf' : 'mp3',
+			fmts = ['opus', 'owa', 'caf', 'mp3'],
+			btns = [];
+
+		if (v === dv)
+			toast.ok(5, L.mt_c2ok);
+		else if (v)
+			toast.inf(10, L.mt_c2nd);
+
+		if ((v == 'opus' && !can_ogg) ||
+			(v == 'caf' && !can_caf) ||
+			(v == 'owa' && !can_owa))
+			toast.warn(15, L.mt_c2ng);
+
+		if (v == 'owa' && IPHONE)
+			toast.err(30, L.mt_xowa);
+
+		for (var a = 0; a < fmts.length; a++) {
+			var btn = ebi('ac2' + fmts[a]);
+			btn.onclick = r.set_ac2;
+			btns.push(btn);
+		}
+		if (!IPHONE)
+			btns[1].style.display = btns[2].style.display = 'none';
+
+		if (v)
+			swrite('acode2', v);
+		else
+			v = dv;
+
+		v = sread('acode2', fmts) || v;
+		for (var a = 0; a < fmts.length; a++)
+			clmod(btns[a], 'on', fmts[a] == v)
+
+		r.ac2 = v;
 	};
 
 	r.pp = function () {
@@ -2483,6 +2558,7 @@ var mpl = (function () {
 	r.unbuffer = function (url) {
 		if (mp.au2 && (!url || mp.au2.rsrc == url)) {
 			mp.au2.src = mp.au2.rsrc = '';
+			mp.au2.ld = 0; //owa
 			mp.au2.load();
 		}
 		if (!url)
@@ -2493,11 +2569,22 @@ var mpl = (function () {
 })();
 
 
-var can_ogg = true;
+var za,
+	can_ogg = true,
+	can_owa = false,
+	can_caf = IPHONE && !/ OS ([1-9]|1[01])_/.test(UA);
 try {
-	can_ogg = new Audio().canPlayType('audio/ogg; codecs=opus') === 'probably';
+	za = new Audio();
+	can_ogg = za.canPlayType('audio/ogg; codecs=opus') === 'probably';
+	can_owa = za.canPlayType('audio/webm; codecs=opus') === 'probably';
 }
 catch (ex) { }
+za = null;
+
+if (can_owa && IPHONE && / OS ([1-9]|1[0-7])_/.test(UA))
+	can_owa = false;
+
+mpl.init_ac2();
 
 
 var re_au_native = (can_ogg || have_acode) ? /\.(aac|flac|m4a|mp3|ogg|opus|wav)$/i : /\.(aac|flac|m4a|mp3|wav)$/i,
@@ -2670,7 +2757,8 @@ function MPlayer() {
 			});
 
 		r.nopause();
-		r.au2.onloadeddata = r.au2.onloadedmetadata = r.nopause;
+		r.au2.ld = 0; //owa
+		r.au2.onloadeddata = r.au2.onloadedmetadata = r.onpreload;
 		r.au2.preload = "auto";
 		r.au2.src = r.au2.rsrc = url;
 
@@ -2685,6 +2773,11 @@ function MPlayer() {
 		r.cd_pause = Date.now();
 	};
 
+	r.onpreload = function () {
+		r.nopause();
+		this.ld++;
+	};
+
 	r.init_fau = function () {
 		if (r.fau || !mpl.fau)
 			return;
@@ -2985,7 +3078,7 @@ var pbar = (function () {
 				return;
 
 			pctx.fillStyle = light ? 'rgba(0,0,0,0.5)' : 'rgba(255,255,255,0.5)';
-			var m = /[?&]th=(opus|caf|mp3)/.exec('' + mp.au.rsrc),
+			var m = /[?&]th=(opus|owa|caf|mp3)/.exec('' + mp.au.rsrc),
 				txt = mp.au.ded ? L.mm_playerr.replace(':', ' ;_;') :
 					m ? L.mm_bconv.format(m[1]) : L.mm_bload;
 
@@ -3064,6 +3157,7 @@ var vbar = (function () {
 		can = r.can.can;
 		ctx = r.can.ctx;
 		ctx.font = '.7em sans-serif';
+		ctx.fontVariantCaps = 'small-caps';
 		w = r.can.w;
 		h = r.can.h;
 		r.draw();
@@ -3089,18 +3183,18 @@ var vbar = (function () {
 		ctx.fillStyle = grad2; ctx.fillRect(0, 0, w, h);
 		ctx.fillStyle = grad1; ctx.fillRect(0, 0, w * mp.vol, h);
 
-		if (Date.now() - lastv > 1000)
+		var vt = 'volume ' + Math.floor(mp.vol * 100),
-			return;
+			tw = ctx.measureText(vt).width,
+			x = w * mp.vol - tw - 8,
+			li = dy;
 
-		var vt = Math.floor(mp.vol * 100),
+		if (mp.vol < 0.5) {
-			tw = ctx.measureText(vt).width;
+			x += tw + 16;
-
-		var li = dy;
-		if (mp.vol < 0.05)
 			li = !li;
+		}
 
 		ctx.fillStyle = li ? '#fff' : '#210';
-		ctx.fillText(vt, Math.max(4, w * mp.vol - tw - 8), h / 3 * 2);
+		ctx.fillText(vt, x, h / 3 * 2);
 
 		clearTimeout(untext);
 		untext = setTimeout(r.draw, 1000);
@@ -3940,16 +4034,20 @@ function play(tid, is_ev, seek) {
 		mp.au = mp.au2;
 		mp.au2 = t;
 		t.onerror = t.onprogress = t.onended = null;
+		t.ld = 0; //owa
 		mp.au.onerror = evau_error;
 		mp.au.onprogress = pbar.drawpos;
 		mp.au.onplaying = mpui.progress_updater;
+		mp.au.onloadeddata = mp.au.onloadedmetadata = mp.nopause;
 		mp.au.onended = next_song;
 		t = mp.au.currentTime;
 		if (isNum(t) && t > 0.1)
 			mp.au.currentTime = 0;
 	}
-	else
+	else {
+		console.log('get ' + url.split('/').pop());
 		mp.au.src = mp.au.rsrc = url;
+	}
 
 	mp.au.osrc = mp.tracks[tid];
 	afilt.apply();
@@ -4035,6 +4133,14 @@ function evau_error(e) {
 			err = L.mm_eabrt;
 			break;
 		case eplaya.error.MEDIA_ERR_NETWORK:
+			if (IPHONE && eplaya.ld === 1 && mpl.ac2 == 'owa' && !eplaya.paused && !eplaya.currentTime) {
+				eplaya.ded = 0;
+				if (!mpl.owaw) {
+					mpl.owaw = 1;
+					console.log('ignored iOS bug; spurious error sent in parallel with preloaded songs starting to play just fine');
+				}
+				return;
+			}
 			err = L.mm_enet;
 			break;
 		case eplaya.error.MEDIA_ERR_DECODE:
@@ -7354,12 +7460,12 @@ var treectl = (function () {
 		xhr.rst = rst;
 		xhr.ts = Date.now();
 		xhr.open('GET', addq(dst, 'tree=' + top + (r.dots ? '&dots' : '') + k), true);
-		xhr.onload = xhr.onerror = recvtree;
+		xhr.onload = xhr.onerror = r.recvtree;
 		xhr.send();
 		enspin('#tree');
 	}
 
-	function recvtree() {
+	r.recvtree = function () {
 		if (!xhrchk(this, L.tl_xe1, L.tl_xe2))
 			return;
 
@@ -7369,10 +7475,10 @@ var treectl = (function () {
 		catch (ex) {
 			return toast.err(30, "bad <code>?tree</code> reply;\nexpected json, got this:\n\n" + esc(this.responseText + ''));
 		}
-		rendertree(res, this.ts, this.top, this.dst, this.rst);
+		r.rendertree(res, this.ts, this.top, this.dst, this.rst);
-	}
+	};
 
-	function rendertree(res, ts, top0, dst, rst) {
+	r.rendertree = function (res, ts, top0, dst, rst) {
 		var cur = ebi('treeul').getAttribute('ts');
 		if (cur && parseInt(cur) > ts + 20 && QS('#treeul>li>a+a')) {
 			console.log("reject tree; " + cur + " / " + (ts - cur));
@@ -7426,7 +7532,7 @@ var treectl = (function () {
 				console.log("dir_cb failed", ex);
 			}
 		}
-	}
+	};
 
 	function reload_tree() {
 		var cdir = r.nextdir || get_vpath(),
@@ -7648,7 +7754,7 @@ var treectl = (function () {
 				dirs.push(dn);
 			}
 
-			rendertree({ "a": dirs }, this.ts, ".", get_evpath() + (dk ? '?k=' + dk : ''));
+			r.rendertree({ "a": dirs }, this.ts, ".", get_evpath() + (dk ? '?k=' + dk : ''));
 		}
 
 		r.gentab(this.top, res);
@@ -7666,9 +7772,9 @@ var treectl = (function () {
 		if (lg1 === Ls.eng.f_empty)
 			lg1 = L.f_empty;
 
-		sandbox(ebi('pro'), sb_lg, '', lg0);
+		sandbox(ebi('pro'), sb_lg, sba_lg,'', lg0);
 		if (dirchg)
-			sandbox(ebi('epi'), sb_lg, '', lg1);
+			sandbox(ebi('epi'), sb_lg, sba_lg, '', lg1);
 
 		clmod(ebi('pro'), 'mdo');
 		clmod(ebi('epi'), 'mdo');
@@ -7679,7 +7785,7 @@ var treectl = (function () {
 		if (md1)
 			show_readme(md1, 1);
 		else if (!dirchg)
-			sandbox(ebi('epi'), sb_lg, '', lg1);
+			sandbox(ebi('epi'), sb_lg, sba_lg, '', lg1);
 
 		if (this.hpush && !this.back) {
 			var ofs = ebi('wrap').offsetTop;
@@ -8634,8 +8740,8 @@ var arcfmt = (function () {
 			["pax", "tar=pax", L.fz_pax],
 			["tgz", "tar=gz", L.fz_targz],
 			["txz", "tar=xz", L.fz_tarxz],
-			["zip", "zip=utf8", L.fz_zip8],
+			["zip", "zip", L.fz_zip8],
-			["zip_dos", "zip", L.fz_zipd],
+			["zip_dos", "zip=dos", L.fz_zipd],
 			["zip_crc", "zip=crc", L.fz_zipc]
 		];
 
@@ -9027,14 +9133,18 @@ var msel = (function () {
 	function cb() {
 		xhrchk(this, L.fsm_xe1, L.fsm_xe2);
 
-		if (this.status < 200 || this.status > 201) {
+		if (this.status < 200 || this.status > 202) {
 			sf.textContent = 'error: ' + hunpre(this.responseText);
 			return;
 		}
 
 		tb.value = '';
 		clmod(sf, 'vis');
-		sf.textContent = 'sent: "' + this.msg + '"';
+		var txt = 'sent: <code>' + esc(this.msg) + '</code>';
+		if (this.status == 202)
+			txt += '<br />&nbsp; got: <code>' + esc(this.responseText) + '</code>';
+
+		sf.innerHTML = txt;
 		setTimeout(function () {
 			treectl.goto();
 		}, 100);
@@ -9155,7 +9265,7 @@ function show_md(md, name, div, url, depth) {
 		if (!have_emp)
 			md_html = DOMPurify.sanitize(md_html);
 
-		if (sandbox(div, sb_md, 'mdo', md_html))
+		if (sandbox(div, sb_md, sba_md, 'mdo', md_html))
 			return;
 
 		ext = md_plug.post;
@@ -9206,7 +9316,7 @@ function show_readme(md, n) {
 	var tgt = ebi(n ? 'epi' : 'pro');
 
 	if (!treectl.ireadme)
-		return sandbox(tgt, '', '', 'a');
+		return sandbox(tgt, '', '', '', 'a');
 
 	show_md(md, n ? 'README.md' : 'PREADME.md', tgt);
 }
@@ -9215,7 +9325,7 @@ for (var a = 0; a < readmes.length; a++)
 		show_readme(readmes[a], a);
 
 
-function sandbox(tgt, rules, cls, html) {
+function sandbox(tgt, rules, allow, cls, html) {
 	if (!treectl.ireadme) {
 		tgt.innerHTML = html ? L.md_off : '';
 		return;
@@ -9271,6 +9381,7 @@ function sandbox(tgt, rules, cls, html) {
 	var fr = mknod('iframe');
 	fr.setAttribute('title', 'folder ' + tid + 'logue');
 	fr.setAttribute('sandbox', rules ? 'allow-' + rules.replace(/ /g, ' allow-') : '');
+	fr.setAttribute('allow', allow);
 	fr.setAttribute('srcdoc', html);
 	tgt.appendChild(fr);
 	treectl.sb_msg = true;
@@ -9320,8 +9431,8 @@ if (sb_lg && logues.length) {
 	if (logues[1] === Ls.eng.f_empty)
 		logues[1] = L.f_empty;
 
-	sandbox(ebi('pro'), sb_lg, '', logues[0]);
+	sandbox(ebi('pro'), sb_lg, sba_lg, '', logues[0]);
-	sandbox(ebi('epi'), sb_lg, '', logues[1]);
+	sandbox(ebi('epi'), sb_lg, sba_lg, '', logues[1]);
 }
 
 

copyparty/web/md.html

@@ -128,9 +128,9 @@ write markdown (most html is 🙆 too)
 
 	<script>
 
-var SR = {{ r|tojson }},
+var SR = "{{ r }}",
 	last_modified = {{ lastmod }},
-	have_emp = {{ have_emp|tojson }},
+	have_emp = {{ "true" if have_emp else "false" }},
 	dfavico = "{{ favico }}";
 
 var md_opt = {

copyparty/web/md.js

@@ -23,8 +23,7 @@ var dbg = function () { };
 
 // dodge browser issues
 (function () {
-    var ua = navigator.userAgent;
+    if (UA.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(UA)) {
-    if (ua.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(ua)) {
         // necessary on ff-68.7 at least
         var s = mknod('style');
         s.innerHTML = '@page { margin: .5in .6in .8in .6in; }';

copyparty/web/md2.js

@@ -450,7 +450,7 @@ function savechk_cb() {
 
 // firefox bug: initial selection offset isn't cleared properly through js
 var ff_clearsel = (function () {
-    if (navigator.userAgent.indexOf(') Gecko/') === -1)
+    if (UA.indexOf(') Gecko/') === -1)
         return function () { }
 
     return function () {

copyparty/web/mde.html

@@ -26,9 +26,9 @@
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR = "{{ r }}",
 	last_modified = {{ lastmod }},
-	have_emp = {{ have_emp|tojson }},
+	have_emp = {{ "true" if have_emp else "false" }},
 	dfavico = "{{ favico }}";
 
 var md_opt = {

copyparty/web/rups.css

@@ -10,16 +10,10 @@ html {
 	padding: 0 1em 3em 1em;
 	line-height: 2.3em;
 }
-form {
-	display: inline;
-	padding-left: 1em;
-}
-input[type=submit],
 a {
 	color: #047;
 	background: #fff;
 	text-decoration: none;
-	border: none;
 	border-bottom: 1px solid #8ab;
 	border-radius: .2em;
 	padding: .2em .6em;
@@ -89,7 +83,6 @@ html.bz {
 	background: #11121d;
 	color: #bbd;
 }
-html.z input[type=submit],
 html.z a {
 	color: #fff;
 	background: #057;

copyparty/web/rups.html

@@ -6,6 +6,7 @@
 	<title>{{ s_doctitle }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/rups.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
@@ -14,14 +15,10 @@
 
 <body>
 	<div id="wrap">
-        <a id="a" href="{{ r }}/?ru" class="af">refresh</a>
+        <a href="#" id="re">refresh</a>
-        <a id="a" href="{{ r }}/?h" class="af">control-panel</a>
+        <a href="{{ r }}/?h">control-panel</a>
-        <form method="get" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ r }}">
+        &nbsp; Filter: <input type="text" id="filter" size="20" placeholder="documents/passwords" />
-            <input type="hidden" name="ru" value="a" />
+        &nbsp; <span id="hits"></span>
-            Filter: <input type="text" name="filter" size="20" placeholder="documents/passwords" value="{{ filt }}" />
-            <input type="submit" />
-        </form>
-        <span id="hits"></span>
         <table id="tab"><thead><tr>
             <th>size</th>
             <th>who</th>
@@ -29,27 +26,12 @@
             <th>age</th>
             <th>dir</th>
             <th>file</th>
-        </tr></thead><tbody>
+        </tr></thead><tbody id="tb"></tbody></table>
-        {% for vp, evp, sz, ip, at in rows %}
-<tr>
-<td>{{ sz }}</td>
-<td>{{ ip }}</td>
-<td>{{ at }}</td>
-<td>{{ (now-at) }}</td>
-<td></td>
-<td><a href="{{ r }}{{ evp }}">{{ vp|e }}</a></td>
-</tr>
-        {% endfor %}
-        </tbody></table>
-        {% if not rows %}
-        (the database is not aware of any uploads)
-        {% endif %}
     </div>
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
-    NOW = {{ now }},
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";
 
@@ -58,6 +40,7 @@ document.documentElement.className = (STG && STG.cpp_thm) || "{{ this.args.theme
 
 </script>
 <script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>
+<script>var V={{ v }};</script>
 <script src="{{ r }}/.cpr/rups.js?_={{ ts }}"></script>
 {%- if js %}
 <script src="{{ js }}_={{ ts }}"></script>

copyparty/web/rups.js

@@ -1,34 +1,66 @@
-(function() {
+function render() {
-    var tab = ebi('tab').tBodies[0],
+    var ups = V.ups, now = V.now, html = [];
-        tr = Array.prototype.slice.call(tab.rows, 0),
+    ebi('filter').value = V.filter;
-        rows = [];
+    ebi('hits').innerHTML = 'showing ' + ups.length + ' files';
 
-    for (var a = 0; a < tr.length; a++) {
+    for (var a = 0; a < ups.length; a++) {
-        var td = tr[a].cells,
+        var f = ups[a],
-            an = td[5].children[0];
+            vsp = vsplit(f.vp.split('?')[0]),
+            dn = esc(uricom_dec(vsp[0])),
+            fn = esc(uricom_dec(vsp[1])),
+            at = f.at,
+            td = now - f.at,
+            ts = !at ? '(?)' : unix2iso(at),
+            sa = !at ? '(?)' : td > 60 ? shumantime(td) : (td + 's'),
+            sz = ('' + f.sz).replace(/\B(?=(\d{3})+(?!\d))/g, " ");
 
-        rows.push([
+        html.push('<tr><td>' + sz +
-            td[0].textContent,
+            '</td><td>' + f.ip +
-            td[2].textContent,
+            '</td><td>' + ts +
-            td[3].textContent,
+            '</td><td>' + sa +
-            an.textContent,
+            '</td><td><a href="' + vsp[0] + '">' + dn +
-            an.getAttribute('href'),
+            '</a></td><td><a href="' + f.vp + '">' + fn +
-        ]);
+            '</a></td></tr>');
     }
-
+    if (!ups.length) {
-    for (var a = 0; a < rows.length; a++) {
+        var t = V.filter ? ' matching the filter' : '';
-        var t = rows[a],
+        html = ['<tr><td colspan="6">there are no uploads' + t + '</td></tr>'];
-            sz = t[0],
-            at = parseInt(t[1]),
-            nam = vsplit(t[3]),
-            dh = vsplit(t[4])[0];
-
-        tr[a].cells[0].innerHTML = sz.replace(/\B(?=(\d{3})+(?!\d))/g, " ");
-        tr[a].cells[2].innerHTML = at ? unix2iso(at) : '(?)';
-        tr[a].cells[3].innerHTML = at ? shumantime(t[2]) : '(?)';
-        tr[a].cells[4].innerHTML = '<a href="' + dh + '">' + nam[0] + '</a>';
-        tr[a].cells[5].children[0].innerHTML = nam[1].split('?')[0];
     }
+    ebi('tb').innerHTML = html.join('');
+}
+render();
 
-    ebi('hits').innerHTML = '-- showing ' + rows.length + ' files';
+var ti;
-})();
+function ask(e) {
+    ev(e);
+    clearTimeout(ti);
+    ebi('hits').innerHTML = 'Loading...';
+
+    var xhr = new XHR(),
+        filter = unsmart(ebi('filter').value);
+
+    hist_replace(get_evpath().split('?')[0] + '?ru&filter=' + uricom_enc(filter));
+
+    xhr.onload = xhr.onerror = function () {
+        try {
+            V = JSON.parse(this.responseText)
+        }
+        catch (ex) {
+            ebi('tb').innerHTML = '<tr><td colspan="6">failed to decode server response as json: <pre>' + esc(this.responseText) + '</pre></td></tr>';
+            return;
+        }
+        render();
+    };
+    xhr.open('GET', SR + '/?ru&j&filter=' + uricom_enc(filter), true);
+    xhr.send();
+}
+ebi('re').onclick = ask;
+ebi('filter').oninput = function () {
+    clearTimeout(ti);
+    ti = setTimeout(ask, 500);
+    ebi('hits').innerHTML = '...';
+};
+ebi('filter').onkeydown = function (e) {
+    if (('' + e.key).endsWith('Enter'))
+        ask();
+};

copyparty/web/shares.html

@@ -6,6 +6,7 @@
 	<title>{{ s_doctitle }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/shares.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
@@ -14,8 +15,8 @@
 
 <body>
 	<div id="wrap">
-		<a id="a" href="{{ r }}/?shares" class="af">refresh</a>
+		<a href="{{ r }}/?shares">refresh</a>
-		<a id="a" href="{{ r }}/?h" class="af">control-panel</a>
+		<a href="{{ r }}/?h">control-panel</a>
 
         <span>axs = perms (read,write,move,delet)</span>
         <span>nf = numFiles (0=dir)</span>
@@ -62,7 +63,7 @@
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
     shr="{{ shr }}",
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";

copyparty/web/splash.html

@@ -168,7 +168,7 @@
 	{%- endif %}
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";
 

copyparty/web/svcs.html

@@ -9,7 +9,7 @@
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/splash.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
-	<style>ul{padding-left:1.3em}li{margin:.4em 0}</style>
+	<style>ul{padding-left:1.3em}li{margin:.4em 0}.txa{float:right;margin:0 0 0 1em}</style>
 {{ html_head }}
 </head>
 
@@ -31,15 +31,22 @@
             <br />
             <span class="os win lin mac">placeholders:</span>
             <span class="os win">
-                {% if accs %}<code><b>{{ pw }}</b></code>=password, {% endif %}<code><b>W:</b></code>=mountpoint
+                {% if accs %}<code><b id="pw0">{{ pw }}</b></code>=password, {% endif %}<code><b>W:</b></code>=mountpoint
             </span>
             <span class="os lin mac">
-                {% if accs %}<code><b>{{ pw }}</b></code>=password, {% endif %}<code><b>mp</b></code>=mountpoint
+                {% if accs %}<code><b id="pw0">{{ pw }}</b></code>=password, {% endif %}<code><b>mp</b></code>=mountpoint
             </span>
+            <a href="#" id="setpw">use real password</a>
         </p>
 
 
 
+        {% if args.idp_h_usr %}
+        <p style="line-height:2em"><b>WARNING:</b> this server is using IdP-based authentication, so this stuff may not work as advertised. Depending on server config, these commands can probably only be used to access areas which don't require authentication, unless you auth using any non-IdP accounts defined in the copyparty config. Please see <a href="https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients">the IdP docs</a></p>
+        {% endif %}
+
+
+
         {% if not args.no_dav %}
         <h1>WebDAV</h1>
 
@@ -229,11 +236,65 @@
 
 
 
+        <div class="os win">
+            <h1>ShareX</h1>
+
+            <p>to upload screenshots using ShareX <a href="https://github.com/ShareX/ShareX/releases/tag/v12.1.1">v12</a> or <a href="https://getsharex.com/">v15+</a>, save this as <code>copyparty.sxcu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.sxcu">
+                { "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>",{% endif %}
+                    "accept": "url"
+                },
+                "DestinationType": "ImageUploader, TextUploader, FileUploader",
+                "FileFormName": "f" }
+            </pre>
+        </div>
+
+
+
+        <div class="os mac">
+            <h1>ishare</h1>
+
+            <p>to upload screenshots using <a href="https://isharemac.app/">ishare</a>, save this as <code>copyparty.iscu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.iscu">
+                { "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>",{% endif %}
+                    "accept": "json"
+                },
+                "ResponseURL": "{{ '{{fileurl}}' }}",
+                "FileFormName": "f" }
+            </pre>
+        </div>
+
+
+
+        <div class="os lin">
+            <h1>flameshot</h1>
+
+            <p>to upload screenshots using <a href="https://flameshot.org/">flameshot</a>, save this as <code>flameshot.sh</code> and run it:</p>
+
+            <pre class="dl" name="flameshot.sh">
+                #!/bin/bash
+                pw="<b>{{ pw }}</b>"
+                url="http{{ s }}://{{ ep }}/{{ rvp }}"
+                filename="$(date +%Y-%m%d-%H%M%S).png"
+                flameshot gui -s -r | curl -sT- "$url$filename?want=url&pw=$pw" | xsel -ib
+            </pre>
+        </div>
+
+
+
     </div>
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
     lang="{{ lang }}",
 	dfavico="{{ favico }}";
 

copyparty/web/svcs.js

@@ -1,11 +1,3 @@
-function QSA(x) {
-    return document.querySelectorAll(x);
-}
-var LINUX = /Linux/.test(navigator.userAgent),
-    MACOS = /[^a-z]mac ?os/i.test(navigator.userAgent),
-    WINDOWS = /Windows/.test(navigator.userAgent);
-
-
 var oa = QSA('pre');
 for (var a = 0; a < oa.length; a++) {
     var html = oa[a].innerHTML,
@@ -15,6 +7,21 @@ for (var a = 0; a < oa.length; a++) {
     oa[a].innerHTML = html.replace(rd, '$1').replace(/[ \r\n]+$/, '').replace(/\r?\n/g, '<br />');
 }
 
+function add_dls() {
+    oa = QSA('pre.dl');
+    for (var a = 0; a < oa.length; a++) {
+        var an = 'ta' + a,
+            o = ebi(an) || mknod('a', an, 'download');
+
+        oa[a].setAttribute('id', 'tx' + a);
+        oa[a].parentNode.insertBefore(o, oa[a]);
+        o.setAttribute('download', oa[a].getAttribute('name'));
+        o.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(oa[a].innerText));
+        clmod(o, 'txa', 1);
+    }
+}
+add_dls();
+
 
 oa = QSA('.ossel a');
 for (var a = 0; a < oa.length; a++)
@@ -40,3 +47,21 @@ function setos(os) {
 }
 
 setos(WINDOWS ? 'win' : LINUX ? 'lin' : MACOS ? 'mac' : 'idk');
+
+
+ebi('setpw').onclick = function (e) {
+    ev(e);
+    modal.prompt('password:', '', function (v) {
+        if (!v)
+            return;
+
+        var pw0 = ebi('pw0').innerHTML,
+            oa = QSA('b');
+        
+        for (var a = 0; a < oa.length; a++)
+            if (oa[a].innerHTML == pw0)
+                oa[a].textContent = v;
+
+        add_dls();
+    });
+}

copyparty/web/up2k.js

@@ -881,7 +881,7 @@ function up2k_init(subtle) {
     bcfg_bind(uc, 'turbo', 'u2turbo', turbolvl > 1, draw_turbo);
     bcfg_bind(uc, 'datechk', 'u2tdate', turbolvl < 3, null);
     bcfg_bind(uc, 'az', 'u2sort', u2sort.indexOf('n') + 1, set_u2sort);
-    bcfg_bind(uc, 'hashw', 'hashw', !!WebAssembly && (!subtle || !CHROME || MOBILE || VCHROME >= 107), set_hashw);
+    bcfg_bind(uc, 'hashw', 'hashw', !!WebAssembly && !(CHROME && MOBILE) && (!subtle || !CHROME), set_hashw);
     bcfg_bind(uc, 'upnag', 'upnag', false, set_upnag);
     bcfg_bind(uc, 'upsfx', 'upsfx', false, set_upsfx);
 
@@ -969,7 +969,7 @@ function up2k_init(subtle) {
             ud = function () { ebi('dir' + fdom_ctr).click(); };
 
         // too buggy on chrome <= 72
-        var m = / Chrome\/([0-9]+)\./.exec(navigator.userAgent);
+        var m = / Chrome\/([0-9]+)\./.exec(UA);
         if (m && parseInt(m[1]) < 73)
             return uf();
 
@@ -1972,32 +1972,84 @@ function up2k_init(subtle) {
             nchunk = 0,
             chunksize = get_chunksize(t.size),
             nchunks = Math.ceil(t.size / chunksize),
+            csz_mib = chunksize / 1048576,
+            tread = t.t_hashing,
+            cache_buf = null,
+            cache_car = 0,
+            cache_cdr = 0,
+            hashers = 0,
             hashtab = {};
 
+        // resolving subtle.digest w/o worker takes 1sec on blur if the actx hack breaks
+        var use_workers = hws.length && !hws_ng && uc.hashw && (nchunks > 1 || document.visibilityState == 'hidden'),
+            hash_par = (!subtle && !use_workers) ? 0 : csz_mib < 48 ? 2 : csz_mib < 96 ? 1 : 0;
+
         pvis.setab(t.n, nchunks);
         pvis.move(t.n, 'bz');
 
-        if (hws.length && !hws_ng && uc.hashw && (nchunks > 1 || document.visibilityState == 'hidden'))
+        if (use_workers)
-            // resolving subtle.digest w/o worker takes 1sec on blur if the actx hack breaks
             return wexec_hash(t, chunksize, nchunks);
 
         var segm_next = function () {
             if (nchunk >= nchunks || bpend)
                 return false;
 
-            var reader = new FileReader(),
+            var nch = nchunk++,
-                nch = nchunk++,
                 car = nch * chunksize,
                 cdr = Math.min(chunksize + car, t.size);
 
             st.bytes.hashed += cdr - car;
             st.etac.h++;
 
-            var orz = function (e) {
+            if (MOBILE && CHROME && st.slow_io === null && nch == 1 && cdr - car >= 1024 * 512) {
-                bpend--;
+                var spd = Math.floor((cdr - car) / (Date.now() + 1 - tread));
-                segm_next();
+                st.slow_io = spd < 40 * 1024;
-                hash_calc(nch, e.target.result);
+                console.log('spd {0}, slow: {1}'.format(spd, st.slow_io));
             }
+
+            if (cdr <= cache_cdr && car >= cache_car) {
+                try {
+                    var ofs = car - cache_car,
+                        ofs2 = ofs + (cdr - car),
+                        buf = cache_buf.subarray(ofs, ofs2);
+
+                    hash_calc(nch, buf);
+                }
+                catch (ex) {
+                    vis_exh(ex + '', 'up2k.js', '', '', ex);
+                }
+                return;
+            }
+
+            var reader = new FileReader(),
+                fr_cdr = cdr;
+
+            if (st.slow_io) {
+                var step = cdr - car,
+                    tgt = 48 * 1048576;
+
+                while (step && fr_cdr - car < tgt)
+                    fr_cdr += step;
+                if (fr_cdr - car > tgt && fr_cdr > cdr)
+                    fr_cdr -= step;
+                if (fr_cdr > t.size)
+                    fr_cdr = t.size;
+            }
+
+            var orz = function (e) {
+                bpend = 0;
+                var buf = e.target.result;
+                if (fr_cdr > cdr) {
+                    cache_buf = new Uint8Array(buf);
+                    cache_car = car;
+                    cache_cdr = fr_cdr;
+                    buf = cache_buf.subarray(0, cdr - car);
+                }
+                if (hashers < hash_par)
+                    segm_next();
+
+                hash_calc(nch, buf);
+            };
             reader.onload = function (e) {
                 try { orz(e); } catch (ex) { vis_exh(ex + '', 'up2k.js', '', '', ex); }
             };
@@ -2024,17 +2076,20 @@ function up2k_init(subtle) {
 
                 toast.err(0, 'y o u   b r o k e    i t\nfile: ' + esc(t.name + '') + '\nerror: ' + err);
             };
-            bpend++;
+            bpend = 1;
-            reader.readAsArrayBuffer(t.fobj.slice(car, cdr));
+            tread = Date.now();
+            reader.readAsArrayBuffer(t.fobj.slice(car, fr_cdr));
 
             return true;
         };
 
         var hash_calc = function (nch, buf) {
+            hashers++;
             var orz = function (hashbuf) {
                 var hslice = new Uint8Array(hashbuf).subarray(0, 33),
                     b64str = buf2b64(hslice);
 
+                hashers--;
                 hashtab[nch] = b64str;
                 t.hash.push(nch);
                 pvis.hashed(t);
@@ -3044,7 +3099,7 @@ function up2k_init(subtle) {
                 new_state = false;
                 fixed = true;
             }
-            if (new_state === undefined)
+            if (new_state === undefined && preferred === undefined)
                 new_state = can_write ? false : have_up2k_idx ? true : undefined;
         }
 

copyparty/web/util.js

@@ -29,14 +29,15 @@ var wah = '',
     HTTPS = ('' + location).indexOf('https:') === 0,
     TOUCH = 'ontouchstart' in window,
     MOBILE = TOUCH,
-    CHROME = !!window.chrome,
+    CHROME = !!window.chrome,  // safari=false
     VCHROME = CHROME ? 1 : 0,
-    IE = /Trident\//.test(navigator.userAgent),
+    UA = '' + navigator.userAgent,
-    FIREFOX = ('netscape' in window) && / rv:/.test(navigator.userAgent),
+    IE = /Trident\//.test(UA),
-    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(navigator.userAgent),
+    FIREFOX = ('netscape' in window) && / rv:/.test(UA),
-    LINUX = /Linux/.test(navigator.userAgent),
+    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(UA),
-    MACOS = /[^a-z]mac ?os/i.test(navigator.userAgent),
+    LINUX = /Linux/.test(UA),
-    WINDOWS = /Windows/.test(navigator.userAgent);
+    MACOS = /Macintosh/.test(UA),
+    WINDOWS = /Windows/.test(UA);
 
 if (!window.WebAssembly || !WebAssembly.Memory)
     window.WebAssembly = false;
@@ -196,7 +197,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
         '<p style="font-size:1.3em;margin:0;line-height:2em">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a> / <a href="?b=u">basic</a></p>',
         '<p style="color:#fff">please send me a screenshot arigathanks gozaimuch: <a href="<ghi>" target="_blank">new github issue</a></p>',
         '<p class="b">' + esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(msg).replace(/\n/g, '<br />') + '</p>',
-        '<p><b>UA:</b> ' + esc(navigator.userAgent + '')
+        '<p><b>UA:</b> ' + esc(UA)
     ];
 
     try {

docs/README.md

@@ -25,6 +25,9 @@
 ## [`changelog.md`](changelog.md)
 * occasionally grabbed from github release notes
 
+## [`synology-dsm.md`](synology-dsm.md)
+* running copyparty on a synology nas
+
 ## [`devnotes.md`](devnotes.md)
 * technical stuff
 

docs/changelog.md

@@ -1,3 +1,131 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0122-2326  `v1.16.9`  ZeroMQ says hello
+
+## 🧪 new features
+
+* event-hooks can send zeromq / zmq / 0mq messages; see [readme](https://github.com/9001/copyparty#zeromq) or `--help-hooks` for examples d9db1534
+* new volflags to specify the [allow-tag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes) of the markdown/logue sandbox, to allow fullscreen and such (see `--help-flags`) 6a0aaaf0
+* new volflag `nosparse` for possibly-better performance in very rare and specific scenarios 917380dd
+  * only enable this if you're uploading to s3 or something like that, and do plenty of benchmarking to make sure that it actually improved performance instead of making it worse
+
+## 🩹 bugfixes
+
+* restrict max-length of filekeys to 72 characters e0cac6fd
+* the hash-calculator mode of the commandline uploader produced incorrect whole-file hashes 4c04798a
+  * each chunk (`--chs`) was okay, but the final sum was not
+
+## 🔧 other changes
+
+* selftest the xml-parser on startup with malicious xml b2e8bf6e
+  * just in case a future python-version suddenly makes it unsafe somehow
+* disable some features if a dangerously misconfigured reverseproxy is detected 3f84b0a0
+* the download-as-zip feature now defaults to utf8 filenames 1231ce19
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0111-1611  `v1.16.8`  android boost
+
+## 🧪 new features
+
+* 10x faster file hashing in android-chrome ec507889
+  * on a recent pixel, speed went from 13 to 139 MiB/s
+  * android's sandboxing makes small reads expensive, so do bigger reads instead
+    * so the browser-tab will use more RAM on android now, maybe around 200 MiB
+    * this only affects chrome-based browsers on android, not firefox
+* PUT/multipart uploads: request-header `Accept: json` makes it return json instead of html, just like `?j` ce0e5be4
+* add config examples for [ishare](https://isharemac.app/), a MacOS screenshot utility inspired by ShareX 0c0d6b2b
+  * also includes a bug-workaround for [ishare#107](https://github.com/castdrian/ishare/issues/107) - copyparty will now include a toplevel json property `fileurl` in the response if exactly one file was uploaded
+  * the [connect-page](https://a.ocv.me/?hc) generates an appropriate `copyparty.iscu` for ishare; [it looks like this](https://github.com/user-attachments/assets/820730ad-2319-4912-8eb2-733755a4cf54)
+
+## 🩹 bugfixes
+
+* fix a potential upload deadlock when...
+  * ...the database (`-e2d`) is **not** enabled for any volume, and...
+  * ...either the shares feature, or user-changeable passwords, is enabled 9e542cf8
+* when loading the partial-uploads registry on startup, a cosmetic desync could occur 467acb47
+
+## 🔧 other changes
+
+* remove some deprecated properties in partial-upload metadata aa2a8fa2
+  * v1.15.7 is now the oldest version which still has any chance of reading a modern up2k.snap
+* #129 added howto: [using webdav when copyparty is behind IdP](https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients) -- thanks @wuast94 !
+* added howto: [install copyparty on a synology nas](https://github.com/9001/copyparty/blob/hovudstraum/docs/synology-dsm.md) 21f93042
+* more examples in the connect-page: 278258ee fb139697
+  * config-file for sharex on windows
+  * config-file for ishare on macos
+  * script for flameshot on linux
+* #75 add recommendation to use the [kamelåså project](https://github.com/steinuil/kameloso) instead of copyparty's [very-bad-idea.py](https://github.com/9001/copyparty/tree/hovudstraum/bin/mtag#dangerous-plugins) 9f84dc42
+* more reverse-proxy examples (haproxy, lighttpd, traefik, caddy) and improved nginx performance ac0a2da3
+  * readme has a [performance comparison](https://github.com/9001/copyparty?tab=readme-ov-file#reverse-proxy-performance) -- `haproxy > caddy > traefik > nginx > apache > lighttpd`
+* copyparty.exe: updated pillow 244e952f
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1223-0005  `v1.16.7`  an idp fix for xmas
+
+# ☃️🎄 **there is still time** 🎅🎁
+
+❄️❄️❄️ please [enjoy some appropriate music](https://a.ocv.me/pub/demo/music/.bonus/#af-55d4554d) -- you'll probably like this more than the idp thing honestly ❄️❄️❄️
+
+## 🧪 new features
+
+* more improvements to the recent-uploads feature 87598dcd
+  * move html rendering to clientside
+    * any changes to the filter-text applies in real-time
+    * loads 50% faster, reduces server-load by 30%
+    * inhibits search engines from indexing it
+
+## 🩹 bugfixes
+
+* using idp without e2d could mess with uploads dd6e9ea7
+* u2c (commandline uploader): fix window title 946a8c5b
+* mDNS/SSDP: fix incorrect log colors when multiple primary IPs are lost 552897ab
+
+## 🔧 other changes
+
+* ui: make it more obvious that the volume-control is a volume-control 7f044372
+* copyparty.exe: update deps (jinja2, markupsafe, pyinstaller) c0dacbc4
+* improve safety of custom plugins 988a7223
+  * if you've made your own plugins which expect certain values (host-header, filekeys) to be html-safe, then you'll want to upgrade
+  * also fixes rss-feed xml if password contains special characters
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1219-0037  `v1.16.6`  merry \x58mas
+
+# ☃️🎄 **it is time** 🎅🎁
+
+❄️❄️❄️ please [enjoy some appropriate music](https://a.ocv.me/pub/demo/music/.bonus/#af-55d4554d) (trust me on this one, you won't regret it) ❄️❄️❄️
+
+## 🧪 new features
+
+* [list of recent uploads](https://a.ocv.me/?ru) eaa4b04a
+  * new button in the controlpanel; can be disabled with `--no-ups-page`
+  * only users with the dot-permission can see dotfiles
+  * only admins can see uploader-ip and upload-times
+    * enable `--ups-when` to let all users see upload-times
+* #125 log decoded request-URLs 73f7249c
+  * non-ascii filenames would make the accesslog a wall of `%E5%B9%BB%E6%83%B3%E9%83%B7` so print [the decoded URL](https://github.com/user-attachments/assets/9d411183-30f3-4cb2-a880-84cf18011183) in addition to the original one, which is left as-is for debugging purposes
+
+## 🩹 bugfixes
+
+* #126 improve dotfile handling 4c4e48ba
+  * was impossible to delete a folder which contained hidden files if the user did not have the permission to see hidden files
+  * would also affect moving, renaming, copying folders, in which case the dotfiles would not be carried over to the new location
+  * now, dotfiles are always deleted, and always moved/copied into a new destination, on the condition that this is safe -- if the user has the dotfile permission in the target loocation but not in the source location, the dotfiles will be left behind to avoid accidentally making then browsable
+* ux: cosmetic eta/idle-timer fixes 01a3eb29
+
+## 🔧 other changes
+
+* warn on ambiguous comments in config files da5ad2ab
+* avoid writing mojibake to the log 3051b131
+  * use `\x`-encoding for unprintable text
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2024-1211-2236  `v1.16.5`  4chrome
 

docs/chunksizes.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+
+# there's far better ways to do this but its 4am and i dont wanna think
+
+# just pypy it my dude
+
+import math
+
+def humansize(sz, terse=False):
+    for unit in ["B", "KiB", "MiB", "GiB", "TiB"]:
+        if sz < 1024:
+            break
+
+        sz /= 1024.0
+
+    ret = " ".join([str(sz)[:4].rstrip("."), unit])
+
+    if not terse:
+        return ret
+
+    return ret.replace("iB", "").replace(" ", "")
+
+
+def up2k_chunksize(filesize):
+    chunksize = 1024 * 1024
+    stepsize = 512 * 1024
+    while True:
+        for mul in [1, 2]:
+            nchunks = math.ceil(filesize * 1.0 / chunksize)
+            if nchunks <= 256 or (chunksize >= 32 * 1024 * 1024 and nchunks <= 4096):
+                return chunksize
+
+            chunksize += stepsize
+            stepsize *= mul
+
+
+def main():
+    prev = 1048576
+    n = n0 = 524288
+    while True:
+        csz = up2k_chunksize(n)
+        if csz > prev:
+            print(f"| {n-n0:>18_} | {humansize(n-n0):>8} | {prev:>13_} | {humansize(prev):>8} |".replace("_", " "))
+            prev = csz
+        n += n0
+
+
+main()

docs/devnotes.md

@@ -6,6 +6,7 @@
     * [up2k](#up2k) - quick outline of the up2k protocol
         * [why not tus](#why-not-tus) - I didn't know about [tus](https://tus.io/)
         * [why chunk-hashes](#why-chunk-hashes) - a single sha512 would be better, right?
+        * [list of chunk-sizes](#list-of-chunk-sizes) - specific chunksizes are enforced
 * [hashed passwords](#hashed-passwords) - regarding the curious decisions
 * [http api](#http-api)
     * [read](#read)
@@ -95,6 +96,44 @@ hashwasm would solve the streaming issue but reduces hashing speed for sha512 (x
 
 * blake2 might be a better choice since xxh is non-cryptographic, but that gets ~15 MiB/s on slower androids
 
+### list of chunk-sizes
+
+specific chunksizes are enforced  depending on total filesize
+
+each pair of filesize/chunksize is the largest filesize which will use its listed chunksize; a 512 MiB file will use chunksize 2 MiB, but if the file is one byte larger than 512 MiB then it becomes 3 MiB
+
+for the purpose of performance (or dodging arbitrary proxy limitations), it is possible to upload combined and/or partial chunks using stitching and/or subchunks respectively
+
+| filesize           | filesize | chunksize     | chunksz |
+| -----------------: | -------: | ------------: | ------: |
+|        268 435 456 |  256 MiB |     1 048 576 | 1.0 MiB |
+|        402 653 184 |  384 MiB |     1 572 864 | 1.5 MiB |
+|        536 870 912 |  512 MiB |     2 097 152 | 2.0 MiB |
+|        805 306 368 |  768 MiB |     3 145 728 | 3.0 MiB |
+|      1 073 741 824 |  1.0 GiB |     4 194 304 | 4.0 MiB |
+|      1 610 612 736 |  1.5 GiB |     6 291 456 | 6.0 MiB |
+|      2 147 483 648 |  2.0 GiB |     8 388 608 | 8.0 MiB |
+|      3 221 225 472 |  3.0 GiB |    12 582 912 |  12 MiB |
+|      4 294 967 296 |  4.0 GiB |    16 777 216 |  16 MiB |
+|      6 442 450 944 |  6.0 GiB |    25 165 824 |  24 MiB |
+|    137 438 953 472 |  128 GiB |    33 554 432 |  32 MiB |
+|    206 158 430 208 |  192 GiB |    50 331 648 |  48 MiB |
+|    274 877 906 944 |  256 GiB |    67 108 864 |  64 MiB |
+|    412 316 860 416 |  384 GiB |   100 663 296 |  96 MiB |
+|    549 755 813 888 |  512 GiB |   134 217 728 | 128 MiB |
+|    824 633 720 832 |  768 GiB |   201 326 592 | 192 MiB |
+|  1 099 511 627 776 |  1.0 TiB |   268 435 456 | 256 MiB |
+|  1 649 267 441 664 |  1.5 TiB |   402 653 184 | 384 MiB |
+|  2 199 023 255 552 |  2.0 TiB |   536 870 912 | 512 MiB |
+|  3 298 534 883 328 |  3.0 TiB |   805 306 368 | 768 MiB |
+|  4 398 046 511 104 |  4.0 TiB | 1 073 741 824 | 1.0 GiB |
+|  6 597 069 766 656 |  6.0 TiB | 1 610 612 736 | 1.5 GiB |
+|  8 796 093 022 208 |  8.0 TiB | 2 147 483 648 | 2.0 GiB |
+| 13 194 139 533 312 | 12.0 TiB | 3 221 225 472 | 3.0 GiB |
+| 17 592 186 044 416 | 16.0 TiB | 4 294 967 296 | 4.0 GiB |
+| 26 388 279 066 624 | 24.0 TiB | 6 442 450 944 | 6.0 GiB |
+| 35 184 372 088 832 | 32.0 TiB | 8 589 934 592 | 8.0 GiB |
+
 
 # hashed passwords
 
@@ -133,8 +172,8 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | GET | `?tar=xz:9` | ...as an xz-level-9 gnu-tar file |
 | GET | `?tar=pax` | ...as a pax-tar file |
 | GET | `?tar=pax,xz` | ...as an xz-level-1 pax-tar file |
-| GET | `?zip=utf-8` | ...as a zip file |
+| GET | `?zip` | ...as a zip file |
-| GET | `?zip` | ...as a WinXP-compatible zip file |
+| GET | `?zip=dos` | ...as a WinXP-compatible zip file |
 | GET | `?zip=crc` | ...as an MSDOS-compatible zip file |
 | GET | `?tar&w` | pregenerate webp thumbnails |
 | GET | `?tar&j` | pregenerate jpg thumbnails |
@@ -173,6 +212,7 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | method | params | body | result |
 |--|--|--|--|
 | PUT | | (binary data) | upload into file at URL |
+| PUT | `?j` | (binary data) | ...and reply with json |
 | PUT | `?ck` | (binary data) | upload without checksum gen (faster) |
 | PUT | `?ck=md5` | (binary data) | return md5 instead of sha512 |
 | PUT | `?gz` | (binary data) | compress with gzip and write into file at URL |
@@ -197,6 +237,7 @@ upload modifiers:
 | http-header | url-param | effect |
 |--|--|--|
 | `Accept: url` | `want=url` | return just the file URL |
+| `Accept: json` | `want=json` | return upload info as json; same as `?j` |
 | `Rand: 4` | `rand=4` | generate random filename with 4 characters |
 | `Life: 30` | `life=30` | delete file after 30 seconds |
 | `CK: no` | `ck` | disable serverside checksum (maybe faster) |
@@ -301,6 +342,7 @@ python3 -m venv .venv
 . .venv/bin/activate
 pip install jinja2 strip_hints  # MANDATORY
 pip install argon2-cffi  # password hashing
+pip install pyzmq  # send 0mq from hooks
 pip install mutagen  # audio metadata
 pip install pyftpdlib  # ftp server
 pip install partftpy  # tftp server

docs/idp.md

@@ -20,3 +20,25 @@ this means that, if an IdP volume is located inside a folder that is readable by
 and likewise -- if the IdP volume is inside a folder that is only accessible by certain users, but the IdP volume is configured to allow access from unauthenticated users, then the contents of the volume will NOT be accessible until it is revived
 
 until this limitation is fixed (if ever), it is recommended to place IdP volumes inside an appropriate parent volume, so they can inherit acceptable permissions until their revival; see the "strategic volumes" at the bottom of [./examples/docker/idp/copyparty.conf](./examples/docker/idp/copyparty.conf)
+
+
+## Connecting webdav clients
+
+If you use only idp and want to connect via rclone you have to adapt a few things.
+The following steps are for Authelia, but should be easy adaptable to other IdPs and clients. There may be better/smarter ways to do this, but this is a known solution.
+
+1. Add a rule for your domain and set it to one factor
+```
+  rules:
+    - domain: 'sub.domain.tld'
+      policy: one_factor
+```
+2. After you created your rclone config find its location with `rclone config file` and add the headers option to it, change the string to `username:password` base64 encoded. Make sure to set the right url location, otherwise you will get a 401 from copyparty.
+```
+[servername-dav]
+type = webdav
+url = https://sub.domain.tld/u/user/priv/
+vendor = owncloud
+pacer_min_sleep = 0.01ms
+headers = Proxy-Authorization,basic base64encodedstring==
+```

docs/notes.sh

@@ -259,6 +259,12 @@ for d in /usr /var; do find $d -type f -size +30M 2>/dev/null; done | while IFS=
 for f in {0..255}; do echo $f; truncate -s 256M $f; b1=$(printf '%02x' $f); for o in {0..255}; do b2=$(printf '%02x' $o); printf "\x$b1\x$b2" | dd of=$f bs=2 seek=$((o*1024*1024)) conv=notrunc 2>/dev/null; done; done
 # create 6.06G file with 16 bytes of unique data at start+end of each 32M chunk
 sz=6509559808; truncate -s $sz f; csz=33554432; sz=$((sz/16)); step=$((csz/16)); ofs=0; while [ $ofs -lt $sz ]; do dd if=/dev/urandom of=f bs=16 count=2 seek=$ofs conv=notrunc iflag=fullblock; [ $ofs = 0 ] && ofs=$((ofs+step-1)) || ofs=$((ofs+step)); done
+# same but for chunksizes 16M (3.1G), 24M (4.1G), 48M (128.1G)
+sz=3321225472;   csz=16777216;
+sz=4394967296;   csz=25165824;
+sz=6509559808;   csz=33554432;
+sz=138438953472; csz=50331648;
+f=csz-$csz; truncate -s $sz $f; sz=$((sz/16)); step=$((csz/16)); ofs=0; while [ $ofs -lt $sz ]; do dd if=/dev/urandom of=$f bs=16 count=2 seek=$ofs conv=notrunc iflag=fullblock; [ $ofs = 0 ] && ofs=$((ofs+step-1)) || ofs=$((ofs+step)); done
 
 # py2 on osx
 brew install python@2

docs/synology-dsm.md

@@ -0,0 +1,140 @@
+# running copyparty on synology dsm nas
+
+![synology-dsm-container-status.png](https://ocv.me/copyparty/doc/pics/dsm.png)
+
+this has been tested on a `Synology ds218+` NAS with 1 SHR storage-pool and 1 volume, but the same steps should work in more advanced setups too
+
+verified on DSM 7.1 and 7.2, but not on 6.x since my flea-market ds218+ refuses to install it for some reason
+
+
+
+# ok let's go
+
+go to controlpanel -> shared-folders, and create the following shared-folders if you don't already have appropriate ones:
+
+* a shared-folder for configuration files, preferably on SSD if you have one
+
+* one or more shared-folders for your actual data/media to share
+
+(btw, when you create the shared-folders, it asks whether you want to enable data checksum and file compression, i would recommend both)
+
+the rest of this doc assumes that these two shared-folders are named `configs` and `media1`, and that you made an empty folder inside the `configs` shared-folder named `cpp`
+
+* your copyparty config file (see below) should be named `something.conf` directly inside that cpp folder, for example `/configs/cpp/copyparty.conf`
+
+* during first start, copyparty will create a folder there named `copyparty`, in other words `/configs/cpp/copyparty` which you should leave alone; that's where copyparty stores its indexes and other runtime config
+
+
+
+## recommended copyparty config
+
+open the Package Center and install `Text Editor` (by Synology Inc.) to create and edit your copyparty config:
+
+![synology-text-editor-copyparty-conf.png](https://ocv.me/copyparty/doc/pics/dsm-cfg.png)
+
+* note the `copyparty` and `hist` folders in that screenshot which are autogenerated by copyparty and to be left alone
+
+```yaml
+[global]
+  e2d, e2t         # remember uploads & read media tags
+  rss, daw, ver    # some other nice-to-have features
+  #dedup            # you may want this, or maybe not
+  hist: /cfg/hist  # don't pollute the shared-folder
+  name: synology   # shows in the browser, can be anything
+
+[accounts]
+  ed: wark   # username ed, password wark
+
+[/]          # share the following at the webroot:
+  /w         # the "/w" docker-volume (the shared-folder)
+  accs:
+    A: ed    # give Admin to username ed
+
+# hide the synology system files by creating a hidden volume
+[/@eaDir]
+  /w/@eaDir
+```
+
+if you ever change the copyparty config file, then [restart the container](https://ocv.me/copyparty/doc/pics/dsm71-02.png) to make the changes take effect
+
+okay now continue with one of these:
+
+* [DSM v7.2 or newer](#dsm-v72-or-newer)
+
+* [all older DSM versions](#dsm-v6x-dsm-v71x-or-older)
+
+
+
+# DSM v7.2 or newer
+
+`Docker` was replaced by `Container Manager` in DSM v7.2 but they're almost the same thing;
+
+* open the `Package Center` and install the [Container Manager package](https://ocv.me/copyparty/doc/pics/dsm72-01.png) by `Docker Inc.`
+* open the `Container Manager` app
+* go to the `Registry` tab and search for `copyparty`
+* [doubleclick copyparty/ac](https://ocv.me/copyparty/doc/pics/dsm72-02.png) and keep the [default `latest`](https://ocv.me/copyparty/doc/pics/dsm72-03.png) when it asks you which tag to use
+* switch to the `Container` tab and click `Create`
+* [choose `copyparty/ac:latest`](https://ocv.me/copyparty/doc/pics/dsm72-04.png) and click `Next`
+
+finally, in the [Advanced Settings](https://ocv.me/copyparty/doc/pics/dsm72-05.png) window,
+
+* under `Port Settings`, type `3923` into the `Local Port` textbox
+* click `Add Folder` and select `/configs/cpp` on your nas (the `cpp` folder in the `configs` shared-folder), and change `Mount path` to `/cfg`
+* click `Add Folder` and select `/media1` on your nas (the shared-folder that copyparty can share in its web-UI) and change `Mount path` to `/w`
+  * if you are adding multiple shared-folders for media, then the `Mount path` of the 2nd folder should be something like `/w/share2` or `/w/music`
+
+copyparty will launch and become available at http://192.168.1.9:3923/ (assuming `192.168.1.9` is your nas ip)
+
+
+# DSM v6.x, DSM v7.1.x or older
+
+if you're using DSM 7.1 or older, then you don't have [Container Manager](https://www.synology.com/en-global/dsm/packages/ContainerManager) yet and you'll have to use [Docker](https://www.synology.com/en-global/dsm/packages/Docker?os_ver=6.2&search=docker) instead. Here's how:
+
+* open the `Package Center` and install the [Docker package](https://ocv.me/copyparty/doc/pics/dsm71-01.png) by `Docker Inc.`
+* open the `Docker` app
+* go to the `Registry` tab and search for `copyparty`
+* [doubleclick copyparty/ac](https://ocv.me/copyparty/doc/pics/dsm71-02.png) and keep the [default `latest`](https://ocv.me/copyparty/doc/pics/dsm71-03.png) when it asks you which tag to use
+* switch to the `Container` tab and click `Create`
+* [choose `copyparty/ac:latest`](https://ocv.me/copyparty/doc/pics/dsm71-04.png) and `Next`
+* in the [Network](https://ocv.me/copyparty/doc/pics/dsm71-05.png) window, keep the default `Use the selected networks: [x] bridge`
+* in the [General Settings](https://ocv.me/copyparty/doc/pics/dsm71-06.png) window, just keep everything default (in other words, everything disabled)
+* in the [Port Settings](https://ocv.me/copyparty/doc/pics/dsm71-07.png) window, change `Local Port` to `3923` (or choose something else, but it cannot be the default `Auto`)
+
+finally, in the [Volume Settings](https://ocv.me/copyparty/doc/pics/dsm71-08.png) window, add a docker volume for copyparty config, and at least one volume for media-files which copyparty can share in its web-UI
+
+* click `Add Folder` and select `/configs/cpp` on your nas (the `cpp` folder in the `configs` shared-folder), and change `Mount path` to `/cfg`
+* click `Add Folder` and select `/media1` on your nas (the shared-folder that copyparty can share in its web-UI) and change `Mount path` to `/w`
+* if you are adding multiple shared-folders for media, then the `Mount path` of the 2nd folder should be something like `/w/share2` or `/w/music`
+
+copyparty will launch and become available at http://192.168.1.9:3923/ (assuming `192.168.1.9` is your nas ip)
+
+
+# misc notes
+
+note that if you only want to share some folders inside your data volume, and not all of it, then you can either give copyparty the whole shared-folder anyways and control/restrict access in the copyparty config file (recommended), or you can add each folder as a new docker volume (not as flexible)
+
+
+
+## regarding ram usage
+
+the ram usage indicator in both `Docker` and `Container Manager` is misleading  because it also counts the kernel disk cache which makes the number insanely high -- the synology resource monitor shows the correct values, usually less than 100 MiB
+
+to see the actual memory usage by copyparty, see `Resource Monitor` -> `Task Manager` -> `Processes` and look at the `Private Memory` of `python3` which is probably copyparty
+
+
+
+## regarding performance
+
+when uploading files to the synology nas with the respective web-UIs,
+
+* `File Station` does about 16 MiB/s,
+
+* `Synology Drive Server` does about 50 MiB/s; deceivingly fast upload speeds at first, but when the file is fully uploaded, there is a lengthy "processing" step at the end, reducing the average speed to about 50% of the initial
+
+* copyparty maxes the HDD write-speeds, 99 MiB/s
+
+when uploading to the synology nas over webdav,
+
+* `WebDAV Server` by `Synology Inc.` in the Package Center does 86 MiB/s
+
+* copyparty does 79 MiB/s; the NAS CPU is a bottleneck because copyparty verifies the upload checksum while `WebDAV Server` doesn't

docs/versus.md

@@ -279,7 +279,7 @@ symbol legend,
 | per-file passwords      | █ |   |   | █ | █ |   | █ |   | █ |   |   |   | █ |
 | unmap subfolders        | █ |   | █ |   |   |   | █ |   |   | █ | ╱ | • |   |
 | index.html blocks list  | ╱ |   |   |   |   |   | █ |   |   | • |   |   |   |
-| write-only folders      | █ |   | █ |   |   |   |   |   |   |   | █ | █ |   |
+| write-only folders      | █ |   | █ |   | █ |   |   |   |   |   | █ | █ |   |
 | files stored as-is      | █ | █ | █ | █ |   | █ | █ |   |   | █ | █ | █ | █ |
 | file versioning         |   |   |   | █ | █ |   |   |   |   |   |   |   |   |
 | file encryption         |   |   |   | █ | █ | █ |   |   |   |   |   | █ |   |
@@ -507,7 +507,6 @@ symbol legend,
 * ⚠️ uploads not resumable / accelerated / integrity-checked
   * ⚠️ on cloudflare: max upload size 100 MiB
 * ⚠️ uploading small files is slow; `4.7` files per sec (copyparty does `670`/sec, 140x faster)
-* ⚠️ no write-only / upload-only folders
 * ⚠️ big folders cannot be zip-downloaded
 * ⚠️ http/webdav only; no ftp, zeroconf
 * ⚠️ less awesome music player
@@ -593,6 +592,7 @@ symbol legend,
 * ✅ user signup
 * ✅ command runner / remote shell
 * ✅ more efficient; can handle around twice as much simultaneous traffic
+* note: keep an eye on [gtsteffaniak's fork](https://github.com/gtsteffaniak/filebrowser)
 
 ## [filegator](https://github.com/filegator/filegator)
 * php; cross-platform (windows, linux, mac)

pyproject.toml

@@ -52,6 +52,7 @@ ftpd = ["pyftpdlib"]
 ftps = ["pyftpdlib", "pyopenssl"]
 tftpd = ["partftpy>=0.4.0"]
 pwhash = ["argon2-cffi"]
+zeromq = ["pyzmq"]
 
 [project.scripts]
 copyparty = "copyparty.__main__:main"

scripts/docker/Dockerfile.ac

@@ -9,7 +9,7 @@ ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk --no-cache add !pyc \
             tzdata wget \
-            py3-jinja2 py3-argon2-cffi py3-pillow \
+            py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
             ffmpeg
 
 COPY    i/dist/copyparty-sfx.py innvikler.sh ./

scripts/docker/Dockerfile.dj

@@ -12,7 +12,8 @@ COPY    i/bin/mtag/audio-bpm.py /mtag/
 COPY    i/bin/mtag/audio-key.py /mtag/
 RUN     apk add -U !pyc \
             tzdata wget \
-            py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
+            py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
+            py3-pip py3-cffi \
             ffmpeg \
             vips-jxl vips-heif vips-poppler vips-magick \
             py3-numpy fftw libsndfile \

scripts/docker/Dockerfile.iv

@@ -9,7 +9,8 @@ ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk add -U !pyc \
             tzdata wget \
-            py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
+            py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
+            py3-pip py3-cffi \
             ffmpeg \
             vips-jxl vips-heif vips-poppler vips-magick \
         && apk add -t .bd \

scripts/help2html.py

@@ -32,6 +32,7 @@ def readclip():
 def cnv(src):
     hostname = str(socket.gethostname()).split(".")[0]
 
+    yield '<!DOCTYPE html>'
     yield '<html style="background:#222;color:#fff"><body>'
     skip_sfx = False
     in_sfx = 0

scripts/pyinstaller/deps.sha512

@@ -1,7 +1,7 @@
 f117016b1e6a7d7e745db30d3e67f1acf7957c443a0dd301b6c5e10b8368f2aa4db6be9782d2d3f84beadd139bfeef4982e40f21ca5d9065cb794eeb0e473e82  altgraph-0.17.4-py2.py3-none-any.whl
 6a624018f30da375581d5751eca0080edbbe37f102f643f856279fcfded3a4379fd1b6fb0661cdb2e72bbbbc726ca714a1f5990cc348df365db62bc53e4c4503  Git-2.45.2-32-bit.exe
 17ce52ba50692a9d964f57a23ac163fb74c77fdeb2ca988a6d439ae1fe91955ff43730c073af97a7b3223093ffea3479a996b9b50ee7fba0869247a56f74baa6  pefile-2023.2.7-py3-none-any.whl
-749a473646c6d4c7939989649733d4c7699fd1c359c27046bf5bc9c070d1a4b8b986bbc65f60d7da725baf16dbfdd75a4c2f5bb8335f2cb5685073f5fee5c2d1  pywin32_ctypes-0.2.2-py3-none-any.whl
+b297ff66ec50cf5a1abcf07d6ac949644c5150ba094ffac974c5d27c81574c3e97ed814a47547f4b03a4c83ea0fb8f026433fca06a3f08e32742dc5c024f3d07  pywin32_ctypes-0.2.3-py3-none-any.whl
 085d39ef4426aa5f097fbc484595becc16e61ca23fc7da4d2a8bba540a3b82e789e390b176c7151bdc67d01735cce22b1562cdb2e31273225a2d3e275851a4ad  setuptools-70.3.0-py3-none-any.whl
 360a141928f4a7ec18a994602cbb28bbf8b5cc7c077a06ac76b54b12fa769ed95ca0333a5cf728923a8e0baeb5cc4d5e73e5b3de2666beb05eb477d8ae719093  upx-4.2.4-win32.zip
 # win7
@@ -23,11 +23,11 @@ ac96786e5d35882e0c5b724794329c9125c2b86ae7847f17acfc49f0d294312c6afc1c3f248655de
 # win10
 0a2cd4cadf0395f0374974cd2bc2407e5cc65c111275acdffb6ecc5a2026eee9e1bb3da528b35c7f0ff4b64563a74857d5c2149051e281cc09ebd0d1968be9aa  en-us_windows_10_enterprise_ltsc_2021_x64_dvd_d289cf96.iso
 16cc0c58b5df6c7040893089f3eb29c074aed61d76dae6cd628d8a89a05f6223ac5d7f3f709a12417c147594a87a94cc808d1e04a6f1e407cc41f7c9f47790d1  virtio-win-0.1.248.iso
-d1420c8417fad7888766dd26b9706a87c63e8f33dceeb8e26d0056d5127b0b3ed9272e44b4b761132d4b3320327252eab1696520488ca64c25958896b41f547b  jinja2-3.1.4-py3-none-any.whl
+18b9e8cfa682da51da1b682612652030bd7f10e4a1d5ea5220ab32bde734b0e6fe1c7dbd903ac37928c0171fd45d5ca602952054de40a4e55e9ed596279516b5  jinja2-3.1.5-py3-none-any.whl
-8e6847bcde75a2736be0214731f834bc1b5854238d703351e68bc4e74d38404b212b8568565ae22c844189e466d3fbe6024836351cb69ffb1824131387644fef  MarkupSafe-2.1.5-cp312-cp312-win_amd64.whl
+6df21f0da408a89f6504417c7cdf9aaafe4ed88cfa13e9b8fa8414f604c0401f885a04bbad0484dc51a29284af5d1548e33c6cc6bfb9896d9992c1b1074f332d  MarkupSafe-3.0.2-cp312-cp312-win_amd64.whl
 8a6e2b13a2ec4ef914a5d62aad3db6464d45e525a82e07f6051ed10474eae959069e165dba011aefb8207cdfd55391d73d6f06362c7eb247b08763106709526e  mutagen-1.47.0-py3-none-any.whl
 0203ec2551c4836696cfab0b2c9fff603352f03fa36e7476e2e1ca7ec57a3a0c24bd791fcd92f342bf817f0887854d9f072e0271c643de4b313d8c9569ba8813  packaging-24.1-py3-none-any.whl
-2be320b4191f208cdd6af183c77ba2cf460ea52164ee45ac3ff17d6dfa57acd9deff016636c2dd42a21f4f6af977d5f72df7dacf599bebcf41757272354d14c1  pillow-10.4.0-cp312-cp312-win_amd64.whl
+12d7921dc7dfd8a4b0ea0fa2bae8f1354fcdd59ece3d7f4e075aed631f9ba791dc142c70b1ccd1e6287c43139df1db26bd57a7a217c8da3a77326036495cdb57  pillow-11.1.0-cp312-cp312-win_amd64.whl
-896ddddbd4b85e86e0600cb65eb4c07fbc7f3802d47e7f660411e20b5500831469b97ed4770f25820f4e75cbfac40308da624fd86d4f62e578149d5c276a9cde  pyinstaller-6.10.0-py3-none-win_amd64.whl
+f0463895e9aee97f31a2003323de235fed1b26289766dc0837261e3f4a594a31162b69e9adbb0e9a31e2e2d4b5f25c762ed1669553df7dc89a8ba4f85d297873  pyinstaller-6.11.1-py3-none-win_amd64.whl
-873781decaeef07f6a79b0ed8b9f35f3fa534a1ea0d866991e40278a10818fa5b60c70b0d5828971b045364f1099694cd1e5d5d60d480acb93fcfbfbced4a09e  pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
+d550a0a14428386945533de2220c4c2e37c0c890fc51a600f626c6ca90a32d39572c121ec04c157ba3a8d6601cb021f8433d871b5c562a3d342c804fffec90c1  pyinstaller_hooks_contrib-2024.11-py3-none-any.whl
 0f623c9ab52d050283e97a986ba626d86b04cd02fa7ffdf352740576940b142b264709abadb5d875c90f625b28103d7210b900e0d77f12c1c140108bd2a159aa  python-3.12.8-amd64.exe

scripts/pyinstaller/notes.txt

@@ -38,7 +38,7 @@ fns=(
   MarkupSafe-2.1.5-cp312-cp312-win_amd64.whl
   mutagen-1.47.0-py3-none-any.whl
   packaging-24.1-py3-none-any.whl
-  pillow-10.4.0-cp312-cp312-win_amd64.whl
+  pillow-11.1.0-cp312-cp312-win_amd64.whl
   pyinstaller-6.10.0-py3-none-win_amd64.whl
   pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
   python-3.12.7-amd64.exe

scripts/toc.sh

@@ -20,7 +20,7 @@ cat $f | awk '
     o{next}
     /^#/{s=1;rs=0;pr()}
     /^#* *(nix package)/{rs=1}
-    /^#* *(themes|install on android|dev env setup|just the sfx|complete release|optional gpl stuff|nixos module)|```/{s=rs}
+    /^#* *(themes|install on android|dev env setup|just the sfx|complete release|optional gpl stuff|nixos module|reverse-proxy perf)|```/{s=rs}
     /^#/{
         lv=length($1);
         sub(/[^ ]+ /,"");

setup.py

@@ -144,6 +144,7 @@ args = {
         "ftps": ["pyftpdlib", "pyopenssl"],
         "tftpd": ["partftpy>=0.4.0"],
         "pwhash": ["argon2-cffi"],
+        "zeromq": ["pyzmq"],
     },
     "entry_points": {"console_scripts": ["copyparty = copyparty.__main__:main"]},
     "scripts": ["bin/partyfuse.py", "bin/u2c.py"],

tests/test_dxml.py

@@ -20,7 +20,8 @@ def _parse(txt):
 
 
 class TestDXML(unittest.TestCase):
-    def test1(self):
+    def test_qbe(self):
+        # allowed by default; verify that we stopped it
         txt = r"""<!DOCTYPE qbe [
 <!ENTITY a "nice_bakuretsu">
 ]>
@@ -28,7 +29,8 @@ class TestDXML(unittest.TestCase):
         _parse(txt)
         ET.fromstring(txt)
 
-    def test2(self):
+    def test_ent_file(self):
+        # NOT allowed by default; should still be blocked
         txt = r"""<!DOCTYPE ext [
 <!ENTITY ee SYSTEM "file:///bin/bash">
 ]>
@@ -40,6 +42,25 @@ class TestDXML(unittest.TestCase):
         except ET.ParseError:
             pass
 
+    def test_ent_ext(self):
+        # NOT allowed by default; should still be blocked
+        txt = r"""<!DOCTYPE ext [
+<!ENTITY ee SYSTEM "http://example.com/a.xml">
+]>
+<root>&ee;</root>"""
+        _parse(txt)
+
+    def test_dtd(self):
+        # allowed by default; verify that we stopped it
+        txt = r"""<!DOCTYPE d SYSTEM "a.dtd">
+<root>a</root>"""
+        _parse(txt)
+        ET.fromstring(txt)
+
+    ##
+    ## end of negative/security tests; the rest is functional
+    ##
+
     def test3(self):
         txt = r"""<?xml version="1.0" ?>
 <propfind xmlns="DAV:">

tests/util.py

@@ -141,13 +141,13 @@ class Cfg(Namespace):
         ex = "hash_mt hsortn safe_dedup srch_time u2abort u2j u2sz"
         ka.update(**{k: 1 for k in ex.split()})
 
-        ex = "au_vol dl_list mtab_age reg_cap s_thead s_tbody th_convt"
+        ex = "au_vol dl_list mtab_age reg_cap s_thead s_tbody th_convt ups_who"
         ka.update(**{k: 9 for k in ex.split()})
 
         ex = "db_act k304 loris no304 re_maxage rproxy rsp_jtr rsp_slp s_wr_slp snap_wri theme themes turbo"
         ka.update(**{k: 0 for k in ex.split()})
 
-        ex = "ah_alg bname chpw_db doctitle df exit favico idp_h_usr ipa html_head lg_sbf log_fk md_sbf name og_desc og_site og_th og_title og_title_a og_title_v og_title_i shr tcolor textfiles unlist vname xff_src R RS SR"
+        ex = "ah_alg bname chpw_db doctitle df exit favico idp_h_usr ipa html_head lg_sba lg_sbf log_fk md_sba md_sbf name og_desc og_site og_th og_title og_title_a og_title_v og_title_i shr tcolor textfiles unlist vname xff_src R RS SR"
         ka.update(**{k: "" for k in ex.split()})
 
         ex = "ban_403 ban_404 ban_422 ban_pw ban_url"
@@ -275,8 +275,6 @@ class VHttpSrv(object):
         self.gurl = Garda("")
 
         self.u2idx = None
-        self.ptn_cc = re.compile(r"[\x00-\x1f]")
-        self.uparam_cc_ok = set("doc move tree".split())
 
     def cachebuster(self):
         return "a"