v1.16.10

also add transcoding from opus to mp3 if client wants mp3,
overriding the feature detection for opus support

README.md

@@ -48,6 +48,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [shares](#shares) - share a file or folder by creating a temporary link
     * [batch rename](#batch-rename) - select some files and press `F2` to bring up the rename UI
     * [rss feeds](#rss-feeds) - monitor a folder with your RSS reader
+    * [recent uploads](#recent-uploads) - list all recent uploads
     * [media player](#media-player) - plays almost every audio format there is
         * [audio equalizer](#audio-equalizer) - and [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression)
         * [fix unreliable playback on android](#fix-unreliable-playback-on-android) - due to phone / app settings
@@ -79,6 +80,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [metadata from audio files](#metadata-from-audio-files) - set `-e2t` to index tags on upload
     * [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags
     * [event hooks](#event-hooks) - trigger a program on uploads, renames etc ([examples](./bin/hooks/))
+        * [zeromq](#zeromq) - event-hooks can send zeromq messages
         * [upload events](#upload-events) - the older, more powerful approach ([examples](./bin/mtag/))
     * [handlers](#handlers) - redefine behavior with plugins ([examples](./bin/handlers/))
     * [ip auth](#ip-auth) - autologin based on IP range (CIDR)
@@ -91,6 +93,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [listen on port 80 and 443](#listen-on-port-80-and-443) - become a *real* webserver
     * [reverse-proxy](#reverse-proxy) - running copyparty next to other websites
         * [real-ip](#real-ip) - teaching copyparty how to see client IPs
+        * [reverse-proxy performance](#reverse-proxy-performance)
     * [prometheus](#prometheus) - metrics/stats can be enabled
     * [other extremely specific features](#other-extremely-specific-features) - you'll never find a use for these
         * [custom mimetypes](#custom-mimetypes) - change the association of a file extension
@@ -139,6 +142,7 @@ just run **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/
 * or if you cannot install python, you can use [copyparty.exe](#copypartyexe) instead
 * or install [on arch](#arch-package) ╱ [on NixOS](#nixos-module) ╱ [through nix](#nix-package)
 * or if you are on android, [install copyparty in termux](#install-on-android)
+* or maybe you have a [synology nas / dsm](./docs/synology-dsm.md)
 * or if your computer is messed up and nothing else works, [try the pyz](#zipapp)
 * or if you prefer to [use docker](./scripts/docker/) 🐋 you can do that too
   * docker has all deps built-in, so skip this step:
@@ -339,6 +343,9 @@ same order here too
 
 * [Chrome issue 1352210](https://bugs.chromium.org/p/chromium/issues/detail?id=1352210) -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive)
 
+* [Chrome issue 383568268](https://issues.chromium.org/issues/383568268) -- filereaders in webworkers can OOM / crash the browser-tab
+  * copyparty has a workaround which seems to work well enough
+
 * [Firefox issue 1790500](https://bugzilla.mozilla.org/show_bug.cgi?id=1790500) -- entire browser can crash after uploading ~4000 small files
 
 * Android: music playback randomly stops due to [battery usage settings](#fix-unreliable-playback-on-android)
@@ -346,10 +353,19 @@ same order here too
 * iPhones: the volume control doesn't work because [apple doesn't want it to](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html#//apple_ref/doc/uid/TP40009523-CH5-SW11)
   * `AudioContext` will probably never be a viable workaround as apple introduces new issues faster than they fix current ones
 
+* iPhones: music volume goes on a rollercoaster during song changes
+  * nothing I can do about it because `AudioContext` is still broken in safari
+
 * iPhones: the preload feature (in the media-player-options tab) can cause a tiny audio glitch 20sec before the end of each song, but disabling it may cause worse iOS bugs to appear instead
   * just a hunch, but disabling preloading may cause playback to stop entirely, or possibly mess with bluetooth speakers
   * tried to add a tooltip regarding this but looks like apple broke my tooltips
 
+* iPhones: preloaded awo files make safari log MEDIA_ERR_NETWORK errors as playback starts, but the song plays just fine so eh whatever
+  * awo, opus-weba, is apple's new take on opus support, replacing opus-caf which was technically limited to cbr opus
+
+* iPhones: preloading another awo file may cause playback to stop
+  * can be somewhat mitigated with `mp.au.play()` in `mp.onpreload` but that can hit a race condition in safari that starts playing the same audio object twice in parallel...
+
 * Windows: folders cannot be accessed if the name ends with `.`
   * python or windows bug
 
@@ -428,7 +444,7 @@ configuring accounts/volumes with arguments:
 
 permissions:
 * `r` (read): browse folder contents, download files, download as zip/tar, see filekeys/dirkeys
-* `w` (write): upload files, move files *into* this folder
+* `w` (write): upload files, move/copy files *into* this folder
 * `m` (move): move files/folders *from* this folder
 * `d` (delete): delete files/folders
 * `.` (dots): user can ask to show dotfiles in directory listings
@@ -508,7 +524,8 @@ the browser has the following hotkeys  (always qwerty)
 * `ESC` close various things
 * `ctrl-K` delete selected files/folders
 * `ctrl-X` cut selected files/folders
-* `ctrl-V` paste
+* `ctrl-C` copy selected files/folders to clipboard
+* `ctrl-V` paste (move/copy)
 * `Y` download selected files
 * `F2` [rename](#batch-rename) selected file/folder
 * when a file/folder is selected (in not-grid-view):
@@ -577,6 +594,7 @@ click the `🌲` or pressing the `B` hotkey to toggle between breadcrumbs path (
 
 press `g` or `田` to toggle grid-view instead of the file listing  and `t` toggles icons / thumbnails
 * can be made default globally with `--grid` or per-volume with volflag `grid`
+* enable by adding `?imgs` to a link, or disable with `?imgs=0`
 
 ![copyparty-thumbs-fs8](https://user-images.githubusercontent.com/241032/129636211-abd20fa2-a953-4366-9423-1c88ebb96ba9.png)
 
@@ -607,8 +625,8 @@ select which type of archive you want in the `[⚙️] config` tab:
 | `pax` | `?tar=pax` | pax-format tar, futureproof, not as fast |
 | `tgz` | `?tar=gz` | gzip compressed gnu-tar (slow), for `curl \| tar -xvz` |
 | `txz` | `?tar=xz` | gnu-tar with xz / lzma compression (v.slow) |
-| `zip` | `?zip=utf8` | works everywhere, glitchy filenames on win7 and older |
+| `zip` | `?zip` | works everywhere, glitchy filenames on win7 and older |
-| `zip_dos` | `?zip` | traditional cp437 (no unicode) to fix glitchy filenames |
+| `zip_dos` | `?zip=dos` | traditional cp437 (no unicode) to fix glitchy filenames |
 | `zip_crc` | `?zip=crc` | cp437 with crc32 computed early for truly ancient software |
 
 * gzip default level is `3` (0=fast, 9=best), change with `?tar=gz:9`
@@ -616,7 +634,7 @@ select which type of archive you want in the `[⚙️] config` tab:
 * bz2 default level is `2` (1=fast, 9=best), change with `?tar=bz2:9`
 * hidden files ([dotfiles](#dotfiles)) are excluded unless account is allowed to list them
   * `up2k.db` and `dir.txt` is always excluded
-* bsdtar supports streaming unzipping: `curl foo?zip=utf8 | bsdtar -xv`
+* bsdtar supports streaming unzipping: `curl foo?zip | bsdtar -xv`
   * good, because copyparty's zip is faster than tar on small files
 * `zip_crc` will take longer to download since the server has to read each file twice
   * this is only to support MS-DOS PKZIP v2.04g (october 1993) and older
@@ -640,7 +658,7 @@ dragdrop is the recommended way, but you may also:
 
 * select some files (not folders) in your file explorer and press CTRL-V inside the browser window
 * use the [command-line uploader](https://github.com/9001/copyparty/tree/hovudstraum/bin#u2cpy)
-* upload using [curl or sharex](#client-examples)
+* upload using [curl, sharex, ishare, ...](#client-examples)
 
 when uploading files through dragdrop or CTRL-V, this initiates an upload using `up2k`; there are two browser-based uploaders available:
 * `[🎈] bup`, the basic uploader, supports almost every browser since netscape 4.0
@@ -712,7 +730,7 @@ files go into `[ok]` if they exist (and you get a link to where it is), otherwis
 
 ### unpost
 
-undo/delete accidental uploads
+undo/delete accidental uploads  using the `[🧯]` tab in the UI
 
 ![copyparty-unpost-fs8](https://user-images.githubusercontent.com/241032/129635368-3afa6634-c20f-418c-90dc-ec411f3b3897.png)
 
@@ -756,10 +774,11 @@ file selection: click somewhere on the line (not the link itself), then:
 * shift-click another line for range-select
 
 * cut: select some files and `ctrl-x`
+* copy: select some files and `ctrl-c`
 * paste: `ctrl-v` in another folder
 * rename: `F2`
 
-you can move files across browser tabs (cut in one tab, paste in another)
+you can copy/move files across browser tabs (cut/copy in one tab, paste in another)
 
 
 ## shares
@@ -870,6 +889,19 @@ url parameters:
   * uppercase = reverse-sort; `M` = oldest file first
 
 
+## recent uploads
+
+list all recent uploads  by clicking "show recent uploads" in the controlpanel
+
+will show uploader IP and upload-time if the visitor has the admin permission
+
+* global-option `--ups-when` makes upload-time visible to all users, and not just admins
+
+* global-option `--ups-who` (volflag `ups_who`) specifies who gets access (0=nobody, 1=admins, 2=everyone), default=2
+
+note that the [🧯 unpost](#unpost) feature is better suited for viewing *your own* recent uploads, as it includes the option to undo/delete them
+
+
 ## media player
 
 plays almost every audio format there is  (if the server has FFmpeg installed for on-demand transcoding)
@@ -906,6 +938,11 @@ open the `[🎺]` media-player-settings tab to configure it,
   * `[aac]` converts `aac` and `m4a` files into opus (if supported by browser) or mp3
   * `[oth]` converts all other known formats into opus (if supported by browser) or mp3
     * `aac|ac3|aif|aiff|alac|alaw|amr|ape|au|dfpwm|dts|flac|gsm|it|m4a|mo3|mod|mp2|mp3|mpc|mptm|mt2|mulaw|ogg|okt|opus|ra|s3m|tak|tta|ulaw|wav|wma|wv|xm|xpk`
+* "transcode to":
+  * `[opus]` produces an `opus` whenever transcoding is necessary (the best choice on Android and PCs)
+  * `[awo]` is `opus` in a `weba` file, good for iPhones (iOS 17.5 and newer) but Apple is still fixing some state-confusion bugs as of iOS 18.2.1
+  * `[caf]` is `opus` in a `caf` file, good for iPhones (iOS 11 through 17), technically unsupported by Apple but works for the mos tpart
+  * `[mp3]` -- the myth, the legend, the undying master of mediocre sound quality that definitely works everywhere
 * "tint" reduces the contrast of the playback bar
 
 
@@ -1086,6 +1123,8 @@ on macos, connect from finder:
 
 in order to grant full write-access to webdav clients, the volflag `daw` must be set and the account must also have delete-access (otherwise the client won't be allowed to replace the contents of existing files, which is how webdav works)
 
+> note: if you have enabled [IdP authentication](#identity-providers) then that may cause issues for some/most webdav clients; see [the webdav section in the IdP docs](https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients)
+
 
 ### connecting to webdav from windows
 
@@ -1094,11 +1133,12 @@ using the GUI  (winXP or later):
   * on winXP only, click the `Sign up for online storage` hyperlink instead and put the URL there
   * providing your password as the username is recommended; the password field can be anything or empty
 
-known client bugs:
+the webdav client that's built into windows has the following list of bugs; you can avoid all of these by connecting with rclone instead:
 * win7+ doesn't actually send the password to the server when reauthenticating after a reboot unless you first try to login with an incorrect password and then switch to the correct password
   * or just type your password into the username field instead to get around it entirely
 * connecting to a folder which allows anonymous read will make writing impossible, as windows has decided it doesn't need to login
   * workaround: connect twice; first to a folder which requires auth, then to the folder you actually want, and leave both of those mounted
+  * or set the server-option `--dav-auth` to force password-auth for all webdav clients
 * win7+ may open a new tcp connection for every file and sometimes forgets to close them, eventually needing a reboot
   * maybe NIC-related (??), happens with win10-ltsc on e1000e but not virtio
 * windows cannot access folders which contain filenames with invalid unicode or forbidden characters (`<>:"/\|?*`), or names ending with `.`
@@ -1265,7 +1305,7 @@ note:
 
 ### exclude-patterns
 
-to save some time,  you can provide a regex pattern for filepaths to only index by filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash \.iso$` or the volflag `:c,nohash=\.iso$`, this has the following consequences:
+to save some time,  you can provide a regex pattern for filepaths to only index by filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash '\.iso$'` or the volflag `:c,nohash=\.iso$`, this has the following consequences:
 * initial indexing is way faster, especially when the volume is on a network disk
 * makes it impossible to [file-search](#file-search)
 * if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected
@@ -1276,6 +1316,8 @@ similarly, you can fully ignore files/folders using `--no-idx [...]` and `:c,noi
 
 if you set `--no-hash [...]` globally, you can enable hashing for specific volumes using flag `:c,nohash=`
 
+to exclude certain filepaths from search-results, use `--srch-excl` or volflag `srch_excl` instead of `--no-idx`, for example `--srch-excl 'password|logs/[0-9]'`
+
 ### filesystem guards
 
 avoid traversing into other filesystems  using `--xdev` / volflag `:c,xdev`, skipping any symlinks or bind-mounts to another HDD for example
@@ -1433,6 +1475,23 @@ there's a bunch of flags and stuff, see `--help-hooks`
 if you want to write your own hooks, see [devnotes](./docs/devnotes.md#event-hooks)
 
 
+### zeromq
+
+event-hooks can send zeromq messages  instead of running programs
+
+to send a 0mq message every time a file is uploaded,
+
+* `--xau zmq:pub:tcp://*:5556` sends a PUB to any/all connected SUB clients
+* `--xau t3,zmq:push:tcp://*:5557` sends a PUSH to exactly one connected PULL client
+* `--xau t3,j,zmq:req:tcp://localhost:5555` sends a REQ to the connected REP client
+
+the PUSH and REQ examples have `t3` (timeout after 3 seconds) because they block if there's no clients to talk to
+
+* the REQ example does `t3,j` to send extended upload-info as json instead of just the filesystem-path
+
+see [zmq-recv.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/zmq-recv.py) if you need something to receive the messages with
+
+
 ### upload events
 
 the older, more powerful approach ([examples](./bin/mtag/)):
@@ -1482,7 +1541,9 @@ replace copyparty passwords with oauth and such
 
 you can disable the built-in password-based login system, and instead replace it with a separate piece of software (an identity provider) which will then handle authenticating / authorizing of users; this makes it possible to login with passkeys / fido2 / webauthn / yubikey / ldap / active directory / oauth / many other single-sign-on contraptions
 
-a popular choice is [Authelia](https://www.authelia.com/) (config-file based), another one is [authentik](https://goauthentik.io/) (GUI-based, more complex)
+* the regular config-defined users will be used as a fallback for requests which don't include a valid (trusted) IdP username header
+
+some popular identity providers are [Authelia](https://www.authelia.com/) (config-file based) and [authentik](https://goauthentik.io/) (GUI-based, more complex)
 
 there is a [docker-compose example](./docs/examples/docker/idp-authelia-traefik) which is hopefully a good starting point (alternatively see [./docs/idp.md](./docs/idp.md) if you're the DIY type)
 
@@ -1518,12 +1579,16 @@ connecting to an aws s3 bucket and similar
 
 there is no built-in support for this, but you can use FUSE-software such as [rclone](https://rclone.org/) / [geesefs](https://github.com/yandex-cloud/geesefs) / [JuiceFS](https://juicefs.com/en/) to first mount your cloud storage as a local disk, and then let copyparty use (a folder in) that disk as a volume
 
-you may experience poor upload performance this way, but that can sometimes be fixed by specifying the volflag `sparse` to force the use of sparse files; this has improved the upload speeds from `1.5 MiB/s` to over `80 MiB/s` in one case, but note that you are also more likely to discover funny bugs in your FUSE software this way, so buckle up
+you will probably get decent speeds with the default config, however most likely restricted to using one TCP connection per file, so the upload-client won't be able to send multiple chunks in parallel
+
+> before [v1.13.5](https://github.com/9001/copyparty/releases/tag/v1.13.5) it was recommended to use the volflag `sparse` to force-allow multiple chunks in parallel; this would improve the upload-speed from `1.5 MiB/s` to over `80 MiB/s` at the risk of provoking latent bugs in S3 or JuiceFS. But v1.13.5 added chunk-stitching, so this is now probably much less important. On the contrary, `nosparse` *may* now increase performance in some cases. Please try all three options (default, `sparse`, `nosparse`) as the optimal choice depends on your network conditions and software stack (both the FUSE-driver and cloud-server)
 
 someone has also tested geesefs in combination with [gocryptfs](https://nuetzlich.net/gocryptfs/) with surprisingly good results, getting 60 MiB/s upload speeds on a gbit line, but JuiceFS won with 80 MiB/s using its built-in encryption
 
 you may improve performance by specifying larger values for `--iobuf` / `--s-rd-sz` / `--s-wr-sz`
 
+> if you've experimented with this and made interesting observations, please share your findings so we can add a section with specific recommendations :-)
+
 
 ## hiding from google
 
@@ -1646,10 +1711,16 @@ some reverse proxies (such as [Caddy](https://caddyserver.com/)) can automatical
 
 for improved security (and a 10% performance boost) consider listening on a unix-socket with `-i unix:770:www:/tmp/party.sock` (permission `770` means only members of group `www` can access it)
 
-example webserver configs:
+example webserver / reverse-proxy configs:
 
-* [nginx config](contrib/nginx/copyparty.conf) -- entire domain/subdomain
+* [apache config](contrib/apache/copyparty.conf)
-* [apache2 config](contrib/apache/copyparty.conf) -- location-based
+* caddy uds: `caddy reverse-proxy --from :8080 --to unix///dev/shm/party.sock`
+* caddy tcp: `caddy reverse-proxy --from :8081 --to http://127.0.0.1:3923`
+* [haproxy config](contrib/haproxy/copyparty.conf)
+* [lighttpd subdomain](contrib/lighttpd/subdomain.conf) -- entire domain/subdomain
+* [lighttpd subpath](contrib/lighttpd/subpath.conf) -- location-based (not optimal, but in case you need it)
+* [nginx config](contrib/nginx/copyparty.conf) -- recommended
+* [traefik config](contrib/traefik/copyparty.yaml)
 
 
 ### real-ip
@@ -1661,6 +1732,38 @@ if you (and maybe everybody else) keep getting a message that says `thank you fo
 for most common setups, there should be a helpful message in the server-log explaining what to do, but see [docs/xff.md](docs/xff.md) if you want to learn more, including a quick hack to **just make it work** (which is **not** recommended, but hey...)
 
 
+### reverse-proxy performance
+
+most reverse-proxies support connecting to copyparty either using uds/unix-sockets (`/dev/shm/party.sock`, faster/recommended) or using tcp (`127.0.0.1`)
+
+with copyparty listening on a uds / unix-socket / unix-domain-socket and the reverse-proxy connecting to that:
+
+| index.html   | upload      | download    | software |
+| ------------ | ----------- | ----------- | -------- |
+| 28'900 req/s | 6'900 MiB/s | 7'400 MiB/s | no-proxy |
+| 18'750 req/s | 3'500 MiB/s | 2'370 MiB/s | haproxy |
+|  9'900 req/s | 3'750 MiB/s | 2'200 MiB/s | caddy |
+| 18'700 req/s | 2'200 MiB/s | 1'570 MiB/s | nginx |
+|  9'700 req/s | 1'750 MiB/s | 1'830 MiB/s | apache |
+|  9'900 req/s | 1'300 MiB/s | 1'470 MiB/s | lighttpd |
+
+when connecting the reverse-proxy to `127.0.0.1` instead (the basic and/or old-fasioned way), speeds are a bit worse:
+
+| index.html   | upload      | download    | software |
+| ------------ | ----------- | ----------- | -------- |
+| 21'200 req/s | 5'700 MiB/s | 6'700 MiB/s | no-proxy |
+| 14'500 req/s | 1'700 MiB/s | 2'170 MiB/s | haproxy |
+| 11'100 req/s | 2'750 MiB/s | 2'000 MiB/s | traefik |
+|  8'400 req/s | 2'300 MiB/s | 1'950 MiB/s | caddy |
+| 13'400 req/s | 1'100 MiB/s | 1'480 MiB/s | nginx |
+|  8'400 req/s | 1'000 MiB/s | 1'000 MiB/s | apache |
+|  6'500 req/s | 1'270 MiB/s | 1'500 MiB/s | lighttpd |
+
+in summary, `haproxy > caddy > traefik > nginx > apache > lighttpd`, and use uds when possible (traefik does not support it yet)
+
+* if these results are bullshit because my config exampels are bad, please submit corrections!
+
+
 ## prometheus
 
 metrics/stats can be enabled  at URL `/.cpr/metrics` for grafana / prometheus / etc (openmetrics 1.0.0)
@@ -1684,6 +1787,7 @@ scrape_configs:
 currently the following metrics are available,
 * `cpp_uptime_seconds` time since last copyparty restart
 * `cpp_boot_unixtime_seconds` same but as an absolute timestamp
+* `cpp_active_dl` number of active downloads
 * `cpp_http_conns` number of open http(s) connections
 * `cpp_http_reqs` number of http(s) requests handled
 * `cpp_sus_reqs` number of 403/422/malicious requests
@@ -1973,7 +2077,8 @@ interact with copyparty using non-browser clients
   * can be downloaded from copyparty: controlpanel -> connect -> [partyfuse.py](http://127.0.0.1:3923/.cpr/a/partyfuse.py)
   * [rclone](https://rclone.org/) as client can give ~5x performance, see [./docs/rclone.md](docs/rclone.md)
 
-* sharex (screenshot utility): see [./contrib/sharex.sxcu](contrib/#sharexsxcu)
+* sharex (screenshot utility): see [./contrib/sharex.sxcu](./contrib/#sharexsxcu)
+  * and for screenshots on macos, see [./contrib/ishare.iscu](./contrib/#ishareiscu)
   * and for screenshots on linux, see [./contrib/flameshot.sh](./contrib/flameshot.sh)
 
 * contextlet (web browser integration); see [contrib contextlet](contrib/#send-to-cppcontextletjson)
@@ -2237,13 +2342,13 @@ mandatory deps:
 
 install these to enable bonus features
 
-enable hashed passwords in config: `argon2-cffi`
+enable [hashed passwords](#password-hashing) in config: `argon2-cffi`
 
-enable ftp-server:
+enable [ftp-server](#ftp-server):
 * for just plaintext FTP, `pyftpdlib` (is built into the SFX)
 * with TLS encryption, `pyftpdlib pyopenssl`
 
-enable music tags:
+enable [music tags](#metadata-from-audio-files):
 * either `mutagen` (fast, pure-python, skips a few tags, makes copyparty GPL? idk)
 * or `ffprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users)
 
@@ -2254,8 +2359,9 @@ enable [thumbnails](#thumbnails) of...
 * **AVIF pictures:** `pyvips` or `ffmpeg` or `pillow-avif-plugin`
 * **JPEG XL pictures:** `pyvips` or `ffmpeg`
 
-enable [smb](#smb-server) support (**not** recommended):
+enable sending [zeromq messages](#zeromq) from event-hooks: `pyzmq`
-* `impacket==0.12.0`
+
+enable [smb](#smb-server) support (**not** recommended): `impacket==0.12.0`
 
 `pyvips` gives higher quality thumbnails than `Pillow` and is 320% faster, using 270% more ram: `sudo apt install libvips42 && python3 -m pip install --user -U pyvips`
 

bin/hooks/README.md

@@ -2,7 +2,7 @@ standalone programs which are executed by copyparty when an event happens (uploa
 
 these programs either take zero arguments, or a filepath (the affected file), or a json message with filepath + additional info
 
-run copyparty with `--help-hooks` for usage details / hook type explanations (xm/xbu/xau/xiu/xbr/xar/xbd/xad/xban)
+run copyparty with `--help-hooks` for usage details / hook type explanations (xm/xbu/xau/xiu/xbc/xac/xbr/xar/xbd/xad/xban)
 
 > **note:** in addition to event hooks (the stuff described here), copyparty has another api to run your programs/scripts while providing way more information such as audio tags / video codecs / etc and optionally daisychaining data between scripts in a processing pipeline; if that's what you want then see [mtp plugins](../mtag/) instead
 
@@ -30,4 +30,5 @@ these are `--xiu` hooks; unlike `xbu` and `xau` (which get executed on every sin
 # on message
 * [wget.py](wget.py) lets you download files by POSTing URLs to copyparty
 * [qbittorrent-magnet.py](qbittorrent-magnet.py) starts downloading a torrent if you post a magnet url
+* [usb-eject.py](usb-eject.py) adds web-UI buttons to safe-remove usb flashdrives shared through copyparty
 * [msg-log.py](msg-log.py) is a guestbook; logs messages to a doc in the same folder

bin/hooks/usb-eject.js

@@ -0,0 +1,54 @@
+// see usb-eject.py for usage
+
+function usbclick() {
+    QS('#treeul a[href="/usb/"]').click();
+}
+
+function eject_cb() {
+    var t = this.responseText;
+    if (t.indexOf('can be safely unplugged') < 0 && t.indexOf('Device can be removed') < 0)
+        return toast.err(30, 'usb eject failed:\n\n' + t);
+
+    toast.ok(5, esc(t.replace(/ - /g, '\n\n')));
+    usbclick(); setTimeout(usbclick, 10);
+};
+
+function add_eject_2(a) {
+    var aw = a.getAttribute('href').split(/\//g);
+    if (aw.length != 4 || aw[3])
+        return;
+
+    var v = aw[2],
+        k = 'umount_' + v;
+
+    qsr('#' + k);
+    a.appendChild(mknod('span', k, '⏏'), a);
+
+    var o = ebi(k);
+    o.style.cssText = 'position:absolute; right:1em; margin-top:-.2em; font-size:1.3em';
+    o.onclick = function (e) {
+        ev(e);
+        var xhr = new XHR();
+        xhr.open('POST', get_evpath(), true);
+        xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded;charset=UTF-8');
+        xhr.send('msg=' + uricom_enc(':usb-eject:' + v + ':'));
+        xhr.onload = xhr.onerror = eject_cb;
+        toast.inf(10, "ejecting " + v + "...");
+    };
+};
+
+function add_eject() {
+    for (var a of QSA('#treeul a[href^="/usb/"]'))
+        add_eject_2(a);
+};
+
+(function() {
+    var f0 = treectl.rendertree;
+    treectl.rendertree = function (res, ts, top0, dst, rst) {
+        var ret = f0(res, ts, top0, dst, rst);
+        add_eject();
+        return ret;
+    };
+})();
+
+setTimeout(add_eject, 50);

bin/hooks/usb-eject.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import stat
+import subprocess as sp
+import sys
+
+
+"""
+if you've found yourself using copyparty to serve flashdrives on a LAN
+and your only wish is that the web-UI had a button to unmount / safely
+remove those flashdrives, then boy howdy are you in the right place :D
+
+put usb-eject.js in the webroot (or somewhere else http-accessible)
+then run copyparty with these args:
+
+   -v /run/media/ed:/usb:A:c,hist=/tmp/junk
+   --xm=c1,bin/hooks/usb-eject.py
+   --js-browser=/usb-eject.js
+
+which does the following respectively,
+
+  * share all of /run/media/ed as /usb with admin for everyone
+     and put the histpath somewhere it won't cause trouble
+  * run the usb-eject hook with stdout redirect to the web-ui
+  * add the complementary usb-eject.js to the browser
+
+"""
+
+
+def main():
+    try:
+        label = sys.argv[1].split(":usb-eject:")[1].split(":")[0]
+        mp = "/run/media/ed/" + label
+        # print("ejecting [%s]... " % (mp,), end="")
+        mp = os.path.abspath(os.path.realpath(mp.encode("utf-8")))
+        st = os.lstat(mp)
+        if not stat.S_ISDIR(st.st_mode):
+            raise Exception("not a regular directory")
+
+        cmd = [b"gio", b"mount", b"-e", mp]
+        print(sp.check_output(cmd).decode("utf-8", "replace").strip())
+
+    except Exception as ex:
+        print("unmount failed: %r" % (ex,))
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/README.md

@@ -31,6 +31,9 @@ plugins in this section should only be used with appropriate precautions:
 * [very-bad-idea.py](./very-bad-idea.py) combined with [meadup.js](https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/meadup.js) converts copyparty into a janky yet extremely flexible chromecast clone
   * also adds a virtual keyboard by @steinuil to the basic-upload tab for comfy couch crowd control
   * anything uploaded through the [android app](https://github.com/9001/party-up) (files or links) are executed on the server, meaning anyone can infect your PC with malware... so protect this with a password and keep it on a LAN!
+  * [kamelåså](https://github.com/steinuil/kameloso) is a much better (and MUCH safer) alternative to this plugin
+    * powered by [chicken-curry-banana-pineapple-peanut pizza](https://a.ocv.me/pub/g/i/2025/01/298437ce-8351-4c8c-861c-fa131d217999.jpg?cache) so you know it's good
+    * and, unlike this plugin, kamelåså even has windows support (nice)
 
 
 # dependencies

bin/mtag/very-bad-idea.py

@@ -6,6 +6,11 @@ WARNING -- DANGEROUS PLUGIN --
   running this plugin, they can execute malware on your machine
   so please keep this on a LAN and protect it with a password
 
+here is a MUCH BETTER ALTERNATIVE (which also works on Windows):
+  https://github.com/steinuil/kameloso
+
+----------------------------------------------------------------------
+
 use copyparty as a chromecast replacement:
   * post a URL and it will open in the default browser
   * upload a file and it will open in the default application

bin/partyfuse.py

@@ -393,7 +393,8 @@ class Gateway(object):
         if r.status != 200:
             self.closeconn()
             info("http error %s reading dir %r", r.status, web_path)
-            raise FuseOSError(errno.ENOENT)
+            err = errno.ENOENT if r.status == 404 else errno.EIO
+            raise FuseOSError(err)
 
         ctype = r.getheader("Content-Type", "")
         if ctype == "application/json":

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "2.5"
+S_VERSION = "2.8"
-S_BUILD_DT = "2024-10-18"
+S_BUILD_DT = "2025-01-21"
 
 """
 u2c.py: upload to copyparty
@@ -189,6 +189,8 @@ class HCli(object):
         return C(self.addr, self.port, timeout=timeout, **args)
 
     def req(self, meth, vpath, hdrs, body=None, ctype=None):
+        now = time.time()
+
         hdrs.update(self.base_hdrs)
         if self.ar.a:
             hdrs["PW"] = self.ar.a
@@ -201,7 +203,9 @@ class HCli(object):
 
         # large timeout for handshakes (safededup)
         conns = self.hconns if ctype == MJ else self.conns
-        c = conns.pop() if conns else self._connect(999 if ctype == MJ else 128)
+        while conns and self.ar.cxp < now - conns[0][0]:
+            conns.pop(0)[1].close()
+        c = conns.pop()[1] if conns else self._connect(999 if ctype == MJ else 128)
         try:
             c.request(meth, vpath, body, hdrs)
             if PY27:
@@ -210,8 +214,15 @@ class HCli(object):
                 rsp = c.getresponse()
 
             data = rsp.read()
-            conns.append(c)
+            conns.append((time.time(), c))
             return rsp.status, data.decode("utf-8")
+        except http_client.BadStatusLine:
+            if self.ar.cxp > 4:
+                t = "\nWARNING: --cxp probably too high; reducing from %d to 4"
+                print(t % (self.ar.cxp,))
+                self.ar.cxp = 4
+            c.close()
+            raise
         except:
             c.close()
             raise
@@ -1022,8 +1033,8 @@ class Ctl(object):
             handshake(self.ar, file, False)
 
     def _fancy(self):
+        atexit.register(self.cleanup_vt100)
         if VT100 and not self.ar.ns:
-            atexit.register(self.cleanup_vt100)
             ss.scroll_region(3)
 
         Daemon(self.hasher)
@@ -1031,6 +1042,7 @@ class Ctl(object):
             Daemon(self.handshaker)
             Daemon(self.uploader)
 
+        last_sp = -1
         while True:
             with self.exit_cond:
                 self.exit_cond.wait(0.07)
@@ -1069,6 +1081,12 @@ class Ctl(object):
             else:
                 txt = " "
 
+            if not VT100:  # OSC9;4 (taskbar-progress)
+                sp = int(self.up_b * 100 / self.nbytes) or 1
+                if last_sp != sp:
+                    last_sp = sp
+                    txt += "\033]9;4;1;%d\033\\" % (sp,)
+
             if not self.up_br:
                 spd = self.hash_b / ((time.time() - self.t0) or 1)
                 eta = (self.nbytes - self.hash_b) / (spd or 1)
@@ -1079,11 +1097,15 @@ class Ctl(object):
 
             spd = humansize(spd)
             self.eta = str(datetime.timedelta(seconds=int(eta)))
+            if eta > 2591999:
+                self.eta = self.eta.split(",")[0]  # truncate HH:MM:SS
             sleft = humansize(self.nbytes - self.up_b)
             nleft = self.nfiles - self.up_f
             tail = "\033[K\033[u" if VT100 and not self.ar.ns else "\r"
 
-            t = "%s eta @ %s/s, %s, %d# left\033[K" % (self.eta, spd, sleft, nleft)
+            t = "%s eta @ %s/s, %s, %d# left" % (self.eta, spd, sleft, nleft)
+            if not self.hash_b:
+                t = " now hashing..."
             eprint(txt + "\033]0;{0}\033\\\r{0}{1}".format(t, tail))
 
         if self.ar.wlist:
@@ -1104,7 +1126,10 @@ class Ctl(object):
             handshake(self.ar, file, False)
 
     def cleanup_vt100(self):
-        ss.scroll_region(None)
+        if VT100:
+            ss.scroll_region(None)
+        else:
+            eprint("\033]9;4;0\033\\")
         eprint("\033[J\033]0;\033\\")
 
     def cb_hasher(self, file, ofs):
@@ -1119,7 +1144,9 @@ class Ctl(object):
             isdir = stat.S_ISDIR(inf.st_mode)
             if self.ar.z or self.ar.drd:
                 rd = rel if isdir else os.path.dirname(rel)
-                srd = rd.decode("utf-8", "replace").replace("\\", "/")
+                srd = rd.decode("utf-8", "replace").replace("\\", "/").rstrip("/")
+                if srd:
+                    srd += "/"
                 if prd != rd:
                     prd = rd
                     ls = {}
@@ -1142,7 +1169,10 @@ class Ctl(object):
 
                     if self.ar.drd:
                         dp = os.path.join(top, rd)
-                        lnodes = set(os.listdir(dp))
+                        try:
+                            lnodes = set(os.listdir(dp))
+                        except:
+                            lnodes = list(ls)  # fs eio; don't delete
                         if ptn:
                             zs = dp.replace(sep, b"/").rstrip(b"/") + b"/"
                             zls = [zs + x for x in lnodes]
@@ -1151,11 +1181,11 @@ class Ctl(object):
                         bnames = [x for x in ls if x not in lnodes and x != b".hist"]
                         vpath = self.ar.url.split("://")[-1].split("/", 1)[-1]
                         names = [x.decode("utf-8", WTF8) for x in bnames]
-                        locs = [vpath + srd + "/" + x for x in names]
+                        locs = [vpath + srd + x for x in names]
                         while locs:
                             req = locs
                             while req:
-                                print("DELETING ~%s/#%s" % (srd, len(req)))
+                                print("DELETING ~%s#%s" % (srd, len(req)))
                                 body = json.dumps(req).encode("utf-8")
                                 sc, txt = web.req(
                                     "POST", self.ar.url + "?delete", {}, body, MJ
@@ -1219,7 +1249,7 @@ class Ctl(object):
                         for n, zsii in enumerate(file.cids)
                     ]
                     print("chs: %s\n%s" % (vp, "\n".join(zsl)))
-                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.kchunks]
+                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.cids]
                 zb = hashlib.sha512("\n".join(zsl).encode("utf-8")).digest()[:33]
                 wark = ub64enc(zb).decode("utf-8")
                 if self.ar.jw:
@@ -1500,6 +1530,7 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("--szm", type=int, metavar="MiB", default=96, help="max size of each POST (default is cloudflare max)")
     ap.add_argument("-nh", action="store_true", help="disable hashing while uploading")
     ap.add_argument("-ns", action="store_true", help="no status panel (for slow consoles and macos)")
+    ap.add_argument("--cxp", type=float, metavar="SEC", default=57, help="assume http connections expired after SEConds")
     ap.add_argument("--cd", type=float, metavar="SEC", default=5, help="delay before reattempting a failed handshake/upload")
     ap.add_argument("--safe", action="store_true", help="use simple fallback approach")
     ap.add_argument("-z", action="store_true", help="ZOOMIN' (skip uploading files if they exist at the destination with the ~same last-modified timestamp, so same as yolo / turbo with date-chk but even faster)")
@@ -1519,6 +1550,38 @@ source file/folder selection uses rsync syntax, meaning that:
             except:
                 pass
 
+    # msys2 doesn't uncygpath absolute paths with whitespace
+    if not VT100:
+        zsl = []
+        for fn in ar.files:
+            if re.search("^/[a-z]/", fn):
+                fn = r"%s:\%s" % (fn[1:2], fn[3:])
+            zsl.append(fn.replace("/", "\\"))
+        ar.files = zsl
+
+    fok = []
+    fng = []
+    for fn in ar.files:
+        if os.path.exists(fn):
+            fok.append(fn)
+        elif VT100:
+            fng.append(fn)
+        else:
+            # windows leaves glob-expansion to the invoked process... okayyy let's get to work
+            from glob import glob
+
+            fns = glob(fn)
+            if fns:
+                fok.extend(fns)
+            else:
+                fng.append(fn)
+
+    if fng:
+        t = "some files/folders were not found:\n  %s"
+        raise Exception(t % ("\n  ".join(fng),))
+
+    ar.files = fok
+
     if ar.drd:
         ar.dr = True
 

bin/zmq-recv.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+
+import sys
+import zmq
+
+"""
+zmq-recv.py: demo zmq receiver
+2025-01-22, v1.0, ed <irc.rizon.net>, MIT-Licensed
+https://github.com/9001/copyparty/blob/hovudstraum/bin/zmq-recv.py
+
+basic zmq-server to receive events from copyparty; try one of
+the below and then "send a message to serverlog" in the web-ui:
+
+1) dumb fire-and-forget to any and all listeners;
+run this script with "sub" and run copyparty with this:
+  --xm zmq:pub:tcp://*:5556
+
+2) one lucky listener gets the message, blocks if no listeners:
+run this script with "pull" and run copyparty with this:
+  --xm t3,zmq:push:tcp://*:5557
+
+3) blocking syn/ack mode, client must ack each message;
+run this script with "rep" and run copyparty with this:
+  --xm t3,zmq:req:tcp://localhost:5555
+
+note: to conditionally block uploads based on message contents,
+use rep_server to answer with "return 1" and run copyparty with
+  --xau t3,c,zmq:req:tcp://localhost:5555
+"""
+
+
+ctx = zmq.Context()
+
+
+def sub_server():
+    # PUB/SUB allows any number of servers/clients, and
+    # messages are fire-and-forget
+    sck = ctx.socket(zmq.SUB)
+    sck.connect("tcp://localhost:5556")
+    sck.setsockopt_string(zmq.SUBSCRIBE, "")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def pull_server():
+    # PUSH/PULL allows any number of servers/clients, and
+    # each message is sent to a exactly one PULL client
+    sck = ctx.socket(zmq.PULL)
+    sck.connect("tcp://localhost:5557")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+
+
+def rep_server():
+    # REP/REQ is a server/client pair where each message must be
+    # acked by the other before another message can be sent, so
+    # copyparty will do a blocking-wait for the ack
+    sck = ctx.socket(zmq.REP)
+    sck.bind("tcp://*:5555")
+    while True:
+        print("copyparty says %r" % (sck.recv_string(),))
+        reply = b"thx"
+        # reply = b"return 1"  # non-zero to block an upload
+        sck.send(reply)
+
+
+mode = sys.argv[1].lower() if len(sys.argv) > 1 else ""
+
+if mode == "sub":
+    sub_server()
+elif mode == "pull":
+    pull_server()
+elif mode == "rep":
+    rep_server()
+else:
+    print("specify mode as first argument:  SUB | PULL | REP")

contrib/README.md

@@ -12,14 +12,19 @@
 * assumes the webserver and copyparty is running on the same server/IP
 * modify `10.13.1.1` as necessary if you wish to support browsers without javascript
 
-### [`sharex.sxcu`](sharex.sxcu)
+### [`sharex.sxcu`](sharex.sxcu) - Windows screenshot uploader
-* sharex config file to upload screenshots and grab the URL
+* [sharex](https://getsharex.com/) config file to upload screenshots and grab the URL
 * `RequestURL`: full URL to the target folder
 * `pw`: password (remove the `pw` line if anon-write)
 * the `act:bput` thing is optional since copyparty v1.9.29
 * using an older sharex version, maybe sharex v12.1.1 for example? dw fam i got your back 👉😎👉 [`sharex12.sxcu`](sharex12.sxcu)
 
-### [`flameshot.sh`](flameshot.sh)
+### [`ishare.iscu`](ishare.iscu) - MacOS screenshot uploader
+* [ishare](https://isharemac.app/) config file to upload screenshots and grab the URL
+* `RequestURL`: full URL to the target folder
+* `pw`: password (remove the `pw` line if anon-write)
+
+### [`flameshot.sh`](flameshot.sh) - Linux screenshot uploader
 * takes a screenshot with [flameshot](https://flameshot.org/) on Linux, uploads it, and writes the URL to clipboard
 
 ### [`send-to-cpp.contextlet.json`](send-to-cpp.contextlet.json)
@@ -53,5 +58,10 @@ init-scripts to start copyparty as a service
 * [`openrc/copyparty`](openrc/copyparty)
 
 # Reverse-proxy
-copyparty has basic support for running behind another webserver
+copyparty supports running behind another webserver
-* [`nginx/copyparty.conf`](nginx/copyparty.conf)
+* [`apache/copyparty.conf`](apache/copyparty.conf)
+* [`haproxy/copyparty.conf`](haproxy/copyparty.conf)
+* [`lighttpd/subdomain.conf`](lighttpd/subdomain.conf)
+* [`lighttpd/subpath.conf`](lighttpd/subpath.conf)
+* [`nginx/copyparty.conf`](nginx/copyparty.conf) -- recommended
+* [`traefik/copyparty.yaml`](traefik/copyparty.yaml)

contrib/apache/copyparty.conf

@@ -1,14 +1,29 @@
-# when running copyparty behind a reverse proxy,
+# if you would like to use unix-sockets (recommended),
-# the following arguments are recommended:
+# you must run copyparty with one of the following:
 #
-#   -i 127.0.0.1    only accept connections from nginx
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
 #
 # if you are doing location-based proxying (such as `/stuff` below)
 # you must run copyparty with --rp-loc=stuff
 #
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 
+
 LoadModule proxy_module modules/mod_proxy.so
-ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"
+
-# do not specify ProxyPassReverse
 RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
+# NOTE: do not specify ProxyPassReverse
+
+
+##
+## then, enable one of the below:
+
+# use subdomain proxying to unix-socket (best)
+ProxyPass "/" "unix:///dev/shm/party.sock|http://whatever/"
+
+# use subdomain proxying to 127.0.0.1 (slower)
+#ProxyPass "/" "http://127.0.0.1:3923/"
+
+# use subpath proxying to 127.0.0.1 (slow and maybe buggy)
+#ProxyPass "/stuff" "http://127.0.0.1:3923/stuff"

contrib/haproxy/copyparty.conf

@@ -0,0 +1,24 @@
+# this config is essentially two separate examples;
+#
+#   foo1 connects to copyparty using tcp, and
+#   foo2 uses unix-sockets for 27% higher performance
+#
+# to use foo2 you must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+
+defaults
+  mode http
+  option forwardfor
+  timeout connect 1s
+  timeout client 610s
+  timeout server 610s
+
+listen foo1
+  bind *:8081
+  server srv1 127.0.0.1:3923 maxconn 512
+
+listen foo2
+  bind *:8082
+  server srv1 /dev/shm/party.sock maxconn 512

contrib/ishare.iscu

@@ -0,0 +1,10 @@
+{
+  "Name": "copyparty",
+  "RequestURL": "http://127.0.0.1:3923/screenshots/",
+  "Headers": {
+    "pw": "PUT_YOUR_PASSWORD_HERE_MY_DUDE",
+    "accept": "json"
+  },
+  "FileFormName": "f",
+  "ResponseURL": "{{fileurl}}"
+}

contrib/lighttpd/subdomain.conf

@@ -0,0 +1,24 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subdomain.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+proxy.forwarded = ( "for" => 1, "proto" => 1 )
+proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+# if you really need to use tcp instead of unix-sockets, do this instead:
+#proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )

contrib/lighttpd/subpath.conf

@@ -0,0 +1,31 @@
+# example usage for benchmarking:
+#
+#   taskset -c 1 lighttpd -Df ~/dev/copyparty/contrib/lighttpd/subpath.conf
+#
+# lighttpd can connect to copyparty using either tcp (127.0.0.1)
+# or a unix-socket, but unix-sockets are 37% faster because
+# lighttpd doesn't reuse tcp connections, so we're doing unix-sockets
+#
+# this means we must run copyparty with one of the following:
+#
+#   -i unix:777:/dev/shm/party.sock
+#   -i unix:777:/dev/shm/party.sock,127.0.0.1
+#
+# also since this example proxies a subpath instead of the
+# recommended subdomain-proxying, we must also specify this:
+#
+#   --rp-loc files
+#
+# on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
+
+server.port = 80
+server.document-root = "/var/empty"
+server.upload-dirs = ( "/dev/shm", "/tmp" )
+server.modules = ( "mod_proxy" )
+$HTTP["url"] =~ "^/files" {
+    proxy.forwarded = ( "for" => 1, "proto" => 1 )
+    proxy.server = ( "" => ( ( "host" => "/dev/shm/party.sock" ) ) )
+
+    # if you really need to use tcp instead of unix-sockets, do this instead:
+    #proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "3923" ) ) )
+}

contrib/nginx/copyparty.conf

@@ -36,9 +36,9 @@ upstream cpp_uds {
 	# but there must be at least one unix-group which both
 	# nginx and copyparty is a member of; if that group is
 	# "www" then run copyparty with the following args:
-	# -i unix:770:www:/tmp/party.sock
+	# -i unix:770:www:/dev/shm/party.sock
 
-	server unix:/tmp/party.sock fail_timeout=1s;
+	server unix:/dev/shm/party.sock fail_timeout=1s;
 	keepalive 1;
 }
 
@@ -61,6 +61,10 @@ server {
 		client_max_body_size 0;
 		proxy_buffering off;
 		proxy_request_buffering off;
+		# improve download speed from 600 to 1500 MiB/s
+		proxy_buffers 32 8k;
+		proxy_buffer_size 16k;
+		proxy_busy_buffers_size 24k;
 
 		proxy_set_header   Host              $host;
 		proxy_set_header   X-Real-IP         $remote_addr;

contrib/package/arch/PKGBUILD

@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.15.9"
+pkgver="1.16.9"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -16,12 +16,13 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
             "libkeyfinder-git: detection of musical keys" 
             "qm-vamp-plugins: BPM detection" 
             "python-pyopenssl: ftps functionality" 
-            "python-argon2_cffi: hashed passwords in config" 
+            "python-pyzmq: send zeromq messages from event-hooks" 
+            "python-argon2-cffi: hashed passwords in config" 
             "python-impacket-git: smb support (bad idea)"
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("f75668c752468cab3e4d7bb93323ab9ac7309ee74a138d3597ef0edabd9235de")
+sha256sums=("3e8f3c24c699aa41e0d51db6d781e453979c77abc34c919063b5bddd64d27bb0")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/default.nix

@@ -1,4 +1,4 @@
-{ lib, stdenv, makeWrapper, fetchurl, utillinux, python, jinja2, impacket, pyftpdlib, pyopenssl, argon2-cffi, pillow, pyvips, ffmpeg, mutagen,
+{ lib, stdenv, makeWrapper, fetchurl, utillinux, python, jinja2, impacket, pyftpdlib, pyopenssl, argon2-cffi, pillow, pyvips, pyzmq, ffmpeg, mutagen,
 
 # use argon2id-hashed passwords in config files (sha2 is always available)
 withHashedPasswords ? true,
@@ -21,6 +21,9 @@ withMediaProcessing ? true,
 # if MediaProcessing is not enabled, you probably want this instead (less accurate, but much safer and faster)
 withBasicAudioMetadata ? false,
 
+# send ZeroMQ messages from event-hooks
+withZeroMQ ? true,
+
 # enable FTPS support in the FTP server
 withFTPS ? false,
 
@@ -43,6 +46,7 @@ let
     ++ lib.optional withMediaProcessing ffmpeg
     ++ lib.optional withBasicAudioMetadata mutagen
     ++ lib.optional withHashedPasswords argon2-cffi
+    ++ lib.optional withZeroMQ pyzmq
     );
 in stdenv.mkDerivation {
   pname = "copyparty";

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.15.9/copyparty-sfx.py",
+    "url": "https://github.com/9001/copyparty/releases/download/v1.16.9/copyparty-sfx.py",
-    "version": "1.15.9",
+    "version": "1.16.9",
-    "hash": "sha256-pxX1RkfO3h1XyPQnrwkFp8laOS4sqXAwbsi2V18VnVY="
+    "hash": "sha256-456L3IHzf8ups3L9pTBZJMQjML8AlsQI66HZohDyEIA="
 }

contrib/traefik/copyparty.yaml

@@ -0,0 +1,12 @@
+# ./traefik --experimental.fastproxy=true --entrypoints.web.address=:8080 --providers.file.filename=copyparty.yaml
+
+http:
+  services:
+    service-cpp:
+      loadBalancer:
+        servers:
+          - url: "http://127.0.0.1:3923/"
+  routers:
+    my-router:
+      rule: "PathPrefix(`/`)"
+      service: service-cpp

copyparty/__init__.py

@@ -80,6 +80,7 @@ web/deps/prismd.css
 web/deps/scp.woff2
 web/deps/sha512.ac.js
 web/deps/sha512.hw.js
+web/iiam.gif
 web/md.css
 web/md.html
 web/md.js
@@ -90,6 +91,9 @@ web/mde.html
 web/mde.js
 web/msg.css
 web/msg.html
+web/rups.css
+web/rups.html
+web/rups.js
 web/shares.css
 web/shares.html
 web/shares.js

copyparty/__main__.py

@@ -50,8 +50,12 @@ from .util import (
     PARTFTPY_VER,
     PY_DESC,
     PYFTPD_VER,
+    RAM_AVAIL,
+    RAM_TOTAL,
     SQLITE_VER,
     UNPLICATIONS,
+    URL_BUG,
+    URL_PRJ,
     Daemon,
     align_tab,
     ansi_re,
@@ -330,17 +334,16 @@ def ensure_webdeps() -> None:
     if has_resource(E, "web/deps/mini-fa.woff"):
         return
 
-    warn(
+    t = """could not find webdeps;
-        """could not find webdeps;
   if you are running the sfx, or exe, or pypi package, or docker image,
   then this is a bug! Please let me know so I can fix it, thanks :-)
-  https://github.com/9001/copyparty/issues/new?labels=bug&template=bug_report.md
+  %s
 
   however, if you are a dev, or running copyparty from source, and you want
   full client functionality, you will need to build or obtain the webdeps:
-  https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#building
+  %s/blob/hovudstraum/docs/devnotes.md#building
     """
-    )
+    warn(t % (URL_BUG, URL_PRJ))
 
 
 def configure_ssl_ver(al: argparse.Namespace) -> None:
@@ -684,6 +687,8 @@ def get_sects():
              \033[36mxbu\033[35m executes CMD before a file upload starts
              \033[36mxau\033[35m executes CMD after  a file upload finishes
              \033[36mxiu\033[35m executes CMD after  all uploads finish and volume is idle
+             \033[36mxbc\033[35m executes CMD before a file copy
+             \033[36mxac\033[35m executes CMD after  a file copy
              \033[36mxbr\033[35m executes CMD before a file rename/move
              \033[36mxar\033[35m executes CMD after  a file rename/move
              \033[36mxbd\033[35m executes CMD before a file delete
@@ -735,6 +740,10 @@ def get_sects():
               the \033[33m,,\033[35m stops copyparty from reading the rest as flags and
               the \033[33m--\033[35m stops notify-send from reading the message as args
               and the alert will be "hey" followed by the messagetext
+
+             \033[36m--xau zmq:pub:tcp://*:5556\033[35m announces uploads on zeromq;
+             \033[36m--xau t3,zmq:push:tcp://*:5557\033[35m also works, and you can
+             \033[36m--xau t3,j,zmq:req:tcp://localhost:5555\033[35m too for example
             \033[0m
             each hook is executed once for each event, except for \033[36mxiu\033[0m
             which builds up a backlog of uploads, running the hook just once
@@ -766,11 +775,22 @@ def get_sects():
             values for --urlform:
               \033[36mstash\033[35m dumps the data to file and returns length + checksum
               \033[36msave,get\033[35m dumps to file and returns the page like a GET
-              \033[36mprint,get\033[35m prints the data in the log and returns GET
+              \033[36mprint    \033[35m prints the data to log and returns an error
-              (leave out the ",get" to return an error instead)\033[0m
+              \033[36mprint,xm \033[35m prints the data to log and returns --xm output
+              \033[36mprint,get\033[35m prints the data to log and returns GET\033[0m
 
-            note that the \033[35m--xm\033[0m hook will only run if \033[35m--urlform\033[0m
+            note that the \033[35m--xm\033[0m hook will only run if \033[35m--urlform\033[0m is
-              is either \033[36mprint\033[0m or the default \033[36mprint,get\033[0m
+              either \033[36mprint\033[0m or \033[36mprint,get\033[0m or the default \033[36mprint,xm\033[0m
+
+            if an \033[35m--xm\033[0m hook returns text, then
+              the response code will be HTTP 202;
+              http/get responses will be HTTP 200
+
+            if there are multiple \033[35m--xm\033[0m hooks defined, then
+              the first hook that produced output is returned
+
+            if there are no \033[35m--xm\033[0m hooks defined, then the default
+              \033[36mprint,xm\033[0m behaves like \033[36mprint,get\033[0m (returning html)
             """
             ),
         ],
@@ -874,8 +894,9 @@ def get_sects():
             use argon2id with timecost 3, 256 MiB, 4 threads, version 19 (0x13/v1.3)
 
             \033[36m--ah-alg scrypt\033[0m  # which is the same as:
-            \033[36m--ah-alg scrypt,13,2,8,4\033[0m
+            \033[36m--ah-alg scrypt,13,2,8,4,32\033[0m
-            use scrypt with cost 2**13, 2 iterations, blocksize 8, 4 threads
+            use scrypt with cost 2**13, 2 iterations, blocksize 8, 4 threads,
+              and allow using up to 32 MiB RAM (ram=cost*blksz roughly)
 
             \033[36m--ah-alg sha2\033[0m  # which is the same as:
             \033[36m--ah-alg sha2,424242\033[0m
@@ -950,7 +971,7 @@ def add_general(ap, nc, srvname):
     ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m], see --help-accounts")
     ap2.add_argument("--grp", metavar="G:N,N", type=u, action="append", help="add group, \033[33mNAME\033[0m:\033[33mUSER1\033[0m,\033[33mUSER2\033[0m,\033[33m...\033[0m; example [\033[32madmins:ed,foo,bar\033[0m]")
     ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files (volflag=dots)")
-    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,xm", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
     ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="server terminal title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
     ap2.add_argument("--name", metavar="TXT", type=u, default=srvname, help="server name (displayed topleft in browser and in mDNS)")
     ap2.add_argument("--mime", metavar="EXT=MIME", type=u, action="append", help="map file \033[33mEXT\033[0mension to \033[33mMIME\033[0mtype, for example [\033[32mjpg=image/jpeg\033[0m]")
@@ -1078,7 +1099,7 @@ def add_cert(ap, cert_path):
 def add_auth(ap):
     ses_db = os.path.join(E.cfg, "sessions.db")
     ap2 = ap.add_argument_group('IdP / identity provider / user authentication options')
-    ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks and assume the request-header \033[33mHN\033[0m contains the username of the requesting user (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
+    ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks if the request-header \033[33mHN\033[0m contains a username to associate the request with (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
     ap2.add_argument("--idp-h-grp", metavar="HN", type=u, default="", help="assume the request-header \033[33mHN\033[0m contains the groupname of the requesting user; can be referenced in config files for group-based access control")
     ap2.add_argument("--idp-h-key", metavar="HN", type=u, default="", help="optional but recommended safeguard; your reverse-proxy will insert a secret header named \033[33mHN\033[0m into all requests, and the other IdP headers will be ignored if this header is not present")
     ap2.add_argument("--idp-gsep", metavar="RE", type=u, default="|:;+,", help="if there are multiple groups in \033[33m--idp-h-grp\033[0m, they are separated by one of the characters in \033[33mRE\033[0m")
@@ -1119,6 +1140,8 @@ def add_zc_mdns(ap):
     ap2.add_argument("--zm6", action="store_true", help="IPv6 only")
     ap2.add_argument("--zmv", action="store_true", help="verbose mdns")
     ap2.add_argument("--zmvv", action="store_true", help="verboser mdns")
+    ap2.add_argument("--zm-no-pe", action="store_true", help="mute parser errors (invalid incoming MDNS packets)")
+    ap2.add_argument("--zm-nwa-1", action="store_true", help="disable workaround for avahi-bug #379 (corruption in Avahi's mDNS reflection feature)")
     ap2.add_argument("--zms", metavar="dhf", type=u, default="", help="list of services to announce -- d=webdav h=http f=ftp s=smb -- lowercase=plaintext uppercase=TLS -- default: all enabled services except http/https (\033[32mDdfs\033[0m if \033[33m--ftp\033[0m and \033[33m--smb\033[0m is set, \033[32mDd\033[0m otherwise)")
     ap2.add_argument("--zm-ld", metavar="PATH", type=u, default="", help="link a specific folder for webdav shares")
     ap2.add_argument("--zm-lh", metavar="PATH", type=u, default="", help="link a specific folder for http shares")
@@ -1201,6 +1224,8 @@ def add_hooks(ap):
     ap2.add_argument("--xbu", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m before a file upload starts")
     ap2.add_argument("--xau", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m after  a file upload finishes")
     ap2.add_argument("--xiu", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m after  all uploads finish and volume is idle")
+    ap2.add_argument("--xbc", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m before a file copy")
+    ap2.add_argument("--xac", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m after  a file copy")
     ap2.add_argument("--xbr", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m before a file move/rename")
     ap2.add_argument("--xar", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m after  a file move/rename")
     ap2.add_argument("--xbd", metavar="CMD", type=u, action="append", help="execute \033[33mCMD\033[0m before a file delete")
@@ -1233,6 +1258,7 @@ def add_optouts(ap):
     ap2.add_argument("--no-dav", action="store_true", help="disable webdav support")
     ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
     ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
+    ap2.add_argument("--no-cp", action="store_true", help="disable copy operations")
     ap2.add_argument("-nth", action="store_true", help="no title hostname; don't show \033[33m--name\033[0m in <title>")
     ap2.add_argument("-nih", action="store_true", help="no info hostname -- don't show in UI")
     ap2.add_argument("-nid", action="store_true", help="no info disk-usage -- don't show in UI")
@@ -1240,7 +1266,6 @@ def add_optouts(ap):
     ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
     ap2.add_argument("--no-tarcmp", action="store_true", help="disable download as compressed tar (?tar=gz, ?tar=bz2, ?tar=xz, ?tar=gz:9, ...)")
     ap2.add_argument("--no-lifetime", action="store_true", help="do not allow clients (or server config) to schedule an upload to be deleted after a given time")
-    ap2.add_argument("--no-up-list", action="store_true", help="don't show list of incoming files in controlpanel")
     ap2.add_argument("--no-pipe", action="store_true", help="disable race-the-beam (lockstep download of files which are currently being uploaded) (volflag=nopipe)")
     ap2.add_argument("--no-db-ip", action="store_true", help="do not write uploader IPs into the database")
 
@@ -1308,7 +1333,7 @@ def add_logging(ap):
     ap2.add_argument("--log-htp", action="store_true", help="debug: print http-server threadpool scaling")
     ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="print request \033[33mHEADER\033[0m; [\033[32m*\033[0m]=all")
     ap2.add_argument("--ohead", metavar="HEADER", type=u, action='append', help="print response \033[33mHEADER\033[0m; [\033[32m*\033[0m]=all")
-    ap2.add_argument("--lf-url", metavar="RE", type=u, default=r"^/\.cpr/|\?th=[wj]$|/\.(_|ql_|DS_Store$|localized$)", help="dont log URLs matching regex \033[33mRE\033[0m")
+    ap2.add_argument("--lf-url", metavar="RE", type=u, default=r"^/\.cpr/|[?&]th=[wjp]|/\.(_|ql_|DS_Store$|localized$)", help="dont log URLs matching regex \033[33mRE\033[0m")
 
 
 def add_admin(ap):
@@ -1316,9 +1341,16 @@ def add_admin(ap):
     ap2.add_argument("--no-reload", action="store_true", help="disable ?reload=cfg (reload users/volumes/volflags from config file)")
     ap2.add_argument("--no-rescan", action="store_true", help="disable ?scan (volume reindexing)")
     ap2.add_argument("--no-stack", action="store_true", help="disable ?stack (list all stacks)")
+    ap2.add_argument("--no-ups-page", action="store_true", help="disable ?ru (list of recent uploads)")
+    ap2.add_argument("--no-up-list", action="store_true", help="don't show list of incoming files in controlpanel")
+    ap2.add_argument("--dl-list", metavar="LVL", type=int, default=2, help="who can see active downloads in the controlpanel? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=everyone")
+    ap2.add_argument("--ups-who", metavar="LVL", type=int, default=2, help="who can see recent uploads on the ?ru page? [\033[32m0\033[0m]=nobody, [\033[32m1\033[0m]=admins, [\033[32m2\033[0m]=everyone (volflag=ups_who)")
+    ap2.add_argument("--ups-when", action="store_true", help="let everyone see upload timestamps on the ?ru page, not just admins")
 
 
 def add_thumbnail(ap):
+    th_ram = (RAM_AVAIL or RAM_TOTAL or 9) * 0.6
+    th_ram = int(max(min(th_ram, 6), 1) * 10) / 10
     ap2 = ap.add_argument_group('thumbnail options')
     ap2.add_argument("--no-thumb", action="store_true", help="disable all thumbnails (volflag=dthumb)")
     ap2.add_argument("--no-vthumb", action="store_true", help="disable video thumbnails (volflag=dvthumb)")
@@ -1326,7 +1358,7 @@ def add_thumbnail(ap):
     ap2.add_argument("--th-size", metavar="WxH", default="320x256", help="thumbnail res (volflag=thsize)")
     ap2.add_argument("--th-mt", metavar="CORES", type=int, default=CORES, help="num cpu cores to use for generating thumbnails")
     ap2.add_argument("--th-convt", metavar="SEC", type=float, default=60.0, help="conversion timeout in seconds (volflag=convt)")
-    ap2.add_argument("--th-ram-max", metavar="GB", type=float, default=6.0, help="max memory usage (GiB) permitted by thumbnailer; not very accurate")
+    ap2.add_argument("--th-ram-max", metavar="GB", type=float, default=th_ram, help="max memory usage (GiB) permitted by thumbnailer; not very accurate")
     ap2.add_argument("--th-crop", metavar="TXT", type=u, default="y", help="crop thumbnails to 4:3 or keep dynamic height; client can override in UI unless force. [\033[32my\033[0m]=crop, [\033[32mn\033[0m]=nocrop, [\033[32mfy\033[0m]=force-y, [\033[32mfn\033[0m]=force-n (volflag=crop)")
     ap2.add_argument("--th-x3", metavar="TXT", type=u, default="n", help="show thumbs at 3x resolution; client can override in UI unless force. [\033[32my\033[0m]=yes, [\033[32mn\033[0m]=no, [\033[32mfy\033[0m]=force-yes, [\033[32mfn\033[0m]=force-no (volflag=th3x)")
     ap2.add_argument("--th-dec", metavar="LIBS", default="vips,pil,ff", help="image decoders, in order of preference")
@@ -1341,18 +1373,20 @@ def add_thumbnail(ap):
     # https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
     # https://github.com/libvips/libvips
     # ffmpeg -hide_banner -demuxers | awk '/^ D  /{print$2}' | while IFS= read -r x; do ffmpeg -hide_banner -h demuxer=$x; done | grep -E '^Demuxer |extensions:'
-    ap2.add_argument("--th-r-pil", metavar="T,T", type=u, default="avif,avifs,blp,bmp,dcx,dds,dib,emf,eps,fits,flc,fli,fpx,gif,heic,heics,heif,heifs,icns,ico,im,j2p,j2k,jp2,jpeg,jpg,jpx,pbm,pcx,pgm,png,pnm,ppm,psd,qoi,sgi,spi,tga,tif,tiff,webp,wmf,xbm,xpm", help="image formats to decode using pillow")
+    ap2.add_argument("--th-r-pil", metavar="T,T", type=u, default="avif,avifs,blp,bmp,cbz,dcx,dds,dib,emf,eps,fits,flc,fli,fpx,gif,heic,heics,heif,heifs,icns,ico,im,j2p,j2k,jp2,jpeg,jpg,jpx,pbm,pcx,pgm,png,pnm,ppm,psd,qoi,sgi,spi,tga,tif,tiff,webp,wmf,xbm,xpm", help="image formats to decode using pillow")
     ap2.add_argument("--th-r-vips", metavar="T,T", type=u, default="avif,exr,fit,fits,fts,gif,hdr,heic,jp2,jpeg,jpg,jpx,jxl,nii,pfm,pgm,png,ppm,svg,tif,tiff,webp", help="image formats to decode using pyvips")
-    ap2.add_argument("--th-r-ffi", metavar="T,T", type=u, default="apng,avif,avifs,bmp,dds,dib,fit,fits,fts,gif,hdr,heic,heics,heif,heifs,icns,ico,jp2,jpeg,jpg,jpx,jxl,pbm,pcx,pfm,pgm,png,pnm,ppm,psd,qoi,sgi,tga,tif,tiff,webp,xbm,xpm", help="image formats to decode using ffmpeg")
+    ap2.add_argument("--th-r-ffi", metavar="T,T", type=u, default="apng,avif,avifs,bmp,cbz,dds,dib,fit,fits,fts,gif,hdr,heic,heics,heif,heifs,icns,ico,jp2,jpeg,jpg,jpx,jxl,pbm,pcx,pfm,pgm,png,pnm,ppm,psd,qoi,sgi,tga,tif,tiff,webp,xbm,xpm", help="image formats to decode using ffmpeg")
     ap2.add_argument("--th-r-ffv", metavar="T,T", type=u, default="3gp,asf,av1,avc,avi,flv,h264,h265,hevc,m4v,mjpeg,mjpg,mkv,mov,mp4,mpeg,mpeg2,mpegts,mpg,mpg2,mts,nut,ogm,ogv,rm,ts,vob,webm,wmv", help="video formats to decode using ffmpeg")
     ap2.add_argument("--th-r-ffa", metavar="T,T", type=u, default="aac,ac3,aif,aiff,alac,alaw,amr,apac,ape,au,bonk,dfpwm,dts,flac,gsm,ilbc,it,itgz,itxz,itz,m4a,mdgz,mdxz,mdz,mo3,mod,mp2,mp3,mpc,mptm,mt2,mulaw,ogg,okt,opus,ra,s3m,s3gz,s3xz,s3z,tak,tta,ulaw,wav,wma,wv,xm,xmgz,xmxz,xmz,xpk", help="audio formats to decode using ffmpeg")
-    ap2.add_argument("--au-unpk", metavar="E=F.C", type=u, default="mdz=mod.zip, mdgz=mod.gz, mdxz=mod.xz, s3z=s3m.zip, s3gz=s3m.gz, s3xz=s3m.xz, xmz=xm.zip, xmgz=xm.gz, xmxz=xm.xz, itz=it.zip, itgz=it.gz, itxz=it.xz", help="audio formats to decompress before passing to ffmpeg")
+    ap2.add_argument("--au-unpk", metavar="E=F.C", type=u, default="mdz=mod.zip, mdgz=mod.gz, mdxz=mod.xz, s3z=s3m.zip, s3gz=s3m.gz, s3xz=s3m.xz, xmz=xm.zip, xmgz=xm.gz, xmxz=xm.xz, itz=it.zip, itgz=it.gz, itxz=it.xz, cbz=jpg.cbz", help="audio/image formats to decompress before passing to ffmpeg")
 
 
 def add_transcoding(ap):
     ap2 = ap.add_argument_group('transcoding options')
     ap2.add_argument("--q-opus", metavar="KBPS", type=int, default=128, help="target bitrate for transcoding to opus; set 0 to disable")
     ap2.add_argument("--q-mp3", metavar="QUALITY", type=u, default="q2", help="target quality for transcoding to mp3, for example [\033[32m192k\033[0m] (CBR) or [\033[32mq0\033[0m] (CQ/CRF, q0=maxquality, q9=smallest); set 0 to disable")
+    ap2.add_argument("--no-caf", action="store_true", help="disable transcoding to caf-opus (affects iOS v12~v17), will use mp3 instead")
+    ap2.add_argument("--no-owa", action="store_true", help="disable transcoding to webm-opus (iOS v18 and later), will use mp3 instead")
     ap2.add_argument("--no-acode", action="store_true", help="disable audio transcoding")
     ap2.add_argument("--no-bacode", action="store_true", help="disable batch audio transcoding by folder download (zip/tar)")
     ap2.add_argument("--ac-maxage", metavar="SEC", type=int, default=86400, help="delete cached transcode output after \033[33mSEC\033[0m seconds")
@@ -1390,6 +1424,7 @@ def add_db_general(ap, hcores):
     ap2.add_argument("--db-act", metavar="SEC", type=float, default=10.0, help="defer any scheduled volume reindexing until \033[33mSEC\033[0m seconds after last db write (uploads, renames, ...)")
     ap2.add_argument("--srch-time", metavar="SEC", type=int, default=45, help="search deadline -- terminate searches running for more than \033[33mSEC\033[0m seconds")
     ap2.add_argument("--srch-hits", metavar="N", type=int, default=7999, help="max search results to allow clients to fetch; 125 results will be shown initially")
+    ap2.add_argument("--srch-excl", metavar="PTN", type=u, default="", help="regex: exclude files from search results if the file-URL matches \033[33mPTN\033[0m (case-sensitive). Example: [\033[32mpassword|logs/[0-9]\033[0m] any URL containing 'password' or 'logs/DIGIT' (volflag=srch_excl)")
     ap2.add_argument("--dotsrch", action="store_true", help="show dotfiles in search results (volflags: dotsrch | nodotsrch)")
 
 
@@ -1446,6 +1481,8 @@ def add_ui(ap, retry):
     ap2.add_argument("--themes", metavar="NUM", type=int, default=8, help="number of themes installed")
     ap2.add_argument("--au-vol", metavar="0-100", type=int, default=50, choices=range(0, 101), help="default audio/video volume percent")
     ap2.add_argument("--sort", metavar="C,C,C", type=u, default="href", help="default sort order, comma-separated column IDs (see header tooltips), prefix with '-' for descending. Examples: \033[32mhref -href ext sz ts tags/Album tags/.tn\033[0m (volflag=sort)")
+    ap2.add_argument("--nsort", action="store_true", help="default-enable natural sort of filenames with leading numbers (volflag=nsort)")
+    ap2.add_argument("--hsortn", metavar="N", type=int, default=2, help="number of sorting rules to include in media URLs by default (volflag=hsortn)")
     ap2.add_argument("--unlist", metavar="REGEX", type=u, default="", help="don't show files matching \033[33mREGEX\033[0m in file list. Purely cosmetic! Does not affect API calls, just the browser. Example: [\033[32m\\.(js|css)$\033[0m] (volflag=unlist)")
     ap2.add_argument("--favico", metavar="TXT", type=u, default="c 000 none" if retry else "🎉 000 none", help="\033[33mfavicon-text\033[0m [ \033[33mforeground\033[0m [ \033[33mbackground\033[0m ] ], set blank to disable")
     ap2.add_argument("--mpmc", metavar="URL", type=u, default="", help="change the mediaplayer-toggle mouse cursor; URL to a folder with {2..5}.png inside (or disable with [\033[32m.\033[0m])")
@@ -1458,12 +1495,14 @@ def add_ui(ap, retry):
     ap2.add_argument("--txt-max", metavar="KiB", type=int, default=64, help="max size of embedded textfiles on ?doc= (anything bigger will be lazy-loaded by JS)")
     ap2.add_argument("--doctitle", metavar="TXT", type=u, default="copyparty @ --name", help="title / service-name to show in html documents")
     ap2.add_argument("--bname", metavar="TXT", type=u, default="--name", help="server name (displayed in filebrowser document title)")
-    ap2.add_argument("--pb-url", metavar="URL", type=u, default="https://github.com/9001/copyparty", help="powered-by link; disable with \033[33m-np\033[0m")
+    ap2.add_argument("--pb-url", metavar="URL", type=u, default=URL_PRJ, help="powered-by link; disable with \033[33m-np\033[0m")
     ap2.add_argument("--ver", action="store_true", help="show version on the control panel (incompatible with \033[33m-nb\033[0m)")
     ap2.add_argument("--k304", metavar="NUM", type=int, default=0, help="configure the option to enable/disable k304 on the controlpanel (workaround for buggy reverse-proxies); [\033[32m0\033[0m] = hidden and default-off, [\033[32m1\033[0m] = visible and default-off, [\033[32m2\033[0m] = visible and default-on")
     ap2.add_argument("--no304", metavar="NUM", type=int, default=0, help="configure the option to enable/disable no304 on the controlpanel (workaround for buggy caching in browsers); [\033[32m0\033[0m] = hidden and default-off, [\033[32m1\033[0m] = visible and default-off, [\033[32m2\033[0m] = visible and default-on")
-    ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#attr-sandbox")
+    ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to allow in the iframe 'sandbox' attribute for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#sandbox")
-    ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for prologue/epilogue docs  (volflag=lg_sbf)")
+    ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to allow in the iframe 'sandbox' attribute for prologue/epilogue docs (volflag=lg_sbf)")
+    ap2.add_argument("--md-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for README.md docs, for example [\033[32mfullscreen\033[0m] (volflag=md_sba)")
+    ap2.add_argument("--lg-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for prologue/epilogue docs (volflag=lg_sba); see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes")
     ap2.add_argument("--no-sb-md", action="store_true", help="don't sandbox README/PREADME.md documents (volflags: no_sb_md | sb_md)")
     ap2.add_argument("--no-sb-lg", action="store_true", help="don't sandbox prologue/epilogue docs (volflags: no_sb_lg | sb_lg); enables non-js support")
 
@@ -1489,6 +1528,7 @@ def add_debug(ap):
     ap2.add_argument("--bf-nc", metavar="NUM", type=int, default=200, help="bak-flips: stop if there's more than \033[33mNUM\033[0m files at \033[33m--kf-dir\033[0m already; default: 6.3 GiB max (200*32M)")
     ap2.add_argument("--bf-dir", metavar="PATH", type=u, default="bf", help="bak-flips: store corrupted chunks at \033[33mPATH\033[0m; default: folder named 'bf' wherever copyparty was started")
     ap2.add_argument("--bf-log", metavar="PATH", type=u, default="", help="bak-flips: log corruption info to a textfile at \033[33mPATH\033[0m")
+    ap2.add_argument("--no-cfg-cmt-warn", action="store_true", help=argparse.SUPPRESS)
 
 
 # fmt: on
@@ -1721,7 +1761,7 @@ def main(argv: Optional[list[str]] = None) -> None:
         except:
             lprint("\nfailed to disable quick-edit-mode:\n" + min_ex() + "\n")
 
-    if al.ansi:
+    if not al.ansi:
         al.wintitle = ""
 
     # propagate implications

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 15, 10)
+VERSION = (1, 16, 10)
-CODENAME = "fill the drives"
+CODENAME = "COPYparty"
-BUILD_DT = (2024, 10, 26)
+BUILD_DT = (2025, 1, 25)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -212,7 +212,7 @@ class Lim(object):
 
             df, du, err = get_df(abspath, True)
             if err:
-                t = "failed to read disk space usage for [%s]: %s"
+                t = "failed to read disk space usage for %r: %s"
                 self.log(t % (abspath, err), 3)
                 self.dfv = 0xAAAAAAAAA  # 42.6 GiB
             else:
@@ -367,18 +367,21 @@ class VFS(object):
         self.ahtml: dict[str, list[str]] = {}
         self.aadmin: dict[str, list[str]] = {}
         self.adot: dict[str, list[str]] = {}
-        self.all_vols: dict[str, VFS] = {}
+        self.js_ls = {}
+        self.js_htm = ""
 
         if realpath:
             rp = realpath + ("" if realpath.endswith(os.sep) else os.sep)
             vp = vpath + ("/" if vpath else "")
             self.histpath = os.path.join(realpath, ".hist")  # db / thumbcache
             self.all_vols = {vpath: self}  # flattened recursive
+            self.all_nodes = {vpath: self}  # also jumpvols
             self.all_aps = [(rp, self)]
             self.all_vps = [(vp, self)]
         else:
             self.histpath = ""
             self.all_vols = {}
+            self.all_nodes = {}
             self.all_aps = []
             self.all_vps = []
 
@@ -396,9 +399,11 @@ class VFS(object):
     def get_all_vols(
         self,
         vols: dict[str, "VFS"],
+        nodes: dict[str, "VFS"],
         aps: list[tuple[str, "VFS"]],
         vps: list[tuple[str, "VFS"]],
     ) -> None:
+        nodes[self.vpath] = self
         if self.realpath:
             vols[self.vpath] = self
             rp = self.realpath
@@ -408,7 +413,7 @@ class VFS(object):
             vps.append((vp, self))
 
         for v in self.nodes.values():
-            v.get_all_vols(vols, aps, vps)
+            v.get_all_vols(vols, nodes, aps, vps)
 
     def add(self, src: str, dst: str) -> "VFS":
         """get existing, or add new path to the vfs"""
@@ -521,7 +526,7 @@ class VFS(object):
         """returns [vfsnode,fs_remainder] if user has the requested permissions"""
         if relchk(vpath):
             if self.log:
-                self.log("vfs", "invalid relpath [{}]".format(vpath))
+                self.log("vfs", "invalid relpath %r @%s" % (vpath, uname))
             raise Pebkac(422)
 
         cvpath = undot(vpath)
@@ -538,11 +543,11 @@ class VFS(object):
             if req and uname not in d and uname != LEELOO_DALLAS:
                 if vpath != cvpath and vpath != "." and self.log:
                     ap = vn.canonical(rem)
-                    t = "{} has no {} in [{}] => [{}] => [{}]"
+                    t = "%s has no %s in %r => %r => %r"
-                    self.log("vfs", t.format(uname, msg, vpath, cvpath, ap), 6)
+                    self.log("vfs", t % (uname, msg, vpath, cvpath, ap), 6)
 
-                t = 'you don\'t have %s-access in "/%s" or below "/%s"'
+                t = "you don't have %s-access in %r or below %r"
-                raise Pebkac(err, t % (msg, cvpath, vn.vpath))
+                raise Pebkac(err, t % (msg, "/" + cvpath, "/" + vn.vpath))
 
         return vn, rem
 
@@ -591,10 +596,11 @@ class VFS(object):
         scandir: bool,
         permsets: list[list[bool]],
         lstat: bool = False,
+        throw: bool = False,
     ) -> tuple[str, list[tuple[str, os.stat_result]], dict[str, "VFS"]]:
         """replaces _ls for certain shares (single-file, or file selection)"""
         vn, rem = self.shr_src  # type: ignore
-        abspath, real, _ = vn.ls(rem, "\n", scandir, permsets, lstat)
+        abspath, real, _ = vn.ls(rem, "\n", scandir, permsets, lstat, throw)
         real = [x for x in real if os.path.basename(x[0]) in self.shr_files]
         return abspath, real, {}
 
@@ -605,11 +611,12 @@ class VFS(object):
         scandir: bool,
         permsets: list[list[bool]],
         lstat: bool = False,
+        throw: bool = False,
     ) -> tuple[str, list[tuple[str, os.stat_result]], dict[str, "VFS"]]:
         """return user-readable [fsdir,real,virt] items at vpath"""
         virt_vis = {}  # nodes readable by user
         abspath = self.canonical(rem)
-        real = list(statdir(self.log, scandir, lstat, abspath))
+        real = list(statdir(self.log, scandir, lstat, abspath, throw))
         real.sort()
         if not rem:
             # no vfs nodes in the list of real inodes
@@ -651,7 +658,7 @@ class VFS(object):
         seen: list[str],
         uname: str,
         permsets: list[list[bool]],
-        wantdots: bool,
+        wantdots: int,
         scandir: bool,
         lstat: bool,
         subvols: bool = True,
@@ -671,6 +678,10 @@ class VFS(object):
         """
         recursively yields from ./rem;
         rel is a unix-style user-defined vpath (not vfs-related)
+
+        NOTE: don't invoke this function from a dbv; subvols are only
+          descended into if rem is blank due to the _ls `if not rem:`
+          which intention is to prevent unintended access to subvols
         """
 
         fsroot, vfs_ls, vfs_virt = self.ls(rem, uname, scandir, permsets, lstat=lstat)
@@ -682,8 +693,8 @@ class VFS(object):
             and fsroot in seen
         ):
             if self.log:
-                t = "bailing from symlink loop,\n  prev: {}\n  curr: {}\n  from: {}/{}"
+                t = "bailing from symlink loop,\n  prev: %r\n  curr: %r\n  from: %r / %r"
-                self.log("vfs.walk", t.format(seen[-1], fsroot, self.vpath, rem), 3)
+                self.log("vfs.walk", t % (seen[-1], fsroot, self.vpath, rem), 3)
             return
 
         if "xdev" in self.flags or "xvol" in self.flags:
@@ -695,7 +706,7 @@ class VFS(object):
                     rm1.append(le)
             _ = [vfs_ls.remove(x) for x in rm1]  # type: ignore
 
-        dots_ok = wantdots and uname in dbv.axs.udot
+        dots_ok = wantdots and (wantdots == 2 or uname in dbv.axs.udot)
         if not dots_ok:
             vfs_ls = [x for x in vfs_ls if "/." not in "/" + x[0]]
 
@@ -749,7 +760,7 @@ class VFS(object):
         # if single folder: the folder itself is the top-level item
         folder = "" if flt or not wrap else (vpath.split("/")[-1].lstrip(".") or "top")
 
-        g = self.walk(folder, vrem, [], uname, [[True, False]], True, scandir, False)
+        g = self.walk(folder, vrem, [], uname, [[True, False]], 1, scandir, False)
         for _, _, vpath, apath, files, rd, vd in g:
             if flt:
                 files = [x for x in files if x[0] in flt]
@@ -807,8 +818,8 @@ class VFS(object):
 
             if vdev != st.st_dev:
                 if self.log:
-                    t = "xdev: {}[{}] => {}[{}]"
+                    t = "xdev: %s[%r] => %s[%r]"
-                    self.log("vfs", t.format(vdev, self.realpath, st.st_dev, ap), 3)
+                    self.log("vfs", t % (vdev, self.realpath, st.st_dev, ap), 3)
 
                 return None
 
@@ -818,7 +829,7 @@ class VFS(object):
                     return vn
 
             if self.log:
-                self.log("vfs", "xvol: [{}]".format(ap), 3)
+                self.log("vfs", "xvol: %r" % (ap,), 3)
 
             return None
 
@@ -903,7 +914,7 @@ class AuthSrv(object):
 
             self.idp_accs[uname] = gnames
 
-            t = "reinitializing due to new user from IdP: [%s:%s]"
+            t = "reinitializing due to new user from IdP: [%r:%r]"
             self.log(t % (uname, gnames), 3)
 
             if not broker:
@@ -911,7 +922,7 @@ class AuthSrv(object):
                 self._reload()
                 return True
 
-        broker.ask("_reload_blocking", False).get()
+        broker.ask("reload", False, True).get()
         return True
 
     def _map_volume_idp(
@@ -1381,7 +1392,7 @@ class AuthSrv(object):
             flags[name] = True
             return
 
-        zs = "mtp on403 on404 xbu xau xiu xbr xar xbd xad xm xban"
+        zs = "mtp on403 on404 xbu xau xiu xbc xac xbr xar xbd xad xm xban"
         if name not in zs.split():
             if value is True:
                 t = "└─add volflag [{}] = {}  ({})"
@@ -1529,10 +1540,11 @@ class AuthSrv(object):
 
         assert vfs  # type: ignore
         vfs.all_vols = {}
+        vfs.all_nodes = {}
         vfs.all_aps = []
         vfs.all_vps = []
-        vfs.get_all_vols(vfs.all_vols, vfs.all_aps, vfs.all_vps)
+        vfs.get_all_vols(vfs.all_vols, vfs.all_nodes, vfs.all_aps, vfs.all_vps)
-        for vol in vfs.all_vols.values():
+        for vol in vfs.all_nodes.values():
             vol.all_aps.sort(key=lambda x: len(x[0]), reverse=True)
             vol.all_vps.sort(key=lambda x: len(x[0]), reverse=True)
             vol.root = vfs
@@ -1556,7 +1568,7 @@ class AuthSrv(object):
                     continue
 
                 if self.args.shr_v:
-                    t = "loading %s share [%s] by [%s] => [%s]"
+                    t = "loading %s share %r by %r => %r"
                     self.log(t % (s_pr, s_k, s_un, s_vp))
 
                 if s_pw:
@@ -1583,7 +1595,7 @@ class AuthSrv(object):
 
             vfs.nodes[shr] = vfs.all_vols[shr] = shv
             for vol in shv.nodes.values():
-                vfs.all_vols[vol.vpath] = vol
+                vfs.all_vols[vol.vpath] = vfs.all_nodes[vol.vpath] = vol
                 vol.get_dbv = vol._get_share_src
                 vol.ls = vol._ls_nope
 
@@ -1726,7 +1738,19 @@ class AuthSrv(object):
 
             self.log("\n\n".join(ta) + "\n", c=3)
 
-        vfs.histtab = {zv.realpath: zv.histpath for zv in vfs.all_vols.values()}
+        rhisttab = {}
+        vfs.histtab = {}
+        for zv in vfs.all_vols.values():
+            histp = zv.histpath
+            is_shr = shr and zv.vpath.split("/")[0] == shr
+            if histp and not is_shr and histp in rhisttab:
+                zv2 = rhisttab[histp]
+                t = "invalid config; multiple volumes share the same histpath (database location):\n  histpath: %s\n  volume 1: /%s  [%s]\n  volume 2: %s  [%s]"
+                t = t % (histp, zv2.vpath, zv2.realpath, zv.vpath, zv.realpath)
+                self.log(t, 1)
+                raise Exception(t)
+            rhisttab[histp] = zv
+            vfs.histtab[zv.realpath] = histp
 
         for vol in vfs.all_vols.values():
             lim = Lim(self.log_func)
@@ -1741,7 +1765,7 @@ class AuthSrv(object):
                 use = True
                 try:
                     _ = float(zs)
-                    zs = "%sg" % (zs)
+                    zs = "%sg" % (zs,)
                 except:
                     pass
                 lim.dfl = unhumanize(zs)
@@ -1785,12 +1809,12 @@ class AuthSrv(object):
                 vol.lim = lim
 
         if self.args.no_robots:
-            for vol in vfs.all_vols.values():
+            for vol in vfs.all_nodes.values():
                 # volflag "robots" overrides global "norobots", allowing indexing by search engines for this vol
                 if not vol.flags.get("robots"):
                     vol.flags["norobots"] = True
 
-        for vol in vfs.all_vols.values():
+        for vol in vfs.all_nodes.values():
             if self.args.no_vthumb:
                 vol.flags["dvthumb"] = True
             if self.args.no_athumb:
@@ -1802,13 +1826,17 @@ class AuthSrv(object):
                 vol.flags["dithumb"] = True
 
         have_fk = False
-        for vol in vfs.all_vols.values():
+        for vol in vfs.all_nodes.values():
             fk = vol.flags.get("fk")
             fka = vol.flags.get("fka")
             if fka and not fk:
                 fk = fka
             if fk:
-                vol.flags["fk"] = int(fk) if fk is not True else 8
+                fk = 8 if fk is True else int(fk)
+                if fk > 72:
+                    t = "max filekey-length is 72; volume /%s specified %d (anything higher than 16 is pointless btw)"
+                    raise Exception(t % (vol.vpath, fk))
+                vol.flags["fk"] = fk
                 have_fk = True
 
             dk = vol.flags.get("dk")
@@ -1834,7 +1862,7 @@ class AuthSrv(object):
             zs = os.path.join(E.cfg, "fk-salt.txt")
             self.log(t % (fk_len, 16, zs), 3)
 
-        for vol in vfs.all_vols.values():
+        for vol in vfs.all_nodes.values():
             if "pk" in vol.flags and "gz" not in vol.flags and "xz" not in vol.flags:
                 vol.flags["gz"] = False  # def.pk
 
@@ -1845,7 +1873,7 @@ class AuthSrv(object):
 
         all_mte = {}
         errors = False
-        for vol in vfs.all_vols.values():
+        for vol in vfs.all_nodes.values():
             if (self.args.e2ds and vol.axs.uwrite) or self.args.e2dsa:
                 vol.flags["e2ds"] = True
 
@@ -1856,6 +1884,7 @@ class AuthSrv(object):
                 ["no_hash", "nohash"],
                 ["no_idx", "noidx"],
                 ["og_ua", "og_ua"],
+                ["srch_excl", "srch_excl"],
             ]:
                 if vf in vol.flags:
                     ptn = re.compile(vol.flags.pop(vf))
@@ -1936,7 +1965,7 @@ class AuthSrv(object):
                     vol.flags[k] = odfusion(getattr(self.args, k), vol.flags[k])
 
             # append additive args from argv to volflags
-            hooks = "xbu xau xiu xbr xar xbd xad xm xban".split()
+            hooks = "xbu xau xiu xbc xac xbr xar xbd xad xm xban".split()
             for name in "mtp on404 on403".split() + hooks:
                 self._read_volflag(vol.flags, name, getattr(self.args, name), True)
 
@@ -2062,8 +2091,24 @@ class AuthSrv(object):
                 self.log(t.format(mtp), 1)
                 errors = True
 
-        have_daw = False
         for vol in vfs.all_vols.values():
+            re1: Optional[re.Pattern] = vol.flags.get("srch_excl")
+            excl = [re1.pattern] if re1 else []
+
+            vpaths = []
+            vtop = vol.vpath
+            for vp2 in vfs.all_vols.keys():
+                if vp2.startswith((vtop + "/").lstrip("/")) and vtop != vp2:
+                    vpaths.append(re.escape(vp2[len(vtop) :].lstrip("/")))
+            if vpaths:
+                excl.append("^(%s)/" % ("|".join(vpaths),))
+
+            vol.flags["srch_re_dots"] = re.compile("|".join(excl or ["^$"]))
+            excl.extend([r"^\.", r"/\."])
+            vol.flags["srch_re_nodot"] = re.compile("|".join(excl))
+
+        have_daw = False
+        for vol in vfs.all_nodes.values():
             daw = vol.flags.get("daw") or self.args.daw
             if daw:
                 vol.flags["daw"] = True
@@ -2078,13 +2123,12 @@ class AuthSrv(object):
             self.log("--smb can only be used when --ah-alg is none", 1)
             errors = True
 
-        for vol in vfs.all_vols.values():
+        for vol in vfs.all_nodes.values():
             for k in list(vol.flags.keys()):
                 if re.match("^-[^-]+$", k):
                     vol.flags.pop(k[1:], None)
                     vol.flags.pop(k)
 
-        for vol in vfs.all_vols.values():
             if vol.flags.get("dots"):
                 for name in vol.axs.uread:
                     vol.axs.udot.add(name)
@@ -2141,11 +2185,11 @@ class AuthSrv(object):
             if not self.args.no_voldump:
                 self.log(t)
 
-            if have_e2d:
+            if have_e2d or self.args.idp_h_usr:
                 t = self.chk_sqlite_threadsafe()
                 if t:
                     self.log("\n\033[{}\033[0m\n".format(t))
-
+            if have_e2d:
                 if not have_e2t:
                     t = "hint: enable multimedia indexing (artist/title/...) with argument -e2ts"
                     self.log(t, 6)
@@ -2226,6 +2270,11 @@ class AuthSrv(object):
                 for x, y in vfs.all_vols.items()
                 if x != shr and not x.startswith(shrs)
             }
+            vfs.all_nodes = {
+                x: y
+                for x, y in vfs.all_nodes.items()
+                if x != shr and not x.startswith(shrs)
+            }
 
             assert db and cur and cur2 and shv  # type: ignore
             for row in cur.execute("select * from sh"):
@@ -2278,6 +2327,74 @@ class AuthSrv(object):
             cur.close()
             db.close()
 
+        self.js_ls = {}
+        self.js_htm = {}
+        for vn in self.vfs.all_nodes.values():
+            vf = vn.flags
+            vn.js_ls = {
+                "idx": "e2d" in vf,
+                "itag": "e2t" in vf,
+                "dnsort": "nsort" in vf,
+                "dhsortn": vf["hsortn"],
+                "dsort": vf["sort"],
+                "dcrop": vf["crop"],
+                "dth3x": vf["th3x"],
+                "u2ts": vf["u2ts"],
+                "frand": bool(vf.get("rand")),
+                "lifetime": vf.get("lifetime") or 0,
+                "unlist": vf.get("unlist") or "",
+                "sb_lg": "" if "no_sb_lg" in vf else (vf.get("lg_sbf") or "y"),
+            }
+            js_htm = {
+                "s_name": self.args.bname,
+                "have_up2k_idx": "e2d" in vf,
+                "have_acode": not self.args.no_acode,
+                "have_shr": self.args.shr,
+                "have_zip": not self.args.no_zip,
+                "have_mv": not self.args.no_mv,
+                "have_del": not self.args.no_del,
+                "have_unpost": int(self.args.unpost),
+                "have_emp": self.args.emp,
+                "sb_md": "" if "no_sb_md" in vf else (vf.get("md_sbf") or "y"),
+                "sba_md": vf.get("md_sba") or "",
+                "sba_lg": vf.get("lg_sba") or "",
+                "txt_ext": self.args.textfiles.replace(",", " "),
+                "def_hcols": list(vf.get("mth") or []),
+                "unlist0": vf.get("unlist") or "",
+                "dgrid": "grid" in vf,
+                "dgsel": "gsel" in vf,
+                "dnsort": "nsort" in vf,
+                "dhsortn": vf["hsortn"],
+                "dsort": vf["sort"],
+                "dcrop": vf["crop"],
+                "dth3x": vf["th3x"],
+                "dvol": self.args.au_vol,
+                "idxh": int(self.args.ih),
+                "themes": self.args.themes,
+                "turbolvl": self.args.turbo,
+                "u2j": self.args.u2j,
+                "u2sz": self.args.u2sz,
+                "u2ts": vf["u2ts"],
+                "frand": bool(vf.get("rand")),
+                "lifetime": vn.js_ls["lifetime"],
+                "u2sort": self.args.u2sort,
+            }
+            vn.js_htm = json.dumps(js_htm)
+
+        vols = list(vfs.all_nodes.values())
+        if enshare:
+            assert shv  # type: ignore  # !rm
+            vols.append(shv)
+            vols.extend(list(shv.nodes.values()))
+
+        for vol in vols:
+            dbv = vol.get_dbv("")[0]
+            vol.js_ls = vol.js_ls or dbv.js_ls or {}
+            vol.js_htm = vol.js_htm or dbv.js_htm or "{}"
+
+            zs = str(vol.flags.get("tcolor") or self.args.tcolor)
+            vol.flags["tcolor"] = zs.lstrip("#")
+
     def load_sessions(self, quiet=False) -> None:
         # mutex me
         if self.args.no_ses:
@@ -2387,7 +2504,7 @@ class AuthSrv(object):
                 self._reload()
                 return True, "new password OK"
 
-        broker.ask("_reload_blocking", False, False).get()
+        broker.ask("reload", False, False).get()
         return True, "new password OK"
 
     def setup_chpw(self, acct: dict[str, str]) -> None:
@@ -2428,7 +2545,7 @@ class AuthSrv(object):
             return
 
         elif self.args.chpw_v == 2:
-            t = "chpw: %d changed" % (len(uok))
+            t = "chpw: %d changed" % (len(uok),)
             if urst:
                 t += ", \033[0munchanged:\033[35m %s" % (", ".join(list(urst)))
 
@@ -2586,7 +2703,7 @@ class AuthSrv(object):
                     [],
                     u,
                     [[True, False]],
-                    True,
+                    1,
                     not self.args.no_scandir,
                     False,
                     False,
@@ -2639,7 +2756,7 @@ class AuthSrv(object):
         ]
 
         csv = set("i p th_covers zm_on zm_off zs_on zs_off".split())
-        zs = "c ihead ohead mtm mtp on403 on404 xad xar xau xiu xban xbd xbr xbu xm"
+        zs = "c ihead ohead mtm mtp on403 on404 xac xad xar xau xiu xban xbc xbd xbr xbu xm"
         lst = set(zs.split())
         askip = set("a v c vc cgen exp_lg exp_md theme".split())
         fskip = set("exp_lg exp_md mv_re_r mv_re_t rm_re_r rm_re_t".split())
@@ -2907,6 +3024,19 @@ def expand_config_file(
 
     ret.append("#\033[36m closed{}\033[0m".format(ipath))
 
+    zsl = []
+    for ln in ret:
+        zs = ln.split("  #")[0]
+        if " #" in zs and zs.split("#")[0].strip():
+            zsl.append(ln)
+    if zsl and "no-cfg-cmt-warn" not in "\n".join(ret):
+        t = "\033[33mWARNING: there is less than two spaces before the # in the following config lines, so instead of assuming that this is a comment, the whole line will become part of the config value:\n\n>>> %s\n\nif you are familiar with this and would like to mute this warning, specify the global-option no-cfg-cmt-warn\n\033[0m"
+        t = t % ("\n>>> ".join(zsl),)
+        if log:
+            log(t)
+        else:
+            print(t, file=sys.stderr)
+
 
 def upgrade_cfg_fmt(
     log: Optional["NamedLogger"], args: argparse.Namespace, orig: list[str], cfg_fp: str

copyparty/broker_mp.py

@@ -43,6 +43,9 @@ class BrokerMp(object):
         self.procs = []
         self.mutex = threading.Lock()
 
+        self.retpend: dict[int, Any] = {}
+        self.retpend_mutex = threading.Lock()
+
         self.num_workers = self.args.j or CORES
         self.log("broker", "booting {} subprocesses".format(self.num_workers))
         for n in range(1, self.num_workers + 1):
@@ -54,6 +57,8 @@ class BrokerMp(object):
             self.procs.append(proc)
             proc.start()
 
+        Daemon(self.periodic, "mp-periodic")
+
     def shutdown(self) -> None:
         self.log("broker", "shutting down")
         for n, proc in enumerate(self.procs):
@@ -90,8 +95,10 @@ class BrokerMp(object):
                 self.log(*args)
 
             elif dest == "retq":
-                # response from previous ipc call
+                with self.retpend_mutex:
-                raise Exception("invalid broker_mp usage")
+                    retq = self.retpend.pop(retq_id)
+
+                retq.put(args[0])
 
             else:
                 # new ipc invoking managed service in hub
@@ -109,7 +116,6 @@ class BrokerMp(object):
                     proc.q_pend.put((retq_id, "retq", rv))
 
     def ask(self, dest: str, *args: Any) -> Union[ExceptionalQueue, NotExQueue]:
-
         # new non-ipc invoking managed service in hub
         obj = self.hub
         for node in dest.split("."):
@@ -121,17 +127,30 @@ class BrokerMp(object):
         retq.put(rv)
         return retq
 
+    def wask(self, dest: str, *args: Any) -> list[Union[ExceptionalQueue, NotExQueue]]:
+        # call from hub to workers
+        ret = []
+        for p in self.procs:
+            retq = ExceptionalQueue(1)
+            retq_id = id(retq)
+            with self.retpend_mutex:
+                self.retpend[retq_id] = retq
+
+            p.q_pend.put((retq_id, dest, list(args)))
+            ret.append(retq)
+        return ret
+
     def say(self, dest: str, *args: Any) -> None:
         """
         send message to non-hub component in other process,
         returns a Queue object which eventually contains the response if want_retval
         (not-impl here since nothing uses it yet)
         """
-        if dest == "listen":
+        if dest == "httpsrv.listen":
             for p in self.procs:
                 p.q_pend.put((0, dest, [args[0], len(self.procs)]))
 
-        elif dest == "set_netdevs":
+        elif dest == "httpsrv.set_netdevs":
             for p in self.procs:
                 p.q_pend.put((0, dest, list(args)))
 
@@ -140,3 +159,19 @@ class BrokerMp(object):
 
         else:
             raise Exception("what is " + str(dest))
+
+    def periodic(self) -> None:
+        while True:
+            time.sleep(1)
+
+            tdli = {}
+            tdls = {}
+            qs = self.wask("httpsrv.read_dls")
+            for q in qs:
+                qr = q.get()
+                dli, dls = qr
+                tdli.update(dli)
+                tdls.update(dls)
+            tdl = (tdli, tdls)
+            for p in self.procs:
+                p.q_pend.put((0, "httpsrv.write_dls", tdl))

copyparty/broker_mpw.py

@@ -82,37 +82,38 @@ class MpWorker(BrokerCli):
         while True:
             retq_id, dest, args = self.q_pend.get()
 
-            # self.logw("work: [{}]".format(d[0]))
+            if dest == "retq":
+                # response from previous ipc call
+                with self.retpend_mutex:
+                    retq = self.retpend.pop(retq_id)
+
+                retq.put(args)
+                continue
+
             if dest == "shutdown":
                 self.httpsrv.shutdown()
                 self.logw("ok bye")
                 sys.exit(0)
                 return
 
-            elif dest == "reload":
+            if dest == "reload":
                 self.logw("mpw.asrv reloading")
                 self.asrv.reload()
                 self.logw("mpw.asrv reloaded")
+                continue
 
-            elif dest == "reload_sessions":
+            if dest == "reload_sessions":
                 with self.asrv.mutex:
                     self.asrv.load_sessions()
+                continue
 
-            elif dest == "listen":
+            obj = self
-                self.httpsrv.listen(args[0], args[1])
+            for node in dest.split("."):
+                obj = getattr(obj, node)
 
-            elif dest == "set_netdevs":
+            rv = obj(*args)  # type: ignore
-                self.httpsrv.set_netdevs(args[0])
+            if retq_id:
-
+                self.say("retq", rv, retq_id=retq_id)
-            elif dest == "retq":
-                # response from previous ipc call
-                with self.retpend_mutex:
-                    retq = self.retpend.pop(retq_id)
-
-                retq.put(args)
-
-            else:
-                raise Exception("what is " + str(dest))
 
     def ask(self, dest: str, *args: Any) -> Union[ExceptionalQueue, NotExQueue]:
         retq = ExceptionalQueue(1)
@@ -123,5 +124,5 @@ class MpWorker(BrokerCli):
         self.q_yield.put((retq_id, dest, list(args)))
         return retq
 
-    def say(self, dest: str, *args: Any) -> None:
+    def say(self, dest: str, *args: Any, retq_id=0) -> None:
-        self.q_yield.put((0, dest, list(args)))
+        self.q_yield.put((retq_id, dest, list(args)))

copyparty/broker_thr.py

@@ -53,11 +53,11 @@ class BrokerThr(BrokerCli):
         return NotExQueue(obj(*args))  # type: ignore
 
     def say(self, dest: str, *args: Any) -> None:
-        if dest == "listen":
+        if dest == "httpsrv.listen":
             self.httpsrv.listen(args[0], 1)
             return
 
-        if dest == "set_netdevs":
+        if dest == "httpsrv.set_netdevs":
             self.httpsrv.set_netdevs(args[0])
             return
 

copyparty/cfg.py

@@ -42,6 +42,7 @@ def vf_bmap() -> dict[str, str]:
         "magic",
         "no_sb_md",
         "no_sb_lg",
+        "nsort",
         "og",
         "og_no_head",
         "og_s_title",
@@ -69,9 +70,12 @@ def vf_vmap() -> dict[str, str]:
     }
     for k in (
         "dbd",
+        "hsortn",
         "html_head",
         "lg_sbf",
         "md_sbf",
+        "lg_sba",
+        "md_sba",
         "nrand",
         "og_desc",
         "og_site",
@@ -89,6 +93,7 @@ def vf_vmap() -> dict[str, str]:
         "unlist",
         "u2abort",
         "u2ts",
+        "ups_who",
     ):
         ret[k] = k
     return ret
@@ -103,10 +108,12 @@ def vf_cmap() -> dict[str, str]:
         "mte",
         "mth",
         "mtp",
+        "xac",
         "xad",
         "xar",
         "xau",
         "xban",
+        "xbc",
         "xbd",
         "xbr",
         "xbu",
@@ -140,6 +147,7 @@ flagcats = {
         "noclone": "take dupe data from clients, even if available on HDD",
         "nodupe": "rejects existing files (instead of linking/cloning them)",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
+        "nosparse": "deny use of sparse files, mainly for slow storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
         "nosub": "forces all uploads into the top folder of the vfs",
         "magic": "enables filetype detection for nameless uploads",
@@ -188,6 +196,7 @@ flagcats = {
         "xvol": "do not follow symlinks leaving the volume root",
         "dotsrch": "show dotfiles in search results",
         "nodotsrch": "hide dotfiles in search results (default)",
+        "srch_excl": "exclude search results with URL matching this regex",
     },
     'database, audio tags\n"mte", "mth", "mtp", "mtm" all work the same as -mte, -mth, ...': {
         "mtp=.bpm=f,audio-bpm.py": 'uses the "audio-bpm.py" program to\ngenerate ".bpm" tags from uploads (f = overwrite tags)',
@@ -212,6 +221,8 @@ flagcats = {
         "xbu=CMD": "execute CMD before a file upload starts",
         "xau=CMD": "execute CMD after  a file upload finishes",
         "xiu=CMD": "execute CMD after  all uploads finish and volume is idle",
+        "xbc=CMD": "execute CMD before a file copy",
+        "xac=CMD": "execute CMD after  a file copy",
         "xbr=CMD": "execute CMD before a file rename/move",
         "xar=CMD": "execute CMD after  a file rename/move",
         "xbd=CMD": "execute CMD before a file delete",
@@ -233,6 +244,8 @@ flagcats = {
         "sb_lg": "enable js sandbox for prologue/epilogue (default)",
         "md_sbf": "list of markdown-sandbox safeguards to disable",
         "lg_sbf": "list of *logue-sandbox safeguards to disable",
+        "md_sba": "value of iframe allow-prop for markdown-sandbox",
+        "lg_sba": "value of iframe allow-prop for *logue-sandbox",
         "nohtml": "return html and markdown as text/html",
     },
     "others": {

copyparty/dxml.py

@@ -1,3 +1,6 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
 import importlib
 import sys
 import xml.etree.ElementTree as ET
@@ -8,6 +11,10 @@ if True:  # pylint: disable=using-constant-test
     from typing import Any, Optional
 
 
+class BadXML(Exception):
+    pass
+
+
 def get_ET() -> ET.XMLParser:
     pn = "xml.etree.ElementTree"
     cn = "_elementtree"
@@ -34,7 +41,7 @@ def get_ET() -> ET.XMLParser:
 XMLParser: ET.XMLParser = get_ET()
 
 
-class DXMLParser(XMLParser):  # type: ignore
+class _DXMLParser(XMLParser):  # type: ignore
     def __init__(self) -> None:
         tb = ET.TreeBuilder()
         super(DXMLParser, self).__init__(target=tb)
@@ -49,8 +56,12 @@ class DXMLParser(XMLParser):  # type: ignore
         raise BadXML("{}, {}".format(a, ka))
 
 
-class BadXML(Exception):
+class _NG(XMLParser):  # type: ignore
-    pass
+    def __int__(self) -> None:
+        raise BadXML("dxml selftest failed")
+
+
+DXMLParser = _DXMLParser
 
 
 def parse_xml(txt: str) -> ET.Element:
@@ -59,6 +70,40 @@ def parse_xml(txt: str) -> ET.Element:
     return parser.close()  # type: ignore
 
 
+def selftest() -> bool:
+    qbe = r"""<!DOCTYPE d [
+<!ENTITY a "nice_bakuretsu">
+]>
+<root>&a;&a;&a;</root>"""
+
+    emb = r"""<!DOCTYPE d [
+<!ENTITY a SYSTEM "file:///etc/hostname">
+]>
+<root>&a;</root>"""
+
+    # future-proofing; there's never been any known vulns
+    # regarding DTDs and ET.XMLParser, but might as well
+    # block them since webdav-clients don't use them
+    dtd = r"""<!DOCTYPE d SYSTEM "a.dtd">
+<root>a</root>"""
+
+    for txt in (qbe, emb, dtd):
+        try:
+            parse_xml(txt)
+            t = "WARNING: dxml selftest failed:\n%s\n"
+            print(t % (txt,), file=sys.stderr)
+            return False
+        except BadXML:
+            pass
+
+    return True
+
+
+DXML_OK = selftest()
+if not DXML_OK:
+    DXMLParser = _NG
+
+
 def mktnod(name: str, text: str) -> ET.Element:
     el = ET.Element(name)
     el.text = text

copyparty/fsutil.py

@@ -42,14 +42,14 @@ class Fstab(object):
             self.cache = {}
 
         fs = "ext4"
-        msg = "failed to determine filesystem at [{}]; assuming {}\n{}"
+        msg = "failed to determine filesystem at %r; assuming %s\n%s"
 
         if ANYWIN:
             fs = "vfat"
             try:
                 path = self._winpath(path)
             except:
-                self.log(msg.format(path, fs, min_ex()), 3)
+                self.log(msg % (path, fs, min_ex()), 3)
                 return fs
 
         path = undot(path)
@@ -61,11 +61,11 @@ class Fstab(object):
         try:
             fs = self.get_w32(path) if ANYWIN else self.get_unix(path)
         except:
-            self.log(msg.format(path, fs, min_ex()), 3)
+            self.log(msg % (path, fs, min_ex()), 3)
 
         fs = fs.lower()
         self.cache[path] = fs
-        self.log("found {} at {}".format(fs, path))
+        self.log("found %s at %r" % (fs, path))
         return fs
 
     def _winpath(self, path: str) -> str:

copyparty/ftpd.py

@@ -296,6 +296,7 @@ class FtpFs(AbstractedFS):
                 self.uname,
                 not self.args.no_scandir,
                 [[True, False], [False, True]],
+                throw=True,
             )
             vfs_ls = [x[0] for x in vfs_ls1]
             vfs_ls.extend(vfs_virt.keys())

copyparty/httpsrv.py

@@ -81,6 +81,7 @@ from .util import (
 )
 
 if TYPE_CHECKING:
+    from .authsrv import VFS
     from .broker_util import BrokerCli
     from .ssdp import SSDPr
 
@@ -130,6 +131,12 @@ class HttpSrv(object):
         self.bans: dict[str, int] = {}
         self.aclose: dict[str, int] = {}
 
+        dli: dict[str, tuple[float, int, "VFS", str, str]] = {}  # info
+        dls: dict[str, tuple[float, int]] = {}  # state
+        self.dli = self.tdli = dli
+        self.dls = self.tdls = dls
+        self.iiam = '<img src="%s.cpr/iiam.gif?cache=i" />' % (self.args.SRS,)
+
         self.bound: set[tuple[str, int]] = set()
         self.name = "hsrv" + nsuf
         self.mutex = threading.Lock()
@@ -165,15 +172,16 @@ class HttpSrv(object):
         env = jinja2.Environment()
         env.loader = jinja2.FunctionLoader(lambda f: load_jinja2_resource(self.E, f))
         jn = [
-            "splash",
-            "shares",
-            "svcs",
             "browser",
             "browser2",
-            "msg",
+            "cf",
             "md",
             "mde",
-            "cf",
+            "msg",
+            "rups",
+            "shares",
+            "splash",
+            "svcs",
         ]
         self.j2 = {x: env.get_template(x + ".html") for x in jn}
         self.prism = has_resource(self.E, "web/deps/prism.js.gz")
@@ -187,10 +195,6 @@ class HttpSrv(object):
         self.xff_nm = build_netmap(self.args.xff_src)
         self.xff_lan = build_netmap("lan")
 
-        self.ptn_cc = re.compile(r"[\x00-\x1f]")
-        self.ptn_hsafe = re.compile(r"[\x00-\x1f<>\"'&]")
-        self.uparam_cc_ok = set("doc move tree".split())
-
         self.mallow = "GET HEAD POST PUT DELETE OPTIONS".split()
         if not self.args.no_dav:
             zs = "PROPFIND PROPPATCH LOCK UNLOCK MKCOL COPY MOVE"
@@ -205,6 +209,9 @@ class HttpSrv(object):
             self.start_threads(4)
 
         if nid:
+            self.tdli = {}
+            self.tdls = {}
+
             if self.args.stackmon:
                 start_stackmon(self.args.stackmon, nid)
 
@@ -579,3 +586,32 @@ class HttpSrv(object):
                 ident += "a"
 
             self.u2idx_free[ident] = u2idx
+
+    def read_dls(
+        self,
+    ) -> tuple[
+        dict[str, tuple[float, int, str, str, str]], dict[str, tuple[float, int]]
+    ]:
+        """
+        mp-broker asking for local dl-info + dl-state;
+        reduce overhead by sending just the vfs vpath
+        """
+        dli = {k: (a, b, c.vpath, d, e) for k, (a, b, c, d, e) in self.dli.items()}
+        return (dli, self.dls)
+
+    def write_dls(
+        self,
+        sdli: dict[str, tuple[float, int, str, str, str]],
+        dls: dict[str, tuple[float, int]],
+    ) -> None:
+        """
+        mp-broker pushing total dl-info + dl-state;
+        swap out the vfs vpath with the vfs node
+        """
+        dli: dict[str, tuple[float, int, "VFS", str, str]] = {}
+        for k, (a, b, c, d, e) in sdli.items():
+            vn = self.asrv.vfs.all_nodes[c]
+            dli[k] = (a, b, vn, d, e)
+
+        self.tdli = dli
+        self.tdls = dls

copyparty/mdns.py

@@ -25,6 +25,7 @@ from .stolen.dnslib import (
     DNSHeader,
     DNSQuestion,
     DNSRecord,
+    set_avahi_379,
 )
 from .util import CachedSet, Daemon, Netdev, list_ips, min_ex
 
@@ -72,6 +73,9 @@ class MDNS(MCast):
         self.ngen = ngen
         self.ttl = 300
 
+        if not self.args.zm_nwa_1:
+            set_avahi_379()
+
         zs = self.args.name + ".local."
         zs = zs.encode("ascii", "replace").decode("ascii", "replace")
         self.hn = "-".join(x for x in zs.split("?") if x) or (
@@ -336,6 +340,9 @@ class MDNS(MCast):
                         self.log("stopped", 2)
                         return
 
+                    if self.args.zm_no_pe:
+                        continue
+
                     t = "{} {} \033[33m|{}| {}\n{}".format(
                         self.srv[sck].name, addr, len(buf), repr(buf)[2:-1], min_ex()
                     )

copyparty/metrics.py

@@ -72,6 +72,9 @@ class Metrics(object):
         v = "{:.3f}".format(self.hsrv.t0)
         addug("cpp_boot_unixtime", "seconds", v, t)
 
+        t = "number of active downloads"
+        addg("cpp_active_dl", str(len(self.hsrv.tdls)), t)
+
         t = "number of open http(s) client connections"
         addg("cpp_http_conns", str(self.hsrv.ncli), t)
 

copyparty/mtag.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import argparse
 import json
 import os
+import re
 import shutil
 import subprocess as sp
 import sys
@@ -62,6 +63,9 @@ def have_ff(scmd: str) -> bool:
 HAVE_FFMPEG = not os.environ.get("PRTY_NO_FFMPEG") and have_ff("ffmpeg")
 HAVE_FFPROBE = not os.environ.get("PRTY_NO_FFPROBE") and have_ff("ffprobe")
 
+CBZ_PICS = set("png jpg jpeg gif bmp tga tif tiff webp avif".split())
+CBZ_01 = re.compile(r"(^|[^0-9v])0+[01]\b")
+
 
 class MParser(object):
     def __init__(self, cmdline: str) -> None:
@@ -126,6 +130,7 @@ def au_unpk(
     log: "NamedLogger", fmt_map: dict[str, str], abspath: str, vn: Optional[VFS] = None
 ) -> str:
     ret = ""
+    maxsz = 1024 * 1024 * 64
     try:
         ext = abspath.split(".")[-1].lower()
         au, pk = fmt_map[ext].split(".")
@@ -148,24 +153,48 @@ def au_unpk(
             zf = zipfile.ZipFile(abspath, "r")
             zil = zf.infolist()
             zil = [x for x in zil if x.filename.lower().split(".")[-1] == au]
+            if not zil:
+                raise Exception("no audio inside zip")
             fi = zf.open(zil[0])
 
+        elif pk == "cbz":
+            import zipfile
+
+            zf = zipfile.ZipFile(abspath, "r")
+            znil = [(x.filename.lower(), x) for x in zf.infolist()]
+            nf = len(znil)
+            znil = [x for x in znil if x[0].split(".")[-1] in CBZ_PICS]
+            znil = [x for x in znil if "cover" in x[0]] or znil
+            znil = [x for x in znil if CBZ_01.search(x[0])] or znil
+            t = "cbz: %d files, %d hits" % (nf, len(znil))
+            if znil:
+                t += ", using " + znil[0][1].filename
+            log(t)
+            if not znil:
+                raise Exception("no images inside cbz")
+            fi = zf.open(znil[0][1])
+
         else:
             raise Exception("unknown compression %s" % (pk,))
 
+        fsz = 0
         with os.fdopen(fd, "wb") as fo:
             while True:
                 buf = fi.read(32768)
                 if not buf:
                     break
 
+                fsz += len(buf)
+                if fsz > maxsz:
+                    raise Exception("zipbomb defused")
+
                 fo.write(buf)
 
         return ret
 
     except Exception as ex:
         if ret:
-            t = "failed to decompress audio file [%s]: %r"
+            t = "failed to decompress audio file %r: %r"
             log(t % (abspath, ex))
             wunlink(log, ret, vn.flags if vn else VF_CAREFUL)
 
@@ -553,7 +582,7 @@ class MTag(object):
                 raise Exception()
         except Exception as ex:
             if self.args.mtag_v:
-                self.log("mutagen-err [{}] @ [{}]".format(ex, abspath), "90")
+                self.log("mutagen-err [%s] @ %r" % (ex, abspath), "90")
 
             return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 
@@ -670,8 +699,8 @@ class MTag(object):
                             ret[tag] = zj[tag]
             except:
                 if self.args.mtag_v:
-                    t = "mtag error: tagname {}, parser {}, file {} => {}"
+                    t = "mtag error: tagname %r, parser %r, file %r => %r"
-                    self.log(t.format(tagname, parser.bin, abspath, min_ex()))
+                    self.log(t % (tagname, parser.bin, abspath, min_ex()), 6)
 
         if ap != abspath:
             wunlink(self.log, ap, VF_CAREFUL)

copyparty/pwhash.py

@@ -24,17 +24,13 @@ class PWHash(object):
     def __init__(self, args: argparse.Namespace):
         self.args = args
 
-        try:
+        zsl = args.ah_alg.split(",")
-            alg, ac = args.ah_alg.split(",")
+        alg = zsl[0]
-        except:
-            alg = args.ah_alg
-            ac = {}
-
         if alg == "none":
             alg = ""
 
         self.alg = alg
-        self.ac = ac
+        self.ac = zsl[1:]
         if not alg:
             self.on = False
             self.hash = unicode
@@ -90,17 +86,23 @@ class PWHash(object):
         its = 2
         blksz = 8
         para = 4
+        ramcap = 0  # openssl 1.1 = 32 MiB
         try:
             cost = 2 << int(self.ac[0])
             its = int(self.ac[1])
             blksz = int(self.ac[2])
             para = int(self.ac[3])
+            ramcap = int(self.ac[4]) * 1024 * 1024
         except:
             pass
 
+        cfg = {"salt": self.salt, "n": cost, "r": blksz, "p": para, "dklen": 24}
+        if ramcap:
+            cfg["maxmem"] = ramcap
+
         ret = plain.encode("utf-8")
         for _ in range(its):
-            ret = hashlib.scrypt(ret, salt=self.salt, n=cost, r=blksz, p=para, dklen=24)
+            ret = hashlib.scrypt(ret, **cfg)
 
         return "+" + base64.urlsafe_b64encode(ret).decode("utf-8")
 

copyparty/smbd.py

@@ -263,7 +263,7 @@ class SMB(object):
                 time.time(),
                 "",
             ):
-                yeet("blocked by xbu server config: " + vpath)
+                yeet("blocked by xbu server config: %r" % (vpath,))
 
         ret = bos.open(ap, flags, *a, mode=chmod, **ka)
         if wr:

copyparty/stolen/dnslib/dns.py

@@ -8,7 +8,7 @@ from itertools import chain
 from .bimap import Bimap, BimapError
 from .bit import get_bits, set_bits
 from .buffer import BufferError
-from .label import DNSBuffer, DNSLabel
+from .label import DNSBuffer, DNSLabel, set_avahi_379
 from .ranges import IP4, IP6, H, I, check_bytes
 
 
@@ -426,7 +426,7 @@ class RR(object):
             if rdlength:
                 rdata = RDMAP.get(QTYPE.get(rtype), RD).parse(buffer, rdlength)
             else:
-                rdata = ""
+                rdata = RD(b"a")
             return cls(rname, rtype, rclass, ttl, rdata)
         except (BufferError, BimapError) as e:
             raise DNSError("Error unpacking RR [offset=%d]: %s" % (buffer.offset, e))

copyparty/stolen/dnslib/label.py

@@ -11,6 +11,23 @@ LDH = set(range(33, 127))
 ESCAPE = re.compile(r"\\([0-9][0-9][0-9])")
 
 
+avahi_379 = 0
+
+
+def set_avahi_379():
+    global avahi_379
+    avahi_379 = 1
+
+
+def log_avahi_379(args):
+    global avahi_379
+    if avahi_379 == 2:
+        return
+    avahi_379 = 2
+    t = "Invalid pointer in DNSLabel [offset=%d,pointer=%d,length=%d];\n\033[35m  NOTE: this is probably avahi-bug #379, packet corruption in Avahi's mDNS-reflection feature. Copyparty has a workaround and is OK, but other devices need either --zm4 or --zm6"
+    raise BufferError(t % args)
+
+
 class DNSLabelError(Exception):
     pass
 
@@ -96,8 +113,11 @@ class DNSBuffer(Buffer):
                     )
                 if pointer < self.offset:
                     self.offset = pointer
+                elif avahi_379:
+                    log_avahi_379((self.offset, pointer, len(self.data)))
+                    label.extend(b"a")
+                    break
                 else:
-
                     raise BufferError(
                         "Invalid pointer in DNSLabel [offset=%d,pointer=%d,length=%d]"
                         % (self.offset, pointer, len(self.data))

copyparty/sutil.py

@@ -110,7 +110,7 @@ def errdesc(
     report = ["copyparty failed to add the following files to the archive:", ""]
 
     for fn, err in errors:
-        report.extend([" file: {}".format(fn), "error: {}".format(err), ""])
+        report.extend([" file: %r" % (fn,), "error: %s" % (err,), ""])
 
     btxt = "\r\n".join(report).encode("utf-8", "replace")
     btxt = vol_san(list(vfs.all_vols.values()), btxt)

copyparty/svchub.py

@@ -50,6 +50,8 @@ from .util import (
     FFMPEG_URL,
     HAVE_PSUTIL,
     HAVE_SQLITE3,
+    HAVE_ZMQ,
+    URL_BUG,
     UTC,
     VERSIONS,
     Daemon,
@@ -60,6 +62,7 @@ from .util import (
     alltrace,
     ansi_re,
     build_netmap,
+    expat_ver,
     load_ipu,
     min_ex,
     mp,
@@ -112,7 +115,7 @@ class SvcHub(object):
         self.stopping = False
         self.stopped = False
         self.reload_req = False
-        self.reloading = 0
+        self.reload_mutex = threading.Lock()
         self.stop_cond = threading.Condition()
         self.nsigs = 3
         self.retcode = 0
@@ -211,6 +214,15 @@ class SvcHub(object):
             t = "WARNING: --s-rd-sz (%d) is larger than --iobuf (%d); this may lead to reduced performance"
             self.log("root", t % (args.s_rd_sz, args.iobuf), 3)
 
+        zs = ""
+        if args.th_ram_max < 0.22:
+            zs = "generate thumbnails"
+        elif args.th_ram_max < 1:
+            zs = "generate audio waveforms or spectrograms"
+        if zs:
+            t = "WARNING: --th-ram-max is very small (%.2f GiB); will not be able to %s"
+            self.log("root", t % (args.th_ram_max, zs), 3)
+
         if args.chpw and args.idp_h_usr:
             t = "ERROR: user-changeable passwords is incompatible with IdP/identity-providers; you must disable either --chpw or --idp-h-usr"
             self.log("root", t, 1)
@@ -630,6 +642,7 @@ class SvcHub(object):
             (HAVE_FFPROBE, "ffprobe", t_ff + ", read audio/media tags"),
             (HAVE_MUTAGEN, "mutagen", "read audio tags (ffprobe is better but slower)"),
             (HAVE_ARGON2, "argon2", "secure password hashing (advanced users only)"),
+            (HAVE_ZMQ, "pyzmq", "send zeromq messages from event-hooks"),
             (HAVE_HEIF, "pillow-heif", "read .heif images with pillow (rarely useful)"),
             (HAVE_AVIF, "pillow-avif", "read .avif images with pillow (rarely useful)"),
         ]
@@ -686,6 +699,15 @@ class SvcHub(object):
             if self.args.bauth_last:
                 self.log("root", "WARNING: ignoring --bauth-last due to --no-bauth", 3)
 
+        if not self.args.no_dav:
+            from .dxml import DXML_OK
+
+            if not DXML_OK:
+                if not self.args.no_dav:
+                    self.args.no_dav = True
+                    t = "WARNING:\nDisabling WebDAV support because dxml selftest failed. Please report this bug;\n%s\n...and include the following information in the bug-report:\n%s | expat %s\n"
+                    self.log("root", t % (URL_BUG, VERSIONS, expat_ver()), 1)
+
     def _process_config(self) -> bool:
         al = self.args
 
@@ -784,7 +806,7 @@ class SvcHub(object):
         al.exp_md = odfusion(exp, al.exp_md.replace(" ", ","))
         al.exp_lg = odfusion(exp, al.exp_lg.replace(" ", ","))
 
-        for k in ["no_hash", "no_idx", "og_ua"]:
+        for k in ["no_hash", "no_idx", "og_ua", "srch_excl"]:
             ptn = getattr(self.args, k)
             if ptn:
                 setattr(self.args, k, re.compile(ptn))
@@ -1004,41 +1026,18 @@ class SvcHub(object):
             except:
                 self.log("root", "ssdp startup failed;\n" + min_ex(), 3)
 
-    def reload(self) -> str:
+    def reload(self, rescan_all_vols: bool, up2k: bool) -> str:
-        with self.up2k.mutex:
+        t = "config has been reloaded"
-            if self.reloading:
+        with self.reload_mutex:
-                return "cannot reload; already in progress"
-            self.reloading = 1
-
-        Daemon(self._reload, "reloading")
-        return "reload initiated"
-
-    def _reload(self, rescan_all_vols: bool = True, up2k: bool = True) -> None:
-        with self.up2k.mutex:
-            if self.reloading != 1:
-                return
-            self.reloading = 2
             self.log("root", "reloading config")
             self.asrv.reload(9 if up2k else 4)
             if up2k:
                 self.up2k.reload(rescan_all_vols)
+                t += "; volumes are now reinitializing"
             else:
                 self.log("root", "reload done")
             self.broker.reload()
-            self.reloading = 0
+        return t
-
-    def _reload_blocking(self, rescan_all_vols: bool = True, up2k: bool = True) -> None:
-        while True:
-            with self.up2k.mutex:
-                if self.reloading < 2:
-                    self.reloading = 1
-                    break
-            time.sleep(0.05)
-
-        # try to handle multiple pending IdP reloads at once:
-        time.sleep(0.2)
-
-        self._reload(rescan_all_vols=rescan_all_vols, up2k=up2k)
 
     def _reload_sessions(self) -> None:
         with self.asrv.mutex:
@@ -1052,7 +1051,7 @@ class SvcHub(object):
 
             if self.reload_req:
                 self.reload_req = False
-                self.reload()
+                self.reload(True, True)
 
         self.shutdown()
 

copyparty/szip.py

@@ -100,7 +100,7 @@ def gen_hdr(
 
     # spec says to put zeros when !crc if bit3 (streaming)
     # however infozip does actual sz and it even works on winxp
-    # (same reasning for z64 extradata later)
+    # (same reasoning for z64 extradata later)
     vsz = 0xFFFFFFFF if z64 else sz
     ret += spack(b"<LL", vsz, vsz)
 

copyparty/tcpsrv.py

@@ -371,7 +371,7 @@ class TcpSrv(object):
             if self.args.q:
                 print(msg)
 
-            self.hub.broker.say("listen", srv)
+            self.hub.broker.say("httpsrv.listen", srv)
 
         self.srv = srvs
         self.bound = bound
@@ -379,7 +379,7 @@ class TcpSrv(object):
         self._distribute_netdevs()
 
     def _distribute_netdevs(self):
-        self.hub.broker.say("set_netdevs", self.netdevs)
+        self.hub.broker.say("httpsrv.set_netdevs", self.netdevs)
         self.hub.start_zeroconf()
         gencert(self.log, self.args, self.netdevs)
         self.hub.restart_ftpd()
@@ -402,17 +402,17 @@ class TcpSrv(object):
             if not netdevs:
                 continue
 
-            added = "nothing"
+            add = []
-            removed = "nothing"
+            rem = []
             for k, v in netdevs.items():
                 if k not in self.netdevs:
-                    added = "{} = {}".format(k, v)
+                    add.append("\n\033[32m  added %s = %s" % (k, v))
             for k, v in self.netdevs.items():
                 if k not in netdevs:
-                    removed = "{} = {}".format(k, v)
+                    rem.append("\n\033[33mremoved %s = %s" % (k, v))
 
-            t = "network change detected:\n  added {}\033[0;33m\nremoved {}"
+            t = "network change detected:%s%s"
-            self.log("tcpsrv", t.format(added, removed), 3)
+            self.log("tcpsrv", t % ("".join(add), "".join(rem)), 3)
             self.netdevs = netdevs
             self._distribute_netdevs()
 

copyparty/tftpd.py

@@ -269,6 +269,7 @@ class Tftpd(object):
             "*",
             not self.args.no_scandir,
             [[True, False]],
+            throw=True,
         )
         dnames = set([x[0] for x in vfs_ls if stat.S_ISDIR(x[1].st_mode)])
         dirs1 = [(v.st_mtime, v.st_size, k + "/") for k, v in vfs_ls if k in dnames]
@@ -356,7 +357,7 @@ class Tftpd(object):
                 time.time(),
                 "",
             ):
-                yeet("blocked by xbu server config: " + vpath)
+                yeet("blocked by xbu server config: %r" % (vpath,))
 
         if not self.args.tftp_nols and bos.path.isdir(ap):
             return self._ls(vpath, "", 0, True)

copyparty/th_cli.py

@@ -6,7 +6,7 @@ import os
 from .__init__ import TYPE_CHECKING
 from .authsrv import VFS
 from .bos import bos
-from .th_srv import HAVE_WEBP, thumb_path
+from .th_srv import EXTS_AC, HAVE_WEBP, thumb_path
 from .util import Cooldown
 
 if True:  # pylint: disable=using-constant-test
@@ -57,13 +57,17 @@ class ThumbCli(object):
         if is_vid and "dvthumb" in dbv.flags:
             return None
 
-        want_opus = fmt in ("opus", "caf", "mp3")
+        want_opus = fmt in EXTS_AC
         is_au = ext in self.fmt_ffa
         is_vau = want_opus and ext in self.fmt_ffv
         if is_au or is_vau:
             if want_opus:
                 if self.args.no_acode:
                     return None
+                elif fmt == "caf" and self.args.no_caf:
+                    fmt = "mp3"
+                elif fmt == "owa" and self.args.no_owa:
+                    fmt = "mp3"
             else:
                 if "dathumb" in dbv.flags:
                     return None
@@ -109,13 +113,13 @@ class ThumbCli(object):
             fmt = sfmt
 
         elif fmt[:1] == "p" and not is_au and not is_vid:
-            t = "cannot thumbnail [%s]: png only allowed for waveforms"
+            t = "cannot thumbnail %r: png only allowed for waveforms"
-            self.log(t % (rem), 6)
+            self.log(t % (rem,), 6)
             return None
 
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)

copyparty/th_srv.py

@@ -20,7 +20,6 @@ from .util import (
     FFMPEG_URL,
     Cooldown,
     Daemon,
-    Pebkac,
     afsenc,
     fsenc,
     min_ex,
@@ -33,7 +32,7 @@ from .util import (
 )
 
 if True:  # pylint: disable=using-constant-test
-    from typing import Optional, Union
+    from typing import Any, Optional, Union
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -47,6 +46,9 @@ HAVE_HEIF = False
 HAVE_AVIF = False
 HAVE_WEBP = False
 
+EXTS_TH = set(["jpg", "webp", "png"])
+EXTS_AC = set(["opus", "owa", "caf", "mp3"])
+
 try:
     if os.environ.get("PRTY_NO_PIL"):
         raise Exception()
@@ -140,7 +142,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
     h = hashlib.sha512(afsenc(fn)).digest()
     fn = ub64enc(h).decode("ascii")[:24]
 
-    if fmt in ("opus", "caf", "mp3"):
+    if fmt in EXTS_AC:
         cat = "ac"
     else:
         fc = fmt[:1]
@@ -164,6 +166,7 @@ class ThumbSrv(object):
         self.ram: dict[str, float] = {}
         self.memcond = threading.Condition(self.mutex)
         self.stopping = False
+        self.rm_nullthumbs = True  # forget failed conversions on startup
         self.nthr = max(1, self.args.th_mt)
 
         self.q: Queue[Optional[tuple[str, str, str, VFS]]] = Queue(self.nthr * 4)
@@ -239,7 +242,7 @@ class ThumbSrv(object):
     def get(self, ptop: str, rem: str, mtime: float, fmt: str) -> Optional[str]:
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
@@ -249,7 +252,7 @@ class ThumbSrv(object):
         with self.mutex:
             try:
                 self.busy[tpath].append(cond)
-                self.log("joined waiting room for %s" % (tpath,))
+                self.log("joined waiting room for %r" % (tpath,))
             except:
                 thdir = os.path.dirname(tpath)
                 bos.makedirs(os.path.join(thdir, "w"))
@@ -266,11 +269,11 @@ class ThumbSrv(object):
             allvols = list(self.asrv.vfs.all_vols.values())
             vn = next((x for x in allvols if x.realpath == ptop), None)
             if not vn:
-                self.log("ptop [{}] not in {}".format(ptop, allvols), 3)
+                self.log("ptop %r not in %s" % (ptop, allvols), 3)
                 vn = self.asrv.vfs.all_aps[0][1]
 
             self.q.put((abspath, tpath, fmt, vn))
-            self.log("conv {} :{} \033[0m{}".format(tpath, fmt, abspath), c=6)
+            self.log("conv %r :%s \033[0m%r" % (tpath, fmt, abspath), 6)
 
         while not self.stopping:
             with self.mutex:
@@ -334,9 +337,10 @@ class ThumbSrv(object):
                 ap_unpk = abspath
 
             if not bos.path.exists(tpath):
-                want_mp3 = tpath.endswith(".mp3")
+                tex = tpath.rsplit(".", 1)[-1]
-                want_opus = tpath.endswith(".opus") or tpath.endswith(".caf")
+                want_mp3 = tex == "mp3"
-                want_png = tpath.endswith(".png")
+                want_opus = tex in ("opus", "owa", "caf")
+                want_png = tex == "png"
                 want_au = want_mp3 or want_opus
                 for lib in self.args.th_dec:
                     can_au = lib == "ff" and (
@@ -375,8 +379,8 @@ class ThumbSrv(object):
                     fun(ap_unpk, ttpath, fmt, vn)
                     break
                 except Exception as ex:
-                    msg = "{} could not create thumbnail of {}\n{}"
+                    msg = "%s could not create thumbnail of %r\n%s"
-                    msg = msg.format(fun.__name__, abspath, min_ex())
+                    msg = msg % (fun.__name__, abspath, min_ex())
                     c: Union[str, int] = 1 if "<Signals.SIG" in msg else "90"
                     self.log(msg, c)
                     if getattr(ex, "returncode", 0) != 321:
@@ -754,47 +758,102 @@ class ThumbSrv(object):
         if "ac" not in tags:
             raise Exception("not audio")
 
+        sq = "%dk" % (self.args.q_opus,)
+        bq = sq.encode("ascii")
+        if tags["ac"][1] == "opus":
+            enc = "-c:a copy"
+        else:
+            enc = "-c:a libopus -b:a " + sq
+
+        fun = self._conv_caf if fmt == "caf" else self._conv_owa
+
+        fun(abspath, tpath, tags, rawtags, enc, bq, vn)
+
+    def _conv_owa(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        if tpath.endswith(".owa"):
+            container = b"webm"
+            tagset = [b"-map_metadata", b"-1"]
+        else:
+            container = b"opus"
+            tagset = self.big_tags(rawtags)
+
+        self.log("conv2 %s [%s]" % (container, enc), 6)
+        benc = enc.encode("ascii").split(b" ")
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+        ] + tagset + [
+            b"-map", b"0:a:0",
+        ] + benc + [
+            b"-f", container,
+            fsenc(tpath)
+        ]
+        # fmt: on
+        self._run_ff(cmd, vn, oom=300)
+
+    def _conv_caf(
+        self,
+        abspath: str,
+        tpath: str,
+        tags: dict[str, tuple[int, Any]],
+        rawtags: dict[str, list[Any]],
+        enc: str,
+        bq: bytes,
+        vn: VFS,
+    ) -> None:
+        tmp_opus = tpath + ".opus"
+        try:
+            wunlink(self.log, tmp_opus, vn.flags)
+        except:
+            pass
+
         try:
             dur = tags[".dur"][1]
         except:
             dur = 0
 
-        src_opus = abspath.lower().endswith(".opus") or tags["ac"][1] == "opus"
+        self.log("conv2 caf-tmp [%s]" % (enc,), 6)
-        want_caf = tpath.endswith(".caf")
+        benc = enc.encode("ascii").split(b" ")
-        tmp_opus = tpath
-        if want_caf:
-            tmp_opus = tpath + ".opus"
-            try:
-                wunlink(self.log, tmp_opus, vn.flags)
-            except:
-                pass
 
-        caf_src = abspath if src_opus else tmp_opus
+        # fmt: off
-        bq = ("%dk" % (self.args.q_opus,)).encode("ascii")
+        cmd = [
-
+            b"ffmpeg",
-        if not want_caf or not src_opus:
+            b"-nostdin",
-            # fmt: off
+            b"-v", b"error",
-            cmd = [
+            b"-hide_banner",
-                b"ffmpeg",
+            b"-i", fsenc(abspath),
-                b"-nostdin",
+            b"-map_metadata", b"-1",
-                b"-v", b"error",
+            b"-map", b"0:a:0",
-                b"-hide_banner",
+        ] + benc + [
-                b"-i", fsenc(abspath),
+            b"-f", b"opus",
-            ] + self.big_tags(rawtags) + [
+            fsenc(tmp_opus)
-                b"-map", b"0:a:0",
+        ]
-                b"-c:a", b"libopus",
+        # fmt: on
-                b"-b:a", bq,
+        self._run_ff(cmd, vn, oom=300)
-                fsenc(tmp_opus)
-            ]
-            # fmt: on
-            self._run_ff(cmd, vn, oom=300)
 
         # iOS fails to play some "insufficiently complex" files
         # (average file shorter than 8 seconds), so of course we
         # fix that by mixing in some inaudible pink noise :^)
         # 6.3 sec seems like the cutoff so lets do 7, and
         # 7 sec of psyqui-musou.opus @ 3:50 is 174 KiB
-        if want_caf and (dur < 20 or bos.path.getsize(caf_src) < 256 * 1024):
+        sz = bos.path.getsize(tmp_opus)
+        if dur < 20 or sz < 256 * 1024:
+            zs = bq.decode("ascii")
+            self.log("conv2 caf-transcode; dur=%d sz=%d q=%s" % (dur, sz, zs), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
@@ -813,15 +872,16 @@ class ThumbSrv(object):
             # fmt: on
             self._run_ff(cmd, vn, oom=300)
 
-        elif want_caf:
+        else:
             # simple remux should be safe
+            self.log("conv2 caf-remux; dur=%d sz=%d" % (dur, sz), 6)
             # fmt: off
             cmd = [
                 b"ffmpeg",
                 b"-nostdin",
                 b"-v", b"error",
                 b"-hide_banner",
-                b"-i", fsenc(abspath if src_opus else tmp_opus),
+                b"-i", fsenc(tmp_opus),
                 b"-map_metadata", b"-1",
                 b"-map", b"0:a:0",
                 b"-c:a", b"copy",
@@ -831,11 +891,10 @@ class ThumbSrv(object):
             # fmt: on
             self._run_ff(cmd, vn, oom=300)
 
-        if tmp_opus != tpath:
+        try:
-            try:
+            wunlink(self.log, tmp_opus, vn.flags)
-                wunlink(self.log, tmp_opus, vn.flags)
+        except:
-            except:
+            pass
-                pass
 
     def big_tags(self, raw_tags: dict[str, list[str]]) -> list[bytes]:
         ret = []
@@ -862,7 +921,6 @@ class ThumbSrv(object):
     def cleaner(self) -> None:
         interval = self.args.th_clean
         while True:
-            time.sleep(interval)
             ndirs = 0
             for vol, histpath in self.asrv.vfs.histtab.items():
                 if histpath.startswith(vol):
@@ -876,6 +934,8 @@ class ThumbSrv(object):
                     self.log("\033[Jcln err in %s: %r" % (histpath, ex), 3)
 
             self.log("\033[Jcln ok; rm {} dirs".format(ndirs))
+            self.rm_nullthumbs = False
+            time.sleep(interval)
 
     def clean(self, histpath: str) -> int:
         ret = 0
@@ -890,13 +950,15 @@ class ThumbSrv(object):
 
     def _clean(self, cat: str, thumbpath: str) -> int:
         # self.log("cln {}".format(thumbpath))
-        exts = ["jpg", "webp", "png"] if cat == "th" else ["opus", "caf", "mp3"]
+        exts = EXTS_TH if cat == "th" else EXTS_AC
         maxage = getattr(self.args, cat + "_maxage")
         now = time.time()
         prev_b64 = None
         prev_fp = ""
         try:
-            t1 = statdir(self.log_func, not self.args.no_scandir, False, thumbpath)
+            t1 = statdir(
+                self.log_func, not self.args.no_scandir, False, thumbpath, False
+            )
             ents = sorted(list(t1))
         except:
             return 0
@@ -937,6 +999,10 @@ class ThumbSrv(object):
 
                 continue
 
+            if self.rm_nullthumbs and not inf.st_size:
+                bos.unlink(fp)
+                continue
+
             if b64 == prev_b64:
                 self.log("rm replaced [{}]".format(fp))
                 bos.unlink(prev_fp)

copyparty/u2idx.py

@@ -70,6 +70,9 @@ class U2idx(object):
         self.log_func("u2idx", msg, c)
 
     def shutdown(self) -> None:
+        if not HAVE_SQLITE3:
+            return
+
         for cur in self.cur.values():
             db = cur.connection
             try:
@@ -80,6 +83,12 @@ class U2idx(object):
             cur.close()
             db.close()
 
+        for cur in (self.mem_cur, self.sh_cur):
+            if cur:
+                db = cur.connection
+                cur.close()
+                db.close()
+
     def fsearch(
         self, uname: str, vols: list[VFS], body: dict[str, Any]
     ) -> list[dict[str, Any]]:
@@ -127,7 +136,7 @@ class U2idx(object):
         ptop = vn.realpath
         histpath = self.asrv.vfs.histtab.get(ptop)
         if not histpath:
-            self.log("no histpath for [{}]".format(ptop))
+            self.log("no histpath for %r" % (ptop,))
             return None
 
         db_path = os.path.join(histpath, "up2k.db")
@@ -142,7 +151,7 @@ class U2idx(object):
                 db = sqlite3.connect(uri, timeout=2, uri=True, check_same_thread=False)
                 cur = db.cursor()
                 cur.execute('pragma table_info("up")').fetchone()
-                self.log("ro: {}".format(db_path))
+                self.log("ro: %r" % (db_path,))
             except:
                 self.log("could not open read-only: {}\n{}".format(uri, min_ex()))
                 # may not fail until the pragma so unset it
@@ -152,7 +161,7 @@ class U2idx(object):
             # on windows, this steals the write-lock from up2k.deferred_init --
             # seen on win 10.0.17763.2686, py 3.10.4, sqlite 3.37.2
             cur = sqlite3.connect(db_path, timeout=2, check_same_thread=False).cursor()
-            self.log("opened {}".format(db_path))
+            self.log("opened %r" % (db_path,))
 
         self.cur[ptop] = cur
         return cur
@@ -315,7 +324,8 @@ class U2idx(object):
         sort: bool,
         lim: int,
     ) -> tuple[list[dict[str, Any]], list[str], bool]:
-        if self.args.srch_dbg:
+        dbg = self.args.srch_dbg
+        if dbg:
             t = "searching across all %s volumes in which the user has 'r' (full read access):\n  %s"
             zs = "\n  ".join(["/%s = %s" % (x.vpath, x.realpath) for x in vols])
             self.log(t % (len(vols), zs), 5)
@@ -358,14 +368,14 @@ class U2idx(object):
             if not cur:
                 continue
 
-            excl = []
+            dots = flags.get("dotsrch") and uname in vol.axs.udot
-            for vp2 in self.asrv.vfs.all_vols.keys():
+            zs = "srch_re_dots" if dots else "srch_re_nodot"
-                if vp2.startswith((vtop + "/").lstrip("/")) and vtop != vp2:
+            rex: re.Pattern = flags.get(zs)  # type: ignore
-                    excl.append(vp2[len(vtop) :].lstrip("/"))
 
-            if self.args.srch_dbg:
+            if dbg:
-                t = "searching in volume /%s (%s), excludelist %s"
+                t = "searching in volume /%s (%s), excluding %s"
-                self.log(t % (vtop, ptop, excl), 5)
+                self.log(t % (vtop, ptop, rex.pattern), 5)
+                rex_cfg: Optional[re.Pattern] = flags.get("srch_excl")
 
             self.active_cur = cur
 
@@ -378,7 +388,6 @@ class U2idx(object):
 
             sret = []
             fk = flags.get("fk")
-            dots = flags.get("dotsrch") and uname in vol.axs.udot
             fk_alg = 2 if "fka" in flags else 1
             c = cur.execute(uq, tuple(vuv))
             for hit in c:
@@ -387,20 +396,23 @@ class U2idx(object):
                 if rd.startswith("//") or fn.startswith("//"):
                     rd, fn = s3dec(rd, fn)
 
-                if rd in excl or any([x for x in excl if rd.startswith(x + "/")]):
+                vp = vjoin(vjoin(vtop, rd), fn)
-                    if self.args.srch_dbg:
+
-                        zs = vjoin(vjoin(vtop, rd), fn)
+                if vp in seen_rps:
-                        t = "database inconsistency in volume '/%s'; ignoring: %s"
-                        self.log(t % (vtop, zs), 1)
                     continue
 
-                rp = quotep("/".join([x for x in [vtop, rd, fn] if x]))
+                if rex.search(vp):
-                if not dots and "/." in ("/" + rp):
+                    if dbg:
-                    continue
+                        if rex_cfg and rex_cfg.search(vp):  # type: ignore
-
+                            self.log("filtered by srch_excl: %s" % (vp,), 6)
-                if rp in seen_rps:
+                        elif not dots and "/." in ("/" + vp):
+                            pass
+                        else:
+                            t = "database inconsistency in volume '/%s'; ignoring: %s"
+                            self.log(t % (vtop, vp), 1)
                     continue
 
+                rp = quotep(vp)
                 if not fk:
                     suf = ""
                 else:
@@ -422,7 +434,7 @@ class U2idx(object):
                 if lim < 0:
                     break
 
-                if self.args.srch_dbg:
+                if dbg:
                     t = "in volume '/%s': hit: %s"
                     self.log(t % (vtop, rp), 5)
 
@@ -452,7 +464,7 @@ class U2idx(object):
             ret.extend(sret)
             # print("[{}] {}".format(ptop, sret))
 
-            if self.args.srch_dbg:
+            if dbg:
                 t = "in volume '/%s': got %d hits, %d total so far"
                 self.log(t % (vtop, len(sret), len(ret)), 5)
 

copyparty/util.py

@@ -120,6 +120,13 @@ try:
 except:
     HAVE_SQLITE3 = False
 
+try:
+    import importlib.util
+
+    HAVE_ZMQ = bool(importlib.util.find_spec("zmq"))
+except:
+    HAVE_ZMQ = False
+
 try:
     if os.environ.get("PRTY_NO_PSUTIL"):
         raise Exception()
@@ -229,9 +236,14 @@ META_NOBOTS = '<meta name="robots" content="noindex, nofollow">\n'
 
 FFMPEG_URL = "https://www.gyan.dev/ffmpeg/builds/ffmpeg-git-full.7z"
 
+URL_PRJ = "https://github.com/9001/copyparty"
+
+URL_BUG = URL_PRJ + "/issues/new?labels=bug&template=bug_report.md"
+
 HTTPCODE = {
     200: "OK",
     201: "Created",
+    202: "Accepted",
     204: "No Content",
     206: "Partial Content",
     207: "Multi-Status",
@@ -319,6 +331,7 @@ DAV_ALLPROPS = set(DAV_ALLPROP_L)
 
 MIMES = {
     "opus": "audio/ogg; codecs=opus",
+    "owa": "audio/webm; codecs=opus",
 }
 
 
@@ -436,6 +449,27 @@ UNHUMANIZE_UNITS = {
 VF_CAREFUL = {"mv_re_t": 5, "rm_re_t": 5, "mv_re_r": 0.1, "rm_re_r": 0.1}
 
 
+def read_ram() -> tuple[float, float]:
+    a = b = 0
+    try:
+        with open("/proc/meminfo", "rb", 0x10000) as f:
+            zsl = f.read(0x10000).decode("ascii", "replace").split("\n")
+
+        p = re.compile("^MemTotal:.* kB")
+        zs = next((x for x in zsl if p.match(x)))
+        a = int((int(zs.split()[1]) / 0x100000) * 100) / 100
+
+        p = re.compile("^MemAvailable:.* kB")
+        zs = next((x for x in zsl if p.match(x)))
+        b = int((int(zs.split()[1]) / 0x100000) * 100) / 100
+    except:
+        pass
+    return a, b
+
+
+RAM_TOTAL, RAM_AVAIL = read_ram()
+
+
 pybin = sys.executable or ""
 if EXE:
     pybin = ""
@@ -470,6 +504,15 @@ def py_desc() -> str:
     )
 
 
+def expat_ver() -> str:
+    try:
+        import pyexpat
+
+        return ".".join([str(x) for x in pyexpat.version_info])
+    except:
+        return "?"
+
+
 def _sqlite_ver() -> str:
     assert sqlite3  # type: ignore  # !rm
     try:
@@ -1003,7 +1046,7 @@ class ProgressPrinter(threading.Thread):
             now = time.time()
             if msg and now - tp > 10:
                 tp = now
-                self.log("progress: %s" % (msg,), 6)
+                self.log("progress: %r" % (msg,), 6)
 
             if no_stdout:
                 continue
@@ -1030,6 +1073,7 @@ class MTHash(object):
         self.sz = 0
         self.csz = 0
         self.stop = False
+        self.readsz = 1024 * 1024 * (2 if (RAM_AVAIL or 2) < 1 else 12)
         self.omutex = threading.Lock()
         self.imutex = threading.Lock()
         self.work_q: Queue[int] = Queue()
@@ -1105,7 +1149,7 @@ class MTHash(object):
         while chunk_rem > 0:
             with self.imutex:
                 f.seek(ofs)
-                buf = f.read(min(chunk_rem, 1024 * 1024 * 12))
+                buf = f.read(min(chunk_rem, self.readsz))
 
             if not buf:
                 raise Exception("EOF at " + str(ofs))
@@ -1604,7 +1648,7 @@ class MultipartParser(object):
         (only the fallback non-js uploader relies on these filenames)
         """
         for ln in read_header(self.sr, 2, 2592000):
-            self.log(ln)
+            self.log(repr(ln))
 
             m = self.re_ctype.match(ln)
             if m:
@@ -1895,11 +1939,11 @@ def gen_filekey_dbg(
             if p2 != fspath:
                 raise Exception()
         except:
-            t = "maybe wrong abspath for filekey;\norig: {}\nreal: {}"
+            t = "maybe wrong abspath for filekey;\norig: %r\nreal: %r"
-            log(t.format(fspath, p2), 1)
+            log(t % (fspath, p2), 1)
 
-        t = "fk({}) salt({}) size({}) inode({}) fspath({}) at({})"
+        t = "fk(%s) salt(%s) size(%d) inode(%d) fspath(%r) at(%s)"
-        log(t.format(ret[:8], salt, fsize, inode, fspath, ctx), 5)
+        log(t % (ret[:8], salt, fsize, inode, fspath, ctx), 5)
 
     return ret
 
@@ -2255,7 +2299,7 @@ def log_reloc(
     rem: str,
 ) -> None:
     nap, nvp, nfn, (nvn, nrem) = pm
-    t = "reloc %s:\nold ap [%s]\nnew ap [%s\033[36m/%s\033[0m]\nold vp [%s]\nnew vp [%s\033[36m/%s\033[0m]\nold fn [%s]\nnew fn [%s]\nold vfs [%s]\nnew vfs [%s]\nold rem [%s]\nnew rem [%s]"
+    t = "reloc %s:\nold ap %r\nnew ap %r\033[36m/%r\033[0m\nold vp %r\nnew vp %r\033[36m/%r\033[0m\nold fn %r\nnew fn %r\nold vfs %r\nnew vfs %r\nold rem %r\nnew rem %r"
     log(t % (re, ap, nap, nfn, vp, nvp, nfn, fn, nfn, vn.vpath, nvn.vpath, rem, nrem))
 
 
@@ -2426,7 +2470,7 @@ def lsof(log: "NamedLogger", abspath: str) -> None:
     try:
         rc, so, se = runcmd([b"lsof", b"-R", fsenc(abspath)], timeout=45)
         zs = (so.strip() + "\n" + se.strip()).strip()
-        log("lsof {} = {}\n{}".format(abspath, rc, zs), 3)
+        log("lsof %r = %s\n%s" % (abspath, rc, zs), 3)
     except:
         log("lsof failed; " + min_ex(), 3)
 
@@ -2462,17 +2506,17 @@ def _fs_mvrm(
     for attempt in range(90210):
         try:
             if ino and os.stat(bsrc).st_ino != ino:
-                t = "src inode changed; aborting %s %s"
+                t = "src inode changed; aborting %s %r"
                 log(t % (act, src), 1)
                 return False
             if (dst and not atomic) and os.path.exists(bdst):
-                t = "something appeared at dst; aborting rename [%s] ==> [%s]"
+                t = "something appeared at dst; aborting rename %r ==> %r"
                 log(t % (src, dst), 1)
                 return False
             osfun(*args)
             if attempt:
                 now = time.time()
-                t = "%sd in %.2f sec, attempt %d: %s"
+                t = "%sd in %.2f sec, attempt %d: %r"
                 log(t % (act, now - t0, attempt + 1, src))
             return True
         except OSError as ex:
@@ -2484,7 +2528,7 @@ def _fs_mvrm(
             if not attempt:
                 if not PY2:
                     ino = os.stat(bsrc).st_ino
-                t = "%s failed (err.%d); retrying for %d sec: [%s]"
+                t = "%s failed (err.%d); retrying for %d sec: %r"
                 log(t % (act, ex.errno, maxtime + 0.99, src))
 
         time.sleep(chill)
@@ -2687,12 +2731,29 @@ def build_netmap(csv: str, defer_mutex: bool = False):
     if csv in ("any", "all", "no", ",", ""):
         return None
 
-    if csv in ("lan", "local", "private", "prvt"):
-        csv = "10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fd00::/8"  # lan
-        csv += ", 169.254.0.0/16, fe80::/10"  # link-local
-        csv += ", 127.0.0.0/8, ::1/128"  # loopback
-
     srcs = [x.strip() for x in csv.split(",") if x.strip()]
+
+    expanded_shorthands = False
+    for shorthand in ("lan", "local", "private", "prvt"):
+        if shorthand in srcs:
+            if not expanded_shorthands:
+                srcs += [
+                    # lan:
+                    "10.0.0.0/8",
+                    "172.16.0.0/12",
+                    "192.168.0.0/16",
+                    "fd00::/8",
+                    # link-local:
+                    "169.254.0.0/16",
+                    "fe80::/10",
+                    # loopback:
+                    "127.0.0.0/8",
+                    "::1/128",
+                ]
+                expanded_shorthands = True
+
+            srcs.remove(shorthand)
+
     if not HAVE_IPV6:
         srcs = [x for x in srcs if ":" not in x]
 
@@ -2757,6 +2818,26 @@ def yieldfile(fn: str, bufsz: int) -> Generator[bytes, None, None]:
             yield buf
 
 
+def justcopy(
+    fin: Generator[bytes, None, None],
+    fout: Union[typing.BinaryIO, typing.IO[Any]],
+    hashobj: Optional["hashlib._Hash"],
+    max_sz: int,
+    slp: float,
+) -> tuple[int, str, str]:
+    tlen = 0
+    for buf in fin:
+        tlen += len(buf)
+        if max_sz and tlen > max_sz:
+            continue
+
+        fout.write(buf)
+        if slp:
+            time.sleep(slp)
+
+    return tlen, "checksum-disabled", "checksum-disabled"
+
+
 def hashcopy(
     fin: Generator[bytes, None, None],
     fout: Union[typing.BinaryIO, typing.IO[Any]],
@@ -2791,7 +2872,10 @@ def sendfile_py(
     bufsz: int,
     slp: float,
     use_poll: bool,
+    dls: dict[str, tuple[float, int]],
+    dl_id: str,
 ) -> int:
+    sent = 0
     remains = upper - lower
     f.seek(lower)
     while remains > 0:
@@ -2808,6 +2892,10 @@ def sendfile_py(
         except:
             return remains
 
+        if dl_id:
+            sent += len(buf)
+            dls[dl_id] = (time.time(), sent)
+
     return 0
 
 
@@ -2820,6 +2908,8 @@ def sendfile_kern(
     bufsz: int,
     slp: float,
     use_poll: bool,
+    dls: dict[str, tuple[float, int]],
+    dl_id: str,
 ) -> int:
     out_fd = s.fileno()
     in_fd = f.fileno()
@@ -2832,7 +2922,7 @@ def sendfile_kern(
     while ofs < upper:
         stuck = stuck or time.time()
         try:
-            req = min(2 ** 30, upper - ofs)
+            req = min(0x2000000, upper - ofs)  # 32 MiB
             if use_poll:
                 poll.poll(10000)
             else:
@@ -2856,13 +2946,16 @@ def sendfile_kern(
             return upper - ofs
 
         ofs += n
+        if dl_id:
+            dls[dl_id] = (time.time(), ofs - lower)
+
         # print("sendfile: ok, sent {} now, {} total, {} remains".format(n, ofs - lower, upper - ofs))
 
     return 0
 
 
 def statdir(
-    logger: Optional["RootLogger"], scandir: bool, lstat: bool, top: str
+    logger: Optional["RootLogger"], scandir: bool, lstat: bool, top: str, throw: bool
 ) -> Generator[tuple[str, os.stat_result], None, None]:
     if lstat and ANYWIN:
         lstat = False
@@ -2898,6 +2991,12 @@ def statdir(
                     logger(src, "[s] {} @ {}".format(repr(ex), fsdec(abspath)), 6)
 
     except Exception as ex:
+        if throw:
+            zi = getattr(ex, "errno", 0)
+            if zi == errno.ENOENT:
+                raise Pebkac(404, str(ex))
+            raise
+
         t = "{} @ {}".format(repr(ex), top)
         if logger:
             logger(src, t, 1)
@@ -2906,7 +3005,7 @@ def statdir(
 
 
 def dir_is_empty(logger: "RootLogger", scandir: bool, top: str):
-    for _ in statdir(logger, scandir, False, top):
+    for _ in statdir(logger, scandir, False, top, False):
         return False
     return True
 
@@ -2919,7 +3018,7 @@ def rmdirs(
         top = os.path.dirname(top)
         depth -= 1
 
-    stats = statdir(logger, scandir, lstat, top)
+    stats = statdir(logger, scandir, lstat, top, False)
     dirs = [x[0] for x in stats if stat.S_ISDIR(x[1].st_mode)]
     dirs = [os.path.join(top, x) for x in dirs]
     ok = []
@@ -3309,6 +3408,7 @@ def _parsehook(
 
 def runihook(
     log: Optional["NamedLogger"],
+    verbose: bool,
     cmd: str,
     vol: "VFS",
     ups: list[tuple[str, int, int, str, str, str, int]],
@@ -3338,6 +3438,17 @@ def runihook(
     else:
         sp_ka["sin"] = b"\n".join(fsenc(x) for x in aps)
 
+    if acmd[0].startswith("zmq:"):
+        try:
+            msg = sp_ka["sin"].decode("utf-8", "replace")
+            _zmq_hook(log, verbose, "xiu", acmd[0][4:].lower(), msg, wait, sp_ka)
+            if verbose and log:
+                log("hook(xiu) %r OK" % (cmd,), 6)
+        except Exception as ex:
+            if log:
+                log("zeromq failed: %r" % (ex,))
+        return True
+
     t0 = time.time()
     if fork:
         Daemon(runcmd, cmd, bcmd, ka=sp_ka)
@@ -3347,15 +3458,126 @@ def runihook(
             retchk(rc, bcmd, err, log, 5)
             return False
 
-    wait -= time.time() - t0
+    if wait:
-    if wait > 0:
+        wait -= time.time() - t0
-        time.sleep(wait)
+        if wait > 0:
+            time.sleep(wait)
 
     return True
 
 
+ZMQ = {}
+ZMQ_DESC = {
+    "pub": "fire-and-forget to all/any connected SUB-clients",
+    "push": "fire-and-forget to one of the connected PULL-clients",
+    "req": "send messages to a REP-server and blocking-wait for ack",
+}
+
+
+def _zmq_hook(
+    log: Optional["NamedLogger"],
+    verbose: bool,
+    src: str,
+    cmd: str,
+    msg: str,
+    wait: float,
+    sp_ka: dict[str, Any],
+) -> tuple[int, str]:
+    import zmq
+
+    try:
+        mtx = ZMQ["mtx"]
+    except:
+        ZMQ["mtx"] = threading.Lock()
+        time.sleep(0.1)
+        mtx = ZMQ["mtx"]
+
+    ret = ""
+    nret = 0
+    t0 = time.time()
+    if verbose and log:
+        log("hook(%s) %r entering zmq-main-lock" % (src, cmd), 6)
+
+    with mtx:
+        try:
+            mode, sck, mtx = ZMQ[cmd]
+        except:
+            mode, uri = cmd.split(":", 1)
+            try:
+                desc = ZMQ_DESC[mode]
+                if log:
+                    t = "libzmq(%s) pyzmq(%s) init(%s); %s"
+                    log(t % (zmq.zmq_version(), zmq.__version__, cmd, desc))
+            except:
+                raise Exception("the only supported ZMQ modes are REQ PUB PUSH")
+
+            try:
+                ctx = ZMQ["ctx"]
+            except:
+                ctx = ZMQ["ctx"] = zmq.Context()
+
+            timeout = sp_ka["timeout"]
+
+            if mode == "pub":
+                sck = ctx.socket(zmq.PUB)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.bind(uri)
+                time.sleep(1)  # give clients time to connect; avoids losing first msg
+            elif mode == "push":
+                sck = ctx.socket(zmq.PUSH)
+                if timeout:
+                    sck.SNDTIMEO = int(timeout * 1000)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.bind(uri)
+            elif mode == "req":
+                sck = ctx.socket(zmq.REQ)
+                if timeout:
+                    sck.RCVTIMEO = int(timeout * 1000)
+                sck.setsockopt(zmq.LINGER, 0)
+                sck.connect(uri)
+            else:
+                raise Exception()
+
+            mtx = threading.Lock()
+            ZMQ[cmd] = (mode, sck, mtx)
+
+    if verbose and log:
+        log("hook(%s) %r entering socket-lock" % (src, cmd), 6)
+
+    with mtx:
+        if verbose and log:
+            log("hook(%s) %r sending |%d|" % (src, cmd, len(msg)), 6)
+
+        sck.send_string(msg)  # PUSH can safely timeout here
+
+        if mode == "req":
+            if verbose and log:
+                log("hook(%s) %r awaiting ack from req" % (src, cmd), 6)
+            try:
+                ret = sck.recv().decode("utf-8", "replace")
+                if ret.startswith("return "):
+                    m = re.search("^return ([0-9]+)", ret[:12])
+                    if m:
+                        nret = int(m.group(1))
+            except:
+                sck.close()
+                del ZMQ[cmd]  # bad state; must reset
+                raise Exception("ack timeout; zmq socket killed")
+
+    if ret and log:
+        log("hook(%s) %r ACK: %r" % (src, cmd, ret), 6)
+
+    if wait:
+        wait -= time.time() - t0
+        if wait > 0:
+            time.sleep(wait)
+
+    return nret, ret
+
+
 def _runhook(
     log: Optional["NamedLogger"],
+    verbose: bool,
     src: str,
     cmd: str,
     ap: str,
@@ -3396,6 +3618,12 @@ def _runhook(
     else:
         arg = txt or ap
 
+    if acmd[0].startswith("zmq:"):
+        zi, zs = _zmq_hook(log, verbose, src, acmd[0][4:].lower(), arg, wait, sp_ka)
+        if zi:
+            raise Exception("zmq says %d" % (zi,))
+        return {"rc": 0, "stdout": zs}
+
     acmd += [arg]
     if acmd[0].endswith(".py"):
         acmd = [pybin] + acmd
@@ -3424,9 +3652,10 @@ def _runhook(
             except:
                 ret = {"rc": rc, "stdout": v}
 
-    wait -= time.time() - t0
+    if wait:
-    if wait > 0:
+        wait -= time.time() - t0
-        time.sleep(wait)
+        if wait > 0:
+            time.sleep(wait)
 
     return ret
 
@@ -3449,17 +3678,17 @@ def runhook(
     txt: str,
 ) -> dict[str, Any]:
     assert broker or up2k  # !rm
-    asrv = (broker or up2k).asrv
     args = (broker or up2k).args
+    verbose = args.hook_v
     vp = vp.replace("\\", "/")
     ret = {"rc": 0}
     for cmd in cmds:
         try:
             hr = _runhook(
-                log, src, cmd, ap, vp, host, uname, perms, mt, sz, ip, at, txt
+                log, verbose, src, cmd, ap, vp, host, uname, perms, mt, sz, ip, at, txt
             )
-            if log and args.hook_v:
+            if verbose and log:
-                log("hook(%s) [%s] => \033[32m%s" % (src, cmd, hr), 6)
+                log("hook(%s) %r => \033[32m%s" % (src, cmd, hr), 6)
             if not hr:
                 return {}
             for k, v in hr.items():
@@ -3474,6 +3703,8 @@ def runhook(
                 elif k in ret:
                     if k == "rc" and v:
                         ret[k] = v
+                    elif k == "stdout" and v and not ret[k]:
+                        ret[k] = v
                 else:
                     ret[k] = v
         except Exception as ex:

copyparty/web/baguettebox.js

@@ -32,7 +32,7 @@ window.baguetteBox = (function () {
         scrollCSS = ['', ''],
         scrollTimer = 0,
         re_i = /^[^?]+\.(a?png|avif|bmp|gif|heif|jpe?g|jfif|svg|webp)(\?|$)/i,
-        re_v = /^[^?]+\.(webm|mkv|mp4|m4v)(\?|$)/i,
+        re_v = /^[^?]+\.(webm|mkv|mp4|m4v|mov)(\?|$)/i,
         anims = ['slideIn', 'fadeIn', 'none'],
         data = {},  // all galleries
         imagesElements = [],

copyparty/web/browser.css

@@ -188,7 +188,6 @@ html.y {
 	--srv-1: #555;
 	--srv-2: #c83;
 	--srv-3: #c0a;
-	--srv-3b: rgba(255,68,204,0.6);
 
 	--tree-bg: #fff;
 
@@ -286,6 +285,7 @@ html.bz {
 	--f-h-b1: #34384e;
 	--mp-sh: #11121d;
 	/*--mp-b-bg: #2c3044;*/
+	--f-play-bg: var(--btn-1-bg);
 }
 html.by {
 	--bg: #f2f2f2;
@@ -389,8 +389,6 @@ html.cy {
 }
 html.dz {
 	--fg: #4d4;
-	--fg-max: #fff;
-	--fg2-max: #fff;
 	--fg-weak: #2a2;
 
 	--bg-u6: #020;
@@ -400,11 +398,9 @@ html.dz {
 	--bg-u2: #020;
 	--bg-u1: #020;
 	--bg: #010;
-	--bgg: var(--bg);
 	--bg-d1: #000;
 	--bg-d2: #020;
 	--bg-d3: #000;
-	--bg-max: #000;
 
 	--tab-alt: #6f6;
 	--row-alt: #030;
@@ -417,45 +413,21 @@ html.dz {
 	--a-dark: #afa;
 	--a-gray: #2a2;
 
-	--btn-fg: var(--a);
 	--btn-bg: rgba(64,128,64,0.15);
-	--btn-h-fg: var(--a-hil);
 	--btn-h-bg: #050;
 	--btn-1-fg: #000;
 	--btn-1-bg: #4f4;
-	--btn-1h-fg: var(--btn-1-fg);
 	--btn-1h-bg: #3f3;
 	--btn-bs: 0 0 0 .1em #080 inset;
 	--btn-1-bs: a;
 
-	--chk-fg: var(--tab-alt);
-	--txt-sh: var(--bg-d2);
-	--txt-bg: var(--btn-bg);
-
-	--op-aa-fg: var(--a);
-	--op-aa-bg: var(--bg-d2);
-	--op-a-sh: rgba(0,0,0,0.5);
-
 	--u2-btn-b1: var(--fg-weak);
 	--u2-sbtn-b1: var(--fg-weak);
-	--u2-txt-bg: var(--bg-u5);
-	--u2-tab-bg: linear-gradient(to bottom, var(--bg), var(--bg-u1));
 	--u2-tab-b1: var(--fg-weak);
 	--u2-tab-1-fg: #fff;
 	--u2-tab-1-bg: linear-gradient(to bottom, #151, var(--bg) 80%);
-	--u2-tab-1-b1: #7c5;
-	--u2-tab-1-b2: #583;
-	--u2-tab-1-sh: #280;
-	--u2-b-fg: #fff;
 	--u2-b1-bg: #3a3;
 	--u2-b2-bg: #3a3;
-	--u2-inf-bg: #07a;
-	--u2-inf-b1: #0be;
-	--u2-ok-bg: #380;
-	--u2-ok-b1: #8e4;
-	--u2-err-bg: #900;
-	--u2-err-b1: #d06;
-	--ud-b1: #888;
 
 	--sort-1: #fff;
 	--sort-2: #3f3;
@@ -467,47 +439,12 @@ html.dz {
 
 	--tree-bg: #010;
 
-	--g-play-bg: #750;
-	--g-play-b1: #c90;
-	--g-play-b2: #da4;
-	--g-play-sh: #b83;
-
-	--g-sel-fg: #fff;
-	--g-sel-bg: #925;
 	--g-sel-b1: #c37;
 	--g-sel-sh: #b36;
-	--g-fsel-bg: #d39;
 	--g-fsel-b1: #d48;
-	--g-fsel-ts: #804;
-	--g-fg: var(--a-hil);
-	--g-bg: var(--bg-u2);
-	--g-b1: var(--bg-u4);
-	--g-b2: var(--bg-u5);
-	--g-g1: var(--bg-u2);
-	--g-g2: var(--bg-u5);
-	--g-f-bg: var(--bg-u4);
-	--g-f-b1: var(--bg-u5);
-	--g-f-fg: var(--a-hil);
-	--g-sh: rgba(0,0,0,0.3);
 
-	--f-sh1: 0.33;
-	--f-sh2: 0.02;
-	--f-sh3: 0.2;
 	--f-h-b1: #3b3;
 
-	--f-play-bg: #fc5;
-	--f-play-fg: #000;
-	--f-sel-sh: #fc0;
-	--f-gray: #999;
-
-	--fm-off: #f6c;
-	--mp-sh: var(--bg-d3);
-
-	--err-fg: #fff;
-	--err-bg: #a20;
-	--err-b1: #f00;
-	--err-ts: #500;
-
 	text-shadow: none;
 	font-family: 'scp', monospace, monospace;
 	font-family: var(--font-mono), 'scp', monospace, monospace;
@@ -1710,6 +1647,18 @@ html.dz .btn {
 	background: var(--btn-1-bg);
 	text-shadow: none;
 }
+#tree ul a.ld::before {
+	font-weight: bold;
+	font-family: sans-serif;
+	display: inline-block;
+	text-align: center;
+	width: 1em;
+	margin: 0 .3em 0 -1.3em;
+	color: var(--fg-max);
+	opacity: 0;
+	content: '◠';
+	animation: .5s linear infinite forwards spin, ease .25s 1 forwards fadein;
+}
 #tree ul a.par {
 	color: var(--fg-max);
 }
@@ -1931,11 +1880,10 @@ html.y #tree.nowrap .ntree a+a:hover {
 #rn_f.m td+td {
 	width: 50%;
 }
-#rn_f .err td {
+#rn_f .err td,
-	background: var(--err-bg);
+#rn_f .err input[readonly],
-	color: var(--fg-max);
+#rui .ng input[readonly] {
-}
+	color: var(--err-fg);
-#rn_f .err input[readonly] {
 	background: var(--err-bg);
 }
 #rui input[readonly] {
@@ -2837,6 +2785,7 @@ html.b #u2conf a.b:hover {
 	padding-left: .2em;
 }
 .fsearch_explain {
+	color: var(--a-dark);
 	padding-left: .7em;
 	font-size: 1.1em;
 	line-height: 0;

copyparty/web/browser.html

@@ -131,17 +131,16 @@
 	<div id="widget"></div>
 
 	<script>
-		var SR = {{ r|tojson }},
+		var SR = "{{ r }}",
+			CGV1 = {{ cgv1 }},
 			CGV = {{ cgv|tojson }},
 			TS = "{{ ts }}",
 			dtheme = "{{ dtheme }}",
 			srvinf = "{{ srv_info }}",
-			s_name = "{{ s_name }}",
 			lang = "{{ lang }}",
 			dfavico = "{{ favico }}",
-			have_tags_idx = {{ have_tags_idx|tojson }},
+			have_tags_idx = {{ have_tags_idx }},
 			sb_lg = "{{ sb_lg }}",
-			txt_ext = "{{ txt_ext }}",
 			logues = {{ logues|tojson if sb_lg else "[]" }},
 			ls0 = {{ ls0|tojson }};
 

copyparty/web/md.html

@@ -128,9 +128,9 @@ write markdown (most html is 🙆 too)
 
 	<script>
 
-var SR = {{ r|tojson }},
+var SR = "{{ r }}",
 	last_modified = {{ lastmod }},
-	have_emp = {{ have_emp|tojson }},
+	have_emp = {{ "true" if have_emp else "false" }},
 	dfavico = "{{ favico }}";
 
 var md_opt = {

copyparty/web/md.js

@@ -17,14 +17,13 @@ var chromedbg = function () { console.log(arguments); }
 var dbg = function () { };
 
 // replace dbg with the real deal here or in the console:
-// dbg = chromedbg
+// dbg = chromedbg;
-// dbg = console.log
+// dbg = console.log;
 
 
 // dodge browser issues
 (function () {
-    var ua = navigator.userAgent;
+    if (UA.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(UA)) {
-    if (ua.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(ua)) {
         // necessary on ff-68.7 at least
         var s = mknod('style');
         s.innerHTML = '@page { margin: .5in .6in .8in .6in; }';

copyparty/web/md2.js

@@ -450,7 +450,7 @@ function savechk_cb() {
 
 // firefox bug: initial selection offset isn't cleared properly through js
 var ff_clearsel = (function () {
-    if (navigator.userAgent.indexOf(') Gecko/') === -1)
+    if (UA.indexOf(') Gecko/') === -1)
         return function () { }
 
     return function () {

copyparty/web/mde.html

@@ -26,9 +26,9 @@
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR = "{{ r }}",
 	last_modified = {{ lastmod }},
-	have_emp = {{ have_emp|tojson }},
+	have_emp = {{ "true" if have_emp else "false" }},
 	dfavico = "{{ favico }}";
 
 var md_opt = {

copyparty/web/rups.css

@@ -0,0 +1,107 @@
+html {
+	color: #333;
+	background: #f7f7f7;
+	font-family: sans-serif;
+	font-family: var(--font-main), sans-serif;
+	touch-action: manipulation;
+}
+#wrap {
+	margin: 2em auto;
+	padding: 0 1em 3em 1em;
+	line-height: 2.3em;
+}
+a {
+	color: #047;
+	background: #fff;
+	text-decoration: none;
+	border-bottom: 1px solid #8ab;
+	border-radius: .2em;
+	padding: .2em .6em;
+	margin: 0 .3em;
+}
+#wrap td a {
+	margin: 0;
+	line-height: 1em;
+	display: inline-block;
+	white-space: initial;
+	font-family: var(--font-main), sans-serif;
+}
+#repl {
+	border: none;
+	background: none;
+	color: inherit;
+	padding: 0;
+	position: fixed;
+	bottom: .25em;
+	left: .2em;
+}
+#wrap table {
+	border-collapse: collapse;
+	position: relative;
+	margin-top: 2em;
+}
+#wrap th {
+	top: -1px;
+	position: sticky;
+	background: #f7f7f7;
+}
+#wrap td {
+	font-family: var(--font-mono), monospace, monospace;
+	white-space: pre; /*date*/
+	overflow: hidden; /*ipv6*/
+}
+#wrap th:first-child,
+#wrap td:first-child {
+	text-align: right;
+}
+#wrap td,
+#wrap th {
+	text-align: left;
+	padding: .3em .6em;
+	max-width: 30vw;
+}
+#wrap tr:hover td {
+	background: #ddd;
+	box-shadow: 0 -1px 0 rgba(128, 128, 128, 0.5) inset;
+}
+#wrap th:first-child,
+#wrap td:first-child {
+	border-radius: .5em 0 0 .5em;
+}
+#wrap th:last-child,
+#wrap td:last-child {
+	border-radius: 0 .5em .5em 0;
+}
+
+
+
+html.z {
+	background: #222;
+	color: #ccc;
+}
+html.bz {
+	background: #11121d;
+	color: #bbd;
+}
+html.z a {
+	color: #fff;
+	background: #057;
+	border-color: #37a;
+}
+html.z input[type=text] {
+	color: #ddd;
+	background: #223;
+	border: none;
+	border-bottom: 1px solid #fc5;
+	border-radius: .2em;
+	padding: .2em .3em;
+}
+html.z #wrap th {
+	background: #222;
+}
+html.bz #wrap th {
+	background: #223;
+}
+html.z #wrap tr:hover td {
+	background: #000;
+}

copyparty/web/rups.html

@@ -0,0 +1,50 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+	<meta charset="utf-8">
+	<title>{{ s_doctitle }}</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
+	<meta name="theme-color" content="#{{ tcolor }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/rups.css?_={{ ts }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
+{{ html_head }}
+</head>
+
+<body>
+	<div id="wrap">
+        <a href="#" id="re">refresh</a>
+        <a href="{{ r }}/?h">control-panel</a>
+        &nbsp; Filter: <input type="text" id="filter" size="20" placeholder="documents/passwords" />
+        &nbsp; <span id="hits"></span>
+        <table id="tab"><thead><tr>
+            <th>size</th>
+            <th>who</th>
+            <th>when</th>
+            <th>age</th>
+            <th>dir</th>
+            <th>file</th>
+        </tr></thead><tbody id="tb"></tbody></table>
+    </div>
+	<a href="#" id="repl">π</a>
+	<script>
+
+var SR="{{ r }}",
+	lang="{{ lang }}",
+	dfavico="{{ favico }}";
+
+var STG = window.localStorage;
+document.documentElement.className = (STG && STG.cpp_thm) || "{{ this.args.theme }}";
+
+</script>
+<script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>
+<script>var V={{ v }};</script>
+<script src="{{ r }}/.cpr/rups.js?_={{ ts }}"></script>
+{%- if js %}
+<script src="{{ js }}_={{ ts }}"></script>
+{%- endif %}
+</body>
+</html>
+

copyparty/web/rups.js

@@ -0,0 +1,66 @@
+function render() {
+    var ups = V.ups, now = V.now, html = [];
+    ebi('filter').value = V.filter;
+    ebi('hits').innerHTML = 'showing ' + ups.length + ' files';
+
+    for (var a = 0; a < ups.length; a++) {
+        var f = ups[a],
+            vsp = vsplit(f.vp.split('?')[0]),
+            dn = esc(uricom_dec(vsp[0])),
+            fn = esc(uricom_dec(vsp[1])),
+            at = f.at,
+            td = now - f.at,
+            ts = !at ? '(?)' : unix2iso(at),
+            sa = !at ? '(?)' : td > 60 ? shumantime(td) : (td + 's'),
+            sz = ('' + f.sz).replace(/\B(?=(\d{3})+(?!\d))/g, " ");
+
+        html.push('<tr><td>' + sz +
+            '</td><td>' + f.ip +
+            '</td><td>' + ts +
+            '</td><td>' + sa +
+            '</td><td><a href="' + vsp[0] + '">' + dn +
+            '</a></td><td><a href="' + f.vp + '">' + fn +
+            '</a></td></tr>');
+    }
+    if (!ups.length) {
+        var t = V.filter ? ' matching the filter' : '';
+        html = ['<tr><td colspan="6">there are no uploads' + t + '</td></tr>'];
+    }
+    ebi('tb').innerHTML = html.join('');
+}
+render();
+
+var ti;
+function ask(e) {
+    ev(e);
+    clearTimeout(ti);
+    ebi('hits').innerHTML = 'Loading...';
+
+    var xhr = new XHR(),
+        filter = unsmart(ebi('filter').value);
+
+    hist_replace(get_evpath().split('?')[0] + '?ru&filter=' + uricom_enc(filter));
+
+    xhr.onload = xhr.onerror = function () {
+        try {
+            V = JSON.parse(this.responseText)
+        }
+        catch (ex) {
+            ebi('tb').innerHTML = '<tr><td colspan="6">failed to decode server response as json: <pre>' + esc(this.responseText) + '</pre></td></tr>';
+            return;
+        }
+        render();
+    };
+    xhr.open('GET', SR + '/?ru&j&filter=' + uricom_enc(filter), true);
+    xhr.send();
+}
+ebi('re').onclick = ask;
+ebi('filter').oninput = function () {
+    clearTimeout(ti);
+    ti = setTimeout(ask, 500);
+    ebi('hits').innerHTML = '...';
+};
+ebi('filter').onkeydown = function (e) {
+    if (('' + e.key).endsWith('Enter'))
+        ask();
+};

copyparty/web/shares.css

@@ -44,9 +44,10 @@ a {
 	bottom: .25em;
 	left: .2em;
 }
-table {
+#wrap table {
 	border-collapse: collapse;
-    position: relative;
+	position: relative;
+	margin-top: 2em;
 }
 th {
 	top: -1px;
@@ -62,6 +63,14 @@ th {
 #wrap td+td+td+td+td+td+td+td {
 	font-family: var(--font-mono), monospace, monospace;
 }
+#wrap th:first-child,
+#wrap td:first-child {
+	border-radius: .5em 0 0 .5em;
+}
+#wrap th:last-child,
+#wrap td:last-child {
+	border-radius: 0 .5em .5em 0;
+}
 
 
 
@@ -81,3 +90,6 @@ html.bz {
 	color: #bbd;
 	background: #11121d;
 }
+html.bz th {
+	background: #223;
+}

copyparty/web/shares.html

@@ -6,6 +6,7 @@
 	<title>{{ s_doctitle }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/shares.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
@@ -14,8 +15,8 @@
 
 <body>
 	<div id="wrap">
-		<a id="a" href="{{ r }}/?shares" class="af">refresh</a>
+		<a href="{{ r }}/?shares">refresh</a>
-		<a id="a" href="{{ r }}/?h" class="af">control-panel</a>
+		<a href="{{ r }}/?h">control-panel</a>
 
         <span>axs = perms (read,write,move,delet)</span>
         <span>nf = numFiles (0=dir)</span>
@@ -58,9 +59,11 @@
         {% if not rows %}
         (you don't have any active shares btw)
         {% endif %}
+    </div>
+	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
     shr="{{ shr }}",
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";

copyparty/web/shares.js

@@ -45,7 +45,7 @@ function qr(e) {
 
 function showqr(href) {
     var vhref = href.replace('?qr&', '?').replace('?qr', '');
-    modal.alert(esc(vhref) + '<img class="b64" src="' + href + '" />');
+    modal.alert(esc(vhref) + '<img class="b64" width="100" height="100" src="' + href + '" />');
 }
 
 (function() {
@@ -71,7 +71,7 @@ function showqr(href) {
         tr[a].cells[11].innerHTML =
             '<button value="1">1min</button> ' +
             '<button value="60">1h</button>';
-    
+
     var btns = QSA('td button'), aa = btns.length;
     for (var a = 0; a < aa; a++)
         btns[a].onclick = bump;

copyparty/web/splash.css

@@ -90,6 +90,13 @@ table {
 	text-align: left;
 	white-space: nowrap;
 }
+.vols td:empty,
+.vols th:empty {
+	padding: 0;
+}
+.vols img {
+	margin: -4px 0;
+}
 .num {
 	border-right: 1px solid #bbb;
 }
@@ -222,3 +229,6 @@ html.bz {
 	color: #bbd;
 	background: #11121d;
 }
+html.bz .vols img {
+	filter: sepia(0.8) hue-rotate(180deg);
+}

copyparty/web/splash.html

@@ -44,6 +44,18 @@
 			</table>
 		{%- endif %}
 
+		{%- if dls %}
+		<h1 id="ae">active downloads:</h1>
+			<table class="vols">
+				<thead><tr><th>%</th><th>sent</th><th>speed</th><th>eta</th><th>idle</th><th></th><th>dir</th><th>file</th></tr></thead>
+				<tbody>
+					{% for u in dls %}
+					<tr><td>{{ u[0] }}</td><td>{{ u[1] }}</td><td>{{ u[2] }}</td><td>{{ u[3] }}</td><td>{{ u[4] }}</td><td>{{ u[5] }}</td><td><a href="{{ u[6] }}">{{ u[7]|e }}</a></td><td>{{ u[8] }}</td></tr>
+					{% endfor %}
+				</tbody>
+			</table>
+		{%- endif %}
+
 		{%- if avol %}
 		<h1>admin panel:</h1>
 		<table><tr><td> <!-- hehehe -->
@@ -145,6 +157,7 @@
 			<blockquote id="ad">enabling no304 will disable all caching; try this if k304 wasn't enough. This will waste a huge amount of network traffic!</blockquote></li>
 			{% endif %}
 
+			<li><a id="af" href="{{ r }}/?ru">show recent uploads</a></li>
 			<li><a id="k" href="{{ r }}/?reset" class="r" onclick="localStorage.clear();return true">reset client settings</a></li>
 		</ul>
 
@@ -155,7 +168,7 @@
 	{%- endif %}
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";
 

copyparty/web/splash.js

@@ -37,6 +37,8 @@ var Ls = {
 		"ab1": "skru av no304",
 		"ac1": "skru på no304",
 		"ad1": "no304 stopper all bruk av cache. Hvis ikke k304 var nok, prøv denne. Vil mangedoble dataforbruk!",
+		"ae1": "utgående:",
+		"af1": "vis nylig opplastede filer",
 	},
 	"eng": {
 		"d2": "shows the state of all active threads",
@@ -86,6 +88,8 @@ var Ls = {
 		"ab1": "关闭 k304",
 		"ac1": "开启 k304",
 		"ad1": "启用 no304 将禁用所有缓存；如果 k304 不够，可以尝试此选项。这将消耗大量的网络流量！", //m
+		"ae1": "正在下载：", //m
+		"af1": "显示最近上传的文件", //m
 	}
 };
 

copyparty/web/svcs.html

@@ -9,7 +9,7 @@
 	<meta name="theme-color" content="#{{ tcolor }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/splash.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
-	<style>ul{padding-left:1.3em}li{margin:.4em 0}</style>
+	<style>ul{padding-left:1.3em}li{margin:.4em 0}.txa{float:right;margin:0 0 0 1em}</style>
 {{ html_head }}
 </head>
 
@@ -31,15 +31,22 @@
             <br />
             <span class="os win lin mac">placeholders:</span>
             <span class="os win">
-                {% if accs %}<code><b>{{ pw }}</b></code>=password, {% endif %}<code><b>W:</b></code>=mountpoint
+                {% if accs %}<code><b id="pw0">{{ pw }}</b></code>=password, {% endif %}<code><b>W:</b></code>=mountpoint
             </span>
             <span class="os lin mac">
-                {% if accs %}<code><b>{{ pw }}</b></code>=password, {% endif %}<code><b>mp</b></code>=mountpoint
+                {% if accs %}<code><b id="pw0">{{ pw }}</b></code>=password, {% endif %}<code><b>mp</b></code>=mountpoint
             </span>
+            <a href="#" id="setpw">use real password</a>
         </p>
 
 
 
+        {% if args.idp_h_usr %}
+        <p style="line-height:2em"><b>WARNING:</b> this server is using IdP-based authentication, so this stuff may not work as advertised. Depending on server config, these commands can probably only be used to access areas which don't require authentication, unless you auth using any non-IdP accounts defined in the copyparty config. Please see <a href="https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients">the IdP docs</a></p>
+        {% endif %}
+
+
+
         {% if not args.no_dav %}
         <h1>WebDAV</h1>
 
@@ -53,7 +60,6 @@
                 {% if s %}
                 <li>running <code>rclone mount</code> on LAN (or just dont have valid certificates)? add <code>--no-check-certificate</code></li>
                 {% endif %}
-                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
                 <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
             </ul>
 
@@ -137,7 +143,6 @@
                 {% if args.ftps %}
                 <li>running on LAN (or just dont have valid certificates)? add <code>no_check_certificate=true</code> to the config command</li>
                 {% endif %}
-                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
                 <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
             </ul>
             <p>if you want to use the native FTP client in windows instead (please dont), press <code>win+R</code> and run this command:</p>
@@ -231,11 +236,65 @@
 
 
 
+        <div class="os win">
+            <h1>ShareX</h1>
+
+            <p>to upload screenshots using ShareX <a href="https://github.com/ShareX/ShareX/releases/tag/v12.1.1">v12</a> or <a href="https://getsharex.com/">v15+</a>, save this as <code>copyparty.sxcu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.sxcu">
+                { "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>",{% endif %}
+                    "accept": "url"
+                },
+                "DestinationType": "ImageUploader, TextUploader, FileUploader",
+                "FileFormName": "f" }
+            </pre>
+        </div>
+
+
+
+        <div class="os mac">
+            <h1>ishare</h1>
+
+            <p>to upload screenshots using <a href="https://isharemac.app/">ishare</a>, save this as <code>copyparty.iscu</code> and run it:</p>
+
+            <pre class="dl" name="copyparty.iscu">
+                { "Name": "copyparty",
+                "RequestURL": "http{{ s }}://{{ ep }}/{{ rvp }}",
+                "Headers": {
+                    {% if accs %}"pw": "<b>{{ pw }}</b>",{% endif %}
+                    "accept": "json"
+                },
+                "ResponseURL": "{{ '{{fileurl}}' }}",
+                "FileFormName": "f" }
+            </pre>
+        </div>
+
+
+
+        <div class="os lin">
+            <h1>flameshot</h1>
+
+            <p>to upload screenshots using <a href="https://flameshot.org/">flameshot</a>, save this as <code>flameshot.sh</code> and run it:</p>
+
+            <pre class="dl" name="flameshot.sh">
+                #!/bin/bash
+                pw="<b>{{ pw }}</b>"
+                url="http{{ s }}://{{ ep }}/{{ rvp }}"
+                filename="$(date +%Y-%m%d-%H%M%S).png"
+                flameshot gui -s -r | curl -sT- "$url$filename?want=url&pw=$pw" | xsel -ib
+            </pre>
+        </div>
+
+
+
     </div>
 	<a href="#" id="repl">π</a>
 	<script>
 
-var SR = {{ r|tojson }},
+var SR="{{ r }}",
     lang="{{ lang }}",
 	dfavico="{{ favico }}";
 

copyparty/web/svcs.js

@@ -1,11 +1,3 @@
-function QSA(x) {
-    return document.querySelectorAll(x);
-}
-var LINUX = /Linux/.test(navigator.userAgent),
-    MACOS = /[^a-z]mac ?os/i.test(navigator.userAgent),
-    WINDOWS = /Windows/.test(navigator.userAgent);
-
-
 var oa = QSA('pre');
 for (var a = 0; a < oa.length; a++) {
     var html = oa[a].innerHTML,
@@ -15,6 +7,21 @@ for (var a = 0; a < oa.length; a++) {
     oa[a].innerHTML = html.replace(rd, '$1').replace(/[ \r\n]+$/, '').replace(/\r?\n/g, '<br />');
 }
 
+function add_dls() {
+    oa = QSA('pre.dl');
+    for (var a = 0; a < oa.length; a++) {
+        var an = 'ta' + a,
+            o = ebi(an) || mknod('a', an, 'download');
+
+        oa[a].setAttribute('id', 'tx' + a);
+        oa[a].parentNode.insertBefore(o, oa[a]);
+        o.setAttribute('download', oa[a].getAttribute('name'));
+        o.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(oa[a].innerText));
+        clmod(o, 'txa', 1);
+    }
+}
+add_dls();
+
 
 oa = QSA('.ossel a');
 for (var a = 0; a < oa.length; a++)
@@ -40,3 +47,21 @@ function setos(os) {
 }
 
 setos(WINDOWS ? 'win' : LINUX ? 'lin' : MACOS ? 'mac' : 'idk');
+
+
+ebi('setpw').onclick = function (e) {
+    ev(e);
+    modal.prompt('password:', '', function (v) {
+        if (!v)
+            return;
+
+        var pw0 = ebi('pw0').innerHTML,
+            oa = QSA('b');
+        
+        for (var a = 0; a < oa.length; a++)
+            if (oa[a].innerHTML == pw0)
+                oa[a].textContent = v;
+
+        add_dls();
+    });
+}

copyparty/web/ui.css

@@ -75,7 +75,7 @@ html {
 	top: 1px;
 	right: 1px;
 	left: 1px;
-	animation: toastt var(--tmtime) steps(var(--tmstep)) forwards;
+	animation: toastt var(--tmtime) 0.07s steps(var(--tmstep)) forwards;
 	transform-origin: right;
 }
 @keyframes toastt {
@@ -322,6 +322,8 @@ html.y #tth {
 	margin: .1em auto;
 	width: 60%;
 	height: 60%;
+	background: #999;
+	background: rgba(128,128,128,0.2);
 }
 #modalb {
 	position: sticky;

copyparty/web/up2k.js

@@ -695,8 +695,9 @@ function Donut(uc, st) {
         }
 
         if (++r.tc >= 10) {
+            var s = r.eta === null ? 'paused' : r.eta > 60 ? shumantime(r.eta) : (r.eta + 's');
             wintitle("{0}%, {1}, #{2}, ".format(
-                f2f(v * 100 / t, 1), shumantime(r.eta), st.files.length - st.nfile.upload), true);
+                f2f(v * 100 / t, 1), s, st.files.length - st.nfile.upload), true);
             r.tc = 0;
         }
 
@@ -880,7 +881,7 @@ function up2k_init(subtle) {
     bcfg_bind(uc, 'turbo', 'u2turbo', turbolvl > 1, draw_turbo);
     bcfg_bind(uc, 'datechk', 'u2tdate', turbolvl < 3, null);
     bcfg_bind(uc, 'az', 'u2sort', u2sort.indexOf('n') + 1, set_u2sort);
-    bcfg_bind(uc, 'hashw', 'hashw', !!WebAssembly && (!subtle || !CHROME || MOBILE || VCHROME >= 107), set_hashw);
+    bcfg_bind(uc, 'hashw', 'hashw', !!WebAssembly && !(CHROME && MOBILE) && (!subtle || !CHROME), set_hashw);
     bcfg_bind(uc, 'upnag', 'upnag', false, set_upnag);
     bcfg_bind(uc, 'upsfx', 'upsfx', false, set_upsfx);
 
@@ -968,7 +969,7 @@ function up2k_init(subtle) {
             ud = function () { ebi('dir' + fdom_ctr).click(); };
 
         // too buggy on chrome <= 72
-        var m = / Chrome\/([0-9]+)\./.exec(navigator.userAgent);
+        var m = / Chrome\/([0-9]+)\./.exec(UA);
         if (m && parseInt(m[1]) < 73)
             return uf();
 
@@ -1359,7 +1360,15 @@ function up2k_init(subtle) {
             draw_each = good_files.length < 50;
 
         if (WebAssembly && !hws.length) {
-            for (var a = 0; a < Math.min(navigator.hardwareConcurrency || 4, 16); a++)
+            var nw = Math.min(navigator.hardwareConcurrency || 4, 16);
+
+            if (CHROME) {
+                // chrome-bug 383568268 // #124
+                nw = Math.max(1, (nw > 4 ? 4 : (nw - 1)));
+                nw = (subtle && !MOBILE && nw > 2) ? 2 : nw;
+            }
+
+            for (var a = 0; a < nw; a++)
                 hws.push(new Worker(SR + '/.cpr/w.hash.js?_=' + TS));
 
             if (!subtle)
@@ -1556,8 +1565,10 @@ function up2k_init(subtle) {
         if (nhash) {
             st.time.hashing += td;
             t.push(['u2etah', st.bytes.hashed, st.bytes.hashed, st.time.hashing]);
-            if (uc.fsearch)
+            if (uc.fsearch) {
+                st.time.busy += td;
                 t.push(['u2etat', st.bytes.hashed, st.bytes.hashed, st.time.hashing]);
+            }
         }
 
         var b_up = st.bytes.inflight + st.bytes.uploaded,
@@ -1961,32 +1972,84 @@ function up2k_init(subtle) {
             nchunk = 0,
             chunksize = get_chunksize(t.size),
             nchunks = Math.ceil(t.size / chunksize),
+            csz_mib = chunksize / 1048576,
+            tread = t.t_hashing,
+            cache_buf = null,
+            cache_car = 0,
+            cache_cdr = 0,
+            hashers = 0,
             hashtab = {};
 
+        // resolving subtle.digest w/o worker takes 1sec on blur if the actx hack breaks
+        var use_workers = hws.length && !hws_ng && uc.hashw && (nchunks > 1 || document.visibilityState == 'hidden'),
+            hash_par = (!subtle && !use_workers) ? 0 : csz_mib < 48 ? 2 : csz_mib < 96 ? 1 : 0;
+
         pvis.setab(t.n, nchunks);
         pvis.move(t.n, 'bz');
 
-        if (hws.length && !hws_ng && uc.hashw && (nchunks > 1 || document.visibilityState == 'hidden'))
+        if (use_workers)
-            // resolving subtle.digest w/o worker takes 1sec on blur if the actx hack breaks
             return wexec_hash(t, chunksize, nchunks);
 
         var segm_next = function () {
             if (nchunk >= nchunks || bpend)
                 return false;
 
-            var reader = new FileReader(),
+            var nch = nchunk++,
-                nch = nchunk++,
                 car = nch * chunksize,
                 cdr = Math.min(chunksize + car, t.size);
 
             st.bytes.hashed += cdr - car;
             st.etac.h++;
 
-            var orz = function (e) {
+            if (MOBILE && CHROME && st.slow_io === null && nch == 1 && cdr - car >= 1024 * 512) {
-                bpend--;
+                var spd = Math.floor((cdr - car) / (Date.now() + 1 - tread));
-                segm_next();
+                st.slow_io = spd < 40 * 1024;
-                hash_calc(nch, e.target.result);
+                console.log('spd {0}, slow: {1}'.format(spd, st.slow_io));
             }
+
+            if (cdr <= cache_cdr && car >= cache_car) {
+                try {
+                    var ofs = car - cache_car,
+                        ofs2 = ofs + (cdr - car),
+                        buf = cache_buf.subarray(ofs, ofs2);
+
+                    hash_calc(nch, buf);
+                }
+                catch (ex) {
+                    vis_exh(ex + '', 'up2k.js', '', '', ex);
+                }
+                return;
+            }
+
+            var reader = new FileReader(),
+                fr_cdr = cdr;
+
+            if (st.slow_io) {
+                var step = cdr - car,
+                    tgt = 48 * 1048576;
+
+                while (step && fr_cdr - car < tgt)
+                    fr_cdr += step;
+                if (fr_cdr - car > tgt && fr_cdr > cdr)
+                    fr_cdr -= step;
+                if (fr_cdr > t.size)
+                    fr_cdr = t.size;
+            }
+
+            var orz = function (e) {
+                bpend = 0;
+                var buf = e.target.result;
+                if (fr_cdr > cdr) {
+                    cache_buf = new Uint8Array(buf);
+                    cache_car = car;
+                    cache_cdr = fr_cdr;
+                    buf = cache_buf.subarray(0, cdr - car);
+                }
+                if (hashers < hash_par)
+                    segm_next();
+
+                hash_calc(nch, buf);
+            };
             reader.onload = function (e) {
                 try { orz(e); } catch (ex) { vis_exh(ex + '', 'up2k.js', '', '', ex); }
             };
@@ -2013,17 +2076,20 @@ function up2k_init(subtle) {
 
                 toast.err(0, 'y o u   b r o k e    i t\nfile: ' + esc(t.name + '') + '\nerror: ' + err);
             };
-            bpend++;
+            bpend = 1;
-            reader.readAsArrayBuffer(t.fobj.slice(car, cdr));
+            tread = Date.now();
+            reader.readAsArrayBuffer(t.fobj.slice(car, fr_cdr));
 
             return true;
         };
 
         var hash_calc = function (nch, buf) {
+            hashers++;
             var orz = function (hashbuf) {
                 var hslice = new Uint8Array(hashbuf).subarray(0, 33),
                     b64str = buf2b64(hslice);
 
+                hashers--;
                 hashtab[nch] = b64str;
                 t.hash.push(nch);
                 pvis.hashed(t);
@@ -2406,6 +2472,9 @@ function up2k_init(subtle) {
                     msg = 'done';
 
                 if (t.postlist.length) {
+                    if (t.rechecks && QS('#opa_del.act'))
+                        toast.inf(30, L.u_started, L.u_unpt);
+
                     var arr = st.todo.upload,
                         sort = arr.length && arr[arr.length - 1].nfile > t.n;
 
@@ -2516,8 +2585,13 @@ function up2k_init(subtle) {
                     if (!t.rechecks && (err_pend || err_srcb)) {
                         t.rechecks = 0;
                         t.want_recheck = true;
-                        err = L.u_dupdefer;
+                        if (st.busy.upload.length || st.busy.handshake.length || st.bytes.uploaded) {
-                        cls = 'defer';
+                            err = L.u_dupdefer;
+                            cls = 'defer';
+                        }
+                    }
+                    if (err_pend) {
+                        err += ' <a href="#" onclick="toast.inf(60, L.ue_ab);" class="fsearch_explain">(' + L.u_expl + ')</a>';
                     }
                 }
 
@@ -3025,7 +3099,7 @@ function up2k_init(subtle) {
                 new_state = false;
                 fixed = true;
             }
-            if (new_state === undefined)
+            if (new_state === undefined && preferred === undefined)
                 new_state = can_write ? false : have_up2k_idx ? true : undefined;
         }
 

copyparty/web/util.js

@@ -5,10 +5,17 @@ if (!window.console || !console.log)
         "log": function (msg) { }
     };
 
+if (!Object.assign)
+    Object.assign = function (a, b) {
+        for (var k in b)
+            a[k] = b[k];
+    };
+
+if (window.CGV1)
+    Object.assign(window, window.CGV1);
 
 if (window.CGV)
-    for (var k in CGV)
+    Object.assign(window, window.CGV);
-        window[k] = CGV[k];
 
 
 var wah = '',
@@ -22,14 +29,15 @@ var wah = '',
     HTTPS = ('' + location).indexOf('https:') === 0,
     TOUCH = 'ontouchstart' in window,
     MOBILE = TOUCH,
-    CHROME = !!window.chrome,
+    CHROME = !!window.chrome,  // safari=false
     VCHROME = CHROME ? 1 : 0,
-    IE = /Trident\//.test(navigator.userAgent),
+    UA = '' + navigator.userAgent,
-    FIREFOX = ('netscape' in window) && / rv:/.test(navigator.userAgent),
+    IE = /Trident\//.test(UA),
-    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(navigator.userAgent),
+    FIREFOX = ('netscape' in window) && / rv:/.test(UA),
-    LINUX = /Linux/.test(navigator.userAgent),
+    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(UA),
-    MACOS = /[^a-z]mac ?os/i.test(navigator.userAgent),
+    LINUX = /Linux/.test(UA),
-    WINDOWS = /Windows/.test(navigator.userAgent);
+    MACOS = /Macintosh/.test(UA),
+    WINDOWS = /Windows/.test(UA);
 
 if (!window.WebAssembly || !WebAssembly.Memory)
     window.WebAssembly = false;
@@ -189,7 +197,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
         '<p style="font-size:1.3em;margin:0;line-height:2em">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a> / <a href="?b=u">basic</a></p>',
         '<p style="color:#fff">please send me a screenshot arigathanks gozaimuch: <a href="<ghi>" target="_blank">new github issue</a></p>',
         '<p class="b">' + esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(msg).replace(/\n/g, '<br />') + '</p>',
-        '<p><b>UA:</b> ' + esc(navigator.userAgent + '')
+        '<p><b>UA:</b> ' + esc(UA)
     ];
 
     try {
@@ -571,7 +579,9 @@ function yscroll() {
 
 function showsort(tab) {
     var v, vn, v1, v2, th = tab.tHead,
-        sopts = jread('fsort', jcp(dsort));
+        sopts = jread('fsort');
+
+    sopts = sopts && sopts.length ? sopts : dsort;
 
     th && (th = th.rows[0]) && (th = th.cells);
 
@@ -608,10 +618,13 @@ function sortTable(table, col, cb) {
         tr = Array.prototype.slice.call(tb.rows, 0),
         i, reverse = /s0[^r]/.exec(th[col].className + ' ') ? -1 : 1;
 
-    var stype = th[col].getAttribute('sort');
+    var kname = th[col].getAttribute('name'),
+        stype = th[col].getAttribute('sort');
     try {
-        var nrules = [], rules = jread("fsort", []);
+        var nrules = [],
-        rules.unshift([th[col].getAttribute('name'), reverse, stype || '']);
+            rules = kname == 'href' ? [] : jread("fsort", []);
+
+        rules.unshift([kname, reverse, stype || '']);
         for (var a = 0; a < rules.length; a++) {
             var add = true;
             for (var b = 0; b < a; b++)
@@ -874,6 +887,11 @@ if (window.Number && Number.isFinite)
 
 function f2f(val, nd) {
     // 10.toFixed(1) returns 10.00 for certain values of 10
+    if (!isNum(val)) {
+        val = parseFloat(val);
+        if (!isNum(val))
+            val = 999;
+    }
     val = (val * Math.pow(10, nd)).toFixed(0).split('.')[0];
     return nd ? (val.slice(0, -nd) || '0') + '.' + val.slice(-nd) : val;
 }
@@ -970,11 +988,33 @@ function apop(arr, v) {
 }
 
 
-function jcp(obj) {
+function jcp1(obj) {
     return JSON.parse(JSON.stringify(obj));
 }
 
 
+function jcp2(src) {
+    if (Array.isArray(src)) {
+        var ret = [];
+        for (var a = 0; a < src.length; ++a) {
+            var sub = src[a];
+            ret.push((sub === null) ? sub : (sub instanceof Date) ? new Date(sub.valueOf()) : (typeof sub === 'object') ? jcp2(sub) : sub);
+        }
+    } else {
+        var ret = {};
+        for (var key in src) {
+            var sub = src[key];
+            ret[key] = sub === null ? sub : (sub instanceof Date) ? new Date(sub.valueOf()) : (typeof sub === 'object') ? jcp2(sub) : sub;
+        }
+    }
+    return ret;
+};
+
+
+// jcp1 50% faster on android-chrome, jcp2 7x everywhere else
+var jcp = MOBILE && CHROME ? jcp1 : jcp2;
+
+
 function sdrop(key) {
     try {
         STG.removeItem(key);
@@ -1542,8 +1582,8 @@ var toast = (function () {
 
         var html = '';
         if (sec) {
-            setcvar('--tmtime', sec + 's');
+            setcvar('--tmtime', (sec - 0.15) + 's');
-            setcvar('--tmstep', sec * 15);
+            setcvar('--tmstep', Math.floor(sec * 20));
             html += '<div id="toastt"></div>';
         }
         obj.innerHTML = html + '<a href="#" id="toastc">x</a><div id="toastb">' + lf2br(txt) + '</div>';

docs/README.md

@@ -25,6 +25,9 @@
 ## [`changelog.md`](changelog.md)
 * occasionally grabbed from github release notes
 
+## [`synology-dsm.md`](synology-dsm.md)
+* running copyparty on a synology nas
+
 ## [`devnotes.md`](devnotes.md)
 * technical stuff
 

docs/changelog.md

@@ -1,3 +1,337 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0122-2326  `v1.16.9`  ZeroMQ says hello
+
+## 🧪 new features
+
+* event-hooks can send zeromq / zmq / 0mq messages; see [readme](https://github.com/9001/copyparty#zeromq) or `--help-hooks` for examples d9db1534
+* new volflags to specify the [allow-tag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes) of the markdown/logue sandbox, to allow fullscreen and such (see `--help-flags`) 6a0aaaf0
+* new volflag `nosparse` for possibly-better performance in very rare and specific scenarios 917380dd
+  * only enable this if you're uploading to s3 or something like that, and do plenty of benchmarking to make sure that it actually improved performance instead of making it worse
+
+## 🩹 bugfixes
+
+* restrict max-length of filekeys to 72 characters e0cac6fd
+* the hash-calculator mode of the commandline uploader produced incorrect whole-file hashes 4c04798a
+  * each chunk (`--chs`) was okay, but the final sum was not
+
+## 🔧 other changes
+
+* selftest the xml-parser on startup with malicious xml b2e8bf6e
+  * just in case a future python-version suddenly makes it unsafe somehow
+* disable some features if a dangerously misconfigured reverseproxy is detected 3f84b0a0
+* the download-as-zip feature now defaults to utf8 filenames 1231ce19
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0111-1611  `v1.16.8`  android boost
+
+## 🧪 new features
+
+* 10x faster file hashing in android-chrome ec507889
+  * on a recent pixel, speed went from 13 to 139 MiB/s
+  * android's sandboxing makes small reads expensive, so do bigger reads instead
+    * so the browser-tab will use more RAM on android now, maybe around 200 MiB
+    * this only affects chrome-based browsers on android, not firefox
+* PUT/multipart uploads: request-header `Accept: json` makes it return json instead of html, just like `?j` ce0e5be4
+* add config examples for [ishare](https://isharemac.app/), a MacOS screenshot utility inspired by ShareX 0c0d6b2b
+  * also includes a bug-workaround for [ishare#107](https://github.com/castdrian/ishare/issues/107) - copyparty will now include a toplevel json property `fileurl` in the response if exactly one file was uploaded
+  * the [connect-page](https://a.ocv.me/?hc) generates an appropriate `copyparty.iscu` for ishare; [it looks like this](https://github.com/user-attachments/assets/820730ad-2319-4912-8eb2-733755a4cf54)
+
+## 🩹 bugfixes
+
+* fix a potential upload deadlock when...
+  * ...the database (`-e2d`) is **not** enabled for any volume, and...
+  * ...either the shares feature, or user-changeable passwords, is enabled 9e542cf8
+* when loading the partial-uploads registry on startup, a cosmetic desync could occur 467acb47
+
+## 🔧 other changes
+
+* remove some deprecated properties in partial-upload metadata aa2a8fa2
+  * v1.15.7 is now the oldest version which still has any chance of reading a modern up2k.snap
+* #129 added howto: [using webdav when copyparty is behind IdP](https://github.com/9001/copyparty/blob/hovudstraum/docs/idp.md#connecting-webdav-clients) -- thanks @wuast94 !
+* added howto: [install copyparty on a synology nas](https://github.com/9001/copyparty/blob/hovudstraum/docs/synology-dsm.md) 21f93042
+* more examples in the connect-page: 278258ee fb139697
+  * config-file for sharex on windows
+  * config-file for ishare on macos
+  * script for flameshot on linux
+* #75 add recommendation to use the [kamelåså project](https://github.com/steinuil/kameloso) instead of copyparty's [very-bad-idea.py](https://github.com/9001/copyparty/tree/hovudstraum/bin/mtag#dangerous-plugins) 9f84dc42
+* more reverse-proxy examples (haproxy, lighttpd, traefik, caddy) and improved nginx performance ac0a2da3
+  * readme has a [performance comparison](https://github.com/9001/copyparty?tab=readme-ov-file#reverse-proxy-performance) -- `haproxy > caddy > traefik > nginx > apache > lighttpd`
+* copyparty.exe: updated pillow 244e952f
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1223-0005  `v1.16.7`  an idp fix for xmas
+
+# ☃️🎄 **there is still time** 🎅🎁
+
+❄️❄️❄️ please [enjoy some appropriate music](https://a.ocv.me/pub/demo/music/.bonus/#af-55d4554d) -- you'll probably like this more than the idp thing honestly ❄️❄️❄️
+
+## 🧪 new features
+
+* more improvements to the recent-uploads feature 87598dcd
+  * move html rendering to clientside
+    * any changes to the filter-text applies in real-time
+    * loads 50% faster, reduces server-load by 30%
+    * inhibits search engines from indexing it
+
+## 🩹 bugfixes
+
+* using idp without e2d could mess with uploads dd6e9ea7
+* u2c (commandline uploader): fix window title 946a8c5b
+* mDNS/SSDP: fix incorrect log colors when multiple primary IPs are lost 552897ab
+
+## 🔧 other changes
+
+* ui: make it more obvious that the volume-control is a volume-control 7f044372
+* copyparty.exe: update deps (jinja2, markupsafe, pyinstaller) c0dacbc4
+* improve safety of custom plugins 988a7223
+  * if you've made your own plugins which expect certain values (host-header, filekeys) to be html-safe, then you'll want to upgrade
+  * also fixes rss-feed xml if password contains special characters
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1219-0037  `v1.16.6`  merry \x58mas
+
+# ☃️🎄 **it is time** 🎅🎁
+
+❄️❄️❄️ please [enjoy some appropriate music](https://a.ocv.me/pub/demo/music/.bonus/#af-55d4554d) (trust me on this one, you won't regret it) ❄️❄️❄️
+
+## 🧪 new features
+
+* [list of recent uploads](https://a.ocv.me/?ru) eaa4b04a
+  * new button in the controlpanel; can be disabled with `--no-ups-page`
+  * only users with the dot-permission can see dotfiles
+  * only admins can see uploader-ip and upload-times
+    * enable `--ups-when` to let all users see upload-times
+* #125 log decoded request-URLs 73f7249c
+  * non-ascii filenames would make the accesslog a wall of `%E5%B9%BB%E6%83%B3%E9%83%B7` so print [the decoded URL](https://github.com/user-attachments/assets/9d411183-30f3-4cb2-a880-84cf18011183) in addition to the original one, which is left as-is for debugging purposes
+
+## 🩹 bugfixes
+
+* #126 improve dotfile handling 4c4e48ba
+  * was impossible to delete a folder which contained hidden files if the user did not have the permission to see hidden files
+  * would also affect moving, renaming, copying folders, in which case the dotfiles would not be carried over to the new location
+  * now, dotfiles are always deleted, and always moved/copied into a new destination, on the condition that this is safe -- if the user has the dotfile permission in the target loocation but not in the source location, the dotfiles will be left behind to avoid accidentally making then browsable
+* ux: cosmetic eta/idle-timer fixes 01a3eb29
+
+## 🔧 other changes
+
+* warn on ambiguous comments in config files da5ad2ab
+* avoid writing mojibake to the log 3051b131
+  * use `\x`-encoding for unprintable text
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1211-2236  `v1.16.5`  4chrome
+
+## 🧪 new features
+
+* #124 add workaround for a chrome bug (crash during upload) 24ce46b3
+  * chrome and chromium-based browsers could OOM
+  * https://issues.chromium.org/issues/383568268
+
+* #122 "hybrid IdP", regular users can still auth while [IdP](https://github.com/9001/copyparty#identity-providers) is enabled 64501fd7
+  * previously, enabling IdP would entirely disable password-based login
+  * now, password-auth is attempted for requests without a valid IdP header
+
+## 🩹 bugfixes
+
+* the terminal window title would only change if `--no-ansi` was specified, which is exactly the opposite of what it should be (and now is) doing db3c0b09
+
+## 🔧 other changes
+
+* mDNS: better log messages when several IPs are added/removed a49bf81f
+* webdeps: update dompurify 06868606
+
+----
+
+this release includes a build of [copyparty-winpe64.exe](https://github.com/9001/copyparty/releases/download/v1.16.5/copyparty-winpe64.exe) since the last one was [almost a year ago](https://github.com/9001/copyparty/releases/tag/v1.10.1)
+
+* winpe64.exe is only for *very* specific usecases, you almost definitely *do not* want to download it, please just grab the regular [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) instead (works on all 64bit machines running win8 or newer)
+
+* the only difference between winpe64.exe and [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is that winpe64.exe works in the win7x64 PE (rescue-env), which makes it *almost* entirely useless, and every bit as dangerous to use as copyparty32.exe
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1207-0024  `v1.16.4`  ux is hard
+
+## 🧪 new features
+
+* improve the upload ui so it explains how to abort an unfinished upload when someone uploads to the wrong folder by accident be6afe2d
+  * also reduces serverload slightly when cloning an incoming file to multiple destinations
+* u2c (commandline uploader): windows improvements 91637800
+  * now supports globbing (filename wildcards) on windows
+  * progressbar in the windows taskbar (requires conemu or the "new windows terminal")
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1204-0003  `v1.16.3`  120%
+
+## 🧪 new features
+
+* #120 add option `--srch-excl` and volflag `srch_excl` for excluding certain paths from search results 697a4fa8
+* mDNS: add workaround for https://github.com/avahi/avahi/issues/379 6c1cf68b 94d1924f
+  * Avahi mDNS Reflection, sometimes used in intricate LAN setups, doesn't understand NSEC records and corrupts them
+  * the workaround makes copyparty able to read the corrupted packets, but clients without a similar workaround will require either `--zm4` or `--zm6` so copyparty doesn't include the usual NSEC records
+    * this is mentioned in a very loud warning in the logs when necessary
+* mDNS: option to silently ignore buggy devices instead of spamming the log with parser errors 395af051
+* webdav: support listing unmapped root with infinite recursion (Depth:0) 21a3f369
+* embed current sort config into media URLs (gallery/music) 0f257c93 4cfdc4c5 01670827
+  * ensures that anyone clicking your link will see the files in the same order as you
+  * can be confgured serverside (`--hsortn`, volflag `hsortn`) and clientside (`#sort` in settings)
+* URL and UI options to disable checksum calculation of PUT, bup, basic uploads c5a000d2
+  * also allows [choosing either md5, sha1, sha256, or blake2](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#write) instead of the default sha512
+  * can give uploads a nice speed boost when copyparty is running on a potato
+
+## 🩹 bugfixes
+
+* webdav: more correct login challenge 2ce82339
+  * the previous behavior could make some clients reluctant to send the password
+* #120 forget metadata of all files (including uploads) when shadowed d168b2ac
+  * thanks to @Gremious for all the debugging to narrow this down!
+* #120 drop volume caches if relevant config is changed (mainly indexing filters) 2f83c6c7
+* #121 couldn't access arbitrary toplevel files from accounts with `h` permission 1f5f42f2
+
+## 🔧 other changes
+
+* exclude thumbnails from accesslog by default 9082c470
+* filesearch: show a final summary of time-elapsed and average hashing speed 8a631f04
+* improve phrasing of debug messages during indexing at startup 127f414e
+* `--license` no longer depends on opensource.org at build time 33c4ccff
+* update deps 6cedcfbf
+  * copyparty.exe: python 3.12.7 => 3.12.8
+  * webdeps: hashwasm, dompurify
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1123-2336  `v1.16.2`  webdav upload fix
+
+## 🧪 new features
+
+* add `--nsort` and volflag `nsort` to default-enable natural sort of filenames with leading digits 8f7ffcf3
+* video-player: support `.mov` files which contain browser-native codecs 2d0cbdf1
+
+## 🩹 bugfixes
+
+* #119 v1.16.0 broke webdav uploads from rclone and possibly other clients 7dfbfc72
+  * a collection of webdav unittests will be added soon to prevent similar issues in the future
+* #118 ip-ranges can be mixed with `lan` when specifying the list of trusted proxies for `x-forwarded-for` with `--xff-src`
+  * found and fixed by @codemicro (thx!) 0e31cfa7
+* ux:
+  * in the grid-view, markdown files would open in the generic text viewer 520ac8f4
+  * qr-codes (create-share, view-share) didn't render on chrome db069c3d
+  * qr-codes could cause layout-shifting 5afb562a
+  * fix layout-shifting for ongoing downloads in controlpanel 9c8507a0
+  * cosmetic eta jank b10843d0
+
+## 🔧 other changes
+
+* upto 7% faster folder listings due to refactoring for more ux knobs 0c43b592 
+* fix resource leaks (only affected tests/debug) 2ab8924e
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1115-2218  `v1.16.1`  cbz thumbnails
+
+## 🧪 new features
+
+* thumbnails of .cbz manga archives 4d15dd6e
+
+## 🩹 bugfixes
+
+* when running with `-j0`, download-ETA could break in complex volume layouts 10fc4768
+* linking to the image gallery didn't quite work if multiselect was enabled 56a04996
+* password-hashing parameters (cpu/ram cost) could not be customized 1f177528
+  * the defaults must be perfect considering nobody ever tried changing them ¯\\_(ツ)_/¯
+
+## 🔧 other changes
+
+* add intentional crash on startup if two volumes are configured to use the same histpath 2b63d7d1
+  * prevents funky deadlocks and an eventual database loss in case of a no-thoughts-head-empty moment, purely hypothetical of course 🗿
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1110-1932  `v1.16.0`  COPYparty
+
+## 🧪 new features
+
+* #46 #115 copy/paste files and folders cacec9c1
+  * cut/paste still exists, but now you can copy too
+  * with a UI to rename files in case of filename collisions 56317b00
+  * files are created according to the dedup settings in the target volume (either full copies or symlinks/hardlinks)
+* show currently active downloads in the controlpanel 8aba5aed
+  * can be made admin-only with `--dl-list=1` or disabled with `--dl-list=0`
+  * hides filenames of hidden files, and files from volumes where the viewer doesn't have access
+* #114 async reinit on new [IdP users](https://github.com/9001/copyparty#identity-providers) 44ee07f0
+  * new IdP users can now always auth, even while a filesystem reindex is running
+* ux:
+  * remember batch-rename settings from last time 6a8d5e17
+  * URL parameters to force grid/thumbs on/off 5718caa9
+
+## 🩹 bugfixes
+
+* folders that fail to list due to a corrupt HDD/filesystem will now return a 404 instead of an empty listing 119e88d8
+  * also fixes similar issues in u2c and partyfuse
+* u2c (commandline uploader): detect and adapt to proxies with short connection keepalives c784e528
+* ui/ux:
+  * show the "switch-to-https" button in 404-messages too efd8a32e
+  * the folder-loading indicator could steal keyboard focus d9962f65
+  * hotkey-help was very trigger-happy 71d9e010
+
+## 🔧 other changes
+
+* choose more conservative defaults when server has less than 1 GiB RAM 2bf9055c
+  * runs okay down to 128 MiB, but thumbnails die below 256 MiB
+* update the [comparison to similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) after years of optimizations on both sides 0ce7cf5e
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-1027-0751  `v1.15.10`  temporary upload links
+
+## 🧪 new features
+
+* [shares](https://github.com/9001/copyparty#shares) can now be uploaded into, and unpost works too 4bdcbc1c
+  * useful to create temporary URLs for other people to upload to
+  * shares can be write-only, so visitors can't browse or see any files
+* #110 HTTP 304 (caching):
+  * support `If-Range` for HTTP 206 159f51b1
+  * add server-side and client-side options to force-disable cache dd6dbdd9
+    * `--no304=1` shows a button in the controlpanel to disable caching
+    * `--no304=2` makes that button auto-enabled
+    * even when `--no304` is not specified, accessing the URL `/?setck=no304=y` force-disables cache
+    * when cache is force-disabled, browsers will waste a lot of network traffic / data usage
+    * might help to avoid bugs in browsers or proxies, for example if media files suddenly stop loading
+      * but such bugs should be exceedingly rare, so do not enable this unless actually necessary 
+
+## 🩹 bugfixes
+
+* #110 HTTP 304 (caching):
+  * remove `Content-Length` and `Content-Type` response headers from 304 replies 91240236
+    * browsers don't need these, and some middlewares might get confused if they're present
+* #113 fix crash on startup if `-j0` was combined with `--ipa` or `--ipu` 3a0d882c
+* #111 fix javascript crash if `--u2sz` was set to an invalid value b13899c6
+
+## 🔧 other changes
+
+* #110 HTTP 304 (caching):
+  * never automatically enable k304 because the `Vary` header killed support for caching in msie anyways 63013cc5
+  * change time comparison for `If-Modified-Since` to require an exact timestamp match, instead of the intended "modified since". This technically violates the http-spec, but should be safer for backdating file mtimes 159f51b1
+* new option `--ohead` to log response headers 7678a91b
+* added [nintendo 3ds](https://github.com/user-attachments/assets/88deab3d-6cad-4017-8841-2f041472b853) to the [list of supported browsers](https://github.com/9001/copyparty#browser-support) cb81f0ad
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2024-1018-2342  `v1.15.9`  rss server
 

docs/chunksizes.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+
+# there's far better ways to do this but its 4am and i dont wanna think
+
+# just pypy it my dude
+
+import math
+
+def humansize(sz, terse=False):
+    for unit in ["B", "KiB", "MiB", "GiB", "TiB"]:
+        if sz < 1024:
+            break
+
+        sz /= 1024.0
+
+    ret = " ".join([str(sz)[:4].rstrip("."), unit])
+
+    if not terse:
+        return ret
+
+    return ret.replace("iB", "").replace(" ", "")
+
+
+def up2k_chunksize(filesize):
+    chunksize = 1024 * 1024
+    stepsize = 512 * 1024
+    while True:
+        for mul in [1, 2]:
+            nchunks = math.ceil(filesize * 1.0 / chunksize)
+            if nchunks <= 256 or (chunksize >= 32 * 1024 * 1024 and nchunks <= 4096):
+                return chunksize
+
+            chunksize += stepsize
+            stepsize *= mul
+
+
+def main():
+    prev = 1048576
+    n = n0 = 524288
+    while True:
+        csz = up2k_chunksize(n)
+        if csz > prev:
+            print(f"| {n-n0:>18_} | {humansize(n-n0):>8} | {prev:>13_} | {humansize(prev):>8} |".replace("_", " "))
+            prev = csz
+        n += n0
+
+
+main()

docs/devnotes.md

@@ -6,6 +6,7 @@
     * [up2k](#up2k) - quick outline of the up2k protocol
         * [why not tus](#why-not-tus) - I didn't know about [tus](https://tus.io/)
         * [why chunk-hashes](#why-chunk-hashes) - a single sha512 would be better, right?
+        * [list of chunk-sizes](#list-of-chunk-sizes) - specific chunksizes are enforced
 * [hashed passwords](#hashed-passwords) - regarding the curious decisions
 * [http api](#http-api)
     * [read](#read)
@@ -95,6 +96,44 @@ hashwasm would solve the streaming issue but reduces hashing speed for sha512 (x
 
 * blake2 might be a better choice since xxh is non-cryptographic, but that gets ~15 MiB/s on slower androids
 
+### list of chunk-sizes
+
+specific chunksizes are enforced  depending on total filesize
+
+each pair of filesize/chunksize is the largest filesize which will use its listed chunksize; a 512 MiB file will use chunksize 2 MiB, but if the file is one byte larger than 512 MiB then it becomes 3 MiB
+
+for the purpose of performance (or dodging arbitrary proxy limitations), it is possible to upload combined and/or partial chunks using stitching and/or subchunks respectively
+
+| filesize           | filesize | chunksize     | chunksz |
+| -----------------: | -------: | ------------: | ------: |
+|        268 435 456 |  256 MiB |     1 048 576 | 1.0 MiB |
+|        402 653 184 |  384 MiB |     1 572 864 | 1.5 MiB |
+|        536 870 912 |  512 MiB |     2 097 152 | 2.0 MiB |
+|        805 306 368 |  768 MiB |     3 145 728 | 3.0 MiB |
+|      1 073 741 824 |  1.0 GiB |     4 194 304 | 4.0 MiB |
+|      1 610 612 736 |  1.5 GiB |     6 291 456 | 6.0 MiB |
+|      2 147 483 648 |  2.0 GiB |     8 388 608 | 8.0 MiB |
+|      3 221 225 472 |  3.0 GiB |    12 582 912 |  12 MiB |
+|      4 294 967 296 |  4.0 GiB |    16 777 216 |  16 MiB |
+|      6 442 450 944 |  6.0 GiB |    25 165 824 |  24 MiB |
+|    137 438 953 472 |  128 GiB |    33 554 432 |  32 MiB |
+|    206 158 430 208 |  192 GiB |    50 331 648 |  48 MiB |
+|    274 877 906 944 |  256 GiB |    67 108 864 |  64 MiB |
+|    412 316 860 416 |  384 GiB |   100 663 296 |  96 MiB |
+|    549 755 813 888 |  512 GiB |   134 217 728 | 128 MiB |
+|    824 633 720 832 |  768 GiB |   201 326 592 | 192 MiB |
+|  1 099 511 627 776 |  1.0 TiB |   268 435 456 | 256 MiB |
+|  1 649 267 441 664 |  1.5 TiB |   402 653 184 | 384 MiB |
+|  2 199 023 255 552 |  2.0 TiB |   536 870 912 | 512 MiB |
+|  3 298 534 883 328 |  3.0 TiB |   805 306 368 | 768 MiB |
+|  4 398 046 511 104 |  4.0 TiB | 1 073 741 824 | 1.0 GiB |
+|  6 597 069 766 656 |  6.0 TiB | 1 610 612 736 | 1.5 GiB |
+|  8 796 093 022 208 |  8.0 TiB | 2 147 483 648 | 2.0 GiB |
+| 13 194 139 533 312 | 12.0 TiB | 3 221 225 472 | 3.0 GiB |
+| 17 592 186 044 416 | 16.0 TiB | 4 294 967 296 | 4.0 GiB |
+| 26 388 279 066 624 | 24.0 TiB | 6 442 450 944 | 6.0 GiB |
+| 35 184 372 088 832 | 32.0 TiB | 8 589 934 592 | 8.0 GiB |
+
 
 # hashed passwords
 
@@ -133,15 +172,19 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | GET | `?tar=xz:9` | ...as an xz-level-9 gnu-tar file |
 | GET | `?tar=pax` | ...as a pax-tar file |
 | GET | `?tar=pax,xz` | ...as an xz-level-1 pax-tar file |
-| GET | `?zip=utf-8` | ...as a zip file |
+| GET | `?zip` | ...as a zip file |
-| GET | `?zip` | ...as a WinXP-compatible zip file |
+| GET | `?zip=dos` | ...as a WinXP-compatible zip file |
 | GET | `?zip=crc` | ...as an MSDOS-compatible zip file |
 | GET | `?tar&w` | pregenerate webp thumbnails |
 | GET | `?tar&j` | pregenerate jpg thumbnails |
 | GET | `?tar&p` | pregenerate audio waveforms |
 | GET | `?shares` | list your shared files/folders |
+| GET | `?dls` | show active downloads (do this as admin) |
 | GET | `?ups` | show recent uploads from your IP |
 | GET | `?ups&filter=f` | ...where URL contains `f` |
+| GET | `?ru` | show all recent uploads |
+| GET | `?ru&filter=f` | ...where URL contains `f` |
+| GET | `?ru&j` | ...as json |
 | GET | `?mime=foo` | specify return mimetype `foo` |
 | GET | `?v` | render markdown file at URL |
 | GET | `?v` | open image/video/audio in mediaplayer |
@@ -163,15 +206,21 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 
 | method | params | result |
 |--|--|--|
+| POST | `?copy=/foo/bar` | copy the file/folder at URL to /foo/bar |
 | POST | `?move=/foo/bar` | move/rename the file/folder at URL to /foo/bar |
 
 | method | params | body | result |
 |--|--|--|--|
 | PUT | | (binary data) | upload into file at URL |
+| PUT | `?j` | (binary data) | ...and reply with json |
+| PUT | `?ck` | (binary data) | upload without checksum gen (faster) |
+| PUT | `?ck=md5` | (binary data) | return md5 instead of sha512 |
 | PUT | `?gz` | (binary data) | compress with gzip and write into file at URL |
 | PUT | `?xz` | (binary data) | compress with xz and write into file at URL |
 | mPOST | | `f=FILE` | upload `FILE` into the folder at URL |
 | mPOST | `?j` | `f=FILE` | ...and reply with json |
+| mPOST | `?ck` | `f=FILE` | ...and disable checksum gen (faster) |
+| mPOST | `?ck=md5` | `f=FILE` | ...and return md5 instead of sha512 |
 | mPOST | `?replace` | `f=FILE` | ...and overwrite existing files |
 | mPOST | `?media` | `f=FILE` | ...and return medialink (not hotlink) |
 | mPOST | | `act=mkdir`, `name=foo` | create directory `foo` at URL |
@@ -188,8 +237,15 @@ upload modifiers:
 | http-header | url-param | effect |
 |--|--|--|
 | `Accept: url` | `want=url` | return just the file URL |
+| `Accept: json` | `want=json` | return upload info as json; same as `?j` |
 | `Rand: 4` | `rand=4` | generate random filename with 4 characters |
 | `Life: 30` | `life=30` | delete file after 30 seconds |
+| `CK: no` | `ck` | disable serverside checksum (maybe faster) |
+| `CK: md5` | `ck=md5` | return md5 checksum instead of sha512 |
+| `CK: sha1` | `ck=sha1` | return sha1 checksum |
+| `CK: sha256` | `ck=sha256` | return sha256 checksum |
+| `CK: b2` | `ck=b2` | return blake2b checksum |
+| `CK: b2s` | `ck=b2s` | return blake2s checksum |
 
 * `life` only has an effect if the volume has a lifetime, and the volume lifetime must be greater than the file's
 
@@ -208,6 +264,12 @@ upload modifiers:
 | method | params | result |
 |--|--|--|
 | GET | `?pw=x` | logout |
+| GET | `?grid` | ui: show grid-view |
+| GET | `?imgs` | ui: show grid-view with thumbnails |
+| GET | `?grid=0` | ui: show list-view |
+| GET | `?imgs=0` | ui: show list-view |
+| GET | `?thumb` | ui, grid-mode: show thumbnails |
+| GET | `?thumb=0` | ui, grid-mode: show icons |
 
 
 # event hooks
@@ -280,6 +342,7 @@ python3 -m venv .venv
 . .venv/bin/activate
 pip install jinja2 strip_hints  # MANDATORY
 pip install argon2-cffi  # password hashing
+pip install pyzmq  # send 0mq from hooks
 pip install mutagen  # audio metadata
 pip install pyftpdlib  # ftp server
 pip install partftpy  # tftp server

docs/idp.md

@@ -20,3 +20,25 @@ this means that, if an IdP volume is located inside a folder that is readable by
 and likewise -- if the IdP volume is inside a folder that is only accessible by certain users, but the IdP volume is configured to allow access from unauthenticated users, then the contents of the volume will NOT be accessible until it is revived
 
 until this limitation is fixed (if ever), it is recommended to place IdP volumes inside an appropriate parent volume, so they can inherit acceptable permissions until their revival; see the "strategic volumes" at the bottom of [./examples/docker/idp/copyparty.conf](./examples/docker/idp/copyparty.conf)
+
+
+## Connecting webdav clients
+
+If you use only idp and want to connect via rclone you have to adapt a few things.
+The following steps are for Authelia, but should be easy adaptable to other IdPs and clients. There may be better/smarter ways to do this, but this is a known solution.
+
+1. Add a rule for your domain and set it to one factor
+```
+  rules:
+    - domain: 'sub.domain.tld'
+      policy: one_factor
+```
+2. After you created your rclone config find its location with `rclone config file` and add the headers option to it, change the string to `username:password` base64 encoded. Make sure to set the right url location, otherwise you will get a 401 from copyparty.
+```
+[servername-dav]
+type = webdav
+url = https://sub.domain.tld/u/user/priv/
+vendor = owncloud
+pacer_min_sleep = 0.01ms
+headers = Proxy-Authorization,basic base64encodedstring==
+```

docs/notes.sh

@@ -259,6 +259,12 @@ for d in /usr /var; do find $d -type f -size +30M 2>/dev/null; done | while IFS=
 for f in {0..255}; do echo $f; truncate -s 256M $f; b1=$(printf '%02x' $f); for o in {0..255}; do b2=$(printf '%02x' $o); printf "\x$b1\x$b2" | dd of=$f bs=2 seek=$((o*1024*1024)) conv=notrunc 2>/dev/null; done; done
 # create 6.06G file with 16 bytes of unique data at start+end of each 32M chunk
 sz=6509559808; truncate -s $sz f; csz=33554432; sz=$((sz/16)); step=$((csz/16)); ofs=0; while [ $ofs -lt $sz ]; do dd if=/dev/urandom of=f bs=16 count=2 seek=$ofs conv=notrunc iflag=fullblock; [ $ofs = 0 ] && ofs=$((ofs+step-1)) || ofs=$((ofs+step)); done
+# same but for chunksizes 16M (3.1G), 24M (4.1G), 48M (128.1G)
+sz=3321225472;   csz=16777216;
+sz=4394967296;   csz=25165824;
+sz=6509559808;   csz=33554432;
+sz=138438953472; csz=50331648;
+f=csz-$csz; truncate -s $sz $f; sz=$((sz/16)); step=$((csz/16)); ofs=0; while [ $ofs -lt $sz ]; do dd if=/dev/urandom of=$f bs=16 count=2 seek=$ofs conv=notrunc iflag=fullblock; [ $ofs = 0 ] && ofs=$((ofs+step-1)) || ofs=$((ofs+step)); done
 
 # py2 on osx
 brew install python@2

docs/synology-dsm.md

@@ -0,0 +1,140 @@
+# running copyparty on synology dsm nas
+
+![synology-dsm-container-status.png](https://ocv.me/copyparty/doc/pics/dsm.png)
+
+this has been tested on a `Synology ds218+` NAS with 1 SHR storage-pool and 1 volume, but the same steps should work in more advanced setups too
+
+verified on DSM 7.1 and 7.2, but not on 6.x since my flea-market ds218+ refuses to install it for some reason
+
+
+
+# ok let's go
+
+go to controlpanel -> shared-folders, and create the following shared-folders if you don't already have appropriate ones:
+
+* a shared-folder for configuration files, preferably on SSD if you have one
+
+* one or more shared-folders for your actual data/media to share
+
+(btw, when you create the shared-folders, it asks whether you want to enable data checksum and file compression, i would recommend both)
+
+the rest of this doc assumes that these two shared-folders are named `configs` and `media1`, and that you made an empty folder inside the `configs` shared-folder named `cpp`
+
+* your copyparty config file (see below) should be named `something.conf` directly inside that cpp folder, for example `/configs/cpp/copyparty.conf`
+
+* during first start, copyparty will create a folder there named `copyparty`, in other words `/configs/cpp/copyparty` which you should leave alone; that's where copyparty stores its indexes and other runtime config
+
+
+
+## recommended copyparty config
+
+open the Package Center and install `Text Editor` (by Synology Inc.) to create and edit your copyparty config:
+
+![synology-text-editor-copyparty-conf.png](https://ocv.me/copyparty/doc/pics/dsm-cfg.png)
+
+* note the `copyparty` and `hist` folders in that screenshot which are autogenerated by copyparty and to be left alone
+
+```yaml
+[global]
+  e2d, e2t         # remember uploads & read media tags
+  rss, daw, ver    # some other nice-to-have features
+  #dedup            # you may want this, or maybe not
+  hist: /cfg/hist  # don't pollute the shared-folder
+  name: synology   # shows in the browser, can be anything
+
+[accounts]
+  ed: wark   # username ed, password wark
+
+[/]          # share the following at the webroot:
+  /w         # the "/w" docker-volume (the shared-folder)
+  accs:
+    A: ed    # give Admin to username ed
+
+# hide the synology system files by creating a hidden volume
+[/@eaDir]
+  /w/@eaDir
+```
+
+if you ever change the copyparty config file, then [restart the container](https://ocv.me/copyparty/doc/pics/dsm71-02.png) to make the changes take effect
+
+okay now continue with one of these:
+
+* [DSM v7.2 or newer](#dsm-v72-or-newer)
+
+* [all older DSM versions](#dsm-v6x-dsm-v71x-or-older)
+
+
+
+# DSM v7.2 or newer
+
+`Docker` was replaced by `Container Manager` in DSM v7.2 but they're almost the same thing;
+
+* open the `Package Center` and install the [Container Manager package](https://ocv.me/copyparty/doc/pics/dsm72-01.png) by `Docker Inc.`
+* open the `Container Manager` app
+* go to the `Registry` tab and search for `copyparty`
+* [doubleclick copyparty/ac](https://ocv.me/copyparty/doc/pics/dsm72-02.png) and keep the [default `latest`](https://ocv.me/copyparty/doc/pics/dsm72-03.png) when it asks you which tag to use
+* switch to the `Container` tab and click `Create`
+* [choose `copyparty/ac:latest`](https://ocv.me/copyparty/doc/pics/dsm72-04.png) and click `Next`
+
+finally, in the [Advanced Settings](https://ocv.me/copyparty/doc/pics/dsm72-05.png) window,
+
+* under `Port Settings`, type `3923` into the `Local Port` textbox
+* click `Add Folder` and select `/configs/cpp` on your nas (the `cpp` folder in the `configs` shared-folder), and change `Mount path` to `/cfg`
+* click `Add Folder` and select `/media1` on your nas (the shared-folder that copyparty can share in its web-UI) and change `Mount path` to `/w`
+  * if you are adding multiple shared-folders for media, then the `Mount path` of the 2nd folder should be something like `/w/share2` or `/w/music`
+
+copyparty will launch and become available at http://192.168.1.9:3923/ (assuming `192.168.1.9` is your nas ip)
+
+
+# DSM v6.x, DSM v7.1.x or older
+
+if you're using DSM 7.1 or older, then you don't have [Container Manager](https://www.synology.com/en-global/dsm/packages/ContainerManager) yet and you'll have to use [Docker](https://www.synology.com/en-global/dsm/packages/Docker?os_ver=6.2&search=docker) instead. Here's how:
+
+* open the `Package Center` and install the [Docker package](https://ocv.me/copyparty/doc/pics/dsm71-01.png) by `Docker Inc.`
+* open the `Docker` app
+* go to the `Registry` tab and search for `copyparty`
+* [doubleclick copyparty/ac](https://ocv.me/copyparty/doc/pics/dsm71-02.png) and keep the [default `latest`](https://ocv.me/copyparty/doc/pics/dsm71-03.png) when it asks you which tag to use
+* switch to the `Container` tab and click `Create`
+* [choose `copyparty/ac:latest`](https://ocv.me/copyparty/doc/pics/dsm71-04.png) and `Next`
+* in the [Network](https://ocv.me/copyparty/doc/pics/dsm71-05.png) window, keep the default `Use the selected networks: [x] bridge`
+* in the [General Settings](https://ocv.me/copyparty/doc/pics/dsm71-06.png) window, just keep everything default (in other words, everything disabled)
+* in the [Port Settings](https://ocv.me/copyparty/doc/pics/dsm71-07.png) window, change `Local Port` to `3923` (or choose something else, but it cannot be the default `Auto`)
+
+finally, in the [Volume Settings](https://ocv.me/copyparty/doc/pics/dsm71-08.png) window, add a docker volume for copyparty config, and at least one volume for media-files which copyparty can share in its web-UI
+
+* click `Add Folder` and select `/configs/cpp` on your nas (the `cpp` folder in the `configs` shared-folder), and change `Mount path` to `/cfg`
+* click `Add Folder` and select `/media1` on your nas (the shared-folder that copyparty can share in its web-UI) and change `Mount path` to `/w`
+* if you are adding multiple shared-folders for media, then the `Mount path` of the 2nd folder should be something like `/w/share2` or `/w/music`
+
+copyparty will launch and become available at http://192.168.1.9:3923/ (assuming `192.168.1.9` is your nas ip)
+
+
+# misc notes
+
+note that if you only want to share some folders inside your data volume, and not all of it, then you can either give copyparty the whole shared-folder anyways and control/restrict access in the copyparty config file (recommended), or you can add each folder as a new docker volume (not as flexible)
+
+
+
+## regarding ram usage
+
+the ram usage indicator in both `Docker` and `Container Manager` is misleading  because it also counts the kernel disk cache which makes the number insanely high -- the synology resource monitor shows the correct values, usually less than 100 MiB
+
+to see the actual memory usage by copyparty, see `Resource Monitor` -> `Task Manager` -> `Processes` and look at the `Private Memory` of `python3` which is probably copyparty
+
+
+
+## regarding performance
+
+when uploading files to the synology nas with the respective web-UIs,
+
+* `File Station` does about 16 MiB/s,
+
+* `Synology Drive Server` does about 50 MiB/s; deceivingly fast upload speeds at first, but when the file is fully uploaded, there is a lengthy "processing" step at the end, reducing the average speed to about 50% of the initial
+
+* copyparty maxes the HDD write-speeds, 99 MiB/s
+
+when uploading to the synology nas over webdav,
+
+* `WebDAV Server` by `Synology Inc.` in the Package Center does 86 MiB/s
+
+* copyparty does 79 MiB/s; the NAS CPU is a bottleneck because copyparty verifies the upload checksum while `WebDAV Server` doesn't

docs/versus.md

@@ -279,7 +279,7 @@ symbol legend,
 | per-file passwords      | █ |   |   | █ | █ |   | █ |   | █ |   |   |   | █ |
 | unmap subfolders        | █ |   | █ |   |   |   | █ |   |   | █ | ╱ | • |   |
 | index.html blocks list  | ╱ |   |   |   |   |   | █ |   |   | • |   |   |   |
-| write-only folders      | █ |   | █ |   |   |   |   |   |   |   | █ | █ |   |
+| write-only folders      | █ |   | █ |   | █ |   |   |   |   |   | █ | █ |   |
 | files stored as-is      | █ | █ | █ | █ |   | █ | █ |   |   | █ | █ | █ | █ |
 | file versioning         |   |   |   | █ | █ |   |   |   |   |   |   |   |   |
 | file encryption         |   |   |   | █ | █ | █ |   |   |   |   |   | █ |   |
@@ -465,11 +465,13 @@ symbol legend,
 ## [hfs3](https://rejetto.com/hfs/)
 * nodejs; cross-platform
 * vfs with gui config, per-volume permissions
+* tested locally, v0.53.2 on archlinux
 * 🔵 uploads are resumable
 * ⚠️ uploads are not segmented; max upload size 100 MiB on cloudflare
 * ⚠️ uploads are not accelerated (copyparty is 3x faster across the atlantic)
 * ⚠️ uploads are not integrity-checked
 * ⚠️ copies the file after upload; need twice filesize free disk space
+* ⚠️ uploading small files is decent; `107` files per sec (copyparty does `670`/sec, 6x faster)
 * ⚠️ doesn't support crazy filenames
 * ✅ config GUI
 * ✅ download counter
@@ -478,11 +480,12 @@ symbol legend,
 
 ## [nextcloud](https://github.com/nextcloud/server)
 * php, mariadb
+* tested locally, [linuxserver/nextcloud](https://hub.docker.com/r/linuxserver/nextcloud) v30.0.2 (sqlite)
 * ⚠️ [isolated on-disk file hierarchy] in per-user folders
   * not that bad, can probably be remedied with bindmounts or maybe symlinks
 * ⚠️ uploads not resumable / accelerated / integrity-checked
   * ⚠️ on cloudflare: max upload size 100 MiB
-* ⚠️ uploading small files is slow; `2.2` files per sec (copyparty does `87`/sec), tested locally with [linuxserver/nextcloud](https://hub.docker.com/r/linuxserver/nextcloud) (sqlite)
+* ⚠️ uploading small files is slow; `4` files per sec (copyparty does `670`/sec, 160x faster)
 * ⚠️ no write-only / upload-only folders
 * ⚠️ http/webdav only; no ftp, zeroconf
 * ⚠️ less awesome music player
@@ -498,12 +501,12 @@ symbol legend,
 
 ## [seafile](https://github.com/haiwen/seafile)
 * c, mariadb
+* tested locally, [official container](https://manual.seafile.com/latest/docker/deploy_seafile_with_docker/) v11.0.13
 * ⚠️ [isolated on-disk file hierarchy](https://manual.seafile.com/maintain/seafile_fsck/), incompatible with other software
   * *much worse than nextcloud* in that regard
 * ⚠️ uploads not resumable / accelerated / integrity-checked
   * ⚠️ on cloudflare: max upload size 100 MiB
-* ⚠️ uploading small files is slow; `2.7` files per sec (copyparty does `87`/sec), tested locally with [official container](https://manual.seafile.com/docker/deploy_seafile_with_docker/)
+* ⚠️ uploading small files is slow; `4.7` files per sec (copyparty does `670`/sec, 140x faster)
-* ⚠️ no write-only / upload-only folders
 * ⚠️ big folders cannot be zip-downloaded
 * ⚠️ http/webdav only; no ftp, zeroconf
 * ⚠️ less awesome music player
@@ -526,9 +529,11 @@ symbol legend,
 
 ## [dufs](https://github.com/sigoden/dufs)
 * rust; cross-platform (windows, linux, macos)
+* tested locally, v0.43.0 on archlinux (plain binary)
 * ⚠️ uploads not resumable / accelerated / integrity-checked
   * ⚠️ on cloudflare: max upload size 100 MiB
   * ⚠️ across the atlantic, copyparty is 3x faster
+* ⚠️ uploading small files is decent; `97` files per sec (copyparty does `670`/sec, 7x faster)
 * ⚠️ doesn't support crazy filenames
 * ✅ per-url access control (copyparty is per-volume)
 * 🔵 basic but really snappy ui
@@ -571,10 +576,12 @@ symbol legend,
 
 ## [filebrowser](https://github.com/filebrowser/filebrowser)
 * go; cross-platform (windows, linux, mac)
+* tested locally, v2.31.2 on archlinux (plain binary)
 * 🔵 uploads are resumable and segmented
 * 🔵 multiple files are uploaded in parallel, but...
   * ⚠️ big files are not accelerated (copyparty is 5x faster across the atlantic)
 * ⚠️ uploads are not integrity-checked
+* ⚠️ uploading small files is decent; `69` files per sec (copyparty does `670`/sec, 9x faster)
 * ⚠️ http only; no webdav / ftp / zeroconf
 * ⚠️ doesn't support crazy filenames
 * ⚠️ no directory tree nav
@@ -585,6 +592,7 @@ symbol legend,
 * ✅ user signup
 * ✅ command runner / remote shell
 * ✅ more efficient; can handle around twice as much simultaneous traffic
+* note: keep an eye on [gtsteffaniak's fork](https://github.com/gtsteffaniak/filebrowser)
 
 ## [filegator](https://github.com/filegator/filegator)
 * php; cross-platform (windows, linux, mac)
@@ -612,6 +620,7 @@ symbol legend,
 * ⚠️ no zeroconf (mdns/ssdp)
 * ⚠️ impractical directory URLs
 * ⚠️ AGPL licensed
+* 🔵 uploading small files is fast; `340` files per sec (copyparty does `670`/sec)
 * 🔵 ftp, ftps, webdav
 * ✅ sftp server
 * ✅ settings gui

pyproject.toml

@@ -52,6 +52,7 @@ ftpd = ["pyftpdlib"]
 ftps = ["pyftpdlib", "pyopenssl"]
 tftpd = ["partftpy>=0.4.0"]
 pwhash = ["argon2-cffi"]
+zeromq = ["pyzmq"]
 
 [project.scripts]
 copyparty = "copyparty.__main__:main"

scripts/deps-docker/Dockerfile

@@ -1,11 +1,11 @@
 FROM    alpine:3.18
 WORKDIR /z
 ENV     ver_asmcrypto=c72492f4a66e17a0e5dd8ad7874de354f3ccdaa5 \
-        ver_hashwasm=4.10.0 \
+        ver_hashwasm=4.12.0 \
         ver_marked=4.3.0 \
-        ver_dompf=3.1.7 \
+        ver_dompf=3.2.3 \
         ver_mde=2.18.0 \
-        ver_codemirror=5.65.16 \
+        ver_codemirror=5.65.18 \
         ver_fontawesome=5.13.0 \
         ver_prism=1.29.0 \
         ver_zopfli=1.0.3
@@ -16,7 +16,7 @@ ENV     ver_asmcrypto=c72492f4a66e17a0e5dd8ad7874de354f3ccdaa5 \
 # https://github.com/codemirror/codemirror5/releases
 # https://github.com/cure53/DOMPurify/releases
 # https://github.com/Daninet/hash-wasm/releases
-# https://github.com/openpgpjs/asmcrypto.js
+# https://github.com/openpgpjs/asmcrypto.js/commits/main/
 # https://github.com/google/zopfli/tags
 
 

scripts/docker/Dockerfile.ac

@@ -9,7 +9,7 @@ ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk --no-cache add !pyc \
             tzdata wget \
-            py3-jinja2 py3-argon2-cffi py3-pillow \
+            py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
             ffmpeg
 
 COPY    i/dist/copyparty-sfx.py innvikler.sh ./

scripts/docker/Dockerfile.dj

@@ -12,7 +12,8 @@ COPY    i/bin/mtag/audio-bpm.py /mtag/
 COPY    i/bin/mtag/audio-key.py /mtag/
 RUN     apk add -U !pyc \
             tzdata wget \
-            py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
+            py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
+            py3-pip py3-cffi \
             ffmpeg \
             vips-jxl vips-heif vips-poppler vips-magick \
             py3-numpy fftw libsndfile \

scripts/docker/Dockerfile.iv

@@ -9,7 +9,8 @@ ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk add -U !pyc \
             tzdata wget \
-            py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
+            py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
+            py3-pip py3-cffi \
             ffmpeg \
             vips-jxl vips-heif vips-poppler vips-magick \
         && apk add -t .bd \

scripts/genlic.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+
+import re, os, sys, codecs
+
+outfile = os.path.realpath(sys.argv[1])
+
+os.chdir(os.path.dirname(__file__))
+
+with open("../docs/lics.txt", "rb") as f:
+    s = f.read().decode("utf-8").rstrip("\n") + "\n\n\n\n"
+    s = re.sub("\nC: ", "\nCopyright (c) ", s)
+    s = re.sub("\nL: ", "\nLicense: ", s)
+    ret = s.split("\n")
+
+lics = [
+    "MIT License",
+    "BSD 2-Clause License",
+    "BSD 3-Clause License",
+    "SIL Open Font License v1.1",
+]
+
+for n, lic in enumerate(lics, 1):
+    with open("lics/%d.r13" % (n,), "rb") as f:
+        s = f.read().decode("utf-8")
+        s = codecs.decode(s, "rot_13")
+        s = "\n--- %s ---\n\n%s" % (lic, s)
+        ret.extend(s.split("\n"))
+
+for n, ln in enumerate(ret):
+    if not ln.startswith("--- "):
+        continue
+    pad = " " * ((80 - len(ln)) // 2)
+    ln = "%s\033[07m%s\033[0m" % (pad, ln)
+    ret[n] = ln
+
+ret.append("")
+ret.append("")
+
+with open(outfile, "wb") as f:
+    f.write(("\n".join(ret)).encode("utf-8"))

scripts/genlic.sh

@@ -1,54 +0,0 @@
-#!/bin/bash
-set -e
-
-outfile="$($(command -v realpath || command -v grealpath) "$1")"
-
-[ -e genlic.sh ] || cd scripts
-[ -e genlic.sh ]
-
-f=../build/mit.txt
-[ -e $f ] ||
-	curl https://opensource.org/licenses/MIT |
-	awk '/div>/{o=0}o>1;o{o++}/;COPYRIGHT HOLDER/{o=1}' |
-	awk '{gsub(/<[^>]+>/,"")};1' >$f
-
-f=../build/isc.txt
-[ -e $f ] ||
-	curl https://opensource.org/licenses/ISC |
-	awk '/div>/{o=0}o>2;o{o++}/;OWNER/{o=1}' |
-	awk '{gsub(/<[^>]+>/,"")};/./{b=0}!/./{b++}b>1{next}1' >$f
-
-f=../build/2bsd.txt
-[ -e $f ] ||
-	curl https://opensource.org/licenses/BSD-2-Clause |
-	awk '/div>/{o=0}o>1;o{o++}/HOLDER/{o=1}' |
-	awk '{gsub(/<[^>]+>/,"")};1' >$f
-
-f=../build/3bsd.txt
-[ -e $f ] ||
-	curl https://opensource.org/licenses/BSD-3-Clause |
-	awk '/div>/{o=0}o>1;o{o++}/HOLDER/{o=1}' |
-	awk '{gsub(/<[^>]+>/,"")};1' >$f
-
-f=../build/ofl.txt
-[ -e $f ] ||
-	curl https://opensource.org/licenses/OFL-1.1 |
-	awk '/PREAMBLE/{o=1}/sil\.org/{o=0}!o{next}/./{printf "%s ",$0;next}{print"\n"}' |
-	awk '{gsub(/<[^>]+>/,"");gsub(/^\s+/,"");gsub(/&amp;/,"\\&")}/./{b=0}!/./{b++}b>1{next}1' >$f
-
-(sed -r 's/^L: /License: /;s/^C: /Copyright (c) /' <../docs/lics.txt
-printf '\n\n--- MIT License ---\n\n'; cat ../build/mit.txt
-printf '\n\n--- ISC License ---\n\n'; cat ../build/isc.txt
-printf '\n\n--- BSD 2-Clause License ---\n\n'; cat ../build/2bsd.txt
-printf '\n\n--- BSD 3-Clause License ---\n\n'; cat ../build/3bsd.txt
-printf '\n\n--- SIL Open Font License v1.1 ---\n\n'; cat ../build/ofl.txt
-) |
-while IFS= read -r x; do
-	[ "${x:0:4}" = "--- " ] || {
-		printf '%s\n' "$x"
-		continue
-	}
-	n=${#x}
-	p=$(( (80-n)/2 ))
-	printf "%${p}s\033[07m%s\033[0m\n" "" "$x"
-done > "$outfile"

scripts/help2html.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 
 import re
+import socket
 import subprocess as sp
 
 
@@ -25,13 +26,21 @@ def readclip():
             return sp.check_output(cmd.split()).decode("utf-8")
         except:
             pass
+    raise Exception("need one of these: xsel xclip pbpaste")
 
 
 def cnv(src):
+    hostname = str(socket.gethostname()).split(".")[0]
+
+    yield '<!DOCTYPE html>'
     yield '<html style="background:#222;color:#fff"><body>'
     skip_sfx = False
     in_sfx = 0
     in_salt = 0
+    in_name = 0
+    in_cores = 0
+    in_hash_mt = False
+    in_th_ram_max = 0
 
     while True:
         ln = next(src)
@@ -43,6 +52,7 @@ def cnv(src):
 
     for ln in src:
         ln = ln.rstrip()
+        t = ln
         if re.search(r"^<font[^>]+>copyparty v[0-9]", ln):
             in_sfx = 3
         if in_sfx:
@@ -56,11 +66,39 @@ def cnv(src):
             in_salt = 3
         if in_salt:
             in_salt -= 1
-            t = ln
             ln = re.sub(r">[0-9a-zA-Z/+]{24}<", ">24-character-autogenerated<", ln)
             ln = re.sub(r">[0-9a-zA-Z/+]{40}<", ">40-character-autogenerated<", ln)
             if t != ln:
                 in_salt = 0
+        if "--name TXT" in ln:
+            in_name = 3
+        if in_name:
+            in_name -= 1
+            ln = ln.replace(">" + hostname + "<", ">hostname<")
+            if t != ln:
+                in_name = 0
+        if "--hash-mt CORES" in ln:
+            in_cores = 3
+            in_hash_mt = True
+        if "--mtag-mt CORES" in ln or "--th-mt CORES" in ln:
+            in_cores = 3
+        if in_cores:
+            in_cores -= 1
+            zs = ">numCores"
+            if in_hash_mt:
+                zs += " if 5 or less"
+            ln = re.sub(r">[0-9]{1,2}<", zs + "<", ln)
+            if t != ln:
+                in_cores = 0
+                in_hash_mt = False
+        if "--th-ram-max GB" in ln:
+            in_th_ram_max = 3
+        if in_th_ram_max:
+            in_th_ram_max -= 1
+            ln = re.sub(r">[0-9]{1,2}\.[0-9]<", ">dynamic<", ln)
+            if t != ln:
+                in_th_ram_max = 0
+
         ln = ln.replace(">/home/ed/", ">~/")
         if ln.startswith("0" * 20):
             skip_sfx = True

scripts/help2txt.sh

@@ -6,6 +6,10 @@ s`/home/ed/`~/`;
 s/uuid:[0-9a-f-]{36}/autogenerated/;
 s/(-salt SALT.*default: )[0-9a-zA-Z/+]{24}\)/\124-character-autogenerated)/;
 s/(-salt SALT.*default: )[0-9a-zA-Z/+]{40}\)/\140-character-autogenerated)/;
+s/(--name TXT.*default: )[^)]+/\1hostname/;
+s/(--hash-mt CORES.*default: )[0-9]+/\1numCores if 5 or less/;
+s/(--mtag-mt|th-mt)( CORES.*default: )[0-9]+/\1\2numCores/;
+s/(--th-ram-max GB.*default: )[0-9\.]+/\1dynamic/;
 ' | awk '
 /^copyparty/{a=1} !a{next}
 /^0{20}/{b=1} b&&/^copyparty v[0-9]+\./{s=3}

scripts/lics/1.r13

@@ -0,0 +1,5 @@
+Crezvffvba vf urerol tenagrq, serr bs punetr, gb nal crefba bognvavat n pbcl bs guvf fbsgjner naq nffbpvngrq qbphzragngvba svyrf (gur "Fbsgjner"), gb qrny va gur Fbsgjner jvgubhg erfgevpgvba, vapyhqvat jvgubhg yvzvgngvba gur evtugf gb hfr, pbcl, zbqvsl, zretr, choyvfu, qvfgevohgr, fhoyvprafr, naq/be fryy pbcvrf bs gur Fbsgjner, naq gb crezvg crefbaf gb jubz gur Fbsgjner vf sheavfurq gb qb fb, fhowrpg gb gur sbyybjvat pbaqvgvbaf:
+
+Gur nobir pbclevtug abgvpr naq guvf crezvffvba abgvpr funyy or vapyhqrq va nyy pbcvrf be fhofgnagvny cbegvbaf bs gur Fbsgjner.
+
+GUR FBSGJNER VF CEBIVQRQ "NF VF", JVGUBHG JNEENAGL BS NAL XVAQ, RKCERFF BE VZCYVRQ, VAPYHQVAT OHG ABG YVZVGRQ GB GUR JNEENAGVRF BS ZREPUNAGNOVYVGL, SVGARFF SBE N CNEGVPHYNE CHECBFR NAQ ABAVASEVATRZRAG. VA AB RIRAG FUNYY GUR NHGUBEF BE PBCLEVTUG UBYQREF OR YVNOYR SBE NAL PYNVZ, QNZNTRF BE BGURE YVNOVYVGL, JURGURE VA NA NPGVBA BS PBAGENPG, GBEG BE BGUREJVFR, NEVFVAT SEBZ, BHG BS BE VA PBAARPGVBA JVGU GUR FBSGJNER BE GUR HFR BE BGURE QRNYVATF VA GUR FBSGJNER.

scripts/lics/2.r13

@@ -0,0 +1,7 @@
+Erqvfgevohgvba naq hfr va fbhepr naq ovanel sbezf, jvgu be jvgubhg zbqvsvpngvba, ner crezvggrq cebivqrq gung gur sbyybjvat pbaqvgvbaf ner zrg:
+
+1. Erqvfgevohgvbaf bs fbhepr pbqr zhfg ergnva gur nobir pbclevtug abgvpr, guvf yvfg bs pbaqvgvbaf naq gur sbyybjvat qvfpynvzre.
+
+2. Erqvfgevohgvbaf va ovanel sbez zhfg ercebqhpr gur nobir pbclevtug abgvpr, guvf yvfg bs pbaqvgvbaf naq gur sbyybjvat qvfpynvzre va gur qbphzragngvba naq/be bgure zngrevnyf cebivqrq jvgu gur qvfgevohgvba.
+
+GUVF FBSGJNER VF CEBIVQRQ OL GUR PBCLEVTUG UBYQREF NAQ PBAGEVOHGBEF "NF VF" NAQ NAL RKCERFF BE VZCYVRQ JNEENAGVRF, VAPYHQVAT, OHG ABG YVZVGRQ GB, GUR VZCYVRQ JNEENAGVRF BS ZREPUNAGNOVYVGL NAQ SVGARFF SBE N CNEGVPHYNE CHECBFR NER QVFPYNVZRQ. VA AB RIRAG FUNYY GUR PBCLEVTUG UBYQRE BE PBAGEVOHGBEF OR YVNOYR SBE NAL QVERPG, VAQVERPG, VAPVQRAGNY, FCRPVNY, RKRZCYNEL, BE PBAFRDHRAGVNY QNZNTRF (VAPYHQVAT, OHG ABG YVZVGRQ GB, CEBPHERZRAG BS FHOFGVGHGR TBBQF BE FREIVPRF; YBFF BS HFR, QNGN, BE CEBSVGF; BE OHFVARFF VAGREEHCGVBA) UBJRIRE PNHFRQ NAQ BA NAL GURBEL BS YVNOVYVGL, JURGURE VA PBAGENPG, FGEVPG YVNOVYVGL, BE GBEG (VAPYHQVAT ARTYVTRAPR BE BGUREJVFR) NEVFVAT VA NAL JNL BHG BS GUR HFR BS GUVF FBSGJNER, RIRA VS NQIVFRQ BS GUR CBFFVOVYVGL BS FHPU QNZNTR.

scripts/lics/3.r13

@@ -0,0 +1,9 @@
+Erqvfgevohgvba naq hfr va fbhepr naq ovanel sbezf, jvgu be jvgubhg zbqvsvpngvba, ner crezvggrq cebivqrq gung gur sbyybjvat pbaqvgvbaf ner zrg:
+
+1. Erqvfgevohgvbaf bs fbhepr pbqr zhfg ergnva gur nobir pbclevtug abgvpr, guvf yvfg bs pbaqvgvbaf naq gur sbyybjvat qvfpynvzre.
+
+2. Erqvfgevohgvbaf va ovanel sbez zhfg ercebqhpr gur nobir pbclevtug abgvpr, guvf yvfg bs pbaqvgvbaf naq gur sbyybjvat qvfpynvzre va gur qbphzragngvba naq/be bgure zngrevnyf cebivqrq jvgu gur qvfgevohgvba.
+
+3. Arvgure gur anzr bs gur pbclevtug ubyqre abe gur anzrf bs vgf pbagevohgbef znl or hfrq gb raqbefr be cebzbgr cebqhpgf qrevirq sebz guvf fbsgjner jvgubhg fcrpvsvp cevbe jevggra crezvffvba.
+
+GUVF FBSGJNER VF CEBIVQRQ OL GUR PBCLEVTUG UBYQREF NAQ PBAGEVOHGBEF "NF VF" NAQ NAL RKCERFF BE VZCYVRQ JNEENAGVRF, VAPYHQVAT, OHG ABG YVZVGRQ GB, GUR VZCYVRQ JNEENAGVRF BS ZREPUNAGNOVYVGL NAQ SVGARFF SBE N CNEGVPHYNE CHECBFR NER QVFPYNVZRQ. VA AB RIRAG FUNYY GUR PBCLEVTUG UBYQRE BE PBAGEVOHGBEF OR YVNOYR SBE NAL QVERPG, VAQVERPG, VAPVQRAGNY, FCRPVNY, RKRZCYNEL, BE PBAFRDHRAGVNY QNZNTRF (VAPYHQVAT, OHG ABG YVZVGRQ GB, CEBPHERZRAG BS FHOFGVGHGR TBBQF BE FREIVPRF; YBFF BS HFR, QNGN, BE CEBSVGF; BE OHFVARFF VAGREEHCGVBA) UBJRIRE PNHFRQ NAQ BA NAL GURBEL BS YVNOVYVGL, JURGURE VA PBAGENPG, FGEVPG YVNOVYVGL, BE GBEG (VAPYHQVAT ARTYVTRAPR BE BGUREJVFR) NEVFVAT VA NAL JNL BHG BS GUR HFR BS GUVF FBSGJNER, RIRA VS NQIVFRQ BS GUR CBFFVOVYVGL BS FHPU QNZNTR.

scripts/lics/4.r13

@@ -0,0 +1,39 @@
+CERNZOYR
+
+Gur tbnyf bs gur Bcra Sbag Yvprafr (BSY) ner gb fgvzhyngr jbeyqjvqr qrirybczrag bs pbyynobengvir sbag cebwrpgf, gb fhccbeg gur sbag perngvba rssbegf bs npnqrzvp naq yvathvfgvp pbzzhavgvrf, naq gb cebivqr n serr naq bcra senzrjbex va juvpu sbagf znl or funerq naq vzcebirq va cnegarefuvc jvgu bguref.
+
+Gur BSY nyybjf gur yvprafrq sbagf gb or hfrq, fghqvrq, zbqvsvrq naq erqvfgevohgrq serryl nf ybat nf gurl ner abg fbyq ol gurzfryirf. Gur sbagf, vapyhqvat nal qrevingvir jbexf, pna or ohaqyrq, rzorqqrq, erqvfgevohgrq naq/be fbyq jvgu nal fbsgjner cebivqrq gung nal erfreirq anzrf ner abg hfrq ol qrevingvir jbexf. Gur sbagf naq qrevingvirf, ubjrire, pnaabg or eryrnfrq haqre nal bgure glcr bs yvprafr. Gur erdhverzrag sbe sbagf gb erznva haqre guvf yvprafr qbrf abg nccyl gb nal qbphzrag perngrq hfvat gur sbagf be gurve qrevingvirf.
+
+QRSVAVGVBAF
+
+"Sbag Fbsgjner" ersref gb gur frg bs svyrf eryrnfrq ol gur Pbclevtug Ubyqre(f) haqre guvf yvprafr naq pyrneyl znexrq nf fhpu. Guvf znl vapyhqr fbhepr svyrf, ohvyq fpevcgf naq qbphzragngvba.
+
+"Erfreirq Sbag Anzr" ersref gb nal anzrf fcrpvsvrq nf fhpu nsgre gur pbclevtug fgngrzrag(f).
+
+"Bevtvany Irefvba" ersref gb gur pbyyrpgvba bs Sbag Fbsgjner pbzcbaragf nf qvfgevohgrq ol gur Pbclevtug Ubyqre(f).
+
+"Zbqvsvrq Irefvba" ersref gb nal qrevingvir znqr ol nqqvat gb, qryrgvat, be fhofgvghgvat - va cneg be va jubyr - nal bs gur pbzcbaragf bs gur Bevtvany Irefvba, ol punatvat sbezngf be ol cbegvat gur Sbag Fbsgjner gb n arj raivebazrag.
+
+"Nhgube" ersref gb nal qrfvtare, ratvarre, cebtenzzre, grpuavpny jevgre be bgure crefba jub pbagevohgrq gb gur Sbag Fbsgjner.
+
+CREZVFFVBA & PBAQVGVBAF
+
+Crezvffvba vf urerol tenagrq, serr bs punetr, gb nal crefba bognvavat n pbcl bs gur Sbag Fbsgjner, gb hfr, fghql, pbcl, zretr, rzorq, zbqvsl, erqvfgevohgr, naq fryy zbqvsvrq naq hazbqvsvrq pbcvrf bs gur Sbag Fbsgjner, fhowrpg gb gur sbyybjvat pbaqvgvbaf:
+
+1) Arvgure gur Sbag Fbsgjner abe nal bs vgf vaqvivqhny pbzcbaragf, va Bevtvany be Zbqvsvrq Irefvbaf, znl or fbyq ol vgfrys.
+
+2) Bevtvany be Zbqvsvrq Irefvbaf bs gur Sbag Fbsgjner znl or ohaqyrq, erqvfgevohgrq naq/be fbyq jvgu nal fbsgjner, cebivqrq gung rnpu pbcl pbagnvaf gur nobir pbclevtug abgvpr naq guvf yvprafr. Gurfr pna or vapyhqrq rvgure nf fgnaq-nybar grkg svyrf, uhzna-ernqnoyr urnqref be va gur nccebcevngr znpuvar-ernqnoyr zrgnqngn svryqf jvguva grkg be ovanel svyrf nf ybat nf gubfr svryqf pna or rnfvyl ivrjrq ol gur hfre.
+
+3) Ab Zbqvsvrq Irefvba bs gur Sbag Fbsgjner znl hfr gur Erfreirq Sbag Anzr(f) hayrff rkcyvpvg jevggra crezvffvba vf tenagrq ol gur pbeerfcbaqvat Pbclevtug Ubyqre. Guvf erfgevpgvba bayl nccyvrf gb gur cevznel sbag anzr nf cerfragrq gb gur hfref.
+
+4) Gur anzr(f) bs gur Pbclevtug Ubyqre(f) be gur Nhgube(f) bs gur Sbag Fbsgjner funyy abg or hfrq gb cebzbgr, raqbefr be nqiregvfr nal Zbqvsvrq Irefvba, rkprcg gb npxabjyrqtr gur pbagevohgvba(f) bs gur Pbclevtug Ubyqre(f) naq gur Nhgube(f) be jvgu gurve rkcyvpvg jevggra crezvffvba.
+
+5) Gur Sbag Fbsgjner, zbqvsvrq be hazbqvsvrq, va cneg be va jubyr, zhfg or qvfgevohgrq ragveryl haqre guvf yvprafr, naq zhfg abg or qvfgevohgrq haqre nal bgure yvprafr. Gur erdhverzrag sbe sbagf gb erznva haqre guvf yvprafr qbrf abg nccyl gb nal qbphzrag perngrq hfvat gur Sbag Fbsgjner.
+
+GREZVANGVBA
+
+Guvf yvprafr orpbzrf ahyy naq ibvq vs nal bs gur nobir pbaqvgvbaf ner abg zrg.
+
+QVFPYNVZRE
+
+GUR SBAG FBSGJNER VF CEBIVQRQ "NF VF", JVGUBHG JNEENAGL BS NAL XVAQ, RKCERFF BE VZCYVRQ, VAPYHQVAT OHG ABG YVZVGRQ GB NAL JNEENAGVRF BS ZREPUNAGNOVYVGL, SVGARFF SBE N CNEGVPHYNE CHECBFR NAQ ABAVASEVATRZRAG BS PBCLEVTUG, CNGRAG, GENQRZNEX, BE BGURE EVTUG. VA AB RIRAG FUNYY GUR PBCLEVTUG UBYQRE OR YVNOYR SBE NAL PYNVZ, QNZNTRF BE BGURE YVNOVYVGL, VAPYHQVAT NAL TRARENY, FCRPVNY, VAQVERPG, VAPVQRAGNY, BE PBAFRDHRAGVNY QNZNTRF, JURGURE VA NA NPGVBA BS PBAGENPG, GBEG BE BGUREJVFR, NEVFVAT SEBZ, BHG BS GUR HFR BE VANOVYVGL GB HFR GUR SBAG FBSGJNER BE SEBZ BGURE QRNYVATF VA GUR SBAG FBSGJNER.

scripts/lics/README.md

@@ -0,0 +1,3 @@
+these are foss licenses in rot13 so scanners don't think copyparty isn't mit
+
+1=mit 2=2bsd 3=3bsd 4=ofl

scripts/lics/rot.py

@@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+
+import os, codecs
+
+for fn in os.listdir("."):
+    if not fn.endswith(".txt"):
+        continue
+    with open(fn, "rb") as f:
+        s = f.read().decode("utf-8")
+    b = codecs.encode(s, "rot_13").encode("utf-8")
+    with open(fn.replace("txt", "r13"), "wb") as f:
+        f.write(b)

scripts/make-pypi-release.sh

@@ -100,14 +100,13 @@ load_env || {
 # cleanup
 rm -rf unt build/pypi
 
-# grab licenses
+# generate license list
-scripts/genlic.sh copyparty/res/COPYING.txt
+scripts/genlic.py copyparty/res/COPYING.txt
 
 # clean-ish packaging env
 rm -rf build/pypi
 mkdir -p build/pypi
 cp -pR pyproject.toml README.md LICENSE copyparty contrib bin scripts/strip_hints build/pypi/
-tar -c docs/lics.txt scripts/genlic.sh build/*.txt | tar -xC build/pypi/
 cd build/pypi
 
 # delete junk

scripts/make-sfx.sh

@@ -308,8 +308,7 @@ necho() {
 	# remove type hints before build instead
 	(cd copyparty; PYTHONPATH="..:$PYTHONPATH" "$pybin" ../../scripts/strip_hints/a.py; rm uh)
 
-	licfile=$(realpath copyparty/res/COPYING.txt)
+	(cd ../scripts; ./genlic.py ../copyparty/res/COPYING.txt)
-	(cd ../scripts; ./genlic.sh "$licfile")
 }
 
 [ ! -e copyparty/web/deps/mini-fa.woff ] && [ $dl_wd ] && {

scripts/make-tgz-release.sh

@@ -64,7 +64,7 @@ git archive hovudstraum | tar -xC "$rls_dir"
 echo ">>> export untracked deps"
 tar -c copyparty/web/deps | tar -xC "$rls_dir"
 
-scripts/genlic.sh "$rls_dir/copyparty/res/COPYING.txt"
+scripts/genlic.py "$rls_dir/copyparty/res/COPYING.txt"
 
 cd "$rls_dir"
 find -type d -exec chmod 755 '{}' \+

scripts/pyinstaller/deps.sha512

@@ -1,7 +1,7 @@
 f117016b1e6a7d7e745db30d3e67f1acf7957c443a0dd301b6c5e10b8368f2aa4db6be9782d2d3f84beadd139bfeef4982e40f21ca5d9065cb794eeb0e473e82  altgraph-0.17.4-py2.py3-none-any.whl
 6a624018f30da375581d5751eca0080edbbe37f102f643f856279fcfded3a4379fd1b6fb0661cdb2e72bbbbc726ca714a1f5990cc348df365db62bc53e4c4503  Git-2.45.2-32-bit.exe
 17ce52ba50692a9d964f57a23ac163fb74c77fdeb2ca988a6d439ae1fe91955ff43730c073af97a7b3223093ffea3479a996b9b50ee7fba0869247a56f74baa6  pefile-2023.2.7-py3-none-any.whl
-749a473646c6d4c7939989649733d4c7699fd1c359c27046bf5bc9c070d1a4b8b986bbc65f60d7da725baf16dbfdd75a4c2f5bb8335f2cb5685073f5fee5c2d1  pywin32_ctypes-0.2.2-py3-none-any.whl
+b297ff66ec50cf5a1abcf07d6ac949644c5150ba094ffac974c5d27c81574c3e97ed814a47547f4b03a4c83ea0fb8f026433fca06a3f08e32742dc5c024f3d07  pywin32_ctypes-0.2.3-py3-none-any.whl
 085d39ef4426aa5f097fbc484595becc16e61ca23fc7da4d2a8bba540a3b82e789e390b176c7151bdc67d01735cce22b1562cdb2e31273225a2d3e275851a4ad  setuptools-70.3.0-py3-none-any.whl
 360a141928f4a7ec18a994602cbb28bbf8b5cc7c077a06ac76b54b12fa769ed95ca0333a5cf728923a8e0baeb5cc4d5e73e5b3de2666beb05eb477d8ae719093  upx-4.2.4-win32.zip
 # win7
@@ -23,11 +23,11 @@ ac96786e5d35882e0c5b724794329c9125c2b86ae7847f17acfc49f0d294312c6afc1c3f248655de
 # win10
 0a2cd4cadf0395f0374974cd2bc2407e5cc65c111275acdffb6ecc5a2026eee9e1bb3da528b35c7f0ff4b64563a74857d5c2149051e281cc09ebd0d1968be9aa  en-us_windows_10_enterprise_ltsc_2021_x64_dvd_d289cf96.iso
 16cc0c58b5df6c7040893089f3eb29c074aed61d76dae6cd628d8a89a05f6223ac5d7f3f709a12417c147594a87a94cc808d1e04a6f1e407cc41f7c9f47790d1  virtio-win-0.1.248.iso
-d1420c8417fad7888766dd26b9706a87c63e8f33dceeb8e26d0056d5127b0b3ed9272e44b4b761132d4b3320327252eab1696520488ca64c25958896b41f547b  jinja2-3.1.4-py3-none-any.whl
+18b9e8cfa682da51da1b682612652030bd7f10e4a1d5ea5220ab32bde734b0e6fe1c7dbd903ac37928c0171fd45d5ca602952054de40a4e55e9ed596279516b5  jinja2-3.1.5-py3-none-any.whl
-8e6847bcde75a2736be0214731f834bc1b5854238d703351e68bc4e74d38404b212b8568565ae22c844189e466d3fbe6024836351cb69ffb1824131387644fef  MarkupSafe-2.1.5-cp312-cp312-win_amd64.whl
+6df21f0da408a89f6504417c7cdf9aaafe4ed88cfa13e9b8fa8414f604c0401f885a04bbad0484dc51a29284af5d1548e33c6cc6bfb9896d9992c1b1074f332d  MarkupSafe-3.0.2-cp312-cp312-win_amd64.whl
 8a6e2b13a2ec4ef914a5d62aad3db6464d45e525a82e07f6051ed10474eae959069e165dba011aefb8207cdfd55391d73d6f06362c7eb247b08763106709526e  mutagen-1.47.0-py3-none-any.whl
 0203ec2551c4836696cfab0b2c9fff603352f03fa36e7476e2e1ca7ec57a3a0c24bd791fcd92f342bf817f0887854d9f072e0271c643de4b313d8c9569ba8813  packaging-24.1-py3-none-any.whl
-2be320b4191f208cdd6af183c77ba2cf460ea52164ee45ac3ff17d6dfa57acd9deff016636c2dd42a21f4f6af977d5f72df7dacf599bebcf41757272354d14c1  pillow-10.4.0-cp312-cp312-win_amd64.whl
+12d7921dc7dfd8a4b0ea0fa2bae8f1354fcdd59ece3d7f4e075aed631f9ba791dc142c70b1ccd1e6287c43139df1db26bd57a7a217c8da3a77326036495cdb57  pillow-11.1.0-cp312-cp312-win_amd64.whl
-896ddddbd4b85e86e0600cb65eb4c07fbc7f3802d47e7f660411e20b5500831469b97ed4770f25820f4e75cbfac40308da624fd86d4f62e578149d5c276a9cde  pyinstaller-6.10.0-py3-none-win_amd64.whl
+f0463895e9aee97f31a2003323de235fed1b26289766dc0837261e3f4a594a31162b69e9adbb0e9a31e2e2d4b5f25c762ed1669553df7dc89a8ba4f85d297873  pyinstaller-6.11.1-py3-none-win_amd64.whl
-873781decaeef07f6a79b0ed8b9f35f3fa534a1ea0d866991e40278a10818fa5b60c70b0d5828971b045364f1099694cd1e5d5d60d480acb93fcfbfbced4a09e  pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
+d550a0a14428386945533de2220c4c2e37c0c890fc51a600f626c6ca90a32d39572c121ec04c157ba3a8d6601cb021f8433d871b5c562a3d342c804fffec90c1  pyinstaller_hooks_contrib-2024.11-py3-none-any.whl
-912b710007c7b29f29c0097aff8f825412166eed7777a7cef135b14316e8fff31b5df56d26d835d8ca090468cc0e914730f201a56caa3dd6dbef2f91088942b1  python-3.12.7-amd64.exe
+0f623c9ab52d050283e97a986ba626d86b04cd02fa7ffdf352740576940b142b264709abadb5d875c90f625b28103d7210b900e0d77f12c1c140108bd2a159aa  python-3.12.8-amd64.exe