v1.9.30

previously it only did volume abspaths

README.md

@@ -26,6 +26,7 @@ turn almost any device into a file server with resumable uploads/downloads using
 * [FAQ](#FAQ) - "frequently" asked questions
 * [accounts and volumes](#accounts-and-volumes) - per-folder, per-user permissions
     * [shadowing](#shadowing) - hiding specific subfolders
+    * [dotfiles](#dotfiles) - unix-style hidden files/folders
 * [the browser](#the-browser) - accessing a copyparty server using a web-browser
     * [tabs](#tabs) - the main tabs in the ui
     * [hotkeys](#hotkeys) - the browser has the following hotkeys
@@ -75,7 +76,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [prometheus](#prometheus) - metrics/stats can be enabled
 * [packages](#packages) - the party might be closer than you think
     * [arch package](#arch-package) - now [available on aur](https://aur.archlinux.org/packages/copyparty) maintained by [@icxes](https://github.com/icxes)
-    * [fedora package](#fedora-package) - now [available on copr-pypi](https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/)
+    * [fedora package](#fedora-package) - currently **NOT** available on [copr-pypi](https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/)
     * [nix package](#nix-package) - `nix profile install github:9001/copyparty`
     * [nixos module](#nixos-module)
 * [browser support](#browser-support) - TLDR: yes
@@ -112,7 +113,7 @@ just run **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/
 
 * or install through pypi: `python3 -m pip install --user -U copyparty`
 * or if you cannot install python, you can use [copyparty.exe](#copypartyexe) instead
-* or install [on arch](#arch-package) ╱ [on fedora](#fedora-package) ╱ [on NixOS](#nixos-module) ╱ [through nix](#nix-package)
+* or install [on arch](#arch-package) ╱ [on NixOS](#nixos-module) ╱ [through nix](#nix-package)
 * or if you are on android, [install copyparty in termux](#install-on-android)
 * or if you prefer to [use docker](./scripts/docker/) 🐋 you can do that too
   * docker has all deps built-in, so skip this step:
@@ -148,9 +149,10 @@ you may also want these, especially on servers:
 
 * [contrib/systemd/copyparty.service](contrib/systemd/copyparty.service) to run copyparty as a systemd service (see guide inside)
 * [contrib/systemd/prisonparty.service](contrib/systemd/prisonparty.service) to run it in a chroot (for extra security)
+* [contrib/openrc/copyparty](contrib/openrc/copyparty) to run copyparty on Alpine / Gentoo
 * [contrib/rc/copyparty](contrib/rc/copyparty) to run copyparty on FreeBSD
-* [contrib/nginx/copyparty.conf](contrib/nginx/copyparty.conf) to [reverse-proxy](#reverse-proxy) behind nginx (for better https)
 * [nixos module](#nixos-module) to run copyparty on NixOS hosts
+* [contrib/nginx/copyparty.conf](contrib/nginx/copyparty.conf) to [reverse-proxy](#reverse-proxy) behind nginx (for better https)
 
 and remember to open the ports you want; here's a complete example including every feature copyparty has to offer:
 ```
@@ -341,7 +343,7 @@ upgrade notes
   * yes, using [hooks](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/wget.py)
 
 * i want to learn python and/or programming and am considering looking at the copyparty source code in that occasion
-    ```bash
+  * ```bash
      _|  _      __   _  _|_
     (_| (_)     | | (_)  |_
     ```
@@ -367,10 +369,12 @@ permissions:
 * `w` (write): upload files, move files *into* this folder
 * `m` (move): move files/folders *from* this folder
 * `d` (delete): delete files/folders
+* `.` (dots): user can ask to show dotfiles in directory listings
 * `g` (get): only download files, cannot see folder contents or zip/tar
 * `G` (upget): same as `g` except uploaders get to see their own [filekeys](#filekeys) (see `fk` in examples below)
 * `h` (html): same as `g` except folders return their index.html, and filekeys are not necessary for index.html
 * `a` (admin): can see upload time, uploader IPs, config-reload
+* `A` ("all"): same as `rwmda.` (read/write/move/delete/admin/dotfiles)
 
 examples:
 * add accounts named u1, u2, u3 with passwords p1, p2, p3: `-a u1:p1 -a u2:p2 -a u3:p3`
@@ -398,6 +402,17 @@ hiding specific subfolders  by mounting another volume on top of them
 for example `-v /mnt::r -v /var/empty:web/certs:r` mounts the server folder `/mnt` as the webroot, but another volume is mounted at `/web/certs` -- so visitors can only see the contents of `/mnt` and `/mnt/web` (at URLs `/` and `/web`), but not `/mnt/web/certs` because URL `/web/certs` is mapped to `/var/empty`
 
 
+## dotfiles
+
+unix-style hidden files/folders  by starting the name with a dot
+
+anyone can access these if they know the name, but they normally don't appear in directory listings
+
+a client can request to see dotfiles in directory listings if global option `-ed` is specified, or the volume has volflag `dots`, or the user has permission `.`
+
+dotfiles do not appear in search results unless one of the above is true, **and** the global option / volflag `dotsrch` is set
+
+
 # the browser
 
 accessing a copyparty server using a web-browser
@@ -510,7 +525,7 @@ it does static images with Pillow / pyvips / FFmpeg, and uses FFmpeg for video f
 audio files are covnerted into spectrograms using FFmpeg unless you `--no-athumb` (and some FFmpeg builds may need `--th-ff-swr`)
 
 images with the following names (see `--th-covers`) become the thumbnail of the folder they're in: `folder.png`, `folder.jpg`, `cover.png`, `cover.jpg`
-* and, if you enable [file indexing](#file-indexing), all remaining folders will also get thumbnails (as long as they contain any pics at all)
+* and, if you enable [file indexing](#file-indexing), it will also try those names as dotfiles (`.folder.jpg` and so), and then fallback on the first picture in the folder (if it has any pictures at all)
 
 in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked
 * indicated by the audio files having the ▶ icon instead of 💾
@@ -538,7 +553,7 @@ select which type of archive you want in the `[⚙️] config` tab:
 * gzip default level is `3` (0=fast, 9=best), change with `?tar=gz:9`
 * xz default level is `1` (0=fast, 9=best), change with `?tar=xz:9`
 * bz2 default level is `2` (1=fast, 9=best), change with `?tar=bz2:9`
-* hidden files (dotfiles) are excluded unless `-ed`
+* hidden files ([dotfiles](#dotfiles)) are excluded unless account is allowed to list them
   * `up2k.db` and `dir.txt` is always excluded
 * bsdtar supports streaming unzipping: `curl foo?zip=utf8 | bsdtar -xv`
   * good, because copyparty's zip is faster than tar on small files
@@ -724,7 +739,8 @@ some hilights:
 click the `play` link next to an audio file, or copy the link target to [share it](https://a.ocv.me/pub/demo/music/Ubiktune%20-%20SOUNDSHOCK%202%20-%20FM%20FUNK%20TERRROR!!/#af-1fbfba61&t=18) (optionally with a timestamp to start playing from, like that example does)
 
 open the `[🎺]` media-player-settings tab to configure it,
-* switches:
+* "switches":
+  * `[🔀]` shuffles the files inside each folder
   * `[preload]` starts loading the next track when it's about to end, reduces the silence between songs
   * `[full]` does a full preload by downloading the entire next file; good for unreliable connections, bad for slow connections
   * `[~s]` toggles the seekbar waveform display
@@ -734,10 +750,12 @@ open the `[🎺]` media-player-settings tab to configure it,
   * `[art]` shows album art on the lockscreen
   * `[🎯]` keeps the playing song scrolled into view (good when using the player as a taskbar dock)
   * `[⟎]` shrinks the playback controls
-* playback mode:
+* "buttons":
+  * `[uncache]` may fix songs that won't play correctly due to bad files in browser cache
+* "at end of folder":
   * `[loop]` keeps looping the folder
   * `[next]` plays into the next folder
-* transcode:
+* "transcode":
   * `[flac]` converts `flac` and `wav` files into opus
   * `[aac]` converts `aac` and `m4a` files into opus
   * `[oth]` converts all other known formats into opus
@@ -823,6 +841,9 @@ using arguments or config files, or a mix of both:
   * or click the `[reload cfg]` button in the control-panel if the user has `a`/admin in any volume
   * changes to the `[global]` config section requires a restart to take effect
 
+**NB:** as humongous as this readme is, there is also a lot of undocumented features. Run copyparty with `--help` to see all available global options; all of those can be used in the `[global]` section of config files, and everything listed in `--help-flags` can be used in volumes as volflags.
+* if running in docker/podman, try this: `docker run --rm -it copyparty/ac --help`
+
 
 ## zeroconf
 
@@ -1010,6 +1031,8 @@ to save some time,  you can provide a regex pattern for filepaths to only index
 
 similarly, you can fully ignore files/folders using `--no-idx [...]` and `:c,noidx=\.iso$`
 
+* when running on macos, all the usual apple metadata files are excluded by default
+
 if you set `--no-hash [...]` globally, you can enable hashing for specific volumes using flag `:c,nohash=`
 
 ### filesystem guards
@@ -1200,9 +1223,9 @@ replace copyparty passwords with oauth and such
 
 work is [ongoing](https://github.com/9001/copyparty/issues/62) to support authenticating / authorizing users based on a separate authentication proxy, which makes it possible to support oauth, single-sign-on, etc.
 
-it is currently possible to specify `--hdr-au-usr x-username`; copyparty will then skip password validation and blindly trust the username specified in the `X-Username` request header
+it is currently possible to specify `--idp-h-usr x-username`; copyparty will then skip password validation and blindly trust the username specified in the `X-Username` request header
 
-the remaining stuff (accepting user groups through another header, creating volumes on the fly) are still to-do
+the remaining stuff (accepting user groups through another header, creating volumes on the fly) are still to-do; configuration will probably [look like this](./docs/examples/docker/idp/copyparty.conf)
 
 
 ## hiding from google
@@ -1264,8 +1287,8 @@ see the top of [./copyparty/web/browser.css](./copyparty/web/browser.css) where
 * anyone can upload, and receive "secret" links for each upload they do:  
   `python copyparty-sfx.py -e2dsa -v .::wG:c,fk=8`
 
-* anyone can browse, only `kevin` (password `okgo`) can upload/move/delete files:  
-  `python copyparty-sfx.py -e2dsa -a kevin:okgo -v .::r:rwmd,kevin`
+* anyone can browse (`r`), only `kevin` (password `okgo`) can upload/move/delete (`A`) files:  
+  `python copyparty-sfx.py -e2dsa -a kevin:okgo -v .::r:A,kevin`
 
 * read-only music server:  
   `python copyparty-sfx.py -v /mnt/nas/music:/music:r -e2dsa -e2ts --no-robots --force-js --theme 2`
@@ -1365,28 +1388,30 @@ note: the following metrics are counted incorrectly if multiprocessing is enable
 
 the party might be closer than you think
 
+if your distro/OS is not mentioned below, there might be some hints in the [«on servers»](#on-servers) section
+
 
 ## arch package
 
 now [available on aur](https://aur.archlinux.org/packages/copyparty) maintained by [@icxes](https://github.com/icxes)
 
+it comes with a [systemd service](./contrib/package/arch/copyparty.service) and expects to find one or more [config files](./docs/example.conf) in `/etc/copyparty.d/`
+
 
 ## fedora package
 
-now [available on copr-pypi](https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/)  , maintained autonomously -- [track record](https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/package/python-copyparty/) seems OK
+currently **NOT** available on [copr-pypi](https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/)  ,  fedora is having issues with their build servers and won't be fixed for several months
+
+if you previously installed copyparty from copr, you may run one of the following commands to upgrade to a more recent version:
 
 ```bash
-dnf copr enable @copr/PyPI
-dnf install python3-copyparty  # just a minimal install, or... 
-dnf install python3-{copyparty,pillow,argon2-cffi,pyftpdlib,pyOpenSSL} ffmpeg  # with recommended deps
+dnf install https://ocv.me/copyparty/fedora/37/python3-copyparty.fc37.noarch.rpm
+dnf install https://ocv.me/copyparty/fedora/38/python3-copyparty.fc38.noarch.rpm
+dnf install https://ocv.me/copyparty/fedora/39/python3-copyparty.fc39.noarch.rpm
 ```
 
-`ffmpeg` comes from [rpmfusion](https://rpmfusion.org/Configuration#Command_Line_Setup_using_rpm) so it's recommended to enable that (you don't want `ffmpeg-free` since it fails to thumbnail most h264/mkv/mp4 videos)
-
 to run copyparty as a service, use the [systemd service scripts](https://github.com/9001/copyparty/tree/hovudstraum/contrib/systemd), just replace `/usr/bin/python3 /usr/local/bin/copyparty-sfx.py` with `/usr/bin/copyparty`
 
-this *may* also work on RHEL but [I'm not paying IBM to verify that](https://www.jeffgeerling.com/blog/2023/dear-red-hat-are-you-dumb)
-
 
 ## nix package
 
@@ -1515,8 +1540,8 @@ TLDR: yes
 | navpane         |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | image viewer    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | video player    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
-| markdown editor |  -  |  -   | yep  | yep  |  yep  | yep  | yep | yep  |
-| markdown viewer |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
+| markdown editor |  -  |  -   | `*2` | `*2` |  yep  | yep  | yep | yep  |
+| markdown viewer |  -  | `*2` | `*2` | `*2` |  yep  | yep  | yep | yep  |
 | play mp3/m4a    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | play ogg/opus   |  -  |  -   |  -   |  -   |  yep  | yep  | `*3` | yep |
 | **= feature =** | ie6 | ie9  | ie10 | ie11 | ff 52 | c 49 | iOS | Andr |
@@ -1524,6 +1549,7 @@ TLDR: yes
 * internet explorer 6 through 8 behave the same
 * firefox 52 and chrome 49 are the final winxp versions
 * `*1` yes, but extremely slow (ie10: `1 MiB/s`, ie11: `270 KiB/s`)
+* `*2` only able to do plaintext documents (no markdown rendering)
 * `*3` iOS 11 and newer, opus only, and requires FFmpeg on the server
 
 quick summary of more eccentric web-browsers trying to view a directory index:
@@ -1549,10 +1575,12 @@ interact with copyparty using non-browser clients
   * `var xhr = new XMLHttpRequest(); xhr.open('POST', '//127.0.0.1:3923/msgs?raw'); xhr.send('foo');`
 
 * curl/wget: upload some files (post=file, chunk=stdin)
-  * `post(){ curl -F act=bput -F f=@"$1" http://127.0.0.1:3923/?pw=wark;}`  
-    `post movie.mkv`
+  * `post(){ curl -F f=@"$1" http://127.0.0.1:3923/?pw=wark;}`  
+    `post movie.mkv`  (gives HTML in return)
+  * `post(){ curl -F f=@"$1" 'http://127.0.0.1:3923/?want=url&pw=wark';}`  
+    `post movie.mkv`  (gives hotlink in return)
   * `post(){ curl -H pw:wark -H rand:8 -T "$1" http://127.0.0.1:3923/;}`  
-    `post movie.mkv`
+    `post movie.mkv`  (randomized filename)
   * `post(){ wget --header='pw: wark' --post-file="$1" -O- http://127.0.0.1:3923/?raw;}`  
     `post movie.mkv`
   * `chunk(){ curl -H pw:wark -T- http://127.0.0.1:3923/;}`  
@@ -1574,6 +1602,10 @@ interact with copyparty using non-browser clients
 
 * sharex (screenshot utility): see [./contrib/sharex.sxcu](contrib/#sharexsxcu)
 
+* contextlet (web browser integration); see [contrib contextlet](contrib/#send-to-cppcontextletjson)
+
+* [igloo irc](https://iglooirc.com/): Method: `post` Host: `https://you.com/up/?want=url&pw=hunter2` Multipart: `yes` File parameter: `f`
+
 copyparty returns a truncated sha512sum of your PUT/POST as base64; you can generate the same checksum locally to verify uplaods:
 
     b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|tr '+/' '-_'|head -c44;}
@@ -1592,7 +1624,7 @@ the commandline uploader [u2c.py](https://github.com/9001/copyparty/tree/hovudst
 
 alternatively there is [rclone](./docs/rclone.md) which allows for bidirectional sync and is *way* more flexible (stream files straight from sftp/s3/gcs to copyparty, ...), although there is no integrity check and it won't work with files over 100 MiB if copyparty is behind cloudflare
 
-* starting from rclone v1.63 (currently [in beta](https://beta.rclone.org/?filter=latest)), rclone will also be faster than u2c.py
+* starting from rclone v1.63, rclone is faster than u2c.py on low-latency connections
 
 
 ## mount as drive
@@ -1601,7 +1633,7 @@ a remote copyparty server as a local filesystem;  go to the control-panel and cl
 
 alternatively, some alternatives roughly sorted by speed (unreproducible benchmark), best first:
 
-* [rclone-webdav](./docs/rclone.md) (25s), read/WRITE ([v1.63-beta](https://beta.rclone.org/?filter=latest))
+* [rclone-webdav](./docs/rclone.md) (25s), read/WRITE (rclone v1.63 or later)
 * [rclone-http](./docs/rclone.md) (26s), read-only
 * [partyfuse.py](./bin/#partyfusepy) (35s), read-only
 * [rclone-ftp](./docs/rclone.md) (47s), read/WRITE

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "1.11"
-S_BUILD_DT = "2023-11-11"
+S_VERSION = "1.13"
+S_BUILD_DT = "2024-01-24"
 
 """
 u2c.py: upload to copyparty
@@ -560,8 +560,11 @@ def handshake(ar, file, search):
     }
     if search:
         req["srch"] = 1
-    elif ar.dr:
-        req["replace"] = True
+    else:
+        if ar.touch:
+            req["umod"] = True
+        if ar.dr:
+            req["replace"] = True
 
     headers = {"Content-Type": "text/plain"}  # <=1.5.1 compat
     if pw:
@@ -907,12 +910,23 @@ class Ctl(object):
                         dp = os.path.join(top, rd)
                         lnodes = set(os.listdir(dp))
                         bnames = [x for x in ls if x not in lnodes]
-                        if bnames:
-                            vpath = self.ar.url.split("://")[-1].split("/", 1)[-1]
-                            names = [x.decode("utf-8", "replace") for x in bnames]
-                            locs = [vpath + srd + "/" + x for x in names]
-                            print("DELETING ~{0}/#{1}".format(srd, len(names)))
-                            req_ses.post(self.ar.url + "?delete", json=locs)
+                        vpath = self.ar.url.split("://")[-1].split("/", 1)[-1]
+                        names = [x.decode("utf-8", "replace") for x in bnames]
+                        locs = [vpath + srd + "/" + x for x in names]
+                        while locs:
+                            req = locs
+                            while req:
+                                print("DELETING ~%s/#%s" % (srd, len(req)))
+                                r = req_ses.post(self.ar.url + "?delete", json=req)
+                                if r.status_code == 413 and "json 2big" in r.text:
+                                    print(" (delete request too big; slicing...)")
+                                    req = req[: len(req) // 2]
+                                    continue
+                                elif not r:
+                                    t = "delete request failed: %r %s"
+                                    raise Exception(t % (r, r.text))
+                                break
+                            locs = locs[len(req) :]
 
             if isdir:
                 continue
@@ -1046,14 +1060,13 @@ class Ctl(object):
                 self.uploader_busy += 1
                 self.t0_up = self.t0_up or time.time()
 
-            zs = "{0}/{1}/{2}/{3} {4}/{5} {6}"
-            stats = zs.format(
+            stats = "%d/%d/%d/%d %d/%d %s" % (
                 self.up_f,
                 len(self.recheck),
                 self.uploader_busy,
                 self.nfiles - self.up_f,
-                int(self.nbytes / (1024 * 1024)),
-                int((self.nbytes - self.up_b) / (1024 * 1024)),
+                self.nbytes // (1024 * 1024),
+                (self.nbytes - self.up_b) // (1024 * 1024),
                 self.eta,
             )
 
@@ -1119,6 +1132,7 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
     ap.add_argument("-x", type=unicode, metavar="REGEX", default="", help="skip file if filesystem-abspath matches REGEX, example: '.*/\\.hist/.*'")
     ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
+    ap.add_argument("--touch", action="store_true", help="if last-modified timestamps differ, push local to server (need write+delete perms)")
     ap.add_argument("--version", action="store_true", help="show version and exit")
 
     ap = app.add_argument_group("compatibility")

contrib/README.md

@@ -22,6 +22,11 @@ however if your copyparty is behind a reverse-proxy, you may want to use [`share
 * `URL`: full URL to the root folder (with trailing slash) followed by `$regex:1|1$`
 * `pw`: password (remove `Parameters` if anon-write)
 
+### [`send-to-cpp.contextlet.json`](send-to-cpp.contextlet.json)
+* browser integration, kind of? custom rightclick actions and stuff
+* rightclick a pic and send it to copyparty straight from your browser
+* for the [contextlet](https://addons.mozilla.org/en-US/firefox/addon/contextlets/) firefox extension
+
 ### [`media-osd-bgone.ps1`](media-osd-bgone.ps1)
 * disables the [windows OSD popup](https://user-images.githubusercontent.com/241032/122821375-0e08df80-d2dd-11eb-9fd9-184e8aacf1d0.png) (the thing on the left) which appears every time you hit media hotkeys to adjust volume or change song while playing music with the copyparty web-ui, or most other audio players really
 

contrib/package/arch/PKGBUILD

@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.9.24"
+pkgver="1.9.29"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -21,7 +21,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("759fbc7b3063c5e39c49a9524200568ca30c664c8323b1e396d5e58378dc8b80")
+sha256sums=("868fa61f9dee57bf2911b1c000b9b49fc40dd141f8d8338b4462ba0af36774d9")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.9.24/copyparty-sfx.py",
-    "version": "1.9.24",
-    "hash": "sha256-GuKeNrH9rcsUlM7c/cQ0QKlpteu+cGfGmzxRp4ACclw="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.9.29/copyparty-sfx.py",
+    "version": "1.9.29",
+    "hash": "sha256-hY0v6sPTL4h7YFdku2Hvg5k1ewU1GK0qro4iKrXBkY8="
 }

contrib/send-to-cpp.contextlet.json

@@ -0,0 +1,11 @@
+{
+    "code": "// https://addons.mozilla.org/en-US/firefox/addon/contextlets/\n// https://github.com/davidmhammond/contextlets\n\nvar url = 'http://partybox.local:3923/';\nvar pw = 'wark';\n\nvar xhr = new XMLHttpRequest();\nxhr.msg = this.info.linkUrl || this.info.srcUrl;\nxhr.open('POST', url, true);\nxhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded;charset=UTF-8');\nxhr.setRequestHeader('PW', pw);\nxhr.send('msg=' + xhr.msg);\n",
+    "contexts": [
+        "link"
+    ],
+    "icons": null,
+    "patterns": "",
+    "scope": "background",
+    "title": "send to cpp",
+    "type": "normal"
+}

copyparty/__main__.py

@@ -19,26 +19,38 @@ import threading
 import time
 import traceback
 import uuid
-from textwrap import dedent
 
-from .__init__ import ANYWIN, CORES, EXE, PY2, VT100, WINDOWS, E, EnvParams, unicode
+from .__init__ import (
+    ANYWIN,
+    CORES,
+    EXE,
+    MACOS,
+    PY2,
+    VT100,
+    WINDOWS,
+    E,
+    EnvParams,
+    unicode,
+)
 from .__version__ import CODENAME, S_BUILD_DT, S_VERSION
-from .authsrv import expand_config_file, re_vol, split_cfg_ln, upgrade_cfg_fmt
+from .authsrv import expand_config_file, split_cfg_ln, upgrade_cfg_fmt
 from .cfg import flagcats, onedash
 from .svchub import SvcHub
 from .util import (
+    APPLESAN_TXT,
     DEF_EXP,
     DEF_MTE,
     DEF_MTH,
     IMPLICATIONS,
     JINJA_VER,
+    PY_DESC,
     PYFTPD_VER,
     SQLITE_VER,
     UNPLICATIONS,
     align_tab,
     ansi_re,
+    dedent,
     min_ex,
-    py_desc,
     pybin,
     termsize,
     wrap,
@@ -436,7 +448,7 @@ def disable_quickedit() -> None:
     if PY2:
         wintypes.LPDWORD = ctypes.POINTER(wintypes.DWORD)
 
-    k32.GetStdHandle.errcheck = ecb    # type: ignore
+    k32.GetStdHandle.errcheck = ecb  # type: ignore
     k32.GetConsoleMode.errcheck = ecb  # type: ignore
     k32.SetConsoleMode.errcheck = ecb  # type: ignore
     k32.GetConsoleMode.argtypes = (wintypes.HANDLE, wintypes.LPDWORD)
@@ -498,7 +510,9 @@ def get_sects():
               "g" (get):    download files, but cannot see folder contents
               "G" (upget):  "get", but can see filekeys of their own uploads
               "h" (html):   "get", but folders return their index.html
+              "." (dots):   user can ask to show dotfiles in listings
               "a" (admin):  can see uploader IPs, config-reload
+              "A" ("all"):  same as "rwmda." (read/write/move/delete/admin/dotfiles)
 
             too many volflags to list here, see --help-flags
 
@@ -705,6 +719,7 @@ def get_sects():
                 \033[36mln\033[0m only prints symlinks leaving the volume mountpoint
                 \033[36mp\033[0m exits 1 if any such symlinks are found
                 \033[36mr\033[0m resumes startup after the listing
+
             examples:
               --ls '**'          # list all files which are possible to read
               --ls '**,*,ln'     # check for dangerous symlinks
@@ -738,9 +753,12 @@ def get_sects():
                 """
             when \033[36m--ah-alg\033[0m is not the default [\033[32mnone\033[0m], all account passwords must be hashed
 
-            passwords can be hashed on the commandline with \033[36m--ah-gen\033[0m, but copyparty will also hash and print any passwords that are non-hashed (password which do not start with '+') and then terminate afterwards
+            passwords can be hashed on the commandline with \033[36m--ah-gen\033[0m, but
+            copyparty will also hash and print any passwords that are non-hashed
+            (password which do not start with '+') and then terminate afterwards
 
-            \033[36m--ah-alg\033[0m specifies the hashing algorithm and a list of optional comma-separated arguments:
+            \033[36m--ah-alg\033[0m specifies the hashing algorithm and a
+               list of optional comma-separated arguments:
 
             \033[36m--ah-alg argon2\033[0m  # which is the same as:
             \033[36m--ah-alg argon2,3,256,4,19\033[0m
@@ -820,10 +838,10 @@ def add_general(ap, nc, srvname):
     ap2.add_argument("-nc", metavar="NUM", type=int, default=nc, help="max num clients")
     ap2.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores, 0=all")
     ap2.add_argument("-a", metavar="ACCT", type=u, action="append", help="add account, \033[33mUSER\033[0m:\033[33mPASS\033[0m; example [\033[32med:wark\033[0m]")
-    ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m]")
-    ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files")
-    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see --help-urlform")
-    ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="window title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
+    ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m], see --help-accounts")
+    ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files (volflag=dots)")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see \033[33m--help-urlform\033[0m")
+    ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="server terminal title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
     ap2.add_argument("--name", metavar="TXT", type=u, default=srvname, help="server name (displayed topleft in browser and in mDNS)")
     ap2.add_argument("--license", action="store_true", help="show licenses and exit")
     ap2.add_argument("--version", action="store_true", help="show versions and exit")
@@ -841,14 +859,20 @@ def add_qr(ap, tty):
     ap2.add_argument("--qrz", metavar="N", type=int, default=0, help="[\033[32m1\033[0m]=1x, [\033[32m2\033[0m]=2x, [\033[32m0\033[0m]=auto (try [\033[32m2\033[0m] on broken fonts)")
 
 
+def add_fs(ap):
+    ap2 = ap.add_argument_group("filesystem options")
+    rm_re_def = "5/0.1" if ANYWIN else "0/0"
+    ap2.add_argument("--rm-retry", metavar="T/R", type=u, default=rm_re_def, help="if a file cannot be deleted because it is busy, continue trying for \033[33mT\033[0m seconds, retry every \033[33mR\033[0m seconds; disable with 0/0 (volflag=rm_retry)")
+
+
 def add_upload(ap):
     ap2 = ap.add_argument_group('upload options')
-    ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads, hiding them from clients unless -ed")
+    ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads, hiding them from clients unless \033[33m-ed\033[0m")
     ap2.add_argument("--plain-ip", action="store_true", help="when avoiding filename collisions by appending the uploader's ip to the filename: append the plaintext ip instead of salting and hashing the ip")
-    ap2.add_argument("--unpost", metavar="SEC", type=int, default=3600*12, help="grace period where uploads can be deleted by the uploader, even without delete permissions; 0=disabled")
+    ap2.add_argument("--unpost", metavar="SEC", type=int, default=3600*12, help="grace period where uploads can be deleted by the uploader, even without delete permissions; 0=disabled, default=12h")
     ap2.add_argument("--blank-wt", metavar="SEC", type=int, default=300, help="file write grace period (any client can write to a blank file last-modified more recently than \033[33mSEC\033[0m seconds ago)")
-    ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without -e2d; roughly 1 MiB RAM per 600")
-    ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (very slow on windows)")
+    ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without \033[33m-e2d\033[0m; roughly 1 MiB RAM per 600")
+    ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (bad idea to enable this on windows and/or cow filesystems)")
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
     ap2.add_argument("--hardlink", action="store_true", help="prefer hardlinks instead of symlinks when possible (within same filesystem) (volflag=hardlink)")
     ap2.add_argument("--never-symlink", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=neversymlink)")
@@ -858,12 +882,13 @@ def add_upload(ap):
     ap2.add_argument("--snap-wri", metavar="SEC", type=int, default=300, help="write upload state to ./hist/up2k.snap every \033[33mSEC\033[0m seconds; allows resuming incomplete uploads after a server crash")
     ap2.add_argument("--snap-drop", metavar="MIN", type=float, default=1440, help="forget unfinished uploads after \033[33mMIN\033[0m minutes; impossible to resume them after that (360=6h, 1440=24h)")
     ap2.add_argument("--u2ts", metavar="TXT", type=u, default="c", help="how to timestamp uploaded files; [\033[32mc\033[0m]=client-last-modified, [\033[32mu\033[0m]=upload-time, [\033[32mfc\033[0m]=force-c, [\033[32mfu\033[0m]=force-u (volflag=u2ts)")
-    ap2.add_argument("--rand", action="store_true", help="force randomized filenames, --nrand chars long (volflag=rand)")
+    ap2.add_argument("--rand", action="store_true", help="force randomized filenames, \033[33m--nrand\033[0m chars long (volflag=rand)")
     ap2.add_argument("--nrand", metavar="NUM", type=int, default=9, help="randomized filenames length (volflag=nrand)")
     ap2.add_argument("--magic", action="store_true", help="enable filetype detection on nameless uploads (volflag=magic)")
     ap2.add_argument("--df", metavar="GiB", type=float, default=0, help="ensure \033[33mGiB\033[0m free disk space by rejecting upload requests")
     ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="windows-only: minimum size of incoming uploads through up2k before they are made into sparse files")
     ap2.add_argument("--turbo", metavar="LVL", type=int, default=0, help="configure turbo-mode in up2k client; [\033[32m-1\033[0m] = forbidden/always-off, [\033[32m0\033[0m] = default-off and warn if enabled, [\033[32m1\033[0m] = default-off, [\033[32m2\033[0m] = on, [\033[32m3\033[0m] = on and disable datecheck")
+    ap2.add_argument("--u2j", metavar="JOBS", type=int, default=2, help="web-client: number of file chunks to upload in parallel; 1 or 2 is good for low-latency (same-country) connections, 4-8 for android clients, 16-32 for cross-atlantic (max=64)")
     ap2.add_argument("--u2sort", metavar="TXT", type=u, default="s", help="upload order; [\033[32ms\033[0m]=smallest-first, [\033[32mn\033[0m]=alphabetical, [\033[32mfs\033[0m]=force-s, [\033[32mfn\033[0m]=force-n -- alphabetical is a bit slower on fiber/LAN but makes it easier to eyeball if everything went fine")
     ap2.add_argument("--write-uplog", action="store_true", help="write POST reports to textfiles in working-directory")
 
@@ -872,12 +897,12 @@ def add_network(ap):
     ap2 = ap.add_argument_group('network options')
     ap2.add_argument("-i", metavar="IP", type=u, default="::", help="ip to bind (comma-sep.), default: all IPv4 and IPv6")
     ap2.add_argument("-p", metavar="PORT", type=u, default="3923", help="ports to bind (comma/range)")
-    ap2.add_argument("--ll", action="store_true", help="include link-local IPv4/IPv6 even if the NIC has routable IPs (breaks some mdns clients)")
-    ap2.add_argument("--rproxy", metavar="DEPTH", type=int, default=1, help="which ip to keep; [\033[32m0\033[0m]=tcp, [\033[32m1\033[0m]=origin (first x-fwd, unsafe), [\033[32m2\033[0m]=outermost-proxy, [\033[32m3\033[0m]=second-proxy, [\033[32m-1\033[0m]=closest-proxy")
-    ap2.add_argument("--xff-hdr", metavar="NAME", type=u, default="x-forwarded-for", help="if reverse-proxied, which http header to read the client's real ip from (argument must be lowercase, but not the actual header)")
-    ap2.add_argument("--xff-src", metavar="IP", type=u, default="127., ::1", help="comma-separated list of trusted reverse-proxy IPs; only accept the real-ip header (--xff-hdr) if the incoming connection is from an IP starting with either of these. Can be disabled with [\033[32many\033[0m] if you are behind cloudflare (or similar) and are using --xff-hdr=cf-connecting-ip (or similar)")
+    ap2.add_argument("--ll", action="store_true", help="include link-local IPv4/IPv6 in mDNS replies, even if the NIC has routable IPs (breaks some mDNS clients)")
+    ap2.add_argument("--rproxy", metavar="DEPTH", type=int, default=1, help="which ip to associate clients with; [\033[32m0\033[0m]=tcp, [\033[32m1\033[0m]=origin (first x-fwd, unsafe), [\033[32m2\033[0m]=outermost-proxy, [\033[32m3\033[0m]=second-proxy, [\033[32m-1\033[0m]=closest-proxy")
+    ap2.add_argument("--xff-hdr", metavar="NAME", type=u, default="x-forwarded-for", help="if reverse-proxied, which http header to read the client's real ip from")
+    ap2.add_argument("--xff-src", metavar="IP", type=u, default="127., ::1", help="comma-separated list of trusted reverse-proxy IPs; only accept the real-ip header (\033[33m--xff-hdr\033[0m) if the incoming connection is from an IP starting with either of these. Can be disabled with [\033[32many\033[0m] if you are behind cloudflare (or similar) and are using \033[32m--xff-hdr=cf-connecting-ip\033[0m (or similar)")
     ap2.add_argument("--ipa", metavar="PREFIX", type=u, default="", help="only accept connections from IP-addresses starting with \033[33mPREFIX\033[0m; example: [\033[32m127., 10.89., 192.168.\033[0m]")
-    ap2.add_argument("--rp-loc", metavar="PATH", type=u, default="", help="if reverse-proxying on a location instead of a dedicated domain/subdomain, provide the base location here (eg. /foo/bar)")
+    ap2.add_argument("--rp-loc", metavar="PATH", type=u, default="", help="if reverse-proxying on a location instead of a dedicated domain/subdomain, provide the base location here; example: [\033[32m/foo/bar\033[0m]")
     if ANYWIN:
         ap2.add_argument("--reuseaddr", action="store_true", help="set reuseaddr on listening sockets on windows; allows rapid restart of copyparty at the expense of being able to accidentally start multiple instances")
     else:
@@ -906,7 +931,7 @@ def add_cert(ap, cert_path):
     ap2 = ap.add_argument_group('TLS certificate generator options')
     ap2.add_argument("--no-crt", action="store_true", help="disable automatic certificate creation")
     ap2.add_argument("--crt-ns", metavar="N,N", type=u, default="", help="comma-separated list of FQDNs (domains) to add into the certificate")
-    ap2.add_argument("--crt-exact", action="store_true", help="do not add wildcard entries for each --crt-ns")
+    ap2.add_argument("--crt-exact", action="store_true", help="do not add wildcard entries for each \033[33m--crt-ns\033[0m")
     ap2.add_argument("--crt-noip", action="store_true", help="do not add autodetected IP addresses into cert")
     ap2.add_argument("--crt-nolo", action="store_true", help="do not add 127.0.0.1 / localhost into cert")
     ap2.add_argument("--crt-nohn", action="store_true", help="do not add mDNS names / hostname into cert")
@@ -917,12 +942,14 @@ def add_cert(ap, cert_path):
     ap2.add_argument("--crt-cnc", metavar="TXT", type=u, default="--crt-cn", help="override CA name")
     ap2.add_argument("--crt-cns", metavar="TXT", type=u, default="--crt-cn cpp", help="override server-cert name")
     ap2.add_argument("--crt-back", metavar="HRS", type=float, default=72, help="backdate in hours")
-    ap2.add_argument("--crt-alg", metavar="S-N", type=u, default="ecdsa-256", help="algorithm and keysize; one of these: ecdsa-256 rsa-4096 rsa-2048")
+    ap2.add_argument("--crt-alg", metavar="S-N", type=u, default="ecdsa-256", help="algorithm and keysize; one of these: \033[32mecdsa-256 rsa-4096 rsa-2048\033[0m")
 
 
 def add_auth(ap):
-    ap2 = ap.add_argument_group('user authentication options')
-    ap2.add_argument("--hdr-au-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks and assume the request-header \033[33mHN\033[0m contains the username of the requesting user (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy. Also, the argument must be lowercase, but not the actual header")
+    ap2 = ap.add_argument_group('IdP / identity provider / user authentication options')
+    ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks and assume the request-header \033[33mHN\033[0m contains the username of the requesting user (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
+    return
+    ap2.add_argument("--idp-h-grp", metavar="HN", type=u, default="", help="assume the request-header \033[33mHN\033[0m contains the groupname of the requesting user; can be referenced in config files for group-based access control")
 
 
 def add_zeroconf(ap):
@@ -932,19 +959,19 @@ def add_zeroconf(ap):
     ap2.add_argument("--z-off", metavar="NETS", type=u, default="", help="disable zeroconf on the comma-separated list of subnets and/or interface names/indexes")
     ap2.add_argument("--z-chk", metavar="SEC", type=int, default=10, help="check for network changes every \033[33mSEC\033[0m seconds (0=disable)")
     ap2.add_argument("-zv", action="store_true", help="verbose all zeroconf backends")
-    ap2.add_argument("--mc-hop", metavar="SEC", type=int, default=0, help="rejoin multicast groups every \033[33mSEC\033[0m seconds (workaround for some switches/routers which cause mDNS to suddenly stop working after some time); try [\033[32m300\033[0m] or [\033[32m180\033[0m]")
+    ap2.add_argument("--mc-hop", metavar="SEC", type=int, default=0, help="rejoin multicast groups every \033[33mSEC\033[0m seconds (workaround for some switches/routers which cause mDNS to suddenly stop working after some time); try [\033[32m300\033[0m] or [\033[32m180\033[0m]\n └─note: can be due to firewalls; make sure UDP port 5353 is open in both directions (on clients too)")
 
 
 def add_zc_mdns(ap):
     ap2 = ap.add_argument_group("Zeroconf-mDNS options; also see --help-zm")
     ap2.add_argument("--zm", action="store_true", help="announce the enabled protocols over mDNS (multicast DNS-SD) -- compatible with KDE, gnome, macOS, ...")
-    ap2.add_argument("--zm-on", metavar="NETS", type=u, default="", help="enable zeroconf ONLY on the comma-separated list of subnets and/or interface names/indexes")
-    ap2.add_argument("--zm-off", metavar="NETS", type=u, default="", help="disable zeroconf on the comma-separated list of subnets and/or interface names/indexes")
+    ap2.add_argument("--zm-on", metavar="NETS", type=u, default="", help="enable mDNS ONLY on the comma-separated list of subnets and/or interface names/indexes")
+    ap2.add_argument("--zm-off", metavar="NETS", type=u, default="", help="disable mDNS on the comma-separated list of subnets and/or interface names/indexes")
     ap2.add_argument("--zm4", action="store_true", help="IPv4 only -- try this if some clients can't connect")
     ap2.add_argument("--zm6", action="store_true", help="IPv6 only")
     ap2.add_argument("--zmv", action="store_true", help="verbose mdns")
     ap2.add_argument("--zmvv", action="store_true", help="verboser mdns")
-    ap2.add_argument("--zms", metavar="dhf", type=u, default="", help="list of services to announce -- d=webdav h=http f=ftp s=smb -- lowercase=plaintext uppercase=TLS -- default: all enabled services except http/https (\033[32mDdfs\033[0m if \033[33m--ftp\033[0m and \033[33m--smb\033[0m is set)")
+    ap2.add_argument("--zms", metavar="dhf", type=u, default="", help="list of services to announce -- d=webdav h=http f=ftp s=smb -- lowercase=plaintext uppercase=TLS -- default: all enabled services except http/https (\033[32mDdfs\033[0m if \033[33m--ftp\033[0m and \033[33m--smb\033[0m is set, \033[32mDd\033[0m otherwise)")
     ap2.add_argument("--zm-ld", metavar="PATH", type=u, default="", help="link a specific folder for webdav shares")
     ap2.add_argument("--zm-lh", metavar="PATH", type=u, default="", help="link a specific folder for http shares")
     ap2.add_argument("--zm-lf", metavar="PATH", type=u, default="", help="link a specific folder for ftp shares")
@@ -952,14 +979,14 @@ def add_zc_mdns(ap):
     ap2.add_argument("--zm-mnic", action="store_true", help="merge NICs which share subnets; assume that same subnet means same network")
     ap2.add_argument("--zm-msub", action="store_true", help="merge subnets on each NIC -- always enabled for ipv6 -- reduces network load, but gnome-gvfs clients may stop working, and clients cannot be in subnets that the server is not")
     ap2.add_argument("--zm-noneg", action="store_true", help="disable NSEC replies -- try this if some clients don't see copyparty")
-    ap2.add_argument("--zm-spam", metavar="SEC", type=float, default=0, help="send unsolicited announce every \033[33mSEC\033[0m; useful if clients have IPs in a subnet which doesn't overlap with the server")
+    ap2.add_argument("--zm-spam", metavar="SEC", type=float, default=0, help="send unsolicited announce every \033[33mSEC\033[0m; useful if clients have IPs in a subnet which doesn't overlap with the server, or to avoid some firewall issues")
 
 
 def add_zc_ssdp(ap):
     ap2 = ap.add_argument_group("Zeroconf-SSDP options")
     ap2.add_argument("--zs", action="store_true", help="announce the enabled protocols over SSDP -- compatible with Windows")
-    ap2.add_argument("--zs-on", metavar="NETS", type=u, default="", help="enable zeroconf ONLY on the comma-separated list of subnets and/or interface names/indexes")
-    ap2.add_argument("--zs-off", metavar="NETS", type=u, default="", help="disable zeroconf on the comma-separated list of subnets and/or interface names/indexes")
+    ap2.add_argument("--zs-on", metavar="NETS", type=u, default="", help="enable SSDP ONLY on the comma-separated list of subnets and/or interface names/indexes")
+    ap2.add_argument("--zs-off", metavar="NETS", type=u, default="", help="disable SSDP on the comma-separated list of subnets and/or interface names/indexes")
     ap2.add_argument("--zsv", action="store_true", help="verbose SSDP")
     ap2.add_argument("--zsl", metavar="PATH", type=u, default="/?hc", help="location to include in the url (or a complete external URL), for example [\033[32mpriv/?pw=hunter2\033[0m] (goes directly to /priv/ with password hunter2) or [\033[32m?hc=priv&pw=hunter2\033[0m] (shows mounting options for /priv/ with password)")
     ap2.add_argument("--zsid", metavar="UUID", type=u, default=zsid, help="USN (device identifier) to announce")
@@ -988,7 +1015,7 @@ def add_webdav(ap):
 
 def add_smb(ap):
     ap2 = ap.add_argument_group('SMB/CIFS options')
-    ap2.add_argument("--smb", action="store_true", help="enable smb (read-only) -- this requires running copyparty as root on linux and macos unless --smb-port is set above 1024 and your OS does port-forwarding from 445 to that.\n\033[1;31mWARNING:\033[0m this protocol is dangerous! Never expose to the internet!")
+    ap2.add_argument("--smb", action="store_true", help="enable smb (read-only) -- this requires running copyparty as root on linux and macos unless \033[33m--smb-port\033[0m is set above 1024 and your OS does port-forwarding from 445 to that.\n\033[1;31mWARNING:\033[0m this protocol is DANGEROUS and buggy! Never expose to the internet!")
     ap2.add_argument("--smbw", action="store_true", help="enable write support (please dont)")
     ap2.add_argument("--smb1", action="store_true", help="disable SMBv2, only enable SMBv1 (CIFS)")
     ap2.add_argument("--smb-port", metavar="PORT", type=int, default=445, help="port to listen on -- if you change this value, you must NAT from TCP:445 to this port using iptables or similar")
@@ -1002,9 +1029,9 @@ def add_smb(ap):
 
 def add_handlers(ap):
     ap2 = ap.add_argument_group('handlers (see --help-handlers)')
-    ap2.add_argument("--on404", metavar="PY", type=u, action="append", help="handle 404s by executing PY file")
-    ap2.add_argument("--on403", metavar="PY", type=u, action="append", help="handle 403s by executing PY file")
-    ap2.add_argument("--hot-handlers", action="store_true", help="reload handlers on each request -- expensive but convenient when hacking on stuff")
+    ap2.add_argument("--on404", metavar="PY", type=u, action="append", help="handle 404s by executing \033[33mPY\033[0m file")
+    ap2.add_argument("--on403", metavar="PY", type=u, action="append", help="handle 403s by executing \033[33mPY\033[0m file")
+    ap2.add_argument("--hot-handlers", action="store_true", help="recompile handlers on each request -- expensive but convenient when hacking on stuff")
 
 
 def add_hooks(ap):
@@ -1039,17 +1066,17 @@ def add_yolo(ap):
 def add_optouts(ap):
     ap2 = ap.add_argument_group('opt-outs')
     ap2.add_argument("-nw", action="store_true", help="never write anything to disk (debug/benchmark)")
-    ap2.add_argument("--keep-qem", action="store_true", help="do not disable quick-edit-mode on windows (it is disabled to avoid accidental text selection which will deadlock copyparty)")
+    ap2.add_argument("--keep-qem", action="store_true", help="do not disable quick-edit-mode on windows (it is disabled to avoid accidental text selection in the terminal window, as this would pause execution)")
     ap2.add_argument("--no-dav", action="store_true", help="disable webdav support")
     ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
     ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
-    ap2.add_argument("-nth", action="store_true", help="no title hostname; don't show --name in <title>")
+    ap2.add_argument("-nth", action="store_true", help="no title hostname; don't show \033[33m--name\033[0m in <title>")
     ap2.add_argument("-nih", action="store_true", help="no info hostname -- don't show in UI")
     ap2.add_argument("-nid", action="store_true", help="no info disk-usage -- don't show in UI")
     ap2.add_argument("-nb", action="store_true", help="no powered-by-copyparty branding in UI")
     ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
     ap2.add_argument("--no-tarcmp", action="store_true", help="disable download as compressed tar (?tar=gz, ?tar=bz2, ?tar=xz, ?tar=gz:9, ...)")
-    ap2.add_argument("--no-lifetime", action="store_true", help="disable automatic deletion of uploads after a certain time (as specified by the 'lifetime' volflag)")
+    ap2.add_argument("--no-lifetime", action="store_true", help="do not allow clients (or server config) to schedule an upload to be deleted after a given time")
 
 
 def add_safety(ap):
@@ -1057,36 +1084,36 @@ def add_safety(ap):
     ap2.add_argument("-s", action="count", default=0, help="increase safety: Disable thumbnails / potentially dangerous software (ffmpeg/pillow/vips), hide partial uploads, avoid crawlers.\n └─Alias of\033[32m --dotpart --no-thumb --no-mtag-ff --no-robots --force-js")
     ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, webdav, 404 on 403, ban on excessive 404s.\n └─Alias of\033[32m -s --unpost=0 --no-del --no-mv --hardlink --vague-403 -nih")
     ap2.add_argument("-sss", action="store_true", help="further increase safety: Enable logging to disk, scan for dangerous symlinks.\n └─Alias of\033[32m -ss --no-dav --no-logues --no-readme -lo=cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz --ls=**,*,ln,p,r")
-    ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="do a sanity/safety check of all volumes on startup; arguments \033[33mUSER\033[0m,\033[33mVOL\033[0m,\033[33mFLAGS\033[0m; example [\033[32m**,*,ln,p,r\033[0m]")
+    ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="do a sanity/safety check of all volumes on startup; arguments \033[33mUSER\033[0m,\033[33mVOL\033[0m,\033[33mFLAGS\033[0m (see \033[33m--help-ls\033[0m); example [\033[32m**,*,ln,p,r\033[0m]")
     ap2.add_argument("--xvol", action="store_true", help="never follow symlinks leaving the volume root, unless the link is into another volume where the user has similar access (volflag=xvol)")
     ap2.add_argument("--xdev", action="store_true", help="stay within the filesystem of the volume root; do not descend into other devices (symlink or bind-mount to another HDD, ...) (volflag=xdev)")
     ap2.add_argument("--no-dot-mv", action="store_true", help="disallow moving dotfiles; makes it impossible to move folders containing dotfiles")
-    ap2.add_argument("--no-dot-ren", action="store_true", help="disallow renaming dotfiles; makes it impossible to make something a dotfile")
+    ap2.add_argument("--no-dot-ren", action="store_true", help="disallow renaming dotfiles; makes it impossible to turn something into a dotfile")
     ap2.add_argument("--no-logues", action="store_true", help="disable rendering .prologue/.epilogue.html into directory listings")
     ap2.add_argument("--no-readme", action="store_true", help="disable rendering readme.md into directory listings")
     ap2.add_argument("--vague-403", action="store_true", help="send 404 instead of 403 (security through ambiguity, very enterprise)")
-    ap2.add_argument("--force-js", action="store_true", help="don't send folder listings as HTML, force clients to use the embedded json instead -- slight protection against misbehaving search engines which ignore --no-robots")
+    ap2.add_argument("--force-js", action="store_true", help="don't send folder listings as HTML, force clients to use the embedded json instead -- slight protection against misbehaving search engines which ignore \033[33m--no-robots\033[0m")
     ap2.add_argument("--no-robots", action="store_true", help="adds http and html headers asking search engines to not index anything (volflag=norobots)")
     ap2.add_argument("--logout", metavar="H", type=float, default="8086", help="logout clients after \033[33mH\033[0m hours of inactivity; [\033[32m0.0028\033[0m]=10sec, [\033[32m0.1\033[0m]=6min, [\033[32m24\033[0m]=day, [\033[32m168\033[0m]=week, [\033[32m720\033[0m]=month, [\033[32m8760\033[0m]=year)")
     ap2.add_argument("--ban-pw", metavar="N,W,B", type=u, default="9,60,1440", help="more than \033[33mN\033[0m wrong passwords in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; disable with [\033[32mno\033[0m]")
     ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="50,60,1440", help="hitting more than \033[33mN\033[0m 404's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; only affects users who cannot see directory listings because their access is either g/G/h")
     ap2.add_argument("--ban-403", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 403's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; [\033[32m1440\033[0m]=day, [\033[32m10080\033[0m]=week, [\033[32m43200\033[0m]=month")
-    ap2.add_argument("--ban-422", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 422's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (422 is server fuzzing, invalid POSTs and so)")
-    ap2.add_argument("--ban-url", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m sus URL's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; applies only to access g/G/h (decent replacement for --ban-404 if that can't be used)")
+    ap2.add_argument("--ban-422", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 422's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (invalid requests, attempted exploits ++)")
+    ap2.add_argument("--ban-url", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m sus URL's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; applies only to permissions g/G/h (decent replacement for \033[33m--ban-404\033[0m if that can't be used)")
     ap2.add_argument("--sus-urls", metavar="R", type=u, default=r"\.php$|(^|/)wp-(admin|content|includes)/", help="URLs which are considered sus / eligible for banning; disable with blank or [\033[32mno\033[0m]")
     ap2.add_argument("--nonsus-urls", metavar="R", type=u, default=r"^(favicon\.ico|robots\.txt)$|^apple-touch-icon|^\.well-known", help="harmless URLs ignored from 404-bans; disable with blank or [\033[32mno\033[0m]")
     ap2.add_argument("--aclose", metavar="MIN", type=int, default=10, help="if a client maxes out the server connection limit, downgrade it from connection:keep-alive to connection:close for \033[33mMIN\033[0m minutes (and also kill its active connections) -- disable with 0")
     ap2.add_argument("--loris", metavar="B", type=int, default=60, help="if a client maxes out the server connection limit without sending headers, ban it for \033[33mB\033[0m minutes; disable with [\033[32m0\033[0m]")
     ap2.add_argument("--acao", metavar="V[,V]", type=u, default="*", help="Access-Control-Allow-Origin; list of origins (domains/IPs without port) to accept requests from; [\033[32mhttps://1.2.3.4\033[0m]. Default [\033[32m*\033[0m] allows requests from all sites but removes cookies and http-auth; only ?pw=hunter2 survives")
-    ap2.add_argument("--acam", metavar="V[,V]", type=u, default="GET,HEAD", help="Access-Control-Allow-Methods; list of methods to accept from offsite ('*' behaves like described in --acao)")
+    ap2.add_argument("--acam", metavar="V[,V]", type=u, default="GET,HEAD", help="Access-Control-Allow-Methods; list of methods to accept from offsite ('*' behaves like \033[33m--acao\033[0m's description)")
 
 
 def add_salt(ap, fk_salt, ah_salt):
     ap2 = ap.add_argument_group('salting options')
-    ap2.add_argument("--ah-alg", metavar="ALG", type=u, default="none", help="account-pw hashing algorithm; one of these, best to worst: argon2 scrypt sha2 none (each optionally followed by alg-specific comma-sep. config)")
-    ap2.add_argument("--ah-salt", metavar="SALT", type=u, default=ah_salt, help="account-pw salt; ignored if --ah-alg is none (default)")
+    ap2.add_argument("--ah-alg", metavar="ALG", type=u, default="none", help="account-pw hashing algorithm; one of these, best to worst: \033[32margon2 scrypt sha2 none\033[0m (each optionally followed by alg-specific comma-sep. config)")
+    ap2.add_argument("--ah-salt", metavar="SALT", type=u, default=ah_salt, help="account-pw salt; ignored if \033[33m--ah-alg\033[0m is none (default)")
     ap2.add_argument("--ah-gen", metavar="PW", type=u, default="", help="generate hashed password for \033[33mPW\033[0m, or read passwords from STDIN if \033[33mPW\033[0m is [\033[32m-\033[0m]")
-    ap2.add_argument("--ah-cli", action="store_true", help="interactive shell which hashes passwords without ever storing or displaying the original passwords")
+    ap2.add_argument("--ah-cli", action="store_true", help="launch an interactive shell which hashes passwords without ever storing or displaying the original passwords")
     ap2.add_argument("--fk-salt", metavar="SALT", type=u, default=fk_salt, help="per-file accesskey salt; used to generate unpredictable URLs for hidden files")
     ap2.add_argument("--warksalt", metavar="SALT", type=u, default="hunter2", help="up2k file-hash salt; serves no purpose, no reason to change this (but delete all databases if you do)")
 
@@ -1100,8 +1127,8 @@ def add_shutdown(ap):
 
 def add_logging(ap):
     ap2 = ap.add_argument_group('logging options')
-    ap2.add_argument("-q", action="store_true", help="quiet")
-    ap2.add_argument("-lo", metavar="PATH", type=u, help="logfile, example: \033[32mcpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz")
+    ap2.add_argument("-q", action="store_true", help="quiet; disable most STDOUT messages")
+    ap2.add_argument("-lo", metavar="PATH", type=u, help="logfile, example: \033[32mcpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz\033[0m (NB: some errors may appear on STDOUT only)")
     ap2.add_argument("--no-ansi", action="store_true", default=not VT100, help="disable colors; same as environment-variable NO_COLOR")
     ap2.add_argument("--ansi", action="store_true", help="force colors; overrides environment-variable NO_COLOR")
     ap2.add_argument("--no-logflush", action="store_true", help="don't flush the logfile after each write; tiny bit faster")
@@ -1110,7 +1137,7 @@ def add_logging(ap):
     ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log failed login attempt passwords: 0=terse, 1=plaintext, 2=hashed")
     ap2.add_argument("--log-conn", action="store_true", help="debug: print tcp-server msgs")
     ap2.add_argument("--log-htp", action="store_true", help="debug: print http-server threadpool scaling")
-    ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header")
+    ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="print request \033[33mHEADER\033[0m; [\033[32m*\033[0m]=all")
     ap2.add_argument("--lf-url", metavar="RE", type=u, default=r"^/\.cpr/|\?th=[wj]$|/\.(_|ql_|DS_Store$|localized$)", help="dont log URLs matching regex \033[33mRE\033[0m")
 
 
@@ -1129,16 +1156,17 @@ def add_thumbnail(ap):
     ap2.add_argument("--th-size", metavar="WxH", default="320x256", help="thumbnail res (volflag=thsize)")
     ap2.add_argument("--th-mt", metavar="CORES", type=int, default=CORES, help="num cpu cores to use for generating thumbnails")
     ap2.add_argument("--th-convt", metavar="SEC", type=float, default=60, help="conversion timeout in seconds (volflag=convt)")
-    ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image by default (volflag=nocrop)")
+    ap2.add_argument("--th-ram-max", metavar="GB", type=float, default=6, help="max memory usage (GiB) permitted by thumbnailer; not very accurate")
+    ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image by default (client can override in UI) (volflag=nocrop)")
     ap2.add_argument("--th-dec", metavar="LIBS", default="vips,pil,ff", help="image decoders, in order of preference")
     ap2.add_argument("--th-no-jpg", action="store_true", help="disable jpg output")
     ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output")
-    ap2.add_argument("--th-ff-jpg", action="store_true", help="force jpg output for video thumbs")
-    ap2.add_argument("--th-ff-swr", action="store_true", help="use swresample instead of soxr for audio thumbs")
+    ap2.add_argument("--th-ff-jpg", action="store_true", help="force jpg output for video thumbs (avoids issues on some FFmpeg builds)")
+    ap2.add_argument("--th-ff-swr", action="store_true", help="use swresample instead of soxr for audio thumbs (faster, lower accuracy, avoids issues on some FFmpeg builds)")
     ap2.add_argument("--th-poke", metavar="SEC", type=int, default=300, help="activity labeling cooldown -- avoids doing keepalive pokes (updating the mtime) on thumbnail folders more often than \033[33mSEC\033[0m seconds")
     ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval; 0=disabled")
-    ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age -- folders which haven't been poked for longer than --th-poke seconds will get deleted every --th-clean seconds")
-    ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat/look for; enabling -e2d will make these case-insensitive, and also automatically select thumbnails for all folders that contain pics, even if none match this pattern")
+    ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age -- folders which haven't been poked for longer than \033[33m--th-poke\033[0m seconds will get deleted every \033[33m--th-clean\033[0m seconds")
+    ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat/look for; enabling \033[33m-e2d\033[0m will make these case-insensitive, and try them as dotfiles (.folder.jpg), and also automatically select thumbnails for all folders that contain pics, even if none match this pattern")
     # https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
     # https://github.com/libvips/libvips
     # ffmpeg -hide_banner -demuxers | awk '/^ D  /{print$2}' | while IFS= read -r x; do ffmpeg -hide_banner -h demuxer=$x; done | grep -E '^Demuxer |extensions:'
@@ -1157,23 +1185,24 @@ def add_transcoding(ap):
 
 
 def add_db_general(ap, hcores):
+    noidx = APPLESAN_TXT if MACOS else ""
     ap2 = ap.add_argument_group('general db options')
     ap2.add_argument("-e2d", action="store_true", help="enable up2k database, making files searchable + enables upload deduplication")
-    ap2.add_argument("-e2ds", action="store_true", help="scan writable folders for new files on startup; sets -e2d")
-    ap2.add_argument("-e2dsa", action="store_true", help="scans all folders on startup; sets -e2ds")
+    ap2.add_argument("-e2ds", action="store_true", help="scan writable folders for new files on startup; sets \033[33m-e2d\033[0m")
+    ap2.add_argument("-e2dsa", action="store_true", help="scans all folders on startup; sets \033[33m-e2ds\033[0m")
     ap2.add_argument("-e2v", action="store_true", help="verify file integrity; rehash all files and compare with db")
     ap2.add_argument("-e2vu", action="store_true", help="on hash mismatch: update the database with the new hash")
     ap2.add_argument("-e2vp", action="store_true", help="on hash mismatch: panic and quit copyparty")
     ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume data (db, thumbs) (volflag=hist)")
     ap2.add_argument("--no-hash", metavar="PTN", type=u, help="regex: disable hashing of matching absolute-filesystem-paths during e2ds folder scans (volflag=nohash)")
-    ap2.add_argument("--no-idx", metavar="PTN", type=u, help="regex: disable indexing of matching absolute-filesystem-paths during e2ds folder scans (volflag=noidx)")
+    ap2.add_argument("--no-idx", metavar="PTN", type=u, default=noidx, help="regex: disable indexing of matching absolute-filesystem-paths during e2ds folder scans (volflag=noidx)")
     ap2.add_argument("--no-dhash", action="store_true", help="disable rescan acceleration; do full database integrity check -- makes the db ~5%% smaller and bootup/rescans 3~10x slower")
     ap2.add_argument("--re-dhash", action="store_true", help="rebuild the cache if it gets out of sync (for example crash on startup during metadata scanning)")
-    ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice (volflag=noforget)")
-    ap2.add_argument("--dbd", metavar="PROFILE", default="wal", help="database durability profile; sets the tradeoff between robustness and speed, see --help-dbd (volflag=dbd)")
+    ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice -- only useful for offloading uploads to a cloud service or something (volflag=noforget)")
+    ap2.add_argument("--dbd", metavar="PROFILE", default="wal", help="database durability profile; sets the tradeoff between robustness and speed, see \033[33m--help-dbd\033[0m (volflag=dbd)")
     ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (volflag=xlink)")
     ap2.add_argument("--hash-mt", metavar="CORES", type=int, default=hcores, help="num cpu cores to use for file hashing; set 0 or 1 for single-core hashing")
-    ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval, 0=off (volflag=scan)")
+    ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="rescan filesystem for changes every \033[33mSEC\033[0m seconds; 0=off (volflag=scan)")
     ap2.add_argument("--db-act", metavar="SEC", type=float, default=10, help="defer any scheduled volume reindexing until \033[33mSEC\033[0m seconds after last db write (uploads, renames, ...)")
     ap2.add_argument("--srch-time", metavar="SEC", type=int, default=45, help="search deadline -- terminate searches running for more than \033[33mSEC\033[0m seconds")
     ap2.add_argument("--srch-hits", metavar="N", type=int, default=7999, help="max search results to allow clients to fetch; 125 results will be shown initially")
@@ -1183,25 +1212,25 @@ def add_db_general(ap, hcores):
 def add_db_metadata(ap):
     ap2 = ap.add_argument_group('metadata db options')
     ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing; makes it possible to search for artist/title/codec/resolution/...")
-    ap2.add_argument("-e2ts", action="store_true", help="scan existing files on startup; sets -e2t")
-    ap2.add_argument("-e2tsr", action="store_true", help="delete all metadata from DB and do a full rescan; sets -e2ts")
-    ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead; will catch more tags")
+    ap2.add_argument("-e2ts", action="store_true", help="scan newly discovered files for metadata on startup; sets \033[33m-e2t\033[0m")
+    ap2.add_argument("-e2tsr", action="store_true", help="delete all metadata from DB and do a full rescan; sets \033[33m-e2ts\033[0m")
+    ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead; will detect more tags")
     ap2.add_argument("--no-mtag-ff", action="store_true", help="never use FFprobe as tag reader; is probably safer")
-    ap2.add_argument("--mtag-to", metavar="SEC", type=int, default=60, help="timeout for ffprobe tag-scan")
+    ap2.add_argument("--mtag-to", metavar="SEC", type=int, default=60, help="timeout for FFprobe tag-scan")
     ap2.add_argument("--mtag-mt", metavar="CORES", type=int, default=CORES, help="num cpu cores to use for tag scanning")
     ap2.add_argument("--mtag-v", action="store_true", help="verbose tag scanning; print errors from mtp subprocesses and such")
-    ap2.add_argument("--mtag-vv", action="store_true", help="debug mtp settings and mutagen/ffprobe parsers")
+    ap2.add_argument("--mtag-vv", action="store_true", help="debug mtp settings and mutagen/FFprobe parsers")
     ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
     ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.); either an entire replacement list, or add/remove stuff on the default-list with +foo or /bar", default=DEF_MTE)
-    ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.); assign/add/remove same as -mte", default=DEF_MTH)
+    ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.); assign/add/remove same as \033[33m-mte\033[0m", default=DEF_MTH)
     ap2.add_argument("-mtp", metavar="M=[f,]BIN", type=u, action="append", help="read tag \033[33mM\033[0m using program \033[33mBIN\033[0m to parse the file")
 
 
 def add_txt(ap):
     ap2 = ap.add_argument_group('textfile options')
-    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="textfile editor checks for serverside changes every \033[33mSEC\033[0m seconds")
+    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="the textfile editor will check for serverside changes every \033[33mSEC\033[0m seconds")
     ap2.add_argument("-emp", action="store_true", help="enable markdown plugins -- neat but dangerous, big XSS risk")
-    ap2.add_argument("--exp", action="store_true", help="enable textfile expansion -- replace {{self.ip}} and such; see --help-exp (volflag=exp)")
+    ap2.add_argument("--exp", action="store_true", help="enable textfile expansion -- replace {{self.ip}} and such; see \033[33m--help-exp\033[0m (volflag=exp)")
     ap2.add_argument("--exp-md", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in markdown files; add/remove stuff on the default list with +hdr_foo or /vf.scan (volflag=exp_md)")
     ap2.add_argument("--exp-lg", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in prologue/epilogue files (volflag=exp_lg)")
 
@@ -1209,8 +1238,8 @@ def add_txt(ap):
 def add_ui(ap, retry):
     ap2 = ap.add_argument_group('ui options')
     ap2.add_argument("--grid", action="store_true", help="show grid/thumbnails by default (volflag=grid)")
-    ap2.add_argument("--lang", metavar="LANG", type=u, default="eng", help="language; one of the following: eng nor")
-    ap2.add_argument("--theme", metavar="NUM", type=int, default=0, help="default theme to use")
+    ap2.add_argument("--lang", metavar="LANG", type=u, default="eng", help="language; one of the following: \033[32meng nor\033[0m")
+    ap2.add_argument("--theme", metavar="NUM", type=int, default=0, help="default theme to use (0..7)")
     ap2.add_argument("--themes", metavar="NUM", type=int, default=8, help="number of themes installed")
     ap2.add_argument("--sort", metavar="C,C,C", type=u, default="href", help="default sort order, comma-separated column IDs (see header tooltips), prefix with '-' for descending. Examples: \033[32mhref -href ext sz ts tags/Album tags/.tn\033[0m (volflag=sort)")
     ap2.add_argument("--unlist", metavar="REGEX", type=u, default="", help="don't show files matching \033[33mREGEX\033[0m in file list. Purely cosmetic! Does not affect API calls, just the browser. Example: [\033[32m\\.(js|css)$\033[0m] (volflag=unlist)")
@@ -1224,8 +1253,8 @@ def add_ui(ap, retry):
     ap2.add_argument("--txt-max", metavar="KiB", type=int, default=64, help="max size of embedded textfiles on ?doc= (anything bigger will be lazy-loaded by JS)")
     ap2.add_argument("--doctitle", metavar="TXT", type=u, default="copyparty @ --name", help="title / service-name to show in html documents")
     ap2.add_argument("--bname", metavar="TXT", type=u, default="--name", help="server name (displayed in filebrowser document title)")
-    ap2.add_argument("--pb-url", metavar="URL", type=u, default="https://github.com/9001/copyparty", help="powered-by link; disable with -np")
-    ap2.add_argument("--ver", action="store_true", help="show version on the control panel (incompatible with -nb)")
+    ap2.add_argument("--pb-url", metavar="URL", type=u, default="https://github.com/9001/copyparty", help="powered-by link; disable with \033[33m-np\033[0m")
+    ap2.add_argument("--ver", action="store_true", help="show version on the control panel (incompatible with \033[33m-nb\033[0m)")
     ap2.add_argument("--md-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for README.md docs (volflag=md_sbf); see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#attr-sandbox")
     ap2.add_argument("--lg-sbf", metavar="FLAGS", type=u, default="downloads forms popups scripts top-navigation-by-user-activation", help="list of capabilities to ALLOW for prologue/epilogue docs  (volflag=lg_sbf)")
     ap2.add_argument("--no-sb-md", action="store_true", help="don't sandbox README.md documents (volflags: no_sb_md | sb_md)")
@@ -1236,17 +1265,17 @@ def add_debug(ap):
     ap2 = ap.add_argument_group('debug options')
     ap2.add_argument("--vc", action="store_true", help="verbose config file parser (explain config)")
     ap2.add_argument("--cgen", action="store_true", help="generate config file from current config (best-effort; probably buggy)")
-    ap2.add_argument("--no-sendfile", action="store_true", help="disable sendfile; instead using a traditional file read loop")
-    ap2.add_argument("--no-scandir", action="store_true", help="disable scandir; instead using listdir + stat on each file")
-    ap2.add_argument("--no-fastboot", action="store_true", help="wait for up2k indexing before starting the httpd")
+    ap2.add_argument("--no-sendfile", action="store_true", help="kernel-bug workaround: disable sendfile; do a safe and slow read-send-loop instead")
+    ap2.add_argument("--no-scandir", action="store_true", help="kernel-bug workaround: disable scandir; do a listdir + stat on each file instead")
+    ap2.add_argument("--no-fastboot", action="store_true", help="wait for initial filesystem indexing before accepting client requests")
     ap2.add_argument("--no-htp", action="store_true", help="disable httpserver threadpool, create threads as-needed instead")
     ap2.add_argument("--srch-dbg", action="store_true", help="explain search processing, and do some extra expensive sanity checks")
     ap2.add_argument("--rclone-mdns", action="store_true", help="use mdns-domain instead of server-ip on /?hc")
     ap2.add_argument("--stackmon", metavar="P,S", type=u, help="write stacktrace to \033[33mP\033[0math every \033[33mS\033[0m second, for example --stackmon=\033[32m./st/%%Y-%%m/%%d/%%H%%M.xz,60")
     ap2.add_argument("--log-thrs", metavar="SEC", type=float, help="list active threads every \033[33mSEC\033[0m")
     ap2.add_argument("--log-fk", metavar="REGEX", type=u, default="", help="log filekey params for files where path matches \033[33mREGEX\033[0m; [\033[32m.\033[0m] (a single dot) = all files")
-    ap2.add_argument("--bak-flips", action="store_true", help="[up2k] if a client uploads a bitflipped/corrupted chunk, store a copy according to --bf-nc and --bf-dir")
-    ap2.add_argument("--bf-nc", metavar="NUM", type=int, default=200, help="bak-flips: stop if there's more than \033[33mNUM\033[0m files at --kf-dir already; default: 6.3 GiB max (200*32M)")
+    ap2.add_argument("--bak-flips", action="store_true", help="[up2k] if a client uploads a bitflipped/corrupted chunk, store a copy according to \033[33m--bf-nc\033[0m and \033[33m--bf-dir\033[0m")
+    ap2.add_argument("--bf-nc", metavar="NUM", type=int, default=200, help="bak-flips: stop if there's more than \033[33mNUM\033[0m files at \033[33m--kf-dir\033[0m already; default: 6.3 GiB max (200*32M)")
     ap2.add_argument("--bf-dir", metavar="PATH", type=u, default="bf", help="bak-flips: store corrupted chunks at \033[33mPATH\033[0m; default: folder named 'bf' wherever copyparty was started")
 
 
@@ -1285,6 +1314,7 @@ def run_argparse(
     add_zeroconf(ap)
     add_zc_mdns(ap)
     add_zc_ssdp(ap)
+    add_fs(ap)
     add_upload(ap)
     add_db_general(ap, hcores)
     add_db_metadata(ap)
@@ -1350,7 +1380,7 @@ def main(argv: Optional[list[str]] = None) -> None:
         S_VERSION,
         CODENAME,
         S_BUILD_DT,
-        py_desc().replace("[", "\033[90m["),
+        PY_DESC.replace("[", "\033[90m["),
         SQLITE_VER,
         JINJA_VER,
         PYFTPD_VER,
@@ -1382,7 +1412,10 @@ def main(argv: Optional[list[str]] = None) -> None:
             supp = args_from_cfg(v)
             argv.extend(supp)
 
-    deprecated: list[tuple[str, str]] = [("--salt", "--warksalt")]
+    deprecated: list[tuple[str, str]] = [
+        ("--salt", "--warksalt"),
+        ("--hdr-au-usr", "--idp-h-usr"),
+    ]
     for dk, nk in deprecated:
         idx = -1
         ov = ""
@@ -1420,7 +1453,7 @@ def main(argv: Optional[list[str]] = None) -> None:
 
         _, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
         if hard > 0:  # -1 == infinite
-            nc = min(nc, hard // 4)
+            nc = min(nc, int(hard / 4))
     except:
         nc = 512
 
@@ -1457,40 +1490,6 @@ def main(argv: Optional[list[str]] = None) -> None:
     if al.ansi:
         al.wintitle = ""
 
-    nstrs: list[str] = []
-    anymod = False
-    for ostr in al.v or []:
-        m = re_vol.match(ostr)
-        if not m:
-            # not our problem
-            nstrs.append(ostr)
-            continue
-
-        src, dst, perms = m.groups()
-        na = [src, dst]
-        mod = False
-        for opt in perms.split(":"):
-            if re.match("c[^,]", opt):
-                mod = True
-                na.append("c," + opt[1:])
-            elif re.sub("^[rwmdgGha]*", "", opt) and "," not in opt:
-                mod = True
-                perm = opt[0]
-                na.append(perm + "," + opt[1:])
-            else:
-                na.append(opt)
-
-        nstr = ":".join(na)
-        nstrs.append(nstr if mod else ostr)
-        if mod:
-            msg = "\033[1;31mWARNING:\033[0;1m\n  -v {} \033[0;33mwas replaced with\033[0;1m\n  -v {} \n\033[0m"
-            lprint(msg.format(ostr, nstr))
-            anymod = True
-
-    if anymod:
-        al.v = nstrs
-        time.sleep(2)
-
     # propagate implications
     for k1, k2 in IMPLICATIONS:
         if getattr(al, k1):
@@ -1546,6 +1545,9 @@ def main(argv: Optional[list[str]] = None) -> None:
     if sys.version_info < (3, 6):
         al.no_scandir = True
 
+    if not hasattr(os, "sendfile"):
+        al.no_sendfile = True
+
     # signal.signal(signal.SIGINT, sighandler)
 
     SvcHub(al, dal, argv, "".join(printed)).run()

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 9, 25)
+VERSION = (1, 9, 30)
 CODENAME = "prometheable"
-BUILD_DT = (2023, 12, 1)
+BUILD_DT = (2024, 1, 25)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -72,6 +72,7 @@ class AXS(object):
         upget: Optional[Union[list[str], set[str]]] = None,
         uhtml: Optional[Union[list[str], set[str]]] = None,
         uadmin: Optional[Union[list[str], set[str]]] = None,
+        udot: Optional[Union[list[str], set[str]]] = None,
     ) -> None:
         self.uread: set[str] = set(uread or [])
         self.uwrite: set[str] = set(uwrite or [])
@@ -81,9 +82,10 @@ class AXS(object):
         self.upget: set[str] = set(upget or [])
         self.uhtml: set[str] = set(uhtml or [])
         self.uadmin: set[str] = set(uadmin or [])
+        self.udot: set[str] = set(udot or [])
 
     def __repr__(self) -> str:
-        ks = "uread uwrite umove udel uget upget uhtml uadmin".split()
+        ks = "uread uwrite umove udel uget upget uhtml uadmin udot".split()
         return "AXS(%s)" % (", ".join("%s=%r" % (k, self.__dict__[k]) for k in ks),)
 
 
@@ -336,6 +338,8 @@ class VFS(object):
         self.apget: dict[str, list[str]] = {}
         self.ahtml: dict[str, list[str]] = {}
         self.aadmin: dict[str, list[str]] = {}
+        self.adot: dict[str, list[str]] = {}
+        self.all_vols: dict[str, VFS] = {}
 
         if realpath:
             rp = realpath + ("" if realpath.endswith(os.sep) else os.sep)
@@ -377,7 +381,7 @@ class VFS(object):
 
     def add(self, src: str, dst: str) -> "VFS":
         """get existing, or add new path to the vfs"""
-        assert not src.endswith("/")  # nosec
+        assert src == "/" or not src.endswith("/")  # nosec
         assert not dst.endswith("/")  # nosec
 
         if "/" in dst:
@@ -445,8 +449,8 @@ class VFS(object):
 
     def can_access(
         self, vpath: str, uname: str
-    ) -> tuple[bool, bool, bool, bool, bool, bool, bool]:
-        """can Read,Write,Move,Delete,Get,Upget,Admin"""
+    ) -> tuple[bool, bool, bool, bool, bool, bool, bool, bool]:
+        """can Read,Write,Move,Delete,Get,Upget,Admin,Dot"""
         if vpath:
             vn, _ = self._find(undot(vpath))
         else:
@@ -454,13 +458,14 @@ class VFS(object):
 
         c = vn.axs
         return (
-            uname in c.uread or "*" in c.uread,
-            uname in c.uwrite or "*" in c.uwrite,
-            uname in c.umove or "*" in c.umove,
-            uname in c.udel or "*" in c.udel,
-            uname in c.uget or "*" in c.uget,
-            uname in c.upget or "*" in c.upget,
-            uname in c.uadmin or "*" in c.uadmin,
+            uname in c.uread,
+            uname in c.uwrite,
+            uname in c.umove,
+            uname in c.udel,
+            uname in c.uget,
+            uname in c.upget,
+            uname in c.uadmin,
+            uname in c.udot,
         )
         # skip uhtml because it's rarely needed
 
@@ -492,7 +497,7 @@ class VFS(object):
             (will_del, c.udel, "delete"),
             (will_get, c.uget, "get"),
         ]:
-            if req and (uname not in d and "*" not in d) and uname != LEELOO_DALLAS:
+            if req and uname not in d and uname != LEELOO_DALLAS:
                 if vpath != cvpath and vpath != "." and self.log:
                     ap = vn.canonical(rem)
                     t = "{} has no {} in [{}] => [{}] => [{}]"
@@ -553,7 +558,7 @@ class VFS(object):
                 for pset in permsets:
                     ok = True
                     for req, lst in zip(pset, axs):
-                        if req and uname not in lst and "*" not in lst:
+                        if req and uname not in lst:
                             ok = False
                     if ok:
                         break
@@ -577,7 +582,7 @@ class VFS(object):
         seen: list[str],
         uname: str,
         permsets: list[list[bool]],
-        dots: bool,
+        wantdots: bool,
         scandir: bool,
         lstat: bool,
         subvols: bool = True,
@@ -621,6 +626,10 @@ class VFS(object):
                     rm1.append(le)
             _ = [vfs_ls.remove(x) for x in rm1]  # type: ignore
 
+        dots_ok = wantdots and uname in dbv.axs.udot
+        if not dots_ok:
+            vfs_ls = [x for x in vfs_ls if "/." not in "/" + x[0]]
+
         seen = seen[:] + [fsroot]
         rfiles = [x for x in vfs_ls if not stat.S_ISDIR(x[1].st_mode)]
         rdirs = [x for x in vfs_ls if stat.S_ISDIR(x[1].st_mode)]
@@ -633,13 +642,13 @@ class VFS(object):
         yield dbv, vrem, rel, fsroot, rfiles, rdirs, vfs_virt
 
         for rdir, _ in rdirs:
-            if not dots and rdir.startswith("."):
+            if not dots_ok and rdir.startswith("."):
                 continue
 
             wrel = (rel + "/" + rdir).lstrip("/")
             wrem = (rem + "/" + rdir).lstrip("/")
             for x in self.walk(
-                wrel, wrem, seen, uname, permsets, dots, scandir, lstat, subvols
+                wrel, wrem, seen, uname, permsets, wantdots, scandir, lstat, subvols
             ):
                 yield x
 
@@ -647,11 +656,13 @@ class VFS(object):
             return
 
         for n, vfs in sorted(vfs_virt.items()):
-            if not dots and n.startswith("."):
+            if not dots_ok and n.startswith("."):
                 continue
 
             wrel = (rel + "/" + n).lstrip("/")
-            for x in vfs.walk(wrel, "", seen, uname, permsets, dots, scandir, lstat):
+            for x in vfs.walk(
+                wrel, "", seen, uname, permsets, wantdots, scandir, lstat
+            ):
                 yield x
 
     def zipgen(
@@ -660,7 +671,6 @@ class VFS(object):
         vrem: str,
         flt: set[str],
         uname: str,
-        dots: bool,
         dirs: bool,
         scandir: bool,
         wrap: bool = True,
@@ -670,7 +680,7 @@ class VFS(object):
         # if single folder: the folder itself is the top-level item
         folder = "" if flt or not wrap else (vpath.split("/")[-1].lstrip(".") or "top")
 
-        g = self.walk(folder, vrem, [], uname, [[True, False]], dots, scandir, False)
+        g = self.walk(folder, vrem, [], uname, [[True, False]], True, scandir, False)
         for _, _, vpath, apath, files, rd, vd in g:
             if flt:
                 files = [x for x in files if x[0] in flt]
@@ -689,18 +699,6 @@ class VFS(object):
             apaths = [os.path.join(apath, n) for n in fnames]
             ret = list(zip(vpaths, apaths, files))
 
-            if not dots:
-                # dotfile filtering based on vpath (intended visibility)
-                ret = [x for x in ret if "/." not in "/" + x[0]]
-
-                zel = [ze for ze in rd if ze[0].startswith(".")]
-                for ze in zel:
-                    rd.remove(ze)
-
-                zsl = [zs for zs in vd.keys() if zs.startswith(".")]
-                for zs in zsl:
-                    del vd[zs]
-
             for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in ret]:
                 yield f
 
@@ -781,7 +779,6 @@ class AuthSrv(object):
         self.warn_anonwrite = warn_anonwrite
         self.line_ctr = 0
         self.indent = ""
-        self.desc = []
 
         self.mutex = threading.Lock()
         self.reload()
@@ -864,7 +861,6 @@ class AuthSrv(object):
         mflags: dict[str, dict[str, Any]],
         mount: dict[str, str],
     ) -> None:
-        self.desc = []
         self.line_ctr = 0
 
         expand_config_file(cfg_lines, fp, "")
@@ -958,16 +954,17 @@ class AuthSrv(object):
                 try:
                     self._l(ln, 5, "volume access config:")
                     sk, sv = ln.split(":")
-                    if re.sub("[rwmdgGha]", "", sk) or not sk:
+                    if re.sub("[rwmdgGhaA.]", "", sk) or not sk:
                         err = "invalid accs permissions list; "
                         raise Exception(err)
                     if " " in re.sub(", *", "", sv).strip():
                         err = "list of users is not comma-separated; "
                         raise Exception(err)
+                    assert vp is not None
                     self._read_vol_str(sk, sv.replace(" ", ""), daxs[vp], mflags[vp])
                     continue
                 except:
-                    err += "accs entries must be 'rwmdgGha: user1, user2, ...'"
+                    err += "accs entries must be 'rwmdgGhaA.: user1, user2, ...'"
                     raise Exception(err + SBADCFG)
 
             if cat == catf:
@@ -986,9 +983,11 @@ class AuthSrv(object):
                             fstr += "," + sk
                         else:
                             fstr += ",{}={}".format(sk, sv)
+                            assert vp is not None
                             self._read_vol_str("c", fstr[1:], daxs[vp], mflags[vp])
                             fstr = ""
                     if fstr:
+                        assert vp is not None
                         self._read_vol_str("c", fstr[1:], daxs[vp], mflags[vp])
                     continue
                 except:
@@ -1003,10 +1002,12 @@ class AuthSrv(object):
     def _read_vol_str(
         self, lvl: str, uname: str, axs: AXS, flags: dict[str, Any]
     ) -> None:
-        if lvl.strip("crwmdgGha"):
-            raise Exception("invalid volflag: {},{}".format(lvl, uname))
+        if lvl.strip("crwmdgGhaA."):
+            t = "%s,%s" % (lvl, uname) if uname else lvl
+            raise Exception("invalid config value (volume or volflag): %s" % (t,))
 
         if lvl == "c":
+            # here, 'uname' is not a username; it is a volflag name... sorry
             cval: Union[bool, str] = True
             try:
                 # volflag with arguments, possibly with a preceding list of bools
@@ -1026,19 +1027,31 @@ class AuthSrv(object):
         if uname == "":
             uname = "*"
 
+        junkset = set()
         for un in uname.replace(",", " ").strip().split():
+            for alias, mapping in [
+                ("h", "gh"),
+                ("G", "gG"),
+                ("A", "rwmda.A"),
+            ]:
+                expanded = ""
+                for ch in mapping:
+                    if ch not in lvl:
+                        expanded += ch
+                    lvl = lvl.replace(alias, expanded + alias)
+
             for ch, al in [
                 ("r", axs.uread),
                 ("w", axs.uwrite),
                 ("m", axs.umove),
                 ("d", axs.udel),
+                (".", axs.udot),
                 ("a", axs.uadmin),
-                ("h", axs.uhtml),
-                ("h", axs.uget),
+                ("A", junkset),
                 ("g", axs.uget),
-                ("G", axs.uget),
                 ("G", axs.upget),
-            ]:  # b bb bbb
+                ("h", axs.uhtml),
+            ]:
                 if ch in lvl:
                     if un == "*":
                         t = "└─add permission [{0}] for [everyone] -- {2}"
@@ -1110,7 +1123,7 @@ class AuthSrv(object):
 
         if self.args.v:
             # list of src:dst:permset:permset:...
-            # permset is <rwmdgGha>[,username][,username] or <c>,<flag>[=args]
+            # permset is <rwmdgGhaA.>[,username][,username] or <c>,<flag>[=args]
             for v_str in self.args.v:
                 m = re_vol.match(v_str)
                 if not m:
@@ -1200,20 +1213,28 @@ class AuthSrv(object):
             vol.all_vps.sort(key=lambda x: len(x[0]), reverse=True)
             vol.root = vfs
 
-        for perm in "read write move del get pget html admin".split():
+        for perm in "read write move del get pget html admin dot".split():
             axs_key = "u" + perm
             unames = ["*"] + list(acct.keys())
+            for vp, vol in vfs.all_vols.items():
+                zx = getattr(vol.axs, axs_key)
+                if "*" in zx:
+                    for usr in unames:
+                        zx.add(usr)
+
+            # aread,... = dict[uname, list[volnames] or []]
             umap: dict[str, list[str]] = {x: [] for x in unames}
             for usr in unames:
                 for vp, vol in vfs.all_vols.items():
                     zx = getattr(vol.axs, axs_key)
-                    if usr in zx or "*" in zx:
+                    if usr in zx:
                         umap[usr].append(vp)
                 umap[usr].sort()
             setattr(vfs, "a" + perm, umap)
 
         all_users = {}
         missing_users = {}
+        associated_users = {}
         for axs in daxs.values():
             for d in [
                 axs.uread,
@@ -1224,11 +1245,14 @@ class AuthSrv(object):
                 axs.upget,
                 axs.uhtml,
                 axs.uadmin,
+                axs.udot,
             ]:
                 for usr in d:
                     all_users[usr] = 1
                     if usr != "*" and usr not in acct:
                         missing_users[usr] = 1
+                    if "*" not in d:
+                        associated_users[usr] = 1
 
         if missing_users:
             self.log(
@@ -1249,6 +1273,16 @@ class AuthSrv(object):
                 raise Exception(BAD_CFG)
             seenpwds[pwd] = usr
 
+        for usr in acct:
+            if usr not in associated_users:
+                if len(vfs.all_vols) > 1:
+                    # user probably familiar enough that the verbose message is not necessary
+                    t = "account [%s] is not mentioned in any volume definitions; see --help-accounts"
+                    self.log(t % (usr,), 1)
+                else:
+                    t = "WARNING: the account [%s] is not mentioned in any volume definitions and thus has the same access-level and privileges that guests have; please see --help-accounts for details.  For example, if you intended to give that user full access to the current directory, you could do this:  -v .::A,%s"
+                    self.log(t % (usr, usr), 1)
+
         promote = []
         demote = []
         for vol in vfs.all_vols.values():
@@ -1459,6 +1493,14 @@ class AuthSrv(object):
                 if k in vol.flags:
                     vol.flags[k] = float(vol.flags[k])
 
+            try:
+                zs1, zs2 = vol.flags["rm_retry"].split("/")
+                vol.flags["rm_re_t"] = float(zs1)
+                vol.flags["rm_re_r"] = float(zs2)
+            except:
+                t = 'volume "/%s" has invalid rm_retry [%s]'
+                raise Exception(t % (vol.vpath, vol.flags.get("rm_retry")))
+
             for k1, k2 in IMPLICATIONS:
                 if k1 in vol.flags:
                     vol.flags[k2] = True
@@ -1470,8 +1512,8 @@ class AuthSrv(object):
             dbds = "acid|swal|wal|yolo"
             vol.flags["dbd"] = dbd = vol.flags.get("dbd") or self.args.dbd
             if dbd not in dbds.split("|"):
-                t = "invalid dbd [{}]; must be one of [{}]"
-                raise Exception(t.format(dbd, dbds))
+                t = 'volume "/%s" has invalid dbd [%s]; must be one of [%s]'
+                raise Exception(t % (vol.vpath, dbd, dbds))
 
             # default tag cfgs if unset
             for k in ("mte", "mth", "exp_md", "exp_lg"):
@@ -1632,6 +1674,11 @@ class AuthSrv(object):
                     vol.flags.pop(k[1:], None)
                     vol.flags.pop(k)
 
+        for vol in vfs.all_vols.values():
+            if vol.flags.get("dots"):
+                for name in vol.axs.uread:
+                    vol.axs.udot.add(name)
+
         if errors:
             sys.exit(1)
 
@@ -1650,12 +1697,14 @@ class AuthSrv(object):
                 [" write", "uwrite"],
                 ["  move", "umove"],
                 ["delete", "udel"],
+                ["  dots", "udot"],
                 ["   get", "uget"],
-                [" upget", "upget"],
+                [" upGet", "upget"],
                 ["  html", "uhtml"],
                 ["uadmin", "uadmin"],
             ]:
                 u = list(sorted(getattr(zv.axs, attr)))
+                u = ["*"] if "*" in u else u
                 u = ", ".join("\033[35meverybody\033[0m" if x == "*" else x for x in u)
                 u = u if u else "\033[36m--none--\033[0m"
                 t += "\n|  {}:  {}".format(txt, u)
@@ -1812,7 +1861,7 @@ class AuthSrv(object):
                 raise Exception("volume not found: " + zs)
 
         self.log(str({"users": users, "vols": vols, "flags": flags}))
-        t = "/{}: read({}) write({}) move({}) del({}) get({}) upget({}) uadmin({})"
+        t = "/{}: read({}) write({}) move({}) del({}) dots({}) get({}) upGet({}) uadmin({})"
         for k, zv in self.vfs.all_vols.items():
             vc = zv.axs
             vs = [
@@ -1821,6 +1870,7 @@ class AuthSrv(object):
                 vc.uwrite,
                 vc.umove,
                 vc.udel,
+                vc.udot,
                 vc.uget,
                 vc.upget,
                 vc.uhtml,
@@ -1963,6 +2013,7 @@ class AuthSrv(object):
                 "w": "uwrite",
                 "m": "umove",
                 "d": "udel",
+                ".": "udot",
                 "g": "uget",
                 "G": "upget",
                 "h": "uhtml",
@@ -2169,7 +2220,7 @@ def upgrade_cfg_fmt(
             else:
                 sn = sn.replace(",", ", ")
             ret.append("    " + sn)
-        elif sn[:1] in "rwmdgGha":
+        elif sn[:1] in "rwmdgGhaA.":
             if cat != catx:
                 cat = catx
                 ret.append(cat)

copyparty/bos/bos.py

@@ -43,6 +43,10 @@ def open(p: str, *a, **ka) -> int:
     return os.open(fsenc(p), *a, **ka)
 
 
+def readlink(p: str) -> str:
+    return fsdec(os.readlink(fsenc(p)))
+
+
 def rename(src: str, dst: str) -> None:
     return os.rename(fsenc(src), fsenc(dst))
 

copyparty/broker_mpw.py

@@ -76,7 +76,7 @@ class MpWorker(BrokerCli):
         pass
 
     def logw(self, msg: str, c: Union[int, str] = 0) -> None:
-        self.log("mp{}".format(self.n), msg, c)
+        self.log("mp%d" % (self.n,), msg, c)
 
     def main(self) -> None:
         while True:

copyparty/cfg.py

@@ -9,6 +9,9 @@ onedash = set(zs.split())
 def vf_bmap() -> dict[str, str]:
     """argv-to-volflag: simple bools"""
     ret = {
+        "dav_auth": "davauth",
+        "dav_rt": "davrt",
+        "ed": "dots",
         "never_symlink": "neversymlink",
         "no_dedup": "copydupes",
         "no_dupe": "nodupe",
@@ -18,8 +21,6 @@ def vf_bmap() -> dict[str, str]:
         "no_vthumb": "dvthumb",
         "no_athumb": "dathumb",
         "th_no_crop": "nocrop",
-        "dav_auth": "davauth",
-        "dav_rt": "davrt",
     }
     for k in (
         "dotsrch",
@@ -61,6 +62,7 @@ def vf_vmap() -> dict[str, str]:
         "lg_sbf",
         "md_sbf",
         "nrand",
+        "rm_retry",
         "sort",
         "unlist",
         "u2ts",
@@ -98,10 +100,12 @@ permdescs = {
     "w": 'write; upload files; need "r" to see the uploads',
     "m": 'move; move files and folders; need "w" at destination',
     "d": "delete; permanently delete files and folders",
+    ".": "dots; user can ask to show dotfiles in listings",
     "g": "get; download files, but cannot see folder contents",
     "G": 'upget; same as "g" but can see filekeys of their own uploads',
     "h": 'html; same as "g" but folders return their index.html',
     "a": "admin; can see uploader IPs, config-reload",
+    "A": "all; same as 'rwmda.' (read/write/move/delete/dotfiles)",
 }
 
 
@@ -202,8 +206,10 @@ flagcats = {
         "nohtml": "return html and markdown as text/html",
     },
     "others": {
+        "dots": "allow all users with read-access to\nenable the option to show dotfiles in listings",
         "fk=8": 'generates per-file accesskeys,\nwhich are then required at the "g" permission;\nkeys are invalidated if filesize or inode changes',
         "fka=8": 'generates slightly weaker per-file accesskeys,\nwhich are then required at the "g" permission;\nnot affected by filesize or inode numbers',
+        "rm_retry": "ms-windows: timeout for deleting busy files",
         "davauth": "ask webdav clients to login for all folders",
         "davrt": "show lastmod time of symlink destination, not the link itself\n(note: this option is always enabled for recursive listings)",
     },

copyparty/ftpd.py

@@ -73,6 +73,7 @@ class FtpAuth(DummyAuthorizer):
         asrv = self.hub.asrv
         uname = "*"
         if username != "anonymous":
+            uname = ""
             for zs in (password, username):
                 zs = asrv.iacct.get(asrv.ah.hash(zs), "")
                 if zs:
@@ -132,7 +133,7 @@ class FtpFs(AbstractedFS):
 
         self.can_read = self.can_write = self.can_move = False
         self.can_delete = self.can_get = self.can_upget = False
-        self.can_admin = False
+        self.can_admin = self.can_dot = False
 
         self.listdirinfo = self.listdir
         self.chdir(".")
@@ -167,7 +168,7 @@ class FtpFs(AbstractedFS):
                 if not avfs:
                     raise FSE(t.format(vpath), 1)
 
-                cr, cw, cm, cd, _, _, _ = avfs.can_access("", self.h.uname)
+                cr, cw, cm, cd, _, _, _, _ = avfs.can_access("", self.h.uname)
                 if r and not cr or w and not cw or m and not cm or d and not cd:
                     raise FSE(t.format(vpath), 1)
 
@@ -243,6 +244,7 @@ class FtpFs(AbstractedFS):
             self.can_get,
             self.can_upget,
             self.can_admin,
+            self.can_dot,
         ) = avfs.can_access("", self.h.uname)
 
     def mkdir(self, path: str) -> None:
@@ -265,7 +267,7 @@ class FtpFs(AbstractedFS):
             vfs_ls = [x[0] for x in vfs_ls1]
             vfs_ls.extend(vfs_virt.keys())
 
-            if not self.args.ed:
+            if not self.can_dot:
                 vfs_ls = exclude_dotfiles(vfs_ls)
 
             vfs_ls.sort()

copyparty/httpcli.py

@@ -37,6 +37,8 @@ from .star import StreamTar
 from .sutil import StreamArc, gfilter
 from .szip import StreamZip
 from .util import (
+    APPLESAN_RE,
+    BITNESS,
     HTTPCODE,
     META_NOBOTS,
     UTC,
@@ -45,6 +47,7 @@ from .util import (
     ODict,
     Pebkac,
     UnrecvEOF,
+    WrongPostKey,
     absreal,
     alltrace,
     atomic_move,
@@ -86,6 +89,7 @@ from .util import (
     vjoin,
     vol_san,
     vsplit,
+    wunlink,
     yieldfile,
 )
 
@@ -154,10 +158,6 @@ class HttpCli(object):
         self.pw = " "
         self.rvol = [" "]
         self.wvol = [" "]
-        self.mvol = [" "]
-        self.dvol = [" "]
-        self.gvol = [" "]
-        self.upvol = [" "]
         self.avol = [" "]
         self.do_log = True
         self.can_read = False
@@ -167,6 +167,7 @@ class HttpCli(object):
         self.can_get = False
         self.can_upget = False
         self.can_admin = False
+        self.can_dot = False
         self.out_headerlist: list[tuple[str, str]] = []
         self.out_headers: dict[str, str] = {}
         self.html_head = " "
@@ -192,7 +193,7 @@ class HttpCli(object):
 
     def unpwd(self, m: Match[str]) -> str:
         a, b, c = m.groups()
-        return "{}\033[7m {} \033[27m{}".format(a, self.asrv.iacct[b], c)
+        return "%s\033[7m %s \033[27m%s" % (a, self.asrv.iacct[b], c)
 
     def _check_nonfatal(self, ex: Pebkac, post: bool) -> bool:
         if post:
@@ -455,9 +456,9 @@ class HttpCli(object):
             except:
                 pass
 
-        if self.args.hdr_au_usr:
+        if self.args.idp_h_usr:
             self.pw = ""
-            self.uname = self.headers.get(self.args.hdr_au_usr) or "*"
+            self.uname = self.headers.get(self.args.idp_h_usr) or "*"
             if self.uname not in self.asrv.vfs.aread:
                 self.log("unknown username: [%s]" % (self.uname), 1)
                 self.uname = "*"
@@ -467,10 +468,6 @@ class HttpCli(object):
 
         self.rvol = self.asrv.vfs.aread[self.uname]
         self.wvol = self.asrv.vfs.awrite[self.uname]
-        self.mvol = self.asrv.vfs.amove[self.uname]
-        self.dvol = self.asrv.vfs.adel[self.uname]
-        self.gvol = self.asrv.vfs.aget[self.uname]
-        self.upvol = self.asrv.vfs.apget[self.uname]
         self.avol = self.asrv.vfs.aadmin[self.uname]
 
         if self.pw and (
@@ -503,8 +500,9 @@ class HttpCli(object):
             self.can_get,
             self.can_upget,
             self.can_admin,
+            self.can_dot,
         ) = (
-            avn.can_access("", self.uname) if avn else [False] * 7
+            avn.can_access("", self.uname) if avn else [False] * 8
         )
         self.avn = avn
         self.vn = vn
@@ -562,16 +560,16 @@ class HttpCli(object):
                     self.keepalive = False
 
                 em = str(ex)
-                msg = em if pex == ex else min_ex()
+                msg = em if pex is ex else min_ex()
                 if pex.code != 404 or self.do_log:
                     self.log(
-                        "{}\033[0m, {}".format(msg, self.vpath),
+                        "%s\033[0m, %s" % (msg, self.vpath),
                         6 if em.startswith("client d/c ") else 3,
                     )
 
-                msg = "{}\r\nURL: {}\r\n".format(em, self.vpath)
+                msg = "%s\r\nURL: %s\r\n" % (em, self.vpath)
                 if self.hint:
-                    msg += "hint: {}\r\n".format(self.hint)
+                    msg += "hint: %s\r\n" % (self.hint,)
 
                 if "database is locked" in em:
                     self.conn.hsrv.broker.say("log_stacks")
@@ -814,7 +812,7 @@ class HttpCli(object):
             if k in skip:
                 continue
 
-            t = "{}={}".format(quotep(k), quotep(v))
+            t = "%s=%s" % (quotep(k), quotep(v))
             ret.append(t.replace(" ", "+").rstrip("="))
 
         if not ret:
@@ -862,21 +860,22 @@ class HttpCli(object):
         oh = self.out_headers
         origin = origin.lower()
         good_origins = self.args.acao + [
-            "{}://{}".format(
+            "%s://%s"
+            % (
                 "https" if self.is_https else "http",
                 self.host.lower().split(":")[0],
             )
         ]
-        if re.sub(r"(:[0-9]{1,5})?/?$", "", origin) in good_origins:
+        if "pw" in ih or re.sub(r"(:[0-9]{1,5})?/?$", "", origin) in good_origins:
             good_origin = True
             bad_hdrs = ("",)
         else:
             good_origin = False
             bad_hdrs = ("", "pw")
 
-        # '*' blocks all credentials (cookies, http-auth);
+        # '*' blocks auth through cookies / WWW-Authenticate;
         # exact-match for Origin is necessary to unlock those,
-        # however yolo-requests (?pw=) are always allowed
+        # but the ?pw= param and PW: header are always allowed
         acah = ih.get("access-control-request-headers", "")
         acao = (origin if good_origin else None) or (
             "*" if "*" in good_origins else None
@@ -1059,7 +1058,7 @@ class HttpCli(object):
             self.can_read = self.can_write = self.can_get = False
 
         if not self.can_read and not self.can_write and not self.can_get:
-            self.log("inaccessible: [{}]".format(self.vpath))
+            self.log("inaccessible: [%s]" % (self.vpath,))
             raise Pebkac(401, "authenticate")
 
         from .dxml import parse_xml
@@ -1131,7 +1130,6 @@ class HttpCli(object):
                 rem,
                 set(),
                 self.uname,
-                self.args.ed,
                 True,
                 not self.args.no_scandir,
                 wrap=False,
@@ -1145,7 +1143,7 @@ class HttpCli(object):
                 [[True, False]],
                 lstat="davrt" not in vn.flags,
             )
-            if not self.args.ed:
+            if not self.can_dot:
                 names = set(exclude_dotfiles([x[0] for x in vfs_ls]))
                 vfs_ls = [x for x in vfs_ls if x[0] in names]
 
@@ -1389,8 +1387,7 @@ class HttpCli(object):
             return False
 
         vp = "/" + self.vpath
-        ptn = r"/\.(_|DS_Store|Spotlight-|fseventsd|Trashes|AppleDouble)|/__MACOS"
-        if re.search(ptn, vp):
+        if re.search(APPLESAN_RE, vp):
             zt = '<?xml version="1.0" encoding="utf-8"?>\n<D:error xmlns:D="DAV:"><D:lock-token-submitted><D:href>{}</D:href></D:lock-token-submitted></D:error>'
             zb = zt.format(vp).encode("utf-8", "replace")
             self.reply(zb, 423, "text/xml; charset=utf-8")
@@ -1410,7 +1407,7 @@ class HttpCli(object):
         if txt and len(txt) == orig_len:
             raise Pebkac(500, "chunk slicing failed")
 
-        buf = "{:x}\r\n".format(len(buf)).encode(enc) + buf
+        buf = ("%x\r\n" % (len(buf),)).encode(enc) + buf
         self.s.sendall(buf + b"\r\n")
         return txt
 
@@ -1696,7 +1693,7 @@ class HttpCli(object):
                 and bos.path.getmtime(path) >= time.time() - self.args.blank_wt
             ):
                 # small toctou, but better than clobbering a hardlink
-                bos.unlink(path)
+                wunlink(self.log, path, vfs.flags)
 
         with ren_open(fn, *open_a, **params) as zfw:
             f, fn = zfw["orz"]
@@ -1710,7 +1707,7 @@ class HttpCli(object):
                 lim.chk_sz(post_sz)
                 lim.chk_vsz(self.conn.hsrv.broker, vfs.realpath, post_sz)
             except:
-                bos.unlink(path)
+                wunlink(self.log, path, vfs.flags)
                 raise
 
         if self.args.nw:
@@ -1763,7 +1760,7 @@ class HttpCli(object):
         ):
             t = "upload blocked by xau server config"
             self.log(t, 1)
-            os.unlink(path)
+            wunlink(self.log, path, vfs.flags)
             raise Pebkac(403, t)
 
         vfs, rem = vfs.get_dbv(rem)
@@ -1869,7 +1866,16 @@ class HttpCli(object):
         self.parser = MultipartParser(self.log, self.sr, self.headers)
         self.parser.parse()
 
-        act = self.parser.require("act", 64)
+        file0: list[tuple[str, Optional[str], Generator[bytes, None, None]]] = []
+        try:
+            act = self.parser.require("act", 64)
+        except WrongPostKey as ex:
+            if ex.got == "f" and ex.fname:
+                self.log("missing 'act', but looks like an upload so assuming that")
+                file0 = [(ex.got, ex.fname, ex.datagen)]
+                act = "bput"
+            else:
+                raise
 
         if act == "login":
             return self.handle_login()
@@ -1882,7 +1888,7 @@ class HttpCli(object):
             return self.handle_new_md()
 
         if act == "bput":
-            return self.handle_plain_upload()
+            return self.handle_plain_upload(file0)
 
         if act == "tput":
             return self.handle_text_upload()
@@ -1910,7 +1916,7 @@ class HttpCli(object):
         items = [unquotep(x) for x in items if items]
 
         self.parser.drop()
-        return self.tx_zip(k, v, "", vn, rem, items, self.args.ed)
+        return self.tx_zip(k, v, "", vn, rem, items)
 
     def handle_post_json(self) -> bool:
         try:
@@ -1996,10 +2002,10 @@ class HttpCli(object):
     def handle_search(self, body: dict[str, Any]) -> bool:
         idx = self.conn.get_u2idx()
         if not idx or not hasattr(idx, "p_end"):
-            raise Pebkac(500, "sqlite3 is not available on the server; cannot search")
+            raise Pebkac(500, "server busy, or sqlite3 not available; cannot search")
 
-        vols = []
-        seen = {}
+        vols: list[VFS] = []
+        seen: dict[VFS, bool] = {}
         for vtop in self.rvol:
             vfs, _ = self.asrv.vfs.get(vtop, self.uname, True, False)
             vfs = vfs.dbv or vfs
@@ -2007,7 +2013,7 @@ class HttpCli(object):
                 continue
 
             seen[vfs] = True
-            vols.append((vfs.vpath, vfs.realpath, vfs.flags))
+            vols.append(vfs)
 
         t0 = time.time()
         if idx.p_end:
@@ -2022,7 +2028,7 @@ class HttpCli(object):
             vbody = copy.deepcopy(body)
             vbody["hash"] = len(vbody["hash"])
             self.log("qj: " + repr(vbody))
-            hits = idx.fsearch(vols, body)
+            hits = idx.fsearch(self.uname, vols, body)
             msg: Any = repr(hits)
             taglist: list[str] = []
             trunc = False
@@ -2031,7 +2037,7 @@ class HttpCli(object):
             q = body["q"]
             n = body.get("n", self.args.srch_hits)
             self.log("qj: {} |{}|".format(q, n))
-            hits, taglist, trunc = idx.search(vols, q, n)
+            hits, taglist, trunc = idx.search(self.uname, vols, q, n)
             msg = len(hits)
 
         idx.p_end = time.time()
@@ -2321,7 +2327,9 @@ class HttpCli(object):
             vfs.flags.get("xau") or [],
         )
 
-    def handle_plain_upload(self) -> bool:
+    def handle_plain_upload(
+        self, file0: list[tuple[str, Optional[str], Generator[bytes, None, None]]]
+    ) -> bool:
         assert self.parser
         nullwrite = self.args.nw
         vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
@@ -2343,11 +2351,13 @@ class HttpCli(object):
         files: list[tuple[int, str, str, str, str, str]] = []
         # sz, sha_hex, sha_b64, p_file, fname, abspath
         errmsg = ""
+        tabspath = ""
         dip = self.dip()
         t0 = time.time()
         try:
             assert self.parser.gen
-            for nfile, (p_field, p_file, p_data) in enumerate(self.parser.gen):
+            gens = itertools.chain(file0, self.parser.gen)
+            for nfile, (p_field, p_file, p_data) in enumerate(gens):
                 if not p_file:
                     self.log("discarding incoming file without filename")
                     # fallthrough
@@ -2431,14 +2441,16 @@ class HttpCli(object):
                             lim.chk_nup(self.ip)
                         except:
                             if not nullwrite:
-                                bos.unlink(tabspath)
-                                bos.unlink(abspath)
+                                wunlink(self.log, tabspath, vfs.flags)
+                                wunlink(self.log, abspath, vfs.flags)
                             fname = os.devnull
                             raise
 
                     if not nullwrite:
                         atomic_move(tabspath, abspath)
 
+                    tabspath = ""
+
                     files.append(
                         (sz, sha_hex, sha_b64, p_file or "(discarded)", fname, abspath)
                     )
@@ -2458,7 +2470,7 @@ class HttpCli(object):
                     ):
                         t = "upload blocked by xau server config"
                         self.log(t, 1)
-                        os.unlink(abspath)
+                        wunlink(self.log, abspath, vfs.flags)
                         raise Pebkac(403, t)
 
                     dbv, vrem = vfs.get_dbv(rem)
@@ -2484,6 +2496,12 @@ class HttpCli(object):
             errmsg = vol_san(
                 list(self.asrv.vfs.all_vols.values()), unicode(ex).encode("utf-8")
             ).decode("utf-8")
+            try:
+                got = bos.path.getsize(tabspath)
+                t = "connection lost after receiving %s of the file"
+                self.log(t % (humansize(got),), 3)
+            except:
+                pass
 
         td = max(0.1, time.time() - t0)
         sz_total = sum(x[0] for x in files)
@@ -2696,7 +2714,7 @@ class HttpCli(object):
                 raise Pebkac(403, t)
 
         if bos.path.exists(fp):
-            bos.unlink(fp)
+            wunlink(self.log, fp, vfs.flags)
 
         with open(fsenc(fp), "wb", 512 * 1024) as f:
             sz, sha512, _ = hashcopy(p_data, f, self.args.s_wr_slp)
@@ -2708,7 +2726,7 @@ class HttpCli(object):
                 lim.chk_sz(sz)
                 lim.chk_vsz(self.conn.hsrv.broker, vfs.realpath, sz)
             except:
-                bos.unlink(fp)
+                wunlink(self.log, fp, vfs.flags)
                 raise
 
         new_lastmod = bos.stat(fp).st_mtime
@@ -2731,7 +2749,7 @@ class HttpCli(object):
         ):
             t = "save blocked by xau server config"
             self.log(t, 1)
-            os.unlink(fp)
+            wunlink(self.log, fp, vfs.flags)
             raise Pebkac(403, t)
 
         vfs, rem = vfs.get_dbv(rem)
@@ -2943,9 +2961,11 @@ class HttpCli(object):
             # 512 kB is optimal for huge files, use 64k
             open_args = [fsenc(fs_path), "rb", 64 * 1024]
             use_sendfile = (
-                not self.tls  #
+                # fmt: off
+                not self.tls
                 and not self.args.no_sendfile
-                and hasattr(os, "sendfile")
+                and (BITNESS > 32 or file_sz < 0x7fffFFFF)
+                # fmt: on
             )
 
         #
@@ -2967,18 +2987,19 @@ class HttpCli(object):
             mime = "text/plain; charset=utf-8"
 
         self.out_headers["Accept-Ranges"] = "bytes"
-        self.send_headers(length=upper - lower, status=status, mime=mime)
-
         logmsg += unicode(status) + logtail
 
         if self.mode == "HEAD" or not do_send:
             if self.do_log:
                 self.log(logmsg)
 
+            self.send_headers(length=upper - lower, status=status, mime=mime)
             return True
 
         ret = True
         with open_func(*open_args) as f:
+            self.send_headers(length=upper - lower, status=status, mime=mime)
+
             sendfun = sendfile_kern if use_sendfile else sendfile_py
             remains = sendfun(
                 self.log, lower, upper, f, self.s, self.args.s_wr_sz, self.args.s_wr_slp
@@ -2986,7 +3007,7 @@ class HttpCli(object):
 
         if remains > 0:
             logmsg += " \033[31m" + unicode(upper - remains) + "\033[0m"
-            self.keepalive = False
+            ret = False
 
         spd = self._spd((upper - lower) - remains)
         if self.do_log:
@@ -3002,7 +3023,6 @@ class HttpCli(object):
         vn: VFS,
         rem: str,
         items: list[str],
-        dots: bool,
     ) -> bool:
         if self.args.no_zip:
             raise Pebkac(400, "not enabled")
@@ -3059,7 +3079,7 @@ class HttpCli(object):
         self.send_headers(None, mime=mime, headers={"Content-Disposition": cdis})
 
         fgen = vn.zipgen(
-            vpath, rem, set(items), self.uname, dots, False, not self.args.no_scandir
+            vpath, rem, set(items), self.uname, False, not self.args.no_scandir
         )
         # for f in fgen: print(repr({k: f[k] for k in ["vp", "ap"]}))
         cfmt = ""
@@ -3370,11 +3390,19 @@ class HttpCli(object):
         rc = 404
         if self.args.vague_403:
             t = '<h1 id="n">404 not found &nbsp;┐( ´ -`)┌</h1><p id="o">or maybe you don\'t have access -- try logging in or <a href="{}/?h">go home</a></p>'
+            pt = "404 not found  ┐( ´ -`)┌   (or maybe you don't have access -- try logging in)"
         elif is_403:
             t = '<h1 id="p">403 forbiddena &nbsp;~┻━┻</h1><p id="q">you\'ll have to log in or <a href="{}/?h">go home</a></p>'
+            pt = "403 forbiddena ~┻━┻   (you'll have to log in)"
             rc = 403
         else:
             t = '<h1 id="n">404 not found &nbsp;┐( ´ -`)┌</h1><p><a id="r" href="{}/?h">go home</a></p>'
+            pt = "404 not found  ┐( ´ -`)┌"
+
+        if self.ua.startswith("curl/") or self.ua.startswith("fetch"):
+            pt = "# acct: %s\n%s\n" % (self.uname, pt)
+            self.reply(pt.encode("utf-8"), status=rc)
+            return True
 
         t = t.format(self.args.SR)
         qv = quotep(self.vpaths) + self.ourlq()
@@ -3473,6 +3501,7 @@ class HttpCli(object):
             ret["k" + quotep(excl)] = sub
 
         vfs = self.asrv.vfs
+        dots = False
         try:
             vn, rem = vfs.get(top, self.uname, True, False)
             fsroot, vfs_ls, vfs_virt = vn.ls(
@@ -3481,6 +3510,7 @@ class HttpCli(object):
                 not self.args.no_scandir,
                 [[True, False], [False, True]],
             )
+            dots = self.uname in vn.axs.udot
         except:
             vfs_ls = []
             vfs_virt = {}
@@ -3489,15 +3519,12 @@ class HttpCli(object):
                 if d1 == top:
                     vfs_virt[d2] = vfs  # typechk, value never read
 
-        dirs = []
+        dirs = [x[0] for x in vfs_ls if stat.S_ISDIR(x[1].st_mode)]
 
-        dirnames = [x[0] for x in vfs_ls if stat.S_ISDIR(x[1].st_mode)]
+        if not dots or "dots" not in self.uparam:
+            dirs = exclude_dotfiles(dirs)
 
-        if not self.args.ed or "dots" not in self.uparam:
-            dirnames = exclude_dotfiles(dirnames)
-
-        for fn in [x for x in dirnames if x != excl]:
-            dirs.append(quotep(fn))
+        dirs = [quotep(x) for x in dirs if x != excl]
 
         for x in vfs_virt:
             if x != excl:
@@ -3529,7 +3556,8 @@ class HttpCli(object):
         fk_vols = {
             vol: (vol.flags["fk"], 2 if "fka" in vol.flags else 1)
             for vp, vol in self.asrv.vfs.all_vols.items()
-            if "fk" in vol.flags and (vp in self.rvol or vp in self.upvol)
+            if "fk" in vol.flags
+            and (self.uname in vol.axs.uread or self.uname in vol.axs.upget)
         }
         for vol in self.asrv.vfs.all_vols.values():
             cur = idx.get_cur(vol.realpath)
@@ -3800,7 +3828,7 @@ class HttpCli(object):
 
         elif self.can_get and self.avn:
             axs = self.avn.axs
-            if self.uname not in axs.uhtml and "*" not in axs.uhtml:
+            if self.uname not in axs.uhtml:
                 pass
             elif is_dir:
                 for fn in ("index.htm", "index.html"):
@@ -3840,7 +3868,8 @@ class HttpCli(object):
                 )[: vn.flags["fk"]]
                 got = self.uparam.get("k")
                 if got != correct:
-                    self.log("wrong filekey, want {}, got {}".format(correct, got))
+                    t = "wrong filekey, want %s, got %s\n  vp: %s\n  ap: %s"
+                    self.log(t % (correct, got, self.req, abspath), 6)
                     return self.tx_404()
 
             if (
@@ -3972,6 +4001,7 @@ class HttpCli(object):
             "dsort": vf["sort"],
             "themes": self.args.themes,
             "turbolvl": self.args.turbo,
+            "u2j": self.args.u2j,
             "idxh": int(self.args.ih),
             "u2sort": self.args.u2sort,
         }
@@ -4021,7 +4051,7 @@ class HttpCli(object):
         for k in ["zip", "tar"]:
             v = self.uparam.get(k)
             if v is not None:
-                return self.tx_zip(k, v, self.vpath, vn, rem, [], self.args.ed)
+                return self.tx_zip(k, v, self.vpath, vn, rem, [])
 
         fsroot, vfs_ls, vfs_virt = vn.ls(
             rem,
@@ -4052,13 +4082,13 @@ class HttpCli(object):
             pass
 
         # show dotfiles if permitted and requested
-        if not self.args.ed or (
+        if not self.can_dot or (
             "dots" not in self.uparam and (is_ls or "dots" not in self.cookies)
         ):
             ls_names = exclude_dotfiles(ls_names)
 
-        add_fk = vn.flags.get("fk")
-        fk_alg = 2 if "fka" in vn.flags else 1
+        add_fk = vf.get("fk")
+        fk_alg = 2 if "fka" in vf else 1
 
         dirs = []
         files = []
@@ -4215,7 +4245,7 @@ class HttpCli(object):
         if icur:
             lmte = list(mte)
             if self.can_admin:
-                lmte += ["up_ip", ".up_at"]
+                lmte.extend(("up_ip", ".up_at"))
 
             taglist = [k for k in lmte if k in tagset]
             for fe in dirs:

copyparty/httpconn.py

@@ -93,7 +93,7 @@ class HttpConn(object):
             self.rproxy = ip
 
         self.ip = ip
-        self.log_src = "{} \033[{}m{}".format(ip, color, self.addr[1]).ljust(26)
+        self.log_src = ("%s \033[%dm%d" % (ip, color, self.addr[1])).ljust(26)
         return self.log_src
 
     def respath(self, res_name: str) -> str:
@@ -112,32 +112,30 @@ class HttpConn(object):
         return self.u2idx
 
     def _detect_https(self) -> bool:
-        method = None
-        if True:
-            try:
-                method = self.s.recv(4, socket.MSG_PEEK)
-            except socket.timeout:
-                return False
-            except AttributeError:
-                # jython does not support msg_peek; forget about https
-                method = self.s.recv(4)
-                self.sr = Util.Unrecv(self.s, self.log)
-                self.sr.buf = method
+        try:
+            method = self.s.recv(4, socket.MSG_PEEK)
+        except socket.timeout:
+            return False
+        except AttributeError:
+            # jython does not support msg_peek; forget about https
+            method = self.s.recv(4)
+            self.sr = Util.Unrecv(self.s, self.log)
+            self.sr.buf = method
 
-                # jython used to do this, they stopped since it's broken
-                # but reimplementing sendall is out of scope for now
-                if not getattr(self.s, "sendall", None):
-                    self.s.sendall = self.s.send  # type: ignore
+            # jython used to do this, they stopped since it's broken
+            # but reimplementing sendall is out of scope for now
+            if not getattr(self.s, "sendall", None):
+                self.s.sendall = self.s.send  # type: ignore
 
-            if len(method) != 4:
-                err = "need at least 4 bytes in the first packet; got {}".format(
-                    len(method)
-                )
-                if method:
-                    self.log(err)
+        if len(method) != 4:
+            err = "need at least 4 bytes in the first packet; got {}".format(
+                len(method)
+            )
+            if method:
+                self.log(err)
 
-                self.s.send(b"HTTP/1.1 400 Bad Request\r\n\r\n" + err.encode("utf-8"))
-                return False
+            self.s.send(b"HTTP/1.1 400 Bad Request\r\n\r\n" + err.encode("utf-8"))
+            return False
 
         return not method or not bool(PTN_HTTP.match(method))
 
@@ -178,7 +176,7 @@ class HttpConn(object):
 
                 self.s = ctx.wrap_socket(self.s, server_side=True)
                 msg = [
-                    "\033[1;3{:d}m{}".format(c, s)
+                    "\033[1;3%dm%s" % (c, s)
                     for c, s in zip([0, 5, 0], self.s.cipher())  # type: ignore
                 ]
                 self.log(" ".join(msg) + "\033[0m")

copyparty/httpsrv.py

@@ -366,7 +366,7 @@ class HttpSrv(object):
             if not self.t_periodic:
                 name = "hsrv-pt"
                 if self.nid:
-                    name += "-{}".format(self.nid)
+                    name += "-%d" % (self.nid,)
 
                 self.t_periodic = Daemon(self.periodic, name)
 
@@ -385,7 +385,7 @@ class HttpSrv(object):
 
         Daemon(
             self.thr_client,
-            "httpconn-{}-{}".format(addr[0].split(".", 2)[-1][-6:], addr[1]),
+            "httpconn-%s-%d" % (addr[0].split(".", 2)[-1][-6:], addr[1]),
             (sck, addr),
         )
 
@@ -402,9 +402,7 @@ class HttpSrv(object):
             try:
                 sck, addr = task
                 me = threading.current_thread()
-                me.name = "httpconn-{}-{}".format(
-                    addr[0].split(".", 2)[-1][-6:], addr[1]
-                )
+                me.name = "httpconn-%s-%d" % (addr[0].split(".", 2)[-1][-6:], addr[1])
                 self.thr_client(sck, addr)
                 me.name = self.name + "-poolw"
             except Exception as ex:

copyparty/ico.py

@@ -27,13 +27,13 @@ class Ico(object):
         c1 = colorsys.hsv_to_rgb(zb[0] / 256.0, 1, 0.3)
         c2 = colorsys.hsv_to_rgb(zb[0] / 256.0, 0.8 if HAVE_PILF else 1, 1)
         ci = [int(x * 255) for x in list(c1) + list(c2)]
-        c = "".join(["{:02x}".format(x) for x in ci])
+        c = "".join(["%02x" % (x,) for x in ci])
 
         w = 100
         h = 30
         if not self.args.th_no_crop and as_thumb:
             sw, sh = self.args.th_size.split("x")
-            h = int(100 / (float(sw) / float(sh)))
+            h = int(100.0 / (float(sw) / float(sh)))
             w = 100
 
         if chrome:
@@ -47,12 +47,12 @@ class Ico(object):
                     # [.lt] are hard to see lowercase / unspaced
                     ext2 = re.sub("(.)", "\\1 ", ext).upper()
 
-                    h = int(128 * h / w)
+                    h = int(128.0 * h / w)
                     w = 128
                     img = Image.new("RGB", (w, h), "#" + c[:6])
                     pb = ImageDraw.Draw(img)
                     _, _, tw, th = pb.textbbox((0, 0), ext2, font_size=16)
-                    xy = ((w - tw) // 2, (h - th) // 2)
+                    xy = (int((w - tw) / 2), int((h - th) / 2))
                     pb.text(xy, ext2, fill="#" + c[6:], font_size=16)
 
                     img = img.resize((w * 2, h * 2), Image.NEAREST)
@@ -68,7 +68,7 @@ class Ico(object):
                 # svg: 3s, cache: 6s, this: 8s
                 from PIL import Image, ImageDraw
 
-                h = int(64 * h / w)
+                h = int(64.0 * h / w)
                 w = 64
                 img = Image.new("RGB", (w, h), "#" + c[:6])
                 pb = ImageDraw.Draw(img)

copyparty/mtag.py

@@ -118,7 +118,7 @@ def ffprobe(
         b"--",
         fsenc(abspath),
     ]
-    rc, so, se = runcmd(cmd, timeout=timeout, nice=True)
+    rc, so, se = runcmd(cmd, timeout=timeout, nice=True, oom=200)
     retchk(rc, cmd, se)
     return parse_ffprobe(so)
 
@@ -240,7 +240,7 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
         if "/" in fps:
             fa, fb = fps.split("/")
             try:
-                fps = int(fa) * 1.0 / int(fb)
+                fps = float(fa) / float(fb)
             except:
                 fps = 9001
 
@@ -564,6 +564,7 @@ class MTag(object):
                 args = {
                     "env": env,
                     "nice": True,
+                    "oom": 300,
                     "timeout": parser.timeout,
                     "kill": parser.kill,
                     "capture": parser.capture,

copyparty/star.py

@@ -65,21 +65,21 @@ class StreamTar(StreamArc):
             cmp = re.sub(r"[^a-z0-9]*pax[^a-z0-9]*", "", cmp)
 
         try:
-            cmp, lv = cmp.replace(":", ",").split(",")
-            lv = int(lv)
+            cmp, zs = cmp.replace(":", ",").split(",")
+            lv = int(zs)
         except:
-            lv = None
+            lv = -1
 
         arg = {"name": None, "fileobj": self.qfile, "mode": "w", "format": fmt}
         if cmp == "gz":
             fun = tarfile.TarFile.gzopen
-            arg["compresslevel"] = lv if lv is not None else 3
+            arg["compresslevel"] = lv if lv >= 0 else 3
         elif cmp == "bz2":
             fun = tarfile.TarFile.bz2open
-            arg["compresslevel"] = lv if lv is not None else 2
+            arg["compresslevel"] = lv if lv >= 0 else 2
         elif cmp == "xz":
             fun = tarfile.TarFile.xzopen
-            arg["preset"] = lv if lv is not None else 1
+            arg["preset"] = lv if lv >= 0 else 1
         else:
             fun = tarfile.open
             arg["mode"] = "w|"

copyparty/stolen/ifaddr/_shared.py

@@ -61,7 +61,7 @@ class Adapter(object):
         )
 
 
-if True:
+if True:  # pylint: disable=using-constant-test
     # Type of an IPv4 address (a string in "xxx.xxx.xxx.xxx" format)
     _IPv4Address = str
 

copyparty/svchub.py

@@ -405,7 +405,11 @@ class SvcHub(object):
         if al.rsp_jtr:
             al.rsp_slp = 0.000001
 
-        al.th_covers = set(al.th_covers.split(","))
+        zsl = al.th_covers.split(",")
+        zsl = [x.strip() for x in zsl]
+        zsl = [x for x in zsl if x]
+        al.th_covers = set(zsl)
+        al.th_coversd = set(zsl + ["." + x for x in zsl])
 
         for k in "c".split(" "):
             vl = getattr(al, k)
@@ -433,6 +437,10 @@ class SvcHub(object):
         elif al.ban_url == "no":
             al.sus_urls = None
 
+        al.xff_hdr = al.xff_hdr.lower()
+        al.idp_h_usr = al.idp_h_usr.lower()
+        # al.idp_h_grp = al.idp_h_grp.lower()
+
         al.xff_re = self._ipa2re(al.xff_src)
         al.ipa_re = self._ipa2re(al.ipa)
         al.ftp_ipa_re = self._ipa2re(al.ftp_ipa or al.ipa)
@@ -452,12 +460,19 @@ class SvcHub(object):
             if ptn:
                 setattr(self.args, k, re.compile(ptn))
 
+        try:
+            zf1, zf2 = self.args.rm_retry.split("/")
+            self.args.rm_re_t = float(zf1)
+            self.args.rm_re_r = float(zf2)
+        except:
+            raise Exception("invalid --rm-retry [%s]" % (self.args.rm_retry,))
+
         return True
 
     def _ipa2re(self, txt) -> Optional[re.Pattern]:
         if txt in ("any", "0", ""):
             return None
-        
+
         zs = txt.replace(" ", "").replace(".", "\\.").replace(",", "|")
         return re.compile("^(?:" + zs + ")")
 
@@ -466,7 +481,7 @@ class SvcHub(object):
             import resource
 
             soft, hard = [
-                x if x > 0 else 1024 * 1024
+                int(x) if x > 0 else 1024 * 1024
                 for x in list(resource.getrlimit(resource.RLIMIT_NOFILE))
             ]
         except:
@@ -778,12 +793,12 @@ class SvcHub(object):
             if "\033" in msg:
                 msg += "\033[0m"
 
-            self.logf.write("@%s [%s] %s\n" % (ts, src, msg))
+            self.logf.write("@%s [%-21s] %s\n" % (ts, src, msg))
             if not self.args.no_logflush:
                 self.logf.flush()
 
             now = time.time()
-            if now >= self.next_day:
+            if int(now) >= self.next_day:
                 self._set_next_day()
 
     def _set_next_day(self) -> None:
@@ -811,7 +826,7 @@ class SvcHub(object):
         """handles logging from all components"""
         with self.log_mutex:
             now = time.time()
-            if now >= self.next_day:
+            if int(now) >= self.next_day:
                 dt = datetime.fromtimestamp(now, UTC)
                 zs = "{}\n" if self.no_ansi else "\033[36m{}\033[0m\n"
                 zs = zs.format(dt.strftime("%Y-%m-%d"))

copyparty/tcpsrv.py

@@ -241,6 +241,11 @@ class TcpSrv(object):
                 raise OSError(E_ADDR_IN_USE[0], "")
             self.srv.append(srv)
         except (OSError, socket.error) as ex:
+            try:
+                srv.close()
+            except:
+                pass
+
             if ex.errno in E_ADDR_IN_USE:
                 e = "\033[1;31mport {} is busy on interface {}\033[0m".format(port, ip)
             elif ex.errno in E_ADDR_NOT_AVAIL:

copyparty/th_srv.py

@@ -28,6 +28,7 @@ from .util import (
     runcmd,
     statdir,
     vsplit,
+    wunlink,
 )
 
 if True:  # pylint: disable=using-constant-test
@@ -102,7 +103,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
     rd += "\n" + fmt
     h = hashlib.sha512(afsenc(rd)).digest()
     b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
-    rd = "{}/{}/".format(b64[:2], b64[2:4]).lower() + b64
+    rd = ("%s/%s/" % (b64[:2], b64[2:4])).lower() + b64
 
     # could keep original filenames but this is safer re pathlen
     h = hashlib.sha512(afsenc(fn)).digest()
@@ -115,7 +116,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
         fmt = "webp" if fc == "w" else "png" if fc == "p" else "jpg"
         cat = "th"
 
-    return "{}/{}/{}/{}.{:x}.{}".format(histpath, cat, rd, fn, int(mtime), fmt)
+    return "%s/%s/%s/%s.%x.%s" % (histpath, cat, rd, fn, int(mtime), fmt)
 
 
 class ThumbSrv(object):
@@ -129,6 +130,8 @@ class ThumbSrv(object):
 
         self.mutex = threading.Lock()
         self.busy: dict[str, list[threading.Condition]] = {}
+        self.ram: dict[str, float] = {}
+        self.memcond = threading.Condition(self.mutex)
         self.stopping = False
         self.nthr = max(1, self.args.th_mt)
 
@@ -214,7 +217,7 @@ class ThumbSrv(object):
         with self.mutex:
             try:
                 self.busy[tpath].append(cond)
-                self.log("wait {}".format(tpath))
+                self.log("joined waiting room for %s" % (tpath,))
             except:
                 thdir = os.path.dirname(tpath)
                 bos.makedirs(os.path.join(thdir, "w"))
@@ -265,6 +268,23 @@ class ThumbSrv(object):
             "ffa": self.fmt_ffa,
         }
 
+    def wait4ram(self, need: float, ttpath: str) -> None:
+        ram = self.args.th_ram_max
+        if need > ram * 0.99:
+            t = "file too big; need %.2f GiB RAM, but --th-ram-max is only %.1f"
+            raise Exception(t % (need, ram))
+
+        while True:
+            with self.mutex:
+                used = sum([v for k, v in self.ram.items() if k != ttpath]) + need
+                if used < ram:
+                    # self.log("XXX self.ram: %s" % (self.ram,), 5)
+                    self.ram[ttpath] = need
+                    return
+            with self.memcond:
+                # self.log("at RAM limit; used %.2f GiB, need %.2f more" % (used-need, need), 1)
+                self.memcond.wait(3)
+
     def worker(self) -> None:
         while not self.stopping:
             task = self.q.get()
@@ -298,7 +318,7 @@ class ThumbSrv(object):
             tdir, tfn = os.path.split(tpath)
             ttpath = os.path.join(tdir, "w", tfn)
             try:
-                bos.unlink(ttpath)
+                wunlink(self.log, ttpath, vn.flags)
             except:
                 pass
 
@@ -318,7 +338,7 @@ class ThumbSrv(object):
                     else:
                         # ffmpeg may spawn empty files on windows
                         try:
-                            os.unlink(ttpath)
+                            wunlink(self.log, ttpath, vn.flags)
                         except:
                             pass
 
@@ -330,11 +350,15 @@ class ThumbSrv(object):
             with self.mutex:
                 subs = self.busy[tpath]
                 del self.busy[tpath]
+                self.ram.pop(ttpath, None)
 
             for x in subs:
                 with x:
                     x.notify_all()
 
+            with self.memcond:
+                self.memcond.notify_all()
+
         with self.mutex:
             self.nthr -= 1
 
@@ -366,6 +390,7 @@ class ThumbSrv(object):
         return im
 
     def conv_pil(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
+        self.wait4ram(0.2, tpath)
         with Image.open(fsenc(abspath)) as im:
             try:
                 im = self.fancy_pillow(im, fmt, vn)
@@ -382,7 +407,7 @@ class ThumbSrv(object):
                 # method 0 = pillow-default, fast
                 # method 4 = ffmpeg-default
                 # method 6 = max, slow
-                fmts += ["RGBA", "LA"]
+                fmts.extend(("RGBA", "LA"))
                 args["method"] = 6
             else:
                 # default q = 75
@@ -395,6 +420,7 @@ class ThumbSrv(object):
             im.save(tpath, **args)
 
     def conv_vips(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
+        self.wait4ram(0.2, tpath)
         crops = ["centre", "none"]
         if fmt.endswith("f"):
             crops = ["none"]
@@ -415,6 +441,7 @@ class ThumbSrv(object):
         img.write_to_file(tpath, Q=40)
 
     def conv_ffmpeg(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
+        self.wait4ram(0.2, tpath)
         ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if not ret:
             return
@@ -467,9 +494,9 @@ class ThumbSrv(object):
         cmd += [fsenc(tpath)]
         self._run_ff(cmd, vn)
 
-    def _run_ff(self, cmd: list[bytes], vn: VFS) -> None:
+    def _run_ff(self, cmd: list[bytes], vn: VFS, oom: int = 400) -> None:
         # self.log((b" ".join(cmd)).decode("utf-8"))
-        ret, _, serr = runcmd(cmd, timeout=vn.flags["convt"], nice=True)
+        ret, _, serr = runcmd(cmd, timeout=vn.flags["convt"], nice=True, oom=oom)
         if not ret:
             return
 
@@ -517,8 +544,21 @@ class ThumbSrv(object):
         if "ac" not in ret:
             raise Exception("not audio")
 
-        flt = (
-            b"[0:a:0]"
+        # jt_versi.xm: 405M/839s
+        dur = ret[".dur"][1] if ".dur" in ret else 300
+        need = 0.2 + dur / 3000
+        speedup = b""
+        if need > self.args.th_ram_max * 0.7:
+            self.log("waves too big (need %.2f GiB); trying to optimize" % (need,))
+            need = 0.2 + dur / 4200  # only helps about this much...
+            speedup = b"aresample=8000,"
+        if need > self.args.th_ram_max * 0.96:
+            raise Exception("file too big; cannot waves")
+
+        self.wait4ram(need, tpath)
+
+        flt = b"[0:a:0]" + speedup
+        flt += (
             b"compand=.3|.3:1|1:-90/-60|-60/-40|-40/-30|-20/-20:6:0:-90:0.2"
             b",volume=2"
             b",showwavespic=s=2048x64:colors=white"
@@ -545,6 +585,15 @@ class ThumbSrv(object):
         if "ac" not in ret:
             raise Exception("not audio")
 
+        # https://trac.ffmpeg.org/ticket/10797
+        # expect 1 GiB every 600 seconds when duration is tricky;
+        # simple filetypes are generally safer so let's special-case those
+        safe = ("flac", "wav", "aif", "aiff", "opus")
+        coeff = 1800 if abspath.split(".")[-1].lower() in safe else 600
+        dur = ret[".dur"][1] if ".dur" in ret else 300
+        need = 0.2 + dur / coeff
+        self.wait4ram(need, tpath)
+
         fc = "[0:a:0]aresample=48000{},showspectrumpic=s=640x512,crop=780:544:70:50[o]"
 
         if self.args.th_ff_swr:
@@ -587,6 +636,7 @@ class ThumbSrv(object):
         if self.args.no_acode:
             raise Exception("disabled in server config")
 
+        self.wait4ram(0.2, tpath)
         ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in ret:
             raise Exception("not audio")
@@ -602,7 +652,7 @@ class ThumbSrv(object):
         if want_caf:
             tmp_opus = tpath + ".opus"
             try:
-                bos.unlink(tmp_opus)
+                wunlink(self.log, tmp_opus, vn.flags)
             except:
                 pass
 
@@ -623,7 +673,7 @@ class ThumbSrv(object):
                 fsenc(tmp_opus)
             ]
             # fmt: on
-            self._run_ff(cmd, vn)
+            self._run_ff(cmd, vn, oom=300)
 
         # iOS fails to play some "insufficiently complex" files
         # (average file shorter than 8 seconds), so of course we
@@ -647,7 +697,7 @@ class ThumbSrv(object):
                 fsenc(tpath)
             ]
             # fmt: on
-            self._run_ff(cmd, vn)
+            self._run_ff(cmd, vn, oom=300)
 
         elif want_caf:
             # simple remux should be safe
@@ -665,11 +715,11 @@ class ThumbSrv(object):
                 fsenc(tpath)
             ]
             # fmt: on
-            self._run_ff(cmd, vn)
+            self._run_ff(cmd, vn, oom=300)
 
         if tmp_opus != tpath:
             try:
-                bos.unlink(tmp_opus)
+                wunlink(self.log, tmp_opus, vn.flags)
             except:
                 pass
 
@@ -696,7 +746,10 @@ class ThumbSrv(object):
                 else:
                     self.log("\033[Jcln {} ({})/\033[A".format(histpath, vol))
 
-                ndirs += self.clean(histpath)
+                try:
+                    ndirs += self.clean(histpath)
+                except Exception as ex:
+                    self.log("\033[Jcln err in %s: %r" % (histpath, ex), 3)
 
             self.log("\033[Jcln ok; rm {} dirs".format(ndirs))
 

copyparty/u2idx.py

@@ -9,7 +9,7 @@ import time
 from operator import itemgetter
 
 from .__init__ import ANYWIN, TYPE_CHECKING, unicode
-from .authsrv import LEELOO_DALLAS
+from .authsrv import LEELOO_DALLAS, VFS
 from .bos import bos
 from .up2k import up2k_wark_from_hashlist
 from .util import (
@@ -63,7 +63,7 @@ class U2idx(object):
         self.log_func("u2idx", msg, c)
 
     def fsearch(
-        self, vols: list[tuple[str, str, dict[str, Any]]], body: dict[str, Any]
+        self, uname: str, vols: list[VFS], body: dict[str, Any]
     ) -> list[dict[str, Any]]:
         """search by up2k hashlist"""
         if not HAVE_SQLITE3:
@@ -77,7 +77,7 @@ class U2idx(object):
         uv: list[Union[str, int]] = [wark[:16], wark]
 
         try:
-            return self.run_query(vols, uq, uv, True, False, 99999)[0]
+            return self.run_query(uname, vols, uq, uv, False, 99999)[0]
         except:
             raise Pebkac(500, min_ex())
 
@@ -103,7 +103,7 @@ class U2idx(object):
             uri = ""
             try:
                 uri = "{}?mode=ro&nolock=1".format(Path(db_path).as_uri())
-                db = sqlite3.connect(uri, 2, uri=True, check_same_thread=False)
+                db = sqlite3.connect(uri, timeout=2, uri=True, check_same_thread=False)
                 cur = db.cursor()
                 cur.execute('pragma table_info("up")').fetchone()
                 self.log("ro: {}".format(db_path))
@@ -115,14 +115,14 @@ class U2idx(object):
         if not cur:
             # on windows, this steals the write-lock from up2k.deferred_init --
             # seen on win 10.0.17763.2686, py 3.10.4, sqlite 3.37.2
-            cur = sqlite3.connect(db_path, 2, check_same_thread=False).cursor()
+            cur = sqlite3.connect(db_path, timeout=2, check_same_thread=False).cursor()
             self.log("opened {}".format(db_path))
 
         self.cur[ptop] = cur
         return cur
 
     def search(
-        self, vols: list[tuple[str, str, dict[str, Any]]], uq: str, lim: int
+        self, uname: str, vols: list[VFS], uq: str, lim: int
     ) -> tuple[list[dict[str, Any]], list[str], bool]:
         """search by query params"""
         if not HAVE_SQLITE3:
@@ -131,7 +131,6 @@ class U2idx(object):
         q = ""
         v: Union[str, int] = ""
         va: list[Union[str, int]] = []
-        have_up = False  # query has up.* operands
         have_mt = False
         is_key = True
         is_size = False
@@ -176,26 +175,21 @@ class U2idx(object):
                 if v == "size":
                     v = "up.sz"
                     is_size = True
-                    have_up = True
 
                 elif v == "date":
                     v = "up.mt"
                     is_date = True
-                    have_up = True
 
                 elif v == "up_at":
                     v = "up.at"
                     is_date = True
-                    have_up = True
 
                 elif v == "path":
                     v = "trim(?||up.rd,'/')"
                     va.append("\nrd")
-                    have_up = True
 
                 elif v == "name":
                     v = "up.fn"
-                    have_up = True
 
                 elif v == "tags" or ptn_mt.match(v):
                     have_mt = True
@@ -271,22 +265,22 @@ class U2idx(object):
                 q += " lower({}) {} ? ) ".format(field, oper)
 
         try:
-            return self.run_query(vols, q, va, have_up, have_mt, lim)
+            return self.run_query(uname, vols, q, va, have_mt, lim)
         except Exception as ex:
             raise Pebkac(500, repr(ex))
 
     def run_query(
         self,
-        vols: list[tuple[str, str, dict[str, Any]]],
+        uname: str,
+        vols: list[VFS],
         uq: str,
         uv: list[Union[str, int]],
-        have_up: bool,
         have_mt: bool,
         lim: int,
     ) -> tuple[list[dict[str, Any]], list[str], bool]:
         if self.args.srch_dbg:
             t = "searching across all %s volumes in which the user has 'r' (full read access):\n  %s"
-            zs = "\n  ".join(["/%s = %s" % (x[0], x[1]) for x in vols])
+            zs = "\n  ".join(["/%s = %s" % (x.vpath, x.realpath) for x in vols])
             self.log(t % (len(vols), zs), 5)
 
         done_flag: list[bool] = []
@@ -315,10 +309,14 @@ class U2idx(object):
             clamped = False
 
         taglist = {}
-        for (vtop, ptop, flags) in vols:
+        for vol in vols:
             if lim < 0:
                 break
 
+            vtop = vol.vpath
+            ptop = vol.realpath
+            flags = vol.flags
+
             cur = self.get_cur(ptop)
             if not cur:
                 continue
@@ -343,7 +341,7 @@ class U2idx(object):
 
             sret = []
             fk = flags.get("fk")
-            dots = flags.get("dotsrch")
+            dots = flags.get("dotsrch") and uname in vol.axs.udot
             fk_alg = 2 if "fka" in flags else 1
             c = cur.execute(uq, tuple(vuv))
             for hit in c:

copyparty/up2k.py

@@ -37,6 +37,7 @@ from .util import (
     absreal,
     atomic_move,
     db_ex_chk,
+    dir_is_empty,
     djoin,
     fsenc,
     gen_filekey,
@@ -63,6 +64,7 @@ from .util import (
     vsplit,
     w8b64dec,
     w8b64enc,
+    wunlink,
 )
 
 try:
@@ -145,9 +147,12 @@ class Up2k(object):
         self.entags: dict[str, set[str]] = {}
         self.mtp_parsers: dict[str, dict[str, MParser]] = {}
         self.pending_tags: list[tuple[set[str], str, str, dict[str, Any]]] = []
-        self.hashq: Queue[tuple[str, str, str, str, str, float, str, bool]] = Queue()
+        self.hashq: Queue[
+            tuple[str, str, dict[str, Any], str, str, str, float, str, bool]
+        ] = Queue()
         self.tagq: Queue[tuple[str, str, str, str, str, float]] = Queue()
         self.tag_event = threading.Condition()
+        self.hashq_mutex = threading.Lock()
         self.n_hashq = 0
         self.n_tagq = 0
         self.mpool_used = False
@@ -419,50 +424,49 @@ class Up2k(object):
     def _check_lifetimes(self) -> float:
         now = time.time()
         timeout = now + 9001
-        if now:  # diff-golf
-            for vp, vol in sorted(self.asrv.vfs.all_vols.items()):
-                lifetime = vol.flags.get("lifetime")
-                if not lifetime:
-                    continue
+        for vp, vol in sorted(self.asrv.vfs.all_vols.items()):
+            lifetime = vol.flags.get("lifetime")
+            if not lifetime:
+                continue
 
-                cur = self.cur.get(vol.realpath)
-                if not cur:
-                    continue
+            cur = self.cur.get(vol.realpath)
+            if not cur:
+                continue
 
-                nrm = 0
-                deadline = time.time() - lifetime
-                timeout = min(timeout, now + lifetime)
-                q = "select rd, fn from up where at > 0 and at < ? limit 100"
-                while True:
-                    with self.mutex:
-                        hits = cur.execute(q, (deadline,)).fetchall()
-
-                    if not hits:
-                        break
-
-                    for rd, fn in hits:
-                        if rd.startswith("//") or fn.startswith("//"):
-                            rd, fn = s3dec(rd, fn)
-
-                        fvp = ("%s/%s" % (rd, fn)).strip("/")
-                        if vp:
-                            fvp = "%s/%s" % (vp, fvp)
-
-                        self._handle_rm(LEELOO_DALLAS, "", fvp, [], True)
-                        nrm += 1
-
-                if nrm:
-                    self.log("{} files graduated in {}".format(nrm, vp))
-
-                if timeout < 10:
-                    continue
-
-                q = "select at from up where at > 0 order by at limit 1"
+            nrm = 0
+            deadline = time.time() - lifetime
+            timeout = min(timeout, now + lifetime)
+            q = "select rd, fn from up where at > 0 and at < ? limit 100"
+            while True:
                 with self.mutex:
-                    hits = cur.execute(q).fetchone()
+                    hits = cur.execute(q, (deadline,)).fetchall()
 
-                if hits:
-                    timeout = min(timeout, now + lifetime - (now - hits[0]))
+                if not hits:
+                    break
+
+                for rd, fn in hits:
+                    if rd.startswith("//") or fn.startswith("//"):
+                        rd, fn = s3dec(rd, fn)
+
+                    fvp = ("%s/%s" % (rd, fn)).strip("/")
+                    if vp:
+                        fvp = "%s/%s" % (vp, fvp)
+
+                    self._handle_rm(LEELOO_DALLAS, "", fvp, [], True)
+                    nrm += 1
+
+            if nrm:
+                self.log("{} files graduated in {}".format(nrm, vp))
+
+            if timeout < 10:
+                continue
+
+            q = "select at from up where at > 0 order by at limit 1"
+            with self.mutex:
+                hits = cur.execute(q).fetchone()
+
+            if hits:
+                timeout = min(timeout, now + lifetime - (now - hits[0]))
 
         return timeout
 
@@ -579,7 +583,7 @@ class Up2k(object):
         if gid:
             self.log("reload #{} running".format(self.gid))
 
-        self.pp = ProgressPrinter()
+        self.pp = ProgressPrinter(self.log, self.args)
         vols = list(all_vols.values())
         t0 = time.time()
         have_e2d = False
@@ -602,7 +606,7 @@ class Up2k(object):
             for vol in vols:
                 try:
                     bos.makedirs(vol.realpath)  # gonna happen at snap anyways
-                    bos.listdir(vol.realpath)
+                    dir_is_empty(self.log_func, not self.args.no_scandir, vol.realpath)
                 except:
                     self.volstate[vol.vpath] = "OFFLINE (cannot access folder)"
                     self.log("cannot access " + vol.realpath, c=1)
@@ -805,7 +809,7 @@ class Up2k(object):
         ft = "\033[0;32m{}{:.0}"
         ff = "\033[0;35m{}{:.0}"
         fv = "\033[0;36m{}:\033[90m{}"
-        fx = set(("html_head",))
+        fx = set(("html_head", "rm_re_t", "rm_re_r"))
         fd = vf_bmap()
         fd.update(vf_cmap())
         fd.update(vf_vmap())
@@ -984,12 +988,12 @@ class Up2k(object):
                 excl = [x.replace("/", "\\") for x in excl]
             else:
                 # ~/.wine/dosdevices/z:/ and such
-                excl += ["/dev", "/proc", "/run", "/sys"]
+                excl.extend(("/dev", "/proc", "/run", "/sys"))
 
             rtop = absreal(top)
             n_add = n_rm = 0
             try:
-                if not bos.listdir(rtop):
+                if dir_is_empty(self.log_func, not self.args.no_scandir, rtop):
                     t = "volume /%s at [%s] is empty; will not be indexed as this could be due to an offline filesystem"
                     self.log(t % (vol.vpath, rtop), 6)
                     return True, False
@@ -1086,7 +1090,7 @@ class Up2k(object):
         cv = ""
 
         assert self.pp and self.mem_cur
-        self.pp.msg = "a{} {}".format(self.pp.n, cdir)
+        self.pp.msg = "a%d %s" % (self.pp.n, cdir)
 
         rd = cdir[len(top) :].strip("/")
         if WINDOWS:
@@ -1119,7 +1123,11 @@ class Up2k(object):
 
             if stat.S_ISDIR(inf.st_mode):
                 rap = absreal(abspath)
-                if dev and inf.st_dev != dev:
+                if (
+                    dev
+                    and inf.st_dev != dev
+                    and not (ANYWIN and bos.stat(rap).st_dev == dev)
+                ):
                     self.log("skip xdev {}->{}: {}".format(dev, inf.st_dev, abspath), 6)
                     continue
                 if abspath in excl or rap in excl:
@@ -1157,8 +1165,8 @@ class Up2k(object):
                     continue
 
                 if not sz and (
-                    "{}.PARTIAL".format(iname) in partials
-                    or ".{}.PARTIAL".format(iname) in partials
+                    "%s.PARTIAL" % (iname,) in partials
+                    or ".%s.PARTIAL" % (iname,) in partials
                 ):
                     # placeholder for unfinished upload
                     continue
@@ -1166,8 +1174,12 @@ class Up2k(object):
                 files.append((sz, lmod, iname))
                 liname = iname.lower()
                 if sz and (
-                    iname in self.args.th_covers
-                    or (not cv and liname.rsplit(".", 1)[-1] in CV_EXTS)
+                    iname in self.args.th_coversd
+                    or (
+                        not cv
+                        and liname.rsplit(".", 1)[-1] in CV_EXTS
+                        and not iname.startswith(".")
+                    )
                 ):
                     cv = iname
 
@@ -1217,72 +1229,76 @@ class Up2k(object):
             abspath = os.path.join(cdir, fn)
             nohash = reh.search(abspath) if reh else False
 
-            if fn:  # diff-golf
+            sql = "select w, mt, sz, ip, at from up where rd = ? and fn = ?"
+            try:
+                c = db.c.execute(sql, (rd, fn))
+            except:
+                c = db.c.execute(sql, s3enc(self.mem_cur, rd, fn))
 
-                sql = "select w, mt, sz, at from up where rd = ? and fn = ?"
-                try:
-                    c = db.c.execute(sql, (rd, fn))
-                except:
-                    c = db.c.execute(sql, s3enc(self.mem_cur, rd, fn))
+            in_db = list(c.fetchall())
+            if in_db:
+                self.pp.n -= 1
+                dw, dts, dsz, ip, at = in_db[0]
+                if len(in_db) > 1:
+                    t = "WARN: multiple entries: [{}] => [{}] |{}|\n{}"
+                    rep_db = "\n".join([repr(x) for x in in_db])
+                    self.log(t.format(top, rp, len(in_db), rep_db))
+                    dts = -1
 
-                in_db = list(c.fetchall())
-                if in_db:
-                    self.pp.n -= 1
-                    dw, dts, dsz, at = in_db[0]
-                    if len(in_db) > 1:
-                        t = "WARN: multiple entries: [{}] => [{}] |{}|\n{}"
-                        rep_db = "\n".join([repr(x) for x in in_db])
-                        self.log(t.format(top, rp, len(in_db), rep_db))
-                        dts = -1
+                if fat32 and abs(dts - lmod) == 1:
+                    dts = lmod
 
-                    if fat32 and abs(dts - lmod) == 1:
-                        dts = lmod
+                if dts == lmod and dsz == sz and (nohash or dw[0] != "#" or not sz):
+                    continue
 
-                    if dts == lmod and dsz == sz and (nohash or dw[0] != "#" or not sz):
-                        continue
-
-                    t = "reindex [{}] => [{}] ({}/{}) ({}/{})".format(
-                        top, rp, dts, lmod, dsz, sz
-                    )
-                    self.log(t)
-                    self.db_rm(db.c, rd, fn, 0)
-                    ret += 1
-                    db.n += 1
-                    in_db = []
-                else:
-                    at = 0
-
-                self.pp.msg = "a{} {}".format(self.pp.n, abspath)
-
-                if nohash or not sz:
-                    wark = up2k_wark_from_metadata(self.salt, sz, lmod, rd, fn)
-                else:
-                    if sz > 1024 * 1024:
-                        self.log("file: {}".format(abspath))
-
-                    try:
-                        hashes = self._hashlist_from_file(
-                            abspath, "a{}, ".format(self.pp.n)
-                        )
-                    except Exception as ex:
-                        self.log("hash: {} @ [{}]".format(repr(ex), abspath))
-                        continue
-
-                    if not hashes:
-                        return -1
-
-                    wark = up2k_wark_from_hashlist(self.salt, sz, hashes)
-
-                # skip upload hooks by not providing vflags
-                self.db_add(db.c, {}, rd, fn, lmod, sz, "", "", wark, "", "", "", at)
-                db.n += 1
+                t = "reindex [{}] => [{}] ({}/{}) ({}/{})".format(
+                    top, rp, dts, lmod, dsz, sz
+                )
+                self.log(t)
+                self.db_rm(db.c, rd, fn, 0)
                 ret += 1
-                td = time.time() - db.t
-                if db.n >= 4096 or td >= 60:
-                    self.log("commit {} new files".format(db.n))
-                    db.c.connection.commit()
-                    db.n = 0
-                    db.t = time.time()
+                db.n += 1
+                in_db = []
+            else:
+                dw = ""
+                ip = ""
+                at = 0
+
+            self.pp.msg = "a%d %s" % (self.pp.n, abspath)
+
+            if nohash or not sz:
+                wark = up2k_wark_from_metadata(self.salt, sz, lmod, rd, fn)
+            else:
+                if sz > 1024 * 1024:
+                    self.log("file: {}".format(abspath))
+
+                try:
+                    hashes = self._hashlist_from_file(
+                        abspath, "a{}, ".format(self.pp.n)
+                    )
+                except Exception as ex:
+                    self.log("hash: {} @ [{}]".format(repr(ex), abspath))
+                    continue
+
+                if not hashes:
+                    return -1
+
+                wark = up2k_wark_from_hashlist(self.salt, sz, hashes)
+
+            if dw and dw != wark:
+                ip = ""
+                at = 0
+
+            # skip upload hooks by not providing vflags
+            self.db_add(db.c, {}, rd, fn, lmod, sz, "", "", wark, "", "", ip, at)
+            db.n += 1
+            ret += 1
+            td = time.time() - db.t
+            if db.n >= 4096 or td >= 60:
+                self.log("commit {} new files".format(db.n))
+                db.c.connection.commit()
+                db.n = 0
+                db.t = time.time()
 
         if not self.args.no_dhash:
             db.c.execute("delete from dh where d = ?", (drd,))  # type: ignore
@@ -1352,7 +1368,7 @@ class Up2k(object):
                 rd = drd
 
             abspath = djoin(top, rd)
-            self.pp.msg = "b{} {}".format(ndirs - nchecked, abspath)
+            self.pp.msg = "b%d %s" % (ndirs - nchecked, abspath)
             try:
                 if os.path.isdir(abspath):
                     continue
@@ -1704,7 +1720,7 @@ class Up2k(object):
                     cur.execute(q, (w[:16],))
 
             abspath = djoin(ptop, rd, fn)
-            self.pp.msg = "c{} {}".format(nq, abspath)
+            self.pp.msg = "c%d %s" % (nq, abspath)
             if not mpool:
                 n_tags = self._tagscan_file(cur, entags, w, abspath, ip, at)
             else:
@@ -1761,7 +1777,7 @@ class Up2k(object):
                     if c2.execute(q, (row[0][:16],)).fetchone():
                         continue
 
-                gf.write("{}\n".format("\x00".join(row)).encode("utf-8"))
+                gf.write(("%s\n" % ("\x00".join(row),)).encode("utf-8"))
                 n += 1
 
         c2.close()
@@ -2140,7 +2156,9 @@ class Up2k(object):
         self.log("ST: {}".format(msg))
 
     def _orz(self, db_path: str) -> "sqlite3.Cursor":
-        c = sqlite3.connect(db_path, self.timeout, check_same_thread=False).cursor()
+        c = sqlite3.connect(
+            db_path, timeout=self.timeout, check_same_thread=False
+        ).cursor()
         # c.connection.set_trace_callback(self._trace)
         return c
 
@@ -2344,7 +2362,8 @@ class Up2k(object):
                 t = "cannot receive uploads right now;\nserver busy with {}.\nPlease wait; the client will retry..."
                 raise Pebkac(503, t.format(self.blocked or "[unknown]"))
         except TypeError:
-            # py2
+            if not PY2:
+                raise
             with self.mutex:
                 self._job_volchk(cj)
 
@@ -2567,12 +2586,13 @@ class Up2k(object):
                             raise Pebkac(403, t)
 
                         if not self.args.nw:
+                            dvf: dict[str, Any] = vfs.flags
                             try:
                                 dvf = self.flags[job["ptop"]]
                                 self._symlink(src, dst, dvf, lmod=cj["lmod"], rm=True)
                             except:
                                 if bos.path.exists(dst):
-                                    bos.unlink(dst)
+                                    wunlink(self.log, dst, dvf)
                                 if not n4g:
                                     raise
 
@@ -2661,12 +2681,7 @@ class Up2k(object):
                 not ret["hash"]
                 and "fk" in vfs.flags
                 and not self.args.nw
-                and (
-                    cj["user"] in vfs.axs.uread
-                    or cj["user"] in vfs.axs.upget
-                    or "*" in vfs.axs.uread
-                    or "*" in vfs.axs.upget
-                )
+                and (cj["user"] in vfs.axs.uread or cj["user"] in vfs.axs.upget)
             ):
                 alg = 2 if "fka" in vfs.flags else 1
                 ap = absreal(djoin(job["ptop"], job["prel"], job["name"]))
@@ -2674,6 +2689,28 @@ class Up2k(object):
                 fk = self.gen_fk(alg, self.args.fk_salt, ap, job["size"], ino)
                 ret["fk"] = fk[: vfs.flags["fk"]]
 
+            if (
+                not ret["hash"]
+                and cur
+                and cj.get("umod")
+                and int(cj["lmod"]) != int(job["lmod"])
+                and not self.args.nw
+                and cj["user"] in vfs.axs.uwrite
+                and cj["user"] in vfs.axs.udel
+            ):
+                sql = "update up set mt=? where substr(w,1,16)=? and +rd=? and +fn=?"
+                try:
+                    cur.execute(sql, (cj["lmod"], wark[:16], job["prel"], job["name"]))
+                    cur.connection.commit()
+
+                    ap = djoin(job["ptop"], job["prel"], job["name"])
+                    times = (int(time.time()), int(cj["lmod"]))
+                    bos.utime(ap, times, False)
+
+                    self.log("touched %s from %d to %d" % (ap, job["lmod"], cj["lmod"]))
+                except Exception as ex:
+                    self.log("umod failed, %r" % (ex,), 3)
+
             return ret
 
     def _untaken(self, fdir: str, job: dict[str, Any], ts: float) -> str:
@@ -2686,14 +2723,14 @@ class Up2k(object):
         fp = djoin(fdir, fname)
         if job.get("replace") and bos.path.exists(fp):
             self.log("replacing existing file at {}".format(fp))
-            bos.unlink(fp)
+            wunlink(self.log, fp, self.flags.get(job["ptop"]) or {})
 
         if self.args.plain_ip:
             dip = ip.replace(":", ".")
         else:
             dip = self.hub.iphash.s(ip)
 
-        suffix = "-{:.6f}-{}".format(ts, dip)
+        suffix = "-%.6f-%s" % (ts, dip)
         with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as zfw:
             return zfw["orz"][1]
 
@@ -2744,7 +2781,7 @@ class Up2k(object):
                 ldst = ldst.replace("/", "\\")
 
             if rm and bos.path.exists(dst):
-                bos.unlink(dst)
+                wunlink(self.log, dst, flags)
 
             try:
                 if "hardlink" in flags:
@@ -2760,7 +2797,7 @@ class Up2k(object):
                     Path(ldst).symlink_to(lsrc)
                     if not bos.path.exists(dst):
                         try:
-                            bos.unlink(dst)
+                            wunlink(self.log, dst, flags)
                         except:
                             pass
                         t = "the created symlink [%s] did not resolve to [%s]"
@@ -2908,7 +2945,7 @@ class Up2k(object):
         except:
             pass
 
-        z2 += [upt]
+        z2.append(upt)
         if self.idx_wark(vflags, *z2):
             del self.registry[ptop][wark]
         else:
@@ -2931,7 +2968,6 @@ class Up2k(object):
 
             self._symlink(dst, d2, self.flags[ptop], lmod=lmod)
             if cur:
-                self.db_rm(cur, rd, fn, job["size"])
                 self.db_add(cur, vflags, rd, fn, lmod, *z2[3:])
 
         if cur:
@@ -2974,7 +3010,6 @@ class Up2k(object):
 
         self.db_act = self.vol_act[ptop] = time.time()
         try:
-            self.db_rm(cur, rd, fn, sz)
             self.db_add(
                 cur,
                 vflags,
@@ -3033,6 +3068,8 @@ class Up2k(object):
         at: float,
         skip_xau: bool = False,
     ) -> None:
+        self.db_rm(db, rd, fn, sz)
+
         sql = "insert into up values (?,?,?,?,?,?,?)"
         v = (wark, int(ts), sz, rd, fn, ip or "", int(at or 0))
         try:
@@ -3063,7 +3100,7 @@ class Up2k(object):
         ):
             t = "upload blocked by xau server config"
             self.log(t, 1)
-            bos.unlink(dst)
+            wunlink(self.log, dst, vflags)
             self.registry[ptop].pop(wark, None)
             raise Pebkac(403, t)
 
@@ -3089,7 +3126,7 @@ class Up2k(object):
                 with self.rescan_cond:
                     self.rescan_cond.notify_all()
 
-        if rd and sz and fn.lower() in self.args.th_covers:
+        if rd and sz and fn.lower() in self.args.th_coversd:
             # wasteful; db_add will re-index actual covers
             # but that won't catch existing files
             crd, cdn = rd.rsplit("/", 1) if "/" in rd else ("", rd)
@@ -3194,10 +3231,16 @@ class Up2k(object):
                         break
 
                 abspath = djoin(adir, fn)
-                st = bos.stat(abspath)
-                volpath = "{}/{}".format(vrem, fn).strip("/")
-                vpath = "{}/{}".format(dbv.vpath, volpath).strip("/")
-                self.log("rm {}\n  {}".format(vpath, abspath))
+                st = stl = bos.lstat(abspath)
+                if stat.S_ISLNK(st.st_mode):
+                    try:
+                        st = bos.stat(abspath)
+                    except:
+                        pass
+
+                volpath = ("%s/%s" % (vrem, fn)).strip("/")
+                vpath = ("%s/%s" % (dbv.vpath, volpath)).strip("/")
+                self.log("rm %s\n  %s" % (vpath, abspath))
                 _ = dbv.get(volpath, uname, *permsets[0])
                 if xbd:
                     if not runhook(
@@ -3207,7 +3250,7 @@ class Up2k(object):
                         vpath,
                         "",
                         uname,
-                        st.st_mtime,
+                        stl.st_mtime,
                         st.st_size,
                         ip,
                         0,
@@ -3228,7 +3271,7 @@ class Up2k(object):
                         if cur:
                             cur.connection.commit()
 
-                bos.unlink(abspath)
+                wunlink(self.log, abspath, dbv.flags)
                 if xad:
                     runhook(
                         self.log,
@@ -3237,7 +3280,7 @@ class Up2k(object):
                         vpath,
                         "",
                         uname,
-                        st.st_mtime,
+                        stl.st_mtime,
                         st.st_size,
                         ip,
                         0,
@@ -3354,28 +3397,27 @@ class Up2k(object):
         if bos.path.exists(dabs):
             raise Pebkac(400, "mv2: target file exists")
 
-        stl = bos.lstat(sabs)
-        try:
-            st = bos.stat(sabs)
-        except:
-            st = stl
+        is_link = is_dirlink = False
+        st = stl = bos.lstat(sabs)
+        if stat.S_ISLNK(stl.st_mode):
+            is_link = True
+            try:
+                st = bos.stat(sabs)
+                is_dirlink = stat.S_ISDIR(st.st_mode)
+            except:
+                pass  # broken symlink; keep as-is
 
         xbr = svn.flags.get("xbr")
         xar = dvn.flags.get("xar")
         if xbr:
             if not runhook(
-                self.log, xbr, sabs, svp, "", uname, st.st_mtime, st.st_size, "", 0, ""
+                self.log, xbr, sabs, svp, "", uname, stl.st_mtime, st.st_size, "", 0, ""
             ):
                 t = "move blocked by xbr server config: {}".format(svp)
                 self.log(t, 1)
                 raise Pebkac(405, t)
 
         is_xvol = svn.realpath != dvn.realpath
-        if stat.S_ISLNK(stl.st_mode):
-            is_dirlink = stat.S_ISDIR(st.st_mode)
-            is_link = True
-        else:
-            is_link = is_dirlink = False
 
         bos.makedirs(os.path.dirname(dabs))
 
@@ -3384,7 +3426,7 @@ class Up2k(object):
             t = "moving symlink from [{}] to [{}], target [{}]"
             self.log(t.format(sabs, dabs, dlabs))
             mt = bos.path.getmtime(sabs, False)
-            bos.unlink(sabs)
+            wunlink(self.log, sabs, svn.flags)
             self._symlink(dlabs, dabs, dvn.flags, False, lmod=mt)
 
             # folders are too scary, schedule rescan of both vols
@@ -3402,7 +3444,7 @@ class Up2k(object):
         c2 = self.cur.get(dvn.realpath)
 
         if ftime_ is None:
-            ftime = st.st_mtime
+            ftime = stl.st_mtime
             fsize = st.st_size
         else:
             ftime = ftime_
@@ -3444,7 +3486,16 @@ class Up2k(object):
             if is_xvol and has_dupes:
                 raise OSError(errno.EXDEV, "src is symlink")
 
-            atomic_move(sabs, dabs)
+            if is_link and st != stl:
+                # relink non-broken symlinks to still work after the move,
+                # but only resolve 1st level to maintain relativity
+                dlink = bos.readlink(sabs)
+                dlink = os.path.join(os.path.dirname(sabs), dlink)
+                dlink = bos.path.abspath(dlink)
+                self._symlink(dlink, dabs, dvn.flags, lmod=ftime)
+                wunlink(self.log, sabs, svn.flags)
+            else:
+                atomic_move(sabs, dabs)
 
         except OSError as ex:
             if ex.errno != errno.EXDEV:
@@ -3457,7 +3508,7 @@ class Up2k(object):
                 shutil.copy2(b1, b2)
             except:
                 try:
-                    os.unlink(b2)
+                    wunlink(self.log, dabs, dvn.flags)
                 except:
                     pass
 
@@ -3469,7 +3520,7 @@ class Up2k(object):
                     zb = os.readlink(b1)
                     os.symlink(zb, b2)
                 except:
-                    os.unlink(b2)
+                    wunlink(self.log, dabs, dvn.flags)
                     raise
 
             if is_link:
@@ -3479,7 +3530,7 @@ class Up2k(object):
                 except:
                     pass
 
-            os.unlink(b1)
+            wunlink(self.log, sabs, svn.flags)
 
         if xar:
             runhook(self.log, xar, dabs, dvp, "", uname, 0, 0, "", 0, "")
@@ -3611,16 +3662,19 @@ class Up2k(object):
             except:
                 self.log("relink: not found: [{}]".format(ap))
 
+        # self.log("full:\n" + "\n".join("  {:90}: {}".format(*x) for x in full.items()))
+        # self.log("links:\n" + "\n".join("  {:90}: {}".format(*x) for x in links.items()))
         if not dabs and not full and links:
             # deleting final remaining full copy; swap it with a symlink
             slabs = list(sorted(links.keys()))[0]
             ptop, rem = links.pop(slabs)
             self.log("linkswap [{}] and [{}]".format(sabs, slabs))
             mt = bos.path.getmtime(slabs, False)
-            bos.unlink(slabs)
+            flags = self.flags.get(ptop) or {}
+            wunlink(self.log, slabs, flags)
             bos.rename(sabs, slabs)
             bos.utime(slabs, (int(time.time()), int(mt)), False)
-            self._symlink(slabs, sabs, self.flags.get(ptop) or {}, False)
+            self._symlink(slabs, sabs, flags, False)
             full[slabs] = (ptop, rem)
             sabs = slabs
 
@@ -3628,18 +3682,51 @@ class Up2k(object):
             dabs = list(sorted(full.keys()))[0]
 
         for alink, parts in links.items():
-            lmod = None
+            lmod = 0.0
             try:
-                if alink != sabs and absreal(alink) != sabs:
-                    continue
+                faulty = False
+                ldst = alink
+                try:
+                    for n in range(40):  # MAXSYMLINKS
+                        zs = bos.readlink(ldst)
+                        ldst = os.path.join(os.path.dirname(ldst), zs)
+                        ldst = bos.path.abspath(ldst)
+                        if not bos.path.islink(ldst):
+                            break
 
-                self.log("relinking [{}] to [{}]".format(alink, dabs))
+                        if ldst == sabs:
+                            t = "relink because level %d would break:"
+                            self.log(t % (n,), 6)
+                            faulty = True
+                except Exception as ex:
+                    self.log("relink because walk failed: %s; %r" % (ex, ex), 3)
+                    faulty = True
+
+                zs = absreal(alink)
+                if ldst != zs:
+                    t = "relink because computed != actual destination:\n  %s\n  %s"
+                    self.log(t % (ldst, zs), 3)
+                    ldst = zs
+                    faulty = True
+
+                if bos.path.islink(ldst):
+                    raise Exception("broken symlink: %s" % (alink,))
+
+                if alink != sabs and ldst != sabs and not faulty:
+                    continue  # original symlink OK; leave it be
+
+            except Exception as ex:
+                t = "relink because symlink verification failed: %s; %r"
+                self.log(t % (ex, ex), 3)
+
+            self.log("relinking [%s] to [%s]" % (alink, dabs))
+            flags = self.flags.get(parts[0]) or {}
+            try:
                 lmod = bos.path.getmtime(alink, False)
-                bos.unlink(alink)
+                wunlink(self.log, alink, flags)
             except:
                 pass
 
-            flags = self.flags.get(parts[0]) or {}
             self._symlink(dabs, alink, flags, False, lmod=lmod or 0)
 
         return len(full) + len(links)
@@ -3686,7 +3773,7 @@ class Up2k(object):
                     return []
 
                 if self.pp:
-                    mb = int(fsz / 1024 / 1024)
+                    mb = fsz // (1024 * 1024)
                     self.pp.msg = prefix + str(mb) + suffix
 
                 hashobj = hashlib.sha512()
@@ -3751,7 +3838,7 @@ class Up2k(object):
         else:
             dip = self.hub.iphash.s(job["addr"])
 
-        suffix = "-{:.6f}-{}".format(job["t0"], dip)
+        suffix = "-%.6f-%s" % (job["t0"], dip)
         with ren_open(tnam, "wb", fdir=pdir, suffix=suffix) as zfw:
             f, job["tnam"] = zfw["orz"]
             abspath = djoin(pdir, job["tnam"])
@@ -3940,16 +4027,16 @@ class Up2k(object):
             self.log("tagged {} ({}+{})".format(abspath, ntags1, len(tags) - ntags1))
 
     def _hasher(self) -> None:
-        with self.mutex:
+        with self.hashq_mutex:
             self.n_hashq += 1
 
         while True:
-            with self.mutex:
+            with self.hashq_mutex:
                 self.n_hashq -= 1
             # self.log("hashq {}".format(self.n_hashq))
 
             task = self.hashq.get()
-            if len(task) != 8:
+            if len(task) != 9:
                 raise Exception("invalid hash task")
 
             try:
@@ -3958,11 +4045,14 @@ class Up2k(object):
             except Exception as ex:
                 self.log("failed to hash %s: %s" % (task, ex), 1)
 
-    def _hash_t(self, task: tuple[str, str, str, str, str, float, str, bool]) -> bool:
-        ptop, vtop, rd, fn, ip, at, usr, skip_xau = task
+    def _hash_t(
+        self, task: tuple[str, str, dict[str, Any], str, str, str, float, str, bool]
+    ) -> bool:
+        ptop, vtop, flags, rd, fn, ip, at, usr, skip_xau = task
         # self.log("hashq {} pop {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
-        if "e2d" not in self.flags[ptop]:
-            return True
+        with self.mutex:
+            if not self.register_vpath(ptop, flags):
+                return True
 
         abspath = djoin(ptop, rd, fn)
         self.log("hashing " + abspath)
@@ -4013,11 +4103,22 @@ class Up2k(object):
         usr: str,
         skip_xau: bool = False,
     ) -> None:
-        with self.mutex:
-            self.register_vpath(ptop, flags)
-            self.hashq.put((ptop, vtop, rd, fn, ip, at, usr, skip_xau))
+        if "e2d" not in flags:
+            return
+
+        if self.n_hashq > 1024:
+            t = "%d files in hashq; taking a nap"
+            self.log(t % (self.n_hashq,), 6)
+
+            for _ in range(self.n_hashq // 1024):
+                time.sleep(0.1)
+                if self.n_hashq < 1024:
+                    break
+
+        zt = (ptop, vtop, flags, rd, fn, ip, at, usr, skip_xau)
+        with self.hashq_mutex:
+            self.hashq.put(zt)
             self.n_hashq += 1
-        # self.log("hashq {} push {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
 
     def shutdown(self) -> None:
         self.stop = True
@@ -4068,6 +4169,6 @@ def up2k_wark_from_hashlist(salt: str, filesize: int, hashes: list[str]) -> str:
 
 
 def up2k_wark_from_metadata(salt: str, sz: int, lastmod: int, rd: str, fn: str) -> str:
-    ret = sfsenc("{}\n{}\n{}\n{}\n{}".format(salt, lastmod, sz, rd, fn))
+    ret = sfsenc("%s\n%d\n%d\n%s\n%s" % (salt, lastmod, sz, rd, fn))
     ret = base64.urlsafe_b64encode(hashlib.sha512(ret).digest())
-    return "#{}".format(ret.decode("ascii"))[:44]
+    return ("#%s" % (ret.decode("ascii"),))[:44]

copyparty/util.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
+import argparse
 import base64
 import contextlib
 import errno
@@ -167,6 +168,12 @@ except:
         return struct.unpack(fmt.decode("ascii"), a)
 
 
+try:
+    BITNESS = struct.calcsize(b"P") * 8
+except:
+    BITNESS = struct.calcsize("P") * 8
+
+
 ansi_re = re.compile("\033\\[[^mK]*[mK]")
 
 
@@ -343,6 +350,11 @@ CMD_EXEB = set(_exestr.encode("utf-8").split())
 CMD_EXES = set(_exestr.split())
 
 
+# mostly from https://github.com/github/gitignore/blob/main/Global/macOS.gitignore
+APPLESAN_TXT = r"/(__MACOS|Icon\r\r)|/\.(_|DS_Store|AppleDouble|LSOverride|DocumentRevisions-|fseventsd|Spotlight-|TemporaryItems|Trashes|VolumeIcon\.icns|com\.apple\.timemachine\.donotpresent|AppleDB|AppleDesktop|apdisk)"
+APPLESAN_RE = re.compile(APPLESAN_TXT)
+
+
 pybin = sys.executable or ""
 if EXE:
     pybin = ""
@@ -366,11 +378,6 @@ def py_desc() -> str:
     if ofs > 0:
         py_ver = py_ver[:ofs]
 
-    try:
-        bitness = struct.calcsize(b"P") * 8
-    except:
-        bitness = struct.calcsize("P") * 8
-
     host_os = platform.system()
     compiler = platform.python_compiler().split("http")[0]
 
@@ -378,7 +385,7 @@ def py_desc() -> str:
     os_ver = m.group(1) if m else ""
 
     return "{:>9} v{} on {}{} {} [{}]".format(
-        interp, py_ver, host_os, bitness, os_ver, compiler
+        interp, py_ver, host_os, BITNESS, os_ver, compiler
     )
 
 
@@ -417,8 +424,10 @@ except:
     PYFTPD_VER = "(None)"
 
 
+PY_DESC = py_desc()
+
 VERSIONS = "copyparty v{} ({})\n{}\n   sqlite v{} | jinja v{} | pyftpd v{}".format(
-    S_VERSION, S_BUILD_DT, py_desc(), SQLITE_VER, JINJA_VER, PYFTPD_VER
+    S_VERSION, S_BUILD_DT, PY_DESC, SQLITE_VER, JINJA_VER, PYFTPD_VER
 )
 
 
@@ -623,9 +632,14 @@ class _Unrecv(object):
             while nbytes > len(ret):
                 ret += self.recv(nbytes - len(ret))
         except OSError:
-            t = "client only sent {} of {} expected bytes".format(len(ret), nbytes)
-            if len(ret) <= 16:
-                t += "; got {!r}".format(ret)
+            t = "client stopped sending data; expected at least %d more bytes"
+            if not ret:
+                t = t % (nbytes,)
+            else:
+                t += ", only got %d"
+                t = t % (nbytes, len(ret))
+                if len(ret) <= 16:
+                    t += "; %r" % (ret,)
 
             if raise_on_trunc:
                 raise UnrecvEOF(5, t)
@@ -775,16 +789,20 @@ class ProgressPrinter(threading.Thread):
     periodically print progress info without linefeeds
     """
 
-    def __init__(self) -> None:
+    def __init__(self, log: "NamedLogger", args: argparse.Namespace) -> None:
         threading.Thread.__init__(self, name="pp")
         self.daemon = True
+        self.log = log
+        self.args = args
         self.msg = ""
         self.end = False
         self.n = -1
         self.start()
 
     def run(self) -> None:
+        tp = 0
         msg = None
+        no_stdout = self.args.q
         fmt = " {}\033[K\r" if VT100 else " {} $\r"
         while not self.end:
             time.sleep(0.1)
@@ -792,10 +810,21 @@ class ProgressPrinter(threading.Thread):
                 continue
 
             msg = self.msg
+            now = time.time()
+            if msg and now - tp > 10:
+                tp = now
+                self.log("progress: %s" % (msg,), 6)
+
+            if no_stdout:
+                continue
+
             uprint(fmt.format(msg))
             if PY2:
                 sys.stdout.flush()
 
+        if no_stdout:
+            return
+
         if VT100:
             print("\033[K", end="")
         elif msg:
@@ -849,7 +878,7 @@ class MTHash(object):
                     ex = ex or str(qe)
 
                 if pp:
-                    mb = int((fsz - nch * chunksz) / 1024 / 1024)
+                    mb = (fsz - nch * chunksz) // (1024 * 1024)
                     pp.msg = prefix + str(mb) + suffix
 
             if ex:
@@ -1071,7 +1100,18 @@ def uprint(msg: str) -> None:
 
 
 def nuprint(msg: str) -> None:
-    uprint("{}\n".format(msg))
+    uprint("%s\n" % (msg,))
+
+
+def dedent(txt: str) -> str:
+    pad = 64
+    lns = txt.replace("\r", "").split("\n")
+    for ln in lns:
+        zs = ln.lstrip()
+        pad2 = len(ln) - len(zs)
+        if zs and pad > pad2:
+            pad = pad2
+    return "\n".join([ln[pad:] for ln in lns])
 
 
 def rice_tid() -> str:
@@ -1083,10 +1123,10 @@ def rice_tid() -> str:
 def trace(*args: Any, **kwargs: Any) -> None:
     t = time.time()
     stack = "".join(
-        "\033[36m{}\033[33m{}".format(x[0].split(os.sep)[-1][:-3], x[1])
+        "\033[36m%s\033[33m%s" % (x[0].split(os.sep)[-1][:-3], x[1])
         for x in traceback.extract_stack()[3:-1]
     )
-    parts = ["{:.6f}".format(t), rice_tid(), stack]
+    parts = ["%.6f" % (t,), rice_tid(), stack]
 
     if args:
         parts.append(repr(args))
@@ -1103,17 +1143,17 @@ def alltrace() -> str:
     threads: dict[str, types.FrameType] = {}
     names = dict([(t.ident, t.name) for t in threading.enumerate()])
     for tid, stack in sys._current_frames().items():
-        name = "{} ({:x})".format(names.get(tid), tid)
+        name = "%s (%x)" % (names.get(tid), tid)
         threads[name] = stack
 
     rret: list[str] = []
     bret: list[str] = []
     for name, stack in sorted(threads.items()):
-        ret = ["\n\n# {}".format(name)]
+        ret = ["\n\n# %s" % (name,)]
         pad = None
         for fn, lno, name, line in traceback.extract_stack(stack):
             fn = os.sep.join(fn.split(os.sep)[-3:])
-            ret.append('File: "{}", line {}, in {}'.format(fn, lno, name))
+            ret.append('File: "%s", line %d, in %s' % (fn, lno, name))
             if line:
                 ret.append("  " + str(line.strip()))
                 if "self.not_empty.wait()" in line:
@@ -1122,7 +1162,7 @@ def alltrace() -> str:
         if pad:
             bret += [ret[0]] + [pad + x for x in ret[1:]]
         else:
-            rret += ret
+            rret.extend(ret)
 
     return "\n".join(rret + bret) + "\n"
 
@@ -1205,12 +1245,20 @@ def log_thrs(log: Callable[[str, str, int], None], ival: float, name: str) -> No
 
 
 def vol_san(vols: list["VFS"], txt: bytes) -> bytes:
+    txt0 = txt
     for vol in vols:
-        txt = txt.replace(vol.realpath.encode("utf-8"), vol.vpath.encode("utf-8"))
-        txt = txt.replace(
-            vol.realpath.encode("utf-8").replace(b"\\", b"\\\\"),
-            vol.vpath.encode("utf-8"),
-        )
+        bap = vol.realpath.encode("utf-8")
+        bhp = vol.histpath.encode("utf-8")
+        bvp = vol.vpath.encode("utf-8")
+        bvph = b"$hist(/" + bvp + b")"
+
+        txt = txt.replace(bap, bvp)
+        txt = txt.replace(bhp, bvph)
+        txt = txt.replace(bap.replace(b"\\", b"\\\\"), bvp)
+        txt = txt.replace(bhp.replace(b"\\", b"\\\\"), bvph)
+
+    if txt != txt0:
+        txt += b"\r\nNOTE: filepaths sanitized; see serverlog for correct values"
 
     return txt
 
@@ -1218,9 +1266,9 @@ def vol_san(vols: list["VFS"], txt: bytes) -> bytes:
 def min_ex(max_lines: int = 8, reverse: bool = False) -> str:
     et, ev, tb = sys.exc_info()
     stb = traceback.extract_tb(tb)
-    fmt = "{} @ {} <{}>: {}"
-    ex = [fmt.format(fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in stb]
-    ex.append("[{}] {}".format(et.__name__ if et else "(anonymous)", ev))
+    fmt = "%s @ %d <%s>: %s"
+    ex = [fmt % (fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in stb]
+    ex.append("[%s] %s" % (et.__name__ if et else "(anonymous)", ev))
     return "\n".join(ex[-max_lines:][:: -1 if reverse else 1])
 
 
@@ -1271,7 +1319,7 @@ def ren_open(
             with fun(fsenc(fpath), *args, **kwargs) as f:
                 if b64:
                     assert fdir
-                    fp2 = "fn-trunc.{}.txt".format(b64)
+                    fp2 = "fn-trunc.%s.txt" % (b64,)
                     fp2 = os.path.join(fdir, fp2)
                     with open(fsenc(fp2), "wb") as f2:
                         f2.write(orig_name.encode("utf-8"))
@@ -1300,7 +1348,7 @@ def ren_open(
                 raise
 
         if not b64:
-            zs = "{}\n{}".format(orig_name, suffix).encode("utf-8", "replace")
+            zs = ("%s\n%s" % (orig_name, suffix)).encode("utf-8", "replace")
             zs = hashlib.sha512(zs).digest()[:12]
             b64 = base64.urlsafe_b64encode(zs).decode("utf-8")
 
@@ -1320,7 +1368,7 @@ def ren_open(
                     # okay do the first letter then
                     ext = "." + ext[2:]
 
-            fname = "{}~{}{}".format(bname, b64, ext)
+            fname = "%s~%s%s" % (bname, b64, ext)
 
 
 class MultipartParser(object):
@@ -1506,15 +1554,20 @@ class MultipartParser(object):
         return ret
 
     def parse(self) -> None:
+        boundary = get_boundary(self.headers)
+        self.log("boundary=%r" % (boundary,))
+
         # spec says there might be junk before the first boundary,
         # can't have the leading \r\n if that's not the case
-        self.boundary = b"--" + get_boundary(self.headers).encode("utf-8")
+        self.boundary = b"--" + boundary.encode("utf-8")
 
         # discard junk before the first boundary
         for junk in self._read_data():
-            self.log(
-                "discarding preamble: [{}]".format(junk.decode("utf-8", "replace"))
-            )
+            if not junk:
+                continue
+
+            jtxt = junk.decode("utf-8", "replace")
+            self.log("discarding preamble |%d| %r" % (len(junk), jtxt))
 
         # nice, now make it fast
         self.boundary = b"\r\n" + self.boundary
@@ -1526,11 +1579,9 @@ class MultipartParser(object):
         raises if the field name is not as expected
         """
         assert self.gen
-        p_field, _, p_data = next(self.gen)
+        p_field, p_fname, p_data = next(self.gen)
         if p_field != field_name:
-            raise Pebkac(
-                422, 'expected field "{}", got "{}"'.format(field_name, p_field)
-            )
+            raise WrongPostKey(field_name, p_field, p_fname, p_data)
 
         return self._read_value(p_data, max_len).decode("utf-8", "surrogateescape")
 
@@ -1599,7 +1650,7 @@ def rand_name(fdir: str, fn: str, rnd: int) -> str:
                 break
 
             nc = rnd + extra
-            nb = int((6 + 6 * nc) / 8)
+            nb = (6 + 6 * nc) // 8
             zb = os.urandom(nb)
             zb = base64.urlsafe_b64encode(zb)
             fn = zb[:nc].decode("utf-8") + ext
@@ -1702,7 +1753,7 @@ def get_spd(nbyte: int, t0: float, t: Optional[float] = None) -> str:
     bps = nbyte / ((t - t0) + 0.001)
     s1 = humansize(nbyte).replace(" ", "\033[33m").replace("iB", "")
     s2 = humansize(bps).replace(" ", "\033[35m").replace("iB", "")
-    return "{} \033[0m{}/s\033[0m".format(s1, s2)
+    return "%s \033[0m%s/s\033[0m" % (s1, s2)
 
 
 def s2hms(s: float, optional_h: bool = False) -> str:
@@ -1710,9 +1761,9 @@ def s2hms(s: float, optional_h: bool = False) -> str:
     h, s = divmod(s, 3600)
     m, s = divmod(s, 60)
     if not h and optional_h:
-        return "{}:{:02}".format(m, s)
+        return "%d:%02d" % (m, s)
 
-    return "{}:{:02}:{:02}".format(h, m, s)
+    return "%d:%02d:%02d" % (h, m, s)
 
 
 def djoin(*paths: str) -> str:
@@ -1823,7 +1874,9 @@ def exclude_dotfiles(filepaths: list[str]) -> list[str]:
     return [x for x in filepaths if not x.split("/")[-1].startswith(".")]
 
 
-def odfusion(base: Union[ODict[str, bool], ODict["LiteralString", bool]], oth: str) -> ODict[str, bool]:
+def odfusion(
+    base: Union[ODict[str, bool], ODict["LiteralString", bool]], oth: str
+) -> ODict[str, bool]:
     # merge an "ordered set" (just a dict really) with another list of keys
     words0 = [x for x in oth.split(",") if x]
     words1 = [x for x in oth[1:].split(",") if x]
@@ -2052,6 +2105,47 @@ def atomic_move(usrc: str, udst: str) -> None:
         os.rename(src, dst)
 
 
+def wunlink(log: "NamedLogger", abspath: str, flags: dict[str, Any]) -> bool:
+    maxtime = flags.get("rm_re_t", 0.0)
+    bpath = fsenc(abspath)
+    if not maxtime:
+        os.unlink(bpath)
+        return True
+
+    chill = flags.get("rm_re_r", 0.0)
+    if chill < 0.001:
+        chill = 0.1
+
+    ino = 0
+    t0 = now = time.time()
+    for attempt in range(90210):
+        try:
+            if ino and os.stat(bpath).st_ino != ino:
+                log("inode changed; aborting delete")
+                return False
+            os.unlink(bpath)
+            if attempt:
+                now = time.time()
+                t = "deleted in %.2f sec, attempt %d"
+                log(t % (now - t0, attempt + 1))
+            return True
+        except OSError as ex:
+            now = time.time()
+            if ex.errno == errno.ENOENT:
+                return False
+            if now - t0 > maxtime or attempt == 90209:
+                raise
+            if not attempt:
+                if not PY2:
+                    ino = os.stat(bpath).st_ino
+                t = "delete failed (err.%d); retrying for %d sec: %s"
+                log(t % (ex.errno, maxtime + 0.99, abspath))
+
+        time.sleep(chill)
+
+    return False  # makes pylance happy
+
+
 def get_df(abspath: str) -> tuple[Optional[int], Optional[int]]:
     try:
         # some fuses misbehave
@@ -2180,7 +2274,7 @@ def read_socket_chunked(
             raise Pebkac(400, t.format(x))
 
         if log:
-            log("receiving {} byte chunk".format(chunklen))
+            log("receiving %d byte chunk" % (chunklen,))
 
         for chunk in read_socket(sr, chunklen):
             yield chunk
@@ -2352,6 +2446,12 @@ def statdir(
             print(t)
 
 
+def dir_is_empty(logger: "RootLogger", scandir: bool, top: str):
+    for _ in statdir(logger, scandir, False, top):
+        return False
+    return True
+
+
 def rmdirs(
     logger: "RootLogger", scandir: bool, lstat: bool, top: str, depth: int
 ) -> tuple[list[str], list[str]]:
@@ -2544,6 +2644,7 @@ def runcmd(
     argv: Union[list[bytes], list[str]], timeout: Optional[float] = None, **ka: Any
 ) -> tuple[int, str, str]:
     isbytes = isinstance(argv[0], (bytes, bytearray))
+    oom = ka.pop("oom", 0)  # 0..1000
     kill = ka.pop("kill", "t")  # [t]ree [m]ain [n]one
     capture = ka.pop("capture", 3)  # 0=none 1=stdout 2=stderr 3=both
 
@@ -2574,6 +2675,14 @@ def runcmd(
                 argv = [NICES] + argv
 
     p = sp.Popen(argv, stdout=cout, stderr=cerr, **ka)
+
+    if oom and not ANYWIN and not MACOS:
+        try:
+            with open("/proc/%d/oom_score_adj" % (p.pid,), "wb") as f:
+                f.write(("%d\n" % (oom,)).encode("utf-8"))
+        except:
+            pass
+
     if not timeout or PY2:
         bout, berr = p.communicate(sin)
     else:
@@ -2721,6 +2830,7 @@ def _parsehook(
     sp_ka = {
         "env": env,
         "nice": True,
+        "oom": 300,
         "timeout": tout,
         "kill": kill,
         "capture": cap,
@@ -2937,7 +3047,7 @@ def visual_length(txt: str) -> int:
                 pend = None
             else:
                 if ch == "\033":
-                    pend = "{0}".format(ch)
+                    pend = "%s" % (ch,)
                 else:
                     co = ord(ch)
                     # the safe parts of latin1 and cp437 (no greek stuff)
@@ -3035,3 +3145,20 @@ class Pebkac(Exception):
 
     def __repr__(self) -> str:
         return "Pebkac({}, {})".format(self.code, repr(self.args))
+
+
+class WrongPostKey(Pebkac):
+    def __init__(
+        self,
+        expected: str,
+        got: str,
+        fname: Optional[str],
+        datagen: Generator[bytes, None, None],
+    ) -> None:
+        msg = 'expected field "{}", got "{}"'.format(expected, got)
+        super(WrongPostKey, self).__init__(422, msg)
+
+        self.expected = expected
+        self.got = got
+        self.fname = fname
+        self.datagen = datagen

copyparty/web/baguettebox.js

@@ -255,19 +255,19 @@ window.baguetteBox = (function () {
         if (anymod(e, true))
             return;
 
-        var k = e.code + '', v = vid(), pos = -1;
+        var k = (e.code || e.key) + '', v = vid(), pos = -1;
 
         if (k == "BracketLeft")
             setloop(1);
         else if (k == "BracketRight")
             setloop(2);
-        else if (e.shiftKey && k != 'KeyR')
+        else if (e.shiftKey && k != "KeyR" && k != "R")
             return;
-        else if (k == "ArrowLeft" || k == "KeyJ")
+        else if (k == "ArrowLeft" || k == "KeyJ" || k == "Left" || k == "j")
             showPreviousImage();
-        else if (k == "ArrowRight" || k == "KeyL")
+        else if (k == "ArrowRight" || k == "KeyL" || k == "Right" || k == "l")
             showNextImage();
-        else if (k == "Escape")
+        else if (k == "Escape" || k == "Esc")
             hideOverlay();
         else if (k == "Home")
             showFirstImage(e);
@@ -295,9 +295,9 @@ window.baguetteBox = (function () {
         }
         else if (k == "KeyF")
             tglfull();
-        else if (k == "KeyS")
+        else if (k == "KeyS" || k == "s")
             tglsel();
-        else if (k == "KeyR")
+        else if (k == "KeyR" || k == "r" || k == "R")
             rotn(e.shiftKey ? -1 : 1);
         else if (k == "KeyY")
             dlpic();

copyparty/web/browser.css

@@ -818,6 +818,10 @@ html.y #path a:hover {
 .logue:empty {
 	display: none;
 }
+.logue.raw {
+	white-space: pre;
+	font-family: 'scp', 'consolas', monospace;
+}
 #doc>iframe,
 .logue>iframe {
 	background: var(--bgg);
@@ -1653,7 +1657,9 @@ html.cz .tgl.btn.on {
 	color: var(--fg-max);
 }
 #tree ul a.hl {
+	color: #fff;
 	color: var(--btn-1-fg);
+	background: #000;
 	background: var(--btn-1-bg);
 	text-shadow: none;
 }
@@ -2174,6 +2180,7 @@ html.y #bbox-overlay figcaption a {
 }
 #bbox-halp {
 	color: var(--fg-max);
+	background: #fff;
 	background: var(--bg);
 	position: absolute;
 	top: 0;

copyparty/web/browser.html

@@ -148,7 +148,8 @@
 			logues = {{ logues|tojson if sb_lg else "[]" }},
 			ls0 = {{ ls0|tojson }};
 
-		document.documentElement.className = localStorage.cpp_thm || dtheme;
+		var STG = window.localStorage;
+		document.documentElement.className = (STG && STG.cpp_thm) || dtheme;
 	</script>
 	<script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>
 	<script src="{{ r }}/.cpr/baguettebox.js?_={{ ts }}"></script>

copyparty/web/browser.js

@@ -200,6 +200,8 @@ var Ls = {
 		"ct_idxh": "show index.html instead of folder listing",
 		"ct_sbars": "show scrollbars",
 
+		"cut_umod": "if a file already exists on the server, update the server's last-modified timestamp to match your local file (requires write+delete permissions)",
+
 		"cut_turbo": "the yolo button, you probably DO NOT want to enable this:$N$Nuse this if you were uploading a huge amount of files and had to restart for some reason, and want to continue the upload ASAP$N$Nthis replaces the hash-check with a simple <em>&quot;does this have the same filesize on the server?&quot;</em> so if the file contents are different it will NOT be uploaded$N$Nyou should turn this off when the upload is done, and then &quot;upload&quot; the same files again to let the client verify them",
 
 		"cut_datechk": "has no effect unless the turbo button is enabled$N$Nreduces the yolo factor by a tiny amount; checks whether the file timestamps on the server matches yours$N$Nshould <em>theoretically</em> catch most unfinished / corrupted uploads, but is not a substitute for doing a verification pass with turbo disabled afterwards",
@@ -229,12 +231,14 @@ var Ls = {
 		"tt_wrap": "word wrap",
 		"tt_hover": "reveal overflowing lines on hover$N( breaks scrolling unless mouse $N&nbsp; cursor is in the left gutter )",
 
-		"ml_pmode": "playback mode",
+		"ml_pmode": "at end of folder...",
+		"ml_btns": "cmds",
 		"ml_tcode": "transcode",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer",
 		"ml_drc": "dynamic range compressor",
 
+		"mt_shuf": "shuffle the songs in each folder\">🔀",
 		"mt_preload": "start loading the next song near the end for gapless playback\">preload",
 		"mt_fullpre": "try to preload the entire song;$N✅ enable on <b>unreliable</b> connections,$N❌ <b>disable</b> on slow connections probably\">full",
 		"mt_waves": "waveform seekbar:$Nshow audio amplitude in the scrubber\">~s",
@@ -244,6 +248,7 @@ var Ls = {
 		"mt_oscv": "show album cover in osd\">art",
 		"mt_follow": "keep the playing track scrolled into view\">🎯",
 		"mt_compact": "compact controls\">⟎",
+		"mt_uncache": "clear cache &nbsp;(try this if your browser cached$Na broken copy of a song so it refuses to play)\">uncache",
 		"mt_mloop": "loop the open folder\">🔁 loop",
 		"mt_mnext": "load the next folder and continue\">📂 next",
 		"mt_cflac": "convert flac / wav to opus\">flac",
@@ -267,6 +272,7 @@ var Ls = {
 		"mm_e403": "Could not play audio; error 403: Access denied.\n\nTry pressing F5 to reload, maybe you got logged out",
 		"mm_e5xx": "Could not play audio; server error ",
 		"mm_nof": "not finding any more audio files nearby",
+		"mm_uncache": "cache cleared; all songs will redownload on next playback",
 		"mm_pwrsv": "<p>it looks like playback is being interrupted by your phone's power-saving settings!</p>" + '<p>please go to <a target="_blank" href="https://user-images.githubusercontent.com/241032/235262121-2ffc51ae-7821-4310-a322-c3b7a507890c.png">the app settings of your browser</a> and then <a target="_blank" href="https://user-images.githubusercontent.com/241032/235262123-c328cca9-3930-4948-bd18-3949b9fd3fcf.png">allow unrestricted battery usage</a> to fix it.</p><p><em>however,</em> it could also be due to the browser\'s autoplay settings;</p><p>Firefox: tap the icon on the left side of the address bar, then select "autoplay" and "allow audio"</p><p>Chrome: the problem will gradually dissipate as you play more music on this site</p>',
 		"mm_iosblk": "<p>your web browser thinks the audio playback is unwanted, and it decided to block playback until you start another track manually... unfortunately we are both powerless in telling it otherwise</p><p>supposedly this will get better as you continue playing music on this site, but I'm unfamiliar with apple devices so idk if that's true</p><p>you could try another browser, maybe firefox or chrome?</p>",
 		"mm_hnf": "that song no longer exists",
@@ -295,6 +301,8 @@ var Ls = {
 		"frb_apply": "APPLY RENAME",
 		"fr_adv": "batch / metadata / pattern renaming\">advanced",
 		"fr_case": "case-sensitive regex\">case",
+		"fr_win": "windows-safe names; replace <code>&lt;&gt;:&quot;\\|?*</code> with japanese fullwidth characters\">win",
+		"fr_slash": "replace <code>/</code> with a character that doesn't cause new folders to be created\">no /",
 		"fr_pdel": "delete",
 		"fr_pnew": "save as",
 		"fr_pname": "provide a name for your new preset",
@@ -304,6 +312,7 @@ var Ls = {
 		"fr_tags": "tags for the selected files (read-only, just for reference):",
 		"fr_busy": "renaming {0} items...\n\n{1}",
 		"fr_efail": "rename failed:\n",
+		"fr_nchg": "{0} of the new names were altered due to <code>win</code> and/or <code>ikke /</code>\n\nOK to continue with these altered new names?",
 
 		"fd_ok": "delete OK",
 		"fd_err": "delete failed:\n",
@@ -369,7 +378,7 @@ var Ls = {
 		"s_t1": "tags contains &nbsp; (^=start, end=$)",
 		"s_a1": "specific metadata properties",
 
-		"md_eshow": "cannot show ",
+		"md_eshow": "cannot render ",
 		"md_off": "[📜<em>readme</em>] disabled in [⚙️] -- document hidden",
 
 		"xhr403": "403: Access denied\n\ntry pressing F5, maybe you got logged out",
@@ -680,6 +689,8 @@ var Ls = {
 		"ct_idxh": "vis index.html istedenfor fil-liste",
 		"ct_sbars": "vis rullgardiner / skrollefelt",
 
+		"cut_umod": "i tilfelle en fil du laster opp allerede finnes på serveren, så skal serverens tidsstempel oppdateres slik at det stemmer overens med din lokale fil (krever rettighetene write+delete)",
+
 		"cut_turbo": "forenklet befaring ved opplastning; bør sannsynlig <em>ikke</em> skrus på:$N$Nnyttig dersom du var midt i en svær opplastning som måtte restartes av en eller annen grunn, og du vil komme igang igjen så raskt som overhodet mulig.$N$Nnår denne er skrudd på så forenkles befaringen kraftig; istedenfor å utføre en trygg sjekk på om filene finnes på serveren i god stand, så sjekkes kun om <em>filstørrelsen</em> stemmer. Så dersom en korrupt fil skulle befinne seg på serveren allerede, på samme sted med samme størrelse og navn, så blir det <em>ikke oppdaget</em>.$N$Ndet anbefales å kun benytte denne funksjonen for å komme seg raskt igjennom selve opplastningen, for så å skru den av, og til slutt &quot;laste opp&quot; de samme filene én gang til -- slik at integriteten kan verifiseres",
 
 		"cut_datechk": "har ingen effekt dersom turbo er avslått$N$Ngjør turbo bittelitt tryggere ved å sjekke datostemplingen på filene (i tillegg til filstørrelse)$N$N<em>burde</em> oppdage og gjenoppta de fleste ufullstendige opplastninger, men er <em>ikke</em> en fullverdig erstatning for å deaktivere turbo og gjøre en skikkelig sjekk",
@@ -709,12 +720,14 @@ var Ls = {
 		"tt_wrap": "linjebryting",
 		"tt_hover": "vis hele mappenavnet når musepekeren treffer mappen$N( gjør dessverre at scrollhjulet fusker dersom musepekeren ikke befinner seg i grøfta )",
 
-		"ml_pmode": "spillemodus",
+		"ml_pmode": "ved enden av mappen",
+		"ml_btns": "knapper",
 		"ml_tcode": "konvertering",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer (tonejustering)",
 		"ml_drc": "compressor (volum-utjevning)",
 
+		"mt_shuf": "sangene i hver mappe$Nspilles i tilfeldig rekkefølge\">🔀",
 		"mt_preload": "hent ned litt av neste sang i forkant,$Nslik at pausen i overgangen blir mindre\">forles",
 		"mt_fullpre": "hent ned hele neste sang, ikke bare litt:$N✅ skru på hvis nettet ditt er <b>ustabilt</b>,$N❌ skru av hvis nettet ditt er <b>tregt</b>\">full",
 		"mt_waves": "waveform seekbar:$Nvis volumkurve i avspillingsfeltet\">~s",
@@ -724,6 +737,7 @@ var Ls = {
 		"mt_oscv": "vis album-cover på infoskjermen\">bilde",
 		"mt_follow": "bla slik at sangen som spilles alltid er synlig\">🎯",
 		"mt_compact": "tettpakket avspillerpanel\">⟎",
+		"mt_uncache": "prøv denne hvis en sang ikke spiller riktig\">uncache",
 		"mt_mloop": "repeter hele mappen\">🔁 gjenta",
 		"mt_mnext": "hopp til neste mappe og fortsett\">📂 neste",
 		"mt_cflac": "konverter flac / wav-filer til opus\">flac",
@@ -747,6 +761,7 @@ var Ls = {
 		"mm_e403": "Avspilling feilet: Tilgang nektet.\n\nKanskje du ble logget ut?\nPrøv å trykk F5 for å laste siden på nytt.",
 		"mm_e5xx": "Avspilling feilet: ",
 		"mm_nof": "finner ikke flere sanger i nærheten",
+		"mm_uncache": "alle sanger vil lastes på nytt ved neste avspilling",
 		"mm_pwrsv": "<p>det ser ut som musikken ble avbrutt av telefonen sine strømsparings-innstillinger!</p>" + '<p>ta en tur innom <a target="_blank" href="https://user-images.githubusercontent.com/241032/235262121-2ffc51ae-7821-4310-a322-c3b7a507890c.png">app-innstillingene til nettleseren din</a> og så <a target="_blank" href="https://user-images.githubusercontent.com/241032/235262123-c328cca9-3930-4948-bd18-3949b9fd3fcf.png">tillat ubegrenset batteriforbruk</a></p><p>NB: det kan også være pga. autoplay-innstillingene, så prøv dette:</p><p>Firefox: klikk på ikonet i venstre side av addressefeltet, velg "autoplay" og "tillat lyd"</p><p>Chrome: problemet vil minske gradvis jo mer musikk du spiller på denne siden</p>',
 		"mm_iosblk": "<p>nettleseren din tror at musikken er uønsket, og den bestemte seg for å stoppe avspillingen slik at du manuelt må velge en ny sang... dessverre er både du og jeg maktesløse når den har bestemt seg.</p><p>det ryktes at problemet vil minske jo mer musikk du spiller på denne siden, men jeg er ikke godt kjent med apple-dingser så jeg er ikke sikker.</p><p>kanskje firefox eller chrome fungerer bedre?</p>",
 		"mm_hnf": "sangen finnes ikke lenger",
@@ -775,6 +790,8 @@ var Ls = {
 		"frb_apply": "IVERKSETT",
 		"fr_adv": "automasjon basert på metadata<br>og / eller mønster (regulære uttrykk)\">avansert",
 		"fr_case": "versalfølsomme uttrykk\">Aa",
+		"fr_win": "bytt ut bokstavene <code>&lt;&gt;:&quot;\\|?*</code> med$Ntilsvarende som windows ikke får panikk av\">win",
+		"fr_slash": "bytt ut bokstaven <code>/</code> slik at den ikke forårsaker at nye mapper opprettes\">ikke /",
 		"fr_pdel": "slett",
 		"fr_pnew": "lagre som",
 		"fr_pname": "gi innstillingene dine et navn",
@@ -784,6 +801,7 @@ var Ls = {
 		"fr_tags": "metadata for de valgte filene (kun for referanse):",
 		"fr_busy": "endrer navn på {0} filer...\n\n{1}",
 		"fr_efail": "endring av navn feilet:\n",
+		"fr_nchg": "{0} av navnene ble justert pga. <code>win</code> og/eller <code>ikke /</code>\n\nvil du fortsette med de nye navnene som ble valgt?",
 
 		"fd_ok": "sletting OK",
 		"fd_err": "sletting feilet:\n",
@@ -849,7 +867,7 @@ var Ls = {
 		"s_t1": "sang-info inneholder",
 		"s_a1": "konkrete egenskaper",
 
-		"md_eshow": "kan ikke vise ",
+		"md_eshow": "viser forenklet ",
 		"md_off": "[📜<em>readme</em>] er avskrudd i [⚙️] -- dokument skjult",
 
 		"xhr403": "403: Tilgang nektet\n\nkanskje du ble logget ut? prøv å trykk F5",
@@ -1077,7 +1095,7 @@ ebi('op_up2k').innerHTML = (
 	'	<tr>\n' +
 	'		<td class="c" data-perm="read">\n' +
 	'			<a href="#" class="b" id="nthread_sub">&ndash;</a><input\n' +
-	'				class="txtbox" id="nthread" value="2" tt="' + L.ut_par + '"/><a\n' +
+	'				class="txtbox" id="nthread" value="" tt="' + L.ut_par + '"/><a\n' +
 	'				href="#" class="b" id="nthread_add">+</a><br />&nbsp;\n' +
 	'		</td>\n' +
 	'	</tr>\n' +
@@ -1177,6 +1195,7 @@ ebi('op_cfg').innerHTML = (
 	'	<h3>' + L.cl_uopts + '</h3>\n' +
 	'	<div>\n' +
 	'		<a id="ask_up" class="tgl btn" href="#" tt="' + L.ut_ask + '">💭</a>\n' +
+	'		<a id="umod" class="tgl btn" href="#" tt="' + L.cut_umod + '">re📅</a>\n' +
 	'		<a id="hashw" class="tgl btn" href="#" tt="' + L.cut_mt + '">mt</a>\n' +
 	'		<a id="u2turbo" class="tgl btn ttb" href="#" tt="' + L.cut_turbo + '">turbo</a>\n' +
 	'		<a id="u2tdate" class="tgl btn ttb" href="#" tt="' + L.cut_datechk + '">date-chk</a>\n' +
@@ -1353,6 +1372,7 @@ function set_files_html(html) {
 
 // actx breaks background album playback on ios
 var ACtx = !IPHONE && (window.AudioContext || window.webkitAudioContext),
+	ACB = sread('au_cbv') || 1,
 	noih = /[?&]v\b/.exec('' + location),
 	hash0 = location.hash,
 	mp;
@@ -1363,6 +1383,7 @@ var mpl = (function () {
 
 	ebi('op_player').innerHTML = (
 		'<div><h3>' + L.cl_opts + '</h3><div>' +
+		'<a href="#" class="tgl btn" id="au_shuf" tt="' + L.mt_shuf + '</a>' +
 		'<a href="#" class="tgl btn" id="au_preload" tt="' + L.mt_preload + '</a>' +
 		'<a href="#" class="tgl btn" id="au_fullpre" tt="' + L.mt_fullpre + '</a>' +
 		'<a href="#" class="tgl btn" id="au_waves" tt="' + L.mt_waves + '</a>' +
@@ -1374,6 +1395,10 @@ var mpl = (function () {
 		'<a href="#" class="tgl btn" id="au_compact" tt="' + L.mt_compact + '</a>' +
 		'</div></div>' +
 
+		'<div><h3>' + L.ml_btns + '</h3><div>' +
+		'<a href="#" class="btn" id="au_uncache" tt="' + L.mt_uncache + '</a>' +
+		'</div></div>' +
+
 		'<div><h3>' + L.ml_pmode + '</h3><div id="pb_mode">' +
 		'<a href="#" class="tgl btn" m="loop" tt="' + L.mt_mloop + '</a>' +
 		'<a href="#" class="tgl btn" m="next" tt="' + L.mt_mnext + '</a>' +
@@ -1400,6 +1425,9 @@ var mpl = (function () {
 		"os_ctl": bcfg_get('au_os_ctl', have_mctl) && have_mctl,
 		'traversals': 0,
 	};
+	bcfg_bind(r, 'shuf', 'au_shuf', false, function () {
+		mp.read_order();  // don't bind
+	});
 	bcfg_bind(r, 'preload', 'au_preload', true);
 	bcfg_bind(r, 'fullpre', 'au_fullpre', false);
 	bcfg_bind(r, 'waves', 'au_waves', true, function (v) {
@@ -1422,6 +1450,14 @@ var mpl = (function () {
 		r.fullpre = false;
 	}
 
+	ebi('au_uncache').onclick = function (e) {
+		ev(e);
+		ACB = (Date.now() % 46656).toString(36);
+		swrite('au_cbv', ACB);
+		reload_mp();
+		toast.inf(5, L.mm_uncache);
+	};
+
 	ebi('au_os_ctl').onclick = function (e) {
 		ev(e);
 		r.os_ctl = !r.os_ctl && have_mctl;
@@ -1644,6 +1680,20 @@ function MPlayer() {
 			r.au.volume = r.expvol(r.vol);
 	};
 
+	r.shuffle = function () {
+		if (!mpl.shuf)
+			return;
+
+		// durstenfeld
+		for (var a = r.order.length - 1; a > 0; a--) {
+			var b = Math.floor(Math.random() * (a + 1)),
+				c = r.order[a];
+			r.order[a] = r.order[b];
+			r.order[b] = c;
+		}
+	};
+	r.shuffle();
+
 	r.read_order = function () {
 		var order = [],
 			links = QSA('#files>tbody>tr>td:nth-child(1)>a');
@@ -1656,6 +1706,7 @@ function MPlayer() {
 			order.push(tid.slice(1));
 		}
 		r.order = order;
+		r.shuffle();
 	};
 
 	r.fdir = 0;
@@ -1712,7 +1763,7 @@ function MPlayer() {
 
 	r.preload = function (url, full) {
 		url = mpl.acode(url);
-		url += (url.indexOf('?') < 0 ? '?' : '&') + 'cache=987';
+		url += (url.indexOf('?') < 0 ? '?' : '&') + 'cache=987&_=' + ACB;
 		mpl.preload_url = full ? url : null;
 		var t0 = Date.now();
 
@@ -2303,7 +2354,7 @@ function dl_song() {
 	}
 
 	var url = mp.tracks[mp.au.tid];
-	url += (url.indexOf('?') < 0 ? '?' : '&') + 'cache=987';
+	url += (url.indexOf('?') < 0 ? '?' : '&') + 'cache=987&_=' + ACB;
 	dl_file(url);
 }
 
@@ -2953,7 +3004,7 @@ function play(tid, is_ev, seek) {
 	}
 
 	var url = mpl.acode(mp.tracks[tid]);
-	url += (url.indexOf('?') < 0 ? '?' : '&') + 'cache=987';
+	url += (url.indexOf('?') < 0 ? '?' : '&') + 'cache=987&_=' + ACB;
 
 	if (mp.au.rsrc == url)
 		mp.au.currentTime = 0;
@@ -3359,8 +3410,8 @@ function sortfiles(nodes) {
 	}
 	catch (ex) {
 		console.log("failed to apply sort config: " + ex);
-		console.log("resetting fsort " + sread('fsort'))
-		localStorage.removeItem('fsort');
+		console.log("resetting fsort " + sread('fsort'));
+		sdrop('fsort');
 	}
 	return nodes;
 }
@@ -3605,6 +3656,8 @@ var fileman = (function () {
 			'<button id="rn_apply">✅ ' + L.frb_apply + '</button>',
 			'<a id="rn_adv" class="tgl btn" href="#" tt="' + L.fr_adv + '</a>',
 			'<a id="rn_case" class="tgl btn" href="#" tt="' + L.fr_case + '</a>',
+			'<a id="rn_win" class="tgl btn" href="#" tt="' + L.fr_win + '</a>',
+			'<a id="rn_slash" class="tgl btn" href="#" tt="' + L.fr_slash + '</a>',
 			'</div>',
 			'<div id="rn_vadv"><table>',
 			'<tr><td>regex</td><td><input type="text" id="rn_re" ' + NOAC + ' tt="regex search pattern to apply to original filenames; capturing groups can be referenced in the format field below like &lt;code&gt;(1)&lt;/code&gt; and &lt;code&gt;(2)&lt;/code&gt; and so on" placeholder="^[0-9]+[\\. ]+(.*) - (.*)" /></td></tr>',
@@ -3656,7 +3709,8 @@ var fileman = (function () {
 				(function (a) {
 					f[a].inew.onkeydown = function (e) {
 						rn_ok(a, true);
-						if (e.key.endsWith('Enter'))
+						var kc = (e.code || e.key) + '';
+						if (kc.endsWith('Enter'))
 							return rn_apply();
 					};
 					QS('.rn_dec' + k).onclick = function (e) {
@@ -3677,6 +3731,8 @@ var fileman = (function () {
 		}
 		bcfg_bind(r, 'adv', 'rn_adv', false, sadv);
 		bcfg_bind(r, 'cs', 'rn_case', false);
+		bcfg_bind(r, 'win', 'rn_win', true);
+		bcfg_bind(r, 'slash', 'rn_slash', true);
 		sadv();
 
 		function rn_ok(n, ok) {
@@ -3748,10 +3804,12 @@ var fileman = (function () {
 		spresets();
 
 		ire.onkeydown = ifmt.onkeydown = function (e) {
-			if (e.key == 'Escape')
+			var k = (e.code || e.key) + '';
+
+			if (k == 'Escape' || k == 'Esc')
 				return rn_cancel();
 
-			if (e.key.endsWith('Enter'))
+			if (k.endsWith('Enter'))
 				return rn_apply();
 		};
 
@@ -3792,6 +3850,24 @@ var fileman = (function () {
 
 		function rn_apply(e) {
 			ev(e);
+			if (r.win || r.slash) {
+				var changed = 0;
+				for (var a = 0; a < f.length; a++) {
+					var ov = f[a].inew.value,
+						nv = namesan(ov, r.win, r.slash);
+
+					if (ov != nv) {
+						f[a].inew.value = nv;
+						changed++;
+					}
+				}
+				if (changed)
+					return modal.confirm(L.fr_nchg.format(changed), rn_apply_loop, null);
+			}
+			rn_apply_loop();
+		}
+
+		function rn_apply_loop() {
 			while (f.length && (!f[0].ok || f[0].ofn == f[0].inew.value))
 				f.shift();
 
@@ -3812,7 +3888,7 @@ var fileman = (function () {
 				}
 
 				f.shift().inew.value = '( OK )';
-				return rn_apply();
+				return rn_apply_loop();
 			}
 
 			var xhr = new XHR();
@@ -4703,7 +4779,7 @@ var thegrid = (function () {
 				}
 				ihref = SR + '/.cpr/ico/' + ext;
 			}
-			ihref += (ihref.indexOf('?') > 0 ? '&' : '?') + 'cache=i';
+			ihref += (ihref.indexOf('?') > 0 ? '&' : '?') + 'cache=i&_=' + ACB;
 
 			html.push('<a href="' + ohref + '" ref="' + ref +
 				'"' + ac + ' ttt="' + esc(name) + '"><img style="height:' +
@@ -4926,14 +5002,14 @@ document.onkeydown = function (e) {
 	if (QS('#bbox-overlay.visible') || modal.busy)
 		return;
 
-	var k = e.code + '', pos = -1, n,
+	var k = (e.code || e.key) + '', pos = -1, n,
 		ae = document.activeElement,
 		aet = ae && ae != document.body ? ae.nodeName.toLowerCase() : '';
 
 	if (e.key == '?')
 		return hkhelp();
 
-	if (k == 'Escape') {
+	if (k == 'Escape' || k == 'Esc') {
 		ae && ae.blur();
 		tt.hide();
 
@@ -5003,23 +5079,23 @@ document.onkeydown = function (e) {
 	if (k.endsWith('Enter') && ae && (ae.onclick || ae.hasAttribute('tabIndex')))
 		return ev(e) && ae.click() || true;
 
-	if (aet && aet != 'a' && aet != 'tr' && aet != 'pre')
+	if (aet && aet != 'a' && aet != 'tr' && aet != 'td' && aet != 'div' && aet != 'pre')
 		return;
 
 	if (ctrl(e)) {
-		if (k == 'KeyX')
+		if (k == 'KeyX' || k == 'x')
 			return fileman.cut();
 
-		if (k == 'KeyV')
+		if (k == 'KeyV' || k == 'v')
 			return fileman.paste();
 
-		if (k == 'KeyK')
+		if (k == 'KeyK' || k == 'k')
 			return fileman.delete();
 
 		return;
 	}
 
-	if (e.shiftKey && k != 'KeyA' && k != 'KeyD')
+	if (e.shiftKey && k != 'KeyA' && k != 'KeyD' && k != 'A' && k != 'D')
 		return;
 
 	if (k.indexOf('Digit') === 0)
@@ -5028,16 +5104,17 @@ document.onkeydown = function (e) {
 	if (pos !== -1)
 		return seek_au_mul(pos) || true;
 
-	if (k == 'KeyJ')
+	if (k == 'KeyJ' || k == 'j')
 		return prev_song() || true;
 
-	if (k == 'KeyL')
+	if (k == 'KeyL' || k == 'l')
 		return next_song() || true;
 
-	if (k == 'KeyP')
+	if (k == 'KeyP' || k == 'p')
 		return playpause() || true;
 
-	n = k == 'KeyU' ? -10 : k == 'KeyO' ? 10 : 0;
+	n = (k == 'KeyU' || k == 'u') ? -10 :
+		(k == 'KeyO' || k == 'o') ? 10 : 0;
 	if (n !== 0)
 		return seek_au_rel(n) || true;
 
@@ -5046,51 +5123,52 @@ document.onkeydown = function (e) {
 			showfile.active() ? ebi('dldoc').click() :
 				dl_song();
 
-	n = k == 'KeyI' ? -1 : k == 'KeyK' ? 1 : 0;
+	n = (k == 'KeyI' || k == 'i') ? -1 :
+		(k == 'KeyK' || k == 'k') ? 1 : 0;
 	if (n !== 0)
 		return tree_neigh(n);
 
-	if (k == 'KeyM')
+	if (k == 'KeyM' || k == 'm')
 		return tree_up();
 
-	if (k == 'KeyB')
+	if (k == 'KeyB' || k == 'b')
 		return treectl.hidden ? treectl.entree() : treectl.detree();
 
-	if (k == 'KeyG')
+	if (k == 'KeyG' || k == 'g')
 		return ebi('griden').click();
 
-	if (k == 'KeyT')
+	if (k == 'KeyT' || k == 't')
 		return ebi('thumbs').click();
 
-	if (k == 'KeyV')
+	if (k == 'KeyV' || k == 'v')
 		return ebi('filetree').click();
 
 	if (k == 'F2')
 		return fileman.rename();
 
 	if (!treectl.hidden && (!e.shiftKey || !thegrid.en)) {
-		if (k == 'KeyA')
+		if (k == 'KeyA' || k == 'a')
 			return QS('#twig').click();
 
-		if (k == 'KeyD')
+		if (k == 'KeyD' || k == 'd')
 			return QS('#twobytwo').click();
 	}
 
 	if (showfile.active()) {
-		if (k == 'KeyS')
+		if (k == 'KeyS' || k == 's')
 			showfile.tglsel();
-		if (k == 'KeyE' && ebi('editdoc').style.display != 'none')
+		if ((k == 'KeyE' || k == 'e') && ebi('editdoc').style.display != 'none')
 			ebi('editdoc').click();
 	}
 
 	if (thegrid.en) {
-		if (k == 'KeyS')
+		if (k == 'KeyS' || k == 's')
 			return ebi('gridsel').click();
 
-		if (k == 'KeyA')
+		if (k == 'KeyA' || k == 'a')
 			return QSA('#ghead a[z]')[0].click();
 
-		if (k == 'KeyD')
+		if (k == 'KeyD' || k == 'd')
 			return QSA('#ghead a[z]')[1].click();
 	}
 };
@@ -5179,9 +5257,11 @@ document.onkeydown = function (e) {
 
 	function ev_search_input() {
 		var v = unsmart(this.value),
-			id = this.getAttribute('id');
+			id = this.getAttribute('id'),
+			is_txt = id.slice(-1) == 'v',
+			is_chk = id.slice(-1) == 'c';
 
-		if (id.slice(-1) == 'v') {
+		if (is_txt) {
 			var chk = ebi(id.slice(0, -1) + 'c');
 			chk.checked = ((v + '').length > 0);
 		}
@@ -5193,12 +5273,15 @@ document.onkeydown = function (e) {
 		cap = 125;
 
 		clearTimeout(defer_timeout);
+		if (is_chk)
+			return do_search();
+
 		defer_timeout = setTimeout(try_search, 2000);
 		try_search(v);
 	}
 
 	function ev_search_keydown(e) {
-		if (e.key.endsWith('Enter'))
+		if ((e.key + '').endsWith('Enter'))
 			do_search();
 	}
 
@@ -5872,6 +5955,9 @@ var treectl = (function () {
 	}
 
 	function bad_proxy(e) {
+		if (ctrl(e))
+			return true;
+
 		ev(e);
 		var dst = this.getAttribute('dst'),
 			k = dst ? 'dst' : 'href',
@@ -5904,6 +5990,9 @@ var treectl = (function () {
 	}
 
 	r.reqls = function (url, hpush, back) {
+		if (IE && !history.pushState)
+			return window.location = url;
+
 		var xhr = new XHR();
 		xhr.top = url.split('?')[0];
 		xhr.back = back
@@ -7349,8 +7438,11 @@ function show_md(md, name, div, url, depth) {
 
 	wfp_debounce.hide();
 	if (!marked) {
-		if (depth)
+		if (depth) {
+			clmod(div, 'raw', 1);
+			div.textContent = "--[ " + name + " ]---------\r\n" + md;
 			return toast.warn(10, errmsg + (window.WebAssembly ? 'failed to load marked.js' : 'your browser is too old'));
+		}
 
 		wfp_debounce.n--;
 		return import_js(SR + '/.cpr/deps/marked.js', function () {

copyparty/web/md.html

@@ -139,16 +139,15 @@ var md_opt = {
 };
 
 (function () {
-	var l = localStorage,
-		drk = l.light != 1,
+	var l = window.localStorage,
+		drk = (l && l.light) != 1,
 		btn = document.getElementById("lightswitch"),
 		f = function (e) {
 if (e) { e.preventDefault(); drk = !drk; }
 document.documentElement.className = drk? "z":"y";
 btn.innerHTML = "go " + (drk ? "light":"dark");
-l.light = drk? 0:1;
+try { l.light = drk? 0:1; } catch (ex) { }
 		};
-	
 	btn.onclick = f;
 	f();
 })();

copyparty/web/md.js

@@ -216,6 +216,11 @@ function convert_markdown(md_text, dest_dom) {
             md_html = DOMPurify.sanitize(md_html);
     }
     catch (ex) {
+        if (IE) {
+            dest_dom.innerHTML = 'IE cannot into markdown ;_;';
+            return;
+        }
+
         if (ext)
             md_plug_err(ex, ext[1]);
 

copyparty/web/md2.js

@@ -163,7 +163,7 @@ redraw = (function () {
     dom_sbs.onclick = setsbs;
     dom_nsbs.onclick = modetoggle;
 
-    onresize();
+    (IE ? modetoggle : onresize)();
     return onresize;
 })();
 
@@ -933,7 +933,7 @@ var set_lno = (function () {
     var keydown = function (ev) {
         if (!ev && window.event) {
             ev = window.event;
-            if (localStorage.dev_fbw == 1) {
+            if (dev_fbw == 1) {
                 toast.warn(10, 'hello from fallback code ;_;\ncheck console trace');
                 console.error('using window.event');
             }
@@ -1009,7 +1009,7 @@ var set_lno = (function () {
                 md_home(ev.shiftKey);
                 return false;
             }
-            if (!ev.shiftKey && (ev.code.endsWith("Enter") || kc == 13)) {
+            if (!ev.shiftKey && ((ev.code + '').endsWith("Enter") || kc == 13)) {
                 return md_newline();
             }
             if (!ev.shiftKey && kc == 8) {

copyparty/web/mde.html

@@ -37,12 +37,12 @@ var md_opt = {
 };
 
 var lightswitch = (function () {
-	var l = localStorage,
-		drk = l.light != 1,
+	var l = window.localStorage,
+		drk = (l && l.light) != 1,
 		f = function (e) {
 if (e) drk = !drk;
 document.documentElement.className = drk? "z":"y";
-l.light = drk? 0:1;
+try { l.light = drk? 0:1; } catch (ex) { }
 		};
 	f();
 	return f;

copyparty/web/splash.html

@@ -110,7 +110,8 @@ var SR = {{ r|tojson }},
 	lang="{{ lang }}",
 	dfavico="{{ favico }}";
 
-document.documentElement.className=localStorage.cpp_thm||"{{ this.args.theme }}";
+var STG = window.localStorage;
+document.documentElement.className = (STG && STG.cpp_thm) || "{{ this.args.theme }}";
 
 </script>
 <script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>

copyparty/web/svcs.html

@@ -238,7 +238,8 @@ var SR = {{ r|tojson }},
     lang="{{ lang }}",
 	dfavico="{{ favico }}";
 
-document.documentElement.className=localStorage.cpp_thm||"{{ args.theme }}";
+var STG = window.localStorage;
+document.documentElement.className = (STG && STG.cpp_thm) || "{{ args.theme }}";
 
 </script>
 <script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>

copyparty/web/ui.css

@@ -105,6 +105,9 @@ html {
 #toast pre {
 	margin: 0;
 }
+#toast.hide {
+	display: none;
+}
 #toast.vis {
 	right: 1.3em;
 	transform: inherit;

copyparty/web/up2k.js

@@ -431,7 +431,7 @@ function U2pvis(act, btns, uc, st) {
         if (sread('potato') === null) {
             btn.click();
             toast.inf(30, L.u_gotpot);
-            localStorage.removeItem('potato');
+            sdrop('potato');
         }
 
         u2f.appendChild(ode);
@@ -852,7 +852,7 @@ function up2k_init(subtle) {
 
     setmsg(suggest_up2k, 'msg');
 
-    var parallel_uploads = icfg_get('nthread'),
+    var parallel_uploads = ebi('nthread').value = icfg_get('nthread', u2j),
         uc = {},
         fdom_ctr = 0,
         biggest_file = 0;
@@ -861,6 +861,7 @@ function up2k_init(subtle) {
     bcfg_bind(uc, 'multitask', 'multitask', true, null, false);
     bcfg_bind(uc, 'potato', 'potato', false, set_potato, false);
     bcfg_bind(uc, 'ask_up', 'ask_up', true, null, false);
+    bcfg_bind(uc, 'umod', 'umod', false, null, false);
     bcfg_bind(uc, 'u2ts', 'u2ts', !u2ts.endsWith('u'), set_u2ts, false);
     bcfg_bind(uc, 'fsearch', 'fsearch', false, set_fsearch, false);
 
@@ -1332,7 +1333,8 @@ function up2k_init(subtle) {
             return modal.confirm(msg.join('') + '</ul>', function () {
                 start_actx();
                 up_them(good_files);
-                toast.inf(15, L.u_unpt, L.u_unpt);
+                if (have_up2k_idx)
+                    toast.inf(15, L.u_unpt, L.u_unpt);
             }, null);
 
         up_them(good_files);
@@ -1391,6 +1393,8 @@ function up2k_init(subtle) {
                 entry.rand = true;
                 entry.name = 'a\n' + entry.name;
             }
+            else if (uc.umod)
+                entry.umod = true;
 
             if (biggest_file < entry.size)
                 biggest_file = entry.size;
@@ -2467,6 +2471,8 @@ function up2k_init(subtle) {
             req.srch = 1;
         else if (t.rand)
             req.rand = true;
+        else if (t.umod)
+            req.umod = true;
 
         xhr.open('POST', t.purl, true);
         xhr.responseType = 'text';
@@ -2685,7 +2691,11 @@ function up2k_init(subtle) {
             }
 
             parallel_uploads = v;
-            swrite('nthread', v);
+            if (v == u2j)
+                sdrop('nthread');
+            else
+                swrite('nthread', v);
+
             clmod(obj, 'err');
             return;
         }
@@ -2699,7 +2709,7 @@ function up2k_init(subtle) {
             parallel_uploads = 16;
 
         obj.value = parallel_uploads;
-        bumpthread({ "target": 1 })
+        bumpthread({ "target": 1 });
     }
 
     function tgl_fsearch() {

copyparty/web/util.js

@@ -12,9 +12,11 @@ if (window.CGV)
 
 
 var wah = '',
+    STG = null,
     NOAC = 'autocorrect="off" autocapitalize="off"',
     L, tt, treectl, thegrid, up2k, asmCrypto, hashwasm, vbar, marked,
-    CB = '?_=' + Date.now(),
+    T0 = Date.now(),
+    CB = '?_=' + Math.floor(T0 / 1000).toString(36),
     R = SR.slice(1),
     RS = R ? "/" + R : "",
     HALFMAX = 8192 * 8192 * 8192 * 8192,
@@ -39,6 +41,16 @@ if (!window.Notification || !Notification.permission)
 if (!window.FormData)
     window.FormData = false;
 
+try {
+    STG = window.localStorage;
+    STG.STG;
+}
+catch (ex) {
+    STG = null;
+    if ((ex + '').indexOf('sandbox') < 0)
+        console.log('no localStorage: ' + ex);
+}
+
 try {
     CB = '?' + document.currentScript.src.split('?').pop();
 
@@ -160,6 +172,9 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
     if (url.indexOf(' > eval') + 1 && !evalex_fatal)
         return;  // md timer
 
+    if (IE && url.indexOf('prism.js') + 1)
+        return;
+
     var ekey = url + '\n' + lineNo + '\n' + msg;
     if (ignexd[ekey] || crashed)
         return;
@@ -202,19 +217,24 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
         }
         ignexd[ekey] = true;
 
-        var ls = jcp(localStorage);
-        if (ls.fman_clip)
-            ls.fman_clip = ls.fman_clip.length + ' items';
+        var ls = {},
+            lsk = Object.keys(localStorage),
+            nka = lsk.length,
+            nk = Math.min(200, nka);
 
-        var lsk = Object.keys(ls);
-        lsk.sort();
-        html.push('<p class="b">');
-        for (var a = 0; a < lsk.length; a++) {
-            if (ls[lsk[a]].length > 9000)
-                continue;
+        for (var a = 0; a < nk; a++) {
+            var k = lsk[a],
+                v = localStorage.getItem(k);
 
-            html.push(' <b>' + esc(lsk[a]) + '</b> <code>' + esc(ls[lsk[a]]) + '</code> ');
+            ls[k] = v.length > 256 ? v.slice(0, 32) + '[...' + v.length + 'b]' : v;
         }
+
+        lsk = Object.keys(ls);
+        lsk.sort();
+        html.push('<p class="b"><b>' + nka + ':&nbsp;</b>');
+        for (var a = 0; a < nk; a++)
+            html.push(' <b>' + esc(lsk[a]) + '</b> <code>' + esc(ls[lsk[a]]) + '</code> ');
+
         html.push('</p>');
     }
     catch (e) { }
@@ -276,10 +296,11 @@ function anymod(e, shift_ok) {
 }
 
 
+var dev_fbw = sread('dev_fbw');
 function ev(e) {
     if (!e && window.event) {
         e = window.event;
-        if (localStorage.dev_fbw == 1) {
+        if (dev_fbw == 1) {
             toast.warn(10, 'hello from fallback code ;_;\ncheck console trace');
             console.error('using window.event');
         }
@@ -370,6 +391,22 @@ catch (ex) {
     }
 }
 
+if (!window.Set)
+    window.Set = function () {
+        var r = this;
+        r.size = 0;
+        r.d = {};
+        r.add = function (k) {
+            if (!r.d[k]) {
+                r.d[k] = 1;
+                r.size++;
+            }
+        };
+        r.has = function (k) {
+            return r.d[k];
+        };
+    };
+
 // https://stackoverflow.com/a/950146
 function import_js(url, cb, ecb) {
     var head = document.head || document.getElementsByTagName('head')[0];
@@ -395,6 +432,25 @@ function unsmart(txt) {
 }
 
 
+function namesan(txt, win, fslash) {
+    if (win)
+        txt = (txt.
+            replace(/</g, "＜").
+            replace(/>/g, "＞").
+            replace(/:/g, "：").
+            replace(/"/g, "＂").
+            replace(/\\/g, "＼").
+            replace(/\|/g, "｜").
+            replace(/\?/g, "？").
+            replace(/\*/g, "＊"));
+
+    if (fslash)
+        txt = txt.replace(/\//g, "／");
+
+    return txt;
+}
+
+
 var crctab = (function () {
     var c, tab = [];
     for (var n = 0; n < 256; n++) {
@@ -881,9 +937,16 @@ function jcp(obj) {
 }
 
 
+function sdrop(key) {
+    try {
+        STG.removeItem(key);
+    }
+    catch (ex) { }
+}
+
 function sread(key, al) {
     try {
-        var ret = localStorage.getItem(key);
+        var ret = STG.getItem(key);
         return (!al || has(al, ret)) ? ret : null;
     }
     catch (e) {
@@ -894,9 +957,9 @@ function sread(key, al) {
 function swrite(key, val) {
     try {
         if (val === undefined || val === null)
-            localStorage.removeItem(key);
+            STG.removeItem(key);
         else
-            localStorage.setItem(key, val);
+            STG.setItem(key, val);
     }
     catch (e) { }
 }
@@ -931,7 +994,7 @@ function fcfg_get(name, defval) {
         val = parseFloat(sread(name));
 
     if (!isNum(val))
-        return parseFloat(o ? o.value : defval);
+        return parseFloat(o && o.value !== '' ? o.value : defval);
 
     if (o)
         o.value = val;
@@ -1057,7 +1120,7 @@ function dl_file(url) {
 
 function cliptxt(txt, ok) {
     var fb = function () {
-        console.log('fb');
+        console.log('clip-fb');
         var o = mknod('input');
         o.value = txt;
         document.body.appendChild(o);
@@ -1398,6 +1461,10 @@ var toast = (function () {
         clmod(obj, 'vis');
         r.visible = false;
         r.tag = obj;
+        if (!window.WebAssembly)
+            te = setTimeout(function () {
+                obj.className = 'hide';
+            }, 500);
     };
 
     r.show = function (cl, sec, txt, tag) {
@@ -1558,7 +1625,7 @@ var modal = (function () {
     };
 
     var onkey = function (e) {
-        var k = e.code,
+        var k = (e.code || e.key) + '',
             eok = ebi('modal-ok'),
             eng = ebi('modal-ng'),
             ae = document.activeElement;
@@ -1573,10 +1640,10 @@ var modal = (function () {
             return ok(e);
         }
 
-        if ((k == 'ArrowLeft' || k == 'ArrowRight') && eng && (ae == eok || ae == eng))
+        if ((k == 'ArrowLeft' || k == 'ArrowRight' || k == 'Left' || k == 'Right') && eng && (ae == eok || ae == eng))
             return (ae == eok ? eng : eok).focus() || ev(e);
 
-        if (k == 'Escape')
+        if (k == 'Escape' || k == 'Esc')
             return ng(e);
     }
 
@@ -1854,21 +1921,17 @@ var favico = (function () {
         var b64;
         try {
             b64 = btoa(svg ? svg_decl + svg : gx(r.txt));
-            //console.log('f1');
         }
         catch (e1) {
             try {
                 b64 = btoa(gx(encodeURIComponent(r.txt).replace(/%([0-9A-F]{2})/g,
                     function x(m, v) { return String.fromCharCode('0x' + v); })));
-                //console.log('f2');
             }
             catch (e2) {
                 try {
                     b64 = btoa(gx(unescape(encodeURIComponent(r.txt))));
-                    //console.log('f3');
                 }
                 catch (e3) {
-                    //console.log('fe');
                     return;
                 }
             }

docs/changelog.md

@@ -1,3 +1,148 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-0114-0629  `v1.9.29`  RAM friendly
+
+## new features
+
+* try to keep track of RAM usage in the thumbnailer 95a59996
+  * very inaccurate, just wild guessing really, but probably good enough:
+  * an attempt to stop FFmpeg from eating all the RAM when generating spectrograms
+  * `--th-ram-max` specifies how much RAM it's allowed to use (default 6 GB), crank it up if thumbnailing is too slow now
+* much faster startup on devices with slow filesystems and lots of files in the volume root (especially android phones) f1358dba
+* `uncache` button (in mediaplayer settings) a55e0d6e
+  * rotates all audio URLs, in case the browser has a cached copy of a broken mp3 or whatnot
+* now possible to POST files without having to set the `act: bput` multipart field 9bc09ce9
+  * mainly to support [igloo irc](https://github.com/9001/copyparty#client-examples) and other simplistic upload clients
+* try to point the linux oom-killer at FFmpeg so it doesn't kill innocent processes instead dc8e621d
+  * only works if copyparty has acces to /proc, so not in prisonparty, and maybe not in docker (todo)
+* UX:
+  * do another search immediately if a search-filter gets unchecked a4239a46
+  * several ie11 fixes (keyboard hotkeys and a working text editor) 2fd2c6b9
+
+## bugfixes
+
+* POSTing files could block for a really long time if the database is busy (filesystem reindexing), now it schedules the indexing for later instead e8a653ca
+* less confusing behavior when reindexing a file (keep uploader-ip/time if file contents turn out to be unmodified, and drop both otherwise) 226c7c30
+
+## other changes
+
+* better log messages when clients decide to disconnect in the middle of a POST 02430359
+* add a warning if copyparty is started with an account definition (`-a`) which isn't used in any volumes e01ba855
+* when running on macos, don't index apple metadata files (`.DS_Store` and such) d0eb014c
+  * they are still downloadable by anyone with read-access, and still appear in directory listings for users with access to see dotfiles
+* added a [log repacker](https://github.com/9001/copyparty/blob/hovudstraum/scripts/logpack.sh) to shrink/optimize old logs dee0950f
+* and a [contextlet](https://github.com/9001/copyparty/blob/hovudstraum/contrib/README.md#send-to-cppcontextletjson) example
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1231-1849  `v1.9.28`  eo2023
+
+was hoping to finish the IdP stuff during 37c3 but that fell through, so here's all the other recent fixes instead -- happy newyears
+
+## new features
+
+* #66 new permission `.` to grant access to see dotfiles (hidden files) to specific users
+  * and new volflag `dots` to grant access to all users with `r`ead
+  * `-ed` still behaves like before (anyone with `r` can see dotfiles in all volumes)
+* #70 new permission `A` (alias of `rwmda.`) grants read/write/move/delete/admin/dotfiles
+* #67 folder thumbnails can be dotfiles (`.cover.jpg`, `.folder.png`) if the database is enabled (`-e2dsa`)
+* new option `--u2j` to specify default number of parallel file uploads in the up2k browser client
+  * default (2) is good on average; 16 can be good when most uploaders are overseas
+* curl gets plaintext 404/403 messages
+
+## bugfixes
+
+* cors-checking is disabled if the `PW` header is provided, just like the [readme](https://github.com/9001/copyparty#cors) always claimed
+* server would return `200 OK` while trying to return a file that is unreadable due to filesystem permissions
+* `--xdev` still doesn't work on windows, but at least now it doesn't entirely break filesystem indexing
+* fix tiny resource leak due to funky dualstack on macos
+
+## other changes
+
+* logfiles are padded to align messages when `-q` is specified, similar to current/previous behavior without `-q`
+* `--hdr-au-usr` was renamed to `--idp-h-usr` in preparation for other `--idp` things
+  * any mentions of `--hdr-au-usr` are translated to the new name on startup
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1208-2133  `v1.9.27`  another dedup bug
+
+so [v1.9.26](https://github.com/9001/copyparty/releases/tag/v1.9.26) fixed how moving a symlink could break other related symlinks, and then it turns out symlinks themselves could also die when moving them to another location, and somehow nobody encountered any of these until now... surely there are no more deduplication-related issues left at this point, yeah?
+
+## bugfixes
+
+* #65 moving deduplicated copies of files (symlinks) from one location to another could make them disappear (break the symlinks)
+
+  * don't worry, we are **not** talking about data loss! but see the [release notes for v1.9.26](https://github.com/9001/copyparty/releases/tag/v1.9.26) which explain how to deal with this issue (how to find, diagnose, and repair broken symlinks)
+
+----
+
+## regarding fedora packages
+
+[copr-pypi](https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/) (fedora's build service) is not building at the moment; ***if you installed copyparty from copr-pypi,*** you can upgrade to this release by running one of the following:
+
+```bash
+dnf install https://ocv.me/copyparty/fedora/37/python3-copyparty.fc37.noarch.rpm
+dnf install https://ocv.me/copyparty/fedora/38/python3-copyparty.fc38.noarch.rpm
+dnf install https://ocv.me/copyparty/fedora/39/python3-copyparty.fc39.noarch.rpm
+```
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1208-0136  `v1.9.26`  dont break symlinks
+
+## new features
+* *tumbleweed*
+
+## bugfixes
+
+* deleting files from the server could make some duplicates of that file unavailable (by breaking nested symlinks)
+
+  * don't worry, we are **not** talking about data loss! but such broken links would disappear from the directory listing and would need to be remedied by replacing the broken links manually, either by using a file explorer or commandline
+
+  * **only** affected linux/macos, did **not** affect servers with `--hardlink` or `--never-symlink` or `--no-dedup`, and **mainly** affected servers with lots of duplicate files (with some dupes in the same folder and some elsewhere)
+
+  * if you want to check for such broken symlinks, the following unix command will find all of them: `find -L -type l`
+
+  * to repair a broken link, first remove it and then replace it: `rm thelink.opus; ln -s /mnt/music/realfile.opus thelink.opus`
+
+  * if you are left with a mystery file and want to know where its duplicates are, you can grep for the filename in the logs and you'll find something like the following line, where the `wark` is the file identifier; grep for that to find all the other copies of that file -- `purl` is the folder/URL which that copy of the file was uploaded to:
+    ```json
+    {"name": "04. GHOST.opus", "purl": "/mu/vt/suisei/still-still-stellar/", "size": 4520986, "lmod": 1697091772, "sprs": true, "hash": [], "wark": "SJMASMtWOa0UZnc002nn5unO5iCBMa-krt2CDcq8eJe9"}
+    ```
+
+* the server would throw an error if you tried to delete a broken symlink
+* prevent warnings about duplicate file entries in the database by preventing that from happening in the first place
+* `u2c.py` (commandline uploader) would fail to delete files from the server if there's more than ~10'000 files to be deleted
+  * and forgot to bump the version number... `1.11 (2nd season)`
+
+## other changes
+* `--help` was slightly improved
+* docker images are now based on alpine v3.19
+* `copyparty.exe` is now based on python v3.11.7
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1201-2326  `v1.9.25`  focus
+
+## new features
+* remember and restore scroll position when leaving the textfile viewer
+
+## bugfixes
+* the request-smuggling detetcor was too strict, blocking access to textfiles with newlines / control-codes in the filename
+* focus and text selection in messageboxes was still jank, mainly in firefox and especially phones
+
+## other changes
+* the banhammer now applies on attempts at request-smuggling and path traversals
+  * these were merely detected and rejected before, might as well bonk them
+* reject bad requests with a terse 500 instead of abruptly disconnecting in some cases
+  * stops firefox from rapidly spamming additional attempts
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2023-1201-0210  `v1.9.24`  header auth
 
@@ -4391,3 +4536,832 @@ nothing really important happened since [v0.11.6](https://github.com/9001/copypa
 
 
 
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0601-0625  `v0.11.6`  vtec
+
+### things to know when upgrading:
+* see release-notes for [v0.11.0](https://github.com/9001/copyparty/releases/tag/v0.11.0) and [v0.11.1](https://github.com/9001/copyparty/releases/tag/v0.11.1) as they introduced new features you may wish to disable
+
+### new features:
+* searching for audio tags is now literally 1000x faster
+  (almost as fast as the version numbers recently)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0601-0155  `v0.11.4`  please upgrade
+
+## important news:
+* this release fixes a missing permission check which could allow users to download write-only folders
+  * this bug was introduced 19 days ago, in `v0.10.17`
+  * the requirement to be affected is write-only folders mounted within readable folders
+  * and the worst part is there was a unit-test exactly for this, https://github.com/9001/copyparty/commit/273ca0c8da0d94f9d06ca16bd86c0301d9d06455 way overdue
+* also fixes minor bugs introduced in `v0.11.1`
+* this version is the same as `v0.11.5` on pypi
+
+----
+
+### things to know when upgrading:
+* see [v0.11.0](https://github.com/9001/copyparty/releases/tag/v0.11.0) and [v0.11.1](https://github.com/9001/copyparty/releases/tag/v0.11.1) as they introduce new features you may wish to disable
+  * especially the `dbtool` part if your database is huge
+
+### new features:
+* filesearch now powered by a boolean query syntax
+  * the regular search interface generates example queries
+  * `size >= 2048 and ( name like *.mp4 or name like *.mkv )`
+
+### bugfixes:
+* scan files on upload (broke in 0.11.1)
+* restore the loud "folder does not exist" warning (another 0.11.1)
+* fix thumbnails in search results (never worked)
+
+#### really minor stuff:
+* increased default thumbnail clean interval from 30min to 12h
+* admin panel also links to the volumes
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0529-2139  `v0.11.1`  do it live
+
+no important bugfixes, just new features
+
+### things to know when upgrading:
+* `--no-rescan` disables `?scan`, a new feature which lets users initiate a recursive scan for new files to hash and read tags from
+  * this is enabled per-volume for users with read+write access
+* `--no-stack` disables `?stack`, a new feature which shows a dump of all the stacks
+  * this is enabled if a user has read+write on at least one folder
+* if you wish to wipe the DB and rebuild it to get the new metadata collected as of v0.11.0, and you have expensive `-mtp` parsers (bpm/key) and a huge database (or a slow server), consider https://github.com/9001/copyparty/tree/master/bin#dbtoolpy
+
+### new features:
+* **live rescan!** no more rebooting if you add/move files outside of copyparty and want to update the database, just hit the rescan button in the new...
+* **admin panel!** access `/?h` (the old control-panel link) to see it
+* **fast startup!** added 40TB of music? no need to wait for the initial scan, it runs in the background now
+  * when this turns out to be buggy you can `--no-fastboot`
+  * uploading is not possible until the initial file hashing has finished and it has started doing tags
+    * you can follow the progress in the new admin panel
+
+### bugfixes:
+* windows: avoid drifting into subvolumes and doublehashing files
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0529-1303  `v0.11.0`  welcome to the grid
+
+no important bugfixes, just new features
+
+### things to know when upgrading:
+* `Pillow` and `FFmpeg` is now used to generate thumbnails
+  * `--no-thumb` disables both
+  * `--no-vthumb` disables just `FFmpeg`
+* new optional dependencies:
+  * `Pillow` to enable thumbnails
+    * `pyheif-pillow-opener` to enable reading HEIF images
+    * `pillow-avif-plugin` to enable reading AVIF images
+    * `ffmpeg` and `ffprobe` to enable video thumbnails
+* if you wish to wipe the DB and rebuild it to get the new metadata collected as of this version, and you have expensive `-mtp` parsers (bpm/key) and a huge database (or a slow server), consider https://github.com/9001/copyparty/tree/master/bin#dbtoolpy
+
+### new features:
+* thumbnails! of both static images and video files
+  * served as webp or jpg depending on browser support
+  * new hotkeys: G, T, S, A/D
+* additional metadata collection with `-e2ts`
+  * audio/video codecs, video/image resolution, fps, ...
+  * if you wanna reindex, do a single run with `-e2tsr` to wipe the DB
+* mtp can collect multiple tags at once
+  * expects json like `{ "tag": "value" }`, see end of https://github.com/9001/copyparty/blob/master/bin/mtag/exe.py
+
+### bugfixes:
+* when sorting by name, show folders first
+* mimetypes for webp and opus on GET
+* mojibake support
+  * up2k into mb folder
+  * indexing files in mb folders
+  * editing markdown in mb folders
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0518-0210  `v0.10.22`  this is a no drift zone
+
+* browser: fix off-by-one which made the page slowly shrink back down when navigating away from a large folder
+* browser/mediaplayer: handle unsupported audio codecs better in some (older?) browsers
+* readme/requirements: firefox 34 and chrome 41 were the first browsers with native sha512 / full speed in up2k
+* and the feature nobody asked for:
+![2021-0518-041625-hexchat-fs8](https://user-images.githubusercontent.com/241032/118581146-67876700-b791-11eb-99c0-f1f5ace50797.png)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0516-1822  `v0.10.21`  fix tagger crash
+
+a
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0516-0551  `v0.10.20`  inspect
+
+nothing important this time, just new bling and some fixes to support old browsers  
+(well except for the basic-uploader summary autoclosing immediately on completion, that was kinda user-confusing)
+
+* add `ad`/`an` flags to `-mtp`; collect and display metadata from any file, not just audio-files
+* up2k speedboost on older iPhones (native hashing on safari 7 through 10)
+* add `--lf-url`, URL regex to exclude from log, defaults to `^/\.cpr/` (static files)
+* add `--ihead` to print specific request headers, `*` prints all
+* ux fixes
+  * include links to the uploaded files in bup summaries
+  * ...also make the bup summary not auto-close
+  * don't link to bup from up2k if read-only access
+  * toggle-switch for tooltips also affects the up2k ui
+  * stop flipping back to up2k on older browsers
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0513-2200  `v0.10.19`  imagine running servers on windows
+
+* fix: uploads when running copyparty on windows (broke in 0.10.18)
+* fix: bup uploads would not get PARTIAL-suffixed if the filename length hits filesystem-max and the client disconnects mid-upload 
+* add `--dotpart` which hides uploads as dotfiles until completed
+* very careful styling of the basic-browser
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0513-1542  `v0.10.18`  just 302 it my dude
+
+* stop trying to be smart, do full redirects instead
+* allow switching to basic-browser using cookie `b=u`
+* fix mode-toggling (upload/search) depending on folder permissions
+* persist/clear the password cookie with expiration
+* slight optimizations for rclone clients
+* other minor ui tweaks
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0512-2139  `v0.10.17`  denoise
+
+* allow navigating to write-only folders using the tree sidebar
+* show logues (prologue/epilogue) in write-only folders as well
+  * rename `.prologue.html` / `.epilogue.html` when uploaded so people can't embed javascript
+* support pyinstaller
+* hide more of the UI while in write-only folders
+  * hide [even more](https://a.ocv.me/pub/g/nerd-stuff/cpp/2021-0513-ui-mod.png) using [lovely hacks](https://github.com/9001/copyparty/blob/master/docs/minimal-up2k.html)
+* add a notice in bup that up2k is generally better
+
+alternative title: [Petit Up2k's - No Gui!](https://www.youtube.com/watch?v=IreeUoI6Kqc)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0502-0718  `v0.10.16`  somebody used -c
+
+* cfg-file: fix shorthand for assigning permissions to anonymous users
+* sfx: `-j` works on python3 (pickle did not enjoy the binary comments)
+* sfx: higher cooldown before it starts deleting tempfiles from old instances
+* sfx: should be a bit smaller (put compressed blobs at the end of the tar)
+* misc minor ui tweaks, mostly the bright-mode theme
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0424-0205  `v0.10.15`  write-only volumes are write-only
+
+good thing it was so obviously broken and/or that nobody ever tried to use it
+* regression test added to keep it fixed
+* can now make a hidden/inaccessible folder (optionally inside a public folder) like `-v /mnt/nas/music:/music:r -v /mnt/nas/music/inc:/music/inc:w`
+
+in other news, minor ui tweaks:
+* clickdrag in the media player sliders doesn't select text any more
+* a few lightmode adjustments
+* less cpu usage? should be
+
+`copyparty-sfx.py` (latest) made from c5db7c1a0c8f6ab23138ad7ea7642a6260e7da9b (v0.10.15-15) fixes `-j` (multiprocessing/high-performance)
+`copyparty-sfx-5a579db.py` (old) made from 5a579dba52e46c202b79c3d80c3b1c996c7b2e4a (v0.10.15-5) reduced the size
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0421-2004  `v0.10.14`  sparse4win
+
+# great stuff
+* firefox no longer leaking memory like crazy during large uploads
+    * not fixed intentionally (the firefox bug still exists i think)
+        * one of the v0.10.x changes are accidentally avoiding it w
+
+# good stuff
+* up2k-cli: conditional readahead based on filereader latency (firefox was not happy)
+* up2k-srv: make sparse files on windows if larger than `--sparse` MiB
+    * files will unsparse when upload completes if win10 or newer
+    * performance gain starts around 32 and up but default is 4 to save the SSDs
+* up2k-cli: fix high cpu usage after returning to idle
+* up2k-cli: ui tweaks
+* browser: give 404 instead of redirecting home when folder is 404 or 403
+* md-srv: stream documents rather than load into memory
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0420-2319  `v0.10.13`  moon
+
+600 MiB/s for both hashing and uploading on a ryzen 3700
+
+* up2k: hashing 2x faster than before
+    * except on android-chrome where it is now slightly slower because the android file api is a meme
+        * ...but android-firefox gained 4x and is now 3x faster than chrome, google pls
+
+this concludes the optimization arc
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0419-1958  `v0.10.12`  rocket
+
+up2k [way faster on large files](https://a.ocv.me/pub/g/nerd-stuff/cpp/2021-0419-up2k.webm) this time
+* js: removed a cpu bottleneck in the up2k client
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0419-1517  `v0.10.10`  blinded by the light
+
+* up2k: fix progress bars
+* up2k: more specific error messages (for example when trying to up a rangelocked file)
+* browser: link to timestamps in media files (media fragment urls)
+* fix crash when trying to -e2ts without the necessary dependencies available
+* since there wasn't enough pointless features that nobody will ever use already: added lightmode
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0416-2329  `v0.10.9`  fasten your seatbelts
+
+* up2k: [way faster](https://a.ocv.me/pub/g/nerd-stuff/cpp/2021-0416-up2k.webm) when uploading a large number of files
+  * 2x faster at 500 files, 3x faster at 1000, **8x at 3000**
+* up2k: show ETA and upload/hashing speeds in realtime
+* browser: hide search tab when database disabled
+* avoid crash on startup when mounting the root of a restricted smb share on windows, [cpython bug](https://bugs.python.org/issue43847)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0411-1926  `v0.10.8`  misc
+
+nothing massive, just a bunch of small things
+
+* browser: fix zip download on iphone/android
+* sfx: prevent StorageSense from deleting copyparty while it's running
+* browser: less tree jitter when scrolling
+* browser: only capture hotkeys without modifiers
+* up2k: add some missing presentational uridecodes
+* browser: add `?b` for an extremely minimal browser
+    * `?b=u` includes the uploader
+* browser: somewhat support `?pw=hunter2` in addition to the cppwd cookie
+* make-sfx: optional argument `gz` to build non-bz2 sfx
+* stop crashing argparse on pythons <= june 2018
+* support http/1.0
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0402-2235  `v0.10.7`  thx exci
+
+up2k-client fixes:
+* uploads getting stuck if more than 128 MiB was rejected as dupes
+* displayed links on rejected uploads
+* displayed upload speed was way off
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0402-0111  `v0.10.6`  enterprise ready
+
+minimal-effort support for really old browsers
+* internet explorer 6 can browse, upload files, mkdir
+* internet explorer 9 can also play mp3s, zip selected files
+* internet explorer 10 and newer has near-full support
+* the final version of chrome and firefox on xp have full support
+* netscape 4.5 works well enough, text is yellow on white
+* [netscape 4.0 segfaults](https://a.ocv.me/pub/g/nerd-stuff/cpp/2021-0402-netscape.png) (rip)
+
+on a more serious note,
+* fix multiselect zip diving into unselected subfolders
+* decode urlform messages to plaintext
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0330-2328  `v0.10.5`  search fix
+
+* fix audio playback in search results (broke in v0.9.9)
+* sort search results according to userdefined order
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0329-1853  `v0.10.4`  stablesort
+
+running out of things to fix so here are nitpicks
+* stable sort when sorting multiple columns
+* default to filenames with directories first (column 2 + 1)
+* remove some console spam
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0329-0247  `v0.10.3`  not slow as tar
+
+nothing too big this time,
+* tar 6x faster (does 1.8 GiB/s now)
+* fix selective archiving of subfolders
+* mute the loadbalancer when `-q`
+* don't show 0:00 as duration for non-audio files
+
+known inconvenience since 0.9.13 that won't ever be fixed:
+if you use the subfolder hiding thing (`-v :foo/bar:cd2d`) it creates intermediate volumes between the actual volume and the hidden subfolder which kinda messes with existing indexes (it will reindex stuff inside the intermediate volumes) but everything still works so it's just a pain
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0327-1703  `v0.10.2`  do i have to think of a name
+
+* select multiple files/folders to download as tar/zip
+* recover from read-errors when zipping things, adding a textfile in the zip explaining what went wrong
+* fix permissions in zip files for linux/macos unpacking
+* make the first browser column sortable
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0327-0144  `v0.10.1`  zip it
+
+* download folders as .zip or .tar files
+* upload entire folders by dropping them in
+* 4x faster response on the first request on each new connection
+
+forgot to explain the zip formats
+
+| name | url-suffix | description |
+|--|--|--|
+| `tar` | `?tar` | a plain gnutar, works great with `curl \| tar -xv` |
+| `zip` | `?zip=utf8` | works everywhere, glitchy filenames on win7 and older |
+| `zip_dos` | `?zip` | traditional cp437 (no unicode) to fix glitchy filenames |
+| `zip_crc` | `?zip=crc` | cp437 with crc32 computed early for truly ancient software |
+
+`zip_crc` will take longer to download since the server has to read each file twice, please let me know if you find a program old enough to actually need it btw, curious
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0323-0113  `v0.9.13`  micromanage
+
+you can skip this version unless your volume setup is crazy advanced
+
+* support hiding specific subfolders with `-v :/foo/bar:cd2d`
+* properly disable db/tags/etc when `cd2d` or `cd2t` volflags are set
+* volume info on startup is prettier
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0321-2105  `v0.9.10`  nurupo
+
+not so strong anymore
+
+* fixes a nullpointer when sorting a folder that contains markdown revisions
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0321-1615  `v0.9.9`  the strongest
+
+## big ones
+* add support for external analysis tools to provide arbitrary tags for the index
+* add example tools for detecting bpm and melodic key
+  * https://github.com/9001/copyparty/tree/master/bin/mtag
+* add range-search (duration/bpm/key/... between min/max values)
+* hotkeys for changing songs + skipping
+  * `0..9`=jump, `J/L`=file, `U/O`=10sec, `K/I`=folder, `P`=parent
+
+## the rest
+* add search timeouts and rate-control on both server/client-side
+* add time markers in the audio player
+* remember the file browser sort order
+  * the initial html retains server order, so use the tree to navigate
+* fix a race in the tag parser when using the multithreaded FFprobe backend
+* fix minor stuff related to volume flags and tag-display options
+* repacker should no longer break the bundled jinja2
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0315-0013  `v0.9.8`  the strongest for a while
+
+nothing more to add or fix for now (barely avoided adding bpm/tempo detection using keyfinder and vamp+qm since thats just too ridiculous)
+
+* browser: correct music playback order after sorting
+* browser: no more glitching on resize in non-tree-mode
+* fuse-client: read password from `some.txt` with `-a $some.txt`
+* sfx: reduce startup time by 20% or so (import rather than shell out)
+* sfx: support pypy, jython, and ironpy
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0308-0251  `v0.9.7`  the strongest hotfix 2nd season
+
+* actually fix it so it doesn't truncate in the first place
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0307-2044  `v0.9.6`  the strongest hotfix
+
+* don't crash the file browser on truncated table rows
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0307-1825  `v0.9.5`  the strongest potions
+
+* better support for mojibake filenames
+* separate scrollbar for the directory tree
+* stop persisting page data in the browser, reload on each navigation
+  * firefox disapproves of storing >= 4 MB of json in sessionStorage
+* normalization of musical keys collected from tags
+* recover from dying tag parsers
+* be nice to rhelics
+  * add support for the 2013 edition of sqlite3 in rhel 7
+  * and fix some py2 issues with `-e2d`, again thx to ^
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0305-0106  `v0.9.4`  the strongest orz
+
+markdown editor works
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0304-2300  `v0.9.3`  the strongest performance
+
+gotta go fast
+
+| | windows | linux | macos |
+|--|--|--|--|
+| file browser / directory listing | *15 times* faster | 2% slower sorry | 15% faster |
+| startup / `-e2ds` verification | 10% faster | even | 10% faster |
+| reading tags with ffprobe | 5 times faster | 4 times faster | 2 times faster |
+
+## new features
+* async scan incoming files for tags (from up2k, basic-upper, PUT)
+* resizable file browser tree
+
+## bugfixes
+* floor mtime so `-e2ds` doesn't keep rescanning
+* use localStorage for pushState data since firefox couldn't handle big folders
+* minor directory rescan semantics
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0303-0028  `v0.9.1`  the strongest bugs
+
+imagine downloading a .0
+* fix file search / search by contents
+* stop spamming responses with `{"tags":["x"]}`
+* recover from missing writable volumes during startup
+* redo search when filter-checkboxes are toggled
+* 1.5x faster client-side sorting
+* 1.02x faster server-side
+
+and i just realized i never added runtime tag scanning so copyparty will have to be restarted to see tags of new uploads, TODO for next ver
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0301-2312  `v0.9.0`  the strongest music server
+
+* grab tags from music files and make them searchable
+* and show the tags in the file browser
+* make all the browser columns minimizable
+* shrink the media player widget thing on big screens
+
+use `-e2dsa` and `-e2ts` to enable the media tag features globally, or enable/disable them per-volume (see readme)
+
+**NOTE:** older fuse clients (from before 5e3775c1afc9438f9930080a9b8542a063ba1765  / older than v0.8.0) must be upgraded for this copyparty release, however the new client still supports connecting to old servers
+
+other changes include
+* support chunked PUT requests from curl
+* fix a pypy memleak which broke sqlite3
+* fix directory tree sidebar breaking when nothing is mounted on `/`
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0222-2058  `v0.8.3`  discovery
+
+forgot to update the release name for 0.8 (which introduced searching and directory trees), good opportunity to name it after a dope album with some absolute bangers
+
+aside from the release name this version is entirely unrevolutionary
+
+* fixed debug prints on xp / win7 / win8 / early win10 versions
+* load prologues/epilogues when switching between folders
+* fix up2k modeswitching between read/write folders
+* additional minor ux tweaks
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0222-0254  `v0.8.1`  the ux update
+
+* search by name/path/size/date
+* search by file contents
+* directory tree sidebar thing
+  * navigate between folders while uploading
+
+NOTE: this will upgrade your `up2k.db` to `v2` but it will leave a backup of the old version in case you need to downgrade or whatever
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0214-0113  `v0.7.7`  trafikklys
+
+* new checkbox in up2k which coordinates uploading from multiple tabs
+  * if one tab is uploading, others will wait
+* fix up2k handshakes so uploads complete faster
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0212-1953  `v0.7.6`  nothing big
+
+* up2k: resume hashing when <= 128 MiB left to upload
+* stop showing `up2k.db/snap` in the file list
+* fix `--ciphers help`
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0212-0706  `v0.7.5`  you can https if you want to
+
+* fix https on python3 after breaking it in v0.6.3
+  * workaround for older versions: `--no-sendfile`
+  * don't use the native https anyways (pls reverse-proxy)
+* that said, added a bunch of ssl/tls/https options
+  * choice to only accept http or https
+  * specify ssl/tls versions and ciphers to allow
+  * log master-secrets to file
+  * print cipher overlap on connect
+* up2k indexer flushes to disk every minute
+* up2k indexer mentions the filepath on errors
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0204-0001  `v0.7.4`  a
+
+* sfx: save 43kB by replacing all docstrings with "a"
+* sfx: upgrade the bundled jinja2 and markupsafe
+    * zero dependencies on python3 as well now
+* do something useful with url-encoded POSTs
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0202-2357  `v0.7.3`  Hey! Listen!
+
+* bind multiple IP's / port ranges
+* dim the connection tracking messages a bit
+* stop gz/br unpacker from being too helpful
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0128-2352  `v0.7.2`  QUALITY
+
+* make up2k confirmations optional
+* let pending uploads stay for 6 hours
+* fix the 0.7.1 regression we won't talk about
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0123-1855  `v0.7.1`  checking it twice
+
+* up2k-client shows an OK/Cancel box before upload starts
+* up2k-client hashes the next pending file at most
+  * previously, all pending uploads were announced immediately
+* fix edgecase when the registry snapshot contained deleted files
+* delete all related files after 1h if an up2k upload was initiated but never started
+  * previously, the `.PARTIAL` (upload data) was kept, even when blank
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0110-1649  `v0.7.0`  keeping track
+
+## remember all uploads using `-e2d` to avoid duplicates
+* `-e2d` stores the up2k registry in a per-volume sqlite3 database at `$VOL/.hist/up2k.db`
+* unfinished uploads are indexed in `$VOL/.hist/up2k.snap` every 30 seconds
+* unfinished uploads which are idle for over 1 hour are forgotten
+* duplicate uploads will be symlinked to the new name (by default) or rejected
+
+## build an index of all existing files at startup using `-e2s`
+* ...so copyparty also knows about files from older versions / other sources
+* this detects deleted/renamed files and updates the database
+
+## reject duplicate uploads instead of symlinking
+* this is a per-volume config option, see the `cnodupe` example in `-h`
+* the uploader gets an error message with the path to the existing file
+
+## other changes
+* uploads temporarily have the extension `.PARTIAL` until the upload is completed
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2021-0107-0009  `v0.6.3`  no nagles beyond this point
+
+* reduce latency of final packet by ~0.2 sec
+* use sendfile(2) when possible (linux and macos)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1214-0328  `v0.6.2`  happy end of 2020
+
+* support uploads with massive filenames
+* list world-readable volumes when logged in
+* up2k-client: ignore rejected dupe uploads
+* sfx-repack: support wget
+* dodge python-bug #7980 
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1201-0158  `v0.6.0`  CHRISTMAAAAAS
+
+https://www.youtube.com/watch?v=rWc9XuqwoLI
+* md cleanup/fixes (thx eslint)
+* fix the sfx repacker
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1130-0201  `v0.5.7`  multiuser notepad
+
+not in the etherpad sense but rather
+* md: poll for changes every `-mcr` sec and warn if doc changed
+* md: prevent closing the tab on unsaved changes
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1129-1849  `v0.5.6`  the extra mile
+
+* use git tag/commit as version when creating sfx
+* md: table prettyprinter compacting properly
+* md/plug: add error handling to the plugins
+* md/plug: new feature to modify the final dom tree
+* md/plug: actually replace the plugin instances rather than keep adding new ones tehe
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1127-0225  `v0.5.5`  far beyond
+
+valvrave-stop.jpg
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1117-2258  `v0.5.4`  edovprim
+
+(get it? becasue reverse proxy haha)
+
+* reverse-proxy support
+* filetype column in the browser
+* md-edit: table formatter more chill
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-1113-0231  `v0.5.3`  improved
+
+* show per-connection and per-transfer speeds
+* restore macos support in sfx.sh
+* http correctness fixes
+  * SameSite=Lax
+  * support multiple cookies in parser
+  * `+` no longer decodes to ` `, goodbye netscape 3.04
+* fuse stuff
+  * python client: mojibake support on windows
+  * python client: https and password support
+  * support rclone as client (windows/linux)
+* new markdown-editor features
+  * table formatter
+  * mojibake/unicode hunter
+  * more predictable behavior
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0818-1822  `v0.5.2`  da setter vi punktum
+
+full disclaimer: `copyparty-sfx.py` was built using `sfx.py` from ~~82e568d4c9f25bfdfd1bf5166f0ebedf058723ee~~ f550a8171d298992f4ef569d2fc99a6037a44ea8
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0817-2155  `v0.5.1`  insert soho joke
+
+* add info-banner with hostname and disk-free
+* make older firefox versions cache less aggressively
+* expect less correctness from cots nas
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0816-2304  `v0.5.0`  fuse jelly
+
+* change default port from `1234` to `3923`
+* fuse 10x faster + add windows support
+* minimal CORS support added
+* PUT stuff from a browser-console or wherever
+* markdown editor improvements again
+  * paragraph-jump with ctrl-cursors
+  * fix firefox not showing the latest ver on F5
+* fix systemd killing the sfx binaries (ﾉ ﾟヮﾟ)ﾉ ~┻━┻
+* not actually related to the tegra exploit
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0517-1446  `v0.4.3`  🌲🪓🎉
+
+* print your documents! kill the trees!
+* drop support for opus/vorbis audio playback on iOS 11 *and older*
+* chrome's now twice as fast in the markdown editor
+  * firefox still wins
+* upgrade to marked.js v1.1.0
+* minor fuse + ux fixes
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0514-2302  `v0.4.2`  still not quite emacs (the editor is too good)
+
+* better editor cursor behavior
+* better editor autoindent
+* less broken fuse client
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0513-2308  `v0.4.1`  Further improvements to overall system stability and other minor adjustments have been made to enhance the user experience
+
+* better editor performance in massive documents
+* better undo/redo cursor positioning
+* better ux on safari
+* better ux on phones
+* better
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0512-2244  `v0.4.0`  NIH
+
+* new "basic" markdown editor
+  * textarea-based, way less buggy on phones
+  * better autoindent + undo/redo
+* smaller sfx (~170k)
+* osx fixes
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0506-2220  `v0.3.1`  v0.3.1
+
+* indicate version history for files in the browser
+* (also move old versions into .hist subfolders)
+* handle uploads with illegal filenames on windows
+* sortable file list
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0505-2302  `v0.3.0`  docuparty
+
+"why does a file server have a markdown editor"
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2020-0119-1512  `v0.2.3`  hello world
+
+

docs/devnotes.md

@@ -162,8 +162,8 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | PUT | | (binary data) | upload into file at URL |
 | PUT | `?gz` | (binary data) | compress with gzip and write into file at URL |
 | PUT | `?xz` | (binary data) | compress with xz and write into file at URL |
-| mPOST | | `act=bput`, `f=FILE` | upload `FILE` into the folder at URL |
-| mPOST | `?j` | `act=bput`, `f=FILE` | ...and reply with json |
+| mPOST | | `f=FILE` | upload `FILE` into the folder at URL |
+| mPOST | `?j` | `f=FILE` | ...and reply with json |
 | mPOST | | `act=mkdir`, `name=foo` | create directory `foo` at URL |
 | POST | `?delete` | | delete URL recursively |
 | jPOST | `?delete` | `["/foo","/bar"]` | delete `/foo` and `/bar` recursively |

docs/examples/docker/idp/copyparty.conf

@@ -0,0 +1,72 @@
+# not actually YAML but lets pretend:
+# -*- mode: yaml -*-
+# vim: ft=yaml:
+
+
+# example config for how copyparty can be used with an identity
+# provider, replacing the built-in authentication/authorization
+# mechanism, and instead expecting the reverse-proxy to provide
+# the requester's username (and possibly a group-name, for
+# optional group-based access control)
+#
+# the filesystem-path `/w` is used as the storage location
+# because that is the data-volume in the docker containers,
+# because a deployment like this (with an IdP) is more commonly
+# seen in containerized environments -- but this is not required
+
+
+[global]
+  e2dsa  # enable file indexing and filesystem scanning
+  e2ts   # enable multimedia indexing
+  ansi   # enable colors in log messages
+
+  # enable IdP support by expecting username/groupname in
+  # http-headers provided by the reverse-proxy; header "X-IdP-User"
+  # will contain the username, "X-IdP-Group" the groupname
+  idp-h-usr: x-idp-user
+  idp-h-grp: x-idp-group
+
+
+[/]      # create a volume at "/" (the webroot), which will
+  /w     # share /w (the docker data volume)
+  accs:
+    rw: *       # everyone gets read-access, but
+    rwmda: @su  # the group "su" gets read-write-move-delete-admin
+
+
+[/u/${u}]    # each user gets their own home-folder at /u/username
+  /w/u/${u}  # which will be "u/username" in the docker data volume
+  accs:
+    r: *              # read-access for anyone, and
+    rwmda: ${u}, @su  # read-write-move-delete-admin for that username + the "su" group
+
+
+[/u/${u}/priv]    # each user also gets a private area at /u/username/priv
+  /w/u/${u}/priv  # stored at DATAVOLUME/u/username/priv
+  accs:
+    rwmda: ${u}, @su  # read-write-move-delete-admin for that username + the "su" group
+
+
+[/lounge/${g}]    # each group gets their own shared volume
+  /w/lounge/${g}  # stored at DATAVOLUME/lounge/groupname
+  accs:
+    r: *               # read-access for anyone, and
+    rwmda: @${g}, @su  # read-write-move-delete-admin for that group + the "su" group
+
+
+[/lounge/${g}/priv]    # and a private area for each group too
+  /w/lounge/${g}/priv  # stored at DATAVOLUME/lounge/groupname/priv
+  accs:
+    rwmda: @${g}, @su  # read-write-move-delete-admin for that group + the "su" group
+
+
+# and create some strategic volumes to prevent anyone from gaining
+# unintended access to priv folders if the users/groups db is lost
+[/u]
+  /w/u
+  accs:
+    rwmda: @su
+[/lounge]
+  /w/lounge
+  accs:
+    rwmda: @su

scripts/bench/filehash.sh

@@ -12,6 +12,11 @@ set -euo pipefail
 #
 # can be adjusted with --hash-mt (but alpine caps out at 5)
 
+fsize=256
+nfiles=128
+pybin=$(command -v python3 || command -v python)
+#pybin=~/.pyenv/versions/nogil-3.9.10-2/bin/python3
+
 [ $# -ge 1 ] || {
 	echo 'need arg 1: path to copyparty-sfx.py'
 	echo ' (remaining args will be passed on to copyparty,'
@@ -22,6 +27,8 @@ sfx="$1"
 shift
 sfx="$(realpath "$sfx" || readlink -e "$sfx" || echo "$sfx")"
 awk=$(command -v gawk || command -v awk)
+uname -s | grep -E MSYS && win=1 || win=
+totalsize=$((fsize*nfiles))
 
 # try to use /dev/shm to avoid hitting filesystems at all,
 # otherwise fallback to mktemp which probably uses /tmp
@@ -30,20 +37,24 @@ mkdir $td || td=$(mktemp -d)
 trap "rm -rf $td" INT TERM EXIT
 cd $td
 
-echo creating 256 MiB testfile in $td
-head -c $((1024*1024*256)) /dev/urandom > 1
+echo creating $fsize MiB testfile in $td
+sz=$((1024*1024*fsize))
+head -c $sz /dev/zero | openssl enc -aes-256-ctr -iter 1 -pass pass:k -nosalt 2>/dev/null >1 || true
+wc -c 1 | awk '$1=='$sz'{r=1}END{exit 1-r}' || head -c $sz /dev/urandom >1
 
-echo creating 127 symlinks to it
-for n in $(seq 2 128); do ln -s 1 $n; done
+echo creating $((nfiles-1)) symlinks to it
+for n in $(seq 2 $nfiles); do MSYS=winsymlinks:nativestrict ln -s 1 $n; done
 
 echo warming up cache
 cat 1 >/dev/null
 
 echo ok lets go
-python3 "$sfx" -p39204 -e2dsa --dbd=yolo --exit=idx -lo=t -q "$@"
+$pybin "$sfx" -p39204 -e2dsa --dbd=yolo --exit=idx -lo=t -q "$@" && err= || err=$?
+[ $win ] && [ $err = 15 ] && err=  # sigterm doesn't hook on windows, ah whatever
+[ $err ] && echo ERROR $err && exit $err
 
 echo and the results are...
-$awk '/1 volumes in / {printf "%s MiB/s\n", 256*128/$(NF-1)}' <t
+LC_ALL=C $awk '/1 volumes in / {s=$(NF-1); printf "speed: %.1f MiB/s  (time=%.2fs)\n", '$totalsize'/s, s}' <t
 
 echo deleting $td and exiting
 
@@ -52,16 +63,30 @@ echo deleting $td and exiting
 
 # MiB/s @ cpu or device  (copyparty, pythonver, distro/os)  // comment
 
+#  3887 @ Ryzen 5 4500U  (cpp 1.9.5, nogil 3.9, fedora 39)  // --hash-mt=6; laptop
+#  3732 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.12.1, fedora 39)  // --hash-mt=6; laptop
 #  3608 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.5, fedora 38)  // --hash-mt=6; laptop
 #  2726 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.5, fedora 38)  // --hash-mt=4 (old-default)
 #  2202 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.5, docker-alpine 3.18.3) ??? alpine slow
 #  2719 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.2, docker-debian 12.1)
 
+#  7746 @ mbp 2023 m3pro (cpp 1.9.5, py 3.11.7, macos 14.1)  // --hash-mt=6
+#  6687 @ mbp 2023 m3pro (cpp 1.9.5, py 3.11.7, macos 14.1)  // --hash-mt=5 (default)
 #  5544 @ Intel i5-12500 (cpp 1.9.5, py 3.11.2, debian 12.0)  // --hash-mt=12; desktop
 #  5197 @ Ryzen 7 3700X  (cpp 1.9.5, py 3.9.18, freebsd 13.2)  // --hash-mt=8; 2u server
+#  4551 @ mbp 2020 m1    (cpp 1.9.5, py 3.11.7, macos 14.2.1)
+#  4190 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.6, fedora 37)  // --hash-mt=8 (vbox-VM on win10-17763.4974)
+#  3028 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.6, fedora 37)  // --hash-mt=5 (vbox-VM on win10-17763.4974)
+#  2629 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.7, win10-ltsc-1809-17763.4974)  // --hash-mt=5 (default)
+#  2576 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.7, win10-ltsc-1809-17763.4974)  // --hash-mt=8 (hello??)
 #  2606 @ Ryzen 7 3700X  (cpp 1.9.5, py 3.9.18, freebsd 13.2)  // --hash-mt=4 (old-default)
 #  1436 @ Ryzen 5 5500U  (cpp 1.9.5, py 3.11.4, alpine 3.18.3)  // nuc
 #  1065 @ Pixel 7        (cpp 1.9.5, py 3.11.5, termux 2023-09)
+#   945 @ Pi 5B v1.0     (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
+#   548 @ Pi 4B v1.5     (cpp 1.9.5, py 3.11.6, debian 11)
+#   435 @ Pi 4B v1.5     (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
+#   212 @ Pi Zero2W v1.0 (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
+#  10.0 @ Pi Zero W v1.1 (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
 
 # notes,
 # podman run --rm -it --shm-size 512m --entrypoint /bin/ash localhost/copyparty-min

scripts/deps-docker/Dockerfile

@@ -1,11 +1,11 @@
 FROM    alpine:3.18
 WORKDIR /z
 ENV     ver_asmcrypto=c72492f4a66e17a0e5dd8ad7874de354f3ccdaa5 \
-        ver_hashwasm=4.9.0 \
+        ver_hashwasm=4.10.0 \
         ver_marked=4.3.0 \
-        ver_dompf=3.0.5 \
+        ver_dompf=3.0.8 \
         ver_mde=2.18.0 \
-        ver_codemirror=5.65.12 \
+        ver_codemirror=5.65.16 \
         ver_fontawesome=5.13.0 \
         ver_prism=1.29.0 \
         ver_zopfli=1.0.3
@@ -80,7 +80,7 @@ RUN     cd asmcrypto.js-$ver_asmcrypto \
 
 
 # build hash-wasm
-RUN     cd hash-wasm \
+RUN     cd hash-wasm/dist \
         && mv sha512.umd.min.js /z/dist/sha512.hw.js
 
 

scripts/deps-docker/zopfli.makefile

@@ -1,6 +1,6 @@
 all: $(addsuffix .gz, $(wildcard *.js *.css))
 
 %.gz: %
-	pigz -11 -I 573 $<
+	pigz -11 -I 2048 $<
 
 # pigz -11 -J 34 -I 100 -F < $< > $@.first

scripts/docker/README.md

@@ -19,7 +19,7 @@ docker run --rm -it -u 1000 -p 3923:3923 -v /mnt/nas:/w -v $PWD/cfgdir:/cfg copy
 
 this example is also available as a podman-compatible [docker-compose yaml](https://github.com/9001/copyparty/blob/hovudstraum/docs/examples/docker/basic-docker-compose); example usage: `docker-compose up` (you may need to `systemctl enable --now podman.socket` or similar)
 
-i'm unfamiliar with docker-compose and alternatives so let me know if this section could be better 🙏
+i'm not very familiar with containers, so let me know if this section could be better 🙏
 
 
 ## configuration

scripts/logpack.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+set -e
+
+# recompress logs so they decompress faster + save some space;
+# * will not recurse into subfolders
+# * each file in current folder gets recompressed to zstd; input file is DELETED
+# * any xz-compressed logfiles are decompressed before converting to zstd
+# * SHOULD ignore and skip files which are currently open; SHOULD be safe to run while copyparty is running
+
+
+# for files larger than $cutoff, compress with `zstd -T0`
+# (otherwise do several files in parallel (scales better))
+cutoff=400M
+
+
+# osx support:
+# port install findutils gsed coreutils
+command -v gfind >/dev/null &&
+command -v gsed  >/dev/null &&
+command -v gsort >/dev/null && {
+    find() { gfind "$@"; }
+    sed()  { gsed  "$@"; }
+    sort() { gsort "$@"; }
+}
+
+packfun() {
+    local jobs=$1 fn="$2"
+    printf '%s\n' "$fn" | grep -qF .zst && return
+
+    local of="$(printf '%s\n' "$fn" | sed -r 's/\.(xz|txt)/.zst/')"
+    [ "$fn" = "$of" ] &&
+        of="$of.zst"
+
+    [ -e "$of" ] &&
+        echo "SKIP: output file exists: $of" &&
+        return
+
+    lsof -- "$fn" 2>/dev/null | grep -E .. &&
+        printf "SKIP: file in use: %s\n\n" $fn &&
+        return
+
+    # determine by header; old copyparty versions would produce xz output without .xz names
+    head -c3 "$fn" | grep -qF 7z &&
+        cmd="xz -dkc" || cmd="cat"
+
+    printf '<%s> T%d: %s\n' "$cmd" $jobs "$of"
+
+    $cmd <"$fn" >/dev/null || {
+        echo "ERROR: uncompress failed: $fn"
+        return
+    }
+
+    $cmd <"$fn" | zstd --long -19 -T$jobs >"$of"
+    touch -r "$fn" -- "$of"
+
+    cmp <($cmd <"$fn") <(zstd -d <"$of") || {
+        echo "ERROR: data mismatch: $of"
+        mv "$of"{,.BAD}
+        return
+    }
+    rm -- "$fn"
+}
+
+# do small files in parallel first (in descending size);
+# each file can use 4 threads in case the cutoff is poor
+export -f packfun
+export -f sed 2>/dev/null || true
+find -maxdepth 1 -type f -size -$cutoff -printf '%s %p\n' |
+sort -nr | sed -r 's`[^ ]+ ``; s`^\./``' | tr '\n' '\0' |
+xargs "$@" -0i -P$(nproc) bash -c 'packfun 4 "$@"' _ {}
+
+# then the big ones, letting each file use the whole cpu
+for f in *; do packfun 0 "$f"; done

scripts/pyinstaller/deps.sha512

@@ -28,5 +28,5 @@ ba91ab0518c61eff13e5612d9e6b532940813f6b56e6ed81ea6c7c4d45acee4d98136a383a250675
 7f8f4daa4f4f2dbf24cdd534b2952ee3fba6334eb42b37465ccda3aa1cccc3d6204aa6bfffb8a83bf42ec59c702b5b5247d4c8ee0d4df906334ae53072ef8c4c  MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl
 8a6e2b13a2ec4ef914a5d62aad3db6464d45e525a82e07f6051ed10474eae959069e165dba011aefb8207cdfd55391d73d6f06362c7eb247b08763106709526e  mutagen-1.47.0-py3-none-any.whl
 656015f5cc2c04aa0653ee5609c39a7e5f0b6a58c84fe26b20bd070c52d20b4effb810132f7fb771168483e9fd975cc3302837dd7a1a687ee058b0460c857cc4  packaging-23.2-py3-none-any.whl
-6401616fdfdd720d1aaa9a0ed1398d00664b28b6d84517dff8d1f9c416452610c6afa64cfb012a78e61d1cf4f6d0784eca6e7610957859e511f15bc6f3b3bd53  Pillow-10.1.0-cp311-cp311-win_amd64.whl
-36442c017d8fc603745d33ca888b5b1194644103cbe1ff53e32d9b0355e290d5efac655fa1ae1b8e552ad8468878dc600d550c1158224260ca463991442e5264  python-3.11.6-amd64.exe
+424e20dc7263a31d524307bc39ed755a9dd82f538086fff68d98dd97e236c9b00777a8ac2e3853081b532b0e93cef44983e74d0ab274877440e8b7341b19358a  pillow-10.2.0-cp311-cp311-win_amd64.whl
+2e6a57bab45b5a825a2073780c73980cbf5aafd99dc3b28660ea3f5f658f04668cd0f01c7de0bb79e362ff4e3b8f01dd4f671d3a2e054d3071baefdcf0b0e4ba  python-3.11.7-amd64.exe

scripts/pyinstaller/notes.txt

@@ -28,8 +28,8 @@ fns=(
   MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl
   mutagen-1.47.0-py3-none-any.whl
   packaging-23.2-py3-none-any.whl
-  Pillow-10.1.0-cp311-cp311-win_amd64.whl
-  python-3.11.6-amd64.exe
+  pillow-10.2.0-cp311-cp311-win_amd64.whl
+  python-3.11.7-amd64.exe
 )
 [ $w7 ] && fns+=(
   pyinstaller-5.13.2-py3-none-win32.whl

scripts/run-tests.sh

@@ -1,6 +1,17 @@
 #!/bin/bash
 set -ex
 
+# osx support
+gtar=$(command -v gtar || command -v gnutar) || true
+[ ! -z "$gtar" ] && command -v gfind >/dev/null && {
+	tar()  { $gtar "$@"; }
+	sed()  { gsed  "$@"; }
+	find() { gfind "$@"; }
+	sort() { gsort "$@"; }
+	command -v grealpath >/dev/null &&
+		realpath() { grealpath "$@"; }
+}
+
 rm -rf unt
 mkdir -p unt/srv
 cp -pR copyparty tests unt/
@@ -30,9 +41,11 @@ for py in python{2,3}; do
     [ "${1:0:6}" = python ] && [ "$1" != $py ] && continue
 
     PYTHONPATH=
-    [ $py = python2 ] && PYTHONPATH=../scripts/py2:../sfx/py37
+    [ $py = python2 ] && PYTHONPATH=../scripts/py2:../sfx/py37:../sfx/j2
     export PYTHONPATH
 
+    [ $py = python2 ] && py=$(command -v python2.7 || echo $py)
+
     nice $py -m unittest discover -s tests >/dev/null &
     pids+=($!)
 done

scripts/sfx.py

@@ -262,7 +262,7 @@ def unpack():
     final = opj(top, name)
     san = opj(final, "copyparty/up2k.py")
     for suf in range(0, 9001):
-        withpid = "{}.{}.{}".format(name, os.getpid(), suf)
+        withpid = "%s.%d.%s" % (name, os.getpid(), suf)
         mine = opj(top, withpid)
         if not ofe(mine):
             break
@@ -285,8 +285,8 @@ def unpack():
 
     ck = hashfile(tar)
     if ck != CKSUM:
-        t = "\n\nexpected {} ({} byte)\nobtained {} ({} byte)\nsfx corrupt"
-        raise Exception(t.format(CKSUM, SIZE, ck, sz))
+        t = "\n\nexpected %s (%d byte)\nobtained %s (%d byte)\nsfx corrupt"
+        raise Exception(t % (CKSUM, SIZE, ck, sz))
 
     with tarfile.open(tar, "r:bz2") as tf:
         # this is safe against traversal

tests/test_dots.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import io
+import os
+import shutil
+import tarfile
+import tempfile
+import unittest
+
+from copyparty.authsrv import AuthSrv
+from copyparty.httpcli import HttpCli
+from copyparty.up2k import Up2k
+from copyparty.u2idx import U2idx
+from tests import util as tu
+from tests.util import Cfg
+
+
+def hdr(query, uname):
+    h = "GET /%s HTTP/1.1\r\nPW: %s\r\nConnection: close\r\n\r\n"
+    return (h % (query, uname)).encode("utf-8")
+
+
+class TestHttpCli(unittest.TestCase):
+    def setUp(self):
+        self.td = tu.get_ramdisk()
+
+    def tearDown(self):
+        os.chdir(tempfile.gettempdir())
+        shutil.rmtree(self.td)
+
+    def test(self):
+        td = os.path.join(self.td, "vfs")
+        os.mkdir(td)
+        os.chdir(td)
+
+        # topDir  volA  volA/*dirA  .volB  .volB/*dirB
+        spaths = " t .t a a/da a/.da .b .b/db .b/.db"
+        for n, dirpath in enumerate(spaths.split(" ")):
+            if dirpath:
+                os.makedirs(dirpath)
+
+            for pfx in "f", ".f":
+                filepath = pfx + str(n)
+                if dirpath:
+                    filepath = os.path.join(dirpath, filepath)
+
+                with open(filepath, "wb") as f:
+                    f.write(filepath.encode("utf-8"))
+
+        vcfg = [
+            ".::r,u1:r.,u2",
+            "a:a:r,u1:r,u2",
+            ".b:.b:r.,u1:r,u2"
+        ]
+        self.args = Cfg(v=vcfg, a=["u1:u1", "u2:u2"], e2dsa=True)
+        self.asrv = AuthSrv(self.args, self.log)
+
+        self.assertEqual(self.tardir("", "u1"), "f0 t/f1 a/f3 a/da/f4")
+        self.assertEqual(self.tardir(".t", "u1"), "f2")
+        self.assertEqual(self.tardir(".b", "u1"), ".f6 f6 .db/.f8 .db/f8 db/.f7 db/f7")
+
+        zs = ".f0 f0 .t/.f2 .t/f2 t/.f1 t/f1 .b/f6 .b/db/f7 a/f3 a/da/f4"
+        self.assertEqual(self.tardir("", "u2"), zs)
+
+        self.assertEqual(self.curl("?tar", "x")[1][:17], "\nJ2EOT")
+
+        # search
+        up2k = Up2k(self)
+        u2idx = U2idx(self)
+        allvols = list(self.asrv.vfs.all_vols.values())
+
+        x = u2idx.search("u1", allvols, "", 999)
+        x = " ".join(sorted([x["rp"] for x in x[0]]))
+        # u1 can see dotfiles in volB so they should be included
+        xe = ".b/.db/.f8 .b/.db/f8 .b/.f6 .b/db/.f7 .b/db/f7 .b/f6 a/da/f4 a/f3 f0 t/f1"
+        self.assertEqual(x, xe)
+
+        x = u2idx.search("u2", allvols, "", 999)
+        x = " ".join(sorted([x["rp"] for x in x[0]]))
+        self.assertEqual(x, ".f0 .t/.f2 .t/f2 a/da/f4 a/f3 f0 t/.f1 t/f1")
+
+        self.args = Cfg(v=vcfg, a=["u1:u1", "u2:u2"], dotsrch=False)
+        self.asrv = AuthSrv(self.args, self.log)
+        u2idx = U2idx(self)
+
+        x = u2idx.search("u1", self.asrv.vfs.all_vols.values(), "", 999)
+        x = " ".join(sorted([x["rp"] for x in x[0]]))
+        # u1 can see dotfiles in volB so they should be included
+        xe = "a/da/f4 a/f3 f0 t/f1"
+        self.assertEqual(x, xe)
+
+    def tardir(self, url, uname):
+        h, b = self.curl("/" + url + "?tar", uname, True)
+        tar = tarfile.open(fileobj=io.BytesIO(b), mode="r|").getnames()
+        top = ("top" if not url else url.lstrip(".").split("/")[0]) + "/"
+        assert len(tar) == len([x for x in tar if x.startswith(top)])
+        return " ".join([x[len(top):] for x in tar])
+
+    def curl(self, url, uname, binary=False):
+        conn = tu.VHttpConn(self.args, self.asrv, self.log, hdr(url, uname))
+        HttpCli(conn).run()
+        if binary:
+            h, b = conn.s._reply.split(b"\r\n\r\n", 1)
+            return [h.decode("utf-8"), b]
+
+        return conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def log(self, src, msg, c=0):
+        print(msg)

tests/test_vfs.py

@@ -7,7 +7,6 @@ import os
 import shutil
 import tempfile
 import unittest
-from textwrap import dedent
 
 from copyparty import util
 from copyparty.authsrv import VFS, AuthSrv
@@ -175,11 +174,11 @@ class TestVFS(unittest.TestCase):
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a")
         self.assertEqual(n.realpath, os.path.join(td, "a"))
-        self.assertAxs(n.axs.uread, ["*"])
+        self.assertAxs(n.axs.uread, ["*", "k"])
         self.assertAxs(n.axs.uwrite, [])
-        perm_na = (False, False, False, False, False, False, False)
-        perm_rw = (True, True, False, False, False, False, False)
-        perm_ro = (True, False, False, False, False, False, False)
+        perm_na = (False, False, False, False, False, False, False, False)
+        perm_rw = (True, True, False, False, False, False, False, False)
+        perm_ro = (True, False, False, False, False, False, False, False)
         self.assertEqual(vfs.can_access("/", "*"), perm_na)
         self.assertEqual(vfs.can_access("/", "k"), perm_rw)
         self.assertEqual(vfs.can_access("/a", "*"), perm_ro)
@@ -232,7 +231,7 @@ class TestVFS(unittest.TestCase):
         cfg_path = os.path.join(self.td, "test.cfg")
         with open(cfg_path, "wb") as f:
             f.write(
-                dedent(
+                util.dedent(
                     """
                     u a:123
                     u asd:fgh:jkl

tests/util.py

@@ -44,6 +44,7 @@ if MACOS:
 from copyparty.__init__ import E
 from copyparty.__main__ import init_E
 from copyparty.util import FHC, Garda, Unrecv
+from copyparty.u2idx import U2idx
 
 init_E(E)
 
@@ -106,51 +107,60 @@ def get_ramdisk():
 
 
 class Cfg(Namespace):
-    def __init__(self, a=None, v=None, c=None):
+    def __init__(self, a=None, v=None, c=None, **ka0):
         ka = {}
 
-        ex = "daw dav_auth dav_inf dav_mac dav_rt dotsrch e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp ed emp exp force_js getmod grid hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_logues no_mv no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw rand smb th_no_crop vague_403 vc ver xdev xlink xvol"
+        ex = "daw dav_auth dav_inf dav_mac dav_rt e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp ed emp exp force_js getmod grid hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_lifetime no_logues no_mv no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw q rand smb srch_dbg stats th_no_crop vague_403 vc ver xdev xlink xvol"
         ka.update(**{k: False for k in ex.split()})
 
-        ex = "dotpart no_rescan no_sendfile no_voldump plain_ip"
+        ex = "dotpart dotsrch no_dhash no_fastboot no_rescan no_sendfile no_voldump re_dhash plain_ip"
         ka.update(**{k: True for k in ex.split()})
 
         ex = "ah_cli ah_gen css_browser hist ipa_re js_browser no_forget no_hash no_idx nonsus_urls"
         ka.update(**{k: None for k in ex.split()})
 
-        ex = "s_thead s_tbody th_convt"
+        ex = "hash_mt srch_time u2j"
+        ka.update(**{k: 1 for k in ex.split()})
+
+        ex = "reg_cap s_thead s_tbody th_convt"
         ka.update(**{k: 9 for k in ex.split()})
 
-        ex = "df loris re_maxage rproxy rsp_jtr rsp_slp s_wr_slp theme themes turbo"
+        ex = "db_act df loris re_maxage rproxy rsp_jtr rsp_slp s_wr_slp snap_wri theme themes turbo"
         ka.update(**{k: 0 for k in ex.split()})
 
-        ex = "ah_alg bname doctitle favico hdr_au_usr html_head lg_sbf log_fk md_sbf name textfiles unlist vname R RS SR"
+        ex = "ah_alg bname doctitle exit favico idp_h_usr html_head lg_sbf log_fk md_sbf name textfiles unlist vname R RS SR"
         ka.update(**{k: "" for k in ex.split()})
 
         ex = "on403 on404 xad xar xau xban xbd xbr xbu xiu xm"
         ka.update(**{k: [] for k in ex.split()})
 
-        ex = "exp_lg exp_md"
+        ex = "exp_lg exp_md th_coversd"
         ka.update(**{k: {} for k in ex.split()})
 
+        ka.update(ka0)
+
         super(Cfg, self).__init__(
             a=a or [],
             v=v or [],
             c=c,
             E=E,
             dbd="wal",
-            s_wr_sz=512 * 1024,
-            th_size="320x256",
             fk_salt="a" * 16,
-            unpost=600,
-            u2sort="s",
-            u2ts="c",
-            sort="href",
-            mtp=[],
+            lang="eng",
+            log_badpwd=1,
+            logout=573,
             mte={"a": True},
             mth={},
-            lang="eng",
-            logout=573,
+            mtp=[],
+            rm_retry="0/0",
+            s_wr_sz=512 * 1024,
+            sort="href",
+            srch_hits=99999,
+            th_size="320x256",
+            u2sort="s",
+            u2ts="c",
+            unpost=600,
+            warksalt="hunter2",
             **ka
         )
 
@@ -186,11 +196,16 @@ class VSock(object):
 
 
 class VHttpSrv(object):
-    def __init__(self):
+    def __init__(self, args, asrv, log):
+        self.args = args
+        self.asrv = asrv
+        self.log = log
+
         self.broker = NullBroker()
         self.prism = None
         self.bans = {}
         self.nreq = 0
+        self.nsus = 0
 
         aliases = ["splash", "browser", "browser2", "msg", "md", "mde"]
         self.j2 = {x: J2_FILES for x in aliases}
@@ -200,31 +215,38 @@ class VHttpSrv(object):
         self.g403 = Garda("")
         self.gurl = Garda("")
 
+        self.u2idx = None
         self.ptn_cc = re.compile(r"[\x00-\x1f]")
 
     def cachebuster(self):
         return "a"
 
+    def get_u2idx(self):
+        self.u2idx = self.u2idx or U2idx(self)
+        return self.u2idx
+
 
 class VHttpConn(object):
     def __init__(self, args, asrv, log, buf):
+        self.t0 = time.time()
         self.s = VSock(buf)
         self.sr = Unrecv(self.s, None)  # type: ignore
+        self.aclose = {}
         self.addr = ("127.0.0.1", "42069")
         self.args = args
         self.asrv = asrv
-        self.nid = None
+        self.bans = {}
+        self.freshen_pwd = 0.0
+        self.hsrv = VHttpSrv(args, asrv, log)
+        self.ico = None
+        self.lf_url = None
         self.log_func = log
         self.log_src = "a"
-        self.lf_url = None
-        self.hsrv = VHttpSrv()
-        self.bans = {}
-        self.aclose = {}
-        self.u2fh = FHC()
         self.mutex = threading.Lock()
-        self.nreq = -1
         self.nbyte = 0
-        self.ico = None
+        self.nid = None
+        self.nreq = -1
         self.thumbcli = None
-        self.freshen_pwd = 0.0
-        self.t0 = time.time()
+        self.u2fh = FHC()
+
+        self.get_u2idx = self.hsrv.get_u2idx