v1.18.4

togglebutton in the ui switches between 2 (off/default) and
1 (on/quick) confirmations; global-option `--qdel` sets the default

setting `--qdel=0` changes the togglebutton to switch
between 1 (off/default) confirmations and 0 (on)

in other words, when the ui-button is enabled, it
always reduces the number of confirmations by one

README.md

@@ -1501,7 +1501,6 @@ the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`,
 note:
 * upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
-* the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
 
 config file example (these options are recommended btw):
 
@@ -1606,7 +1605,7 @@ config file example:
     w: *    # anyone can upload here
     rw: ed  # only user "ed" can read-write
   flags:
-    e2ds:      # filesystem indexing is required for many of these:
+    e2ds       # filesystem indexing is required for many of these:
     sz: 1k-3m  # accept upload only if filesize in this range
     df: 4g     # free disk space cannot go lower than this
     vmaxb: 1g  # volume can never exceed 1 GiB
@@ -1663,6 +1662,8 @@ this can instead be kept in a single place using the `--hist` argument, or the `
 
 by default, the per-volume `up2k.db` sqlite3-database for `-e2d` and `-e2t` is stored next to the thumbnails according to the `--hist` option, but the global-option `--dbpath` and/or volflag `dbpath` can be used to put the database somewhere else
 
+if your storage backend is unreliable (NFS or bad HDDs), you can specify one or more "landmarks" to look for before doing anything database-related. A landmark is a file which is always expected to exist inside the volume. This avoids spurious filesystem rescans in the event of an outage. One line per landmark (see example below)
+
 note:
 * putting the hist-folders on an SSD is strongly recommended for performance
 * markdown edits are always stored in a local `.hist` subdirectory
@@ -1680,6 +1681,8 @@ config file example:
   flags:
     hist: -  # restore the default (/mnt/nas/pics/.hist/)
     hist: /mnt/nas/cache/pics/  # can be absolute path
+    landmark: me.jpg  # /mnt/nas/pics/me.jpg must be readable to enable db
+    landmark: info/a.txt^=ok  # and this textfile must start with "ok"
 ```
 
 
@@ -2365,8 +2368,10 @@ TLDR: yes
 | send message    | yep | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | set sort order  |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | zip selection   |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
+| file search     |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | file rename     |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | file cut/paste  |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
+| unpost uploads  |  -  |  -   | yep  | yep  |  yep  | yep  | yep | yep  |
 | navpane         |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | image viewer    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | video player    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
@@ -2728,7 +2733,7 @@ enable [thumbnails](#thumbnails) of...
 * **images:** `Pillow` and/or `pyvips` and/or `ffmpeg` (requires py2.7 or py3.5+)
 * **videos/audio:** `ffmpeg` and `ffprobe` somewhere in `$PATH`
 * **HEIF pictures:** `pyvips` or `ffmpeg` or `pyheif-pillow-opener` (requires Linux or a C compiler)
-* **AVIF pictures:** `pyvips` or `ffmpeg` or `pillow-avif-plugin`
+* **AVIF pictures:** `pyvips` or `ffmpeg` or `pillow-avif-plugin` or pillow v11.3+
 * **JPEG XL pictures:** `pyvips` or `ffmpeg`
 
 enable sending [zeromq messages](#zeromq) from event-hooks: `pyzmq`
@@ -2759,7 +2764,7 @@ set any of the following environment variables to disable its associated optiona
 | `PRTY_NO_MUTAGEN`    | do not use [mutagen](https://pypi.org/project/mutagen/) for reading metadata from media files; will fallback to ffprobe |
 | `PRTY_NO_PIL`        | disable all [Pillow](https://pypi.org/project/pillow/)-based thumbnail support; will fallback to libvips or ffmpeg |
 | `PRTY_NO_PILF`       | disable Pillow `ImageFont` text rendering, used for folder thumbnails |
-| `PRTY_NO_PIL_AVIF`   | disable 3rd-party Pillow plugin for [AVIF support](https://pypi.org/project/pillow-avif-plugin/) |
+| `PRTY_NO_PIL_AVIF`   | disable Pillow avif support (internal and/or [plugin](https://pypi.org/project/pillow-avif-plugin/)) |
 | `PRTY_NO_PIL_HEIF`   | disable 3rd-party Pillow plugin for [HEIF support](https://pypi.org/project/pyheif-pillow-opener/) |
 | `PRTY_NO_PIL_WEBP`   | disable use of native webp support in Pillow |
 | `PRTY_NO_PSUTIL`     | do not use [psutil](https://pypi.org/project/psutil/) for reaping stuck hooks and plugins on Windows |

bin/hooks/usb-eject.py

@@ -4,6 +4,7 @@ import os
 import stat
 import subprocess as sp
 import sys
+from urllib.parse import unquote_to_bytes as unquote
 
 
 """
@@ -28,14 +29,17 @@ which does the following respectively,
 """
 
 
+MOUNT_BASE = b"/run/media/egon/"
+
+
 def main():
     try:
         label = sys.argv[1].split(":usb-eject:")[1].split(":")[0]
-        mp = "/run/media/egon/" + label
+        mp = MOUNT_BASE + unquote(label)
         # print("ejecting [%s]... " % (mp,), end="")
-        mp = os.path.abspath(os.path.realpath(mp.encode("utf-8")))
+        mp = os.path.abspath(os.path.realpath(mp))
         st = os.lstat(mp)
-        if not stat.S_ISDIR(st.st_mode):
+        if not stat.S_ISDIR(st.st_mode) or not mp.startswith(MOUNT_BASE):
             raise Exception("not a regular directory")
 
         # if you're running copyparty as root (thx for the faith)

bin/mtag/install-deps.sh

@@ -22,6 +22,8 @@ set -e
 #   modifies the keyfinder python lib to load the .so in ~/pe
 
 
+export FORCE_COLOR=1
+
 linux=1
 
 win=
@@ -187,11 +189,14 @@ install_keyfinder() {
 		exit 1
 	}
 
+	x=${-//[^x]/}; set -x; cat /etc/alpine-release
 	# rm -rf /Users/ed/Library/Python/3.9/lib/python/site-packages/*keyfinder*
 	CFLAGS="-I$h/pe/keyfinder/include -I/opt/local/include -I/usr/include/ffmpeg" \
+	CXXFLAGS="-I$h/pe/keyfinder/include -I/opt/local/include -I/usr/include/ffmpeg" \
 	LDFLAGS="-L$h/pe/keyfinder/lib -L$h/pe/keyfinder/lib64 -L/opt/local/lib" \
-	PKG_CONFIG_PATH=/c/msys64/mingw64/lib/pkgconfig \
+	PKG_CONFIG_PATH="/c/msys64/mingw64/lib/pkgconfig:$h/pe/keyfinder/lib/pkgconfig" \
 	$pybin -m pip install --user keyfinder
+	[ "$x" ] || set +x
 
 	pypath="$($pybin -c 'import keyfinder; print(keyfinder.__file__)')"
 	for pyso in "${pypath%/*}"/*.so; do

contrib/package/arch/PKGBUILD

@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.17.2"
+pkgver="1.18.3"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -22,7 +22,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("20af4a9b3188fee235c505af4a09190088d0094ab594e37ca1eabbda41c8912d")
+sha256sums=("aa12f4779cf5c014cc9503798ac63872dac840ca91ddf122daa6befb4c883d48")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.17.2/copyparty-sfx.py",
-    "version": "1.17.2",
-    "hash": "sha256-qY3QTLthJLpvO9yo2kiwM2cT3eqg5/cGvGzp+JHy4Ko="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.18.3/copyparty-sfx.py",
+    "version": "1.18.3",
+    "hash": "sha256-INqErls4gyhBAlDlY1vfNboKrrqHmeiyB+RAuuYRISQ="
 }

copyparty/__init__.py

@@ -80,6 +80,7 @@ web/deps/prismd.css
 web/deps/scp.woff2
 web/deps/sha512.ac.js
 web/deps/sha512.hw.js
+web/idp.html
 web/iiam.gif
 web/md.css
 web/md.html

copyparty/__main__.py

@@ -863,6 +863,43 @@ def get_sects():
             """
             ),
         ],
+        [
+            "chmod",
+            "file/folder permissions",
+            dedent(
+                """
+            global-option \033[33m--chmod-f\033[0m and volflag \033[33mchmod_f\033[0m specifies the unix-permission to use when creating a new file
+
+            similarly, \033[33m--chmod-d\033[0m and \033[33mchmod_d\033[0m sets the directory/folder perm
+
+            the value is a three-digit octal number such as 755, 750, 644, etc.
+
+            first digit = "User"; permission for the unix-user
+            second digit = "Group"; permission for the unix-group
+            third digit = "Other"; permission for all other users/groups
+
+            for files:
+            0 = --- = no access
+            1 = --x = can execute the file as a program
+            2 = -w- = can write
+            3 = -wx = can write and execute
+            4 = r-- = can read
+            5 = r-x = can read and execute
+            6 = rw- = can read and write
+            7 = rwx = can read, write, execute
+
+            for directories/folders:
+            0 = --- = no access
+            1 = --x = can read files in folder but not list contents
+            2 = -w- = n/a
+            3 = -wx = can create files but not list
+            4 = r-- = can list, but not read/write
+            5 = r-x = can list and read files
+            6 = rw- = n/a
+            7 = rwx = can read, write, list
+            """
+            ),
+        ],
         [
             "pwhash",
             "password hashing",
@@ -1013,6 +1050,8 @@ def add_upload(ap):
     ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without \033[33m-e2d\033[0m; roughly 1 MiB RAM per 600")
     ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (bad idea to enable this on windows and/or cow filesystems)")
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
+    ap2.add_argument("--chmod-f", metavar="UGO", type=u, default="", help="unix file permissions to use when creating files; default is probably 644 (OS-decided), see --help-chmod. Examples: [\033[32m644\033[0m] = owner-RW + all-R, [\033[32m755\033[0m] = owner-RWX + all-RX, [\033[32m777\033[0m] = full-yolo (volflag=chmod_f)")
+    ap2.add_argument("--chmod-d", metavar="UGO", type=u, default="755", help="unix file permissions to use when creating directories; see --help-chmod. Examples: [\033[32m755\033[0m] = owner-RW + all-R, [\033[32m777\033[0m] = full-yolo (volflag=chmod_d)")
     ap2.add_argument("--dedup", action="store_true", help="enable symlink-based upload deduplication (volflag=dedup)")
     ap2.add_argument("--safe-dedup", metavar="N", type=int, default=50, help="how careful to be when deduplicating files; [\033[32m1\033[0m] = just verify the filesize, [\033[32m50\033[0m] = verify file contents have not been altered (volflag=safededup)")
     ap2.add_argument("--hardlink", action="store_true", help="enable hardlink-based dedup; will fallback on symlinks when that is impossible (across filesystems) (volflag=hardlink)")
@@ -1049,7 +1088,7 @@ def add_network(ap):
     ap2.add_argument("--rp-loc", metavar="PATH", type=u, default="", help="if reverse-proxying on a location instead of a dedicated domain/subdomain, provide the base location here; example: [\033[32m/foo/bar\033[0m]")
     if ANYWIN:
         ap2.add_argument("--reuseaddr", action="store_true", help="set reuseaddr on listening sockets on windows; allows rapid restart of copyparty at the expense of being able to accidentally start multiple instances")
-    else:
+    elif not MACOS:
         ap2.add_argument("--freebind", action="store_true", help="allow listening on IPs which do not yet exist, for example if the network interfaces haven't finished going up. Only makes sense for IPs other than '0.0.0.0', '127.0.0.1', '::', and '::1'. May require running as root (unless net.ipv6.ip_nonlocal_bind)")
     ap2.add_argument("--wr-h-eps", metavar="PATH", type=u, default="", help="write list of listening-on ip:port to textfile at \033[33mPATH\033[0m when http-servers have started")
     ap2.add_argument("--wr-h-aon", metavar="PATH", type=u, default="", help="write list of accessible-on ip:port to textfile at \033[33mPATH\033[0m when http-servers have started")
@@ -1093,12 +1132,16 @@ def add_cert(ap, cert_path):
 
 
 def add_auth(ap):
+    idp_db = os.path.join(E.cfg, "idp.db")
     ses_db = os.path.join(E.cfg, "sessions.db")
     ap2 = ap.add_argument_group('IdP / identity provider / user authentication options')
     ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks if the request-header \033[33mHN\033[0m contains a username to associate the request with (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
     ap2.add_argument("--idp-h-grp", metavar="HN", type=u, default="", help="assume the request-header \033[33mHN\033[0m contains the groupname of the requesting user; can be referenced in config files for group-based access control")
     ap2.add_argument("--idp-h-key", metavar="HN", type=u, default="", help="optional but recommended safeguard; your reverse-proxy will insert a secret header named \033[33mHN\033[0m into all requests, and the other IdP headers will be ignored if this header is not present")
     ap2.add_argument("--idp-gsep", metavar="RE", type=u, default="|:;+,", help="if there are multiple groups in \033[33m--idp-h-grp\033[0m, they are separated by one of the characters in \033[33mRE\033[0m")
+    ap2.add_argument("--idp-db", metavar="PATH", type=u, default=idp_db, help="where to store the known IdP users/groups (if you run multiple copyparty instances, make sure they use different DBs)")
+    ap2.add_argument("--idp-store", metavar="N", type=int, default=1, help="how to use \033[33m--idp-db\033[0m; [\033[32m0\033[0m] = entirely disable, [\033[32m1\033[0m] = write-only (effectively disabled), [\033[32m2\033[0m] = remember users, [\033[32m3\033[0m] = remember users and groups.\nNOTE: Will remember and restore the IdP-volumes of all users for all eternity if set to 2 or 3, even when user is deleted from your IdP")
+    ap2.add_argument("--idp-adm", metavar="U,U", type=u, default="", help="comma-separated list of users allowed to use /?idp (the cache management UI)")
     ap2.add_argument("--no-bauth", action="store_true", help="disable basic-authentication support; do not accept passwords from the 'Authenticate' header at all. NOTE: This breaks support for the android app")
     ap2.add_argument("--bauth-last", action="store_true", help="keeps basic-authentication enabled, but only as a last-resort; if a cookie is also provided then the cookie wins")
     ap2.add_argument("--ses-db", metavar="PATH", type=u, default=ses_db, help="where to store the sessions database (if you run multiple copyparty instances, make sure they use different DBs)")
@@ -1507,6 +1550,7 @@ def add_ui(ap, retry):
     ap2.add_argument("--nsort", action="store_true", help="default-enable natural sort of filenames with leading numbers (volflag=nsort)")
     ap2.add_argument("--hsortn", metavar="N", type=int, default=2, help="number of sorting rules to include in media URLs by default (volflag=hsortn)")
     ap2.add_argument("--see-dots", action="store_true", help="default-enable seeing dotfiles; only takes effect if user has the necessary permissions")
+    ap2.add_argument("--qdel", metavar="LVL", type=int, default=2, help="number of confirmations to show when deleting files (2/1/0)")
     ap2.add_argument("--unlist", metavar="REGEX", type=u, default="", help="don't show files matching \033[33mREGEX\033[0m in file list. Purely cosmetic! Does not affect API calls, just the browser. Example: [\033[32m\\.(js|css)$\033[0m] (volflag=unlist)")
     ap2.add_argument("--favico", metavar="TXT", type=u, default="c 000 none" if retry else "🎉 000 none", help="\033[33mfavicon-text\033[0m [ \033[33mforeground\033[0m [ \033[33mbackground\033[0m ] ], set blank to disable")
     ap2.add_argument("--ext-th", metavar="E=VP", type=u, action="append", help="use thumbnail-image \033[33mVP\033[0m for file-extension \033[33mE\033[0m, example: [\033[32mexe=/.res/exe.png\033[0m] (volflag=ext_th)")
@@ -1531,6 +1575,7 @@ def add_ui(ap, retry):
     ap2.add_argument("--lg-sba", metavar="TXT", type=u, default="", help="the value of the iframe 'allow' attribute for prologue/epilogue docs (volflag=lg_sba); see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy#iframes")
     ap2.add_argument("--no-sb-md", action="store_true", help="don't sandbox README/PREADME.md documents (volflags: no_sb_md | sb_md)")
     ap2.add_argument("--no-sb-lg", action="store_true", help="don't sandbox prologue/epilogue docs (volflags: no_sb_lg | sb_lg); enables non-js support")
+    ap2.add_argument("--have-unlistc", action="store_true", help=argparse.SUPPRESS)
 
 
 def add_debug(ap):

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 18, 0)
+VERSION = (1, 18, 4)
 CODENAME = "logtail"
-BUILD_DT = (2025, 6, 22)
+BUILD_DT = (2025, 7, 25)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -21,6 +21,7 @@ from .util import (
     DEF_MTE,
     DEF_MTH,
     EXTS,
+    HAVE_SQLITE3,
     IMPLICATIONS,
     MIMES,
     SQLITE_VER,
@@ -32,6 +33,7 @@ from .util import (
     afsenc,
     get_df,
     humansize,
+    min_ex,
     odfusion,
     read_utf8,
     relchk,
@@ -44,6 +46,9 @@ from .util import (
     vsplit,
 )
 
+if HAVE_SQLITE3:
+    import sqlite3
+
 if True:  # pylint: disable=using-constant-test
     from collections.abc import Iterable
 
@@ -115,6 +120,8 @@ class Lim(object):
 
         self.reg: Optional[dict[str, dict[str, Any]]] = None  # up2k registry
 
+        self.chmod_d = 0o755
+
         self.nups: dict[str, list[float]] = {}  # num tracker
         self.bups: dict[str, list[tuple[float, int]]] = {}  # byte tracker list
         self.bupc: dict[str, int] = {}  # byte tracker cache
@@ -275,7 +282,7 @@ class Lim(object):
         if not dirs:
             # no branches yet; make one
             sub = os.path.join(path, "0")
-            bos.mkdir(sub)
+            bos.mkdir(sub, self.chmod_d)
         else:
             # try newest branch only
             sub = os.path.join(path, str(dirs[-1]))
@@ -290,7 +297,7 @@ class Lim(object):
 
         # make a branch
         sub = os.path.join(path, str(dirs[-1] + 1))
-        bos.mkdir(sub)
+        bos.mkdir(sub, self.chmod_d)
         ret = self.dive(sub, lvs - 1)
         if ret is None:
             raise Pebkac(500, "rotation bug")
@@ -367,6 +374,7 @@ class VFS(object):
         self.shr_src: Optional[tuple[VFS, str]] = None  # source vfs+rem of a share
         self.shr_files: set[str] = set()  # filenames to include from shr_src
         self.shr_owner: str = ""  # uname
+        self.shr_all_aps: list[tuple[str, list[VFS]]] = []
         self.aread: dict[str, list[str]] = {}
         self.awrite: dict[str, list[str]] = {}
         self.amove: dict[str, list[str]] = {}
@@ -378,20 +386,20 @@ class VFS(object):
         self.adot: dict[str, list[str]] = {}
         self.js_ls = {}
         self.js_htm = ""
+        self.all_vols: dict[str, VFS] = {}  # flattened recursive
+        self.all_nodes: dict[str, VFS] = {}  # also jumpvols/shares
 
         if realpath:
             rp = realpath + ("" if realpath.endswith(os.sep) else os.sep)
             vp = vpath + ("/" if vpath else "")
             self.histpath = os.path.join(realpath, ".hist")  # db / thumbcache
             self.dbpath = self.histpath
-            self.all_vols = {vpath: self}  # flattened recursive
-            self.all_nodes = {vpath: self}  # also jumpvols/shares
-            self.all_aps = [(rp, self)]
+            self.all_vols[vpath] = self
+            self.all_nodes[vpath] = self
+            self.all_aps = [(rp, [self])]
             self.all_vps = [(vp, self)]
         else:
             self.histpath = self.dbpath = ""
-            self.all_vols = {}
-            self.all_nodes = {}
             self.all_aps = []
             self.all_vps = []
 
@@ -410,7 +418,7 @@ class VFS(object):
         self,
         vols: dict[str, "VFS"],
         nodes: dict[str, "VFS"],
-        aps: list[tuple[str, "VFS"]],
+        aps: list[tuple[str, list["VFS"]]],
         vps: list[tuple[str, "VFS"]],
     ) -> None:
         nodes[self.vpath] = self
@@ -419,7 +427,11 @@ class VFS(object):
             rp = self.realpath
             rp += "" if rp.endswith(os.sep) else os.sep
             vp = self.vpath + ("/" if self.vpath else "")
-            aps.append((rp, self))
+            hit = next((x[1] for x in aps if x[0] == rp), None)
+            if hit:
+                hit.append(self)
+            else:
+                aps.append((rp, [self]))
             vps.append((vp, self))
 
         for v in self.nodes.values():
@@ -843,9 +855,11 @@ class VFS(object):
                 return None
 
         if "xvol" in self.flags:
-            for vap, vn in self.root.all_aps:
+            all_aps = self.shr_all_aps or self.root.all_aps
+
+            for vap, vns in all_aps:
                 if aps.startswith(vap):
-                    return vn
+                    return self if self in vns else vns[0]
 
             if self.log:
                 self.log("vfs", "xvol: %r" % (ap,), 3)
@@ -854,6 +868,53 @@ class VFS(object):
 
         return self
 
+    def check_landmarks(self) -> bool:
+        if self.dbv:
+            return True
+
+        vps = self.flags.get("landmark") or []
+        if not vps:
+            return True
+
+        failed = ""
+        for vp in vps:
+            if "^=" in vp:
+                vp, zs = vp.split("^=", 1)
+                expect = zs.encode("utf-8")
+            else:
+                expect = b""
+
+            if self.log:
+                t = "checking [/%s] landmark [%s]"
+                self.log("vfs", t % (self.vpath, vp), 6)
+
+            ap = "?"
+            try:
+                ap = self.canonical(vp)
+                with open(ap, "rb") as f:
+                    buf = f.read(4096)
+                    if not buf.startswith(expect):
+                        t = "file [%s] does not start with the expected bytes %s"
+                        failed = t % (ap, expect)
+                        break
+            except Exception as ex:
+                t = "%r while trying to read [%s] => [%s]"
+                failed = t % (ex, vp, ap)
+                break
+
+        if not failed:
+            return True
+
+        if self.log:
+            t = "WARNING: landmark verification failed; %s; will now disable up2k database for volume [/%s]"
+            self.log("vfs", t % (failed, self.vpath), 3)
+
+        for rm in "e2d e2t e2v".split():
+            self.flags = {k: v for k, v in self.flags.items() if not k.startswith(rm)}
+        self.flags["d2d"] = True
+        self.flags["d2t"] = True
+        return False
+
 
 if WINDOWS:
     re_vol = re.compile(r"^([a-zA-Z]:[\\/][^:]*|[^:]*):([^:]*):(.*)$")
@@ -917,6 +978,9 @@ class AuthSrv(object):
 
         yield prev, True
 
+    def vf0(self):
+        return {"d2d": True, "tcolor": self.args.tcolor}
+
     def idp_checkin(
         self, broker: Optional["BrokerCli"], uname: str, gname: str
     ) -> bool:
@@ -935,6 +999,10 @@ class AuthSrv(object):
                 return False
 
             self.idp_accs[uname] = gnames
+            try:
+                self._update_idp_db(uname, gname)
+            except:
+                self.log("failed to update the --idp-db:\n%s" % (min_ex(),), 3)
 
             t = "reinitializing due to new user from IdP: [%r:%r]"
             self.log(t % (uname, gnames), 3)
@@ -947,6 +1015,22 @@ class AuthSrv(object):
         broker.ask("reload", False, True).get()
         return True
 
+    def _update_idp_db(self, uname: str, gname: str) -> None:
+        if not self.args.idp_store:
+            return
+
+        assert sqlite3  # type: ignore  # !rm
+
+        db = sqlite3.connect(self.args.idp_db)
+        cur = db.cursor()
+
+        cur.execute("delete from us where un = ?", (uname,))
+        cur.execute("insert into us values (?,?)", (uname, gname))
+
+        db.commit()
+        cur.close()
+        db.close()
+
     def _map_volume_idp(
         self,
         src: str,
@@ -1095,6 +1179,7 @@ class AuthSrv(object):
          * any non-zero value from IdP group header
          * otherwise take --grps / [groups]
         """
+        self.load_idp_db(bool(self.idp_accs))
         ret = {un: gns[:] for un, gns in self.idp_accs.items()}
         ret.update({zs: [""] for zs in acct if zs not in ret})
         for gn, uns in grps.items():
@@ -1463,7 +1548,7 @@ class AuthSrv(object):
             flags[name] = True
             return
 
-        zs = "ext_th mtp on403 on404 xbu xau xiu xbc xac xbr xar xbd xad xm xban"
+        zs = "ext_th landmark mtp on403 on404 xbu xau xiu xbc xac xbr xar xbd xad xm xban"
         if name not in zs.split():
             if value is True:
                 t = "└─add volflag [{}] = {}  ({})"
@@ -1602,13 +1687,12 @@ class AuthSrv(object):
                 t = "Read-access has been disabled due to failsafe: No volumes were defined by the config-file. This failsafe is to prevent unintended access if this is due to accidental loss of config. You can override this safeguard and allow read/write to the working-directory by adding the following arguments:  -v .::rw"
                 self.log(t, 1)
                 axs = AXS()
-            vfs = VFS(self.log_func, absreal("."), "", "", axs, {})
+            vfs = VFS(self.log_func, absreal("."), "", "", axs, self.vf0())
             if not axs.uread:
                 self.badcfg1 = True
         elif "" not in mount:
             # there's volumes but no root; make root inaccessible
-            zsd = {"d2d": True, "tcolor": self.args.tcolor}
-            vfs = VFS(self.log_func, "", "", "", AXS(), zsd)
+            vfs = VFS(self.log_func, "", "", "", AXS(), self.vf0())
 
         maxdepth = 0
         for dst in sorted(mount.keys(), key=lambda x: (x.count("/"), len(x))):
@@ -1655,10 +1739,9 @@ class AuthSrv(object):
         shr = enshare[1:-1]
         shrs = enshare[1:]
         if enshare:
-            import sqlite3
+            assert sqlite3  # type: ignore  # !rm
 
-            zsd = {"d2d": True, "tcolor": self.args.tcolor}
-            shv = VFS(self.log_func, "", shr, shr, AXS(), zsd)
+            shv = VFS(self.log_func, "", shr, shr, AXS(), self.vf0())
 
             db_path = self.args.shr_db
             db = sqlite3.connect(db_path)
@@ -2036,8 +2119,11 @@ class AuthSrv(object):
             elif self.args.re_maxage:
                 vol.flags["scan"] = self.args.re_maxage
 
+        self.args.have_unlistc = False
+
         all_mte = {}
         errors = False
+        free_umask = False
         for vol in vfs.all_nodes.values():
             if (self.args.e2ds and vol.axs.uwrite) or self.args.e2dsa:
                 vol.flags["e2ds"] = True
@@ -2094,9 +2180,33 @@ class AuthSrv(object):
                     t = 'volume "/%s" has invalid %stry [%s]'
                     raise Exception(t % (vol.vpath, k, vol.flags.get(k + "try")))
 
+            for k in ("chmod_d", "chmod_f"):
+                is_d = k == "chmod_d"
+                zs = vol.flags.get(k, "")
+                if not zs and is_d:
+                    zs = "755"
+                if not zs:
+                    vol.flags.pop(k, None)
+                    continue
+                if not re.match("^[0-7]{3}$", zs):
+                    t = "config-option '%s' must be a three-digit octal value such as [755] or [644] but the value was [%s]"
+                    t = t % (k, zs)
+                    self.log(t, 1)
+                    raise Exception(t)
+                zi = int(zs, 8)
+                vol.flags[k] = zi
+                if (is_d and zi != 0o755) or not is_d:
+                    free_umask = True
+
+            if vol.lim:
+                vol.lim.chmod_d = vol.flags["chmod_d"]
+
             if vol.flags.get("og"):
                 self.args.uqe = True
 
+            if "unlistcr" in vol.flags or "unlistcw" in vol.flags:
+                self.args.have_unlistc = True
+
             zs = str(vol.flags.get("tcolor", "")).lstrip("#")
             if len(zs) == 3:  # fc5 => ffcc55
                 vol.flags["tcolor"] = "".join([x * 2 for x in zs])
@@ -2174,6 +2284,8 @@ class AuthSrv(object):
                 t = "WARNING: volume [/%s]: invalid value specified for ext-th: %s"
                 self.log(t % (vol.vpath, etv), 3)
 
+            vol.check_landmarks()
+
             # d2d drops all database features for a volume
             for grp, rm in [["d2d", "e2d"], ["d2t", "e2t"], ["d2d", "e2v"]]:
                 if not vol.flags.get(grp, False):
@@ -2320,6 +2432,10 @@ class AuthSrv(object):
         if errors:
             sys.exit(1)
 
+        setattr(self.args, "free_umask", free_umask)
+        if free_umask:
+            os.umask(0)
+
         vfs.bubble_flags()
 
         have_e2d = False
@@ -2522,6 +2638,28 @@ class AuthSrv(object):
                 shn.shr_src = (s_vfs, s_rem)
                 shn.realpath = s_vfs.canonical(s_rem)
 
+                # root.all_aps doesn't include any shares, so make a copy where the
+                # share appears in all abspaths it can provide (for example for chk_ap)
+                ap = shn.realpath
+                if not ap.endswith(os.sep):
+                    ap += os.sep
+                shn.shr_all_aps = [(x, y[:]) for x, y in vfs.all_aps]
+                exact = False
+                for ap2, vns in shn.shr_all_aps:
+                    if ap == ap2:
+                        exact = True
+                    if ap2.startswith(ap):
+                        try:
+                            vp2 = vjoin(s_rem, ap2[len(ap) :])
+                            vn2, _ = s_vfs.get(vp2, "*", False, False)
+                            if vn2 == s_vfs or vn2.dbv == s_vfs:
+                                vns.append(shn)
+                        except:
+                            pass
+                if not exact:
+                    shn.shr_all_aps.append((ap, [shn]))
+                shn.shr_all_aps.sort(key=lambda x: len(x[0]), reverse=True)
+
                 if self.args.shr_v:
                     t = "mapped %s share [%s] by [%s] => [%s] => [%s]"
                     self.log(t % (s_pr, s_k, s_un, s_vp, shn.realpath))
@@ -2536,7 +2674,7 @@ class AuthSrv(object):
                     continue  # also fine
                 for zs in svn.nodes.keys():
                     # hide subvolume
-                    vn.nodes[zs] = VFS(self.log_func, "", "", "", AXS(), {})
+                    vn.nodes[zs] = VFS(self.log_func, "", "", "", AXS(), self.vf0())
 
             cur2.close()
             cur.close()
@@ -2581,6 +2719,7 @@ class AuthSrv(object):
                 "def_hcols": list(vf.get("mth") or []),
                 "unlist0": vf.get("unlist") or "",
                 "see_dots": self.args.see_dots,
+                "dqdel": self.args.qdel,
                 "dgrid": "grid" in vf,
                 "dgsel": "gsel" in vf,
                 "dnsort": "nsort" in vf,
@@ -2621,6 +2760,43 @@ class AuthSrv(object):
             zs = str(vol.flags.get("tcolor") or self.args.tcolor)
             vol.flags["tcolor"] = zs.lstrip("#")
 
+    def load_idp_db(self, quiet=False) -> None:
+        # mutex me
+        level = self.args.idp_store
+        if level < 2 or not self.args.idp_h_usr:
+            return
+
+        assert sqlite3  # type: ignore  # !rm
+
+        db = sqlite3.connect(self.args.idp_db)
+        cur = db.cursor()
+        from_cache = cur.execute("select un, gs from us").fetchall()
+        cur.close()
+        db.close()
+
+        self.idp_accs.clear()
+        self.idp_usr_gh.clear()
+
+        gsep = self.args.idp_gsep
+        n = []
+        for uname, gname in from_cache:
+            if level < 3:
+                if uname in self.idp_accs:
+                    continue
+                gname = ""
+            gnames = [x.strip() for x in gsep.split(gname)]
+            gnames.sort()
+
+            # self.idp_usr_gh[uname] = gname
+            self.idp_accs[uname] = gnames
+            n.append(uname)
+
+        if n and not quiet:
+            t = ", ".join(n[:9])
+            if len(n) > 9:
+                t += "..."
+            self.log("found %d IdP users in db (%s)" % (len(n), t))
+
     def load_sessions(self, quiet=False) -> None:
         # mutex me
         if self.args.no_ses:
@@ -2628,7 +2804,7 @@ class AuthSrv(object):
             self.sesa = {}
             return
 
-        import sqlite3
+        assert sqlite3  # type: ignore  # !rm
 
         ases = {}
         blen = (self.args.ses_len // 4) * 4  # 3 bytes in 4 chars
@@ -2675,7 +2851,7 @@ class AuthSrv(object):
         if self.args.no_ses:
             return
 
-        import sqlite3
+        assert sqlite3  # type: ignore  # !rm
 
         db = sqlite3.connect(self.args.ses_db)
         cur = db.cursor()

copyparty/bos/bos.py

@@ -25,14 +25,26 @@ def listdir(p: str = ".") -> list[str]:
 
 
 def makedirs(name: str, mode: int = 0o755, exist_ok: bool = True) -> bool:
+    # os.makedirs does 777 for all but leaf; this does mode on all
+    todo = []
     bname = fsenc(name)
-    try:
-        os.makedirs(bname, mode)
-        return True
-    except:
-        if not exist_ok or not os.path.isdir(bname):
-            raise
+    while bname:
+        if os.path.isdir(bname):
+            break
+        todo.append(bname)
+        bname = os.path.dirname(bname)
+    if not todo:
+        if not exist_ok:
+            os.mkdir(bname)  # to throw
         return False
+    for zb in todo[::-1]:
+        try:
+            os.mkdir(zb, mode)
+        except:
+            if os.path.isdir(zb):
+                continue
+            raise
+    return True
 
 
 def mkdir(p: str, mode: int = 0o755) -> None:

copyparty/cfg.py

@@ -78,6 +78,8 @@ def vf_vmap() -> dict[str, str]:
     }
     for k in (
         "bup_ck",
+        "chmod_d",
+        "chmod_f",
         "dbd",
         "forget_ip",
         "hsortn",
@@ -169,6 +171,8 @@ flagcats = {
         "safededup": "verify on-disk data before using it for dedup",
         "noclone": "take dupe data from clients, even if available on HDD",
         "nodupe": "rejects existing files (instead of linking/cloning them)",
+        "chmod_d=755": "unix-permission for new dirs/folders",
+        "chmod_f=644": "unix-permission for new files",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
         "nosparse": "deny use of sparse files, mainly for slow storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
@@ -218,6 +222,7 @@ flagcats = {
         "d2d": "disables all database stuff, overrides -e2*",
         "hist=/tmp/cdb": "puts thumbnails and indexes at that location",
         "dbpath=/tmp/cdb": "puts indexes at that location",
+        "landmark=foo": "disable db if file foo doesn't exist",
         "scan=60": "scan for new files every 60sec, same as --re-maxage",
         "nohash=\\.iso$": "skips hashing file contents if path matches *.iso",
         "noidx=\\.iso$": "fully ignores the contents at paths matching *.iso",
@@ -280,6 +285,8 @@ flagcats = {
         "nodirsz": "don't show total folder size",
         "robots": "allows indexing by search engines (default)",
         "norobots": "kindly asks search engines to leave",
+        "unlistcr": "don't list read-access in controlpanel",
+        "unlistcw": "don't list write-access in controlpanel",
         "no_sb_md": "disable js sandbox for markdown files",
         "no_sb_lg": "disable js sandbox for prologue/epilogue",
         "sb_md": "enable js sandbox for markdown files (default)",

copyparty/ftpd.py

@@ -229,7 +229,7 @@ class FtpFs(AbstractedFS):
         r = "r" in mode
         w = "w" in mode or "a" in mode or "+" in mode
 
-        ap = self.rv2a(filename, r, w)[0]
+        ap, vfs, _ = self.rv2a(filename, r, w)
         self.validpath(ap)
         if w:
             try:
@@ -261,7 +261,11 @@ class FtpFs(AbstractedFS):
 
             wunlink(self.log, ap, VF_CAREFUL)
 
-        return open(fsenc(ap), mode, self.args.iobuf)
+        ret = open(fsenc(ap), mode, self.args.iobuf)
+        if w and "chmod_f" in vfs.flags:
+            os.fchmod(ret.fileno(), vfs.flags["chmod_f"])
+
+        return ret
 
     def chdir(self, path: str) -> None:
         nwd = join(self.cwd, path)
@@ -292,8 +296,9 @@ class FtpFs(AbstractedFS):
         ) = avfs.can_access("", self.h.uname)
 
     def mkdir(self, path: str) -> None:
-        ap = self.rv2a(path, w=True)[0]
-        bos.makedirs(ap)  # filezilla expects this
+        ap, vfs, _ = self.rv2a(path, w=True)
+        chmod = vfs.flags["chmod_d"]
+        bos.makedirs(ap, chmod)  # filezilla expects this
 
     def listdir(self, path: str) -> list[str]:
         vpath = join(self.cwd, path)

copyparty/httpcli.py

@@ -45,6 +45,7 @@ from .util import (
     APPLESAN_RE,
     BITNESS,
     DAV_ALLPROPS,
+    E_SCK_WR,
     FN_EMB,
     HAVE_SQLITE3,
     HTTPCODE,
@@ -1299,6 +1300,9 @@ class HttpCli(object):
             if "ru" in self.uparam:
                 return self.tx_rups()
 
+            if "idp" in self.uparam:
+                return self.tx_idp()
+
         if "h" in self.uparam:
             return self.tx_mounts()
 
@@ -1371,12 +1375,13 @@ class HttpCli(object):
         title = self.uparam.get("title") or self.vpath.split("/")[-1]
         etitle = html_escape(title, True, True)
 
-        baseurl = "%s://%s%s" % (
+        baseurl = "%s://%s/" % (
             "https" if self.is_https else "http",
             self.host,
-            self.args.SRS,
         )
-        feed = "%s%s" % (baseurl, self.req[1:])
+        feed = baseurl + self.req[1:]
+        if self.is_vproxied:
+            baseurl += self.args.RS
         efeed = html_escape(feed, True, True)
         edirlink = efeed.split("?")[0] + q_pw
 
@@ -1389,7 +1394,7 @@ class HttpCli(object):
 \t\t<title>%s</title>
 \t\t<description></description>
 \t\t<link>%s</link>
-\t\t<generator>copyparty-1</generator>
+\t\t<generator>copyparty-2</generator>
 """
             % (efeed, etitle, edirlink)
         ]
@@ -2063,7 +2068,7 @@ class HttpCli(object):
             fdir, fn = os.path.split(fdir)
             rem, _ = vsplit(rem)
 
-        bos.makedirs(fdir)
+        bos.makedirs(fdir, vfs.flags["chmod_d"])
 
         open_ka: dict[str, Any] = {"fun": open}
         open_a = ["wb", self.args.iobuf]
@@ -2122,6 +2127,8 @@ class HttpCli(object):
             fn = vfs.flags["put_name2"].format(now=time.time(), cip=self.dip())
 
         params = {"suffix": suffix, "fdir": fdir}
+        if "chmod_f" in vfs.flags:
+            params["chmod"] = vfs.flags["chmod_f"]
         if self.args.nw:
             params = {}
             fn = os.devnull
@@ -2170,7 +2177,7 @@ class HttpCli(object):
                     if self.args.nw:
                         fn = os.devnull
                     else:
-                        bos.makedirs(fdir)
+                        bos.makedirs(fdir, vfs.flags["chmod_d"])
                         path = os.path.join(fdir, fn)
                         if not nameless:
                             self.vpath = vjoin(self.vpath, fn)
@@ -2302,7 +2309,7 @@ class HttpCli(object):
                     if self.args.hook_v:
                         log_reloc(self.log, hr["reloc"], x, path, vp, fn, vfs, rem)
                     fdir, self.vpath, fn, (vfs, rem) = x
-                    bos.makedirs(fdir)
+                    bos.makedirs(fdir, vfs.flags["chmod_d"])
                     path2 = os.path.join(fdir, fn)
                     atomic_move(self.log, path, path2, vfs.flags)
                     path = path2
@@ -2588,7 +2595,7 @@ class HttpCli(object):
             dst = vfs.canonical(rem)
             try:
                 if not bos.path.isdir(dst):
-                    bos.makedirs(dst)
+                    bos.makedirs(dst, vfs.flags["chmod_d"])
             except OSError as ex:
                 self.log("makedirs failed %r" % (dst,))
                 if not bos.path.isdir(dst):
@@ -3023,7 +3030,7 @@ class HttpCli(object):
                 raise Pebkac(405, 'folder "/%s" already exists' % (vpath,))
 
             try:
-                bos.makedirs(fn)
+                bos.makedirs(fn, vfs.flags["chmod_d"])
             except OSError as ex:
                 if ex.errno == errno.EACCES:
                     raise Pebkac(500, "the server OS denied write-access")
@@ -3065,6 +3072,8 @@ class HttpCli(object):
 
             with open(fsenc(fn), "wb") as f:
                 f.write(b"`GRUNNUR`\n")
+                if "chmod_f" in vfs.flags:
+                    os.fchmod(f.fileno(), vfs.flags["chmod_f"])
 
         vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
         self.redirect(vpath, "?edit")
@@ -3138,7 +3147,7 @@ class HttpCli(object):
             )
             upload_vpath = "{}/{}".format(vfs.vpath, rem).strip("/")
             if not nullwrite:
-                bos.makedirs(fdir_base)
+                bos.makedirs(fdir_base, vfs.flags["chmod_d"])
 
         rnd, lifetime, xbu, xau = self.upload_flags(vfs)
         zs = self.uparam.get("want") or self.headers.get("accept") or ""
@@ -3233,8 +3242,11 @@ class HttpCli(object):
                             else:
                                 open_args["fdir"] = fdir
 
+                if "chmod_f" in vfs.flags:
+                    open_args["chmod"] = vfs.flags["chmod_f"]
+
                 if p_file and not nullwrite:
-                    bos.makedirs(fdir)
+                    bos.makedirs(fdir, vfs.flags["chmod_d"])
 
                     # reserve destination filename
                     f, fname = ren_open(fname, "wb", fdir=fdir, suffix=suffix)
@@ -3338,7 +3350,7 @@ class HttpCli(object):
                                 if nullwrite:
                                     fdir = ap2 = ""
                                 else:
-                                    bos.makedirs(fdir)
+                                    bos.makedirs(fdir, vfs.flags["chmod_d"])
                                     atomic_move(self.log, abspath, ap2, vfs.flags)
                                 abspath = ap2
                         sz = bos.path.getsize(abspath)
@@ -3459,6 +3471,8 @@ class HttpCli(object):
                     ft = "{}:{}".format(self.ip, self.addr[1])
                     ft = "{}\n{}\n{}\n".format(ft, msg.rstrip(), errmsg)
                     f.write(ft.encode("utf-8"))
+                    if "chmod_f" in vfs.flags:
+                        os.fchmod(f.fileno(), vfs.flags["chmod_f"])
             except Exception as ex:
                 suf = "\nfailed to write the upload report: {}".format(ex)
 
@@ -3509,7 +3523,7 @@ class HttpCli(object):
         lim = vfs.get_dbv(rem)[0].lim
         if lim:
             fp, rp = lim.all(self.ip, rp, clen, vfs.realpath, fp, self.conn.hsrv.broker)
-            bos.makedirs(fp)
+            bos.makedirs(fp, vfs.flags["chmod_d"])
 
         fp = os.path.join(fp, fn)
         rem = "{}/{}".format(rp, fn).strip("/")
@@ -3577,13 +3591,15 @@ class HttpCli(object):
                 zs = ub64enc(zb).decode("ascii")[:24].lower()
                 dp = "%s/md/%s/%s/%s" % (dbv.histpath, zs[:2], zs[2:4], zs)
                 self.log("moving old version to %s/%s" % (dp, mfile2))
-                if bos.makedirs(dp):
+                if bos.makedirs(dp, vfs.flags["chmod_d"]):
                     with open(os.path.join(dp, "dir.txt"), "wb") as f:
                         f.write(afsenc(vrd))
+                        if "chmod_f" in vfs.flags:
+                            os.fchmod(f.fileno(), vfs.flags["chmod_f"])
             elif hist_cfg == "s":
                 dp = os.path.join(mdir, ".hist")
                 try:
-                    bos.mkdir(dp)
+                    bos.mkdir(dp, vfs.flags["chmod_d"])
                     hidedir(dp)
                 except:
                     pass
@@ -3622,6 +3638,8 @@ class HttpCli(object):
             wunlink(self.log, fp, vfs.flags)
 
         with open(fsenc(fp), "wb", self.args.iobuf) as f:
+            if "chmod_f" in vfs.flags:
+                os.fchmod(f.fileno(), vfs.flags["chmod_f"])
             sz, sha512, _ = hashcopy(p_data, f, None, 0, self.args.s_wr_slp)
 
         if lim:
@@ -4361,7 +4379,7 @@ class HttpCli(object):
                             self.log("file deleted; disconnecting")
                             break
         except IOError as ex:
-            if ex.errno not in (errno.EPIPE, errno.ESHUTDOWN, errno.EBADFD):
+            if ex.errno not in E_SCK_WR:
                 raise
         finally:
             if f:
@@ -4946,6 +4964,11 @@ class HttpCli(object):
             fn = html_escape(fn) if fn else self.conn.hsrv.iiam
             dls.append((perc, hsent, spd, eta, idle, usr, erd, rds, fn))
 
+        if self.args.have_unlistc:
+            allvols = self.asrv.vfs.all_vols
+            rvol = [x for x in rvol if "unlistcr" not in allvols[x[1:-1]].flags]
+            wvol = [x for x in wvol if "unlistcw" not in allvols[x[1:-1]].flags]
+
         fmt = self.uparam.get("ls", "")
         if not fmt and (self.ua.startswith("curl/") or self.ua.startswith("fetch")):
             fmt = "v"
@@ -5100,15 +5123,24 @@ class HttpCli(object):
         return ""  # unhandled / fallthrough
 
     def scanvol(self) -> bool:
-        if not self.can_admin:
-            raise Pebkac(403, "'scanvol' not allowed for user " + self.uname)
-
         if self.args.no_rescan:
             raise Pebkac(403, "the rescan feature is disabled in server config")
 
-        vn, _ = self.asrv.vfs.get(self.vpath, self.uname, True, True)
+        vpaths = self.uparam["scan"].split(",/")
+        if vpaths == [""]:
+            vpaths = [self.vpath]
 
-        args = [self.asrv.vfs.all_vols, [vn.vpath], False, True]
+        vols = []
+        for vpath in vpaths:
+            vn, _ = self.asrv.vfs.get(vpath, self.uname, True, True)
+            vols.append(vn.vpath)
+            if self.uname not in vn.axs.uadmin:
+                self.log("rejected scanning [%s] => [%s];" % (vpath, vn.vpath), 3)
+                raise Pebkac(403, "'scanvol' not allowed for user " + self.uname)
+
+        self.log("trying to rescan %d volumes: %r" % (len(vols), vols))
+
+        args = [self.asrv.vfs.all_vols, vols, False, True]
 
         x = self.conn.hsrv.broker.ask("up2k.rescan", *args)
         err = x.get()
@@ -5528,6 +5560,32 @@ class HttpCli(object):
         self.reply(html.encode("utf-8"), status=200)
         return True
 
+    def tx_idp(self) -> bool:
+        if self.uname.lower() not in self.args.idp_adm_set:
+            raise Pebkac(403, "'idp' not allowed for user " + self.uname)
+
+        cmd = self.uparam["idp"]
+        if cmd.startswith("rm="):
+            import sqlite3
+
+            db = sqlite3.connect(self.args.idp_db)
+            db.execute("delete from us where un=?", (cmd[3:],))
+            db.commit()
+            db.close()
+
+            self.conn.hsrv.broker.ask("reload", False, True).get()
+
+            self.redirect("", "?idp")
+            return True
+
+        rows = [
+            [k, "[%s]" % ("], [".join(v))]
+            for k, v in sorted(self.asrv.idp_accs.items())
+        ]
+        html = self.j2s("idp", this=self, rows=rows, now=int(time.time()))
+        self.reply(html.encode("utf-8"), status=200)
+        return True
+
     def tx_shares(self) -> bool:
         if self.uname == "*":
             self.loud_reply("you're not logged in")
@@ -5599,10 +5657,10 @@ class HttpCli(object):
 
         cur.connection.commit()
         if reload:
-            self.conn.hsrv.broker.ask("reload", False, False).get()
+            self.conn.hsrv.broker.ask("reload", False, True).get()
             self.conn.hsrv.broker.ask("up2k.wake_rescanner").get()
 
-        self.redirect(self.args.SRS + "?shares")
+        self.redirect("", "?shares")
         return True
 
     def handle_share(self, req: dict[str, str]) -> bool:
@@ -5691,7 +5749,7 @@ class HttpCli(object):
             cur.execute(q, (skey, fn))
 
         cur.connection.commit()
-        self.conn.hsrv.broker.ask("reload", False, False).get()
+        self.conn.hsrv.broker.ask("reload", False, True).get()
         self.conn.hsrv.broker.ask("up2k.wake_rescanner").get()
 
         fn = quotep(fns[0]) if len(fns) == 1 else ""

copyparty/httpconn.py

@@ -224,3 +224,6 @@ class HttpConn(object):
             if self.u2idx:
                 self.hsrv.put_u2idx(str(self.addr), self.u2idx)
                 self.u2idx = None
+
+            if self.rproxy:
+                self.set_rproxy()

copyparty/httpsrv.py

@@ -175,6 +175,7 @@ class HttpSrv(object):
             "browser",
             "browser2",
             "cf",
+            "idp",
             "md",
             "mde",
             "msg",
@@ -313,6 +314,8 @@ class HttpSrv(object):
 
         Daemon(self.broker.say, "sig-hsrv-up1", ("cb_httpsrv_up",))
 
+        saddr = ("", 0)  # fwd-decl for `except TypeError as ex:`
+
         while not self.stopping:
             if self.args.log_conn:
                 self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="90")
@@ -394,6 +397,19 @@ class HttpSrv(object):
                 self.log(self.name, "accept({}): {}".format(fno, ex), c=6)
                 time.sleep(0.02)
                 continue
+            except TypeError as ex:
+                # on macOS, accept() may return a None saddr if blocked by LittleSnitch;
+                # unicode(saddr[0]) ==> TypeError: 'NoneType' object is not subscriptable
+                if tcp and not saddr:
+                    t = "accept(%s): failed to accept connection from client due to firewall or network issue"
+                    self.log(self.name, t % (fno,), c=3)
+                    try:
+                        sck.close()  # type: ignore
+                    except:
+                        pass
+                    time.sleep(0.02)
+                    continue
+                raise
 
             if self.args.log_conn:
                 t = "|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(

copyparty/smbd.py

@@ -320,7 +320,7 @@ class SMB(object):
 
         self.hub.up2k.handle_mv(uname, "1.7.6.2", vp1, vp2)
         try:
-            bos.makedirs(ap2)
+            bos.makedirs(ap2, vfs2.flags["chmod_d"])
         except:
             pass
 
@@ -334,7 +334,7 @@ class SMB(object):
             t = "blocked mkdir (no-write-acc %s): /%s @%s"
             yeet(t % (vfs.axs.uwrite, vpath, uname))
 
-        return bos.mkdir(ap)
+        return bos.mkdir(ap, vfs.flags["chmod_d"])
 
     def _stat(self, vpath: str, *a: Any, **ka: Any) -> os.stat_result:
         try:

copyparty/svchub.py

@@ -27,6 +27,7 @@ if True:  # pylint: disable=using-constant-test
 
 from .__init__ import ANYWIN, EXE, MACOS, PY2, TYPE_CHECKING, E, EnvParams, unicode
 from .authsrv import BAD_CFG, AuthSrv
+from .bos import bos
 from .cert import ensure_cert
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, HAVE_MUTAGEN
 from .pwhash import HAVE_ARGON2
@@ -88,6 +89,7 @@ if PY2:
     range = xrange  # type: ignore
 
 
+VER_IDP_DB = 1
 VER_SESSION_DB = 1
 VER_SHARES_DB = 2
 
@@ -258,11 +260,15 @@ class SvcHub(object):
                 self.log("root", "effective %s is %s" % (zs, getattr(args, zs)))
 
         if args.ah_cli or args.ah_gen:
+            args.idp_store = 0
             args.no_ses = True
             args.shr = ""
 
+        if args.idp_store and args.idp_h_usr:
+            self.setup_db("idp")
+
         if not self.args.no_ses:
-            self.setup_session_db()
+            self.setup_db("ses")
 
         args.shr1 = ""
         if args.shr:
@@ -421,26 +427,58 @@ class SvcHub(object):
             except:
                 pass
 
-    def setup_session_db(self) -> None:
-        if not HAVE_SQLITE3:
+    def _db_onfail_ses(self) -> None:
         self.args.no_ses = True
-            t = "WARNING: sqlite3 not available; disabling sessions, will use plaintext passwords in cookies"
-            self.log("root", t, 3)
+
+    def _db_onfail_idp(self) -> None:
+        self.args.idp_store = 0
+
+    def setup_db(self, which: str) -> None:
+        """
+        the "non-mission-critical" databases; if something looks broken then just nuke it
+        """
+        if which == "ses":
+            native_ver = VER_SESSION_DB
+            db_path = self.args.ses_db
+            desc = "sessions-db"
+            pathopt = "ses-db"
+            sanchk_q = "select count(*) from us"
+            createfun = self._create_session_db
+            failfun = self._db_onfail_ses
+        elif which == "idp":
+            native_ver = VER_IDP_DB
+            db_path = self.args.idp_db
+            desc = "idp-db"
+            pathopt = "idp-db"
+            sanchk_q = "select count(*) from us"
+            createfun = self._create_idp_db
+            failfun = self._db_onfail_idp
+        else:
+            raise Exception("unknown cachetype")
+
+        if not db_path.endswith(".db"):
+            zs = "config option --%s (the %s) was configured to [%s] which is invalid; must be a filepath ending with .db"
+            self.log("root", zs % (pathopt, desc, db_path), 1)
+            raise Exception(BAD_CFG)
+
+        if not HAVE_SQLITE3:
+            failfun()
+            if which == "ses":
+                zs = "disabling sessions, will use plaintext passwords in cookies"
+            elif which == "idp":
+                zs = "disabling idp-db, will be unable to remember IdP-volumes after a restart"
+            self.log("root", "WARNING: sqlite3 not available; %s" % (zs,), 3)
             return
 
         assert sqlite3  # type: ignore  # !rm
 
-        # policy:
-        # the sessions-db is whatever, if something looks broken then just nuke it
-
-        db_path = self.args.ses_db
         db_lock = db_path + ".lock"
         try:
             create = not os.path.getsize(db_path)
         except:
             create = True
         zs = "creating new" if create else "opening"
-        self.log("root", "%s sessions-db %s" % (zs, db_path))
+        self.log("root", "%s %s %s" % (zs, desc, db_path))
 
         for tries in range(2):
             sver = 0
@@ -450,17 +488,19 @@ class SvcHub(object):
                 try:
                     zs = "select v from kv where k='sver'"
                     sver = cur.execute(zs).fetchall()[0][0]
-                    if sver > VER_SESSION_DB:
-                        zs = "this version of copyparty only understands session-db v%d and older; the db is v%d"
-                        raise Exception(zs % (VER_SESSION_DB, sver))
+                    if sver > native_ver:
+                        zs = "this version of copyparty only understands %s v%d and older; the db is v%d"
+                        raise Exception(zs % (desc, native_ver, sver))
 
-                    cur.execute("select count(*) from us").fetchone()
+                    cur.execute(sanchk_q).fetchone()
                 except:
                     if sver:
                         raise
-                    sver = 1
-                    self._create_session_db(cur)
-                err = self._verify_session_db(cur, sver, db_path)
+                    sver = createfun(cur)
+
+                err = self._verify_db(
+                    cur, which, pathopt, db_path, desc, sver, native_ver
+                )
                 if err:
                     tries = 99
                     self.args.no_ses = True
@@ -468,10 +508,10 @@ class SvcHub(object):
                 break
 
             except Exception as ex:
-                if tries or sver > VER_SESSION_DB:
+                if tries or sver > native_ver:
                     raise
-                t = "sessions-db is unusable; deleting and recreating: %r"
-                self.log("root", t % (ex,), 3)
+                t = "%s is unusable; deleting and recreating: %r"
+                self.log("root", t % (desc, ex), 3)
                 try:
                     cur.close()  # type: ignore
                 except:
@@ -486,7 +526,7 @@ class SvcHub(object):
                     pass
                 os.unlink(db_path)
 
-    def _create_session_db(self, cur: "sqlite3.Cursor") -> None:
+    def _create_session_db(self, cur: "sqlite3.Cursor") -> int:
         sch = [
             r"create table kv (k text, v int)",
             r"create table us (un text, si text, t0 int)",
@@ -499,8 +539,31 @@ class SvcHub(object):
         for cmd in sch:
             cur.execute(cmd)
         self.log("root", "created new sessions-db")
+        return 1
 
-    def _verify_session_db(self, cur: "sqlite3.Cursor", sver: int, db_path: str) -> str:
+    def _create_idp_db(self, cur: "sqlite3.Cursor") -> int:
+        sch = [
+            r"create table kv (k text, v int)",
+            r"create table us (un text, gs text)",
+            # username, groups
+            r"create index us_un on us(un)",
+            r"insert into kv values ('sver', 1)",
+        ]
+        for cmd in sch:
+            cur.execute(cmd)
+        self.log("root", "created new idp-db")
+        return 1
+
+    def _verify_db(
+        self,
+        cur: "sqlite3.Cursor",
+        which: str,
+        pathopt: str,
+        db_path: str,
+        desc: str,
+        sver: int,
+        native_ver: int,
+    ) -> str:
         # ensure writable (maybe owned by other user)
         db = cur.connection
 
@@ -512,9 +575,16 @@ class SvcHub(object):
         except:
             owner = 0
 
+        if which == "ses":
+            cons = "Will now disable sessions and instead use plaintext passwords in cookies."
+        elif which == "idp":
+            cons = "Each IdP-volume will not become available until its associated user sends their first request."
+        else:
+            raise Exception()
+
         if not lock_file(db_path + ".lock"):
-            t = "the sessions-db [%s] is already in use by another copyparty instance (pid:%d). This is not supported; please provide another database with --ses-db or give this copyparty-instance its entirely separate config-folder by setting another path in the XDG_CONFIG_HOME env-var. You can also disable this safeguard by setting env-var PRTY_NO_DB_LOCK=1. Will now disable sessions and instead use plaintext passwords in cookies."
-            return t % (db_path, owner)
+            t = "the %s [%s] is already in use by another copyparty instance (pid:%d). This is not supported; please provide another database with --%s or give this copyparty-instance its entirely separate config-folder by setting another path in the XDG_CONFIG_HOME env-var. You can also disable this safeguard by setting env-var PRTY_NO_DB_LOCK=1. %s"
+            return t % (desc, db_path, owner, pathopt, cons)
 
         vars = (("pid", os.getpid()), ("ts", int(time.time() * 1000)))
         if owner:
@@ -526,9 +596,9 @@ class SvcHub(object):
             for k, v in vars:
                 cur.execute("insert into kv values(?, ?)", (k, v))
 
-        if sver < VER_SESSION_DB:
+        if sver < native_ver:
             cur.execute("delete from kv where k='sver'")
-            cur.execute("insert into kv values('sver',?)", (VER_SESSION_DB,))
+            cur.execute("insert into kv values('sver',?)", (native_ver,))
 
         db.commit()
         cur.close()
@@ -880,6 +950,12 @@ class SvcHub(object):
                 vs = os.path.expandvars(os.path.expanduser(vs))
                 setattr(al, k, vs)
 
+        for k in "idp_adm".split(" "):
+            vs = getattr(al, k)
+            vsa = [x.strip() for x in vs.split(",")]
+            vsa = [x.lower() for x in vsa if x]
+            setattr(al, k + "_set", set(vsa))
+
         zs = "dav_ua1 sus_urls nonsus_urls ua_nodoc ua_nozip"
         for k in zs.split(" "):
             vs = getattr(al, k)
@@ -1043,7 +1119,7 @@ class SvcHub(object):
 
         fn = sel_fn
         try:
-            os.makedirs(os.path.dirname(fn))
+            bos.makedirs(os.path.dirname(fn))
         except:
             pass
 
@@ -1060,6 +1136,9 @@ class SvcHub(object):
 
             lh = codecs.open(fn, "w", encoding="utf-8", errors="replace")
 
+        if getattr(self.args, "free_umask", False):
+            os.fchmod(lh.fileno(), 0o644)
+
         argv = [pybin] + self.argv
         if hasattr(shlex, "quote"):
             argv = [shlex.quote(x) for x in argv]

copyparty/tcpsrv.py

@@ -282,7 +282,7 @@ class TcpSrv(object):
         except:
             pass  # will create another ipv4 socket instead
 
-        if not ANYWIN and self.args.freebind:
+        if getattr(self.args, "freebind", False):
             srv.setsockopt(socket.SOL_IP, socket.IP_FREEBIND, 1)
 
         try:

copyparty/tftpd.py

@@ -387,14 +387,18 @@ class Tftpd(object):
         if not a:
             a = (self.args.iobuf,)
 
-        return open(ap, mode, *a, **ka)
+        ret = open(ap, mode, *a, **ka)
+        if wr and "chmod_f" in vfs.flags:
+            os.fchmod(ret.fileno(), vfs.flags["chmod_f"])
+
+        return ret
 
     def _mkdir(self, vpath: str, *a) -> None:
         vfs, _, ap = self._v2a("mkdir", vpath, [False, True])
         if "*" not in vfs.axs.uwrite:
             yeet("blocked mkdir; folder not world-writable: /%s" % (vpath,))
 
-        return bos.mkdir(ap)
+        return bos.mkdir(ap, vfs.flags["chmod_d"])
 
     def _unlink(self, vpath: str) -> None:
         # return bos.unlink(self._v2a("stat", vpath, *a)[1])

copyparty/th_srv.py

@@ -96,6 +96,10 @@ try:
         if os.environ.get("PRTY_NO_PIL_AVIF"):
             raise Exception()
 
+        if ".avif" in Image.registered_extensions():
+            HAVE_AVIF = True
+            raise Exception()
+
         import pillow_avif  # noqa: F401  # pylint: disable=unused-import
 
         HAVE_AVIF = True
@@ -265,7 +269,8 @@ class ThumbSrv(object):
                 self.log("joined waiting room for %r" % (tpath,))
             except:
                 thdir = os.path.dirname(tpath)
-                bos.makedirs(os.path.join(thdir, "w"))
+                chmod = 0o700 if self.args.free_umask else 0o755
+                bos.makedirs(os.path.join(thdir, "w"), chmod)
 
                 inf_path = os.path.join(thdir, "dir.txt")
                 if not bos.path.exists(inf_path):
@@ -280,7 +285,7 @@ class ThumbSrv(object):
             vn = next((x for x in allvols if x.realpath == ptop), None)
             if not vn:
                 self.log("ptop %r not in %s" % (ptop, allvols), 3)
-                vn = self.asrv.vfs.all_aps[0][1]
+                vn = self.asrv.vfs.all_aps[0][1][0]
 
             self.q.put((abspath, tpath, fmt, vn))
             self.log("conv %r :%s \033[0m%r" % (tpath, fmt, abspath), 6)

copyparty/up2k.py

@@ -915,7 +915,8 @@ class Up2k(object):
             # only need to protect register_vpath but all in one go feels right
             for vol in vols:
                 try:
-                    bos.makedirs(vol.realpath)  # gonna happen at snap anyways
+                    # mkdir gonna happen at snap anyways;
+                    bos.makedirs(vol.realpath, vol.flags["chmod_d"])
                     dir_is_empty(self.log_func, not self.args.no_scandir, vol.realpath)
                 except Exception as ex:
                     self.volstate[vol.vpath] = "OFFLINE (cannot access folder)"
@@ -1141,6 +1142,20 @@ class Up2k(object):
                 del fl[k1]
             else:
                 fl[k1] = ",".join(x for x in fl[k1])
+
+        if fl["chmod_d"] == int(self.args.chmod_d, 8):
+            fl.pop("chmod_d")
+        try:
+            if fl["chmod_f"] == int(self.args.chmod_f or "-1", 8):
+                fl.pop("chmod_f")
+        except:
+            pass
+        for k in ("chmod_f", "chmod_d"):
+            try:
+                fl[k] = "%o" % (fl[k])
+            except:
+                pass
+
         a = [
             (ft if v is True else ff if v is False else fv).format(k, str(v))
             for k, v in fl.items()
@@ -1364,6 +1379,10 @@ class Up2k(object):
                     t = "volume /%s at [%s] is empty; will not be indexed as this could be due to an offline filesystem"
                     self.log(t % (vol.vpath, rtop), 6)
                     return True, False
+                if not vol.check_landmarks():
+                    t = "volume /%s at [%s] will not be indexed due to bad landmarks"
+                    self.log(t % (vol.vpath, rtop), 6)
+                    return True, False
 
                 n_add, _, _ = self._build_dir(
                     db,
@@ -3290,7 +3309,7 @@ class Up2k(object):
                         reg,
                         "up2k._get_volsize",
                     )
-                    bos.makedirs(ap2)
+                    bos.makedirs(ap2, vfs.flags["chmod_d"])
                     vfs.lim.nup(cj["addr"])
                     vfs.lim.bup(cj["addr"], cj["size"])
 
@@ -3397,11 +3416,11 @@ class Up2k(object):
                 self.log(t % (mts - mtc, mts, mtc, fp))
                 ow = False
 
+        ptop = job["ptop"]
+        vf = self.flags.get(ptop) or {}
         if ow:
             self.log("replacing existing file at %r" % (fp,))
             cur = None
-            ptop = job["ptop"]
-            vf = self.flags.get(ptop) or {}
             st = bos.stat(fp)
             try:
                 vrel = vjoin(job["prel"], fname)
@@ -3421,8 +3440,13 @@ class Up2k(object):
         else:
             dip = self.hub.iphash.s(ip)
 
-        suffix = "-%.6f-%s" % (ts, dip)
-        f, ret = ren_open(fname, "wb", fdir=fdir, suffix=suffix)
+        f, ret = ren_open(
+            fname,
+            "wb",
+            fdir=fdir,
+            suffix="-%.6f-%s" % (ts, dip),
+            chmod=vf.get("chmod_f", -1),
+        )
         f.close()
         return ret
 
@@ -4277,7 +4301,7 @@ class Up2k(object):
                 self.log(t, 1)
                 raise Pebkac(405, t)
 
-        bos.makedirs(os.path.dirname(dabs))
+        bos.makedirs(os.path.dirname(dabs), dvn.flags["chmod_d"])
 
         c1, w, ftime_, fsize_, ip, at = self._find_from_vpath(
             svn_dbv.realpath, srem_dbv
@@ -4453,7 +4477,7 @@ class Up2k(object):
                 vp = vjoin(dvp, rem)
                 try:
                     dvn, drem = self.vfs.get(vp, uname, False, True)
-                    bos.mkdir(dvn.canonical(drem))
+                    bos.mkdir(dvn.canonical(drem), dvn.flags["chmod_d"])
                 except:
                     pass
 
@@ -4523,7 +4547,7 @@ class Up2k(object):
 
         is_xvol = svn.realpath != dvn.realpath
 
-        bos.makedirs(os.path.dirname(dabs))
+        bos.makedirs(os.path.dirname(dabs), dvn.flags["chmod_d"])
 
         if is_dirlink:
             dlabs = absreal(sabs)
@@ -5030,8 +5054,13 @@ class Up2k(object):
         else:
             dip = self.hub.iphash.s(job["addr"])
 
-        suffix = "-%.6f-%s" % (job["t0"], dip)
-        f, job["tnam"] = ren_open(tnam, "wb", fdir=pdir, suffix=suffix)
+        f, job["tnam"] = ren_open(
+            tnam,
+            "wb",
+            fdir=pdir,
+            suffix="-%.6f-%s" % (job["t0"], dip),
+            chmod=vf.get("chmod_f", -1),
+        )
         try:
             abspath = djoin(pdir, job["tnam"])
             sprs = job["sprs"]

copyparty/util.py

@@ -105,6 +105,7 @@ def _ens(want: str) -> tuple[int, ...]:
 # WSAENOTSOCK - no longer a socket
 # EUNATCH - can't assign requested address (wifi down)
 E_SCK = _ens("ENOTCONN EUNATCH EBADF WSAENOTSOCK WSAECONNRESET")
+E_SCK_WR = _ens("EPIPE ESHUTDOWN EBADFD")
 E_ADDR_NOT_AVAIL = _ens("EADDRNOTAVAIL WSAEADDRNOTAVAIL")
 E_ADDR_IN_USE = _ens("EADDRINUSE WSAEADDRINUSE")
 E_ACCESS = _ens("EACCES WSAEACCES")
@@ -1584,6 +1585,7 @@ def ren_open(fname: str, *args: Any, **kwargs: Any) -> tuple[typing.IO[Any], str
     fun = kwargs.pop("fun", open)
     fdir = kwargs.pop("fdir", None)
     suffix = kwargs.pop("suffix", None)
+    chmod = kwargs.pop("chmod", -1)
 
     if fname == os.devnull:
         return fun(fname, *args, **kwargs), fname
@@ -1627,6 +1629,11 @@ def ren_open(fname: str, *args: Any, **kwargs: Any) -> tuple[typing.IO[Any], str
                 fp2 = os.path.join(fdir, fp2)
                 with open(fsenc(fp2), "wb") as f2:
                     f2.write(orig_name.encode("utf-8"))
+                    if chmod >= 0:
+                        os.fchmod(f2.fileno(), chmod)
+
+            if chmod >= 0:
+                os.fchmod(f.fileno(), chmod)
 
             return f, fname
 
@@ -1967,7 +1974,7 @@ def rand_name(fdir: str, fn: str, rnd: int) -> str:
     return fn
 
 
-def gen_filekey(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str:
+def _gen_filekey(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str:
     if alg == 1:
         zs = "%s %s %s %s" % (salt, fspath, fsize, inode)
     else:
@@ -1977,6 +1984,13 @@ def gen_filekey(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str
     return ub64enc(hashlib.sha512(zb).digest()).decode("ascii")
 
 
+def _gen_filekey_w(alg: int, salt: str, fspath: str, fsize: int, inode: int) -> str:
+    return _gen_filekey(alg, salt, fspath.replace("/", "\\"), fsize, inode)
+
+
+gen_filekey = _gen_filekey_w if ANYWIN else _gen_filekey
+
+
 def gen_filekey_dbg(
     alg: int,
     salt: str,
@@ -2400,11 +2414,11 @@ def pathmod(
 
         # try to map abspath to vpath
         np = np.replace("/", os.sep)
-        for vn_ap, vn in vfs.all_aps:
+        for vn_ap, vns in vfs.all_aps:
             if not np.startswith(vn_ap):
                 continue
             zs = np[len(vn_ap) :].replace(os.sep, "/")
-            nvp = vjoin(vn.vpath, zs)
+            nvp = vjoin(vns[0].vpath, zs)
             break
 
     if nvp == "\n":

copyparty/web/baguettebox.js

@@ -592,9 +592,7 @@ window.baguetteBox = (function () {
             preloadPrev(currentIndex);
         });
 
-        clmod(ebi('bbox-btns'), 'off');
-        clmod(btnPrev, 'off');
-        clmod(btnNext, 'off');
+        show_buttons(0);
 
         updateOffset();
         overlay.style.display = 'block';
@@ -776,6 +774,8 @@ window.baguetteBox = (function () {
         if (is_vid) {
             image.volume = clamp(fcfg_get('vol', dvol / 100), 0, 1);
             image.setAttribute('controls', 'controls');
+            image.setAttribute('playsinline', '1');
+            // ios ignores poster
             image.onended = vidEnd;
             image.onplay = function () { show_buttons(1); };
             image.onpause = function () { show_buttons(); };

copyparty/web/browser.css

@@ -4,6 +4,8 @@
 	--grid-sz: 10em;
 	--grid-ln: 3;
 	--nav-sz: 16em;
+	--sbw: 0.5em;
+	--sbh: 0.5em;
 
 	--fg: #ccc;
 	--fg-max: #fff;
@@ -1558,8 +1560,8 @@ html {
 	z-index: 1;
 	position: fixed;
 	background: var(--tree-bg);
-	left: -.98em;
-	width: calc(var(--nav-sz) - 0.5em);
+	left: -.96em;
+	width: calc(.3em + var(--nav-sz) - var(--sbw));
 	border-bottom: 1px solid var(--bg-u5);
 	overflow: hidden;
 }
@@ -2033,6 +2035,9 @@ a.btn,
 	font-family: 'scp', monospace, monospace;
 	font-family: var(--font-mono), 'scp', monospace, monospace;
 }
+#hkhelp b {
+	text-shadow: 1px 0 0 var(--fg), -1px 0 0 var(--fg), 0 -1px 0 var(--fg);
+}
 html.noscroll,
 html.noscroll .sbar {
 	scrollbar-width: none;
@@ -2194,18 +2199,25 @@ html.y #bbox-overlay figcaption a {
 	top: calc(50% - 30px);
 	width: 44px;
 	height: 60px;
+	transition: background-color .3s ease, color .3s ease, left .3s ease, right .3s ease;
+}
+#bbox-btns button {
+	transition: background-color .3s ease, color .3s ease;
+}
+#bbox-btns {
+	transition: top .3s ease;
 }
 .bbox-btn {
 	position: fixed;
 }
-.bbox-btn,
-#bbox-btns {
-	opacity: 1;
-	animation: opacity .2s infinite ease-in-out;
+#bbox-next.off {
+	right: -2.6em;
+}
+#bbox-prev.off {
+	left: -2.6em;
 }
-.bbox-btn.off,
 #bbox-btns.off {
-	opacity: 0;
+	top: -2.2em;
 }
 #bbox-overlay button {
 	cursor: pointer;
@@ -2216,8 +2228,6 @@ html.y #bbox-overlay figcaption a {
 	border-radius: 15%;
 	background: rgba(50, 50, 50, 0.5);
 	color: rgba(255,255,255,0.7);
-	transition: background-color .3s ease;
-	transition: color .3s ease;
 	font-size: 1.4em;
 	line-height: 1.4em;
 	vertical-align: top;
@@ -3066,7 +3076,8 @@ html.b .ntree a {
 	padding: .6em .2em;
 }
 html.b #treepar {
-	margin-left: .62em;
+	margin-left: .63em;
+	width: calc(.1em + var(--nav-sz) - var(--sbw));
 	border-bottom: .2em solid var(--f-h-b1);
 }
 html.b #wrap {
@@ -3261,4 +3272,9 @@ html.d #treepar {
 	.dropdesc>div>div {
 		transition: none;
 	}
+	#bbox-next,
+	#bbox-prev,
+	#bbox-btns {
+		transition: background-color .3s ease, color .3s ease;
+	}
 }

copyparty/web/browser.js

@@ -35,7 +35,7 @@ var Ls = {
 				"file-manager",
 				["G", "toggle list / grid view"],
 				["T", "toggle thumbnails / icons"],
-				["🡅 A/D", "thumbnail size"],
+				["⇧ A/D", "thumbnail size"],
 				["ctrl-K", "delete selected"],
 				["ctrl-X", "cut selection to clipboard"],
 				["ctrl-C", "copy selection to clipboard"],
@@ -45,9 +45,9 @@ var Ls = {
 
 				"file-list-sel",
 				["space", "toggle file selection"],
-				["🡑/🡓", "move selection cursor"],
-				["ctrl 🡑/🡓", "move cursor and viewport"],
-				["🡅 🡑/🡓", "select prev/next file"],
+				["↑/↓", "move selection cursor"],
+				["ctrl ↑/↓", "move cursor and viewport"],
+				["⇧ ↑/↓", "select prev/next file"],
 				["ctrl-A", "select all files / folders"],
 			], [
 				"navigation",
@@ -70,7 +70,7 @@ var Ls = {
 				["Home/End", "first/last pic"],
 				["F", "fullscreen"],
 				["R", "rotate clockwise"],
-				["🡅 R", "rotate ccw"],
+				["⇧ R", "rotate ccw"],
 				["S", "select pic"],
 				["Y", "download pic"],
 			], [
@@ -226,6 +226,7 @@ var Ls = {
 		"ct_csel": 'use CTRL and SHIFT for file selection in grid-view">sel',
 		"ct_ihop": 'when the image viewer is closed, scroll down to the last viewed file">g⮯',
 		"ct_dots": 'show hidden files (if server permits)">dotfiles',
+		"ct_qdel": 'when deleting files, only ask for confirmation once">qdel',
 		"ct_dir1st": 'sort folders before files">📁 first',
 		"ct_nsort": 'natural sort (for filenames with leading digits)">nsort',
 		"ct_readme": 'show README.md in folder listings">📜 readme',
@@ -658,7 +659,7 @@ var Ls = {
 				"filbehandler",
 				["G", "listevisning eller ikoner"],
 				["T", "miniatyrbilder på/av"],
-				["🡅 A/D", "ikonstørrelse"],
+				["⇧ A/D", "ikonstørrelse"],
 				["ctrl-K", "slett valgte"],
 				["ctrl-X", "klipp ut valgte"],
 				["ctrl-C", "kopiér til utklippstavle"],
@@ -668,9 +669,9 @@ var Ls = {
 
 				"filmarkering",
 				["space", "marker fil"],
-				["🡑/🡓", "flytt markør"],
-				["ctrl 🡑/🡓", "flytt markør og scroll"],
-				["🡅 🡑/🡓", "velg forr./neste fil"],
+				["↑/↓", "flytt markør"],
+				["ctrl ↑/↓", "flytt markør og scroll"],
+				["⇧ ↑/↓", "velg forr./neste fil"],
 				["ctrl-A", "velg alle filer / mapper"],
 			], [
 				"navigering",
@@ -693,7 +694,7 @@ var Ls = {
 				["Home/End", "første/siste bilde"],
 				["F", "fullskjermvisning"],
 				["R", "rotere mot høyre"],
-				["🡅 R", "rotere mot venstre"],
+				["⇧ R", "rotere mot venstre"],
 				["S", "marker bilde"],
 				["Y", "last ned bilde"],
 			], [
@@ -850,6 +851,7 @@ var Ls = {
 		"ct_csel": 'bruk tastene CTRL og SHIFT for markering av filer i ikonvisning">merk',
 		"ct_ihop": 'bla ned til sist viste bilde når bildeviseren lukkes">g⮯',
 		"ct_dots": 'vis skjulte filer (gitt at serveren tillater det)">.synlig',
+		"ct_qdel": 'sletteknappen spør bare én gang om bekreftelse">hurtig🗑️',
 		"ct_dir1st": 'sorter slik at mapper kommer foran filer">📁 først',
 		"ct_nsort": 'naturlig sortering (forstår tall i filnavn)">nsort',
 		"ct_readme": 'vis README.md nedenfor filene">📜 readme',
@@ -1283,7 +1285,7 @@ var Ls = {
 				"file-manager",
 				["G", "切换列表 / 网格视图"],
 				["T", "切换缩略图 / 图标"],
-				["🡅 A/D", "缩略图大小"],
+				["⇧ A/D", "缩略图大小"],
 				["ctrl-K", "删除选中项"],
 				["ctrl-X", "剪切选中项"],
 				["ctrl-C", "复制选中项"], //m
@@ -1293,9 +1295,9 @@ var Ls = {
 
 				"file-list-sel",
 				["space", "切换文件选择"],
-				["🡑/🡓", "移动选择光标"],
-				["ctrl 🡑/🡓", "移动光标和视图"],
-				["🡅 🡑/🡓", "选择上一个/下一个文件"],
+				["↑/↓", "移动选择光标"],
+				["ctrl ↑/↓", "移动光标和视图"],
+				["⇧ ↑/↓", "选择上一个/下一个文件"],
 				["ctrl-A", "选择所有文件 / 文件夹"]
 			], [
 				"navigation",
@@ -1318,7 +1320,7 @@ var Ls = {
 				["Home/End", "第一张/最后一张图片"],
 				["F", "全屏"],
 				["R", "顺时针旋转"],
-				["🡅 R", "逆时针旋转"],
+				["⇧ R", "逆时针旋转"],
 				["S", "选择图片"], //m
 				["Y", "下载图片"]
 			], [
@@ -1474,6 +1476,7 @@ var Ls = {
 		"ct_csel": '在网格视图中使用 CTRL 和 SHIFT 进行文件选择">CTRL',
 		"ct_ihop": '当图像查看器关闭时，滚动到最后查看的文件">滚动',
 		"ct_dots": '显示隐藏文件（如果服务器允许）">隐藏文件',
+		"ct_qdel": '删除文件时，只需确认一次">快删', //m
 		"ct_dir1st": '在文件之前排序文件夹">📁 排序',
 		"ct_nsort": '正确排序以数字开头的文件名">数字排序', //m
 		"ct_readme": '在文件夹列表中显示 README.md">📜 readme',
@@ -2090,6 +2093,7 @@ ebi('op_cfg').innerHTML = (
 	'		<a id="csel" class="tgl btn" href="#" tt="' + L.ct_csel + '</a>\n' +
 	'		<a id="ihop" class="tgl btn" href="#" tt="' + L.ct_ihop + '</a>\n' +
 	'		<a id="dotfiles" class="tgl btn" href="#" tt="' + L.ct_dots + '</a>\n' +
+	'		<a id="qdel" class="tgl btn" href="#" tt="' + L.ct_qdel + '</a>\n' +
 	'		<a id="dir1st" class="tgl btn" href="#" tt="' + L.ct_dir1st + '</a>\n' +
 	'		<a id="nsort" class="tgl btn" href="#" tt="' + L.ct_nsort + '</a>\n' +
 	'		<a id="ireadme" class="tgl btn" href="#" tt="' + L.ct_readme + '</a>\n' +
@@ -2258,14 +2262,17 @@ SPINNER = m[0];
 
 
 var SBW, SBH;  // scrollbar size
-(function () {
+function read_sbw() {
 	var el = mknod('div');
-	el.style.cssText = 'overflow:scroll;width:100px;height:100px';
+	el.style.cssText = 'overflow:scroll;width:100px;height:100px;position:absolute;top:0;left:0';
 	document.body.appendChild(el);
 	SBW = el.offsetWidth - el.clientWidth;
 	SBH = el.offsetHeight - el.clientHeight;
 	document.body.removeChild(el);
-})();
+	setcvar('--sbw', SBW + 'px');
+	setcvar('--sbh', SBH + 'px');
+}
+onresize100.add(read_sbw, true);
 
 
 var have_webp = sread('have_webp');
@@ -3680,7 +3687,7 @@ var mpui = (function () {
 					var oi = mp.order.indexOf(mp.au.tid) + 1,
 						evp = get_evpath();
 
-					if (mpl.pb_mode == 'loop' || mp.au.evp != evp)
+					if (mpl.pb_mode == 'loop' || mp.au.evp != evp || ebi('unsearch'))
 						oi = 0;
 
 					if (oi >= mp.order.length) {
@@ -5450,7 +5457,16 @@ var fileman = (function () {
 			deleter();
 		}
 
+		var asks = r.qdel ? 1 : 2;
+		if (dqdel === 0)
+			asks -= 1;
+
+		if (!asks)
+			return deleter();
+
 		modal.confirm('<h6 style="color:#900">' + L.danger + '</h6>\n<b>' + L.fd_warn1.format(vps.length) + '</b><ul>' + uricom_adec(vps, true).join('') + '</ul>', function () {
+			if (asks === 1)
+				return deleter();
 			modal.confirm(L.fd_warn2, deleter, null);
 		}, null);
 	};
@@ -5811,6 +5827,8 @@ var fileman = (function () {
 		r.bus.onmessage();
 	};
 
+	bcfg_bind(r, 'qdel', 'qdel', dqdel == 1);
+
 	bren.onclick = r.rename;
 	bdel.onclick = r.delete;
 	bcut.onclick = r.cut;
@@ -6882,8 +6900,10 @@ function hkhelp() {
 			try {
 				if (c[a].length != 2)
 					html.push('<tr><th colspan="2">' + esc(c[a]) + '</th></tr>');
-				else
-					html.push('<tr><td>{0}</td><td>{1}</td></tr>'.format(c[a][0], c[a][1]));
+				else {
+					var t1 = c[a][0].replace('⇧', '<b>⇧</b>');
+					html.push('<tr><td>{0}</td><td>{1}</td></tr>'.format(t1, c[a][1]));
+				}
 			}
 			catch (ex) {
 				html.push(">>> " + c[a]);

copyparty/web/idp.html

@@ -0,0 +1,55 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+	<meta charset="utf-8">
+	<title>{{ s_doctitle }}</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+    <meta name="robots" content="noindex, nofollow">
+	<meta name="theme-color" content="#{{ tcolor }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/shares.css?_={{ ts }}">
+	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
+{{ html_head }}
+</head>
+
+<body>
+	<div id="wrap">
+		<a href="{{ r }}/?idp">refresh</a>
+		<a href="{{ r }}/?h">control-panel</a>
+
+        <table id="tab"><thead><tr>
+            <th>forget</th>
+            <th>user</th>
+            <th>groups</th>
+        </tr></thead><tbody>
+        {% for un, gn in rows %}
+        <tr>
+            <td><a href="{{ r }}/?idp=rm={{ un|e }}">forget</a></td>
+            <td>{{ un|e }}</td>
+            <td>{{ gn|e }}</td>
+        </tr>
+        {% endfor %}
+        </tbody></table>
+        {% if not rows %}
+        (there are no IdP users in the cache)
+        {% endif %}
+    </div>
+	<a href="#" id="repl">π</a>
+	<script>
+
+var SR="{{ r }}",
+	lang="{{ lang }}",
+	dfavico="{{ favico }}";
+
+var STG = window.localStorage;
+document.documentElement.className = (STG && STG.cpp_thm) || "{{ this.args.theme }}";
+
+</script>
+<script src="{{ r }}/.cpr/util.js?_={{ ts }}"></script>
+{%- if js %}
+<script src="{{ js }}_={{ ts }}"></script>
+{%- endif %}
+</body>
+</html>
+

copyparty/web/splash.html

@@ -135,6 +135,10 @@
 
 		<h1 id="cc">other stuff:</h1>
 		<ul>
+			{%- if this.uname in this.args.idp_adm_set %}
+			<li><a id="ag" href="{{ r }}/?idp">view idp cache</a></li>
+			{% endif %}
+
 			{%- if this.uname != '*' and this.args.shr %}
 			<li><a id="y" href="{{ r }}/?shares">edit shares</a></li>
 			{% endif %}

copyparty/web/splash.js

@@ -39,6 +39,7 @@ var Ls = {
 		"ad1": "no304 stopper all bruk av cache. Hvis ikke k304 var nok, prøv denne. Vil mangedoble dataforbruk!",
 		"ae1": "utgående:",
 		"af1": "vis nylig opplastede filer",
+		"ag1": "vis kjente IdP-brukere",
 	},
 	"eng": {
 		"d2": "shows the state of all active threads",
@@ -90,6 +91,7 @@ var Ls = {
 		"ad1": "启用 no304 将禁用所有缓存；如果 k304 不够，可以尝试此选项。这将消耗大量的网络流量！", //m
 		"ae1": "正在下载：", //m
 		"af1": "显示最近上传的文件", //m
+		"ag1": "查看已知 IdP 用户", //m
 	}
 };
 

copyparty/web/up2k.js

@@ -2431,8 +2431,8 @@ function up2k_init(subtle) {
             try { orz(e); } catch (ex) { vis_exh(ex + '', 'up2k.js', '', '', ex); }
         };
 
-        xhr.timeout = 34000;
         xhr.open('HEAD', t.purl + uricom_enc(t.name), true);
+        xhr.timeout = 34000;
         xhr.send();
     }
 
@@ -2911,6 +2911,7 @@ function up2k_init(subtle) {
 
                 st.bytes.inflight += db;
                 xhr.bsent = nb;
+                if (!IE)
                     xhr.timeout = 64000 + Date.now() - xhr.t0;
                 pvis.prog(t, pcar, nb);
             };
@@ -2959,7 +2960,7 @@ function up2k_init(subtle) {
 
             xhr.bsent = 0;
             xhr.t0 = Date.now();
-            xhr.timeout = 42000;
+            xhr.timeout = 1000 * (IE ? 1234 : 42);
             xhr.responseType = 'text';
             xhr.send(t.fobj.slice(car, cdr));
         }

copyparty/web/util.js

@@ -32,7 +32,7 @@ var wah = '',
     CHROME = !!window.chrome,  // safari=false
     VCHROME = CHROME ? 1 : 0,
     UA = '' + navigator.userAgent,
-    IE = /Trident\//.test(UA),
+    IE = !!document.documentMode,
     FIREFOX = ('netscape' in window) && / rv:/.test(UA),
     IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(UA),
     LINUX = /Linux/.test(UA),

docs/changelog.md

@@ -1,3 +1,96 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0721-2307  `v1.18.3`  drop the umask
+
+## 🧪 new features
+
+* #181 the default chmod (unix-permissions) of new files and folders can now be changed 9921c43e
+  * `--chmod-d` or volflag `chmod_d` sets directory permissions; default is 755
+  * `--chmod-f` or volflag `chmod_f` sets file permissions; default is usually 644 (OS-defined)
+  * see `--help-chmod` which explains the numbers
+
+## 🩹 bugfixes
+
+* #179 couldn't combine `--shr` (shares) and `--xvol` (symlink-guard) 0f0f8d90
+* #180 gallery buttons could still be clicked when faded-out 8c32b0e7
+* rss-feeds were slightly busted when combined with rp-loc (location-based proxying) 56d3bcf5
+* music-playback within search-results no longer jumps into the next folder at end-of-list 9bc4c5d2
+* video-playback on iOS now behaves like on all other platforms 78605d9a
+  * (it would force-switch into fullscreen because that's their default)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0707-1419  `v1.18.2`  idp-vol persistence
+
+## 🧪 new features
+
+* IdP-volumes can optionally be persisted across restarts d162502c
+  * there is a UI to manage the cached users/groups 4f264a0a
+    * only available to users listed in the new option `--idp-adm`
+* api for manually rescanning several volumes at once 42c199e7
+  * `/some/path/?scan` does that one volume like before
+  * `/any/path/?scan=/vol1,/another/vol2` rescans `/vol1` and `/another/vol2`
+* volflag to hide volume from listing in controlpanel fd7c71d6
+
+## 🩹 bugfixes
+
+* macos: fix confusing crash when blocked by [Little Snitch](https://www.obdev.at/products/littlesnitch/) bf11b2a4
+* unpost could break in some hairy reverseproxy setups 1b2d3985
+* copyparty32.exe: fix segfault on win7 c9fafb20
+* ui: fix navpane overlapping the scrollbar (still a bit jank but eh) 7ef6fd13
+* usb-eject: support all volume names ed908b98
+* docker: ensure clean slate deb6711b
+* fix up2k on ie11 d2714434
+
+## 🔧 other changes
+
+* update buildscript for keyfinder to support llvm 65c4e035
+* #175 add `python-magic` into the `iv` and `dj` docker flavors (thx @Morganamilo) 77274e9d
+* properly killed the experimental docker flavors to avoid confusion 8306e3d9
+* copyparty.exe: updated pillow 299cff3f f6be3905
+  * avif support was removed to save 2 MiB
+
+## 🌠 fun facts
+
+* this release was slightly delayed due to a [norwegian traffic jam](https://a.ocv.me/pub/g/2025/07/PXL_20250706_143558381.jpg)
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2025-0622-0020  `v1.18.0`  Logtail
+
+## 🧪 new features
+
+* textfile-viewer can now livestream logfiles (and other growing files) 17fa4906 77df17d1 a1c7a095 6ecf4fdc
+  * see [readme](https://github.com/9001/copyparty/#textfile-viewer) and the [live demo](https://a.ocv.me/pub/demo/logtail/)
+* IdP-volumes: extend syntax for excluding certain users/groups 2e53f797
+  * the commit-message explains it well enough
+* new option `--see-dots` to show dotfiles in the web-ui by default c599e2aa
+* #171 automatic mimetype detection for files without extensions (thx @Morganamilo!) ec05f8cc 9dd5dec0
+  * default-disabled since it has a performance impact on webdav
+    * there are plans to fix this by using the db instead
+* #170 improve custom filetype icons
+  * be less strict; if a thumbnail is set for `.gz` files, use it for `.tar.gz` too c75b0c25
+  * improve config docs fa5845ff
+
+## 🩹 bugfixes
+
+* cosmetic: get rid of some noise along the bottom of some cards in the gridview 8cae7a71
+* cosmetic: satisfy a new syntax warning in cpython-3.14 5ac38648
+
+## 🔧 other changes
+
+* properly document how to [build from source](https://github.com/9001/copyparty/blob/hovudstraum/docs/devnotes.md#build-from-scratch) / build from scratch f61511d8
+* update deps
+  * copyparty.exe: python 3.13 1eff87c3
+  * webdeps: dompurify 7eca90cc
+
+## 🌠 fun facts
+
+* this release was cooked up in a [swedish forest cabin](https://a.ocv.me/pub/g/nerd-stuff/forestparty.jpg)
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2025-0527-1939  `v1.17.2`  pushing chrome to the limits (and then some)
 

docs/devnotes.md

@@ -261,6 +261,7 @@ upload modifiers:
 |--|--|--|
 | GET | `?reload=cfg` | reload config files and rescan volumes |
 | GET | `?scan` | initiate a rescan of the volume which provides URL |
+| GET | `?scan=/a,/b` | initiate a rescan of volumes `/a` and `/b` |
 | GET | `?stack` | show a stacktrace of all threads |
 
 ## general
@@ -354,7 +355,7 @@ pip install mutagen  # audio metadata
 pip install pyftpdlib  # ftp server
 pip install partftpy  # tftp server
 pip install impacket  # smb server -- disable Windows Defender if you REALLY need this on windows
-pip install Pillow pyheif-pillow-opener pillow-avif-plugin  # thumbnails
+pip install Pillow pyheif-pillow-opener  # thumbnails
 pip install pyvips  # faster thumbnails
 pip install psutil  # better cleanup of stuck metadata parsers on windows 
 pip install black==21.12b0 click==8.0.2 bandit pylint flake8 isort mypy  # vscode tooling

scripts/docker/Dockerfile.dj

@@ -15,6 +15,7 @@ RUN     apk add -U !pyc \
             py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
             py3-pip py3-cffi \
             ffmpeg \
+            py3-magic \
             vips-jxl vips-heif vips-poppler vips-magick \
             py3-numpy fftw libsndfile \
             vamp-sdk vamp-sdk-libs \

scripts/docker/Dockerfile.djd

@@ -1,4 +1,7 @@
-FROM    debian:12-slim
+FROM    DO_NOT_USE_THIS_DOCKER_IMAGE
+# this image is an unmaintained experiment to see whether alpine was the correct choice (it was)
+
+#FROM    debian:12-slim
 WORKDIR /z
 LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
         org.opencontainers.image.source="https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker" \

scripts/docker/Dockerfile.djf

@@ -1,4 +1,7 @@
-FROM    fedora:39
+FROM    DO_NOT_USE_THIS_DOCKER_IMAGE
+# this image is an unmaintained experiment to see whether alpine was the correct choice (it was)
+
+#FROM    fedora:39
 WORKDIR /z
 LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
         org.opencontainers.image.source="https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker" \

scripts/docker/Dockerfile.djff

@@ -1,4 +1,7 @@
-FROM    fedora:38
+FROM    DO_NOT_USE_THIS_DOCKER_IMAGE
+# this image is an unmaintained experiment to see whether alpine was the correct choice (it was)
+
+#FROM    fedora:38
 WORKDIR /z
 LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
         org.opencontainers.image.source="https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker" \

scripts/docker/Dockerfile.dju

@@ -1,4 +1,7 @@
-FROM    ubuntu:23.04
+FROM    DO_NOT_USE_THIS_DOCKER_IMAGE
+# this image is an unmaintained experiment to see whether alpine was the correct choice (it was)
+
+#FROM    ubuntu:23.04
 WORKDIR /z
 LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
         org.opencontainers.image.source="https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker" \

scripts/docker/Dockerfile.iv

@@ -12,6 +12,7 @@ RUN     apk add -U !pyc \
             py3-jinja2 py3-argon2-cffi py3-pyzmq py3-pillow \
             py3-pip py3-cffi \
             ffmpeg \
+            py3-magic \
             vips-jxl vips-heif vips-poppler vips-magick \
         && apk add -t .bd \
             bash wget gcc g++ make cmake patchelf \

scripts/docker/innvikler.sh

@@ -63,12 +63,13 @@ python3 -m copyparty \
   --ign-ebind -p$((1024+RANDOM)),$((1024+RANDOM)),$((1024+RANDOM)) \
   -v .::r --no-crt -qi127.1 --wr-h-eps $t & pid=$!
 
-for n in $(seq 1 200); do sleep 0.2
+for n in $(seq 1 900); do sleep 0.2
   v=$(awk '/^127/{print;n=1;exit}END{exit n-1}' $t) && break
 done
 [ -z "$v" ] && echo SNAAAAAKE && exit 1
+rm $t
 
-for n in $(seq 1 200); do sleep 0.2
+for n in $(seq 1 900); do sleep 0.2
   wget -O- http://${v/ /:}/?tar=gz:1 >tf && break
 done
 tar -xzO top/innvikler.sh <tf | cmp innvikler.sh
@@ -79,7 +80,7 @@ kill $pid; wait $pid
 ########################################################################
 
 # output from -e2d
-rm -rf .hist
+rm -rf .hist /cfg/copyparty
 
 # goodbye
 exec rm innvikler.sh

scripts/help2html.py

@@ -7,7 +7,7 @@ import subprocess as sp
 
 # to convert the copyparty --help to html, run this in xfce4-terminal @ 140x43:
 _ = r""""
-echo; for a in '' -bind -accounts -flags -handlers -hooks -urlform -exp -ls -dbd -pwhash -zm; do
+echo; for a in '' -bind -accounts -flags -handlers -hooks -urlform -exp -ls -dbd -chmod -pwhash -zm; do
 ./copyparty-sfx.py --help$a 2>/dev/null; printf '\n\n\n%0139d\n\n\n'; done  # xfce4-terminal @ 140x43
 """
 # click [edit] => [select all]

scripts/help2txt.sh

@@ -23,7 +23,7 @@ exit 0
 
 
 # first open an infinitely wide console (this is why you own an ultrawide) and copypaste this into it:
-for a in '' -bind -accounts -flags -handlers -hooks -urlform -exp -ls -dbd -pwhash -zm; do
+for a in '' -bind -accounts -flags -handlers -hooks -urlform -exp -ls -dbd -chmod -pwhash -zm; do
 ./copyparty-sfx.py --help$a 2>/dev/null; printf '\n\n\n%0255d\n\n\n'; done
 
 # then copypaste all of the output by pressing ctrl-shift-a, ctrl-shift-c

scripts/pyinstaller/build.sh

@@ -14,6 +14,7 @@ clean=--clean
 
 uname -s | grep WOW64 && m=64 || m=32
 uname -s | grep NT-10 && w10=1 || w7=1
+[ $w7 ] && export PRTY_NO_MAGIC=1
 [ $w7 ] && [ -e up2k.sh ] && [ ! "$1" ] && ./up2k.sh
 
 [ $w7 ] && pyv=37 || pyv=313
@@ -94,6 +95,7 @@ excl=(
     _pyrepl
     distutils
     setuptools
+    PIL._avif
     PIL.ImageQt
     PIL.ImageShow
     PIL.ImageTk

scripts/pyinstaller/deps.sha512

@@ -27,7 +27,7 @@ ac96786e5d35882e0c5b724794329c9125c2b86ae7847f17acfc49f0d294312c6afc1c3f248655de
 00731cfdd9d5c12efef04a7161c90c1e5ed1dc4677aa88a1d4054aff836f3430df4da5262ed4289c21637358a9e10e5df16f76743cbf5a29bb3a44b146c19cf3  MarkupSafe-3.0.2-cp313-cp313-win_amd64.whl
 8a6e2b13a2ec4ef914a5d62aad3db6464d45e525a82e07f6051ed10474eae959069e165dba011aefb8207cdfd55391d73d6f06362c7eb247b08763106709526e  mutagen-1.47.0-py3-none-any.whl
 a726fb46cce24f781fc8b55a3e6dea0a884ebc3b2b400ea74aa02333699f4955a5dc1e2ec5927ac72f35a624401f3f3b442882ba1cc4cadaf9c88558b5b8bdae  packaging-25.0-py3-none-any.whl
-9265164114db16c7f4286f188d6ebc7f3e2c9a9aca7144f92bc66c320d7d5db44e2c3a93e5e8f8cc12b8ffb30d0402a9510716ed7dbd10df39d19f5cf7ed7486  pillow-11.2.1-cp313-cp313-win_amd64.whl
+3e39ea6e16b502d99a2e6544579095d0f7c6097761cd85135d5e929b9dec1b32e80669a846f94ee8c2cca9be2f5fe728625d09453988864c04e16bb8445c3f91  pillow-11.3.0-cp313-cp313-win_amd64.whl
 59fbbcae044f4ee73d203ac74b553b27bfad3e6b2f3fb290fd3f8774753c6b545176b6b3399c240b092d131d152290ce732750accd962dc1e48e930be85f5e53  pyinstaller-6.14.1-py3-none-win_amd64.whl
 fc6f3e144c5f5b662412de07cb8bf0c2eb3b3be21d19ec448aef3c4244d779b9ab8027fd67a4871e6e13823b248ea0f5a7a9241a53aef30f3b51a6d3cb5bdb3f  pyinstaller_hooks_contrib-2025.5-py3-none-any.whl
 2c7a52e223b8186c21009d3fa5ed6a856d8eb4ef3b98f5d24c378c6a1afbfa1378bd7a51d6addc500e263d7989efb544c862bf920055e740f137c702dfd9d18b  python-3.13.5-amd64.exe

scripts/pyinstaller/notes.txt

@@ -37,7 +37,7 @@ fns=(
   MarkupSafe-3.0.2-cp313-cp313-win_amd64.whl
   mutagen-1.47.0-py3-none-any.whl
   packaging-25.0-py3-none-any.whl
-  pillow-11.2.1-cp313-cp313-win_amd64.whl
+  pillow-11.3.0-cp313-cp313-win_amd64.whl
   pyinstaller-6.14.1-py3-none-win_amd64.whl
   pyinstaller_hooks_contrib-2025.5-py3-none-any.whl
   python-3.13.5-amd64.exe

scripts/sfx.ls

@@ -94,6 +94,7 @@ copyparty/web/deps/prismd.css,
 copyparty/web/deps/scp.woff2,
 copyparty/web/deps/sha512.ac.js,
 copyparty/web/deps/sha512.hw.js,
+copyparty/web/idp.html,
 copyparty/web/iiam.gif,
 copyparty/web/md.css,
 copyparty/web/md.html,

scripts/tl.js

@@ -121,7 +121,7 @@ var tl_browser = {
 				"file-manager",
 				["G", "toggle list / grid view"],
 				["T", "toggle thumbnails / icons"],
-				["🡅 A/D", "thumbnail size"],
+				["⇧ A/D", "thumbnail size"],
 				["ctrl-K", "delete selected"],
 				["ctrl-X", "cut selection to clipboard"],
 				["ctrl-C", "copy selection to clipboard"],
@@ -131,9 +131,9 @@ var tl_browser = {
 
 				"file-list-sel",
 				["space", "toggle file selection"],
-				["🡑/🡓", "move selection cursor"],
-				["ctrl 🡑/🡓", "move cursor and viewport"],
-				["🡅 🡑/🡓", "select prev/next file"],
+				["↑/↓", "move selection cursor"],
+				["ctrl ↑/↓", "move cursor and viewport"],
+				["⇧ ↑/↓", "select prev/next file"],
 				["ctrl-A", "select all files / folders"],
 			], [
 				"navigation",
@@ -156,7 +156,7 @@ var tl_browser = {
 				["Home/End", "first/last pic"],
 				["F", "fullscreen"],
 				["R", "rotate clockwise"],
-				["🡅 R", "rotate ccw"],
+				["⇧ R", "rotate ccw"],
 				["S", "select pic"],
 				["Y", "download pic"],
 			], [
@@ -312,6 +312,7 @@ var tl_browser = {
 		"ct_csel": 'use CTRL and SHIFT for file selection in grid-view">sel',
 		"ct_ihop": 'when the image viewer is closed, scroll down to the last viewed file">g⮯',
 		"ct_dots": 'show hidden files (if server permits)">dotfiles',
+		"ct_qdel": 'when deleting files, only ask for confirmation once">qdel',
 		"ct_dir1st": 'sort folders before files">📁 first',
 		"ct_nsort": 'natural sort (for filenames with leading digits)">nsort',
 		"ct_readme": 'show README.md in folder listings">📜 readme',

tests/test_shr.py

@@ -0,0 +1,229 @@
+#!/usr/bin/env python3
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import json
+import os
+import shutil
+import sqlite3
+import tempfile
+import unittest
+
+from copyparty.__init__ import ANYWIN
+from copyparty.authsrv import AuthSrv
+from copyparty.httpcli import HttpCli
+from copyparty.util import absreal
+from tests import util as tu
+from tests.util import Cfg
+
+
+class TestShr(unittest.TestCase):
+    def log(self, src, msg, c=0):
+        m = "%s" % (msg,)
+        if (
+            "warning: filesystem-path does not exist:" in m
+            or "you are sharing a system directory:" in m
+            or "symlink-based deduplication is enabled" in m
+            or m.startswith("hint: argument")
+        ):
+            return
+
+        print(("[%s] %s" % (src, msg)).encode("ascii", "replace").decode("ascii"))
+
+    def assertLD(self, url, auth, els, edl):
+        ls = self.ls(url, auth)
+        self.assertEqual(ls[0], len(els) == 2)
+        if not ls[0]:
+            return
+        a = [list(sorted(els[0])), list(sorted(els[1]))]
+        b = [list(sorted(ls[1])), list(sorted(ls[2]))]
+        self.assertEqual(a, b)
+
+        if edl is None:
+            edl = els[1]
+        can_dl = []
+        for fn in b[1]:
+            if fn == "a.db":
+                continue
+            furl = url + "/" + fn
+            if auth:
+                furl += "?pw=p1"
+            h, zb = self.curl(furl, True)
+            if h.startswith("HTTP/1.1 200 "):
+                can_dl.append(fn)
+        self.assertEqual(edl, can_dl)
+
+    def setUp(self):
+        self.td = tu.get_ramdisk()
+        td = os.path.join(self.td, "vfs")
+        os.mkdir(td)
+        os.chdir(td)
+        os.mkdir("d1")
+        os.mkdir("d2")
+        os.mkdir("d2/d3")
+        for zs in ("d1/f1", "d2/f2", "d2/d3/f3"):
+            with open(zs, "wb") as f:
+                f.write(zs.encode("utf-8"))
+            for dst in ("d1", "d2", "d2/d3"):
+                src, fn = zs.rsplit("/", 1)
+                os.symlink(absreal(zs), dst + "/l" + fn[-1:])
+
+        db = sqlite3.connect("a.db")
+        with db:
+            zs = r"create table sh (k text, pw text, vp text, pr text, st int, un text, t0 int, t1 int)"
+            db.execute(zs)
+        db.close()
+
+    def tearDown(self):
+        os.chdir(tempfile.gettempdir())
+        shutil.rmtree(self.td)
+
+    def cinit(self):
+        self.asrv = AuthSrv(self.args, self.log)
+        self.conn = tu.VHttpConn(self.args, self.asrv, self.log, b"", True)
+
+    def test1(self):
+        self.args = Cfg(
+            a=["u1:p1"],
+            v=["::A,u1", "d1:v1:A,u1", "d2/d3:d2/d3:A,u1"],
+            shr="/shr/",
+            shr1="shr/",
+            shr_db="a.db",
+            shr_v=False,
+        )
+        self.cinit()
+
+        self.assertLD("", True, [["d1", "d2", "v1"], ["a.db"]], [])
+        self.assertLD("d1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("v1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("d2", True, [["d3"], ["f2", "l1", "l2", "l3"]], None)
+        self.assertLD("d2/d3", True, [[], ["f3", "l1", "l2", "l3"]], None)
+        self.assertLD("d3", True, [], [])
+
+        jt = {
+            "k": "r",
+            "vp": ["/"],
+            "pw": "",
+            "exp": "99",
+            "perms": ["read"],
+        }
+        print(self.post_json("?pw=p1&share", jt)[1])
+        jt = {
+            "k": "d2",
+            "vp": ["/d2/"],
+            "pw": "",
+            "exp": "99",
+            "perms": ["read"],
+        }
+        print(self.post_json("?pw=p1&share", jt)[1])
+        self.conn.shutdown()
+        self.cinit()
+
+        self.assertLD("", True, [["d1", "d2", "v1"], ["a.db"]], [])
+        self.assertLD("d1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("v1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("d2", True, [["d3"], ["f2", "l1", "l2", "l3"]], None)
+        self.assertLD("d2/d3", True, [[], ["f3", "l1", "l2", "l3"]], None)
+        self.assertLD("d3", True, [], [])
+
+        self.assertLD("shr/d2", False, [[], ["f2", "l1", "l2", "l3"]], None)
+        self.assertLD("shr/d2/d3", False, [], None)
+
+        self.assertLD("shr/r", False, [["d1"], ["a.db"]], [])
+        self.assertLD("shr/r/d1", False, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("shr/r/d2", False, [], None)  # unfortunate
+        self.assertLD("shr/r/d2/d3", False, [], None)
+
+        self.conn.shutdown()
+
+    def test2(self):
+        self.args = Cfg(
+            a=["u1:p1"],
+            v=["::A,u1", "d1:v1:A,u1", "d2/d3:d2/d3:A,u1"],
+            shr="/shr/",
+            shr1="shr/",
+            shr_db="a.db",
+            shr_v=False,
+            xvol=True,
+        )
+        self.cinit()
+
+        self.assertLD("", True, [["d1", "d2", "v1"], ["a.db"]], [])
+        self.assertLD("d1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("v1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("d2", True, [["d3"], ["f2", "l1", "l2", "l3"]], None)
+        self.assertLD("d2/d3", True, [[], ["f3", "l1", "l2", "l3"]], None)
+        self.assertLD("d3", True, [], [])
+
+        jt = {
+            "k": "r",
+            "vp": ["/"],
+            "pw": "",
+            "exp": "99",
+            "perms": ["read"],
+        }
+        print(self.post_json("?pw=p1&share", jt)[1])
+        jt = {
+            "k": "d2",
+            "vp": ["/d2/"],
+            "pw": "",
+            "exp": "99",
+            "perms": ["read"],
+        }
+        print(self.post_json("?pw=p1&share", jt)[1])
+        self.conn.shutdown()
+        self.cinit()
+
+        self.assertLD("", True, [["d1", "d2", "v1"], ["a.db"]], [])
+        self.assertLD("d1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("v1", True, [[], ["f1", "l1", "l2", "l3"]], None)
+        self.assertLD("d2", True, [["d3"], ["f2", "l1", "l2", "l3"]], None)
+        self.assertLD("d2/d3", True, [[], ["f3", "l1", "l2", "l3"]], None)
+        self.assertLD("d3", True, [], [])
+
+        self.assertLD("shr/d2", False, [[], ["f2", "l1", "l2", "l3"]], ["f2", "l2"])
+        self.assertLD("shr/d2/d3", False, [], [])
+
+        self.assertLD("shr/r", False, [["d1"], ["a.db"]], [])
+        self.assertLD(
+            "shr/r/d1", False, [[], ["f1", "l1", "l2", "l3"]], ["f1", "l1", "l2"]
+        )
+        self.assertLD("shr/r/d2", False, [], [])  # unfortunate
+        self.assertLD("shr/r/d2/d3", False, [], [])
+
+        self.conn.shutdown()
+
+    def ls(self, url: str, auth: bool):
+        zs = url + "?ls" + ("&pw=p1" if auth else "")
+        h, b = self.curl(zs)
+        if not h.startswith("HTTP/1.1 200 "):
+            return (False, [], [])
+        jo = json.loads(b)
+        return (
+            True,
+            [x["href"].rstrip("/") for x in jo.get("dirs") or {}],
+            [x["href"] for x in jo.get("files") or {}],
+        )
+
+    def curl(self, url: str, binary=False):
+        h = "GET /%s HTTP/1.1\r\nConnection: close\r\n\r\n"
+        HttpCli(self.conn.setbuf((h % (url,)).encode("utf-8"))).run()
+        if binary:
+            h, b = self.conn.s._reply.split(b"\r\n\r\n", 1)
+            return [h.decode("utf-8"), b]
+
+        return self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def post_json(self, url: str, data):
+        buf = json.dumps(data).encode("utf-8")
+        msg = [
+            "POST /%s HTTP/1.1" % (url,),
+            "Connection: close",
+            "Content-Type: application/json",
+            "Content-Length: %d" % (len(buf),),
+            "\r\n",
+        ]
+        buf = "\r\n".join(msg).encode("utf-8") + buf
+        print("PUT -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        return self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)

tests/util.py

@@ -152,16 +152,16 @@ class Cfg(Namespace):
         ex = "ah_cli ah_gen css_browser dbpath hist ipu js_browser js_other mime mimes no_forget no_hash no_idx nonsus_urls og_tpl og_ua ua_nodoc ua_nozip"
         ka.update(**{k: None for k in ex.split()})
 
-        ex = "hash_mt hsortn safe_dedup srch_time tail_fd tail_rate u2abort u2j u2sz"
+        ex = "hash_mt hsortn qdel safe_dedup srch_time tail_fd tail_rate u2abort u2j u2sz"
         ka.update(**{k: 1 for k in ex.split()})
 
         ex = "au_vol dl_list mtab_age reg_cap s_thead s_tbody tail_tmax tail_who th_convt ups_who zip_who"
         ka.update(**{k: 9 for k in ex.split()})
 
-        ex = "db_act forget_ip k304 loris no304 nosubtle re_maxage rproxy rsp_jtr rsp_slp s_wr_slp snap_wri theme themes turbo u2ow zipmaxn zipmaxs"
+        ex = "db_act forget_ip idp_store k304 loris no304 nosubtle re_maxage rproxy rsp_jtr rsp_slp s_wr_slp snap_wri theme themes turbo u2ow zipmaxn zipmaxs"
         ka.update(**{k: 0 for k in ex.split()})
 
-        ex = "ah_alg bname chpw_db doctitle df exit favico idp_h_usr ipa html_head lg_sba lg_sbf log_fk md_sba md_sbf name og_desc og_site og_th og_title og_title_a og_title_v og_title_i shr tcolor textfiles unlist vname xff_src zipmaxt R RS SR"
+        ex = "ah_alg bname chmod_f chpw_db doctitle df exit favico idp_h_usr ipa html_head lg_sba lg_sbf log_fk md_sba md_sbf name og_desc og_site og_th og_title og_title_a og_title_v og_title_i shr tcolor textfiles unlist vname xff_src zipmaxt R RS SR"
         ka.update(**{k: "" for k in ex.split()})
 
         ex = "ban_403 ban_404 ban_422 ban_pw ban_url spinner"
@@ -181,6 +181,7 @@ class Cfg(Namespace):
             c=c,
             E=E,
             bup_ck="sha512",
+            chmod_d="755",
             dbd="wal",
             dk_salt="b" * 16,
             fk_salt="a" * 16,
@@ -260,6 +261,9 @@ class VHub(object):
         self.is_dut = True
         self.up2k = Up2k(self)
 
+    def reload(self, a, b):
+        pass
+
 
 class VBrokerThr(BrokerThr):
     def __init__(self, hub):