filter apple metadata files when uploading with up2k.js:

abandoned because it is ultimately pointless; browsers running on macos already do a decent job filtering these out, and there is only limited benefit of doing this for all other users, probably more harm than good in case of false positives
warn if a user doesn't have privileges anywhere
2024-01-11 23:42:31 +01:00 · 2024-01-11 00:24:34 +00:00 · 2024-01-10 23:59:43 +00:00 · 2024-01-10 23:20:42 +01:00 · 2024-01-09 21:47:02 +01:00 · 2024-01-08 22:10:16 +00:00
23 changed files with 374 additions and 133 deletions
--- a/README.md
+++ b/README.md
@@ -1569,10 +1569,12 @@ interact with copyparty using non-browser clients
  * `var xhr = new XMLHttpRequest(); xhr.open('POST', '//127.0.0.1:3923/msgs?raw'); xhr.send('foo');`

 * curl/wget: upload some files (post=file, chunk=stdin)
-  * `post(){ curl -F act=bput -F f=@"$1" http://127.0.0.1:3923/?pw=wark;}`  
-    `post movie.mkv`
+  * `post(){ curl -F f=@"$1" http://127.0.0.1:3923/?pw=wark;}`  
+    `post movie.mkv`  (gives HTML in return)
+  * `post(){ curl -F f=@"$1" 'http://127.0.0.1:3923/?want=url&pw=wark';}`  
+    `post movie.mkv`  (gives hotlink in return)
  * `post(){ curl -H pw:wark -H rand:8 -T "$1" http://127.0.0.1:3923/;}`  
-    `post movie.mkv`
+    `post movie.mkv`  (randomized filename)
  * `post(){ wget --header='pw: wark' --post-file="$1" -O- http://127.0.0.1:3923/?raw;}`  
    `post movie.mkv`
  * `chunk(){ curl -H pw:wark -T- http://127.0.0.1:3923/;}`  
--- a/bin/u2c.py
+++ b/bin/u2c.py
@@ -1057,14 +1057,13 @@ class Ctl(object):
                self.uploader_busy += 1
                self.t0_up = self.t0_up or time.time()

-            zs = "{0}/{1}/{2}/{3} {4}/{5} {6}"
-            stats = zs.format(
+            stats = "%d/%d/%d/%d %d/%d %s" % (
                self.up_f,
                len(self.recheck),
                self.uploader_busy,
                self.nfiles - self.up_f,
-                int(self.nbytes / (1024 * 1024)),
-                int((self.nbytes - self.up_b) / (1024 * 1024)),
+                self.nbytes // (1024 * 1024),
+                (self.nbytes - self.up_b) // (1024 * 1024),
                self.eta,
            )

--- a/contrib/package/arch/PKGBUILD
+++ b/contrib/package/arch/PKGBUILD
@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.9.27"
+pkgver="1.9.28"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -21,7 +21,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("0bcf9362bc1bd9c85c228312c8341fcb9a37e383af6d8bee123e3a84e66394be")
+sha256sums=("0134f2298b8793f3a8a0a9ba6e7e5a5bf3dc12da5501b100c2f17d061f167fb9")

 build() {
    cd "${srcdir}/${pkgname}-${pkgver}"
--- a/contrib/package/nix/copyparty/pin.json
+++ b/contrib/package/nix/copyparty/pin.json
@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.9.27/copyparty-sfx.py",
-    "version": "1.9.27",
-    "hash": "sha256-o06o/gUIkQhDw9a9SLjuAyQUoLMfFCqkIeonUFzezds="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.9.28/copyparty-sfx.py",
+    "version": "1.9.28",
+    "hash": "sha256-3pHlJYmEu0C/aHPN8JL9fb6tEHYpB3/gON7SJko3tUY="
 }
--- a/copyparty/main.py
+++ b/copyparty/main.py
@@ -1432,7 +1432,7 @@ def main(argv: Optional[list[str]] = None) -> None:

        _, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
        if hard > 0:  # -1 == infinite
-            nc = min(nc, hard // 4)
+            nc = min(nc, int(hard / 4))
    except:
        nc = 512

--- a/copyparty/authsrv.py
+++ b/copyparty/authsrv.py
@@ -1235,6 +1235,7 @@ class AuthSrv(object):

        all_users = {}
        missing_users = {}
+        associated_users = {}
        for axs in daxs.values():
            for d in [
                axs.uread,
@@ -1251,6 +1252,8 @@ class AuthSrv(object):
                    all_users[usr] = 1
                    if usr != "*" and usr not in acct:
                        missing_users[usr] = 1
+                    if "*" not in d:
+                        associated_users[usr] = 1

        if missing_users:
            self.log(
@@ -1271,6 +1274,16 @@ class AuthSrv(object):
                raise Exception(BAD_CFG)
            seenpwds[pwd] = usr

+        for usr in acct:
+            if usr not in associated_users:
+                if len(vfs.all_vols) > 1:
+                    # user probably familiar enough that the verbose message is not necessary
+                    t = "account [%s] is not mentioned in any volume definitions; see --help-accounts"
+                    self.log(t % (usr,), 1)
+                else:
+                    t = "WARNING: the account [%s] is not mentioned in any volume definitions and thus has the same access-level and privileges that guests have; please see --help-accounts for details.  For example, if you intended to give that user full access to the current directory, you could do this:  -v .::A,%s"
+                    self.log(t % (usr, usr), 1)
+
        promote = []
        demote = []
        for vol in vfs.all_vols.values():
--- a/copyparty/broker_mpw.py
+++ b/copyparty/broker_mpw.py
@@ -76,7 +76,7 @@ class MpWorker(BrokerCli):
        pass

    def logw(self, msg: str, c: Union[int, str] = 0) -> None:
-        self.log("mp{}".format(self.n), msg, c)
+        self.log("mp%d" % (self.n,), msg, c)

    def main(self) -> None:
        while True:
--- a/copyparty/httpcli.py
+++ b/copyparty/httpcli.py
@@ -45,6 +45,7 @@ from .util import (
    ODict,
    Pebkac,
    UnrecvEOF,
+    WrongPostKey,
    absreal,
    alltrace,
    atomic_move,
@@ -189,7 +190,7 @@ class HttpCli(object):

    def unpwd(self, m: Match[str]) -> str:
        a, b, c = m.groups()
-        return "{}\033[7m {} \033[27m{}".format(a, self.asrv.iacct[b], c)
+        return "%s\033[7m %s \033[27m%s" % (a, self.asrv.iacct[b], c)

    def _check_nonfatal(self, ex: Pebkac, post: bool) -> bool:
        if post:
@@ -556,16 +557,16 @@ class HttpCli(object):
                    self.keepalive = False

                em = str(ex)
-                msg = em if pex == ex else min_ex()
+                msg = em if pex is ex else min_ex()
                if pex.code != 404 or self.do_log:
                    self.log(
-                        "{}\033[0m, {}".format(msg, self.vpath),
+                        "%s\033[0m, %s" % (msg, self.vpath),
                        6 if em.startswith("client d/c ") else 3,
                    )

-                msg = "{}\r\nURL: {}\r\n".format(em, self.vpath)
+                msg = "%s\r\nURL: %s\r\n" % (em, self.vpath)
                if self.hint:
-                    msg += "hint: {}\r\n".format(self.hint)
+                    msg += "hint: %s\r\n" % (self.hint,)

                if "database is locked" in em:
                    self.conn.hsrv.broker.say("log_stacks")
@@ -808,7 +809,7 @@ class HttpCli(object):
            if k in skip:
                continue

-            t = "{}={}".format(quotep(k), quotep(v))
+            t = "%s=%s" % (quotep(k), quotep(v))
            ret.append(t.replace(" ", "+").rstrip("="))

        if not ret:
@@ -856,7 +857,8 @@ class HttpCli(object):
        oh = self.out_headers
        origin = origin.lower()
        good_origins = self.args.acao + [
-            "{}://{}".format(
+            "%s://%s"
+            % (
                "https" if self.is_https else "http",
                self.host.lower().split(":")[0],
            )
@@ -1053,7 +1055,7 @@ class HttpCli(object):
            self.can_read = self.can_write = self.can_get = False

        if not self.can_read and not self.can_write and not self.can_get:
-            self.log("inaccessible: [{}]".format(self.vpath))
+            self.log("inaccessible: [%s]" % (self.vpath,))
            raise Pebkac(401, "authenticate")

        from .dxml import parse_xml
@@ -1403,7 +1405,7 @@ class HttpCli(object):
        if txt and len(txt) == orig_len:
            raise Pebkac(500, "chunk slicing failed")

-        buf = "{:x}\r\n".format(len(buf)).encode(enc) + buf
+        buf = ("%x\r\n" % (len(buf),)).encode(enc) + buf
        self.s.sendall(buf + b"\r\n")
        return txt

@@ -1862,7 +1864,16 @@ class HttpCli(object):
        self.parser = MultipartParser(self.log, self.sr, self.headers)
        self.parser.parse()

-        act = self.parser.require("act", 64)
+        file0: list[tuple[str, Optional[str], Generator[bytes, None, None]]] = []
+        try:
+            act = self.parser.require("act", 64)
+        except WrongPostKey as ex:
+            if ex.got == "f" and ex.fname:
+                self.log("missing 'act', but looks like an upload so assuming that")
+                file0 = [(ex.got, ex.fname, ex.datagen)]
+                act = "bput"
+            else:
+                raise

        if act == "login":
            return self.handle_login()
@@ -1875,7 +1886,7 @@ class HttpCli(object):
            return self.handle_new_md()

        if act == "bput":
-            return self.handle_plain_upload()
+            return self.handle_plain_upload(file0)

        if act == "tput":
            return self.handle_text_upload()
@@ -2314,7 +2325,9 @@ class HttpCli(object):
            vfs.flags.get("xau") or [],
        )

-    def handle_plain_upload(self) -> bool:
+    def handle_plain_upload(
+        self, file0: list[tuple[str, Optional[str], Generator[bytes, None, None]]]
+    ) -> bool:
        assert self.parser
        nullwrite = self.args.nw
        vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
@@ -2336,11 +2349,13 @@ class HttpCli(object):
        files: list[tuple[int, str, str, str, str, str]] = []
        # sz, sha_hex, sha_b64, p_file, fname, abspath
        errmsg = ""
+        tabspath = ""
        dip = self.dip()
        t0 = time.time()
        try:
            assert self.parser.gen
-            for nfile, (p_field, p_file, p_data) in enumerate(self.parser.gen):
+            gens = itertools.chain(file0, self.parser.gen)
+            for nfile, (p_field, p_file, p_data) in enumerate(gens):
                if not p_file:
                    self.log("discarding incoming file without filename")
                    # fallthrough
@@ -2432,6 +2447,8 @@ class HttpCli(object):
                    if not nullwrite:
                        atomic_move(tabspath, abspath)

+                    tabspath = ""
+
                    files.append(
                        (sz, sha_hex, sha_b64, p_file or "(discarded)", fname, abspath)
                    )
@@ -2477,6 +2494,12 @@ class HttpCli(object):
            errmsg = vol_san(
                list(self.asrv.vfs.all_vols.values()), unicode(ex).encode("utf-8")
            ).decode("utf-8")
+            try:
+                got = bos.path.getsize(tabspath)
+                t = "connection lost after receiving %s of the file"
+                self.log(t % (humansize(got),), 3)
+            except:
+                pass

        td = max(0.1, time.time() - t0)
        sz_total = sum(x[0] for x in files)
@@ -4218,7 +4241,7 @@ class HttpCli(object):
        if icur:
            lmte = list(mte)
            if self.can_admin:
-                lmte += ["up_ip", ".up_at"]
+                lmte.extend(("up_ip", ".up_at"))

            taglist = [k for k in lmte if k in tagset]
            for fe in dirs:
--- a/copyparty/httpconn.py
+++ b/copyparty/httpconn.py
@@ -93,7 +93,7 @@ class HttpConn(object):
            self.rproxy = ip

        self.ip = ip
-        self.log_src = "{} \033[{}m{}".format(ip, color, self.addr[1]).ljust(26)
+        self.log_src = ("%s \033[%dm%d" % (ip, color, self.addr[1])).ljust(26)
        return self.log_src

    def respath(self, res_name: str) -> str:
@@ -176,7 +176,7 @@ class HttpConn(object):

                self.s = ctx.wrap_socket(self.s, server_side=True)
                msg = [
-                    "\033[1;3{:d}m{}".format(c, s)
+                    "\033[1;3%dm%s" % (c, s)
                    for c, s in zip([0, 5, 0], self.s.cipher())  # type: ignore
                ]
                self.log(" ".join(msg) + "\033[0m")
--- a/copyparty/httpsrv.py
+++ b/copyparty/httpsrv.py
@@ -366,7 +366,7 @@ class HttpSrv(object):
            if not self.t_periodic:
                name = "hsrv-pt"
                if self.nid:
-                    name += "-{}".format(self.nid)
+                    name += "-%d" % (self.nid,)

                self.t_periodic = Daemon(self.periodic, name)

@@ -385,7 +385,7 @@ class HttpSrv(object):

        Daemon(
            self.thr_client,
-            "httpconn-{}-{}".format(addr[0].split(".", 2)[-1][-6:], addr[1]),
+            "httpconn-%s-%d" % (addr[0].split(".", 2)[-1][-6:], addr[1]),
            (sck, addr),
        )

@@ -402,9 +402,7 @@ class HttpSrv(object):
            try:
                sck, addr = task
                me = threading.current_thread()
-                me.name = "httpconn-{}-{}".format(
-                    addr[0].split(".", 2)[-1][-6:], addr[1]
-                )
+                me.name = "httpconn-%s-%d" % (addr[0].split(".", 2)[-1][-6:], addr[1])
                self.thr_client(sck, addr)
                me.name = self.name + "-poolw"
            except Exception as ex:
--- a/copyparty/ico.py
+++ b/copyparty/ico.py
@@ -27,13 +27,13 @@ class Ico(object):
        c1 = colorsys.hsv_to_rgb(zb[0] / 256.0, 1, 0.3)
        c2 = colorsys.hsv_to_rgb(zb[0] / 256.0, 0.8 if HAVE_PILF else 1, 1)
        ci = [int(x * 255) for x in list(c1) + list(c2)]
-        c = "".join(["{:02x}".format(x) for x in ci])
+        c = "".join(["%02x" % (x,) for x in ci])

        w = 100
        h = 30
        if not self.args.th_no_crop and as_thumb:
            sw, sh = self.args.th_size.split("x")
-            h = int(100 / (float(sw) / float(sh)))
+            h = int(100.0 / (float(sw) / float(sh)))
            w = 100

        if chrome:
@@ -47,12 +47,12 @@ class Ico(object):
                    # [.lt] are hard to see lowercase / unspaced
                    ext2 = re.sub("(.)", "\\1 ", ext).upper()

-                    h = int(128 * h / w)
+                    h = int(128.0 * h / w)
                    w = 128
                    img = Image.new("RGB", (w, h), "#" + c[:6])
                    pb = ImageDraw.Draw(img)
                    _, _, tw, th = pb.textbbox((0, 0), ext2, font_size=16)
-                    xy = ((w - tw) // 2, (h - th) // 2)
+                    xy = (int((w - tw) / 2), int((h - th) / 2))
                    pb.text(xy, ext2, fill="#" + c[6:], font_size=16)

                    img = img.resize((w * 2, h * 2), Image.NEAREST)
@@ -68,7 +68,7 @@ class Ico(object):
                # svg: 3s, cache: 6s, this: 8s
                from PIL import Image, ImageDraw

-                h = int(64 * h / w)
+                h = int(64.0 * h / w)
                w = 64
                img = Image.new("RGB", (w, h), "#" + c[:6])
                pb = ImageDraw.Draw(img)
--- a/copyparty/mtag.py
+++ b/copyparty/mtag.py
@@ -118,7 +118,7 @@ def ffprobe(
        b"--",
        fsenc(abspath),
    ]
-    rc, so, se = runcmd(cmd, timeout=timeout, nice=True)
+    rc, so, se = runcmd(cmd, timeout=timeout, nice=True, oom=200)
    retchk(rc, cmd, se)
    return parse_ffprobe(so)

@@ -240,7 +240,7 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
        if "/" in fps:
            fa, fb = fps.split("/")
            try:
-                fps = int(fa) * 1.0 / int(fb)
+                fps = float(fa) / float(fb)
            except:
                fps = 9001

@@ -564,6 +564,7 @@ class MTag(object):
                args = {
                    "env": env,
                    "nice": True,
+                    "oom": 300,
                    "timeout": parser.timeout,
                    "kill": parser.kill,
                    "capture": parser.capture,
--- a/copyparty/star.py
+++ b/copyparty/star.py
@@ -65,21 +65,21 @@ class StreamTar(StreamArc):
            cmp = re.sub(r"[^a-z0-9]*pax[^a-z0-9]*", "", cmp)

        try:
-            cmp, lv = cmp.replace(":", ",").split(",")
-            lv = int(lv)
+            cmp, zs = cmp.replace(":", ",").split(",")
+            lv = int(zs)
        except:
-            lv = None
+            lv = -1

        arg = {"name": None, "fileobj": self.qfile, "mode": "w", "format": fmt}
        if cmp == "gz":
            fun = tarfile.TarFile.gzopen
-            arg["compresslevel"] = lv if lv is not None else 3
+            arg["compresslevel"] = lv if lv >= 0 else 3
        elif cmp == "bz2":
            fun = tarfile.TarFile.bz2open
-            arg["compresslevel"] = lv if lv is not None else 2
+            arg["compresslevel"] = lv if lv >= 0 else 2
        elif cmp == "xz":
            fun = tarfile.TarFile.xzopen
-            arg["preset"] = lv if lv is not None else 1
+            arg["preset"] = lv if lv >= 0 else 1
        else:
            fun = tarfile.open
            arg["mode"] = "w|"
--- a/copyparty/svchub.py
+++ b/copyparty/svchub.py
@@ -474,7 +474,7 @@ class SvcHub(object):
            import resource

            soft, hard = [
-                x if x > 0 else 1024 * 1024
+                int(x) if x > 0 else 1024 * 1024
                for x in list(resource.getrlimit(resource.RLIMIT_NOFILE))
            ]
        except:
@@ -791,7 +791,7 @@ class SvcHub(object):
                self.logf.flush()

            now = time.time()
-            if now >= self.next_day:
+            if int(now) >= self.next_day:
                self._set_next_day()

    def _set_next_day(self) -> None:
@@ -819,7 +819,7 @@ class SvcHub(object):
        """handles logging from all components"""
        with self.log_mutex:
            now = time.time()
-            if now >= self.next_day:
+            if int(now) >= self.next_day:
                dt = datetime.fromtimestamp(now, UTC)
                zs = "{}\n" if self.no_ansi else "\033[36m{}\033[0m\n"
                zs = zs.format(dt.strftime("%Y-%m-%d"))
--- a/copyparty/th_srv.py
+++ b/copyparty/th_srv.py
@@ -102,7 +102,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
    rd += "\n" + fmt
    h = hashlib.sha512(afsenc(rd)).digest()
    b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
-    rd = "{}/{}/".format(b64[:2], b64[2:4]).lower() + b64
+    rd = ("%s/%s/" % (b64[:2], b64[2:4])).lower() + b64

    # could keep original filenames but this is safer re pathlen
    h = hashlib.sha512(afsenc(fn)).digest()
@@ -115,7 +115,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -
        fmt = "webp" if fc == "w" else "png" if fc == "p" else "jpg"
        cat = "th"

-    return "{}/{}/{}/{}.{:x}.{}".format(histpath, cat, rd, fn, int(mtime), fmt)
+    return "%s/%s/%s/%s.%x.%s" % (histpath, cat, rd, fn, int(mtime), fmt)


 class ThumbSrv(object):
@@ -382,7 +382,7 @@ class ThumbSrv(object):
                # method 0 = pillow-default, fast
                # method 4 = ffmpeg-default
                # method 6 = max, slow
-                fmts += ["RGBA", "LA"]
+                fmts.extend(("RGBA", "LA"))
                args["method"] = 6
            else:
                # default q = 75
@@ -467,9 +467,9 @@ class ThumbSrv(object):
        cmd += [fsenc(tpath)]
        self._run_ff(cmd, vn)

-    def _run_ff(self, cmd: list[bytes], vn: VFS) -> None:
+    def _run_ff(self, cmd: list[bytes], vn: VFS, oom: int = 400) -> None:
        # self.log((b" ".join(cmd)).decode("utf-8"))
-        ret, _, serr = runcmd(cmd, timeout=vn.flags["convt"], nice=True)
+        ret, _, serr = runcmd(cmd, timeout=vn.flags["convt"], nice=True, oom=oom)
        if not ret:
            return

@@ -623,7 +623,7 @@ class ThumbSrv(object):
                fsenc(tmp_opus)
            ]
            # fmt: on
-            self._run_ff(cmd, vn)
+            self._run_ff(cmd, vn, oom=300)

        # iOS fails to play some "insufficiently complex" files
        # (average file shorter than 8 seconds), so of course we
@@ -647,7 +647,7 @@ class ThumbSrv(object):
                fsenc(tpath)
            ]
            # fmt: on
-            self._run_ff(cmd, vn)
+            self._run_ff(cmd, vn, oom=300)

        elif want_caf:
            # simple remux should be safe
@@ -665,7 +665,7 @@ class ThumbSrv(object):
                fsenc(tpath)
            ]
            # fmt: on
-            self._run_ff(cmd, vn)
+            self._run_ff(cmd, vn, oom=300)

        if tmp_opus != tpath:
            try:
--- a/copyparty/up2k.py
+++ b/copyparty/up2k.py
@@ -37,6 +37,7 @@ from .util import (
    absreal,
    atomic_move,
    db_ex_chk,
+    dir_is_empty,
    djoin,
    fsenc,
    gen_filekey,
@@ -145,9 +146,12 @@ class Up2k(object):
        self.entags: dict[str, set[str]] = {}
        self.mtp_parsers: dict[str, dict[str, MParser]] = {}
        self.pending_tags: list[tuple[set[str], str, str, dict[str, Any]]] = []
-        self.hashq: Queue[tuple[str, str, str, str, str, float, str, bool]] = Queue()
+        self.hashq: Queue[
+            tuple[str, str, dict[str, Any], str, str, str, float, str, bool]
+        ] = Queue()
        self.tagq: Queue[tuple[str, str, str, str, str, float]] = Queue()
        self.tag_event = threading.Condition()
+        self.hashq_mutex = threading.Lock()
        self.n_hashq = 0
        self.n_tagq = 0
        self.mpool_used = False
@@ -601,7 +605,7 @@ class Up2k(object):
            for vol in vols:
                try:
                    bos.makedirs(vol.realpath)  # gonna happen at snap anyways
-                    bos.listdir(vol.realpath)
+                    dir_is_empty(self.log_func, not self.args.no_scandir, vol.realpath)
                except:
                    self.volstate[vol.vpath] = "OFFLINE (cannot access folder)"
                    self.log("cannot access " + vol.realpath, c=1)
@@ -983,12 +987,12 @@ class Up2k(object):
                excl = [x.replace("/", "\\") for x in excl]
            else:
                # ~/.wine/dosdevices/z:/ and such
-                excl += ["/dev", "/proc", "/run", "/sys"]
+                excl.extend(("/dev", "/proc", "/run", "/sys"))

            rtop = absreal(top)
            n_add = n_rm = 0
            try:
-                if not bos.listdir(rtop):
+                if dir_is_empty(self.log_func, not self.args.no_scandir, rtop):
                    t = "volume /%s at [%s] is empty; will not be indexed as this could be due to an offline filesystem"
                    self.log(t % (vol.vpath, rtop), 6)
                    return True, False
@@ -1085,7 +1089,7 @@ class Up2k(object):
        cv = ""

        assert self.pp and self.mem_cur
-        self.pp.msg = "a{} {}".format(self.pp.n, cdir)
+        self.pp.msg = "a%d %s" % (self.pp.n, cdir)

        rd = cdir[len(top) :].strip("/")
        if WINDOWS:
@@ -1160,8 +1164,8 @@ class Up2k(object):
                    continue

                if not sz and (
-                    "{}.PARTIAL".format(iname) in partials
-                    or ".{}.PARTIAL".format(iname) in partials
+                    "%s.PARTIAL" % (iname,) in partials
+                    or ".%s.PARTIAL" % (iname,) in partials
                ):
                    # placeholder for unfinished upload
                    continue
@@ -1257,7 +1261,7 @@ class Up2k(object):
            else:
                at = 0

-            self.pp.msg = "a{} {}".format(self.pp.n, abspath)
+            self.pp.msg = "a%d %s" % (self.pp.n, abspath)

            if nohash or not sz:
                wark = up2k_wark_from_metadata(self.salt, sz, lmod, rd, fn)
@@ -1357,7 +1361,7 @@ class Up2k(object):
                rd = drd

            abspath = djoin(top, rd)
-            self.pp.msg = "b{} {}".format(ndirs - nchecked, abspath)
+            self.pp.msg = "b%d %s" % (ndirs - nchecked, abspath)
            try:
                if os.path.isdir(abspath):
                    continue
@@ -1709,7 +1713,7 @@ class Up2k(object):
                    cur.execute(q, (w[:16],))

            abspath = djoin(ptop, rd, fn)
-            self.pp.msg = "c{} {}".format(nq, abspath)
+            self.pp.msg = "c%d %s" % (nq, abspath)
            if not mpool:
                n_tags = self._tagscan_file(cur, entags, w, abspath, ip, at)
            else:
@@ -1766,7 +1770,7 @@ class Up2k(object):
                    if c2.execute(q, (row[0][:16],)).fetchone():
                        continue

-                gf.write("{}\n".format("\x00".join(row)).encode("utf-8"))
+                gf.write(("%s\n" % ("\x00".join(row),)).encode("utf-8"))
                n += 1

        c2.close()
@@ -2351,7 +2355,8 @@ class Up2k(object):
                t = "cannot receive uploads right now;\nserver busy with {}.\nPlease wait; the client will retry..."
                raise Pebkac(503, t.format(self.blocked or "[unknown]"))
        except TypeError:
-            # py2
+            if not PY2:
+                raise
            with self.mutex:
                self._job_volchk(cj)

@@ -2695,7 +2700,7 @@ class Up2k(object):
        else:
            dip = self.hub.iphash.s(ip)

-        suffix = "-{:.6f}-{}".format(ts, dip)
+        suffix = "-%.6f-%s" % (ts, dip)
        with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as zfw:
            return zfw["orz"][1]

@@ -2910,7 +2915,7 @@ class Up2k(object):
        except:
            pass

-        z2 += [upt]
+        z2.append(upt)
        if self.idx_wark(vflags, *z2):
            del self.registry[ptop][wark]
        else:
@@ -3203,9 +3208,9 @@ class Up2k(object):
                    except:
                        pass

-                volpath = "{}/{}".format(vrem, fn).strip("/")
-                vpath = "{}/{}".format(dbv.vpath, volpath).strip("/")
-                self.log("rm {}\n  {}".format(vpath, abspath))
+                volpath = ("%s/%s" % (vrem, fn)).strip("/")
+                vpath = ("%s/%s" % (dbv.vpath, volpath)).strip("/")
+                self.log("rm %s\n  %s" % (vpath, abspath))
                _ = dbv.get(volpath, uname, *permsets[0])
                if xbd:
                    if not runhook(
@@ -3737,7 +3742,7 @@ class Up2k(object):
                    return []

                if self.pp:
-                    mb = int(fsz / 1024 / 1024)
+                    mb = fsz // (1024 * 1024)
                    self.pp.msg = prefix + str(mb) + suffix

                hashobj = hashlib.sha512()
@@ -3802,7 +3807,7 @@ class Up2k(object):
        else:
            dip = self.hub.iphash.s(job["addr"])

-        suffix = "-{:.6f}-{}".format(job["t0"], dip)
+        suffix = "-%.6f-%s" % (job["t0"], dip)
        with ren_open(tnam, "wb", fdir=pdir, suffix=suffix) as zfw:
            f, job["tnam"] = zfw["orz"]
            abspath = djoin(pdir, job["tnam"])
@@ -3991,16 +3996,16 @@ class Up2k(object):
            self.log("tagged {} ({}+{})".format(abspath, ntags1, len(tags) - ntags1))

    def _hasher(self) -> None:
-        with self.mutex:
+        with self.hashq_mutex:
            self.n_hashq += 1

        while True:
-            with self.mutex:
+            with self.hashq_mutex:
                self.n_hashq -= 1
            # self.log("hashq {}".format(self.n_hashq))

            task = self.hashq.get()
-            if len(task) != 8:
+            if len(task) != 9:
                raise Exception("invalid hash task")

            try:
@@ -4009,11 +4014,14 @@ class Up2k(object):
            except Exception as ex:
                self.log("failed to hash %s: %s" % (task, ex), 1)

-    def _hash_t(self, task: tuple[str, str, str, str, str, float, str, bool]) -> bool:
-        ptop, vtop, rd, fn, ip, at, usr, skip_xau = task
+    def _hash_t(
+        self, task: tuple[str, str, dict[str, Any], str, str, str, float, str, bool]
+    ) -> bool:
+        ptop, vtop, flags, rd, fn, ip, at, usr, skip_xau = task
        # self.log("hashq {} pop {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
-        if "e2d" not in self.flags[ptop]:
-            return True
+        with self.mutex:
+            if not self.register_vpath(ptop, flags):
+                return True

        abspath = djoin(ptop, rd, fn)
        self.log("hashing " + abspath)
@@ -4064,11 +4072,22 @@ class Up2k(object):
        usr: str,
        skip_xau: bool = False,
    ) -> None:
-        with self.mutex:
-            self.register_vpath(ptop, flags)
-            self.hashq.put((ptop, vtop, rd, fn, ip, at, usr, skip_xau))
+        if "e2d" not in flags:
+            return
+
+        if self.n_hashq > 1024:
+            t = "%d files in hashq; taking a nap"
+            self.log(t % (self.n_hashq,), 6)
+
+            for _ in range(self.n_hashq // 1024):
+                time.sleep(0.1)
+                if self.n_hashq < 1024:
+                    break
+
+        zt = (ptop, vtop, flags, rd, fn, ip, at, usr, skip_xau)
+        with self.hashq_mutex:
+            self.hashq.put(zt)
            self.n_hashq += 1
-        # self.log("hashq {} push {}/{}/{}".format(self.n_hashq, ptop, rd, fn))

    def shutdown(self) -> None:
        self.stop = True
@@ -4119,6 +4138,6 @@ def up2k_wark_from_hashlist(salt: str, filesize: int, hashes: list[str]) -> str:


 def up2k_wark_from_metadata(salt: str, sz: int, lastmod: int, rd: str, fn: str) -> str:
-    ret = sfsenc("{}\n{}\n{}\n{}\n{}".format(salt, lastmod, sz, rd, fn))
+    ret = sfsenc("%s\n%d\n%d\n%s\n%s" % (salt, lastmod, sz, rd, fn))
    ret = base64.urlsafe_b64encode(hashlib.sha512(ret).digest())
-    return "#{}".format(ret.decode("ascii"))[:44]
+    return ("#%s" % (ret.decode("ascii"),))[:44]
--- a/copyparty/util.py
+++ b/copyparty/util.py
@@ -623,9 +623,14 @@ class _Unrecv(object):
            while nbytes > len(ret):
                ret += self.recv(nbytes - len(ret))
        except OSError:
-            t = "client only sent {} of {} expected bytes".format(len(ret), nbytes)
-            if len(ret) <= 16:
-                t += "; got {!r}".format(ret)
+            t = "client stopped sending data; expected at least %d more bytes"
+            if not ret:
+                t = t % (nbytes,)
+            else:
+                t += ", only got %d"
+                t = t % (nbytes, len(ret))
+                if len(ret) <= 16:
+                    t += "; %r" % (ret,)

            if raise_on_trunc:
                raise UnrecvEOF(5, t)
@@ -849,7 +854,7 @@ class MTHash(object):
                    ex = ex or str(qe)

                if pp:
-                    mb = int((fsz - nch * chunksz) / 1024 / 1024)
+                    mb = (fsz - nch * chunksz) // (1024 * 1024)
                    pp.msg = prefix + str(mb) + suffix

            if ex:
@@ -1071,7 +1076,7 @@ def uprint(msg: str) -> None:


 def nuprint(msg: str) -> None:
-    uprint("{}\n".format(msg))
+    uprint("%s\n" % (msg,))


 def dedent(txt: str) -> str:
@@ -1094,10 +1099,10 @@ def rice_tid() -> str:
 def trace(*args: Any, **kwargs: Any) -> None:
    t = time.time()
    stack = "".join(
-        "\033[36m{}\033[33m{}".format(x[0].split(os.sep)[-1][:-3], x[1])
+        "\033[36m%s\033[33m%s" % (x[0].split(os.sep)[-1][:-3], x[1])
        for x in traceback.extract_stack()[3:-1]
    )
-    parts = ["{:.6f}".format(t), rice_tid(), stack]
+    parts = ["%.6f" % (t,), rice_tid(), stack]

    if args:
        parts.append(repr(args))
@@ -1114,17 +1119,17 @@ def alltrace() -> str:
    threads: dict[str, types.FrameType] = {}
    names = dict([(t.ident, t.name) for t in threading.enumerate()])
    for tid, stack in sys._current_frames().items():
-        name = "{} ({:x})".format(names.get(tid), tid)
+        name = "%s (%x)" % (names.get(tid), tid)
        threads[name] = stack

    rret: list[str] = []
    bret: list[str] = []
    for name, stack in sorted(threads.items()):
-        ret = ["\n\n# {}".format(name)]
+        ret = ["\n\n# %s" % (name,)]
        pad = None
        for fn, lno, name, line in traceback.extract_stack(stack):
            fn = os.sep.join(fn.split(os.sep)[-3:])
-            ret.append('File: "{}", line {}, in {}'.format(fn, lno, name))
+            ret.append('File: "%s", line %d, in %s' % (fn, lno, name))
            if line:
                ret.append("  " + str(line.strip()))
                if "self.not_empty.wait()" in line:
@@ -1133,7 +1138,7 @@ def alltrace() -> str:
        if pad:
            bret += [ret[0]] + [pad + x for x in ret[1:]]
        else:
-            rret += ret
+            rret.extend(ret)

    return "\n".join(rret + bret) + "\n"

@@ -1229,9 +1234,9 @@ def vol_san(vols: list["VFS"], txt: bytes) -> bytes:
 def min_ex(max_lines: int = 8, reverse: bool = False) -> str:
    et, ev, tb = sys.exc_info()
    stb = traceback.extract_tb(tb)
-    fmt = "{} @ {} <{}>: {}"
-    ex = [fmt.format(fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in stb]
-    ex.append("[{}] {}".format(et.__name__ if et else "(anonymous)", ev))
+    fmt = "%s @ %d <%s>: %s"
+    ex = [fmt % (fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in stb]
+    ex.append("[%s] %s" % (et.__name__ if et else "(anonymous)", ev))
    return "\n".join(ex[-max_lines:][:: -1 if reverse else 1])


@@ -1282,7 +1287,7 @@ def ren_open(
            with fun(fsenc(fpath), *args, **kwargs) as f:
                if b64:
                    assert fdir
-                    fp2 = "fn-trunc.{}.txt".format(b64)
+                    fp2 = "fn-trunc.%s.txt" % (b64,)
                    fp2 = os.path.join(fdir, fp2)
                    with open(fsenc(fp2), "wb") as f2:
                        f2.write(orig_name.encode("utf-8"))
@@ -1311,7 +1316,7 @@ def ren_open(
                raise

        if not b64:
-            zs = "{}\n{}".format(orig_name, suffix).encode("utf-8", "replace")
+            zs = ("%s\n%s" % (orig_name, suffix)).encode("utf-8", "replace")
            zs = hashlib.sha512(zs).digest()[:12]
            b64 = base64.urlsafe_b64encode(zs).decode("utf-8")

@@ -1331,7 +1336,7 @@ def ren_open(
                    # okay do the first letter then
                    ext = "." + ext[2:]

-            fname = "{}~{}{}".format(bname, b64, ext)
+            fname = "%s~%s%s" % (bname, b64, ext)


 class MultipartParser(object):
@@ -1517,15 +1522,20 @@ class MultipartParser(object):
        return ret

    def parse(self) -> None:
+        boundary = get_boundary(self.headers)
+        self.log("boundary=%r" % (boundary,))
+
        # spec says there might be junk before the first boundary,
        # can't have the leading \r\n if that's not the case
-        self.boundary = b"--" + get_boundary(self.headers).encode("utf-8")
+        self.boundary = b"--" + boundary.encode("utf-8")

        # discard junk before the first boundary
        for junk in self._read_data():
-            self.log(
-                "discarding preamble: [{}]".format(junk.decode("utf-8", "replace"))
-            )
+            if not junk:
+                continue
+
+            jtxt = junk.decode("utf-8", "replace")
+            self.log("discarding preamble |%d| %r" % (len(junk), jtxt))

        # nice, now make it fast
        self.boundary = b"\r\n" + self.boundary
@@ -1537,11 +1547,9 @@ class MultipartParser(object):
        raises if the field name is not as expected
        """
        assert self.gen
-        p_field, _, p_data = next(self.gen)
+        p_field, p_fname, p_data = next(self.gen)
        if p_field != field_name:
-            raise Pebkac(
-                422, 'expected field "{}", got "{}"'.format(field_name, p_field)
-            )
+            raise WrongPostKey(field_name, p_field, p_fname, p_data)

        return self._read_value(p_data, max_len).decode("utf-8", "surrogateescape")

@@ -1610,7 +1618,7 @@ def rand_name(fdir: str, fn: str, rnd: int) -> str:
                break

            nc = rnd + extra
-            nb = int((6 + 6 * nc) / 8)
+            nb = (6 + 6 * nc) // 8
            zb = os.urandom(nb)
            zb = base64.urlsafe_b64encode(zb)
            fn = zb[:nc].decode("utf-8") + ext
@@ -1713,7 +1721,7 @@ def get_spd(nbyte: int, t0: float, t: Optional[float] = None) -> str:
    bps = nbyte / ((t - t0) + 0.001)
    s1 = humansize(nbyte).replace(" ", "\033[33m").replace("iB", "")
    s2 = humansize(bps).replace(" ", "\033[35m").replace("iB", "")
-    return "{} \033[0m{}/s\033[0m".format(s1, s2)
+    return "%s \033[0m%s/s\033[0m" % (s1, s2)


 def s2hms(s: float, optional_h: bool = False) -> str:
@@ -1721,9 +1729,9 @@ def s2hms(s: float, optional_h: bool = False) -> str:
    h, s = divmod(s, 3600)
    m, s = divmod(s, 60)
    if not h and optional_h:
-        return "{}:{:02}".format(m, s)
+        return "%d:%02d" % (m, s)

-    return "{}:{:02}:{:02}".format(h, m, s)
+    return "%d:%02d:%02d" % (h, m, s)


 def djoin(*paths: str) -> str:
@@ -2193,7 +2201,7 @@ def read_socket_chunked(
            raise Pebkac(400, t.format(x))

        if log:
-            log("receiving {} byte chunk".format(chunklen))
+            log("receiving %d byte chunk" % (chunklen,))

        for chunk in read_socket(sr, chunklen):
            yield chunk
@@ -2365,6 +2373,12 @@ def statdir(
            print(t)


+def dir_is_empty(logger: "RootLogger", scandir: bool, top: str):
+    for _ in statdir(logger, scandir, False, top):
+        return False
+    return True
+
+
 def rmdirs(
    logger: "RootLogger", scandir: bool, lstat: bool, top: str, depth: int
 ) -> tuple[list[str], list[str]]:
@@ -2557,6 +2571,7 @@ def runcmd(
    argv: Union[list[bytes], list[str]], timeout: Optional[float] = None, **ka: Any
 ) -> tuple[int, str, str]:
    isbytes = isinstance(argv[0], (bytes, bytearray))
+    oom = ka.pop("oom", 0)  # 0..1000
    kill = ka.pop("kill", "t")  # [t]ree [m]ain [n]one
    capture = ka.pop("capture", 3)  # 0=none 1=stdout 2=stderr 3=both

@@ -2587,6 +2602,14 @@ def runcmd(
                argv = [NICES] + argv

    p = sp.Popen(argv, stdout=cout, stderr=cerr, **ka)
+
+    if oom and not ANYWIN and not MACOS:
+        try:
+            with open("/proc/%d/oom_score_adj" % (p.pid,), "wb") as f:
+                f.write(("%d\n" % (oom,)).encode("utf-8"))
+        except:
+            pass
+
    if not timeout or PY2:
        bout, berr = p.communicate(sin)
    else:
@@ -2734,6 +2757,7 @@ def _parsehook(
    sp_ka = {
        "env": env,
        "nice": True,
+        "oom": 300,
        "timeout": tout,
        "kill": kill,
        "capture": cap,
@@ -2950,7 +2974,7 @@ def visual_length(txt: str) -> int:
                pend = None
            else:
                if ch == "\033":
-                    pend = "{0}".format(ch)
+                    pend = "%s" % (ch,)
                else:
                    co = ord(ch)
                    # the safe parts of latin1 and cp437 (no greek stuff)
@@ -3048,3 +3072,20 @@ class Pebkac(Exception):

    def __repr__(self) -> str:
        return "Pebkac({}, {})".format(self.code, repr(self.args))
+
+
+class WrongPostKey(Pebkac):
+    def __init__(
+        self,
+        expected: str,
+        got: str,
+        fname: Optional[str],
+        datagen: Generator[bytes, None, None],
+    ) -> None:
+        msg = 'expected field "{}", got "{}"'.format(expected, got)
+        super(WrongPostKey, self).__init__(422, msg)
+
+        self.expected = expected
+        self.got = got
+        self.fname = fname
+        self.datagen = datagen
--- a/copyparty/web/browser.js
+++ b/copyparty/web/browser.js
@@ -204,6 +204,8 @@ var Ls = {

 		"cut_datechk": "has no effect unless the turbo button is enabled$N$Nreduces the yolo factor by a tiny amount; checks whether the file timestamps on the server matches yours$N$Nshould <em>theoretically</em> catch most unfinished / corrupted uploads, but is not a substitute for doing a verification pass with turbo disabled afterwards",

+		"cut_appl": "skip uploading apple/macos metadata files$Nsuch as __MACOSX, .DS_Store, .fseventsd",
+
 		"cut_flag": "ensure only one tab is uploading at a time $N -- other tabs must have this enabled too $N -- only affects tabs on the same domain",

 		"cut_az": "upload files in alphabetical order, rather than smallest-file-first$N$Nalphabetical order can make it easier to eyeball if something went wrong on the server, but it makes uploading slightly slower on fiber / LAN",
@@ -428,6 +430,7 @@ var Ls = {
 		"u_enoi": 'file-search is not enabled in server config',
 		"u_badf": 'These {0} files (of {1} total) were skipped, possibly due to filesystem permissions:\n\n',
 		"u_blankf": 'These {0} files (of {1} total) are blank / empty; upload them anyways?\n\n',
+		"u_applef": 'Because the apple/macos filter is enabled ("up2k switches" in settings),\nsome files ({0} out of {1} in total) will be excluded / skipped.\n\nPress <code>OK/Enter</code> to SKIP the following files,\nPress <code>Cancel/ESC</code> to NOT exclude, and UPLOAD those as well:\n\n',
 		"u_just1": '\nMaybe it works better if you select just one file',
 		"u_ff_many": "This amount of files <em>may</em> cause Firefox to skip some files, or crash.\nPlease try again with fewer files (or use Chrome) if that happens.",
 		"u_up_life": "This upload will be deleted from the server\n{0} after it completes",
@@ -684,6 +687,8 @@ var Ls = {

 		"cut_datechk": "har ingen effekt dersom turbo er avslått$N$Ngjør turbo bittelitt tryggere ved å sjekke datostemplingen på filene (i tillegg til filstørrelse)$N$N<em>burde</em> oppdage og gjenoppta de fleste ufullstendige opplastninger, men er <em>ikke</em> en fullverdig erstatning for å deaktivere turbo og gjøre en skikkelig sjekk",

+		"cut_appl": "filtrer vekk apple/macos tilleggsfiler før opplastning,$Nf.eks. __MACOSX, .DS_Store, .fseventsd",
+
 		"cut_flag": "samkjører nettleserfaner slik at bare én $N kan holde på med befaring / opplastning $N -- andre faner må også ha denne skrudd på $N -- fungerer kun innenfor samme domene",

 		"cut_az": "last opp filer i alfabetisk rekkefølge, istedenfor minste-fil-først$N$Nalfabetisk kan gjøre det lettere å anslå om alt gikk bra, men er bittelitt tregere på fiber / LAN",
@@ -908,6 +913,7 @@ var Ls = {
 		"u_enoi": 'filsøk er deaktivert i serverkonfigurasjonen',
 		"u_badf": 'Disse {0} filene (av totalt {1}) kan ikke leses, kanskje pga rettighetsproblemer i filsystemet på datamaskinen din:\n\n',
 		"u_blankf": 'Disse {0} filene (av totalt {1}) er blanke / uten innhold; ønsker du å laste dem opp uansett?\n\n',
+		"u_applef": 'Fordi apple/macos-filteret er aktivert (eple-knappen i innstillinger),\nså er følgende {0} (av totalt {1}) filer anbefalt ekskludert.\n\nTrykk <code>OK/Enter</code> for å HOPPE OVER disse filene,\nTrykk <code>Avbryt/ESC</code> for å LASTE OPP disse filene også:\n\n',
 		"u_just1": '\nFunker kanskje bedre hvis du bare tar én fil om gangen',
 		"u_ff_many": "Det var mange filer! Mulig at Firefox kommer til å krasje, eller\nhoppe over et par av dem. Smart å ha Chrome på lur i tilfelle.",
 		"u_up_life": "Filene slettes fra serveren {0}\netter at opplastningen er fullført",
@@ -1180,6 +1186,7 @@ ebi('op_cfg').innerHTML = (
 	'		<a id="hashw" class="tgl btn" href="#" tt="' + L.cut_mt + '">mt</a>\n' +
 	'		<a id="u2turbo" class="tgl btn ttb" href="#" tt="' + L.cut_turbo + '">turbo</a>\n' +
 	'		<a id="u2tdate" class="tgl btn ttb" href="#" tt="' + L.cut_datechk + '">date-chk</a>\n' +
+	'		<a id="u2appl" class="tgl btn" href="#" tt="' + L.cut_appl + '">🍎</a>\n' +
 	'		<a id="flag_en" class="tgl btn" href="#" tt="' + L.cut_flag + '">💤</a>\n' +
 	'		<a id="u2sort" class="tgl btn" href="#" tt="' + L.cut_az + '">az</a>\n' +
 	'		<a id="upnag" class="tgl btn" href="#" tt="' + L.cut_nag + '">🔔</a>\n' +
--- a/copyparty/web/up2k.js
+++ b/copyparty/web/up2k.js
@@ -867,6 +867,7 @@ function up2k_init(subtle) {
    bcfg_bind(uc, 'flag_en', 'flag_en', false, apply_flag_cfg);
    bcfg_bind(uc, 'turbo', 'u2turbo', turbolvl > 1, draw_turbo);
    bcfg_bind(uc, 'datechk', 'u2tdate', turbolvl < 3, null);
+    bcfg_bind(uc, 'appl', 'u2appl', true, null);
    bcfg_bind(uc, 'az', 'u2sort', u2sort.indexOf('n') + 1, set_u2sort);
    bcfg_bind(uc, 'hashw', 'hashw', !!window.WebAssembly && (!subtle || !CHROME || MOBILE || VCHROME >= 107), set_hashw);
    bcfg_bind(uc, 'upnag', 'upnag', false, set_upnag);
@@ -1310,6 +1311,45 @@ function up2k_init(subtle) {
            });
        }

+        applefilter(good_files);
+    }
+
+    function applefilter(good_files) {
+        if (!uc.appl)
+            return gotallfiles2(good_files);
+
+        var junk = [], rmi = [];
+        for (var a = 0; a < good_files.length; a++) {
+            if (/\/(__MACOS|Icon\r\r)|\/\.(_|DS_Store|AppleDouble|LSOverride|DocumentRevisions-|fseventsd|Spotlight-|TemporaryItems|Trashes|VolumeIcon\.icns|com\.apple\.timemachine\.donotpresent|AppleDB|AppleDesktop|apdisk)/.exec(good_files[a][1])) {
+                junk.push(good_files[a]);
+                rmi.push(a);
+            }
+        }
+
+        if (!junk.length)
+            return gotallfiles2(good_files);
+
+        // todo: if there's hits so far, start looking for appledoubles
+        //   (files named ._foo where foo is another file in the list)
+        //   and get rid of those too
+
+        var msg = L.u_applef.format(junk.length, good_files.length);
+        for (var a = 0, aa = Math.min(20, junk.length); a < aa; a++)
+            msg += '-- ' + junk[a][1] + '\n';
+
+        return modal.confirm(msg, function () {
+            for (var a = rmi.length - 1; a >= 0; a--)
+                good_files.splice(rmi[a], 1);
+
+            start_actx();
+            gotallfiles2(good_files);
+        }, function () {
+            start_actx();
+            gotallfiles2(good_files);
+        });
+    }
+
+    function gotallfiles2(good_files) {
        good_files.sort(function (a, b) {
            a = a[1];
            b = b[1];
--- a/docs/devnotes.md
+++ b/docs/devnotes.md
@@ -162,8 +162,8 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | PUT | | (binary data) | upload into file at URL |
 | PUT | `?gz` | (binary data) | compress with gzip and write into file at URL |
 | PUT | `?xz` | (binary data) | compress with xz and write into file at URL |
-| mPOST | | `act=bput`, `f=FILE` | upload `FILE` into the folder at URL |
-| mPOST | `?j` | `act=bput`, `f=FILE` | ...and reply with json |
+| mPOST | | `f=FILE` | upload `FILE` into the folder at URL |
+| mPOST | `?j` | `f=FILE` | ...and reply with json |
 | mPOST | | `act=mkdir`, `name=foo` | create directory `foo` at URL |
 | POST | `?delete` | | delete URL recursively |
 | jPOST | `?delete` | `["/foo","/bar"]` | delete `/foo` and `/bar` recursively |
--- a/scripts/bench/filehash.sh
+++ b/scripts/bench/filehash.sh
@@ -12,6 +12,11 @@ set -euo pipefail
 #
 # can be adjusted with --hash-mt (but alpine caps out at 5)

+fsize=256
+nfiles=128
+pybin=$(command -v python3 || command -v python)
+#pybin=~/.pyenv/versions/nogil-3.9.10-2/bin/python3
+
 [ $# -ge 1 ] || {
 	echo 'need arg 1: path to copyparty-sfx.py'
 	echo ' (remaining args will be passed on to copyparty,'
@@ -22,6 +27,8 @@ sfx="$1"
 shift
 sfx="$(realpath "$sfx" || readlink -e "$sfx" || echo "$sfx")"
 awk=$(command -v gawk || command -v awk)
+uname -s | grep -E MSYS && win=1 || win=
+totalsize=$((fsize*nfiles))

 # try to use /dev/shm to avoid hitting filesystems at all,
 # otherwise fallback to mktemp which probably uses /tmp
@@ -30,20 +37,24 @@ mkdir $td || td=$(mktemp -d)
 trap "rm -rf $td" INT TERM EXIT
 cd $td

-echo creating 256 MiB testfile in $td
-head -c $((1024*1024*256)) /dev/urandom > 1
+echo creating $fsize MiB testfile in $td
+sz=$((1024*1024*fsize))
+head -c $sz /dev/zero | openssl enc -aes-256-ctr -iter 1 -pass pass:k -nosalt 2>/dev/null >1 || true
+wc -c 1 | awk '$1=='$sz'{r=1}END{exit 1-r}' || head -c $sz /dev/urandom >1

-echo creating 127 symlinks to it
-for n in $(seq 2 128); do ln -s 1 $n; done
+echo creating $((nfiles-1)) symlinks to it
+for n in $(seq 2 $nfiles); do MSYS=winsymlinks:nativestrict ln -s 1 $n; done

 echo warming up cache
 cat 1 >/dev/null

 echo ok lets go
-python3 "$sfx" -p39204 -e2dsa --dbd=yolo --exit=idx -lo=t -q "$@"
+$pybin "$sfx" -p39204 -e2dsa --dbd=yolo --exit=idx -lo=t -q "$@" && err= || err=$?
+[ $win ] && [ $err = 15 ] && err=  # sigterm doesn't hook on windows, ah whatever
+[ $err ] && echo ERROR $err && exit $err

 echo and the results are...
-$awk '/1 volumes in / {printf "%s MiB/s\n", 256*128/$(NF-1)}' <t
+LC_ALL=C $awk '/1 volumes in / {s=$(NF-1); printf "speed: %.1f MiB/s  (time=%.2fs)\n", '$totalsize'/s, s}' <t

 echo deleting $td and exiting

@@ -52,16 +63,30 @@ echo deleting $td and exiting

 # MiB/s @ cpu or device  (copyparty, pythonver, distro/os)  // comment

+#  3887 @ Ryzen 5 4500U  (cpp 1.9.5, nogil 3.9, fedora 39)  // --hash-mt=6; laptop
+#  3732 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.12.1, fedora 39)  // --hash-mt=6; laptop
 #  3608 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.5, fedora 38)  // --hash-mt=6; laptop
 #  2726 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.5, fedora 38)  // --hash-mt=4 (old-default)
 #  2202 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.5, docker-alpine 3.18.3) ??? alpine slow
 #  2719 @ Ryzen 5 4500U  (cpp 1.9.5, py 3.11.2, docker-debian 12.1)

+#  7746 @ mbp 2023 m3pro (cpp 1.9.5, py 3.11.7, macos 14.1)  // --hash-mt=6
+#  6687 @ mbp 2023 m3pro (cpp 1.9.5, py 3.11.7, macos 14.1)  // --hash-mt=5 (default)
 #  5544 @ Intel i5-12500 (cpp 1.9.5, py 3.11.2, debian 12.0)  // --hash-mt=12; desktop
 #  5197 @ Ryzen 7 3700X  (cpp 1.9.5, py 3.9.18, freebsd 13.2)  // --hash-mt=8; 2u server
+#  4551 @ mbp 2020 m1    (cpp 1.9.5, py 3.11.7, macos 14.2.1)
+#  4190 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.6, fedora 37)  // --hash-mt=8 (vbox-VM on win10-17763.4974)
+#  3028 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.6, fedora 37)  // --hash-mt=5 (vbox-VM on win10-17763.4974)
+#  2629 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.7, win10-ltsc-1809-17763.4974)  // --hash-mt=5 (default)
+#  2576 @ Ryzen 7 5800X  (cpp 1.9.5, py 3.11.7, win10-ltsc-1809-17763.4974)  // --hash-mt=8 (hello??)
 #  2606 @ Ryzen 7 3700X  (cpp 1.9.5, py 3.9.18, freebsd 13.2)  // --hash-mt=4 (old-default)
 #  1436 @ Ryzen 5 5500U  (cpp 1.9.5, py 3.11.4, alpine 3.18.3)  // nuc
 #  1065 @ Pixel 7        (cpp 1.9.5, py 3.11.5, termux 2023-09)
+#   945 @ Pi 5B v1.0     (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
+#   548 @ Pi 4B v1.5     (cpp 1.9.5, py 3.11.6, debian 11)
+#   435 @ Pi 4B v1.5     (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
+#   212 @ Pi Zero2W v1.0 (cpp 1.9.5, py 3.11.6, alpine 3.19.0)
+#  10.0 @ Pi Zero W v1.1 (cpp 1.9.5, py 3.11.6, alpine 3.19.0)

 # notes,
 # podman run --rm -it --shm-size 512m --entrypoint /bin/ash localhost/copyparty-min
--- a/scripts/logpack.sh
+++ b/scripts/logpack.sh
@@ -0,0 +1,73 @@
+#!/bin/bash
+set -e
+
+# recompress logs so they decompress faster + save some space;
+# * will not recurse into subfolders
+# * each file in current folder gets recompressed to zstd; input file is DELETED
+# * any xz-compressed logfiles are decompressed before converting to zstd
+# * SHOULD ignore and skip files which are currently open; SHOULD be safe to run while copyparty is running
+
+
+# for files larger than $cutoff, compress with `zstd -T0`
+# (otherwise do several files in parallel (scales better))
+cutoff=400M
+
+
+# osx support:
+# port install findutils gsed coreutils
+command -v gfind >/dev/null &&
+command -v gsed  >/dev/null &&
+command -v gsort >/dev/null && {
+    find() { gfind "$@"; }
+    sed()  { gsed  "$@"; }
+    sort() { gsort "$@"; }
+}
+
+packfun() {
+    local jobs=$1 fn="$2"
+    printf '%s\n' "$fn" | grep -qF .zst && return
+
+    local of="$(printf '%s\n' "$fn" | sed -r 's/\.(xz|txt)/.zst/')"
+    [ "$fn" = "$of" ] &&
+        of="$of.zst"
+
+    [ -e "$of" ] &&
+        echo "SKIP: output file exists: $of" &&
+        return
+
+    lsof -- "$fn" 2>/dev/null | grep -E .. &&
+        printf "SKIP: file in use: %s\n\n" $fn &&
+        return
+
+    # determine by header; old copyparty versions would produce xz output without .xz names
+    head -c3 "$fn" | grep -qF 7z &&
+        cmd="xz -dkc" || cmd="cat"
+
+    printf '<%s> T%d: %s\n' "$cmd" $jobs "$of"
+
+    $cmd <"$fn" >/dev/null || {
+        echo "ERROR: uncompress failed: $fn"
+        return
+    }
+
+    $cmd <"$fn" | zstd --long -19 -T$jobs >"$of"
+    touch -r "$fn" -- "$of"
+
+    cmp <($cmd <"$fn") <(zstd -d <"$of") || {
+        echo "ERROR: data mismatch: $of"
+        mv "$of"{,.BAD}
+        return
+    }
+    rm -- "$fn"
+}
+
+# do small files in parallel first (in descending size);
+# each file can use 4 threads in case the cutoff is poor
+export -f packfun
+export -f sed 2>/dev/null || true
+find -maxdepth 1 -type f -size -$cutoff -printf '%s %p\n' |
+sort -nr | sed -r 's`[^ ]+ ``; s`^\./``' | tr '\n' '\0' |
+xargs "$@" -0i -P$(nproc) bash -c 'packfun 4 "$@"' _ {}
+
+# then the big ones, letting each file use the whole cpu
+for f in *; do packfun 0 "$f"; done
--- a/scripts/sfx.py
+++ b/scripts/sfx.py
@@ -262,7 +262,7 @@ def unpack():
    final = opj(top, name)
    san = opj(final, "copyparty/up2k.py")
    for suf in range(0, 9001):
-        withpid = "{}.{}.{}".format(name, os.getpid(), suf)
+        withpid = "%s.%d.%s" % (name, os.getpid(), suf)
        mine = opj(top, withpid)
        if not ofe(mine):
            break
@@ -285,8 +285,8 @@ def unpack():

    ck = hashfile(tar)
    if ck != CKSUM:
-        t = "\n\nexpected {} ({} byte)\nobtained {} ({} byte)\nsfx corrupt"
-        raise Exception(t.format(CKSUM, SIZE, ck, sz))
+        t = "\n\nexpected %s (%d byte)\nobtained %s (%d byte)\nsfx corrupt"
+        raise Exception(t % (CKSUM, SIZE, ck, sz))

    with tarfile.open(tar, "r:bz2") as tf:
        # this is safe against traversal
Author	SHA1	Message	Date
ed	6987692021	filter apple metadata files when uploading with up2k.js: abandoned because it is ultimately pointless; browsers running on macos already do a decent job filtering these out, and there is only limited benefit of doing this for all other users, probably more harm than good in case of false positives	2024-01-11 23:42:31 +01:00
ed	e01ba8552a	warn if a user doesn't have privileges anywhere (since the account system isn't super-inutitive and at least one dude figured that -a would default to giving admin rights)	2024-01-11 00:24:34 +00:00
ed	024303592a	improved logging when a client dies mid-POST; igloo irc has an absolute time limit of 2 minutes before it just disconnects mid-upload and that kinda looked like it had a buggy multipart generator instead of just being funny anticipating similar events in the future, also log the client-selected boundary value to eyeball its yoloness	2024-01-10 23:59:43 +00:00
ed	86419b8f47	suboptimizations and some future safeguards	2024-01-10 23:20:42 +01:00
ed	f1358dbaba	use scandir for volume smoketests during up2k init; gives much faster startup on filesystems that are extremely slow (TLNote: android sdcardfs)	2024-01-09 21:47:02 +01:00
ed	e8a653ca0c	don't block non-up2k uploads during indexing; due to all upload APIs invoking up2k.hash_file to index uploads, the uploads could block during a rescan for a crazy long time (past most gateway timeouts); now this is mostly fire-and-forget "mostly" because this also adds a conditional slowdown to help the hasher churn through if the queue gets too big worst case, if the server is restarted before it catches up, this would rely on filesystem reindexing to eventually index the files after a restart or on a schedule, meaning uploader info would be lost on shutdown, but this is usually fine anyways (and this was also the case until now)	2024-01-08 22:10:16 +00:00
ed	9bc09ce949	accept file POSTs without specifying the act field; primarily to support uploading from Igloo IRC but also generally useful (not actually tested with Igloo IRC yet because it's a paid feature so just gonna wait for spiky to wake up and tell me it didn't work)	2024-01-08 19:09:53 +00:00
ed	dc8e621d7c	increase OOM kill-score for FFmpeg and mtp's; discourage Linux from killing innocent processes when FFmpeg decides to allocate 1 TiB of RAM	2024-01-07 17:52:10 +00:00
ed	dee0950f74	misc; * scripts: add log repacker * bench/filehash: msys support + add more stats	2024-01-06 01:15:43 +00:00
ed	143f72fe36	bench/filehash: fix locale + add more stats	2024-01-03 02:41:18 +01:00
ed	a7889fb6a2	update pkgs to 1.9.28	2023-12-31 19:44:24 +00:00