v1.15.0

up2k.js: bump handshake timeout for safededup
disable upload deduplication by default;
2024-09-08 19:25:46 +00:00 · 2024-09-08 18:06:37 +00:00 · 2024-09-08 17:09:14 +00:00 · 2024-09-08 12:55:27 +00:00 · 2024-09-08 12:12:34 +00:00 · 2024-09-08 00:44:22 +00:00
28 changed files with 820 additions and 246 deletions
--- a/README.md
+++ b/README.md
@@ -65,7 +65,8 @@ turn almost any device into a file server with resumable uploads/downloads using
    * [smb server](#smb-server) - unsafe, slow, not recommended for wan
    * [browser ux](#browser-ux) - tweaking the ui
    * [opengraph](#opengraph) - discord and social-media embeds
-    * [file indexing](#file-indexing) - enables dedup and music search ++
+    * [file deduplication](#file-deduplication) - enable symlink-based upload deduplication
+    * [file indexing](#file-indexing) - enable music search, upload-undo, and better dedup
        * [exclude-patterns](#exclude-patterns) - to save some time
        * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
        * [periodic rescan](#periodic-rescan) - filesystem monitoring
@@ -1155,9 +1156,41 @@ NOTE: because discord (and maybe others) strip query args such as `?raw` in open
 if you want to entirely replace the copyparty response with your own jinja2 template, give the template filepath to `--og-tpl` or volflag `og_tpl` (all members of `HttpCli` are available through the `this` object)


+## file deduplication
+
+enable symlink-based upload deduplication  globally with `--dedup` or per-volume with volflag `dedup`
+
+when someone tries to upload a file that already exists on the server, the upload will be politely declined and a symlink is created instead, pointing to the nearest copy on disk, thus reducinc disk space usage
+
+**warning:** when enabling dedup, you should also:
+* enable indexing with `-e2dsa` or volflag `e2dsa` (see [file indexing](#file-indexing) section below); strongly recommended
+* ...and/or `--hardlink-only` to use hardlink-based deduplication instead of symlinks; see explanation below
+
+it will not be safe to rename/delete files if you only enable dedup and none of the above; if you enable indexing then it is not *necessary* to also do hardlinks (but you may still want to)
+
+by default, deduplication is done based on symlinks (symbolic links); these are tiny files which are pointers to the nearest full copy of the file
+
+you can choose to use hardlinks instead of softlinks, globally with `--hardlink-only` or volflag `hardlinkonly`;
+
+advantages of using hardlinks:
+* hardlinks are more compatible with other software; they behave entirely like regular files
+* you can safely move and rename files using other file managers
+  * symlinks need to be managed by copyparty to ensure the destinations remain correct
+
+advantages of using symlinks (default):
+* each symlink can have its own last-modified timestamp, but a single timestamp is shared by all hardlinks
+* symlinks make it more obvious to other software that the file is not a regular file, so this can be less dangerous
+  * hardlinks look like regular files, so other software may assume they are safe to edit without affecting the other copies
+
+**warning:** if you edit the contents of a deduplicated file, then you will also edit all other copies of that file! This is especially surprising with hardlinks, because they look like regular files, but that same file exists in multiple locations
+
+global-option `--xlink` / volflag `xlink` additionally enables deduplication across volumes, but this is probably buggy and not recommended
+
+
+
 ## file indexing

-enables dedup and music search ++
+enable music search, upload-undo, and better dedup

 file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volflags, or a mix of both.

@@ -1171,7 +1204,6 @@ through arguments:
 * `-e2v` verfies file integrity at startup, comparing hashes from the db
 * `-e2vu` patches the database with the new hashes from the filesystem
 * `-e2vp` panics and kills copyparty instead
-* `--xlink` enables deduplication across volumes

 the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts`, `d2v` for disabling:
 * `-v ~/music::r:c,e2ds,e2tsr` does a full reindex of everything on startup
@@ -1184,7 +1216,6 @@ note:
 * upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
-* deduplication is possible on windows if you run copyparty as administrator (not saying you should!)

 ### exclude-patterns

@@ -1924,6 +1955,8 @@ below are some tweaks roughly ordered by usefulness:
 * `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
 * `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
  * and also makes thumbnails load faster, regardless of e2d/e2t
+* `--dedup` enables deduplication and thus avoids writing to the HDD if someone uploads a dupe
+* `--safe-dedup 1` makes deduplication much faster during upload by skipping verification of file contents; safe if there is no other software editing/moving the files in the volumes
 * `--no-hash .` when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable
 * if your volumes are on a network-disk such as NFS / SMB / s3, specifying larger values for `--iobuf` and/or `--s-rd-sz` and/or `--s-wr-sz` may help; try setting all of them to `524288` or `1048576` or `4194304`
 * `--no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1` minimizes the number of threads; can help in some eccentric environments (like the vscode debugger)
@@ -1978,7 +2011,7 @@ safety profiles:
  * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
    * however note if you edit one file it will also affect the other copies
  * `--vague-403` returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
-  * `--nih` removes the server hostname from directory listings
+  * `-nih` removes the server hostname from directory listings

 * option `-sss` is a shortcut for the above plus:
  * `--no-dav` disables webdav support
--- a/bin/u2c.py
+++ b/bin/u2c.py
@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals

-S_VERSION = "1.23"
-S_BUILD_DT = "2024-08-22"
+S_VERSION = "1.24"
+S_BUILD_DT = "2024-09-05"

 """
 u2c.py: upload to copyparty
@@ -41,19 +41,25 @@ except:

 try:
    import requests
+
+    req_ses = requests.Session()
 except ImportError as ex:
-    if EXE:
+    if "-" in sys.argv or "-h" in sys.argv:
+        m = ""
+    elif EXE:
        raise
    elif sys.version_info > (2, 7):
-        m = "\nERROR: need 'requests'; please run this command:\n {0} -m pip install --user requests\n"
+        m = "\nERROR: need 'requests'{0}; please run this command:\n {1} -m pip install --user requests\n"
    else:
        m = "requests/2.18.4 urllib3/1.23 chardet/3.0.4 certifi/2020.4.5.1 idna/2.7"
        m = ["   https://pypi.org/project/" + x + "/#files" for x in m.split()]
-        m = "\n  ERROR: need these:\n" + "\n".join(m) + "\n"
+        m = "\n  ERROR: need these{0}:\n" + "\n".join(m) + "\n"
        m += "\n  for f in *.whl; do unzip $f; done; rm -r *.dist-info\n"

-    print(m.format(sys.executable), "\nspecifically,", ex)
-    sys.exit(1)
+    if m:
+        t = " when not running with '-h' or url '-'"
+        print(m.format(t, sys.executable), "\nspecifically,", ex)
+        sys.exit(1)


 # from copyparty/__init__.py
@@ -76,7 +82,22 @@ else:
 VT100 = platform.system() != "Windows"


-req_ses = requests.Session()
+try:
+    UTC = datetime.timezone.utc
+except:
+    TD_ZERO = datetime.timedelta(0)
+
+    class _UTC(datetime.tzinfo):
+        def utcoffset(self, dt):
+            return TD_ZERO
+
+        def tzname(self, dt):
+            return "UTC"
+
+        def dst(self, dt):
+            return TD_ZERO
+
+    UTC = _UTC()


 class Daemon(threading.Thread):
@@ -271,6 +292,12 @@ class MTHash(object):
 _print = print


+def safe_print(*a, **ka):
+    ka["end"] = ""
+    zs = " ".join([unicode(x) for x in a])
+    _print(zs + "\n", **ka)
+
+
 def eprint(*a, **ka):
    ka["file"] = sys.stderr
    ka["end"] = ""
@@ -284,18 +311,17 @@ def eprint(*a, **ka):

 def flushing_print(*a, **ka):
    try:
-        _print(*a, **ka)
+        safe_print(*a, **ka)
    except:
        v = " ".join(str(x) for x in a)
        v = v.encode("ascii", "replace").decode("ascii")
-        _print(v, **ka)
+        safe_print(v, **ka)

    if "flush" not in ka:
        sys.stdout.flush()


-if not VT100:
-    print = flushing_print
+print = safe_print if VT100 else flushing_print


 def termsize():
@@ -770,8 +796,6 @@ class Ctl(object):
            self.up_c = 0
            self.up_b = 0
            self.up_br = 0
-            self.hasher_busy = 1
-            self.handshaker_busy = 0
            self.uploader_busy = 0
            self.serialized = False

@@ -781,6 +805,9 @@ class Ctl(object):
            self.eta = "99:99:99"

            self.mutex = threading.Lock()
+            self.exit_cond = threading.Condition()
+            self.uploader_alive = ar.j
+            self.handshaker_alive = ar.j
            self.q_handshake = Queue()  # type: Queue[File]
            self.q_upload = Queue()  # type: Queue[FileSlice]

@@ -851,27 +878,21 @@ class Ctl(object):
            Daemon(self.handshaker)
            Daemon(self.uploader)

-        idles = 0
-        while idles < 3:
-            time.sleep(0.07)
+        while True:
+            with self.exit_cond:
+                self.exit_cond.wait(0.07)
            with self.mutex:
-                if (
-                    self.q_handshake.empty()
-                    and self.q_upload.empty()
-                    and not self.hasher_busy
-                    and not self.handshaker_busy
-                    and not self.uploader_busy
-                ):
-                    idles += 1
-                else:
-                    idles = 0
+                if not self.handshaker_alive and not self.uploader_alive:
+                    break
+                st_hash = self.st_hash[:]
+                st_up = self.st_up[:]

            if VT100 and not self.ar.ns:
                maxlen = ss.w - len(str(self.nfiles)) - 14
                txt = "\033[s\033[{0}H".format(ss.g)
                for y, k, st, f in [
-                    [0, "hash", self.st_hash, self.hash_f],
-                    [1, "send", self.st_up, self.up_f],
+                    [0, "hash", st_hash, self.hash_f],
+                    [1, "send", st_up, self.up_f],
                ]:
                    txt += "\033[{0}H{1}:".format(ss.g + y, k)
                    file, arg = st
@@ -1027,11 +1048,42 @@ class Ctl(object):
                self.hash_f += 1
                self.hash_c += len(file.cids)
                self.hash_b += file.size
+                if self.ar.wlist:
+                    self.up_f = self.hash_f
+                    self.up_c = self.hash_c
+                    self.up_b = self.hash_b
+
+            if self.ar.wlist:
+                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.kchunks]
+                zb = hashlib.sha512("\n".join(zsl).encode("utf-8")).digest()[:33]
+                wark = base64.urlsafe_b64encode(zb).decode("utf-8")
+                vp = file.rel.decode("utf-8")
+                if self.ar.jw:
+                    print("%s  %s" % (wark, vp))
+                else:
+                    zd = datetime.datetime.fromtimestamp(file.lmod, UTC)
+                    dt = "%04d-%02d-%02d %02d:%02d:%02d" % (
+                        zd.year,
+                        zd.month,
+                        zd.day,
+                        zd.hour,
+                        zd.minute,
+                        zd.second,
+                    )
+                    print("%s %12d %s %s" % (dt, file.size, wark, vp))
+                continue

            self.q_handshake.put(file)

-        self.hasher_busy = 0
        self.st_hash = [None, "(finished)"]
+        self._check_if_done()
+
+    def _check_if_done(self):
+        with self.mutex:
+            if self.nfiles - self.up_f:
+                return
+        for _ in range(self.ar.j):
+            self.q_handshake.put(None)

    def handshaker(self):
        search = self.ar.s
@@ -1039,8 +1091,10 @@ class Ctl(object):
        while True:
            file = self.q_handshake.get()
            if not file:
+                with self.mutex:
+                    self.handshaker_alive -= 1
                self.q_upload.put(None)
-                break
+                return

            upath = file.abs.decode("utf-8", "replace")
            if not VT100:
@@ -1052,9 +1106,6 @@ class Ctl(object):
                self.errs += 1
                continue

-            with self.mutex:
-                self.handshaker_busy += 1
-
            while time.time() < file.cd:
                time.sleep(0.1)

@@ -1062,17 +1113,17 @@ class Ctl(object):
            if search:
                if hs:
                    for hit in hs:
-                        t = "found: {0}\n  {1}{2}\n"
-                        print(t.format(upath, burl, hit["rp"]), end="")
+                        t = "found: {0}\n  {1}{2}"
+                        print(t.format(upath, burl, hit["rp"]))
                else:
-                    print("NOT found: {0}\n".format(upath), end="")
+                    print("NOT found: {0}".format(upath))

                with self.mutex:
                    self.up_f += 1
                    self.up_c += len(file.cids)
                    self.up_b += file.size
-                    self.handshaker_busy -= 1

+                self._check_if_done()
                continue

            if file.recheck:
@@ -1104,7 +1155,6 @@ class Ctl(object):
                        file.up_b -= sz

                file.ucids = hs
-                self.handshaker_busy -= 1

            if not hs:
                self.at_hash += file.t_hash
@@ -1130,6 +1180,9 @@ class Ctl(object):
                    kw = "uploaded" if file.up_b else "   found"
                    print("{0} {1}".format(kw, upath))

+                self._check_if_done()
+                continue
+
            chunksz = up2k_chunksize(file.size)
            njoin = (self.ar.sz * 1024 * 1024) // chunksz
            cs = hs[:]
@@ -1149,8 +1202,16 @@ class Ctl(object):
        while True:
            fsl = self.q_upload.get()
            if not fsl:
-                self.st_up = [None, "(finished)"]
-                break
+                done = False
+                with self.mutex:
+                    self.uploader_alive -= 1
+                    if not self.uploader_alive:
+                        done = not self.handshaker_alive
+                        self.st_up = [None, "(finished)"]
+                if done:
+                    with self.exit_cond:
+                        self.exit_cond.notify_all()
+                return

            file = fsl.file
            cids = fsl.cids
@@ -1252,6 +1313,10 @@ source file/folder selection uses rsync syntax, meaning that:
    ap.add_argument("--dr", action="store_true", help="delete remote files which don't exist locally (implies --ow)")
    ap.add_argument("--drd", action="store_true", help="delete remote files during upload instead of afterwards; reduces peak disk space usage, but will reupload instead of detecting renames")

+    ap = app.add_argument_group("file-ID calculator; enable with url '-' to list warks (file identifiers) instead of upload/search")
+    ap.add_argument("--wsalt", type=unicode, metavar="S", default="hunter2", help="salt to use when creating warks; must match server config")
+    ap.add_argument("--jw", action="store_true", help="just identifier+filepath, not mtime/size too")
+
    ap = app.add_argument_group("performance tweaks")
    ap.add_argument("-j", type=int, metavar="CONNS", default=2, help="parallel connections")
    ap.add_argument("-J", type=int, metavar="CORES", default=hcores, help="num cpu-cores to use for hashing; set 0 or 1 for single-core hashing")
@@ -1285,7 +1350,9 @@ source file/folder selection uses rsync syntax, meaning that:

    ar.x = "|".join(ar.x or [])

-    for k in "dl dr drd".split():
+    setattr(ar, "wlist", ar.url == "-")
+
+    for k in "dl dr drd wlist".split():
        errs = []
        if ar.safe and getattr(ar, k):
            errs.append(k)
--- a/contrib/package/arch/PKGBUILD
+++ b/contrib/package/arch/PKGBUILD
@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.14.3"
+pkgver="1.14.4"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -21,7 +21,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("7fcdcec0d7b118bf17a98b1a409331dcfc0fbbd431b362b991f5800006fd8c98")
+sha256sums=("1e8004e4369e59487c47a0a9949668de704b1884beda0421887e342edcff0961")

 build() {
    cd "${srcdir}/${pkgname}-${pkgver}"
--- a/contrib/package/nix/copyparty/pin.json
+++ b/contrib/package/nix/copyparty/pin.json
@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.14.3/copyparty-sfx.py",
-    "version": "1.14.3",
-    "hash": "sha256-1yVeJfYnyNNKYX3KdmYP0ECx7K8EjuWvApMw0diJ1sk="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.14.4/copyparty-sfx.py",
+    "version": "1.14.4",
+    "hash": "sha256-nfcSXddM0jeHr7nGwoguOv/eq750bpHrImKmmODxK6E="
 }
--- a/copyparty/main.py
+++ b/copyparty/main.py
@@ -992,9 +992,10 @@ def add_upload(ap):
    ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without \033[33m-e2d\033[0m; roughly 1 MiB RAM per 600")
    ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (bad idea to enable this on windows and/or cow filesystems)")
    ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
-    ap2.add_argument("--hardlink", action="store_true", help="prefer hardlinks instead of symlinks when possible (within same filesystem) (volflag=hardlink)")
-    ap2.add_argument("--never-symlink", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=neversymlink)")
-    ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead (volflag=copydupes)")
+    ap2.add_argument("--dedup", action="store_true", help="enable symlink-based upload deduplication (volflag=dedup)")
+    ap2.add_argument("--safe-dedup", metavar="N", type=int, default=50, help="how careful to be when deduplicating files; [\033[32m1\033[0m] = just verify the filesize, [\033[32m50\033[0m] = verify file contents have not been altered (volflag=safededup)")
+    ap2.add_argument("--hardlink", action="store_true", help="enable hardlink-based dedup; will fallback on symlinks when that is impossible (across filesystems) (volflag=hardlink)")
+    ap2.add_argument("--hardlink-only", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=hardlinkonly)")
    ap2.add_argument("--no-dupe", action="store_true", help="reject duplicate files during upload; only matches within the same volume (volflag=nodupe)")
    ap2.add_argument("--no-snap", action="store_true", help="disable snapshots -- forget unfinished uploads on shutdown; don't create .hist/up2k.snap files -- abandoned/interrupted uploads must be cleaned up manually")
    ap2.add_argument("--snap-wri", metavar="SEC", type=int, default=300, help="write upload state to ./hist/up2k.snap every \033[33mSEC\033[0m seconds; allows resuming incomplete uploads after a server crash")
@@ -1285,6 +1286,7 @@ def add_logging(ap):
    ap2.add_argument("--ansi", action="store_true", help="force colors; overrides environment-variable NO_COLOR")
    ap2.add_argument("--no-logflush", action="store_true", help="don't flush the logfile after each write; tiny bit faster")
    ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
+    ap2.add_argument("--log-utc", action="store_true", help="do not use local timezone; assume the TZ env-var is UTC (tiny bit faster)")
    ap2.add_argument("--log-tdec", metavar="N", type=int, default=3, help="timestamp resolution / number of timestamp decimals")
    ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log failed login attempt passwords: 0=terse, 1=plaintext, 2=hashed")
    ap2.add_argument("--log-conn", action="store_true", help="debug: print tcp-server msgs")
@@ -1343,7 +1345,7 @@ def add_transcoding(ap):
 def add_db_general(ap, hcores):
    noidx = APPLESAN_TXT if MACOS else ""
    ap2 = ap.add_argument_group('general db options')
-    ap2.add_argument("-e2d", action="store_true", help="enable up2k database, making files searchable + enables upload deduplication")
+    ap2.add_argument("-e2d", action="store_true", help="enable up2k database; this enables file search, upload-undo, improves deduplication")
    ap2.add_argument("-e2ds", action="store_true", help="scan writable folders for new files on startup; sets \033[33m-e2d\033[0m")
    ap2.add_argument("-e2dsa", action="store_true", help="scans all folders on startup; sets \033[33m-e2ds\033[0m")
    ap2.add_argument("-e2v", action="store_true", help="verify file integrity; rehash all files and compare with db")
@@ -1356,7 +1358,7 @@ def add_db_general(ap, hcores):
    ap2.add_argument("--re-dhash", action="store_true", help="force a cache rebuild on startup; enable this once if it gets out of sync (should never be necessary)")
    ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice -- only useful for offloading uploads to a cloud service or something (volflag=noforget)")
    ap2.add_argument("--dbd", metavar="PROFILE", default="wal", help="database durability profile; sets the tradeoff between robustness and speed, see \033[33m--help-dbd\033[0m (volflag=dbd)")
-    ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (volflag=xlink)")
+    ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (probably buggy, not recommended) (volflag=xlink)")
    ap2.add_argument("--hash-mt", metavar="CORES", type=int, default=hcores, help="num cpu cores to use for file hashing; set 0 or 1 for single-core hashing")
    ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="rescan filesystem for changes every \033[33mSEC\033[0m seconds; 0=off (volflag=scan)")
    ap2.add_argument("--db-act", metavar="SEC", type=float, default=10.0, help="defer any scheduled volume reindexing until \033[33mSEC\033[0m seconds after last db write (uploads, renames, ...)")
@@ -1619,6 +1621,7 @@ def main(argv: Optional[list[str]] = None, rsrc: Optional[str] = None) -> None:
        ("--hdr-au-usr", "--idp-h-usr"),
        ("--idp-h-sep", "--idp-gsep"),
        ("--th-no-crop", "--th-crop=n"),
+        ("--never-symlink", "--hardlink-only"),
    ]
    for dk, nk in deprecated:
        idx = -1
@@ -1643,7 +1646,7 @@ def main(argv: Optional[list[str]] = None, rsrc: Optional[str] = None) -> None:
            argv.extend(["--qr"])
            if ANYWIN or not os.geteuid():
                # win10 allows symlinks if admin; can be unexpected
-                argv.extend(["-p80,443,3923", "--ign-ebind", "--no-dedup"])
+                argv.extend(["-p80,443,3923", "--ign-ebind"])
    except:
        pass

--- a/copyparty/version.py
+++ b/copyparty/version.py
@@ -1,8 +1,8 @@
 # coding: utf-8

-VERSION = (1, 14, 4)
-CODENAME = "one step forward"
-BUILD_DT = (2024, 9, 2)
+VERSION = (1, 15, 0)
+CODENAME = "fill the drives"
+BUILD_DT = (2024, 9, 8)

 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)
--- a/copyparty/authsrv.py
+++ b/copyparty/authsrv.py
@@ -1891,6 +1891,11 @@ class AuthSrv(object):
            if len(zs) == 3:  # fc5 => ffcc55
                vol.flags["tcolor"] = "".join([x * 2 for x in zs])

+            if vol.flags.get("neversymlink"):
+                vol.flags["hardlinkonly"] = True  # was renamed
+            if vol.flags.get("hardlinkonly"):
+                vol.flags["hardlink"] = True
+
            for k1, k2 in IMPLICATIONS:
                if k1 in vol.flags:
                    vol.flags[k2] = True
@@ -1995,9 +2000,6 @@ class AuthSrv(object):
                    for x in drop:
                        vol.flags.pop(x)

-            if vol.flags.get("neversymlink") and not vol.flags.get("hardlink"):
-                vol.flags["copydupes"] = True
-
            # verify tags mentioned by -mt[mp] are used by -mte
            local_mtp = {}
            local_only_mtp = {}
@@ -2076,6 +2078,8 @@ class AuthSrv(object):

        have_e2d = False
        have_e2t = False
+        have_dedup = False
+        unsafe_dedup = []
        t = "volumes and permissions:\n"
        for zv in vfs.all_vols.values():
            if not self.warn_anonwrite or verbosity < 5:
@@ -2108,6 +2112,11 @@ class AuthSrv(object):
            if "e2t" in zv.flags:
                have_e2t = True

+            if "dedup" in zv.flags:
+                have_dedup = True
+                if "e2d" not in zv.flags and "hardlink" not in zv.flags:
+                    unsafe_dedup.append("/" + zv.vpath)
+
            t += "\n"

        if self.warn_anonwrite and verbosity > 4:
@@ -2120,10 +2129,17 @@ class AuthSrv(object):
                    self.log("\n\033[{}\033[0m\n".format(t))

                if not have_e2t:
-                    t = "hint: argument -e2ts enables multimedia indexing (artist/title/...)"
+                    t = "hint: enable multimedia indexing (artist/title/...) with argument -e2ts"
                    self.log(t, 6)
            else:
-                t = "hint: argument -e2dsa enables searching, upload-undo, and better deduplication"
+                t = "hint: enable searching and upload-undo with argument -e2dsa"
+                self.log(t, 6)
+
+            if unsafe_dedup:
+                t = "WARNING: symlink-based deduplication is enabled for some volumes, but without indexing. Please enable -e2dsa and/or --hardlink to avoid problems when moving/renaming files. Affected volumes: %s"
+                self.log(t % (", ".join(unsafe_dedup)), 3)
+            elif not have_dedup:
+                t = "hint: enable upload deduplication with --dedup (but see readme for consequences)"
                self.log(t, 6)

            zv, _ = vfs.get("/", "*", False, False)
--- a/copyparty/cfg.py
+++ b/copyparty/cfg.py
@@ -12,8 +12,7 @@ def vf_bmap() -> dict[str, str]:
        "dav_auth": "davauth",
        "dav_rt": "davrt",
        "ed": "dots",
-        "never_symlink": "neversymlink",
-        "no_dedup": "copydupes",
+        "hardlink_only": "hardlinkonly",
        "no_dupe": "nodupe",
        "no_forget": "noforget",
        "no_pipe": "nopipe",
@@ -23,6 +22,7 @@ def vf_bmap() -> dict[str, str]:
        "no_athumb": "dathumb",
    }
    for k in (
+        "dedup",
        "dotsrch",
        "e2d",
        "e2ds",
@@ -58,6 +58,7 @@ def vf_vmap() -> dict[str, str]:
        "no_hash": "nohash",
        "no_idx": "noidx",
        "re_maxage": "scan",
+        "safe_dedup": "safededup",
        "th_convt": "convt",
        "th_size": "thsize",
        "th_crop": "crop",
@@ -129,10 +130,11 @@ permdescs = {

 flagcats = {
    "uploads, general": {
+        "dedup": "enable symlink-based file deduplication",
+        "hardlink": "enable hardlink-based file deduplication,\nwith fallback on symlinks when that is impossible",
+        "hardlinkonly": "dedup with hardlink only, never symlink;\nmake a full copy if hardlink is impossible",
+        "safededup": "verify on-disk data before using it for dedup",
        "nodupe": "rejects existing files (instead of symlinking them)",
-        "hardlink": "does dedup with hardlinks instead of symlinks",
-        "neversymlink": "disables symlink fallback; full copy instead",
-        "copydupes": "disables dedup, always saves full copies of dupes",
        "sparse": "force use of sparse files, mainly for s3-backed storage",
        "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
        "nosub": "forces all uploads into the top folder of the vfs",
@@ -159,7 +161,7 @@ flagcats = {
        "lifetime=3600": "uploads are deleted after 1 hour",
    },
    "database, general": {
-        "e2d": "enable database; makes files searchable + enables upload dedup",
+        "e2d": "enable database; makes files searchable + enables upload-undo",
        "e2ds": "scan writable folders for new files on startup; also sets -e2d",
        "e2dsa": "scans all folders for new files on startup; also sets -e2d",
        "e2t": "enable multimedia indexing; makes it possible to search for tags",
@@ -177,7 +179,7 @@ flagcats = {
        "noforget": "don't forget files when deleted from disk",
        "fat32": "avoid excessive reindexing on android sdcardfs",
        "dbd=[acid|swal|wal|yolo]": "database speed-durability tradeoff",
-        "xlink": "cross-volume dupe detection / linking",
+        "xlink": "cross-volume dupe detection / linking (dangerous)",
        "xdev": "do not descend into other filesystems",
        "xvol": "do not follow symlinks leaving the volume root",
        "dotsrch": "show dotfiles in search results",
--- a/copyparty/svchub.py
+++ b/copyparty/svchub.py
@@ -3,7 +3,6 @@ from __future__ import print_function, unicode_literals

 import argparse
 import base64
-import calendar
 import errno
 import gzip
 import logging
@@ -16,7 +15,7 @@ import string
 import sys
 import threading
 import time
-from datetime import datetime, timedelta
+from datetime import datetime

 # from inspect import currentframe
 # print(currentframe().f_lineno)
@@ -104,6 +103,7 @@ class SvcHub(object):
        self.argv = argv
        self.E: EnvParams = args.E
        self.no_ansi = args.no_ansi
+        self.tz = UTC if args.log_utc else None
        self.logf: Optional[typing.TextIO] = None
        self.logf_base_fn = ""
        self.is_dut = False  # running in unittest; always False
@@ -118,7 +118,8 @@ class SvcHub(object):
        self.httpsrv_up = 0

        self.log_mutex = threading.Lock()
-        self.next_day = 0
+        self.cday = 0
+        self.cmon = 0
        self.tstack = 0.0

        self.iphash = HMaccas(os.path.join(self.E.cfg, "iphash"), 8)
@@ -791,7 +792,7 @@ class SvcHub(object):
            self.args.nc = min(self.args.nc, soft // 2)

    def _logname(self) -> str:
-        dt = datetime.now(UTC)
+        dt = datetime.now(self.tz)
        fn = str(self.args.lo)
        for fs in "YmdHMS":
            fs = "%" + fs
@@ -1064,12 +1065,12 @@ class SvcHub(object):
            return

        with self.log_mutex:
-            zd = datetime.now(UTC)
+            dt = datetime.now(self.tz)
            ts = self.log_dfmt % (
-                zd.year,
-                zd.month * 100 + zd.day,
-                (zd.hour * 100 + zd.minute) * 100 + zd.second,
-                zd.microsecond // self.log_div,
+                dt.year,
+                dt.month * 100 + dt.day,
+                (dt.hour * 100 + dt.minute) * 100 + dt.second,
+                dt.microsecond // self.log_div,
            )

            if c and not self.args.no_ansi:
@@ -1090,41 +1091,26 @@ class SvcHub(object):
            if not self.args.no_logflush:
                self.logf.flush()

-            now = time.time()
-            if int(now) >= self.next_day:
-                self._set_next_day()
+            if dt.day != self.cday or dt.month != self.cmon:
+                self._set_next_day(dt)

-    def _set_next_day(self) -> None:
-        if self.next_day and self.logf and self.logf_base_fn != self._logname():
+    def _set_next_day(self, dt: datetime) -> None:
+        if self.cday and self.logf and self.logf_base_fn != self._logname():
            self.logf.close()
            self._setup_logfile("")

-        dt = datetime.now(UTC)
-
-        # unix timestamp of next 00:00:00 (leap-seconds safe)
-        day_now = dt.day
-        while dt.day == day_now:
-            dt += timedelta(hours=12)
-
-        dt = dt.replace(hour=0, minute=0, second=0)
-        try:
-            tt = dt.utctimetuple()
-        except:
-            # still makes me hella uncomfortable
-            tt = dt.timetuple()
-
-        self.next_day = calendar.timegm(tt)
+        self.cday = dt.day
+        self.cmon = dt.month

    def _log_enabled(self, src: str, msg: str, c: Union[int, str] = 0) -> None:
        """handles logging from all components"""
        with self.log_mutex:
-            now = time.time()
-            if int(now) >= self.next_day:
-                dt = datetime.fromtimestamp(now, UTC)
+            dt = datetime.now(self.tz)
+            if dt.day != self.cday or dt.month != self.cmon:
                zs = "{}\n" if self.no_ansi else "\033[36m{}\033[0m\n"
                zs = zs.format(dt.strftime("%Y-%m-%d"))
                print(zs, end="")
-                self._set_next_day()
+                self._set_next_day(dt)
                if self.logf:
                    self.logf.write(zs)

@@ -1143,12 +1129,11 @@ class SvcHub(object):
                else:
                    msg = "%s%s\033[0m" % (c, msg)

-            zd = datetime.fromtimestamp(now, UTC)
            ts = self.log_efmt % (
-                zd.hour,
-                zd.minute,
-                zd.second,
-                zd.microsecond // self.log_div,
+                dt.hour,
+                dt.minute,
+                dt.second,
+                dt.microsecond // self.log_div,
            )
            msg = fmt % (ts, src, msg)
            try:
--- a/copyparty/up2k.py
+++ b/copyparty/up2k.py
@@ -1462,7 +1462,7 @@ class Up2k(object):
                    self.log("file: {}".format(abspath))

                try:
-                    hashes = self._hashlist_from_file(
+                    hashes, _ = self._hashlist_from_file(
                        abspath, "a{}, ".format(self.pp.n)
                    )
                except Exception as ex:
@@ -1656,6 +1656,7 @@ class Up2k(object):
            qex = " where " + qex

        rewark: list[tuple[str, str, str, int, int]] = []
+        f404: list[tuple[str, str, str]] = []

        with self.mutex:
            b_left = 0
@@ -1672,7 +1673,8 @@ class Up2k(object):
                if self.stop:
                    return -1

-                w, drd, dfn = zb[:-1].decode("utf-8").split("\x00")
+                zs = zb[:-1].decode("utf-8").replace("\x00\x02", "\n")
+                w, drd, dfn = zs.split("\x00\x01")
                with self.mutex:
                    q = "select mt, sz from up where rd=? and fn=? and +w=?"
                    try:
@@ -1698,9 +1700,14 @@ class Up2k(object):
                pf = "v{}, {:.0f}+".format(n_left, b_left / 1024 / 1024)
                self.pp.msg = pf + abspath

-                # throws on broken symlinks (always did)
-                stl = bos.lstat(abspath)
-                st = bos.stat(abspath) if stat.S_ISLNK(stl.st_mode) else stl
+                try:
+                    stl = bos.lstat(abspath)
+                    st = bos.stat(abspath) if stat.S_ISLNK(stl.st_mode) else stl
+                except Exception as ex:
+                    self.log("missing file: %s" % (abspath,), 3)
+                    f404.append((drd, dfn, w))
+                    continue
+
                mt2 = int(stl.st_mtime)
                sz2 = st.st_size

@@ -1711,7 +1718,7 @@ class Up2k(object):
                        self.log("file: {}".format(abspath))

                    try:
-                        hashes = self._hashlist_from_file(abspath, pf)
+                        hashes, _ = self._hashlist_from_file(abspath, pf)
                    except Exception as ex:
                        self.log("hash: {} @ [{}]".format(repr(ex), abspath))
                        continue
@@ -1737,12 +1744,15 @@ class Up2k(object):
                t = t.format(abspath, w, sz, mt, w2, sz2, mt2)
                self.log(t, 1)

-        if e2vp and rewark:
+        if e2vp and (rewark or f404):
            self.hub.retcode = 1
            Daemon(self.hub.sigterm)
-            raise Exception("{} files have incorrect hashes".format(len(rewark)))
+            t = "in volume /%s:  %s files missing, %s files have incorrect hashes"
+            t = t % (vol.vpath, len(f404), len(rewark))
+            self.log(t, 1)
+            raise Exception(t)

-        if not e2vu or not rewark:
+        if not e2vu or (not rewark and not f404):
            return 0

        with self.mutex:
@@ -1750,9 +1760,13 @@ class Up2k(object):
                q = "update up set w = ?, sz = ?, mt = ? where rd = ? and fn = ? limit 1"
                cur.execute(q, (w, sz, int(mt), rd, fn))

+            for _, _, w in f404:
+                q = "delete from up where w = ? limit 1"
+                cur.execute(q, (w,))
+
            cur.connection.commit()

-        return len(rewark)
+        return len(rewark) + len(f404)

    def _build_tags_index(self, vol: VFS) -> tuple[int, int, bool]:
        ptop = vol.realpath
@@ -1967,7 +1981,8 @@ class Up2k(object):
                    if c2.execute(q, (row[0][:16],)).fetchone():
                        continue

-                gf.write(("%s\n" % ("\x00".join(row),)).encode("utf-8"))
+                zs = "\x00\x01".join(row).replace("\n", "\x00\x02")
+                gf.write((zs + "\n").encode("utf-8"))
                n += 1

        c2.close()
@@ -2666,10 +2681,13 @@ class Up2k(object):
            jcur = self.cur.get(ptop)
            reg = self.registry[ptop]
            vfs = self.asrv.vfs.all_vols[cj["vtop"]]
-            n4g = vfs.flags.get("noforget")
+            n4g = bool(vfs.flags.get("noforget"))
            rand = vfs.flags.get("rand") or cj.get("rand")
            lost: list[tuple["sqlite3.Cursor", str, str]] = []

+            safe_dedup = vfs.flags.get("safededup") or 50
+            data_ok = safe_dedup < 10 or n4g
+
            vols = [(ptop, jcur)] if jcur else []
            if vfs.flags.get("xlink"):
                vols += [(k, v) for k, v in self.cur.items() if k != ptop]
@@ -2677,7 +2695,7 @@ class Up2k(object):
                # force upload time rather than last-modified
                cj["lmod"] = int(time.time())

-            alts: list[tuple[int, int, dict[str, Any]]] = []
+            alts: list[tuple[int, int, dict[str, Any], "sqlite3.Cursor", str, str]] = []
            for ptop, cur in vols:
                allv = self.asrv.vfs.all_vols
                cvfs = next((v for v in allv.values() if v.realpath == ptop), vfs)
@@ -2707,13 +2725,12 @@ class Up2k(object):
                                wark, st.st_size, dsize, st.st_mtime, dtime, dp_abs
                            )
                            self.log(t)
-                            raise Exception("desync")
+                            raise Exception()
                    except Exception as ex:
                        if n4g:
                            st = os.stat_result((0, -1, -1, 0, 0, 0, 0, 0, 0, 0))
                        else:
-                            if str(ex) != "desync":
-                                lost.append((cur, dp_dir, dp_fn))
+                            lost.append((cur, dp_dir, dp_fn))
                            continue

                    j = {
@@ -2736,18 +2753,42 @@ class Up2k(object):
                        if k in cj:
                            j[k] = cj[k]

+                    # offset of 1st diff in vpaths
+                    zig = (
+                        n + 1
+                        for n, (c1, c2) in enumerate(
+                            zip(dp_dir + "\r", cj["prel"] + "\n")
+                        )
+                        if c1 != c2
+                    )
                    score = (
-                        (3 if st.st_dev == dev else 0)
-                        + (2 if dp_dir == cj["prel"] else 0)
+                        (6969 if st.st_dev == dev else 0)
+                        + (3210 if dp_dir == cj["prel"] else next(zig))
                        + (1 if dp_fn == cj["name"] else 0)
                    )
-                    alts.append((score, -len(alts), j))
+                    alts.append((score, -len(alts), j, cur, dp_dir, dp_fn))

-            if alts:
-                best = sorted(alts, reverse=True)[0]
-                job = best[2]
-            else:
-                job = None
+            job = None
+            inc_ap = djoin(cj["ptop"], cj["prel"], cj["name"])
+            for dupe in sorted(alts, reverse=True):
+                rj = dupe[2]
+                orig_ap = djoin(rj["ptop"], rj["prel"], rj["name"])
+                if data_ok or inc_ap == orig_ap:
+                    data_ok = True
+                    job = rj
+                    break
+                else:
+                    self.log("asserting contents of %s" % (orig_ap,))
+                    dhashes, st = self._hashlist_from_file(orig_ap)
+                    dwark = up2k_wark_from_hashlist(self.salt, st.st_size, dhashes)
+                    if wark != dwark:
+                        t = "will not dedup (fs index desync): fs=%s, db=%s, file: %s"
+                        self.log(t % (dwark, wark, orig_ap))
+                        lost.append(dupe[3:])
+                        continue
+                    data_ok = True
+                    job = rj
+                    break

            if job and wark in reg:
                # self.log("pop " + wark + "  " + job["name"] + " handle_json db", 4)
@@ -2756,7 +2797,7 @@ class Up2k(object):
            if lost:
                c2 = None
                for cur, dp_dir, dp_fn in lost:
-                    t = "forgetting deleted file: /{}"
+                    t = "forgetting desynced db entry: /{}"
                    self.log(t.format(vjoin(vjoin(vfs.vpath, dp_dir), dp_fn)))
                    self.db_rm(cur, dp_dir, dp_fn, cj["size"])
                    if c2 and c2 != cur:
@@ -2791,7 +2832,13 @@ class Up2k(object):
                            del reg[wark]
                        break

-                if st and not self.args.nw and not n4g and st.st_size != rj["size"]:
+                inc_ap = djoin(cj["ptop"], cj["prel"], cj["name"])
+                orig_ap = djoin(rj["ptop"], rj["prel"], rj["name"])
+
+                if self.args.nw or n4g or not st:
+                    pass
+
+                elif st.st_size != rj["size"]:
                    t = "will not dedup (fs index desync): {}, size fs={} db={}, mtime fs={} db={}, file: {}"
                    t = t.format(
                        wark, st.st_size, rj["size"], st.st_mtime, rj["lmod"], path
@@ -2799,6 +2846,15 @@ class Up2k(object):
                    self.log(t)
                    del reg[wark]

+                elif inc_ap != orig_ap and not data_ok:
+                    self.log("asserting contents of %s" % (orig_ap,))
+                    dhashes, _ = self._hashlist_from_file(orig_ap)
+                    dwark = up2k_wark_from_hashlist(self.salt, st.st_size, dhashes)
+                    if wark != dwark:
+                        t = "will not dedup (fs index desync): fs=%s, idx=%s, file: %s"
+                        self.log(t % (dwark, wark, orig_ap))
+                        del reg[wark]
+
            if job or wark in reg:
                job = job or reg[wark]
                if (
@@ -3070,17 +3126,25 @@ class Up2k(object):
        verbose: bool = True,
        rm: bool = False,
        lmod: float = 0,
+        fsrc: Optional[str] = None,
    ) -> None:
+        if src == dst or (fsrc and fsrc == dst):
+            t = "symlinking a file to itself?? orig(%s) fsrc(%s) link(%s)"
+            raise Exception(t % (src, fsrc, dst))
+
        if verbose:
-            self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+            t = "linking dupe:\n  point-to: {0}\n  link-loc: {1}"
+            if fsrc:
+                t += "\n  data-src: {2}"
+            self.log(t.format(src, dst, fsrc))

        if self.args.nw:
            return

        linked = False
        try:
-            if "copydupes" in flags:
-                raise Exception("disabled in config")
+            if not flags.get("dedup"):
+                raise Exception("dedup is disabled in config")

            lsrc = src
            ldst = dst
@@ -3117,7 +3181,7 @@ class Up2k(object):
                    linked = True
            except Exception as ex:
                self.log("cannot hardlink: " + repr(ex))
-                if "neversymlink" in flags:
+                if "hardlinkonly" in flags:
                    raise Exception("symlink-fallback disabled in cfg")

            if not linked:
@@ -3136,7 +3200,15 @@ class Up2k(object):
                linked = True
        except Exception as ex:
            self.log("cannot link; creating copy: " + repr(ex))
-            shutil.copy2(fsenc(src), fsenc(dst))
+            if bos.path.isfile(src):
+                csrc = src
+            elif fsrc and bos.path.isfile(fsrc):
+                csrc = fsrc
+            else:
+                t = "BUG: no valid sources to link from! orig(%s) fsrc(%s) link(%s)"
+                self.log(t, 1)
+                raise Exception(t % (src, fsrc, dst))
+            shutil.copy2(fsenc(csrc), fsenc(dst))

        if lmod and (not linked or SYMTIME):
            times = (int(time.time()), int(lmod))
@@ -3698,8 +3770,11 @@ class Up2k(object):
                    cur = None
                    try:
                        ptop = dbv.realpath
+                        xlink = bool(dbv.flags.get("xlink"))
                        cur, wark, _, _, _, _ = self._find_from_vpath(ptop, volpath)
-                        self._forget_file(ptop, volpath, cur, wark, True, st.st_size)
+                        self._forget_file(
+                            ptop, volpath, cur, wark, True, st.st_size, xlink
+                        )
                    finally:
                        if cur:
                            cur.connection.commit()
@@ -3923,13 +3998,15 @@ class Up2k(object):
            if c2 and c2 != c1:
                self._copy_tags(c1, c2, w)

+            xlink = bool(svn.flags.get("xlink"))
+
            with self.reg_mutex:
                has_dupes = self._forget_file(
-                    svn.realpath, srem, c1, w, is_xvol, fsize_ or fsize
+                    svn.realpath, srem, c1, w, is_xvol, fsize_ or fsize, xlink
                )

            if not is_xvol:
-                has_dupes = self._relink(w, svn.realpath, srem, dabs)
+                has_dupes = self._relink(w, svn.realpath, srem, dabs, c1, xlink)

            curs.add(c1)

@@ -4072,6 +4149,7 @@ class Up2k(object):
        wark: Optional[str],
        drop_tags: bool,
        sz: int,
+        xlink: bool,
    ) -> bool:
        """
        mutex(main,reg) me
@@ -4083,7 +4161,7 @@ class Up2k(object):
        if wark and cur:
            self.log("found {} in db".format(wark))
            if drop_tags:
-                if self._relink(wark, ptop, vrem, ""):
+                if self._relink(wark, ptop, vrem, "", cur, xlink):
                    has_dupes = True
                    drop_tags = False

@@ -4115,7 +4193,15 @@ class Up2k(object):

        return has_dupes

-    def _relink(self, wark: str, sptop: str, srem: str, dabs: str) -> int:
+    def _relink(
+        self,
+        wark: str,
+        sptop: str,
+        srem: str,
+        dabs: str,
+        vcur: Optional["sqlite3.Cursor"],
+        xlink: bool,
+    ) -> int:
        """
        update symlinks from file at svn/srem to dabs (rename),
        or to first remaining full if no dabs (delete)
@@ -4131,6 +4217,8 @@ class Up2k(object):
            argv = (wark[:16], wark)

        for ptop, cur in self.cur.items():
+            if not xlink and cur and cur != vcur:
+                continue
            for rd, fn in cur.execute(q, argv):
                if rd.startswith("//") or fn.startswith("//"):
                    rd, fn = s3dec(rd, fn)
@@ -4217,7 +4305,13 @@ class Up2k(object):
            except:
                pass

-            self._symlink(dabs, alink, flags, False, lmod=lmod or 0)
+            # this creates a link pointing from dabs to alink; alink may
+            # not exist yet, which becomes problematic if the symlinking
+            # fails and it has to fall back on hardlinking/copying files
+            # (for example a volume with symlinked dupes but no --dedup);
+            # fsrc=sabs is then a source that currently resolves to copy
+
+            self._symlink(dabs, alink, flags, False, lmod=lmod or 0, fsrc=sabs)

        return len(full) + len(links)

@@ -4246,8 +4340,11 @@ class Up2k(object):

        return wark

-    def _hashlist_from_file(self, path: str, prefix: str = "") -> list[str]:
-        fsz = bos.path.getsize(path)
+    def _hashlist_from_file(
+        self, path: str, prefix: str = ""
+    ) -> tuple[list[str], os.stat_result]:
+        st = bos.stat(path)
+        fsz = st.st_size
        csz = up2k_chunksize(fsz)
        ret = []
        suffix = " MB, {}".format(path)
@@ -4260,7 +4357,7 @@ class Up2k(object):
            while fsz > 0:
                # same as `hash_at` except for `imutex` / bufsz
                if self.stop:
-                    return []
+                    return [], st

                if self.pp:
                    mb = fsz // (1024 * 1024)
@@ -4281,7 +4378,7 @@ class Up2k(object):
                digest = base64.urlsafe_b64encode(digest)
                ret.append(digest.decode("utf-8"))

-        return ret
+        return ret, st

    def _new_upload(self, job: dict[str, Any], vfs: VFS, depth: int) -> dict[str, str]:
        pdir = djoin(job["ptop"], job["prel"])
@@ -4582,7 +4679,7 @@ class Up2k(object):
                self.salt, inf.st_size, int(inf.st_mtime), rd, fn
            )
        else:
-            hashes = self._hashlist_from_file(abspath)
+            hashes, _ = self._hashlist_from_file(abspath)
            if not hashes:
                return False

--- a/copyparty/util.py
+++ b/copyparty/util.py
@@ -260,6 +260,8 @@ IMPLICATIONS = [
    ["e2vu", "e2v"],
    ["e2vp", "e2v"],
    ["e2v", "e2d"],
+    ["hardlink_only", "hardlink"],
+    ["hardlink", "dedup"],
    ["tftpvv", "tftpv"],
    ["smbw", "smb"],
    ["smb1", "smb"],
--- a/copyparty/web/browser.css
+++ b/copyparty/web/browser.css
@@ -64,7 +64,7 @@
 	--u2-tab-bg: linear-gradient(to bottom, var(--bg), var(--bg-u1));
 	--u2-tab-b1: rgba(128,128,128,0.8);
 	--u2-tab-1-fg: #fd7;
-	--u2-tab-1-bg: linear-gradient(to bottom, var(#353), var(--bg) 80%);
+	--u2-tab-1-bg: linear-gradient(to bottom, #353, var(--bg) 80%);
 	--u2-tab-1-b1: #7c5;
 	--u2-tab-1-b2: #583;
 	--u2-tab-1-sh: #280;
@@ -345,6 +345,7 @@ html.cz {
 	--srv-3: #fff;

 	--u2-tab-b1: var(--bg-d3);
+	--u2-tab-1-bg: a;
 }
 html.cy {
 	--fg: #fff;
@@ -374,12 +375,17 @@ html.cy {
 	--btn-bs: 0 .25em 0 #f00;
 	--chk-fg: #fd0;

+	--txt-bg: #000;
 	--srv-1: #f00;
 	--srv-3: #fff;
 	--op-aa-bg: #fff;

 	--u2-b1-bg: #f00;
 	--u2-b2-bg: #f00;
+
+	--g-sel-fg: #fff;
+	--g-sel-bg: #aaa;
+	--g-fsel-bg: #aaa;
 }
 html.dz {
 	--fg: #4d4;
@@ -436,7 +442,7 @@ html.dz {
 	--u2-tab-bg: linear-gradient(to bottom, var(--bg), var(--bg-u1));
 	--u2-tab-b1: var(--fg-weak);
 	--u2-tab-1-fg: #fff;
-	--u2-tab-1-bg: linear-gradient(to bottom, var(#353), var(--bg) 80%);
+	--u2-tab-1-bg: linear-gradient(to bottom, #151, var(--bg) 80%);
 	--u2-tab-1-b1: #7c5;
 	--u2-tab-1-b2: #583;
 	--u2-tab-1-sh: #280;
@@ -3112,18 +3118,30 @@ html.by #u2cards a.act {



-html.cy #wrap {
-	color: #000;
-}
 html.cy .mdo a {
 	background: #f00;
 }
+html.cy #wrap,
+html.cy #acc_info a,
 html.cy #op_up2k,
 html.cy #files,
 html.cy #files a,
 html.cy #files tbody div a:last-child {
 	color: #000;
 }
+html.cy #u2tab a,
+html.cy #u2cards a {
+	color: #f00;
+}
+html.cy #unpost a {
+	color: #ff0;
+}
+html.cy #barbuf {
+	filter: hue-rotate(267deg) brightness(0.8) contrast(4);
+}
+html.cy #pvol {
+	filter: hue-rotate(4deg) contrast(2.2);
+}



--- a/copyparty/web/browser.js
+++ b/copyparty/web/browser.js
@@ -90,6 +90,10 @@ var Ls = {
 		"m_ok": "OK",
 		"m_ng": "Cancel",

+		"enable": "Enable",
+		"danger": "DANGER",
+		"clipped": "copied to clipboard",
+
 		"ht_s": "second!s",
 		"ht_m": "minute!s",
 		"ht_h": "hour!s",
@@ -170,6 +174,15 @@ var Ls = {
 		"utl_stat": "status",
 		"utl_prog": "progress",

+		// keep short:
+		"utl_404": "404",
+		"utl_err": "ERROR",
+		"utl_oserr": "OS-error",
+		"utl_found": "found",
+		"utl_defer": "defer",
+		"utl_yolo": "YOLO",
+		"utl_done": "done",
+
 		"ul_flagblk": "the files were added to the queue</b><br>however there is a busy up2k in another browser tab,<br>so waiting for that to finish first",
 		"ul_btnlk": "the server configuration has locked this switch into this state",

@@ -195,6 +208,7 @@ var Ls = {
 		"cl_hcancel": "column hiding aborted",

 		"ct_grid": '田 the grid',
+		"ct_ttips": '◔ ◡ ◔">ℹ️ tooltips',
 		"ct_thumb": 'in grid-view, toggle icons or thumbnails$NHotkey: T">🖼️ thumbs',
 		"ct_csel": 'use CTRL and SHIFT for file selection in grid-view">sel',
 		"ct_ihop": 'when the image viewer is closed, scroll down to the last viewed file">g⮯',
@@ -326,6 +340,7 @@ var Ls = {
 		"fs_pname": "optional link name; will be random if blank",
 		"fs_tsrc": "the file or folder to share",
 		"fs_ppwd": "optional password",
+		"fs_w8": "creating share...",
 		"fs_ok": "<h6>share-URL created</h6>\npress <code>Enter/OK</code> to Clipboard\npress <code>ESC/Cancel</code> to Close\n\n",

 		"frt_dec": "may fix some cases of broken filenames\">url-decode",
@@ -336,6 +351,8 @@ var Ls = {
 		"fr_case": "case-sensitive regex\">case",
 		"fr_win": "windows-safe names; replace <code>&lt;&gt;:&quot;\\|?*</code> with japanese fullwidth characters\">win",
 		"fr_slash": "replace <code>/</code> with a character that doesn't cause new folders to be created\">no /",
+		"fr_re": "regex search pattern to apply to original filenames; capturing groups can be referenced in the format field below like &lt;code&gt;(1)&lt;/code&gt; and &lt;code&gt;(2)&lt;/code&gt; and so on",
+		"fr_fmt": "inspired by foobar2000:$N&lt;code&gt;(title)&lt;/code&gt; is replaced by song title,$N&lt;code&gt;[(artist) - ](title)&lt;/code&gt; skips [this] part if artist is blank$N&lt;code&gt;$lpad((tn),2,0)&lt;/code&gt; pads tracknumber to 2 digits",
 		"fr_pdel": "delete",
 		"fr_pnew": "save as",
 		"fr_pname": "provide a name for your new preset",
@@ -345,7 +362,7 @@ var Ls = {
 		"fr_tags": "tags for the selected files (read-only, just for reference):",
 		"fr_busy": "renaming {0} items...\n\n{1}",
 		"fr_efail": "rename failed:\n",
-		"fr_nchg": "{0} of the new names were altered due to <code>win</code> and/or <code>ikke /</code>\n\nOK to continue with these altered new names?",
+		"fr_nchg": "{0} of the new names were altered due to <code>win</code> and/or <code>no /</code>\n\nOK to continue with these altered new names?",

 		"fd_ok": "delete OK",
 		"fd_err": "delete failed:\n",
@@ -395,6 +412,7 @@ var Ls = {
 		"gt_c1": "truncate filenames more (show less)",
 		"gt_c2": "truncate filenames less (show more)",

+		"sm_w8": "searching...",
 		"sm_prev": "search results below are from a previous query:\n  ",
 		"sl_close": "close search results",
 		"sl_hits": "showing {0} hits",
@@ -467,16 +485,20 @@ var Ls = {
 		"un_del": "delete",
 		"un_m3": "loading your recent uploads...",
 		"un_busy": "deleting {0} files...",
+		"un_clip": "{0} links copied to clipboard",

 		"u_https1": "you should",
 		"u_https2": "switch to https",
 		"u_https3": "for better performance",
 		"u_ancient": 'your browser is impressively ancient -- maybe you should <a href="#" onclick="goto(\'bup\')">use bup instead</a>',
 		"u_nowork": "need firefox 53+ or chrome 57+ or iOS 11+",
+		"u_nodrop": 'your browser is too old for drag-and-drop uploading',
+		"u_notdir": "that's not a folder!\n\nyour browser is too old,\nplease try dragdrop instead",
 		"u_uri": "to dragdrop images from other browser windows,\nplease drop it onto the big upload button",
 		"u_enpot": 'switch to <a href="#">potato UI</a> (may improve upload speed)',
 		"u_depot": 'switch to <a href="#">fancy UI</a> (may reduce upload speed)',
 		"u_gotpot": 'switching to the potato UI for improved upload speed,\n\nfeel free to disagree and switch back!',
+		"u_pott": "<p>files: &nbsp; <b>{0}</b> finished, &nbsp; <b>{1}</b> failed, &nbsp; <b>{2}</b> busy, &nbsp; <b>{3}</b> queued</p>",
 		"u_ever": "this is the basic uploader; up2k needs at least<br>chrome 21 // firefox 13 // edge 12 // opera 12 // safari 5.1",
 		"u_su2k": 'this is the basic uploader; <a href="#" id="u2yea">up2k</a> is better',
 		"u_ewrite": 'you do not have write-access to this folder',
@@ -489,10 +511,16 @@ var Ls = {
 		"u_up_life": "This upload will be deleted from the server\n{0} after it completes",
 		"u_asku": 'upload these {0} files to <code>{1}</code>',
 		"u_unpt": "you can undo / delete this upload using the top-left 🧯",
+		"u_bigtab": 'about to show {0} files\n\nthis may crash your browser, are you sure?',
+		"u_scan": 'Scanning files...',
+		"u_dirstuck": 'directory iterator got stuck trying to access the following {0} items; will skip:',
 		"u_etadone": 'Done ({0}, {1} files)',
 		"u_etaprep": '(preparing to upload)',
 		"u_hashdone": 'hashing done',
 		"u_hashing": 'hash',
+		"u_hs": 'handshaking...',
+		"u_dupdefer": "duplicate; will be processed after all other files",
+		"u_actx": "click this text to prevent loss of<br />performance when switching to other windows/tabs",
 		"u_fixed": "OK!&nbsp; Fixed it 👍",
 		"u_cuerr": "failed to upload chunk {0} of {1};\nprobably harmless, continuing\n\nfile: {2}",
 		"u_cuerr2": "server rejected upload (chunk {0} of {1});\nwill retry later\n\nfile: {2}\n\nerror ",
@@ -626,6 +654,10 @@ var Ls = {
 		"m_ok": "OK",
 		"m_ng": "Avbryt",

+		"enable": "Aktiv",
+		"danger": "VARSKU",
+		"clipped": "kopiert til utklippstavlen",
+
 		"ht_s": "sekund!er",
 		"ht_m": "minutt!er",
 		"ht_h": "time!r",
@@ -706,6 +738,15 @@ var Ls = {
 		"utl_stat": "status",
 		"utl_prog": "fremdrift",

+		// må være korte:
+		"utl_404": "404",
+		"utl_err": "FEIL!",
+		"utl_oserr": "OS-feil",
+		"utl_found": "funnet",
+		"utl_defer": "senere",
+		"utl_yolo": "YOLO",
+		"utl_done": "ferdig",
+
 		"ul_flagblk": "filene har blitt lagt i køen</b><br>men det er en annen nettleserfane som holder på med befaring eller opplastning akkurat nå,<br>så venter til den er ferdig først",
 		"ul_btnlk": "bryteren har blitt låst til denne tilstanden i serverens konfigurasjon",

@@ -731,6 +772,7 @@ var Ls = {
 		"cl_hcancel": "kolonne-skjuling avbrutt",

 		"ct_grid": '田 ikoner',
+		"ct_ttips": 'hvis hjelpetekst ved å holde musen over ting">ℹ️ tips',
 		"ct_thumb": 'vis miniatyrbilder istedenfor ikoner$NSnarvei: T">🖼️ bilder',
 		"ct_csel": 'bruk tastene CTRL og SHIFT for markering av filer i ikonvisning">merk',
 		"ct_ihop": 'bla ned til sist viste bilde når bildeviseren lukkes">g⮯',
@@ -862,6 +904,7 @@ var Ls = {
 		"fs_pname": "frivillig navn (blir noe tilfeldig ellers)",
 		"fs_tsrc": "fil/mappe som skal deles",
 		"fs_ppwd": "frivillig passord",
+		"fs_w8": "oppretter deling...",
 		"fs_ok": "<h6>URL opprettet</h6>\ntrykk <code>Enter/OK</code> for å kopiere linken (for CTRL-V)\ntrykk <code>ESC/Avbryt</code> for å bare bekrefte\n\n",

 		"frt_dec": "kan korrigere visse ødelagte filnavn\">url-decode",
@@ -872,6 +915,8 @@ var Ls = {
 		"fr_case": "versalfølsomme uttrykk\">Aa",
 		"fr_win": "bytt ut bokstavene <code>&lt;&gt;:&quot;\\|?*</code> med$Ntilsvarende som windows ikke får panikk av\">win",
 		"fr_slash": "bytt ut bokstaven <code>/</code> slik at den ikke forårsaker at nye mapper opprettes\">ikke /",
+		"fr_re": "regex-mønster som kjøres på hvert filnavn. Grupper kan leses ut i format-feltet nedenfor, f.eks. &lt;code&gt;(1)&lt;/code&gt; og &lt;code&gt;(2)&lt;/code&gt; osv.",
+		"fr_fmt": "inspirert av foobar2000:$N&lt;code&gt;(title)&lt;/code&gt; byttes ut med sangtittel,$N&lt;code&gt;[(artist) - ](title)&lt;/code&gt; dropper [dette] hvis artist er blank$N&lt;code&gt;$lpad((tn),2,0)&lt;/code&gt; viser sangnr. med 2 siffer",
 		"fr_pdel": "slett",
 		"fr_pnew": "lagre som",
 		"fr_pname": "gi innstillingene dine et navn",
@@ -931,6 +976,7 @@ var Ls = {
 		"gt_c1": "reduser maks-lengde på filnavn",
 		"gt_c2": "øk maks-lengde på filnavn",

+		"sm_w8": "søker...",
 		"sm_prev": "søkeresultatene er fra et tidligere søk:\n  ",
 		"sl_close": "lukk søkeresultater",
 		"sl_hits": "viser {0} treff",
@@ -1003,16 +1049,20 @@ var Ls = {
 		"un_del": "slett",
 		"un_m3": "henter listen med nylig opplastede filer...",
 		"un_busy": "sletter {0} filer...",
+		"un_clip": "{0} lenker kopiert til utklippstavlen",

 		"u_https1": "du burde",
 		"u_https2": "bytte til https",
 		"u_https3": "for høyere hastighet",
 		"u_ancient": 'nettleseren din er prehistorisk -- mulig du burde <a href="#" onclick="goto(\'bup\')">bruke bup istedenfor</a>',
 		"u_nowork": "krever firefox 53+, chrome 57+, eller iOS 11+",
+		"u_nodrop": 'nettleseren din er for gammel til å laste opp filer ved å dra dem inn i vinduet',
+		"u_notdir": "mottok ikke mappen!\n\nnettleseren din er for gammel,\nprøv å dra mappen inn i vinduet istedenfor",
 		"u_uri": "for å laste opp bilder ifra andre nettleservinduer,\nslipp bildet rett på den store last-opp-knappen",
 		"u_enpot": 'bytt til <a href="#">enkelt UI</a> (gir sannsynlig raskere opplastning)',
 		"u_depot": 'bytt til <a href="#">snæsent UI</a> (gir sannsynlig tregere opplastning)',
 		"u_gotpot": 'byttet til et enklere UI for å laste opp raskere,\n\ndu kan gjerne bytte tilbake altså!',
+		"u_pott": "<p>filer: &nbsp; <b>{0}</b> ferdig, &nbsp; <b>{1}</b> feilet, &nbsp; <b>{2}</b> behandles, &nbsp; <b>{3}</b> i kø</p>",
 		"u_ever": "dette er den primitive opplasteren; up2k krever minst:<br>chrome 21 // firefox 13 // edge 12 // opera 12 // safari 5.1",
 		"u_su2k": 'dette er den primitive opplasteren; <a href="#" id="u2yea">up2k</a> er bedre',
 		"u_ewrite": 'du har ikke skrivetilgang i denne mappen',
@@ -1025,10 +1075,16 @@ var Ls = {
 		"u_up_life": "Filene slettes fra serveren {0}\netter at opplastningen er fullført",
 		"u_asku": 'Laste opp disse {0} filene til <code>{1}</code>',
 		"u_unpt": "Du kan angre / slette opplastningen med 🧯 oppe til venstre",
+		"u_bigtab": 'Vil nå vise {0} filer...\n\nDette kan krasje nettleseren din. Fortsette?',
+		"u_scan": 'Leser mappene...',
+		"u_dirstuck": 'Nettleseren din fikk ikke tilgang til å lese følgende {0} filer/mapper, så de blir hoppet over:',
 		"u_etadone": 'Ferdig ({0}, {1} filer)',
 		"u_etaprep": '(forbereder opplastning)',
 		"u_hashdone": 'befaring ferdig',
 		"u_hashing": 'les',
+		"u_hs": 'serveren tenker...',
+		"u_dupdefer": "duplikat; vil bli håndtert til slutt",
+		"u_actx": "klikk her for å forhindre tap av<br />ytelse ved bytte til andre vinduer/faner",
 		"u_fixed": "OK!&nbsp; Løste seg 👍",
 		"u_cuerr": "kunne ikke laste opp del {0} av {1};\nsikkert greit, fortsetter\n\nfil: {2}",
 		"u_cuerr2": "server nektet opplastningen (del {0} av {1});\nprøver igjen senere\n\nfil: {2}\n\nerror ",
@@ -1162,6 +1218,10 @@ var Ls = {
 		"m_ok": "确定",
 		"m_ng": "取消",

+		"enable": "启用", //m
+		"danger": "危险", //m
+		"clipped": "已复制到剪贴板", //m
+
 		"ht_s": "秒",
 		"ht_m": "分",
 		"ht_h": "时",
@@ -1242,6 +1302,15 @@ var Ls = {
 		"utl_stat": "状态",
 		"utl_prog": "进度",

+		// 保持简短:
+		"utl_404": "404", //m
+		"utl_err": "故障", //m
+		"utl_oserr": "OS故障", //m
+		"utl_found": "已找到", //m
+		"utl_defer": "延期", //m
+		"utl_yolo": "加速", //m
+		"utl_done": "完成",
+
 		"ul_flagblk": "文件已添加到队列</b><br>但另一个浏览器标签中有一个繁忙的 up2k，<br>因此等待它完成",
 		"ul_btnlk": "服务器配置已将此开关锁定到此状态",

@@ -1267,6 +1336,7 @@ var Ls = {
 		"cl_hcancel": "列隐藏已取消",

 		"ct_grid": '网格视图',
+		"ct_ttips": '◔ ◡ ◔">ℹ️ 工具提示', //m
 		"ct_thumb": '在网格视图中，切换图标或缩略图$N快捷键: T">🖼️ 缩略图',
 		"ct_csel": '在网格视图中使用 CTRL 和 SHIFT 进行文件选择">CTRL',
 		"ct_ihop": '当图像查看器关闭时，滚动到最后查看的文件">滚动',
@@ -1398,6 +1468,7 @@ var Ls = {
 		"fs_pname": "链接名称可选；如果为空则随机",
 		"fs_tsrc": "共享的文件或文件夹",
 		"fs_ppwd": "密码可选",
+		"fs_w8": "正在创建文件共享...", //m
 		"fs_ok": "<h6>分享链接已创建</h6>\n按 <code>Enter/OK</code> 复制到剪贴板\n按 <code>ESC/Cancel</code> 关闭\n\n",

 		"frt_dec": "可能修复一些损坏的文件名\">url-decode",
@@ -1408,6 +1479,8 @@ var Ls = {
 		"fr_case": "区分大小写的正则表达式\">case",
 		"fr_win": "Windows 安全名称；将 <code>&lt;&gt;:&quot;\\|?*</code> 替换为日文全角字符\">win",
 		"fr_slash": "将 <code>/</code> 替换为不会导致新文件夹创建的字符\">不使用 /",
+		"fr_re": "正则表达式搜索模式应用于原始文件名；$N可以在下面的格式字段中引用捕获组，如&lt;code&gt;(1)&lt;/code&gt;和&lt;code&gt;(2)&lt;/code&gt;等等。", //m
+		"fr_fmt": "受到 foobar2000 的启发：$N&lt;code&gt;(title)&lt;/code&gt; 被歌曲名称替换,$N&lt;code&gt;[(artist) - ](title)&lt;/code&gt; 仅当歌曲艺术家不为空时才包含&lt;code&gt;[此]&lt;/code&gt;部分$N&lt;code&gt;$lpad((tn),2,0)&lt;/code&gt; 将曲目编号填充为 2 位数字", //m
 		"fr_pdel": "删除",
 		"fr_pnew": "另存为",
 		"fr_pname": "为你的新预设提供一个名称",
@@ -1417,7 +1490,7 @@ var Ls = {
 		"fr_tags": "选定文件的标签（只读，仅供参考）：",
 		"fr_busy": "正在重命名 {0} 项...\n\n{1}",
 		"fr_efail": "重命名失败：\n",
-		"fr_nchg": "{0} 个新名称由于 <code>win</code> 和/或 <code>ikke /</code> 被更改\n\n确定继续使用这些更改的新名称？",
+		"fr_nchg": "{0} 个新名称由于 <code>win</code> 和/或 <code>不使用 /</code> 被更改\n\n确定继续使用这些更改的新名称？",

 		"fd_ok": "删除成功",
 		"fd_err": "删除失败：\n",
@@ -1467,6 +1540,7 @@ var Ls = {
 		"gt_c1": "截断文件名更多（显示更少）",
 		"gt_c2": "截断文件名更少（显示更多）",

+		"sm_w8": "正在搜寻匹配...", //m
 		"sm_prev": "以下是来自先前查询的搜索结果：\n  ",
 		"sl_close": "关闭搜索结果",
 		"sl_hits": "显示 {0} 个结果",
@@ -1539,16 +1613,20 @@ var Ls = {
 		"un_del": "删除",
 		"un_m3": "正在加载你的近期上传...",
 		"un_busy": "正在删除 {0} 个文件...",
+		"un_clip": "{0} 个链接已复制到剪贴板", //m

 		"u_https1": "你应该",
 		"u_https2": "切换到 https",
 		"u_https3": "以获得更好的性能",
 		"u_ancient": '你的浏览器非常古老 -- 也许你应该 <a href="#" onclick="goto(\'bup\')">改用 bup</a>',
 		"u_nowork": "需要 Firefox 53+ 或 Chrome 57+ 或 iOS 11+",
+		"u_nodrop": '您的浏览器太旧，不支持通过拖动文件到窗口来上传文件', //m
+		"u_notdir": "未收到文件夹！\n\n您的浏览器太旧；\n请尝试将文件夹拖入窗口", //m
 		"u_uri": "要从其他浏览器窗口拖放图片，\n请将其拖放到大的上传按钮上",
 		"u_enpot": '切换到 <a href="#">简约 UI</a>（可能提高上传速度）',
 		"u_depot": '切换到 <a href="#">精美 UI</a>（可能降低上传速度）',
 		"u_gotpot": '切换到土豆 UI 以提高上传速度，\n\n随时可以不同意并切换回去！',
+		"u_pott": "<p>个文件： &nbsp; <b>{0}</b> 已完成， &nbsp; <b>{1}</b> 失败， &nbsp; <b>{2}</b> 正在处理， &nbsp; <b>{3}</b> 已排队</p>", //m
 		"u_ever": "这是基本的上传工具； up2k 需要至少<br>chrome 21 // firefox 13 // edge 12 // opera 12 // safari 5.1",
 		"u_su2k": '这是基本的上传工具；<a href="#" id="u2yea">up2k</a> 更好',
 		"u_ewrite": '你对这个文件夹没有写入权限',
@@ -1561,10 +1639,16 @@ var Ls = {
 		"u_up_life": "此上传将在 {0} 后从服务器删除",
 		"u_asku": '将这些 {0} 个文件上传到 <code>{1}</code>',
 		"u_unpt": "你可以使用左上角的 🧯 撤销/删除此上传",
+		"u_bigtab": '即将显示 {0} 个文件。这可能会导致您的浏览器崩溃。您确定吗？', //m
+		"u_scan": '正在扫描文件...', //m
+		"u_dirstuck": '您的浏览器无法访问以下 {0} 个文件/文件夹，因此它们将被跳过：', //m
 		"u_etadone": '完成 ({0}, {1} 个文件)',
 		"u_etaprep": '(准备上传)',
 		"u_hashdone": '哈希完成',
 		"u_hashing": '哈希',
+		"u_hs": '正在等待服务器...', //m
+		"u_dupdefer": "这是一个重复文件。它将在所有其他文件上传后进行处理", //m
+		"u_actx": "单击此文本以防止切换到其他窗口/选项卡时性能下降", //m
 		"u_fixed": "好！&nbsp;已修复 👍",
 		"u_cuerr": "上传块 {0} 的 {1} 失败；\n可能无害，继续中\n\n文件：{2}",
 		"u_cuerr2": "服务器拒绝上传（块 {0} 的 {1}）；\n稍后重试\n\n文件：{2}\n\n错误 ",
@@ -1810,7 +1894,7 @@ ebi('op_cfg').innerHTML = (
 	'<div>\n' +
 	'	<h3>' + L.cl_opts + '</h3>\n' +
 	'	<div>\n' +
-	'		<a id="tooltips" class="tgl btn" href="#" tt="◔ ◡ ◔">ℹ️ tooltips</a>\n' +
+	'		<a id="tooltips" class="tgl btn" href="#" tt="' + L.ct_ttips + '</a>\n' +
 	'		<a id="griden" class="tgl btn" href="#" tt="' + L.wt_grid + '">' + L.ct_grid + '</a>\n' +
 	'		<a id="thumbs" class="tgl btn" href="#" tt="' + L.ct_thumb + '</a>\n' +
 	'		<a id="csel" class="tgl btn" href="#" tt="' + L.ct_csel + '</a>\n' +
@@ -2587,7 +2671,7 @@ var widget = (function () {
 		m += '[' + cv + s2ms(mp.au.currentTime) + ck + '/' + cv + s2ms(mp.au.duration) + ck + ']';

 		cliptxt(m, function () {
-			toast.ok(1, 'copied to clipboard', null, 'top');
+			toast.ok(1, L.clipped, null, 'top');
 		});
 	};
 	r.set(sread('au_open') == 1);
@@ -3607,7 +3691,7 @@ var afilt = (function () {
 	}

 	var html = ['<table><tr><td rowspan="4">',
-		'<a id="au_eq" class="tgl btn" href="#" tt="' + L.mt_eq + '">enable</a></td>'],
+		'<a id="au_eq" class="tgl btn" href="#" tt="' + L.mt_eq + '">' + L.enable + '</a></td>'],
 		h2 = [], h3 = [], h4 = [];

 	var vs = [];
@@ -3637,7 +3721,7 @@ var afilt = (function () {

 	h2 = [];
 	html = ['<table><tr><td rowspan="2">',
-		'<a id="au_drc" class="tgl btn" href="#" tt="' + L.mt_drc + '">enable</a></td>'];
+		'<a id="au_drc" class="tgl btn" href="#" tt="' + L.mt_drc + '">' + L.enable + '</a></td>'];

 	for (var a = 0; a < r.drch.length; a++) {
 		html.push('<td>' + r.drch[a] + '</td>');
@@ -3951,11 +4035,12 @@ function eval_hash() {
 				break;
 			}

-		if (mi && img_re.exec(og_fn))
-			hash0 = '#g' + mi.id;
+		var ch = !mi ? '' :
+			img_re.exec(og_fn) ? 'g' :
+			ebi('a' + mi.id) ? 'a' :
+			'';

-		if (ebi('a' + mi.id))
-			hash0 = '#a' + mi.id;
+		hash0 = ch ? ('#' + ch + mi.id) : '';
 	}

 	var v = hash0;
@@ -3996,14 +4081,14 @@ function eval_hash() {
 		}
 	}

-	if (v.indexOf('#q=') === 0) {
+	if (v.startsWith('#q=')) {
 		goto('search');
 		var i = ebi('q_raw');
 		i.value = uricom_dec(v.slice(3));
 		return i.onkeydown({ 'key': 'Enter' });
 	}

-	if (v.indexOf('#v=') === 0) {
+	if (v.startsWith('#v=')) {
 		goto(v.slice(3));
 		return;
 	}
@@ -4501,7 +4586,7 @@ var fileman = (function () {
 			surl = surl.slice(15);
 			modal.confirm(L.fs_ok + esc(surl), function() {
 				cliptxt(surl, function () {
-					toast.ok(2, 'copied to clipboard');
+					toast.ok(2, L.clipped);
 				});
 			});
 		}
@@ -4516,7 +4601,7 @@ var fileman = (function () {
 					plist.push(pbtns[a].textContent);

 			shui.style.display = 'none';
-			toast.inf(30, "creating share...");
+			toast.inf(30, L.fs_w8);

 			var body = {
 				"k": sh_k.value,
@@ -4604,8 +4689,8 @@ var fileman = (function () {
 			'<a id="rn_slash" class="tgl btn" href="#" tt="' + L.fr_slash + '</a>',
 			'</div>',
 			'<div id="rn_vadv"><table>',
-			'<tr><td>regex</td><td><input type="text" id="rn_re" ' + NOAC + ' tt="regex search pattern to apply to original filenames; capturing groups can be referenced in the format field below like &lt;code&gt;(1)&lt;/code&gt; and &lt;code&gt;(2)&lt;/code&gt; and so on" placeholder="^[0-9]+[\\. ]+(.*) - (.*)" /></td></tr>',
-			'<tr><td>format</td><td><input type="text" id="rn_fmt" ' + NOAC + ' tt="inspired by foobar2000:$N&lt;code&gt;(title)&lt;/code&gt; is replaced by song title,$N&lt;code&gt;[(artist) - ](title)&lt;/code&gt; skips the first part if artist is blank$N&lt;code&gt;$lpad((tn),2,0)&lt;/code&gt; pads tracknumber to 2 digits" placeholder="[(artist) - ](title).(ext)" /></td></tr>',
+			'<tr><td>regex</td><td><input type="text" id="rn_re" ' + NOAC + ' tt="' + L.fr_re + '" placeholder="^[0-9]+[\\. ]+(.*) - (.*)" /></td></tr>',
+			'<tr><td>format</td><td><input type="text" id="rn_fmt" ' + NOAC + ' tt="' + L.fr_fmt + '" placeholder="[(artist) - ](title).(ext)" /></td></tr>',
 			'<tr><td>preset</td><td><select id="rn_pre"></select>',
 			'<button id="rn_pdel">❌ ' + L.fr_pdel + '</button>',
 			'<button id="rn_pnew">💾 ' + L.fr_pnew + '</button>',
@@ -4613,7 +4698,9 @@ var fileman = (function () {
 			'</table></div>'
 		]);

-		var cheap = f.length > 500;
+		var cheap = f.length > 500,
+			t_rst = L.frt_rst.split('>').pop();
+
 		if (sel.length == 1)
 			html.push(
 				'<div><table id="rn_f">\n' +
@@ -4628,7 +4715,7 @@ var fileman = (function () {
 					'<tr><td>' +
 					(cheap ? '</td>' :
 						'<button class="rn_dec" n="' + a + '">decode</button>' +
-						'<button class="rn_reset" n="' + a + '">↺ reset</button></td>') +
+						'<button class="rn_reset" n="' + a + '">' + t_rst + '</button></td>') +
 					'<td><input type="text" id="rn_new" n="' + a + '" /></td>' +
 					'<td><input type="text" id="rn_old" n="' + a + '" readonly /></td></tr>');
 		}
@@ -4886,7 +4973,7 @@ var fileman = (function () {
 			deleter();
 		}

-		modal.confirm('<h6 style="color:#900">DANGER</h6>\n<b>' + L.fd_warn1.format(vps.length) + '</b><ul>' + uricom_adec(vps, true).join('') + '</ul>', function () {
+		modal.confirm('<h6 style="color:#900">' + L.danger + '</h6>\n<b>' + L.fd_warn1.format(vps.length) + '</b><ul>' + uricom_adec(vps, true).join('') + '</ul>', function () {
 			modal.confirm(L.fd_warn2, deleter, null);
 		}, null);
 	};
@@ -5161,6 +5248,12 @@ var showfile = (function () {
 		return srch.split(/[?&]doc=/)[1].split('&')[0];
 	};

+	if (window.og_fn) {
+		var ext = og_fn.split(/\./g).pop();
+		if (r.map['.' + ext])
+			hist_replace(get_evpath() + '?doc=' + og_fn);
+	}
+
 	window.Prism = { 'manual': true };
 	var em = QS('#bdoc>pre');
 	if (em)
@@ -6502,7 +6595,7 @@ var ahotkeys = function (e) {

 	function do_search() {
 		search_in_progress = Date.now();
-		srch_msg(false, "searching...");
+		srch_msg(false, L.sm_w8);
 		clearTimeout(search_timeout);

 		var xhr = new XHR();
@@ -9112,7 +9205,7 @@ var unpost = (function () {
 		ev(e);
 		var txt = linklist();
 		cliptxt(txt + '\n', function () {
-			toast.inf(5, txt.split('\n').length + ' links copied to clipboard');
+			toast.inf(5, L.un_clip.format(txt.split('\n').length));
 		});
 	};

--- a/copyparty/web/up2k.js
+++ b/copyparty/web/up2k.js
@@ -152,12 +152,13 @@ function U2pvis(act, btns, uc, st) {
    r.mod0 = null;

    var markup = {
-        '404': '<span class="err">404</span>',
-        'ERROR': '<span class="err">ERROR</span>',
-        'OS-error': '<span class="err">OS-error</span>',
-        'found': '<span class="inf">found</span>',
-        'YOLO': '<span class="inf">YOLO</span>',
-        'done': '<span class="ok">done</span>',
+        '404': '<span class="err">' + L.utl_404 + '</span>',
+        'ERROR': '<span class="err">' + L.utl_err + '</span>',
+        'OS-error': '<span class="err">' + L.utl_oserr + '</span>',
+        'found': '<span class="inf">' + L.utl_found + '</span>',
+        'defer': '<span class="inf">' + L.utl_defer + '</span>',
+        'YOLO': '<span class="inf">' + L.utl_yolo + '</span>',
+        'done': '<span class="ok">' + L.utl_done + '</span>',
    };

    r.addfile = function (entry, sz, draw) {
@@ -445,9 +446,7 @@ function U2pvis(act, btns, uc, st) {
            return;

        r.npotato = 0;
-        var html = [
-            "<p>files: &nbsp; <b>{0}</b> finished, &nbsp; <b>{1}</b> failed, &nbsp; <b>{2}</b> busy, &nbsp; <b>{3}</b> queued</p>".format(
-                r.ctr.ok, r.ctr.ng, r.ctr.bz, r.ctr.q)];
+        var html = [L.u_pott.format(r.ctr.ok, r.ctr.ng, r.ctr.bz, r.ctr.q)];

        while (r.head < r.tab.length && has(["ok", "ng"], r.tab[r.head].in))
            r.head++;
@@ -602,7 +601,7 @@ function U2pvis(act, btns, uc, st) {
            if (nf < 9000)
                return go();

-            modal.confirm('about to show ' + nf + ' files\n\nthis may crash your browser, are you sure?', go, null);
+            modal.confirm(L.u_bigtab.format(nf), go, null);
        };
    }

@@ -1037,7 +1036,7 @@ function up2k_init(subtle) {
        }
        catch (ex) {
            document.body.ondragenter = document.body.ondragleave = document.body.ondragover = null;
-            return modal.alert('your browser does not support drag-and-drop uploading');
+            return modal.alert(L.u_nodrop);
        }
        if (btn)
            return;
@@ -1104,7 +1103,7 @@ function up2k_init(subtle) {
        }

        if (!good_files.length && bad_files.length)
-            return toast.err(30, "that's not a folder!\n\nyour browser is too old,\nplease try dragdrop instead");
+            return toast.err(30, L.u_notdir);

        return read_dirs(null, [], [], good_files, nil_files, bad_files);
    }
@@ -1122,7 +1121,7 @@ function up2k_init(subtle) {
        if (err)
            return modal.alert('sorry, ' + err);

-        toast.inf(0, 'Scanning files...');
+        toast.inf(0, L.u_scan);

        if ((dz == 'up_dz' && uc.fsearch) || (dz == 'srch_dz' && !uc.fsearch))
            tgl_fsearch();
@@ -1210,7 +1209,7 @@ function up2k_init(subtle) {
                    match = false;

            if (match) {
-                var msg = ['directory iterator got stuck trying to access the following {0} items; will skip:<ul>'.format(missing.length)];
+                var msg = [L.u_dirstuck.format(missing.length) + '<ul>'];
                for (var a = 0; a < Math.min(20, missing.length); a++)
                    msg.push('<li>' + esc(missing[a]) + '</li>');

@@ -1281,7 +1280,7 @@ function up2k_init(subtle) {
    }

    function gotallfiles(good_files, nil_files, bad_files) {
-        if (toast.txt == 'Scanning files...')
+        if (toast.txt == L.u_scan)
            toast.hide();

        if (uc.fsearch && !uc.turbo)
@@ -1437,7 +1436,7 @@ function up2k_init(subtle) {
            if (!actx || actx.state != 'suspended' || toast.visible)
                return;

-            toast.warn(30, "<div onclick=\"start_actx();toast.inf(3,'thanks!')\">please click this text to<br />unlock full upload speed</div>");
+            toast.warn(30, "<div onclick=\"start_actx();toast.inf(3,'thanks!')\">" + L.u_actx + "</div>");
        }, 500);
    }

@@ -1479,7 +1478,7 @@ function up2k_init(subtle) {
        ev(e);
        var txt = linklist();
        cliptxt(txt + '\n', function () {
-            toast.inf(5, txt.split('\n').length + ' links copied to clipboard');
+            toast.inf(5, un_clip.format(txt.split('\n').length));
        });
    };

@@ -1746,14 +1745,6 @@ function up2k_init(subtle) {

                var mou_ikkai = false;

-                if (st.busy.handshake.length &&
-                    st.busy.handshake[0].t_busied < now - 30 * 1000
-                ) {
-                    console.log("retrying stuck handshake");
-                    var t = st.busy.handshake.shift();
-                    st.todo.handshake.unshift(t);
-                }
-
                var nprev = -1;
                for (var a = 0; a < st.todo.upload.length; a++) {
                    var nf = st.todo.upload[a].nfile;
@@ -2255,6 +2246,9 @@ function up2k_init(subtle) {
        if (keepalive)
            console.log("sending keepalive handshake", t.name, t);

+        if (!t.srch && !t.t_handshake)
+            pvis.seth(t.n, 2, L.u_hs);
+
        var xhr = new XMLHttpRequest();
        xhr.onerror = xhr.ontimeout = function () {
            if (t.t_busied != me)  // t.done ok
@@ -2281,7 +2275,8 @@ function up2k_init(subtle) {
                    apop(st.busy.handshake, t);
                    st.todo.handshake.unshift(t);
                    t.cooldown = Date.now() + 5000 + Math.floor(Math.random() * 3000);
-                    return toast.err(0, 'Handshake error; will retry...\n\n' + L.badreply + ':\n\n' + unpre(xhr.responseText));
+                    var txt = t.t_uploading ? L.u_ehsfin : t.srch ? L.u_ehssrch : L.u_ehsinit;
+                    return toast.err(0, txt + '\n\n' + L.badreply + ':\n\n' + unpre(xhr.responseText));
                }

                t.t_handshake = Date.now();
@@ -2460,6 +2455,7 @@ function up2k_init(subtle) {
                pvis.seth(t.n, 2, L.u_ehstmp, t);

                var err = "",
+                    cls = "ERROR",
                    rsp = unpre(xhr.responseText),
                    ofs = rsp.lastIndexOf('\nURL: ');

@@ -2489,6 +2485,8 @@ function up2k_init(subtle) {
                    if (!t.rechecks && (err_pend || err_srcb)) {
                        t.rechecks = 0;
                        t.want_recheck = true;
+                        err = L.u_dupdefer;
+                        cls = 'defer';
                    }
                }
                if (rsp.indexOf('server HDD is full') + 1)
@@ -2498,7 +2496,7 @@ function up2k_init(subtle) {
                    if (!t.t_uploading)
                        st.bytes.finished += t.size;

-                    pvis.seth(t.n, 1, "ERROR");
+                    pvis.seth(t.n, 1, cls);
                    pvis.seth(t.n, 2, err);
                    pvis.move(t.n, 'ng');

@@ -2530,7 +2528,8 @@ function up2k_init(subtle) {

        xhr.open('POST', t.purl, true);
        xhr.responseType = 'text';
-        xhr.timeout = 42000;
+        xhr.timeout = 42000 + (t.srch || t.t_uploaded ? 0 :
+            (t.size / (1048 * 20))); // safededup 20M/s hdd
        xhr.send(JSON.stringify(req));
    }

--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -1,3 +1,16 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-0902-0108  `v1.14.4`  another
+
+## recent important news
+
+* [v1.14.3 (2024-08-30)](https://github.com/9001/copyparty/releases/tag/v1.14.3) fixed a bug that was introduced in v1.13.8 (2024-08-13); this bug could lead to **data loss** -- see the v1.14.3 release-notes for details
+
+## bugfixes
+
+* a network glitch could cause the uploader UI to panic d9e95262
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2024-0830-2311  `v1.14.3`  important dedup fix

--- a/docs/examples/windows.md
+++ b/docs/examples/windows.md
@@ -16,7 +16,7 @@ open up notepad and save the following as `c:\users\you\documents\party.conf` (f
 ```yaml
 [global]
  lo: ~/logs/cpp-%Y-%m%d.xz  # log to c:\users\you\logs\
-  e2dsa, e2ts, no-dedup, z   # sets 4 flags; see expl.
+  e2dsa, e2ts, z    # sets 3 flags; see explanation
  p: 80, 443  # listen on ports 80 and 443, not 3923
  theme: 2    # default theme: protonmail-monokai
  lang: nor   # default language: viking
@@ -46,11 +46,10 @@ open up notepad and save the following as `c:\users\you\documents\party.conf` (f

 ### config explained: [global]

-the `[global]` section accepts any config parameters [listed here](https://ocv.me/copyparty/helptext.html), also viewable by running copyparty (either the exe or the sfx.py) with `--help`, so this is the same as running copyparty with arguments `--lo c:\users\you\logs\copyparty-%Y-%m%d.xz -e2dsa -e2ts --no-dedup -z -p 80,443 --theme 2 --lang nor`
+the `[global]` section accepts any config parameters [listed here](https://ocv.me/copyparty/helptext.html), also viewable by running copyparty (either the exe or the sfx.py) with `--help`, so this is the same as running copyparty with arguments `--lo c:\users\you\logs\copyparty-%Y-%m%d.xz -e2dsa -e2ts -z -p 80,443 --theme 2 --lang nor`
 * `lo: ~/logs/cpp-%Y-%m%d.xz` writes compressed logs (the compression will make them delayed)
-* `e2dsa` enables the upload deduplicator and file indexer, which enables searching
+* `e2dsa` enables the file indexer, which enables searching and upload-undo
 * `e2ts` enables music metadata indexing, making albums / titles etc. searchable too
-* `no-dedup` writes full dupes to disk instead of symlinking, since lots of windows software doesn't handle symlinks well
  * but the improved upload speed from `e2dsa` is not affected
 * `z` enables zeroconf, making the server available at `http://HOSTNAME.local/` from any other machine in the LAN
 * `p: 80,443` listens on the ports `80` and `443` instead of the default `3923`
--- a/scripts/docker/Dockerfile.ac
+++ b/scripts/docker/Dockerfile.ac
@@ -8,7 +8,7 @@ LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
 ENV     XDG_CONFIG_HOME=/cfg

 RUN     apk --no-cache add !pyc \
-            wget \
+            tzdata wget \
            py3-jinja2 py3-argon2-cffi py3-pillow \
            ffmpeg

--- a/scripts/docker/Dockerfile.dj
+++ b/scripts/docker/Dockerfile.dj
@@ -11,7 +11,7 @@ COPY    i/bin/mtag/install-deps.sh ./
 COPY    i/bin/mtag/audio-bpm.py /mtag/
 COPY    i/bin/mtag/audio-key.py /mtag/
 RUN     apk add -U !pyc \
-            wget \
+            tzdata wget \
            py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
            ffmpeg \
            vips-jxl vips-heif vips-poppler vips-magick \
--- a/scripts/docker/Dockerfile.im
+++ b/scripts/docker/Dockerfile.im
@@ -8,7 +8,7 @@ LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
 ENV     XDG_CONFIG_HOME=/cfg

 RUN     apk --no-cache add !pyc \
-            wget \
+            tzdata wget \
            py3-jinja2 py3-argon2-cffi py3-pillow py3-mutagen

 COPY    i/dist/copyparty-sfx.py innvikler.sh ./
--- a/scripts/docker/Dockerfile.iv
+++ b/scripts/docker/Dockerfile.iv
@@ -8,7 +8,7 @@ LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
 ENV     XDG_CONFIG_HOME=/cfg

 RUN     apk add -U !pyc \
-            wget \
+            tzdata wget \
            py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
            ffmpeg \
            vips-jxl vips-heif vips-poppler vips-magick \
--- a/scripts/docker/make.sh
+++ b/scripts/docker/make.sh
@@ -100,11 +100,11 @@ filt=
            aa="$(printf '%11s' $a-$i)"

            # arm takes forever so make it top priority
-            [ ${a::3} == arm ] && nice= || nice=nice
+            [ ${a::3} == arm ] && nice= || nice=-n20

            # --pull=never does nothing at all btw
            (set -x
-            $nice podman build \
+            nice $nice podman build \
                --squash \
                --pull=never \
                --from localhost/alpine-$a \
--- a/scripts/make-sfx.sh
+++ b/scripts/make-sfx.sh
@@ -609,7 +609,7 @@ pc="bzip2 -"; pe=bz2
 [ $use_gzz ] && pc="pigz -11 -I$use_gzz" && pe=gz

 echo compressing tar
-for n in {2..9}; do cp tar t.$n; nice $pc$n t.$n & done; wait
+for n in {2..9}; do cp tar t.$n; nice -n20 $pc$n t.$n & done; wait
 minf=$(for f in t.*.$pe; do
 	s1=$(wc -c <$f)
 	s2=$(tr -d '\r\n\0' <$f | wc -c)
--- a/scripts/pyinstaller/deps.sha512
+++ b/scripts/pyinstaller/deps.sha512
@@ -36,4 +36,4 @@ d1420c8417fad7888766dd26b9706a87c63e8f33dceeb8e26d0056d5127b0b3ed9272e44b4b76113
 2be320b4191f208cdd6af183c77ba2cf460ea52164ee45ac3ff17d6dfa57acd9deff016636c2dd42a21f4f6af977d5f72df7dacf599bebcf41757272354d14c1  pillow-10.4.0-cp312-cp312-win_amd64.whl
 896ddddbd4b85e86e0600cb65eb4c07fbc7f3802d47e7f660411e20b5500831469b97ed4770f25820f4e75cbfac40308da624fd86d4f62e578149d5c276a9cde  pyinstaller-6.10.0-py3-none-win_amd64.whl
 873781decaeef07f6a79b0ed8b9f35f3fa534a1ea0d866991e40278a10818fa5b60c70b0d5828971b045364f1099694cd1e5d5d60d480acb93fcfbfbced4a09e  pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
-0572c6345f6a4f7f3e5c2ff858e3ca7ca54ae4478f3d59d8e18cb0f596e61dcf12aef579db229e83d63b30f15d6684ee6bb3feaea9413e5e636a503933057678  python-3.12.5-amd64.exe
+37fa7250b10b0c03b87d800bf4f920589649309cb4fbd25864475084bb7873d62b809a4fdeabd06c79f03f33614218eb7e01a9bd796de29dd3b141f1906d588c  python-3.12.6-amd64.exe
--- a/scripts/pyinstaller/notes.txt
+++ b/scripts/pyinstaller/notes.txt
@@ -41,7 +41,7 @@ fns=(
  pillow-10.4.0-cp312-cp312-win_amd64.whl
  pyinstaller-6.10.0-py3-none-win_amd64.whl
  pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
-  python-3.12.5-amd64.exe
+  python-3.12.6-amd64.exe
 )
 [ $w7 ] && fns+=(  # u2c stuff
  certifi-2024.2.2-py3-none-any.whl
--- a/scripts/toc.sh
+++ b/scripts/toc.sh
@@ -20,7 +20,7 @@ cat $f | awk '
    o{next}
    /^#/{s=1;rs=0;pr()}
    /^#* *(nix package)/{rs=1}
-    /^#* *(install on android|dev env setup|just the sfx|complete release|optional gpl stuff|nixos module)|`$/{s=rs}
+    /^#* *(themes|install on android|dev env setup|just the sfx|complete release|optional gpl stuff|nixos module)|```/{s=rs}
    /^#/{
        lv=length($1);
        sub(/[^ ]+ /,"");
--- a/tests/test_dedup.py
+++ b/tests/test_dedup.py
@@ -31,6 +31,15 @@ class TestDedup(unittest.TestCase):
        os.chdir(td)
        return td

+    def cinit(self):
+        if self.conn:
+            self.fstab = self.conn.hsrv.hub.up2k.fstab
+            self.conn.hsrv.hub.up2k.shutdown()
+        self.asrv = AuthSrv(self.args, self.log)
+        self.conn = tu.VHttpConn(self.args, self.asrv, self.log, b"", True)
+        if self.fstab:
+            self.conn.hsrv.hub.up2k.fstab = self.fstab
+
    def test(self):
        quick = True  # sufficient for regular smoketests
        # quick = False
@@ -51,32 +60,70 @@ class TestDedup(unittest.TestCase):
        ]
        # (data, chash, wark)

-        # 3072 uploads in total
-        self.ctr = 3072
+        self.ctr = 336 if quick else 2016  # estimated total num uploads
        self.conn = None
-        fstab = None
+        self.fstab = None
        for e2d in [True, False]:
            self.args = Cfg(v=[".::A"], a=[], e2d=e2d)
-            for dn1, fn1, f1 in product(dirnames, filenames, files):
-                for dn2, fn2, f2 in product(dirnames, filenames, files):
-                    for dn3, fn3, f3 in product(dirnames, filenames, files):
-                        self.reset()
-                        if self.conn:
-                            fstab = self.conn.hsrv.hub.up2k.fstab
-                            self.conn.hsrv.hub.up2k.shutdown()
-                        self.asrv = AuthSrv(self.args, self.log)
-                        self.conn = tu.VHttpConn(
-                            self.args, self.asrv, self.log, b"", True
-                        )
-                        if fstab:
-                            self.conn.hsrv.hub.up2k.fstab = fstab
+            for cm1 in product(dirnames, filenames, files):
+                for cm2 in product(dirnames, filenames, files):
+                    if cm1 == cm2:
+                        continue
+                    for cm3 in product(dirnames, filenames, files):
+                        if cm3 in (cm1, cm2):
+                            continue
+
+                        f1 = cm1[2]
+                        f2 = cm2[2]
+                        f3 = cm3[2]
+                        if not e2d:
+                            rms = [-1]
+                        elif f1 == f2:
+                            if f1 == f3:
+                                rms = [0, 1, 2]
+                            else:
+                                rms = [0, 1]
+                        elif f1 == f3:
+                            rms = [0, 2]
+                        else:
+                            rms = [1, 2]
+
+                        for rm in rms:
+                            self.do_tc(cm1, cm2, cm3, rm)

-                        self.do_post(dn1, fn1, f1, True)
-                        self.do_post(dn2, fn2, f2, False)
-                        self.do_post(dn3, fn3, f3, False)
                        if quick:
                            break

+    def do_tc(self, cm1, cm2, cm3, irm):
+        dn1, fn1, f1 = cm1
+        dn2, fn2, f2 = cm2
+        dn3, fn3, f3 = cm3
+
+        self.reset()
+        self.cinit()
+
+        fn1 = self.do_post(dn1, fn1, f1, True)
+        fn2 = self.do_post(dn2, fn2, f2, False)
+        fn3 = self.do_post(dn3, fn3, f3, False)
+
+        if irm < 0:
+            return
+
+        cms = [(dn1, fn1, f1), (dn2, fn2, f2), (dn3, fn3, f3)]
+        rm = cms[irm]
+        dn, fn, _ = rm
+        h, b = self.curl("%s/%s?delete" % (dn, fn), meth="POST")
+        self.assertIn(" 200 OK", h)
+        self.assertIn("deleted 1 files", b)
+        h, b = self.curl("%s/%s" % (dn, fn))
+        self.assertIn(" 404 Not Fo", h)
+        for cm in cms:
+            if cm == rm:
+                continue
+            dn, fn, f = cm
+            h, b = self.curl("%s/%s" % (dn, fn))
+            self.assertEqual(b, f[0])
+
    def do_post(self, dn, fn, fi, first):
        print("\n\n# do_post", self.ctr, repr((dn, fn, fi, first)))
        self.ctr -= 1
@@ -101,6 +148,7 @@ class TestDedup(unittest.TestCase):

        h, b = self.curl("%s/%s" % (dn, sfn))
        self.assertEqual(b, data)
+        return sfn

    def handshake(self, dn, fn, fi):
        hdr = "POST /%s/ HTTP/1.1\r\nConnection: close\r\nContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n"
@@ -130,9 +178,10 @@ class TestDedup(unittest.TestCase):
        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
        self.assertEqual(ret[1], "thank")

-    def curl(self, url, binary=False):
-        h = "GET /%s HTTP/1.1\r\nConnection: close\r\n\r\n"
-        HttpCli(self.conn.setbuf((h % (url,)).encode("utf-8"))).run()
+    def curl(self, url, binary=False, meth=None):
+        h = "%s /%s HTTP/1.1\r\nConnection: close\r\n\r\n"
+        h = h % (meth or "GET", url)
+        HttpCli(self.conn.setbuf(h.encode("utf-8"))).run()
        if binary:
            h, b = self.conn.s._reply.split(b"\r\n\r\n", 1)
            return [h.decode("utf-8"), b]
--- a/tests/test_mv.py
+++ b/tests/test_mv.py
@@ -0,0 +1,198 @@
+#!/usr/bin/env python3
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import json
+import os
+import shutil
+import tempfile
+import unittest
+from itertools import product
+
+from copyparty.__init__ import PY2
+from copyparty.authsrv import AuthSrv
+from copyparty.httpcli import HttpCli
+from tests import util as tu
+from tests.util import Cfg
+
+"""
+TODO inject tags into db and verify ls
+"""
+
+
+class TestDedup(unittest.TestCase):
+    def setUp(self):
+        self.td = tu.get_ramdisk()
+
+    def tearDown(self):
+        os.chdir(tempfile.gettempdir())
+        shutil.rmtree(self.td)
+
+    def reset(self):
+        td = os.path.join(self.td, "vfs")
+        if os.path.exists(td):
+            shutil.rmtree(td)
+        os.mkdir(td)
+        os.chdir(td)
+        return td
+
+    def cinit(self):
+        if self.conn:
+            self.fstab = self.conn.hsrv.hub.up2k.fstab
+            self.conn.hsrv.hub.up2k.shutdown()
+        self.asrv = AuthSrv(self.args, self.log)
+        self.conn = tu.VHttpConn(self.args, self.asrv, self.log, b"", True)
+        if self.fstab:
+            self.conn.hsrv.hub.up2k.fstab = self.fstab
+
+    def test(self):
+        if PY2:
+            raise unittest.SkipTest()
+
+        # tc_e2d = [True, False]  # maybe-TODO only known symlinks are translated
+        tc_e2d = [True]
+        tc_dedup = ["sym", "no", "sym-no"]
+        tc_vols = [["::A"], ["::A", "d1:d1:A"]]
+        dirs = ["d1", "d1/d2", "d1/d2/d3", "d1/d4"]
+        files = [
+            (
+                "one",
+                "BfcDQQeKz2oG1CPSFyD5ZD1flTYm2IoCY23DqeeVgq6w",
+                "XMbpLRqVdtGmgggqjUI6uSoNMTqZVX4K6zr74XA1BRKc",
+            )
+        ]
+        # (data, chash, wark)
+
+        self.conn = None
+        self.fstab = None
+        self.ctr = 0  # 2304
+        tcgen = product(tc_e2d, tc_dedup, tc_vols, dirs, ["d9", "../d9"])
+        for e2d, dedup, vols, mv_from, dst in tcgen:
+            if "/" not in mv_from and dst.startswith(".."):
+                continue  # would move past top of fs
+            if len(vols) > 1 and mv_from == "d1":
+                continue  # cannot move a vol
+
+            # print(e2d, dedup, vols, mv_from, dst)
+            ka = {"e2d": e2d}
+            if dedup == "hard":
+                ka["hardlink"] = True
+            elif dedup == "no":
+                ka["no_dedup"] = True
+            self.args = Cfg(v=vols[:], a=[], **ka)
+
+            for u1, u2, u3, u4 in product(dirs, dirs, dirs, dirs):
+                ups = (u1, u2, u3, u4)
+                if len(set(ups)) < 4:
+                    continue  # not unique
+
+                t = "e2d:%s dedup:%s vols:%d from:%s to:%s"
+                t = t % (e2d, dedup, len(vols), mv_from, dst)
+                print("\n\n\033[0;7m# files:", ups, t, "\033[0m")
+
+                self.reset()
+                self.cinit()
+
+                for up in [u1, u2, u3, u4]:
+                    self.do_post(up, "fn", files[0], up == u1)
+
+                restore_args = None
+                if dedup == "sym-no":
+                    restore_args = self.args
+                    ka = {"e2d": e2d, "no_dedup": True}
+                    self.args = Cfg(v=vols[:], a=[], **ka)
+                    self.cinit()
+
+                mv_to = mv_from
+                for _ in range(2 if dst.startswith("../") else 1):
+                    mv_to = mv_from.rsplit("/", 1)[0] if "/" in mv_from else ""
+                mv_to += "/" + dst.lstrip("./")
+
+                self.do_mv(mv_from, mv_to)
+
+                for dirpath in [u1, u2, u3, u4]:
+                    if dirpath == mv_from:
+                        dirpath = mv_to
+                    elif dirpath.startswith(mv_from):
+                        dirpath = mv_to + dirpath[len(mv_from) :]
+                    h, b = self.curl(dirpath + "/fn")
+                    self.assertEqual(b, "one")
+
+                if restore_args:
+                    self.args = restore_args
+
+    def do_mv(self, src, dst):
+        hdr = "POST /%s?move=/%s HTTP/1.1\r\nConnection: close\r\nContent-Length: 0\r\n\r\n"
+        buf = (hdr % (src, dst)).encode("utf-8")
+        print("MV [%s] => [%s]" % (src, dst))
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        print("MV <-- ", ret)
+        self.assertIn(" 201 Created", ret[0])
+        self.assertEqual("k\r\n", ret[1])
+        return ret
+
+    def do_post(self, dn, fn, fi, first):
+        print("\n# do_post", self.ctr, repr((dn, fn, fi, first)))
+        self.ctr -= 1
+
+        data, chash, wark = fi
+        hs = self.handshake(dn, fn, fi)
+        self.assertEqual(hs["wark"], wark)
+
+        sfn = hs["name"]
+        if sfn == fn:
+            print("using original name " + fn)
+        else:
+            print(fn + " got renamed to " + sfn)
+            if first:
+                raise Exception("wait what")
+
+        if hs["hash"]:
+            self.assertEqual(hs["hash"][0], chash)
+            self.put_chunk(dn, wark, chash, data)
+        elif first:
+            raise Exception("found first; %r, %r" % ((dn, fn, fi), hs))
+
+        h, b = self.curl("%s/%s" % (dn, sfn))
+        self.assertEqual(b, data)
+
+    def handshake(self, dn, fn, fi):
+        hdr = "POST /%s/ HTTP/1.1\r\nConnection: close\r\nContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n"
+        msg = {"name": fn, "size": 3, "lmod": 1234567890, "life": 0, "hash": [fi[1]]}
+        buf = json.dumps(msg).encode("utf-8")
+        buf = (hdr % (dn, len(buf))).encode("utf-8") + buf
+        print("HS -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        print("HS <--", ret)
+        return json.loads(ret[1])
+
+    def put_chunk(self, dn, wark, chash, data):
+        msg = [
+            "POST /%s/ HTTP/1.1" % (dn,),
+            "Connection: close",
+            "Content-Type: application/octet-stream",
+            "Content-Length: 3",
+            "X-Up2k-Hash: " + chash,
+            "X-Up2k-Wark: " + wark,
+            "",
+            data,
+        ]
+        buf = "\r\n".join(msg).encode("utf-8")
+        print("PUT -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        self.assertEqual(ret[1], "thank")
+
+    def curl(self, url, binary=False):
+        h = "GET /%s HTTP/1.1\r\nConnection: close\r\n\r\n"
+        HttpCli(self.conn.setbuf((h % (url,)).encode("utf-8"))).run()
+        if binary:
+            h, b = self.conn.s._reply.split(b"\r\n\r\n", 1)
+            return [h.decode("utf-8"), b]
+
+        return self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def log(self, src, msg, c=0):
+        print(msg)
--- a/tests/util.py
+++ b/tests/util.py
@@ -117,16 +117,16 @@ class Cfg(Namespace):
    def __init__(self, a=None, v=None, c=None, **ka0):
        ka = {}

-        ex = "chpw daw dav_auth dav_inf dav_mac dav_rt e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp early_ban ed emp exp force_js getmod grid gsel hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_db_ip no_dedup no_del no_dupe no_lifetime no_logues no_mv no_pipe no_poll no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw og og_no_head og_s_title q rand smb srch_dbg stats uqe vague_403 vc ver write_uplog xdev xlink xvol zs"
+        ex = "chpw daw dav_auth dav_inf dav_mac dav_rt e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp early_ban ed emp exp force_js getmod grid gsel hardlink ih ihead magic hardlink_only nid nih no_acode no_athumb no_dav no_db_ip no_del no_dupe no_lifetime no_logues no_mv no_pipe no_poll no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw og og_no_head og_s_title q rand smb srch_dbg stats uqe vague_403 vc ver write_uplog xdev xlink xvol zs"
        ka.update(**{k: False for k in ex.split()})

-        ex = "dotpart dotsrch hook_v no_dhash no_fastboot no_fpool no_htp no_rescan no_sendfile no_snap no_voldump re_dhash plain_ip"
+        ex = "dedup dotpart dotsrch hook_v no_dhash no_fastboot no_fpool no_htp no_rescan no_sendfile no_snap no_voldump re_dhash plain_ip"
        ka.update(**{k: True for k in ex.split()})

        ex = "ah_cli ah_gen css_browser hist js_browser js_other mime mimes no_forget no_hash no_idx nonsus_urls og_tpl og_ua"
        ka.update(**{k: None for k in ex.split()})

-        ex = "hash_mt srch_time u2abort u2j u2sz"
+        ex = "hash_mt safe_dedup srch_time u2abort u2j u2sz"
        ka.update(**{k: 1 for k in ex.split()})

        ex = "au_vol mtab_age reg_cap s_thead s_tbody th_convt"
Author	SHA1	Message	Date
ed	c140eeee6b	v1.15.0	2024-09-08 19:25:46 +00:00
ed	c5988a04f9	up2k.js: bump handshake timeout for safededup	2024-09-08 18:06:37 +00:00
ed	a2e0f98693	disable upload deduplication by default; dedup is still encouraged and fully supported, but being default-enabled has caused too many surprises enabling `--dedup` restores the previous default behavior also renames `--never-symlink` to `--hardlink-only`	2024-09-08 17:09:14 +00:00
ed	1111153f06	test dedup relinking	2024-09-08 12:55:27 +00:00
ed	e5a836cb7d	og: fix links to textfiles	2024-09-08 12:12:34 +00:00
ed	b0de84cbc5	db-verify: support newlines in filenames + flag 404s	2024-09-08 00:44:22 +00:00
ed	cbb718e10d	css fixes: * improve hotdog-stand theme * fix up2k tabs glow (went poof in a syntax error)	2024-09-07 19:29:40 +00:00
ed	b5ad9369fe	confine xlink behavior behind its volflag symlinks between volumes will only be created if xlink is enabled, so such symlinks should be ignored if xlink is disabled, as they might originate from other software this prevents accidental rewriting of non-dedup symlinks	2024-09-07 19:17:32 +00:00
ed	4401de0413	fix mv with `--no-dedup` in volumes with dupes; if --no-dedup was enabled in a volume which already contained symlinked duplicate files, renaming/moving folders could fail this is due to folder contents being moved one file at a time (which is how symlink breakage is prevented) except the links are moved assuming the final directory layout, meaning they may be intermittently broken during the movie with no-dedup, the symlinks are converted into full files as each symlink is encountered, but a temporarily broken symlink would crash the procedure fix this by giving `_symlink` a new parameter `fsrc` which is a known valid inode for data copying purposes	2024-09-07 00:47:12 +00:00
ed	6e671c5245	verify on-disk contents before dedup; previously, the assumption was made that the database and filesystem would not desync, and that an upload could safely be substituted with a symlink to an existing copy on-disk, assuming said copy still existed on-disk at all this is fine if copyparty is the only software that makes changes to the filesystem, but that is a shitty assumption to make in hindsight add `--safe-dedup` which takes a "safety level", and by default (50) it will no longer blindly expect that the filesystem has not been altered through other means; the file contents will now be hashed and compared to the database deduplication can be much slower as a result, but definitely worth it as this avoids some potentially very unpleasant surprises the previous behavior can be restored with `--safe-dedup 1`	2024-09-06 19:08:14 +00:00
ed	08848be784	u2c: add hashgen mode + fix shutdown lag	2024-09-06 00:31:25 +00:00
ed	b599fbae97	use local timezone in log messages; closes #96 timezone can be changed with `export TZ=Europe/Oslo` before launch using naive timestamps like this appears to be safe as of 3.13-rc1, no deprecation warnings, just a tiny bit slower than assuming UTC	2024-09-05 19:31:33 +00:00
ed	a8dabc99f6	add more translations	2024-09-04 23:46:32 +00:00
ed	f1130db131	fix confusing message when uploading dupes due to deduplication, it is intentionally impossible to upload several identical copies of a file in parallel by default, the up2k client will upload files sorted by size, which usually leads to dupes being grouped together, and it will try to do just that this is by design, as it improves performance on average, but it also shows the confusing (but technically-correct) message "resume the partial upload into the original path" fix this with a more appropriate message note that this approach was selected in favor of pausing handshakes while the initial copy finishes uploading, because that could severely reduce upload performance by preventing optimal use of multiple connections	2024-09-04 22:03:26 +00:00
ed	735ec35546	update pkgs to 1.14.4	2024-09-02 01:21:07 +00:00