v1.15.0

dedup is still encouraged and fully supported, but
being default-enabled has caused too many surprises

enabling `--dedup` restores the previous default behavior

also renames `--never-symlink` to `--hardlink-only`

README.md

@@ -1,4 +1,6 @@
-# 💾🎉 copyparty
+<img src="docs/logo.svg" width="250" align="right"/>
+
+### 💾🎉 copyparty
 
 turn almost any device into a file server with resumable uploads/downloads using [*any*](#browser-support) web browser
 
@@ -63,7 +65,8 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [smb server](#smb-server) - unsafe, slow, not recommended for wan
     * [browser ux](#browser-ux) - tweaking the ui
     * [opengraph](#opengraph) - discord and social-media embeds
-    * [file indexing](#file-indexing) - enables dedup and music search ++
+    * [file deduplication](#file-deduplication) - enable symlink-based upload deduplication
+    * [file indexing](#file-indexing) - enable music search, upload-undo, and better dedup
         * [exclude-patterns](#exclude-patterns) - to save some time
         * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
         * [periodic rescan](#periodic-rescan) - filesystem monitoring
@@ -750,14 +753,16 @@ you can move files across browser tabs (cut in one tab, paste in another)
 
 share a file or folder by creating a temporary link
 
-when enabled in the server settings (`--shr`), click the bottom-right `share` button to share the folder you're currently in, or select a file first to share only that file
+when enabled in the server settings (`--shr`), click the bottom-right `share` button to share the folder you're currently in, or alternatively:
+* select a folder first to share that folder instead
+* select one or more files to share only those files
 
 this feature was made with [identity providers](#identity-providers) in mind -- configure your reverseproxy to skip the IdP's access-control for a given URL prefix and use that to safely share specific files/folders sans the usual auth checks
 
 when creating a share, the creator can choose any of the following options:
 
 * password-protection
-* expire after a certain time
+* expire after a certain time; `0` or blank means infinite
 * allow visitors to upload (if the user who creates the share has write-access)
 
 semi-intentional limitations:
@@ -768,10 +773,17 @@ semi-intentional limitations:
   * when linking something to discord (for example) it'll get accessed by their scraper and that would count as a hit
   * browsers wouldn't be able to resume a broken download unless the requester's IP gets allowlisted for X minutes (ref. tricky)
 
-the links are created inside a specific toplevel folder which must be specified with server-config `--shr`, for example `--shr /share/` (this also enables the feature)
+specify `--shr /foobar` to enable this feature; a toplevel virtual folder named `foobar` is then created, and that's where all the shares will be served from
+
+* you can name it whatever, `foobar` is just an example
+* if you're using config files, put `shr: /foobar` inside the `[global]` section instead
 
 users can delete their own shares in the controlpanel, and a list of privileged users (`--shr-adm`) are allowed to see and/or delet any share on the server
 
+after a share has expired, it remains visible in the controlpanel for `--shr-rt` minutes (default is 1 day), and the owner can revive it by extending the expiration time there
+
+**security note:** using this feature does not mean that you can skip the [accounts and volumes](#accounts-and-volumes) section -- you still need to restrict access to volumes that you do not intend to share with unauthenticated users! it is not sufficient to use rules in the reverseproxy to restrict access to just the `/share` folder.
+
 
 ## batch rename
 
@@ -1144,9 +1156,41 @@ NOTE: because discord (and maybe others) strip query args such as `?raw` in open
 if you want to entirely replace the copyparty response with your own jinja2 template, give the template filepath to `--og-tpl` or volflag `og_tpl` (all members of `HttpCli` are available through the `this` object)
 
 
+## file deduplication
+
+enable symlink-based upload deduplication  globally with `--dedup` or per-volume with volflag `dedup`
+
+when someone tries to upload a file that already exists on the server, the upload will be politely declined and a symlink is created instead, pointing to the nearest copy on disk, thus reducinc disk space usage
+
+**warning:** when enabling dedup, you should also:
+* enable indexing with `-e2dsa` or volflag `e2dsa` (see [file indexing](#file-indexing) section below); strongly recommended
+* ...and/or `--hardlink-only` to use hardlink-based deduplication instead of symlinks; see explanation below
+
+it will not be safe to rename/delete files if you only enable dedup and none of the above; if you enable indexing then it is not *necessary* to also do hardlinks (but you may still want to)
+
+by default, deduplication is done based on symlinks (symbolic links); these are tiny files which are pointers to the nearest full copy of the file
+
+you can choose to use hardlinks instead of softlinks, globally with `--hardlink-only` or volflag `hardlinkonly`;
+
+advantages of using hardlinks:
+* hardlinks are more compatible with other software; they behave entirely like regular files
+* you can safely move and rename files using other file managers
+  * symlinks need to be managed by copyparty to ensure the destinations remain correct
+
+advantages of using symlinks (default):
+* each symlink can have its own last-modified timestamp, but a single timestamp is shared by all hardlinks
+* symlinks make it more obvious to other software that the file is not a regular file, so this can be less dangerous
+  * hardlinks look like regular files, so other software may assume they are safe to edit without affecting the other copies
+
+**warning:** if you edit the contents of a deduplicated file, then you will also edit all other copies of that file! This is especially surprising with hardlinks, because they look like regular files, but that same file exists in multiple locations
+
+global-option `--xlink` / volflag `xlink` additionally enables deduplication across volumes, but this is probably buggy and not recommended
+
+
+
 ## file indexing
 
-enables dedup and music search ++
+enable music search, upload-undo, and better dedup
 
 file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volflags, or a mix of both.
 
@@ -1160,7 +1204,6 @@ through arguments:
 * `-e2v` verfies file integrity at startup, comparing hashes from the db
 * `-e2vu` patches the database with the new hashes from the filesystem
 * `-e2vp` panics and kills copyparty instead
-* `--xlink` enables deduplication across volumes
 
 the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts`, `d2v` for disabling:
 * `-v ~/music::r:c,e2ds,e2tsr` does a full reindex of everything on startup
@@ -1173,7 +1216,6 @@ note:
 * upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
-* deduplication is possible on windows if you run copyparty as administrator (not saying you should!)
 
 ### exclude-patterns
 
@@ -1913,6 +1955,8 @@ below are some tweaks roughly ordered by usefulness:
 * `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
 * `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
   * and also makes thumbnails load faster, regardless of e2d/e2t
+* `--dedup` enables deduplication and thus avoids writing to the HDD if someone uploads a dupe
+* `--safe-dedup 1` makes deduplication much faster during upload by skipping verification of file contents; safe if there is no other software editing/moving the files in the volumes
 * `--no-hash .` when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable
 * if your volumes are on a network-disk such as NFS / SMB / s3, specifying larger values for `--iobuf` and/or `--s-rd-sz` and/or `--s-wr-sz` may help; try setting all of them to `524288` or `1048576` or `4194304`
 * `--no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1` minimizes the number of threads; can help in some eccentric environments (like the vscode debugger)
@@ -1967,7 +2011,7 @@ safety profiles:
   * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
     * however note if you edit one file it will also affect the other copies
   * `--vague-403` returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
-  * `--nih` removes the server hostname from directory listings
+  * `-nih` removes the server hostname from directory listings
 
 * option `-sss` is a shortcut for the above plus:
   * `--no-dav` disables webdav support

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "1.22"
-S_BUILD_DT = "2024-08-08"
+S_VERSION = "1.24"
+S_BUILD_DT = "2024-09-05"
 
 """
 u2c.py: upload to copyparty
@@ -41,19 +41,25 @@ except:
 
 try:
     import requests
+
+    req_ses = requests.Session()
 except ImportError as ex:
-    if EXE:
+    if "-" in sys.argv or "-h" in sys.argv:
+        m = ""
+    elif EXE:
         raise
     elif sys.version_info > (2, 7):
-        m = "\nERROR: need 'requests'; please run this command:\n {0} -m pip install --user requests\n"
+        m = "\nERROR: need 'requests'{0}; please run this command:\n {1} -m pip install --user requests\n"
     else:
         m = "requests/2.18.4 urllib3/1.23 chardet/3.0.4 certifi/2020.4.5.1 idna/2.7"
         m = ["   https://pypi.org/project/" + x + "/#files" for x in m.split()]
-        m = "\n  ERROR: need these:\n" + "\n".join(m) + "\n"
+        m = "\n  ERROR: need these{0}:\n" + "\n".join(m) + "\n"
         m += "\n  for f in *.whl; do unzip $f; done; rm -r *.dist-info\n"
 
-    print(m.format(sys.executable), "\nspecifically,", ex)
-    sys.exit(1)
+    if m:
+        t = " when not running with '-h' or url '-'"
+        print(m.format(t, sys.executable), "\nspecifically,", ex)
+        sys.exit(1)
 
 
 # from copyparty/__init__.py
@@ -76,7 +82,22 @@ else:
 VT100 = platform.system() != "Windows"
 
 
-req_ses = requests.Session()
+try:
+    UTC = datetime.timezone.utc
+except:
+    TD_ZERO = datetime.timedelta(0)
+
+    class _UTC(datetime.tzinfo):
+        def utcoffset(self, dt):
+            return TD_ZERO
+
+        def tzname(self, dt):
+            return "UTC"
+
+        def dst(self, dt):
+            return TD_ZERO
+
+    UTC = _UTC()
 
 
 class Daemon(threading.Thread):
@@ -271,6 +292,12 @@ class MTHash(object):
 _print = print
 
 
+def safe_print(*a, **ka):
+    ka["end"] = ""
+    zs = " ".join([unicode(x) for x in a])
+    _print(zs + "\n", **ka)
+
+
 def eprint(*a, **ka):
     ka["file"] = sys.stderr
     ka["end"] = ""
@@ -284,18 +311,17 @@ def eprint(*a, **ka):
 
 def flushing_print(*a, **ka):
     try:
-        _print(*a, **ka)
+        safe_print(*a, **ka)
     except:
         v = " ".join(str(x) for x in a)
         v = v.encode("ascii", "replace").decode("ascii")
-        _print(v, **ka)
+        safe_print(v, **ka)
 
     if "flush" not in ka:
         sys.stdout.flush()
 
 
-if not VT100:
-    print = flushing_print
+print = safe_print if VT100 else flushing_print
 
 
 def termsize():
@@ -770,8 +796,6 @@ class Ctl(object):
             self.up_c = 0
             self.up_b = 0
             self.up_br = 0
-            self.hasher_busy = 1
-            self.handshaker_busy = 0
             self.uploader_busy = 0
             self.serialized = False
 
@@ -781,6 +805,9 @@ class Ctl(object):
             self.eta = "99:99:99"
 
             self.mutex = threading.Lock()
+            self.exit_cond = threading.Condition()
+            self.uploader_alive = ar.j
+            self.handshaker_alive = ar.j
             self.q_handshake = Queue()  # type: Queue[File]
             self.q_upload = Queue()  # type: Queue[FileSlice]
 
@@ -851,27 +878,21 @@ class Ctl(object):
             Daemon(self.handshaker)
             Daemon(self.uploader)
 
-        idles = 0
-        while idles < 3:
-            time.sleep(0.07)
+        while True:
+            with self.exit_cond:
+                self.exit_cond.wait(0.07)
             with self.mutex:
-                if (
-                    self.q_handshake.empty()
-                    and self.q_upload.empty()
-                    and not self.hasher_busy
-                    and not self.handshaker_busy
-                    and not self.uploader_busy
-                ):
-                    idles += 1
-                else:
-                    idles = 0
+                if not self.handshaker_alive and not self.uploader_alive:
+                    break
+                st_hash = self.st_hash[:]
+                st_up = self.st_up[:]
 
             if VT100 and not self.ar.ns:
                 maxlen = ss.w - len(str(self.nfiles)) - 14
                 txt = "\033[s\033[{0}H".format(ss.g)
                 for y, k, st, f in [
-                    [0, "hash", self.st_hash, self.hash_f],
-                    [1, "send", self.st_up, self.up_f],
+                    [0, "hash", st_hash, self.hash_f],
+                    [1, "send", st_up, self.up_f],
                 ]:
                     txt += "\033[{0}H{1}:".format(ss.g + y, k)
                     file, arg = st
@@ -1027,11 +1048,42 @@ class Ctl(object):
                 self.hash_f += 1
                 self.hash_c += len(file.cids)
                 self.hash_b += file.size
+                if self.ar.wlist:
+                    self.up_f = self.hash_f
+                    self.up_c = self.hash_c
+                    self.up_b = self.hash_b
+
+            if self.ar.wlist:
+                zsl = [self.ar.wsalt, str(file.size)] + [x[0] for x in file.kchunks]
+                zb = hashlib.sha512("\n".join(zsl).encode("utf-8")).digest()[:33]
+                wark = base64.urlsafe_b64encode(zb).decode("utf-8")
+                vp = file.rel.decode("utf-8")
+                if self.ar.jw:
+                    print("%s  %s" % (wark, vp))
+                else:
+                    zd = datetime.datetime.fromtimestamp(file.lmod, UTC)
+                    dt = "%04d-%02d-%02d %02d:%02d:%02d" % (
+                        zd.year,
+                        zd.month,
+                        zd.day,
+                        zd.hour,
+                        zd.minute,
+                        zd.second,
+                    )
+                    print("%s %12d %s %s" % (dt, file.size, wark, vp))
+                continue
 
             self.q_handshake.put(file)
 
-        self.hasher_busy = 0
         self.st_hash = [None, "(finished)"]
+        self._check_if_done()
+
+    def _check_if_done(self):
+        with self.mutex:
+            if self.nfiles - self.up_f:
+                return
+        for _ in range(self.ar.j):
+            self.q_handshake.put(None)
 
     def handshaker(self):
         search = self.ar.s
@@ -1039,8 +1091,10 @@ class Ctl(object):
         while True:
             file = self.q_handshake.get()
             if not file:
+                with self.mutex:
+                    self.handshaker_alive -= 1
                 self.q_upload.put(None)
-                break
+                return
 
             upath = file.abs.decode("utf-8", "replace")
             if not VT100:
@@ -1052,9 +1106,6 @@ class Ctl(object):
                 self.errs += 1
                 continue
 
-            with self.mutex:
-                self.handshaker_busy += 1
-
             while time.time() < file.cd:
                 time.sleep(0.1)
 
@@ -1062,17 +1113,17 @@ class Ctl(object):
             if search:
                 if hs:
                     for hit in hs:
-                        t = "found: {0}\n  {1}{2}\n"
-                        print(t.format(upath, burl, hit["rp"]), end="")
+                        t = "found: {0}\n  {1}{2}"
+                        print(t.format(upath, burl, hit["rp"]))
                 else:
-                    print("NOT found: {0}\n".format(upath), end="")
+                    print("NOT found: {0}".format(upath))
 
                 with self.mutex:
                     self.up_f += 1
                     self.up_c += len(file.cids)
                     self.up_b += file.size
-                    self.handshaker_busy -= 1
 
+                self._check_if_done()
                 continue
 
             if file.recheck:
@@ -1104,7 +1155,6 @@ class Ctl(object):
                         file.up_b -= sz
 
                 file.ucids = hs
-                self.handshaker_busy -= 1
 
             if not hs:
                 self.at_hash += file.t_hash
@@ -1130,6 +1180,9 @@ class Ctl(object):
                     kw = "uploaded" if file.up_b else "   found"
                     print("{0} {1}".format(kw, upath))
 
+                self._check_if_done()
+                continue
+
             chunksz = up2k_chunksize(file.size)
             njoin = (self.ar.sz * 1024 * 1024) // chunksz
             cs = hs[:]
@@ -1149,8 +1202,16 @@ class Ctl(object):
         while True:
             fsl = self.q_upload.get()
             if not fsl:
-                self.st_up = [None, "(finished)"]
-                break
+                done = False
+                with self.mutex:
+                    self.uploader_alive -= 1
+                    if not self.uploader_alive:
+                        done = not self.handshaker_alive
+                        self.st_up = [None, "(finished)"]
+                if done:
+                    with self.exit_cond:
+                        self.exit_cond.notify_all()
+                return
 
             file = fsl.file
             cids = fsl.cids
@@ -1236,7 +1297,7 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("-v", action="store_true", help="verbose")
     ap.add_argument("-a", metavar="PASSWD", help="password or $filepath")
     ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
-    ap.add_argument("-x", type=unicode, metavar="REGEX", default="", help="skip file if filesystem-abspath matches REGEX, example: '.*/\\.hist/.*'")
+    ap.add_argument("-x", type=unicode, metavar="REGEX", action="append", help="skip file if filesystem-abspath matches REGEX (option can be repeated), example: '.*/\\.hist/.*'")
     ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
     ap.add_argument("--touch", action="store_true", help="if last-modified timestamps differ, push local to server (need write+delete perms)")
     ap.add_argument("--ow", action="store_true", help="overwrite existing files instead of autorenaming")
@@ -1252,6 +1313,10 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("--dr", action="store_true", help="delete remote files which don't exist locally (implies --ow)")
     ap.add_argument("--drd", action="store_true", help="delete remote files during upload instead of afterwards; reduces peak disk space usage, but will reupload instead of detecting renames")
 
+    ap = app.add_argument_group("file-ID calculator; enable with url '-' to list warks (file identifiers) instead of upload/search")
+    ap.add_argument("--wsalt", type=unicode, metavar="S", default="hunter2", help="salt to use when creating warks; must match server config")
+    ap.add_argument("--jw", action="store_true", help="just identifier+filepath, not mtime/size too")
+
     ap = app.add_argument_group("performance tweaks")
     ap.add_argument("-j", type=int, metavar="CONNS", default=2, help="parallel connections")
     ap.add_argument("-J", type=int, metavar="CORES", default=hcores, help="num cpu-cores to use for hashing; set 0 or 1 for single-core hashing")
@@ -1283,7 +1348,11 @@ source file/folder selection uses rsync syntax, meaning that:
     if ar.dr:
         ar.ow = True
 
-    for k in "dl dr drd".split():
+    ar.x = "|".join(ar.x or [])
+
+    setattr(ar, "wlist", ar.url == "-")
+
+    for k in "dl dr drd wlist".split():
         errs = []
         if ar.safe and getattr(ar, k):
             errs.append(k)

contrib/package/arch/PKGBUILD

@@ -1,6 +1,6 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.13.8"
+pkgver="1.14.4"
 pkgrel=1
 pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
@@ -21,7 +21,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("ddb7a7247cff7aa72254036c9f9ad66bbd45afdde5241ee60ca51d0d355b5392")
+sha256sums=("1e8004e4369e59487c47a0a9949668de704b1884beda0421887e342edcff0961")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.13.8/copyparty-sfx.py",
-    "version": "1.13.8",
-    "hash": "sha256-J2m9dK0lGG3twNvPPkGWUpzD7OLTEskBUmtwPoZ2qEE="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.14.4/copyparty-sfx.py",
+    "version": "1.14.4",
+    "hash": "sha256-nfcSXddM0jeHr7nGwoguOv/eq750bpHrImKmmODxK6E="
 }

contrib/plugins/banner.js

@@ -1,8 +1,8 @@
 (function() {
 
 // usage: copy this to '.banner.js' in your webroot,
-// and run copyparty with the following argument:
-// --body-foot '<script src="/.banner.js"></script>'
+// and run copyparty with the following arguments:
+// --js-browser /.banner.js  --js-other /.banner.js
 
 
 

copyparty/__main__.py

@@ -975,9 +975,10 @@ def add_fs(ap):
 def add_share(ap):
     db_path = os.path.join(E.cfg, "shares.db")
     ap2 = ap.add_argument_group('share-url options')
-    ap2.add_argument("--shr", metavar="URL", default="", help="base url for shared files, for example [\033[32m/share\033[0m] (must be a toplevel subfolder)")
-    ap2.add_argument("--shr-db", metavar="PATH", default=db_path, help="database to store shares in")
-    ap2.add_argument("--shr-adm", metavar="U,U", default="", help="comma-separated list of users allowed to view/delete any share")
+    ap2.add_argument("--shr", metavar="DIR", type=u, default="", help="toplevel virtual folder for shared files/folders, for example [\033[32m/share\033[0m]")
+    ap2.add_argument("--shr-db", metavar="FILE", type=u, default=db_path, help="database to store shares in")
+    ap2.add_argument("--shr-adm", metavar="U,U", type=u, default="", help="comma-separated list of users allowed to view/delete any share")
+    ap2.add_argument("--shr-rt", metavar="MIN", type=int, default=1440, help="shares can be revived by their owner if they expired less than MIN minutes ago; [\033[32m60\033[0m]=hour, [\033[32m1440\033[0m]=day, [\033[32m10080\033[0m]=week")
     ap2.add_argument("--shr-v", action="store_true", help="debug")
 
 
@@ -991,9 +992,10 @@ def add_upload(ap):
     ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without \033[33m-e2d\033[0m; roughly 1 MiB RAM per 600")
     ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (bad idea to enable this on windows and/or cow filesystems)")
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
-    ap2.add_argument("--hardlink", action="store_true", help="prefer hardlinks instead of symlinks when possible (within same filesystem) (volflag=hardlink)")
-    ap2.add_argument("--never-symlink", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=neversymlink)")
-    ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead (volflag=copydupes)")
+    ap2.add_argument("--dedup", action="store_true", help="enable symlink-based upload deduplication (volflag=dedup)")
+    ap2.add_argument("--safe-dedup", metavar="N", type=int, default=50, help="how careful to be when deduplicating files; [\033[32m1\033[0m] = just verify the filesize, [\033[32m50\033[0m] = verify file contents have not been altered (volflag=safededup)")
+    ap2.add_argument("--hardlink", action="store_true", help="enable hardlink-based dedup; will fallback on symlinks when that is impossible (across filesystems) (volflag=hardlink)")
+    ap2.add_argument("--hardlink-only", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=hardlinkonly)")
     ap2.add_argument("--no-dupe", action="store_true", help="reject duplicate files during upload; only matches within the same volume (volflag=nodupe)")
     ap2.add_argument("--no-snap", action="store_true", help="disable snapshots -- forget unfinished uploads on shutdown; don't create .hist/up2k.snap files -- abandoned/interrupted uploads must be cleaned up manually")
     ap2.add_argument("--snap-wri", metavar="SEC", type=int, default=300, help="write upload state to ./hist/up2k.snap every \033[33mSEC\033[0m seconds; allows resuming incomplete uploads after a server crash")
@@ -1284,6 +1286,7 @@ def add_logging(ap):
     ap2.add_argument("--ansi", action="store_true", help="force colors; overrides environment-variable NO_COLOR")
     ap2.add_argument("--no-logflush", action="store_true", help="don't flush the logfile after each write; tiny bit faster")
     ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
+    ap2.add_argument("--log-utc", action="store_true", help="do not use local timezone; assume the TZ env-var is UTC (tiny bit faster)")
     ap2.add_argument("--log-tdec", metavar="N", type=int, default=3, help="timestamp resolution / number of timestamp decimals")
     ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log failed login attempt passwords: 0=terse, 1=plaintext, 2=hashed")
     ap2.add_argument("--log-conn", action="store_true", help="debug: print tcp-server msgs")
@@ -1342,7 +1345,7 @@ def add_transcoding(ap):
 def add_db_general(ap, hcores):
     noidx = APPLESAN_TXT if MACOS else ""
     ap2 = ap.add_argument_group('general db options')
-    ap2.add_argument("-e2d", action="store_true", help="enable up2k database, making files searchable + enables upload deduplication")
+    ap2.add_argument("-e2d", action="store_true", help="enable up2k database; this enables file search, upload-undo, improves deduplication")
     ap2.add_argument("-e2ds", action="store_true", help="scan writable folders for new files on startup; sets \033[33m-e2d\033[0m")
     ap2.add_argument("-e2dsa", action="store_true", help="scans all folders on startup; sets \033[33m-e2ds\033[0m")
     ap2.add_argument("-e2v", action="store_true", help="verify file integrity; rehash all files and compare with db")
@@ -1355,7 +1358,7 @@ def add_db_general(ap, hcores):
     ap2.add_argument("--re-dhash", action="store_true", help="force a cache rebuild on startup; enable this once if it gets out of sync (should never be necessary)")
     ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice -- only useful for offloading uploads to a cloud service or something (volflag=noforget)")
     ap2.add_argument("--dbd", metavar="PROFILE", default="wal", help="database durability profile; sets the tradeoff between robustness and speed, see \033[33m--help-dbd\033[0m (volflag=dbd)")
-    ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (volflag=xlink)")
+    ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (probably buggy, not recommended) (volflag=xlink)")
     ap2.add_argument("--hash-mt", metavar="CORES", type=int, default=hcores, help="num cpu cores to use for file hashing; set 0 or 1 for single-core hashing")
     ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="rescan filesystem for changes every \033[33mSEC\033[0m seconds; 0=off (volflag=scan)")
     ap2.add_argument("--db-act", metavar="SEC", type=float, default=10.0, help="defer any scheduled volume reindexing until \033[33mSEC\033[0m seconds after last db write (uploads, renames, ...)")
@@ -1412,7 +1415,7 @@ def add_ui(ap, retry):
     ap2 = ap.add_argument_group('ui options')
     ap2.add_argument("--grid", action="store_true", help="show grid/thumbnails by default (volflag=grid)")
     ap2.add_argument("--gsel", action="store_true", help="select files in grid by ctrl-click (volflag=gsel)")
-    ap2.add_argument("--lang", metavar="LANG", type=u, default="eng", help="language; one of the following: \033[32meng nor\033[0m")
+    ap2.add_argument("--lang", metavar="LANG", type=u, default="eng", help="language; one of the following: \033[32meng nor chi\033[0m")
     ap2.add_argument("--theme", metavar="NUM", type=int, default=0, help="default theme to use (0..7)")
     ap2.add_argument("--themes", metavar="NUM", type=int, default=8, help="number of themes installed")
     ap2.add_argument("--au-vol", metavar="0-100", type=int, default=50, choices=range(0, 101), help="default audio/video volume percent")
@@ -1618,6 +1621,7 @@ def main(argv: Optional[list[str]] = None, rsrc: Optional[str] = None) -> None:
         ("--hdr-au-usr", "--idp-h-usr"),
         ("--idp-h-sep", "--idp-gsep"),
         ("--th-no-crop", "--th-crop=n"),
+        ("--never-symlink", "--hardlink-only"),
     ]
     for dk, nk in deprecated:
         idx = -1
@@ -1642,7 +1646,7 @@ def main(argv: Optional[list[str]] = None, rsrc: Optional[str] = None) -> None:
             argv.extend(["--qr"])
             if ANYWIN or not os.geteuid():
                 # win10 allows symlinks if admin; can be unexpected
-                argv.extend(["-p80,443,3923", "--ign-ebind", "--no-dedup"])
+                argv.extend(["-p80,443,3923", "--ign-ebind"])
     except:
         pass
 

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 14, 0)
-CODENAME = "one step forward"
-BUILD_DT = (2024, 8, 18)
+VERSION = (1, 15, 0)
+CODENAME = "fill the drives"
+BUILD_DT = (2024, 9, 8)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -35,6 +35,7 @@ from .util import (
     odfusion,
     relchk,
     statdir,
+    ub64enc,
     uncyg,
     undot,
     unhumanize,
@@ -344,6 +345,7 @@ class VFS(object):
         self.dbv: Optional[VFS] = None  # closest full/non-jump parent
         self.lim: Optional[Lim] = None  # upload limits; only set for dbv
         self.shr_src: Optional[tuple[VFS, str]] = None  # source vfs+rem of a share
+        self.shr_files: set[str] = set()  # filenames to include from shr_src
         self.aread: dict[str, list[str]] = {}
         self.awrite: dict[str, list[str]] = {}
         self.amove: dict[str, list[str]] = {}
@@ -369,6 +371,7 @@ class VFS(object):
             self.all_vps = []
 
         self.get_dbv = self._get_dbv
+        self.ls = self._ls
 
     def __repr__(self) -> str:
         return "VFS(%s)" % (
@@ -565,7 +568,26 @@ class VFS(object):
         ad, fn = os.path.split(ap)
         return os.path.join(absreal(ad), fn)
 
-    def ls(
+    def _ls_nope(
+        self, *a, **ka
+    ) -> tuple[str, list[tuple[str, os.stat_result]], dict[str, "VFS"]]:
+        raise Pebkac(500, "nope.avi")
+
+    def _ls_shr(
+        self,
+        rem: str,
+        uname: str,
+        scandir: bool,
+        permsets: list[list[bool]],
+        lstat: bool = False,
+    ) -> tuple[str, list[tuple[str, os.stat_result]], dict[str, "VFS"]]:
+        """replaces _ls for certain shares (single-file, or file selection)"""
+        vn, rem = self.shr_src  # type: ignore
+        abspath, real, _ = vn.ls(rem, "\n", scandir, permsets, lstat)
+        real = [x for x in real if os.path.basename(x[0]) in self.shr_files]
+        return abspath, real, {}
+
+    def _ls(
         self,
         rem: str,
         uname: str,
@@ -1508,14 +1530,14 @@ class AuthSrv(object):
             import sqlite3
 
             shv = VFS(self.log_func, "", shr, AXS(), {"d2d": True})
-            par = vfs.all_vols[""]
 
             db_path = self.args.shr_db
             db = sqlite3.connect(db_path)
             cur = db.cursor()
+            cur2 = db.cursor()
             now = time.time()
             for row in cur.execute("select * from sh"):
-                s_k, s_pw, s_vp, s_pr, s_st, s_un, s_t0, s_t1 = row
+                s_k, s_pw, s_vp, s_pr, s_nf, s_un, s_t0, s_t1 = row
                 if s_t1 and s_t1 < now:
                     continue
 
@@ -1524,7 +1546,10 @@ class AuthSrv(object):
                     self.log(t % (s_pr, s_k, s_un, s_vp))
 
                 if s_pw:
-                    sun = "s_%s" % (s_k,)
+                    # gotta reuse the "account" for all shares with this pw,
+                    # so do a light scramble as this appears in the web-ui
+                    zs = ub64enc(hashlib.sha512(s_pw.encode("utf-8")).digest())[4:16]
+                    sun = "s_%s" % (zs.decode("utf-8"),)
                     acct[sun] = s_pw
                 else:
                     sun = "*"
@@ -1539,13 +1564,14 @@ class AuthSrv(object):
                 # don't know the abspath yet + wanna ensure the user
                 # still has the privs they granted, so nullmap it
                 shv.nodes[s_k] = VFS(
-                    self.log_func, "", "%s/%s" % (shr, s_k), s_axs, par.flags.copy()
+                    self.log_func, "", "%s/%s" % (shr, s_k), s_axs, shv.flags.copy()
                 )
 
             vfs.nodes[shr] = vfs.all_vols[shr] = shv
             for vol in shv.nodes.values():
                 vfs.all_vols[vol.vpath] = vol
                 vol.get_dbv = vol._get_share_src
+                vol.ls = vol._ls_nope
 
         zss = set(acct)
         zss.update(self.idp_accs)
@@ -1865,6 +1891,11 @@ class AuthSrv(object):
             if len(zs) == 3:  # fc5 => ffcc55
                 vol.flags["tcolor"] = "".join([x * 2 for x in zs])
 
+            if vol.flags.get("neversymlink"):
+                vol.flags["hardlinkonly"] = True  # was renamed
+            if vol.flags.get("hardlinkonly"):
+                vol.flags["hardlink"] = True
+
             for k1, k2 in IMPLICATIONS:
                 if k1 in vol.flags:
                     vol.flags[k2] = True
@@ -1969,9 +2000,6 @@ class AuthSrv(object):
                     for x in drop:
                         vol.flags.pop(x)
 
-            if vol.flags.get("neversymlink") and not vol.flags.get("hardlink"):
-                vol.flags["copydupes"] = True
-
             # verify tags mentioned by -mt[mp] are used by -mte
             local_mtp = {}
             local_only_mtp = {}
@@ -2050,11 +2078,16 @@ class AuthSrv(object):
 
         have_e2d = False
         have_e2t = False
+        have_dedup = False
+        unsafe_dedup = []
         t = "volumes and permissions:\n"
         for zv in vfs.all_vols.values():
             if not self.warn_anonwrite or verbosity < 5:
                 break
 
+            if enshare and (zv.vpath == shr or zv.vpath.startswith(shrs)):
+                continue
+
             t += '\n\033[36m"/{}"  \033[33m{}\033[0m'.format(zv.vpath, zv.realpath)
             for txt, attr in [
                 ["  read", "uread"],
@@ -2079,6 +2112,11 @@ class AuthSrv(object):
             if "e2t" in zv.flags:
                 have_e2t = True
 
+            if "dedup" in zv.flags:
+                have_dedup = True
+                if "e2d" not in zv.flags and "hardlink" not in zv.flags:
+                    unsafe_dedup.append("/" + zv.vpath)
+
             t += "\n"
 
         if self.warn_anonwrite and verbosity > 4:
@@ -2091,10 +2129,17 @@ class AuthSrv(object):
                     self.log("\n\033[{}\033[0m\n".format(t))
 
                 if not have_e2t:
-                    t = "hint: argument -e2ts enables multimedia indexing (artist/title/...)"
+                    t = "hint: enable multimedia indexing (artist/title/...) with argument -e2ts"
                     self.log(t, 6)
             else:
-                t = "hint: argument -e2dsa enables searching, upload-undo, and better deduplication"
+                t = "hint: enable searching and upload-undo with argument -e2dsa"
+                self.log(t, 6)
+
+            if unsafe_dedup:
+                t = "WARNING: symlink-based deduplication is enabled for some volumes, but without indexing. Please enable -e2dsa and/or --hardlink to avoid problems when moving/renaming files. Affected volumes: %s"
+                self.log(t % (", ".join(unsafe_dedup)), 3)
+            elif not have_dedup:
+                t = "hint: enable upload deduplication with --dedup (but see readme for consequences)"
                 self.log(t, 6)
 
             zv, _ = vfs.get("/", "*", False, False)
@@ -2161,10 +2206,9 @@ class AuthSrv(object):
                 if x != shr and not x.startswith(shrs)
             }
 
-            assert cur  # type: ignore
-            assert shv  # type: ignore
+            assert db and cur and cur2 and shv  # type: ignore
             for row in cur.execute("select * from sh"):
-                s_k, s_pw, s_vp, s_pr, s_st, s_un, s_t0, s_t1 = row
+                s_k, s_pw, s_vp, s_pr, s_nf, s_un, s_t0, s_t1 = row
                 shn = shv.nodes.get(s_k, None)
                 if not shn:
                     continue
@@ -2179,6 +2223,17 @@ class AuthSrv(object):
                     shv.nodes.pop(s_k)
                     continue
 
+                fns = []
+                if s_nf:
+                    q = "select vp from sf where k = ?"
+                    for (s_fn,) in cur2.execute(q, (s_k,)):
+                        fns.append(s_fn)
+
+                    shn.shr_files = set(fns)
+                    shn.ls = shn._ls_shr
+                else:
+                    shn.ls = shn._ls
+
                 shn.shr_src = (s_vfs, s_rem)
                 shn.realpath = s_vfs.canonical(s_rem)
 
@@ -2198,6 +2253,10 @@ class AuthSrv(object):
                     # hide subvolume
                     vn.nodes[zs] = VFS(self.log_func, "", "", AXS(), {})
 
+            cur2.close()
+            cur.close()
+            db.close()
+
     def chpw(self, broker: Optional["BrokerCli"], uname, pw) -> tuple[bool, str]:
         if not self.args.chpw:
             return False, "feature disabled in server config"

copyparty/cfg.py

@@ -12,8 +12,7 @@ def vf_bmap() -> dict[str, str]:
         "dav_auth": "davauth",
         "dav_rt": "davrt",
         "ed": "dots",
-        "never_symlink": "neversymlink",
-        "no_dedup": "copydupes",
+        "hardlink_only": "hardlinkonly",
         "no_dupe": "nodupe",
         "no_forget": "noforget",
         "no_pipe": "nopipe",
@@ -23,6 +22,7 @@ def vf_bmap() -> dict[str, str]:
         "no_athumb": "dathumb",
     }
     for k in (
+        "dedup",
         "dotsrch",
         "e2d",
         "e2ds",
@@ -58,6 +58,7 @@ def vf_vmap() -> dict[str, str]:
         "no_hash": "nohash",
         "no_idx": "noidx",
         "re_maxage": "scan",
+        "safe_dedup": "safededup",
         "th_convt": "convt",
         "th_size": "thsize",
         "th_crop": "crop",
@@ -129,10 +130,11 @@ permdescs = {
 
 flagcats = {
     "uploads, general": {
+        "dedup": "enable symlink-based file deduplication",
+        "hardlink": "enable hardlink-based file deduplication,\nwith fallback on symlinks when that is impossible",
+        "hardlinkonly": "dedup with hardlink only, never symlink;\nmake a full copy if hardlink is impossible",
+        "safededup": "verify on-disk data before using it for dedup",
         "nodupe": "rejects existing files (instead of symlinking them)",
-        "hardlink": "does dedup with hardlinks instead of symlinks",
-        "neversymlink": "disables symlink fallback; full copy instead",
-        "copydupes": "disables dedup, always saves full copies of dupes",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
         "nosub": "forces all uploads into the top folder of the vfs",
@@ -159,7 +161,7 @@ flagcats = {
         "lifetime=3600": "uploads are deleted after 1 hour",
     },
     "database, general": {
-        "e2d": "enable database; makes files searchable + enables upload dedup",
+        "e2d": "enable database; makes files searchable + enables upload-undo",
         "e2ds": "scan writable folders for new files on startup; also sets -e2d",
         "e2dsa": "scans all folders for new files on startup; also sets -e2d",
         "e2t": "enable multimedia indexing; makes it possible to search for tags",
@@ -177,7 +179,7 @@ flagcats = {
         "noforget": "don't forget files when deleted from disk",
         "fat32": "avoid excessive reindexing on android sdcardfs",
         "dbd=[acid|swal|wal|yolo]": "database speed-durability tradeoff",
-        "xlink": "cross-volume dupe detection / linking",
+        "xlink": "cross-volume dupe detection / linking (dangerous)",
         "xdev": "do not descend into other filesystems",
         "xvol": "do not follow symlinks leaving the volume root",
         "dotsrch": "show dotfiles in search results",

copyparty/httpcli.py

@@ -1611,8 +1611,8 @@ class HttpCli(object):
         if "delete" in self.uparam:
             return self.handle_rm([])
 
-        if "unshare" in self.uparam:
-            return self.handle_unshare()
+        if "eshare" in self.uparam:
+            return self.handle_eshare()
 
         if "application/octet-stream" in ctype:
             return self.handle_post_binary()
@@ -3266,7 +3266,8 @@ class HttpCli(object):
                     raise Exception("not found in registry")
                 self.pipes.set(req_path, job)
             except Exception as ex:
-                self.log("will not pipe [%s]; %s" % (ap_data, ex), 6)
+                if getattr(ex, "errno", 0) != errno.ENOENT:
+                    self.log("will not pipe [%s]; %s" % (ap_data, ex), 6)
                 ptop = None
 
         #
@@ -3958,6 +3959,7 @@ class HttpCli(object):
             rvol=rvol,
             wvol=wvol,
             avol=avol,
+            in_shr=self.args.shr and self.vpath.startswith(self.args.shr[1:]),
             vstate=vstate,
             scanning=vs["scanning"],
             hashq=vs["hashq"],
@@ -4006,10 +4008,10 @@ class HttpCli(object):
     def tx_404(self, is_403: bool = False) -> bool:
         rc = 404
         if self.args.vague_403:
-            t = '<h1 id="n">404 not found &nbsp;┐( ´ -`)┌</h1><p id="o">or maybe you don\'t have access -- try logging in or <a href="{}/?h">go home</a></p>'
-            pt = "404 not found  ┐( ´ -`)┌   (or maybe you don't have access -- try logging in)"
+            t = '<h1 id="n">404 not found &nbsp;┐( ´ -`)┌</h1><p id="o">or maybe you don\'t have access -- try a password or <a href="{}/?h">go home</a></p>'
+            pt = "404 not found  ┐( ´ -`)┌   (or maybe you don't have access -- try a password)"
         elif is_403:
-            t = '<h1 id="p">403 forbiddena &nbsp;~┻━┻</h1><p id="q">you\'ll have to log in or <a href="{}/?h">go home</a></p>'
+            t = '<h1 id="p">403 forbiddena &nbsp;~┻━┻</h1><p id="q">use a password or <a href="{}/?h">go home</a></p>'
             pt = "403 forbiddena ~┻━┻   (you'll have to log in)"
             rc = 403
         else:
@@ -4026,7 +4028,8 @@ class HttpCli(object):
 
         t = t.format(self.args.SR)
         qv = quotep(self.vpaths) + self.ourlq()
-        html = self.j2s("splash", this=self, qvpath=qv, msg=t)
+        in_shr = self.args.shr and self.vpath.startswith(self.args.shr[1:])
+        html = self.j2s("splash", this=self, qvpath=qv, in_shr=in_shr, msg=t)
         self.reply(html.encode("utf-8"), status=rc)
         return True
 
@@ -4301,7 +4304,7 @@ class HttpCli(object):
         self.reply(html.encode("utf-8"), status=200)
         return True
 
-    def handle_unshare(self) -> bool:
+    def handle_eshare(self) -> bool:
         idx = self.conn.get_u2idx()
         if not idx or not hasattr(idx, "p_end"):
             if not HAVE_SQLITE3:
@@ -4309,7 +4312,7 @@ class HttpCli(object):
             raise Pebkac(500, "server busy, cannot create share; please retry in a bit")
 
         if self.args.shr_v:
-            self.log("handle_unshare: " + self.req)
+            self.log("handle_eshare: " + self.req)
 
         cur = idx.get_shr()
         if not cur:
@@ -4317,18 +4320,36 @@ class HttpCli(object):
 
         skey = self.vpath.split("/")[-1]
 
-        uns = cur.execute("select un from sh where k = ?", (skey,)).fetchall()
-        un = uns[0][0] if uns and uns[0] else ""
+        rows = cur.execute("select un, t1 from sh where k = ?", (skey,)).fetchall()
+        un = rows[0][0] if rows and rows[0] else ""
 
         if not un:
             raise Pebkac(400, "that sharekey didn't match anything")
 
+        expiry = rows[0][1]
+
         if un != self.uname and self.uname != self.args.shr_adm:
             t = "your username (%r) does not match the sharekey's owner (%r) and you're not admin"
             raise Pebkac(400, t % (self.uname, un))
 
-        cur.execute("delete from sh where k = ?", (skey,))
+        reload = False
+        act = self.uparam["eshare"]
+        if act == "rm":
+            cur.execute("delete from sh where k = ?", (skey,))
+            if skey in self.asrv.vfs.nodes[self.args.shr.strip("/")].nodes:
+                reload = True
+        else:
+            now = time.time()
+            if expiry < now:
+                expiry = now
+                reload = True
+            expiry += int(act) * 60
+            cur.execute("update sh set t1 = ? where k = ?", (expiry, skey))
+
         cur.connection.commit()
+        if reload:
+            self.conn.hsrv.broker.ask("_reload_blocking", False, False).get()
+            self.conn.hsrv.broker.ask("up2k.wake_rescanner").get()
 
         self.redirect(self.args.SRS + "?shares")
         return True
@@ -4344,11 +4365,31 @@ class HttpCli(object):
             self.log("handle_share: " + json.dumps(req, indent=4))
 
         skey = req["k"]
-        vp = req["vp"].strip("/")
+        vps = req["vp"]
+        fns = []
+        if len(vps) == 1:
+            vp = vps[0]
+            if not vp.endswith("/"):
+                vp, zs = vp.rsplit("/", 1)
+                fns = [zs]
+        else:
+            for zs in vps:
+                if zs.endswith("/"):
+                    t = "you cannot select more than one folder, or mix flies and folders in one selection"
+                    raise Pebkac(400, t)
+            vp = vps[0].rsplit("/", 1)[0]
+            for zs in vps:
+                vp2, fn = zs.rsplit("/", 1)
+                fns.append(fn)
+                if vp != vp2:
+                    t = "mismatching base paths in selection:\n  [%s]\n  [%s]"
+                    raise Pebkac(400, t % (vp, vp2))
+
+        vp = vp.strip("/")
         if self.is_vproxied and (vp == self.args.R or vp.startswith(self.args.RS)):
             vp = vp[len(self.args.RS) :]
 
-        m = re.search(r"([^0-9a-zA-Z_\.-]|\.\.|^\.)", skey)
+        m = re.search(r"([^0-9a-zA-Z_-])", skey)
         if m:
             raise Pebkac(400, "sharekey has illegal character [%s]" % (m[1],))
 
@@ -4375,29 +4416,41 @@ class HttpCli(object):
         except:
             raise Pebkac(400, "you dont have all the perms you tried to grant")
 
-        ap = vfs.canonical(rem)
-        st = bos.stat(ap)
-        ist = 2 if stat.S_ISDIR(st.st_mode) else 1
+        ap, reals, _ = vfs.ls(
+            rem, self.uname, not self.args.no_scandir, [[s_rd, s_wr, s_mv, s_del]]
+        )
+        rfns = set([x[0] for x in reals])
+        for fn in fns:
+            if fn not in rfns:
+                raise Pebkac(400, "selected file not found on disk: [%s]" % (fn,))
 
         pw = req.get("pw") or ""
         now = int(time.time())
         sexp = req["exp"]
-        exp = now + int(sexp) * 60 if sexp else 0
+        exp = int(sexp) if sexp else 0
+        exp = now + exp * 60 if exp else 0
         pr = "".join(zc for zc, zb in zip("rwmd", (s_rd, s_wr, s_mv, s_del)) if zb)
 
         q = "insert into sh values (?,?,?,?,?,?,?,?)"
-        cur.execute(q, (skey, pw, vp, pr, ist, self.uname, now, exp))
-        cur.connection.commit()
+        cur.execute(q, (skey, pw, vp, pr, len(fns), self.uname, now, exp))
 
+        q = "insert into sf values (?,?)"
+        for fn in fns:
+            cur.execute(q, (skey, fn))
+
+        cur.connection.commit()
         self.conn.hsrv.broker.ask("_reload_blocking", False, False).get()
         self.conn.hsrv.broker.ask("up2k.wake_rescanner").get()
 
-        surl = "%s://%s%s%s%s" % (
+        fn = quotep(fns[0]) if len(fns) == 1 else ""
+
+        surl = "created share: %s://%s%s%s%s/%s" % (
             "https" if self.is_https else "http",
             self.host,
             self.args.SR,
             self.args.shr,
             skey,
+            fn,
         )
         self.loud_reply(surl, status=201)
         return True

copyparty/svchub.py

@@ -3,7 +3,6 @@ from __future__ import print_function, unicode_literals
 
 import argparse
 import base64
-import calendar
 import errno
 import gzip
 import logging
@@ -16,7 +15,7 @@ import string
 import sys
 import threading
 import time
-from datetime import datetime, timedelta
+from datetime import datetime
 
 # from inspect import currentframe
 # print(currentframe().f_lineno)
@@ -104,8 +103,10 @@ class SvcHub(object):
         self.argv = argv
         self.E: EnvParams = args.E
         self.no_ansi = args.no_ansi
+        self.tz = UTC if args.log_utc else None
         self.logf: Optional[typing.TextIO] = None
         self.logf_base_fn = ""
+        self.is_dut = False  # running in unittest; always False
         self.stop_req = False
         self.stopping = False
         self.stopped = False
@@ -117,7 +118,8 @@ class SvcHub(object):
         self.httpsrv_up = 0
 
         self.log_mutex = threading.Lock()
-        self.next_day = 0
+        self.cday = 0
+        self.cmon = 0
         self.tstack = 0.0
 
         self.iphash = HMaccas(os.path.join(self.E.cfg, "iphash"), 8)
@@ -376,11 +378,18 @@ class SvcHub(object):
 
         import sqlite3
 
-        al.shr = "/%s/" % (al.shr.strip("/"))
+        al.shr = al.shr.strip("/")
+        if "/" in al.shr or not al.shr:
+            t = "config error: --shr must be the name of a virtual toplevel directory to put shares inside"
+            self.log("root", t, 1)
+            raise Exception(t)
+
+        al.shr = "/%s/" % (al.shr,)
 
         create = True
+        modified = False
         db_path = self.args.shr_db
-        self.log("root", "initializing shares-db %s" % (db_path,))
+        self.log("root", "opening shares-db %s" % (db_path,))
         for n in range(2):
             try:
                 db = sqlite3.connect(db_path)
@@ -406,18 +415,43 @@ class SvcHub(object):
                     pass
                 os.unlink(db_path)
 
+        sch1 = [
+            r"create table kv (k text, v int)",
+            r"create table sh (k text, pw text, vp text, pr text, st int, un text, t0 int, t1 int)",
+            # sharekey, password, src, perms, numFiles, owner, created, expires
+        ]
+        sch2 = [
+            r"create table sf (k text, vp text)",
+            r"create index sf_k on sf(k)",
+            r"create index sh_k on sh(k)",
+            r"create index sh_t1 on sh(t1)",
+        ]
+
         assert db  # type: ignore
         assert cur  # type: ignore
         if create:
+            dver = 2
+            modified = True
+            for cmd in sch1 + sch2:
+                cur.execute(cmd)
+            self.log("root", "created new shares-db")
+        else:
+            (dver,) = cur.execute("select v from kv where k = 'sver'").fetchall()[0]
+
+        if dver == 1:
+            modified = True
+            for cmd in sch2:
+                cur.execute(cmd)
+            cur.execute("update sh set st = 0")
+            self.log("root", "shares-db schema upgrade ok")
+
+        if modified:
             for cmd in [
-                # sharekey, password, src, perms, type, owner, created, expires
-                r"create table sh (k text, pw text, vp text, pr text, st int, un text, t0 int, t1 int)",
-                r"create table kv (k text, v int)",
-                r"insert into kv values ('sver', {})".format(1),
+                r"delete from kv where k = 'sver'",
+                r"insert into kv values ('sver', %d)" % (2,),
             ]:
                 cur.execute(cmd)
             db.commit()
-            self.log("root", "created new shares-db")
 
         cur.close()
         db.close()
@@ -758,7 +792,7 @@ class SvcHub(object):
             self.args.nc = min(self.args.nc, soft // 2)
 
     def _logname(self) -> str:
-        dt = datetime.now(UTC)
+        dt = datetime.now(self.tz)
         fn = str(self.args.lo)
         for fs in "YmdHMS":
             fs = "%" + fs
@@ -1031,12 +1065,12 @@ class SvcHub(object):
             return
 
         with self.log_mutex:
-            zd = datetime.now(UTC)
+            dt = datetime.now(self.tz)
             ts = self.log_dfmt % (
-                zd.year,
-                zd.month * 100 + zd.day,
-                (zd.hour * 100 + zd.minute) * 100 + zd.second,
-                zd.microsecond // self.log_div,
+                dt.year,
+                dt.month * 100 + dt.day,
+                (dt.hour * 100 + dt.minute) * 100 + dt.second,
+                dt.microsecond // self.log_div,
             )
 
             if c and not self.args.no_ansi:
@@ -1057,41 +1091,26 @@ class SvcHub(object):
             if not self.args.no_logflush:
                 self.logf.flush()
 
-            now = time.time()
-            if int(now) >= self.next_day:
-                self._set_next_day()
+            if dt.day != self.cday or dt.month != self.cmon:
+                self._set_next_day(dt)
 
-    def _set_next_day(self) -> None:
-        if self.next_day and self.logf and self.logf_base_fn != self._logname():
+    def _set_next_day(self, dt: datetime) -> None:
+        if self.cday and self.logf and self.logf_base_fn != self._logname():
             self.logf.close()
             self._setup_logfile("")
 
-        dt = datetime.now(UTC)
-
-        # unix timestamp of next 00:00:00 (leap-seconds safe)
-        day_now = dt.day
-        while dt.day == day_now:
-            dt += timedelta(hours=12)
-
-        dt = dt.replace(hour=0, minute=0, second=0)
-        try:
-            tt = dt.utctimetuple()
-        except:
-            # still makes me hella uncomfortable
-            tt = dt.timetuple()
-
-        self.next_day = calendar.timegm(tt)
+        self.cday = dt.day
+        self.cmon = dt.month
 
     def _log_enabled(self, src: str, msg: str, c: Union[int, str] = 0) -> None:
         """handles logging from all components"""
         with self.log_mutex:
-            now = time.time()
-            if int(now) >= self.next_day:
-                dt = datetime.fromtimestamp(now, UTC)
+            dt = datetime.now(self.tz)
+            if dt.day != self.cday or dt.month != self.cmon:
                 zs = "{}\n" if self.no_ansi else "\033[36m{}\033[0m\n"
                 zs = zs.format(dt.strftime("%Y-%m-%d"))
                 print(zs, end="")
-                self._set_next_day()
+                self._set_next_day(dt)
                 if self.logf:
                     self.logf.write(zs)
 
@@ -1110,12 +1129,11 @@ class SvcHub(object):
                 else:
                     msg = "%s%s\033[0m" % (c, msg)
 
-            zd = datetime.fromtimestamp(now, UTC)
             ts = self.log_efmt % (
-                zd.hour,
-                zd.minute,
-                zd.second,
-                zd.microsecond // self.log_div,
+                dt.hour,
+                dt.minute,
+                dt.second,
+                dt.microsecond // self.log_div,
             )
             msg = fmt % (ts, src, msg)
             try:

copyparty/tftpd.py

@@ -403,7 +403,7 @@ class Tftpd(object):
             bos.stat(ap)
             return True
         except:
-            return False
+            return vpath == "/"
 
     def _p_isdir(self, vpath: str) -> bool:
         try:
@@ -411,7 +411,7 @@ class Tftpd(object):
             ret = stat.S_ISDIR(st.st_mode)
             return ret
         except:
-            return False
+            return vpath == "/"
 
     def _hook(self, *a: Any, **ka: Any) -> None:
         src = inspect.currentframe().f_back.f_code.co_name

copyparty/up2k.py

@@ -236,6 +236,9 @@ class Up2k(object):
         if not self.pp and self.args.exit == "idx":
             return self.hub.sigterm()
 
+        if self.hub.is_dut:
+            return
+
         Daemon(self._snapshot, "up2k-snapshot")
         if have_e2d:
             Daemon(self._hasher, "up2k-hasher")
@@ -432,7 +435,7 @@ class Up2k(object):
     def _sched_rescan(self) -> None:
         volage = {}
         cooldown = timeout = time.time() + 3.0
-        while True:
+        while not self.stop:
             now = time.time()
             timeout = max(timeout, cooldown)
             wait = timeout - time.time()
@@ -440,6 +443,9 @@ class Up2k(object):
             with self.rescan_cond:
                 self.rescan_cond.wait(wait)
 
+            if self.stop:
+                return
+
             now = time.time()
             if now < cooldown:
                 # self.log("SR: cd - now = {:.2f}".format(cooldown - now), 5)
@@ -460,8 +466,10 @@ class Up2k(object):
                 # important; not deferred by db_act
                 timeout = self._check_lifetimes()
                 try:
-                    timeout = min(self._check_shares(), timeout)
+                    if self.args.shr:
+                        timeout = min(self._check_shares(), timeout)
                 except Exception as ex:
+                    timeout = min(timeout, now + 60)
                     t = "could not check for expiring shares: %r"
                     self.log(t % (ex,), 1)
 
@@ -571,27 +579,53 @@ class Up2k(object):
 
         now = time.time()
         timeout = now + 9001
+        maxage = self.args.shr_rt * 60
+        low = now - maxage
+
+        vn = self.asrv.vfs.nodes.get(self.args.shr.strip("/"))
+        active = vn and vn.nodes
 
         db = sqlite3.connect(self.args.shr_db, timeout=2)
         cur = db.cursor()
 
         q = "select k from sh where t1 and t1 <= ?"
-        rm = [x[0] for x in cur.execute(q, (now,))]
+        rm = [x[0] for x in cur.execute(q, (now,))] if active else []
+        if rm:
+            assert vn and vn.nodes  # type: ignore
+            # self.log("chk_shr: %d" % (len(rm),))
+            zss = set(rm)
+            rm = [zs for zs in vn.nodes if zs in zss]
+        reload = bool(rm)
+        if reload:
+            self.log("disabling expired shares %s" % (rm,))
+
+        rm = [x[0] for x in cur.execute(q, (low,))]
         if rm:
             self.log("forgetting expired shares %s" % (rm,))
-            q = "delete from sh where k=?"
-            cur.executemany(q, [(x,) for x in rm])
+            cur.executemany("delete from sh where k=?", [(x,) for x in rm])
+            cur.executemany("delete from sf where k=?", [(x,) for x in rm])
             db.commit()
+
+        if reload:
             Daemon(self.hub._reload_blocking, "sharedrop", (False, False))
 
-        q = "select min(t1) from sh where t1 > 1"
-        (earliest,) = cur.execute(q).fetchone()
+        q = "select min(t1) from sh where t1 > ?"
+        (earliest,) = cur.execute(q, (1,)).fetchone()
         if earliest:
-            timeout = earliest - now
+            # deadline for revoking regular access
+            timeout = min(timeout, earliest + maxage)
+
+            (earliest,) = cur.execute(q, (now - 2,)).fetchone()
+            if earliest:
+                # deadline for revival; drop entirely
+                timeout = min(timeout, earliest)
 
         cur.close()
         db.close()
 
+        if self.args.shr_v:
+            self.log("next shr_chk = %d (%d)" % (timeout, timeout - time.time()))
+
         return timeout
 
     def _check_xiu(self) -> float:
@@ -715,6 +749,8 @@ class Up2k(object):
                     continue
 
                 self.pp = ProgressPrinter(self.log, self.args)
+                if not self.hub.is_dut:
+                    self.pp.start()
 
             break
 
@@ -1404,7 +1440,7 @@ class Up2k(object):
                 if dts == lmod and dsz == sz and (nohash or dw[0] != "#" or not sz):
                     continue
 
-                t = "reindex [{}] => [{}] ({}/{}) ({}/{})".format(
+                t = "reindex [{}] => [{}] mtime({}/{}) size({}/{})".format(
                     top, rp, dts, lmod, dsz, sz
                 )
                 self.log(t)
@@ -1426,7 +1462,7 @@ class Up2k(object):
                     self.log("file: {}".format(abspath))
 
                 try:
-                    hashes = self._hashlist_from_file(
+                    hashes, _ = self._hashlist_from_file(
                         abspath, "a{}, ".format(self.pp.n)
                     )
                 except Exception as ex:
@@ -1620,6 +1656,7 @@ class Up2k(object):
             qex = " where " + qex
 
         rewark: list[tuple[str, str, str, int, int]] = []
+        f404: list[tuple[str, str, str]] = []
 
         with self.mutex:
             b_left = 0
@@ -1636,7 +1673,8 @@ class Up2k(object):
                 if self.stop:
                     return -1
 
-                w, drd, dfn = zb[:-1].decode("utf-8").split("\x00")
+                zs = zb[:-1].decode("utf-8").replace("\x00\x02", "\n")
+                w, drd, dfn = zs.split("\x00\x01")
                 with self.mutex:
                     q = "select mt, sz from up where rd=? and fn=? and +w=?"
                     try:
@@ -1662,9 +1700,14 @@ class Up2k(object):
                 pf = "v{}, {:.0f}+".format(n_left, b_left / 1024 / 1024)
                 self.pp.msg = pf + abspath
 
-                # throws on broken symlinks (always did)
-                stl = bos.lstat(abspath)
-                st = bos.stat(abspath) if stat.S_ISLNK(stl.st_mode) else stl
+                try:
+                    stl = bos.lstat(abspath)
+                    st = bos.stat(abspath) if stat.S_ISLNK(stl.st_mode) else stl
+                except Exception as ex:
+                    self.log("missing file: %s" % (abspath,), 3)
+                    f404.append((drd, dfn, w))
+                    continue
+
                 mt2 = int(stl.st_mtime)
                 sz2 = st.st_size
 
@@ -1675,7 +1718,7 @@ class Up2k(object):
                         self.log("file: {}".format(abspath))
 
                     try:
-                        hashes = self._hashlist_from_file(abspath, pf)
+                        hashes, _ = self._hashlist_from_file(abspath, pf)
                     except Exception as ex:
                         self.log("hash: {} @ [{}]".format(repr(ex), abspath))
                         continue
@@ -1701,12 +1744,15 @@ class Up2k(object):
                 t = t.format(abspath, w, sz, mt, w2, sz2, mt2)
                 self.log(t, 1)
 
-        if e2vp and rewark:
+        if e2vp and (rewark or f404):
             self.hub.retcode = 1
             Daemon(self.hub.sigterm)
-            raise Exception("{} files have incorrect hashes".format(len(rewark)))
+            t = "in volume /%s:  %s files missing, %s files have incorrect hashes"
+            t = t % (vol.vpath, len(f404), len(rewark))
+            self.log(t, 1)
+            raise Exception(t)
 
-        if not e2vu or not rewark:
+        if not e2vu or (not rewark and not f404):
             return 0
 
         with self.mutex:
@@ -1714,9 +1760,13 @@ class Up2k(object):
                 q = "update up set w = ?, sz = ?, mt = ? where rd = ? and fn = ? limit 1"
                 cur.execute(q, (w, sz, int(mt), rd, fn))
 
+            for _, _, w in f404:
+                q = "delete from up where w = ? limit 1"
+                cur.execute(q, (w,))
+
             cur.connection.commit()
 
-        return len(rewark)
+        return len(rewark) + len(f404)
 
     def _build_tags_index(self, vol: VFS) -> tuple[int, int, bool]:
         ptop = vol.realpath
@@ -1931,7 +1981,8 @@ class Up2k(object):
                     if c2.execute(q, (row[0][:16],)).fetchone():
                         continue
 
-                gf.write(("%s\n" % ("\x00".join(row),)).encode("utf-8"))
+                zs = "\x00\x01".join(row).replace("\n", "\x00\x02")
+                gf.write((zs + "\n").encode("utf-8"))
                 n += 1
 
         c2.close()
@@ -2319,7 +2370,9 @@ class Up2k(object):
 
     def _open_db_wd(self, db_path: str) -> "sqlite3.Cursor":
         ok: list[int] = []
-        Daemon(self._open_db_timeout, "opendb_watchdog", [db_path, ok])
+        if not self.hub.is_dut:
+            Daemon(self._open_db_timeout, "opendb_watchdog", [db_path, ok])
+
         try:
             return self._open_db(db_path)
         finally:
@@ -2628,10 +2681,13 @@ class Up2k(object):
             jcur = self.cur.get(ptop)
             reg = self.registry[ptop]
             vfs = self.asrv.vfs.all_vols[cj["vtop"]]
-            n4g = vfs.flags.get("noforget")
+            n4g = bool(vfs.flags.get("noforget"))
             rand = vfs.flags.get("rand") or cj.get("rand")
             lost: list[tuple["sqlite3.Cursor", str, str]] = []
 
+            safe_dedup = vfs.flags.get("safededup") or 50
+            data_ok = safe_dedup < 10 or n4g
+
             vols = [(ptop, jcur)] if jcur else []
             if vfs.flags.get("xlink"):
                 vols += [(k, v) for k, v in self.cur.items() if k != ptop]
@@ -2639,7 +2695,7 @@ class Up2k(object):
                 # force upload time rather than last-modified
                 cj["lmod"] = int(time.time())
 
-            alts: list[tuple[int, int, dict[str, Any]]] = []
+            alts: list[tuple[int, int, dict[str, Any], "sqlite3.Cursor", str, str]] = []
             for ptop, cur in vols:
                 allv = self.asrv.vfs.all_vols
                 cvfs = next((v for v in allv.values() if v.realpath == ptop), vfs)
@@ -2663,7 +2719,14 @@ class Up2k(object):
                         if stat.S_ISLNK(st.st_mode):
                             # broken symlink
                             raise Exception()
-                    except:
+                        if st.st_size != dsize:
+                            t = "candidate ignored (db/fs desync): {}, size fs={} db={}, mtime fs={} db={}, file: {}"
+                            t = t.format(
+                                wark, st.st_size, dsize, st.st_mtime, dtime, dp_abs
+                            )
+                            self.log(t)
+                            raise Exception()
+                    except Exception as ex:
                         if n4g:
                             st = os.stat_result((0, -1, -1, 0, 0, 0, 0, 0, 0, 0))
                         else:
@@ -2690,18 +2753,42 @@ class Up2k(object):
                         if k in cj:
                             j[k] = cj[k]
 
+                    # offset of 1st diff in vpaths
+                    zig = (
+                        n + 1
+                        for n, (c1, c2) in enumerate(
+                            zip(dp_dir + "\r", cj["prel"] + "\n")
+                        )
+                        if c1 != c2
+                    )
                     score = (
-                        (3 if st.st_dev == dev else 0)
-                        + (2 if dp_dir == cj["prel"] else 0)
+                        (6969 if st.st_dev == dev else 0)
+                        + (3210 if dp_dir == cj["prel"] else next(zig))
                         + (1 if dp_fn == cj["name"] else 0)
                     )
-                    alts.append((score, -len(alts), j))
+                    alts.append((score, -len(alts), j, cur, dp_dir, dp_fn))
 
-            if alts:
-                best = sorted(alts, reverse=True)[0]
-                job = best[2]
-            else:
-                job = None
+            job = None
+            inc_ap = djoin(cj["ptop"], cj["prel"], cj["name"])
+            for dupe in sorted(alts, reverse=True):
+                rj = dupe[2]
+                orig_ap = djoin(rj["ptop"], rj["prel"], rj["name"])
+                if data_ok or inc_ap == orig_ap:
+                    data_ok = True
+                    job = rj
+                    break
+                else:
+                    self.log("asserting contents of %s" % (orig_ap,))
+                    dhashes, st = self._hashlist_from_file(orig_ap)
+                    dwark = up2k_wark_from_hashlist(self.salt, st.st_size, dhashes)
+                    if wark != dwark:
+                        t = "will not dedup (fs index desync): fs=%s, db=%s, file: %s"
+                        self.log(t % (dwark, wark, orig_ap))
+                        lost.append(dupe[3:])
+                        continue
+                    data_ok = True
+                    job = rj
+                    break
 
             if job and wark in reg:
                 # self.log("pop " + wark + "  " + job["name"] + " handle_json db", 4)
@@ -2710,7 +2797,7 @@ class Up2k(object):
             if lost:
                 c2 = None
                 for cur, dp_dir, dp_fn in lost:
-                    t = "forgetting deleted file: /{}"
+                    t = "forgetting desynced db entry: /{}"
                     self.log(t.format(vjoin(vjoin(vfs.vpath, dp_dir), dp_fn)))
                     self.db_rm(cur, dp_dir, dp_fn, cj["size"])
                     if c2 and c2 != cur:
@@ -2725,13 +2812,16 @@ class Up2k(object):
             ptop = None  # use cj or job as appropriate
 
             if not job and wark in reg:
-                # ensure the files haven't been deleted manually
+                # ensure the files haven't been edited or deleted
+                path = ""
+                st = None
                 rj = reg[wark]
                 names = [rj[x] for x in ["name", "tnam"] if x in rj]
                 for fn in names:
                     path = djoin(rj["ptop"], rj["prel"], fn)
                     try:
-                        if bos.path.getsize(path) > 0 or not rj["need"]:
+                        st = bos.stat(path)
+                        if st.st_size > 0 or not rj["need"]:
                             # upload completed or both present
                             break
                     except:
@@ -2742,6 +2832,29 @@ class Up2k(object):
                             del reg[wark]
                         break
 
+                inc_ap = djoin(cj["ptop"], cj["prel"], cj["name"])
+                orig_ap = djoin(rj["ptop"], rj["prel"], rj["name"])
+
+                if self.args.nw or n4g or not st:
+                    pass
+
+                elif st.st_size != rj["size"]:
+                    t = "will not dedup (fs index desync): {}, size fs={} db={}, mtime fs={} db={}, file: {}"
+                    t = t.format(
+                        wark, st.st_size, rj["size"], st.st_mtime, rj["lmod"], path
+                    )
+                    self.log(t)
+                    del reg[wark]
+
+                elif inc_ap != orig_ap and not data_ok:
+                    self.log("asserting contents of %s" % (orig_ap,))
+                    dhashes, _ = self._hashlist_from_file(orig_ap)
+                    dwark = up2k_wark_from_hashlist(self.salt, st.st_size, dhashes)
+                    if wark != dwark:
+                        t = "will not dedup (fs index desync): fs=%s, idx=%s, file: %s"
+                        self.log(t % (dwark, wark, orig_ap))
+                        del reg[wark]
+
             if job or wark in reg:
                 job = job or reg[wark]
                 if (
@@ -2849,6 +2962,7 @@ class Up2k(object):
                                         return self._handle_json(job, depth + 1)
 
                         job["name"] = self._untaken(pdir, job, now)
+                        dst = djoin(job["ptop"], job["prel"], job["name"])
 
                         if not self.args.nw:
                             dvf: dict[str, Any] = vfs.flags
@@ -3012,17 +3126,25 @@ class Up2k(object):
         verbose: bool = True,
         rm: bool = False,
         lmod: float = 0,
+        fsrc: Optional[str] = None,
     ) -> None:
+        if src == dst or (fsrc and fsrc == dst):
+            t = "symlinking a file to itself?? orig(%s) fsrc(%s) link(%s)"
+            raise Exception(t % (src, fsrc, dst))
+
         if verbose:
-            self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+            t = "linking dupe:\n  point-to: {0}\n  link-loc: {1}"
+            if fsrc:
+                t += "\n  data-src: {2}"
+            self.log(t.format(src, dst, fsrc))
 
         if self.args.nw:
             return
 
         linked = False
         try:
-            if "copydupes" in flags:
-                raise Exception("disabled in config")
+            if not flags.get("dedup"):
+                raise Exception("dedup is disabled in config")
 
             lsrc = src
             ldst = dst
@@ -3059,7 +3181,7 @@ class Up2k(object):
                     linked = True
             except Exception as ex:
                 self.log("cannot hardlink: " + repr(ex))
-                if "neversymlink" in flags:
+                if "hardlinkonly" in flags:
                     raise Exception("symlink-fallback disabled in cfg")
 
             if not linked:
@@ -3078,7 +3200,15 @@ class Up2k(object):
                 linked = True
         except Exception as ex:
             self.log("cannot link; creating copy: " + repr(ex))
-            shutil.copy2(fsenc(src), fsenc(dst))
+            if bos.path.isfile(src):
+                csrc = src
+            elif fsrc and bos.path.isfile(fsrc):
+                csrc = fsrc
+            else:
+                t = "BUG: no valid sources to link from! orig(%s) fsrc(%s) link(%s)"
+                self.log(t, 1)
+                raise Exception(t % (src, fsrc, dst))
+            shutil.copy2(fsenc(csrc), fsenc(dst))
 
         if lmod and (not linked or SYMTIME):
             times = (int(time.time()), int(lmod))
@@ -3640,8 +3770,11 @@ class Up2k(object):
                     cur = None
                     try:
                         ptop = dbv.realpath
+                        xlink = bool(dbv.flags.get("xlink"))
                         cur, wark, _, _, _, _ = self._find_from_vpath(ptop, volpath)
-                        self._forget_file(ptop, volpath, cur, wark, True, st.st_size)
+                        self._forget_file(
+                            ptop, volpath, cur, wark, True, st.st_size, xlink
+                        )
                     finally:
                         if cur:
                             cur.connection.commit()
@@ -3865,13 +3998,15 @@ class Up2k(object):
             if c2 and c2 != c1:
                 self._copy_tags(c1, c2, w)
 
+            xlink = bool(svn.flags.get("xlink"))
+
             with self.reg_mutex:
                 has_dupes = self._forget_file(
-                    svn.realpath, srem, c1, w, is_xvol, fsize_ or fsize
+                    svn.realpath, srem, c1, w, is_xvol, fsize_ or fsize, xlink
                 )
 
             if not is_xvol:
-                has_dupes = self._relink(w, svn.realpath, srem, dabs)
+                has_dupes = self._relink(w, svn.realpath, srem, dabs, c1, xlink)
 
             curs.add(c1)
 
@@ -4014,6 +4149,7 @@ class Up2k(object):
         wark: Optional[str],
         drop_tags: bool,
         sz: int,
+        xlink: bool,
     ) -> bool:
         """
         mutex(main,reg) me
@@ -4025,7 +4161,7 @@ class Up2k(object):
         if wark and cur:
             self.log("found {} in db".format(wark))
             if drop_tags:
-                if self._relink(wark, ptop, vrem, ""):
+                if self._relink(wark, ptop, vrem, "", cur, xlink):
                     has_dupes = True
                     drop_tags = False
 
@@ -4057,7 +4193,15 @@ class Up2k(object):
 
         return has_dupes
 
-    def _relink(self, wark: str, sptop: str, srem: str, dabs: str) -> int:
+    def _relink(
+        self,
+        wark: str,
+        sptop: str,
+        srem: str,
+        dabs: str,
+        vcur: Optional["sqlite3.Cursor"],
+        xlink: bool,
+    ) -> int:
         """
         update symlinks from file at svn/srem to dabs (rename),
         or to first remaining full if no dabs (delete)
@@ -4073,6 +4217,8 @@ class Up2k(object):
             argv = (wark[:16], wark)
 
         for ptop, cur in self.cur.items():
+            if not xlink and cur and cur != vcur:
+                continue
             for rd, fn in cur.execute(q, argv):
                 if rd.startswith("//") or fn.startswith("//"):
                     rd, fn = s3dec(rd, fn)
@@ -4159,7 +4305,13 @@ class Up2k(object):
             except:
                 pass
 
-            self._symlink(dabs, alink, flags, False, lmod=lmod or 0)
+            # this creates a link pointing from dabs to alink; alink may
+            # not exist yet, which becomes problematic if the symlinking
+            # fails and it has to fall back on hardlinking/copying files
+            # (for example a volume with symlinked dupes but no --dedup);
+            # fsrc=sabs is then a source that currently resolves to copy
+
+            self._symlink(dabs, alink, flags, False, lmod=lmod or 0, fsrc=sabs)
 
         return len(full) + len(links)
 
@@ -4188,8 +4340,11 @@ class Up2k(object):
 
         return wark
 
-    def _hashlist_from_file(self, path: str, prefix: str = "") -> list[str]:
-        fsz = bos.path.getsize(path)
+    def _hashlist_from_file(
+        self, path: str, prefix: str = ""
+    ) -> tuple[list[str], os.stat_result]:
+        st = bos.stat(path)
+        fsz = st.st_size
         csz = up2k_chunksize(fsz)
         ret = []
         suffix = " MB, {}".format(path)
@@ -4202,7 +4357,7 @@ class Up2k(object):
             while fsz > 0:
                 # same as `hash_at` except for `imutex` / bufsz
                 if self.stop:
-                    return []
+                    return [], st
 
                 if self.pp:
                     mb = fsz // (1024 * 1024)
@@ -4223,7 +4378,7 @@ class Up2k(object):
                 digest = base64.urlsafe_b64encode(digest)
                 ret.append(digest.decode("utf-8"))
 
-        return ret
+        return ret, st
 
     def _new_upload(self, job: dict[str, Any], vfs: VFS, depth: int) -> dict[str, str]:
         pdir = djoin(job["ptop"], job["prel"])
@@ -4524,7 +4679,7 @@ class Up2k(object):
                 self.salt, inf.st_size, int(inf.st_mtime), rd, fn
             )
         else:
-            hashes = self._hashlist_from_file(abspath)
+            hashes, _ = self._hashlist_from_file(abspath)
             if not hashes:
                 return False
 

copyparty/util.py

@@ -260,6 +260,8 @@ IMPLICATIONS = [
     ["e2vu", "e2v"],
     ["e2vp", "e2v"],
     ["e2v", "e2d"],
+    ["hardlink_only", "hardlink"],
+    ["hardlink", "dedup"],
     ["tftpvv", "tftpv"],
     ["smbw", "smb"],
     ["smb1", "smb"],
@@ -915,7 +917,6 @@ class ProgressPrinter(threading.Thread):
         self.msg = ""
         self.end = False
         self.n = -1
-        self.start()
 
     def run(self) -> None:
         sigblock()

copyparty/web/browser.css

@@ -64,7 +64,7 @@
 	--u2-tab-bg: linear-gradient(to bottom, var(--bg), var(--bg-u1));
 	--u2-tab-b1: rgba(128,128,128,0.8);
 	--u2-tab-1-fg: #fd7;
-	--u2-tab-1-bg: linear-gradient(to bottom, var(#353), var(--bg) 80%);
+	--u2-tab-1-bg: linear-gradient(to bottom, #353, var(--bg) 80%);
 	--u2-tab-1-b1: #7c5;
 	--u2-tab-1-b2: #583;
 	--u2-tab-1-sh: #280;
@@ -345,6 +345,7 @@ html.cz {
 	--srv-3: #fff;
 
 	--u2-tab-b1: var(--bg-d3);
+	--u2-tab-1-bg: a;
 }
 html.cy {
 	--fg: #fff;
@@ -374,12 +375,17 @@ html.cy {
 	--btn-bs: 0 .25em 0 #f00;
 	--chk-fg: #fd0;
 
+	--txt-bg: #000;
 	--srv-1: #f00;
 	--srv-3: #fff;
 	--op-aa-bg: #fff;
 
 	--u2-b1-bg: #f00;
 	--u2-b2-bg: #f00;
+
+	--g-sel-fg: #fff;
+	--g-sel-bg: #aaa;
+	--g-fsel-bg: #aaa;
 }
 html.dz {
 	--fg: #4d4;
@@ -436,7 +442,7 @@ html.dz {
 	--u2-tab-bg: linear-gradient(to bottom, var(--bg), var(--bg-u1));
 	--u2-tab-b1: var(--fg-weak);
 	--u2-tab-1-fg: #fff;
-	--u2-tab-1-bg: linear-gradient(to bottom, var(#353), var(--bg) 80%);
+	--u2-tab-1-bg: linear-gradient(to bottom, #151, var(--bg) 80%);
 	--u2-tab-1-b1: #7c5;
 	--u2-tab-1-b2: #583;
 	--u2-tab-1-sh: #280;
@@ -626,6 +632,7 @@ pre, code, tt, #doc, #doc>code {
 	overflow: hidden;
 	width: 0;
 	height: 0;
+	color: var(--bg);
 }
 html .ayjump:focus {
 	z-index: 80386;
@@ -3111,18 +3118,30 @@ html.by #u2cards a.act {
 
 
 
-html.cy #wrap {
-	color: #000;
-}
 html.cy .mdo a {
 	background: #f00;
 }
+html.cy #wrap,
+html.cy #acc_info a,
 html.cy #op_up2k,
 html.cy #files,
 html.cy #files a,
 html.cy #files tbody div a:last-child {
 	color: #000;
 }
+html.cy #u2tab a,
+html.cy #u2cards a {
+	color: #f00;
+}
+html.cy #unpost a {
+	color: #ff0;
+}
+html.cy #barbuf {
+	filter: hue-rotate(267deg) brightness(0.8) contrast(4);
+}
+html.cy #pvol {
+	filter: hue-rotate(4deg) contrast(2.2);
+}
 
 
 

copyparty/web/shares.css

@@ -58,6 +58,9 @@ td, th {
 	text-align: left;
 	white-space: nowrap;
 }
+td+td+td+td+td+td+td+td {
+	font-family: var(--font-mono), monospace, monospace;
+}
 
 
 

copyparty/web/shares.html

@@ -15,25 +15,26 @@
 <body>
 	<div id="wrap">
 		<a id="a" href="{{ r }}/?shares" class="af">refresh</a>
-		<a id="a" href="{{ r }}/?h" class="af">controlpanel</a>
+		<a id="a" href="{{ r }}/?h" class="af">control-panel</a>
 
         <span>axs = perms (read,write,move,delet)</span>
-        <span>st 1=file 2=dir</span>
+        <span>nf = numFiles (0=dir)</span>
         <span>min/hrs = time left</span>
 
-        <table><tr>
+        <table id="tab"><thead><tr>
             <th>delete</th>
             <th>sharekey</th>
             <th>pw</th>
             <th>source</th>
             <th>axs</th>
-            <th>st</th>
+            <th>nf</th>
             <th>user</th>
             <th>created</th>
             <th>expires</th>
             <th>min</th>
             <th>hrs</th>
-        </tr>
+            <th>add time</th>
+        </tr></thead><tbody>
         {% for k, pw, vp, pr, st, un, t0, t1 in rows %}
         <tr>
             <td><a href="#" k="{{ k }}">delete</a></td>
@@ -45,11 +46,12 @@
             <td>{{ un|e }}</td>
             <td>{{ t0 }}</td>
             <td>{{ t1 }}</td>
-            <td>{{ (t1 - now) // 60 if t1 else "never" }}</td>
-            <td>{{ (t1 - now) // 3600 if t1 else "never" }}</td>
+            <td>{{ "inf" if not t1 else "dead" if t1 < now else ((t1 - now) / 60)   | round(1) }}</td>
+            <td>{{ "inf" if not t1 else "dead" if t1 < now else ((t1 - now) / 3600) | round(1) }}</td>
+            <td></td>
         </tr>
         {% endfor %}
-        </table>
+        </tbody></table>
         {% if not rows %}
         (you don't have any active shares btw)
         {% endif %}

copyparty/web/shares.js

@@ -3,9 +3,19 @@ for (var a = 0; a < t.length; a++)
     t[a].onclick = rm;
 
 function rm() {
-    var u = SR + shr + uricom_enc(this.getAttribute('k')) + '?unshare',
+    var u = SR + shr + uricom_enc(this.getAttribute('k')) + '?eshare=rm',
         xhr = new XHR();
-    
+
+    xhr.open('POST', u, true);
+    xhr.onload = xhr.onerror = cb;
+    xhr.send();
+}
+
+function bump() {
+    var k = this.closest('tr').getElementsByTagName('a')[0].getAttribute('k'),
+        u = SR + shr + uricom_enc(k) + '?eshare=' + this.value,
+        xhr = new XHR();
+
     xhr.open('POST', u, true);
     xhr.onload = xhr.onerror = cb;
     xhr.send();
@@ -14,6 +24,33 @@ function rm() {
 function cb() {
     if (this.status !== 200)
         return modal.alert('<h6>server error</h6>' + esc(unpre(this.responseText)));
-    
+
     document.location = '?shares';
 }
+
+(function() {
+    var tab = ebi('tab').tBodies[0],
+        tr = Array.prototype.slice.call(tab.rows, 0);
+
+    var buf = [];
+    for (var a = 0; a < tr.length; a++)
+        for (var b = 7; b < 9; b++)
+            buf.push(parseInt(tr[a].cells[b].innerHTML));
+
+    var ibuf = 0;
+    for (var a = 0; a < tr.length; a++)
+        for (var b = 7; b < 9; b++) {
+            var v = buf[ibuf++];
+            tr[a].cells[b].innerHTML =
+                v ? unix2iso(v).replace(' ', ',&nbsp;') : 'never';
+        }
+
+    for (var a = 0; a < tr.length; a++)
+        tr[a].cells[11].innerHTML =
+            '<button value="1">1min</button> ' +
+            '<button value="60">1h</button>';
+    
+    var btns = QSA('td button'), aa = btns.length;
+    for (var a = 0; a < aa; a++)
+        btns[a].onclick = bump;
+})();

copyparty/web/splash.html

@@ -14,6 +14,7 @@
 
 <body>
 	<div id="wrap">
+		{%- if not in_shr %}
 		<a id="a" href="{{ r }}/?h" class="af">refresh</a>
 		<a id="v" href="{{ r }}/?hc" class="af">connect</a>
 
@@ -23,6 +24,7 @@
 			<a id="c" href="{{ r }}/?pw=x" class="logout">logout</a>
 			<p><span id="m">welcome back,</span> <strong>{{ this.uname|e }}</strong></p>
 		{%- endif %}
+		{%- endif %}
 
 		{%- if msg %}
 		<div id="msg">
@@ -76,6 +78,37 @@
 		</ul>
 		{%- endif %}
 
+		{%- if in_shr %}
+		<h1 id="z">unlock this share:</h1>
+		<div>
+			<form id="lf" method="post" enctype="multipart/form-data" action="{{ r }}/{{ qvpath }}">
+				<input type="hidden" id="la" name="act" value="login" />
+				<input type="password" id="lp" name="cppwd" placeholder=" password" />
+				<input type="hidden" name="uhash" id="uhash" value="x" />
+				<input type="submit" id="ls" value="Unlock" />
+				{% if ahttps %}
+				<a id="w" href="{{ ahttps }}">switch to https</a>
+				{% endif %}
+			</form>
+		</div>
+		{%- else %}
+		<h1 id="l">login for more:</h1>
+		<div>
+			<form id="lf" method="post" enctype="multipart/form-data" action="{{ r }}/{{ qvpath }}">
+				<input type="hidden" id="la" name="act" value="login" />
+				<input type="password" id="lp" name="cppwd" placeholder=" password" />
+				<input type="hidden" name="uhash" id="uhash" value="x" />
+				<input type="submit" id="ls" value="Login" />
+				{% if chpw %}
+				<a id="x" href="#">change password</a>
+				{% endif %}
+				{% if ahttps %}
+				<a id="w" href="{{ ahttps }}">switch to https</a>
+				{% endif %}
+			</form>
+		</div>
+		{%- endif %}
+
 		<h1 id="cc">other stuff:</h1>
 		<ul>
 			{%- if this.uname != '*' and this.args.shr %}
@@ -94,21 +127,6 @@
 			<li><a id="k" href="{{ r }}/?reset" class="r" onclick="localStorage.clear();return true">reset client settings</a></li>
 		</ul>
 
-		<h1 id="l">login for more:</h1>
-		<div>
-			<form id="lf" method="post" enctype="multipart/form-data" action="{{ r }}/{{ qvpath }}">
-				<input type="hidden" id="la" name="act" value="login" />
-				<input type="password" id="lp" name="cppwd" placeholder=" password" />
-				<input type="hidden" name="uhash" id="uhash" value="x" />
-				<input type="submit" id="ls" value="Login" />
-				{% if chpw %}
-				<a id="x" href="#">change password</a>
-				{% endif %}
-				{% if ahttps %}
-				<a id="w" href="{{ ahttps }}">switch to https</a>
-				{% endif %}
-			</form>
-		</div>
 	</div>
 	<a href="#" id="repl">π</a>
 	{%- if not this.args.nb %}

copyparty/web/splash.js

@@ -17,9 +17,9 @@ var Ls = {
 		"l1": "logg inn:",
 		"m1": "velkommen tilbake,",
 		"n1": "404: filen finnes ikke  ┐( ´ -`)┌",
-		"o1": 'eller kanskje du ikke har tilgang? prøv å logge inn eller <a href="' + SR + '/?h">gå hjem</a>',
+		"o1": 'eller kanskje du ikke har tilgang? prøv et passord eller <a href="' + SR + '/?h">gå hjem</a>',
 		"p1": "403: tilgang nektet &nbsp;~┻━┻",
-		"q1": 'du må logge inn eller <a href="' + SR + '/?h">gå hjem</a>',
+		"q1": 'prøv et passord eller <a href="' + SR + '/?h">gå hjem</a>',
 		"r1": "gå hjem",
 		".s1": "kartlegg",
 		"t1": "handling",
@@ -29,6 +29,7 @@ var Ls = {
 		"w1": "bytt til https",
 		"x1": "bytt passord",
 		"y1": "dine delinger",
+		"z1": "lås opp område",
 		"ta1": "du må skrive et nytt passord først",
 		"ta2": "gjenta for å bekrefte nytt passord:",
 		"ta3": "fant en skrivefeil; vennligst prøv igjen",
@@ -41,15 +42,50 @@ var Ls = {
 		"ta1": "fill in your new password first",
 		"ta2": "repeat to confirm new password:",
 		"ta3": "found a typo; please try again",
+	},
+
+	"chi": {
+		"a1": "更新",
+		"b1": "你好 &nbsp; <small>(你尚未登录)</small>",
+		"c1": "登出",
+		"d1": "状态",
+		"d2": "显示所有活动线程的状态",
+		"e1": "重新加载配置",
+		"e2": "重新加载配置文件（账户/卷/卷标），$N并重新扫描所有 e2ds 卷$N$N注意：任何全局设置的更改$N都需要完全重启才能生效",
+		"f1": "你可以查看：",
+		"g1": "你可以上传到：",
+		"cc1": "开关等",
+		"h1": "关闭 k304",
+		"i1": "开启 k304",
+		"j1": "k304 会在每个 HTTP 304 时断开连接。这有助于避免某些代理服务器卡住或突然停止加载页面，但也会显著降低性能。",
+		"k1": "重置设置",
+		"l1": "登录：",
+		"m1": "欢迎回来，",
+		"n1": "404: 文件不存在 &nbsp;┐( ´ -`)┌",
+		"o1": '或者你可能没有权限？尝试输入密码或 <a href="' + SR + '/?h">回家</a>',
+		"p1": "403: 访问被拒绝 &nbsp;~┻━┻",
+		"q1": '尝试输入密码或 <a href="' + SR + '/?h">回家</a>',
+		"r1": "回家",
+		".s1": "映射",
+		"t1": "操作",
+		"u2": "自上次服务器写入的时间$N( 上传 / 重命名 / ... )$N$N17d = 17 天$N1h23 = 1 小时 23 分钟$N4m56 = 4 分钟 56 秒",
+		"v1": "连接",
+		"v2": "将此服务器用作本地硬盘$N$N警告：这将显示你的密码！",
+		"w1": "切换到 https",
+		"x1": "更改密码",
+		"y1": "你的分享",
+		"z1": "解锁区域",
+		"ta1": "请先输入新密码",
+		"ta2": "重复以确认新密码：",
+		"ta3": "发现拼写错误；请重试",
 	}
 };
 
-var LANGS = ["eng", "nor"];
-
 if (window.langmod)
 	langmod();
 
-var d = Ls[sread("cpp_lang", LANGS) || lang] || Ls.eng || Ls.nor;
+var d = Ls[sread("cpp_lang", Object.keys(Ls)) || lang] ||
+			Ls.eng || Ls.nor || Ls.chi;
 
 for (var k in (d || {})) {
 	var f = k.slice(-1),

copyparty/web/ui.css

@@ -385,6 +385,7 @@ html.y textarea:focus {
 }
 .mdo pre,
 .mdo code,
+.mdo code[class*="language-"],
 .mdo tt {
 	font-family: 'scp', monospace, monospace;
 	font-family: var(--font-mono), 'scp', monospace, monospace;

copyparty/web/up2k.js

@@ -152,12 +152,13 @@ function U2pvis(act, btns, uc, st) {
     r.mod0 = null;
 
     var markup = {
-        '404': '<span class="err">404</span>',
-        'ERROR': '<span class="err">ERROR</span>',
-        'OS-error': '<span class="err">OS-error</span>',
-        'found': '<span class="inf">found</span>',
-        'YOLO': '<span class="inf">YOLO</span>',
-        'done': '<span class="ok">done</span>',
+        '404': '<span class="err">' + L.utl_404 + '</span>',
+        'ERROR': '<span class="err">' + L.utl_err + '</span>',
+        'OS-error': '<span class="err">' + L.utl_oserr + '</span>',
+        'found': '<span class="inf">' + L.utl_found + '</span>',
+        'defer': '<span class="inf">' + L.utl_defer + '</span>',
+        'YOLO': '<span class="inf">' + L.utl_yolo + '</span>',
+        'done': '<span class="ok">' + L.utl_done + '</span>',
     };
 
     r.addfile = function (entry, sz, draw) {
@@ -445,9 +446,7 @@ function U2pvis(act, btns, uc, st) {
             return;
 
         r.npotato = 0;
-        var html = [
-            "<p>files: &nbsp; <b>{0}</b> finished, &nbsp; <b>{1}</b> failed, &nbsp; <b>{2}</b> busy, &nbsp; <b>{3}</b> queued</p>".format(
-                r.ctr.ok, r.ctr.ng, r.ctr.bz, r.ctr.q)];
+        var html = [L.u_pott.format(r.ctr.ok, r.ctr.ng, r.ctr.bz, r.ctr.q)];
 
         while (r.head < r.tab.length && has(["ok", "ng"], r.tab[r.head].in))
             r.head++;
@@ -602,7 +601,7 @@ function U2pvis(act, btns, uc, st) {
             if (nf < 9000)
                 return go();
 
-            modal.confirm('about to show ' + nf + ' files\n\nthis may crash your browser, are you sure?', go, null);
+            modal.confirm(L.u_bigtab.format(nf), go, null);
         };
     }
 
@@ -1037,7 +1036,7 @@ function up2k_init(subtle) {
         }
         catch (ex) {
             document.body.ondragenter = document.body.ondragleave = document.body.ondragover = null;
-            return modal.alert('your browser does not support drag-and-drop uploading');
+            return modal.alert(L.u_nodrop);
         }
         if (btn)
             return;
@@ -1104,7 +1103,7 @@ function up2k_init(subtle) {
         }
 
         if (!good_files.length && bad_files.length)
-            return toast.err(30, "that's not a folder!\n\nyour browser is too old,\nplease try dragdrop instead");
+            return toast.err(30, L.u_notdir);
 
         return read_dirs(null, [], [], good_files, nil_files, bad_files);
     }
@@ -1122,7 +1121,7 @@ function up2k_init(subtle) {
         if (err)
             return modal.alert('sorry, ' + err);
 
-        toast.inf(0, 'Scanning files...');
+        toast.inf(0, L.u_scan);
 
         if ((dz == 'up_dz' && uc.fsearch) || (dz == 'srch_dz' && !uc.fsearch))
             tgl_fsearch();
@@ -1210,7 +1209,7 @@ function up2k_init(subtle) {
                     match = false;
 
             if (match) {
-                var msg = ['directory iterator got stuck trying to access the following {0} items; will skip:<ul>'.format(missing.length)];
+                var msg = [L.u_dirstuck.format(missing.length) + '<ul>'];
                 for (var a = 0; a < Math.min(20, missing.length); a++)
                     msg.push('<li>' + esc(missing[a]) + '</li>');
 
@@ -1281,7 +1280,7 @@ function up2k_init(subtle) {
     }
 
     function gotallfiles(good_files, nil_files, bad_files) {
-        if (toast.txt == 'Scanning files...')
+        if (toast.txt == L.u_scan)
             toast.hide();
 
         if (uc.fsearch && !uc.turbo)
@@ -1437,7 +1436,7 @@ function up2k_init(subtle) {
             if (!actx || actx.state != 'suspended' || toast.visible)
                 return;
 
-            toast.warn(30, "<div onclick=\"start_actx();toast.inf(3,'thanks!')\">please click this text to<br />unlock full upload speed</div>");
+            toast.warn(30, "<div onclick=\"start_actx();toast.inf(3,'thanks!')\">" + L.u_actx + "</div>");
         }, 500);
     }
 
@@ -1479,7 +1478,7 @@ function up2k_init(subtle) {
         ev(e);
         var txt = linklist();
         cliptxt(txt + '\n', function () {
-            toast.inf(5, txt.split('\n').length + ' links copied to clipboard');
+            toast.inf(5, un_clip.format(txt.split('\n').length));
         });
     };
 
@@ -1746,14 +1745,6 @@ function up2k_init(subtle) {
 
                 var mou_ikkai = false;
 
-                if (st.busy.handshake.length &&
-                    st.busy.handshake[0].t_busied < now - 30 * 1000
-                ) {
-                    console.log("retrying stuck handshake");
-                    var t = st.busy.handshake.shift();
-                    st.todo.handshake.unshift(t);
-                }
-
                 var nprev = -1;
                 for (var a = 0; a < st.todo.upload.length; a++) {
                     var nf = st.todo.upload[a].nfile;
@@ -2255,6 +2246,9 @@ function up2k_init(subtle) {
         if (keepalive)
             console.log("sending keepalive handshake", t.name, t);
 
+        if (!t.srch && !t.t_handshake)
+            pvis.seth(t.n, 2, L.u_hs);
+
         var xhr = new XMLHttpRequest();
         xhr.onerror = xhr.ontimeout = function () {
             if (t.t_busied != me)  // t.done ok
@@ -2281,7 +2275,8 @@ function up2k_init(subtle) {
                     apop(st.busy.handshake, t);
                     st.todo.handshake.unshift(t);
                     t.cooldown = Date.now() + 5000 + Math.floor(Math.random() * 3000);
-                    return toast.err(0, 'Handshake error; will retry...\n\n' + L.badreply + ':\n\n' + unpre(xhr.responseText));
+                    var txt = t.t_uploading ? L.u_ehsfin : t.srch ? L.u_ehssrch : L.u_ehsinit;
+                    return toast.err(0, txt + '\n\n' + L.badreply + ':\n\n' + unpre(xhr.responseText));
                 }
 
                 t.t_handshake = Date.now();
@@ -2460,6 +2455,7 @@ function up2k_init(subtle) {
                 pvis.seth(t.n, 2, L.u_ehstmp, t);
 
                 var err = "",
+                    cls = "ERROR",
                     rsp = unpre(xhr.responseText),
                     ofs = rsp.lastIndexOf('\nURL: ');
 
@@ -2489,6 +2485,8 @@ function up2k_init(subtle) {
                     if (!t.rechecks && (err_pend || err_srcb)) {
                         t.rechecks = 0;
                         t.want_recheck = true;
+                        err = L.u_dupdefer;
+                        cls = 'defer';
                     }
                 }
                 if (rsp.indexOf('server HDD is full') + 1)
@@ -2498,7 +2496,7 @@ function up2k_init(subtle) {
                     if (!t.t_uploading)
                         st.bytes.finished += t.size;
 
-                    pvis.seth(t.n, 1, "ERROR");
+                    pvis.seth(t.n, 1, cls);
                     pvis.seth(t.n, 2, err);
                     pvis.move(t.n, 'ng');
 
@@ -2530,7 +2528,8 @@ function up2k_init(subtle) {
 
         xhr.open('POST', t.purl, true);
         xhr.responseType = 'text';
-        xhr.timeout = 42000;
+        xhr.timeout = 42000 + (t.srch || t.t_uploaded ? 0 :
+            (t.size / (1048 * 20))); // safededup 20M/s hdd
         xhr.send(JSON.stringify(req));
     }
 
@@ -2625,7 +2624,7 @@ function up2k_init(subtle) {
                 t.nojoin = t.nojoin || t.postlist.length;
                 console.log("ignoring dupe-segment with backoff", t.nojoin, t.name, t);
                 if (!toast.visible && st.todo.upload.length < 4)
-                    toast.msg(10, L.u_cbusy);
+                    toast.inf(10, L.u_cbusy);
             }
             else {
                 xhrchk(xhr, L.u_cuerr2.format(snpart, Math.ceil(t.size / chunksize), t.name), "404, target folder not found (???)", "warn", t);

docs/changelog.md

@@ -1,3 +1,142 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-0902-0108  `v1.14.4`  another
+
+## recent important news
+
+* [v1.14.3 (2024-08-30)](https://github.com/9001/copyparty/releases/tag/v1.14.3) fixed a bug that was introduced in v1.13.8 (2024-08-13); this bug could lead to **data loss** -- see the v1.14.3 release-notes for details
+
+## bugfixes
+
+* a network glitch could cause the uploader UI to panic d9e95262
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-0830-2311  `v1.14.3`  important dedup fix
+
+<img src="https://github.com/9001/copyparty/raw/hovudstraum/docs/logo.svg" width="250" align="right"/>
+
+* read-only demo server at https://a.ocv.me/pub/demo/
+* [docker image](https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker) ╱ [similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) ╱ [client testbed](https://cd.ocv.me/b/)
+
+there is a [discord server](https://discord.gg/25J8CdTT6G) with an `@everyone` in case of future important updates, such as [vulnerabilities](https://github.com/9001/copyparty/security) (most recently 2023-07-23)
+
+# important bugfix ☢️
+
+this version fixes a file deduplication bug which was introduced in [v1.13.8](https://github.com/9001/copyparty/releases/tag/v1.13.8), released 2024-08-13
+
+its worst-case outcome is **loss of data** in the following scenario:
+* someone uploads a file into a folder where that filename is already taken, but the file contents are different, and the server already has a copy of that new file elsewhere under a different name
+
+specific example:
+* the server has two existing files, `logo.png` and `logo-v2.png`, in the same volume but not necessarily in the same folder, and those files contain different data
+* you have a local copy of `logo-v2.png` on your laptop, but your local filename is `logo.png`
+* you upload your local `logo.png` onto the server, into the same folder as the server's `logo.png`
+* because the files contain different data, the server accidentally replaces the contents of `logo.png` with your version
+
+if you have been using the database feature (globally with `-e2dsa` or volflag `e2ds`), and you suspect you may have hit this bug, then it is a good idea to make a backup of the up2k databases for all your volumes (the files with names starting with `up2k.db`) before restarting copyparty and before you do anything else, especially if you do not have serverlogs from far back in time -- if you have either the databases and/or the serverlogs, then it is possible to identify replaced files with some manual work
+
+you can check if you hit the bug using one of the following two approaches:
+* if your OS has the [gnu find](https://linux.die.net/man/1/find) command, do a search for empty files with `find -type f -size 0`
+* using copyparty (any OS), do the following steps:
+  * make sure that reindex-on-startup is enabled; either globally with `-e2dsa` or volflag `e2ds`
+  * then install this new copyparty version
+  * click the search tab `[🔎]` and type the number `0` into the `maximum MiB` textbox
+
+if you find any empty files with a filename that indicates it was autogenerated to avoid a name collision, for example `logo.png-1725040569.239207-kbt0xteO.png`, and the value of the number after `logo.png` is larger than `1723507200` (unixtime for 2024-08-13), then this indicates that `logo.png` may have been replaced by another upload
+
+if you have the serverlogs from when the original upload of `logo.png` was made, then this can be used to identify the original contents of the file that was replaced, and to look for other copies. Please get in touch on the discord for assistance if necessary
+
+----
+
+## new features
+
+* shares: add revival and expiration extension ad2371f8
+  * share-owners can revive expired shares for `--shr-rt` minutes (default 1 day)
+  * ...and extend expiration time by adding 1 minute or 1 hour to the timer
+* [sfx customizer](https://github.com/9001/copyparty/blob/hovudstraum/scripts/make-sfx.sh) improvements 03b13e8a
+  * improved translations stripper
+  * add more examples
+
+## bugfixes
+
+* the dedup bug 3da62ec2
+* tftp: support unmapped root 01233991
+
+## other changes
+
+* copyparty.exe: update to pyinstaller 6.10.0 
+* textviewer wordwrapping c4e2b0f9
+* add logo 7037e736 ee359742
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-0823-2307  `v1.14.2`  bing chilling
+
+## new features
+
+* #94 @ultwcz translated the UI to Chinese (thx!)  92edea1d
+* #84 improvements to [shares](https://github.com/9001/copyparty#shares): 8122dded
+  * if one or more files are selected for sharing, they are placed into a virtual folder
+  * more appropriate password UI for accessing protected shares
+  * human-readable timestamps in shares listing
+* u2c (commandline uploader): support multiple exclusion patterns f356faa2
+
+## bugfixes
+
+* remove confusing logmessage when downloading a zerobyte file 9f034d9c
+* shares: 7ff46966
+  * fix crash if the root volume is unmapped
+  * log-spam on config reload
+  * password coalescing
+  * add chrome support
+
+## other changes
+
+* #93 add html IDs to the tabstrip 461f3158
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2024-0819-0014  `v1.14.1`  one step forward
+
+[if i turn back now, then this will always follow... one step forward, forward](https://youtu.be/xe3Wkzc0O3k?t=27)
+
+* read-only demo server at https://a.ocv.me/pub/demo/
+* [docker image](https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker) ╱ [similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) ╱ [client testbed](https://cd.ocv.me/b/)
+
+there is a [discord server](https://discord.gg/25J8CdTT6G) with an `@everyone` in case of future important updates, such as [vulnerabilities](https://github.com/9001/copyparty/security) (most recently 2023-07-23)
+
+## new features
+
+* #92 users can change their own passwords 83fb569d 00da7440
+  * this feature is default-disabled; see [readme](https://github.com/9001/copyparty#user-changeable-passwords)
+* #84 share files/folders by creating a temporary url 7c2beba5
+  * inspired by other file servers; click the share-button to create a link like `example.com/share/enkz8g374o8g`
+    * primary usecase is to sneak past authentication services (see issue description)
+  * the create-share UI has options to accept uploads into the share, and/or set expiration time
+  * this feature is default-disabled; see [readme](https://github.com/9001/copyparty#shares)
+
+## bugfixes
+
+* #93 fixes for vproxy / location-based / not-vhost-based reverse-proxying 0b46b1a6
+  * using `--rp-loc` to reverse-proxy from a subfolder made some UI stuff break
+* listening on unix-sockets: 687df2fa
+  * fix `x-forwarded-for` support, and avoid a possible container-specific collision
+  * new syntax which allows setting unix-permissions and unix-group
+    * `-i unix:770:www:/tmp/party.sock` (see `--help-bind` for more examples)
+* using relocation hooks (introduced in previous ver) could cause dedup issues c8f4aeae b0af4b37
+* custom fonts using `@import` css statements 5a62cb48
+* invert volume scrollwheel 7d8d9438
+
+## other changes
+
+* changed the button colors in theme 2 (pm-monokai) from red to yellow 5153db6b
+  * the red buttons look better, but are too confusing because usually red means off
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2024-0813-0008  `v1.13.8`  hook into place
 
@@ -791,7 +930,7 @@ dnf install https://ocv.me/copyparty/fedora/39/python3-copyparty.fc39.noarch.rpm
 ## other changes
 * improved [systemd example](https://github.com/9001/copyparty/tree/hovudstraum/contrib/systemd) with hardening and a better example config
 * logfiles are flushed for every line written; can be disabled with `--no-logflush` for ~3% more performance best-case
-* iphones probably won't broadcast cover-art to car stereos over bluetooth anymore since the thingamajig in iOS that's in charge of that doesn't have cookie-access, and strapping in the auth is too funky so let's stop doing that b7723ac245b8b3e38d6410891ef1aa92d4772114
+* iphones probably won't broadcast cover-art to car stereos over bluetooth anymore since the thingamajig in iOS that's in charge of that doesn't have cookie-access, and strapping in the auth is too funky so let's stop doing that b7723ac2
   * can be remedied by enabling filekeys and granting unauthenticated people access that way, but that's too much effort for anyone to bother with I'm sure
 
 
@@ -1060,12 +1199,12 @@ okay, i swear this is the last version for weeks! probably
     * [r0c is much better](https://github.com/9001/r0c) than this joke
 
 ## bugfixes
-* 163e3fce46122d64bf824762b6733ff2c3551ba5 the `x-forwarded-for` header was ignored if the nearest reverse-proxy is not asking from 127.0.0.1, which broke client IPs in containerized deployments
+* 163e3fce the `x-forwarded-for` header was ignored if the nearest reverse-proxy is not asking from 127.0.0.1, which broke client IPs in containerized deployments
   * the serverlog will now explain how to trust the reverse-proxy to provide client IPs, but basically,
   * `--xff-hdr` specifies which header to read the client's real ip from
   * `--xff-src` is an allowlist of IP-addresses to trust that header from
-* a62f744a187bc9f821b540e8bb4e0b9a67bd01c8 if copyparty was started while an external HDD was not connected, and that volume's index was stored elsewhere, then the index would get wiped (since all the files are gone)
-* 3b8f66c0d5c27a68841814ec06f1758f146a5ff5 javascript could crash while uploading from a very unreliable internet connection
+* a62f744a if copyparty was started while an external HDD was not connected, and that volume's index was stored elsewhere, then the index would get wiped (since all the files are gone)
+* 3b8f66c0 javascript could crash while uploading from a very unreliable internet connection
 
 ## other changes
 * copyparty.exe: updated pillow to 10.0.1 which fixes the webp cve
@@ -1111,7 +1250,7 @@ hello! it's been a while, an entire day even...
   * default compression levels are gz:3, bz2:2, xz:1; override with `?tar=gz:9`
 
 # bugfixes
-* c1efd227b7377144a5760bc6cff64f4e87b626d9 symlink-deduplicated files got indexed with the wrong last-modified timestamp
+* c1efd227 symlink-deduplicated files got indexed with the wrong last-modified timestamp
   * mostly inconsequential; would cause the dupe's uploader-ip to be forgotten on the next server restart since it would reindex to "fix" the timestamp
 * when linking [a search query](https://a.ocv.me/pub/#q=tags%20like%20soundsho*) it loads the results faster
 
@@ -1126,12 +1265,12 @@ hello! it's been a while, an entire day even...
 
 ## new features
 * iPhones and iPads are now able to...
-  * 9986136dfb2364edb35aa9fbb87410641c6d6af3 play entire albums while the screen is off without the music randomly stopping
+  * 9986136d play entire albums while the screen is off without the music randomly stopping
     * apple keeps breaking AudioContext in new and interesting ways; time to give up (no more equalizer)
-  * 1c0d978979a703edeb792e552b18d3b7695b2d90 perform search queries and execude js code
+  * 1c0d9789 perform search queries and execude js code
     * by translating [smart-quotes](https://stackoverflow.com/questions/48678359/ios-11-safari-html-disable-smart-punctuation) into regular `'` and `"` characters
 * python 3.12 support
-  * technically a bugfix since it was added [a year ago](https://github.com/9001/copyparty/commit/32e22dfe84d5e0b13914b4d0e15c1b8c9725a76d) way before the first py3.12 alpha was released but turns out i botched it, oh well
+  * technically a bugfix since it was added [a year ago](https://github.com/9001/copyparty/commit/32e22dfe) way before the first py3.12 alpha was released but turns out i botched it, oh well
 * filter error messages so they never include the filesystem path where copyparty's python files reside
 * print more context in server logs if someone hits an unexpected permission-denied
 
@@ -1352,8 +1491,8 @@ Thanks for flying copyparty! And especially if you decide to continue doing so :
       ```bash
       (gzip -dc access.log.*.gz; cat access.log) | sed -r 's/" [0-9]+ .*//' | grep -E 'cpr/.*%2[^0]' | grep -vF data:image/svg
       ```
-* 77f1e5144455eb946db7368792ea11c934f0f6da fixes an extremely unlikely race-condition (see the commit for details)
-* 8f59afb1593a75b8ce8c91ceee304097a07aea6e fixes another race-condition which is a bit worse:
+* 77f1e514 fixes an extremely unlikely race-condition (see the commit for details)
+* 8f59afb1 fixes another race-condition which is a bit worse:
   * the unpost feature could collide with other database activity, with the worst-case outcome being aborted batch operations, for example a directory move or a batch-rename which stops halfways
 
 ----
@@ -1527,7 +1666,7 @@ don't get excited! nothing new and revolutionary, but `xvol` and `xdev` changed
 # 2023-0426-2300  `v1.6.15`  unexpected boost
 
 ## new features
-* 30% faster folder listings due to [the very last thing](https://github.com/9001/copyparty/commit/55c74ad164633a0a64dceb51f7f534da0422cbb5) i'd ever expect to be a bottleneck, [thx perf](https://docs.python.org/3.12/howto/perf_profiling.html)
+* 30% faster folder listings due to [the very last thing](https://github.com/9001/copyparty/commit/55c74ad1) i'd ever expect to be a bottleneck, [thx perf](https://docs.python.org/3.12/howto/perf_profiling.html)
 * option to see the lastmod timestamps of symlinks instead of the target files
   * makes the turbo mode of [u2cli, the commandline uploader and folder-sync tool](https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py) more turbo since copyparty dedupes uploads by symlinking to an existing copy and the symlink is stamped with the deduped file's lastmod
   * **webdav:** enabled by default (because rclone will want this), can be disabled with arg `--dav-rt` or volflag `davrt`
@@ -1731,7 +1870,7 @@ don't get excited! nothing new and revolutionary, but `xvol` and `xdev` changed
 the commandline up2k upload / filesearch client, now as a standalone windows exe
 * based on python 3.7 so it runs on 32bit windows7 or anything newer
 * *no https support* (saves space + the python3.7 openssl is getting old)
-* built from b39ff92f34e3fca389c78109d20d5454af761f8e so it can do long filepaths and mojibake
+* built from b39ff92f so it can do long filepaths and mojibake
 
 ----
 
@@ -1938,7 +2077,7 @@ but nothing is affected (that i know of):
 * tar/zip-download of hidden folders
 * unpost filtering was buggy for non-ascii characters
 * moving a deduplicated file on a volume where deduplication was since disabled
-* improved the [linux 6.0.16](https://utcc.utoronto.ca/~cks/space/blog/linux/KernelBindBugIn6016) kernel bug [workaround](https://github.com/9001/copyparty/commit/9065226c3d634a9fc15b14a768116158bc1761ad) because there is similar funk in 5.x
+* improved the [linux 6.0.16](https://utcc.utoronto.ca/~cks/space/blog/linux/KernelBindBugIn6016) kernel bug [workaround](https://github.com/9001/copyparty/commit/9065226c) because there is similar funk in 5.x
 * add custom text selection colors because chrome is currently broken on fedora
 * blockdevs (`/dev/nvme0n1`) couldn't be downloaded as files
 * misc fixes for location-based reverse-proxying
@@ -1967,7 +2106,7 @@ hello from warsaw airport (goodbye japan ;_;)
 * browser ui didn't allow specifying number of threads for file search
 * dont panic if a digit key is pressed while viewing an image
 * workaround [linux kernel bug](https://utcc.utoronto.ca/~cks/space/blog/linux/KernelBindBugIn6016) causing log spam on dualstack
-  * ~~related issue (also mostly harmless) will be fixed next relese 010770684db95bece206943768621f2c7c27bace~~ 
+  * ~~related issue (also mostly harmless) will be fixed next relese 01077068~~ 
     * they fixed it in linux 6.1 so these workarounds will be gone too
 
 
@@ -2933,7 +3072,7 @@ fixed another dumdum, sorry for the spam
 * the ftp server is not compatible with python 3.12 (releasing october 2023)
   * will be fixed in a [future version of pyftpdlib](https://github.com/giampaolo/pyftpdlib/issues/560)
 
-the sfx was built from https://github.com/9001/copyparty/commit/39e7a7a2311ab8da43b2a9a18ae39d06202105e3
+the sfx was built from https://github.com/9001/copyparty/commit/39e7a7a2
 
 
 
@@ -3714,7 +3853,7 @@ we did it reddit 👉😎👉
 * latest gzip edition of the sfx: [v0.11.18](https://github.com/9001/copyparty/releases/tag/v0.11.18)
 * if upgrading from v0.11.x or before, see [v0.12.4](https://github.com/9001/copyparty/releases/tag/v0.12.4)
 
-note: `copyparty-sfx.py` is https://github.com/9001/copyparty/commit/5955940b82adddb7149125a60463aba22f1c8c31 which fixes upload eta
+note: `copyparty-sfx.py` is https://github.com/9001/copyparty/commit/5955940b which fixes upload eta
 
 ## new features
 * provide password using basic-authentication
@@ -5184,7 +5323,7 @@ nothing really important happened since [v0.11.6](https://github.com/9001/copypa
 * this release fixes a missing permission check which could allow users to download write-only folders
   * this bug was introduced 19 days ago, in `v0.10.17`
   * the requirement to be affected is write-only folders mounted within readable folders
-  * and the worst part is there was a unit-test exactly for this, https://github.com/9001/copyparty/commit/273ca0c8da0d94f9d06ca16bd86c0301d9d06455 way overdue
+  * and the worst part is there was a unit-test exactly for this, https://github.com/9001/copyparty/commit/273ca0c8 way overdue
 * also fixes minor bugs introduced in `v0.11.1`
 * this version is the same as `v0.11.5` on pypi
 
@@ -5368,8 +5507,8 @@ in other news, minor ui tweaks:
 * a few lightmode adjustments
 * less cpu usage? should be
 
-`copyparty-sfx.py` (latest) made from c5db7c1a0c8f6ab23138ad7ea7642a6260e7da9b (v0.10.15-15) fixes `-j` (multiprocessing/high-performance)
-`copyparty-sfx-5a579db.py` (old) made from 5a579dba52e46c202b79c3d80c3b1c996c7b2e4a (v0.10.15-5) reduced the size
+`copyparty-sfx.py` (latest) made from c5db7c1a (v0.10.15-15) fixes `-j` (multiprocessing/high-performance)
+`copyparty-sfx-5a579db.py` (old) made from 5a579dba (v0.10.15-5) reduced the size
 
 
 
@@ -5682,7 +5821,7 @@ and i just realized i never added runtime tag scanning so copyparty will have to
 
 use `-e2dsa` and `-e2ts` to enable the media tag features globally, or enable/disable them per-volume (see readme)
 
-**NOTE:** older fuse clients (from before 5e3775c1afc9438f9930080a9b8542a063ba1765  / older than v0.8.0) must be upgraded for this copyparty release, however the new client still supports connecting to old servers
+**NOTE:** older fuse clients (from before 5e3775c1  / older than v0.8.0) must be upgraded for this copyparty release, however the new client still supports connecting to old servers
 
 other changes include
 * support chunked PUT requests from curl
@@ -5902,7 +6041,7 @@ valvrave-stop.jpg
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2020-0818-1822  `v0.5.2`  da setter vi punktum
 
-full disclaimer: `copyparty-sfx.py` was built using `sfx.py` from ~~82e568d4c9f25bfdfd1bf5166f0ebedf058723ee~~ f550a8171d298992f4ef569d2fc99a6037a44ea8
+full disclaimer: `copyparty-sfx.py` was built using `sfx.py` from ~~82e568d4~~ f550a817
 
 
 

docs/devnotes.md

@@ -176,7 +176,8 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | mPOST | `?media` | `f=FILE` | ...and return medialink (not hotlink) |
 | mPOST | | `act=mkdir`, `name=foo` | create directory `foo` at URL |
 | POST | `?delete` | | delete URL recursively |
-| POST | `?unshare` | | stop sharing a file/folder |
+| POST | `?eshare=rm` | | stop sharing a file/folder |
+| POST | `?eshare=3` | | set expiration to 3 minutes |
 | jPOST | `?share` | (complicated) | create temp URL for file/folder |
 | jPOST | `?delete` | `["/foo","/bar"]` | delete `/foo` and `/bar` recursively |
 | uPOST | | `msg=foo` | send message `foo` into server log |

docs/examples/windows.md

@@ -16,7 +16,7 @@ open up notepad and save the following as `c:\users\you\documents\party.conf` (f
 ```yaml
 [global]
   lo: ~/logs/cpp-%Y-%m%d.xz  # log to c:\users\you\logs\
-  e2dsa, e2ts, no-dedup, z   # sets 4 flags; see expl.
+  e2dsa, e2ts, z    # sets 3 flags; see explanation
   p: 80, 443  # listen on ports 80 and 443, not 3923
   theme: 2    # default theme: protonmail-monokai
   lang: nor   # default language: viking
@@ -46,11 +46,10 @@ open up notepad and save the following as `c:\users\you\documents\party.conf` (f
 
 ### config explained: [global]
 
-the `[global]` section accepts any config parameters [listed here](https://ocv.me/copyparty/helptext.html), also viewable by running copyparty (either the exe or the sfx.py) with `--help`, so this is the same as running copyparty with arguments `--lo c:\users\you\logs\copyparty-%Y-%m%d.xz -e2dsa -e2ts --no-dedup -z -p 80,443 --theme 2 --lang nor`
+the `[global]` section accepts any config parameters [listed here](https://ocv.me/copyparty/helptext.html), also viewable by running copyparty (either the exe or the sfx.py) with `--help`, so this is the same as running copyparty with arguments `--lo c:\users\you\logs\copyparty-%Y-%m%d.xz -e2dsa -e2ts -z -p 80,443 --theme 2 --lang nor`
 * `lo: ~/logs/cpp-%Y-%m%d.xz` writes compressed logs (the compression will make them delayed)
-* `e2dsa` enables the upload deduplicator and file indexer, which enables searching
+* `e2dsa` enables the file indexer, which enables searching and upload-undo
 * `e2ts` enables music metadata indexing, making albums / titles etc. searchable too
-* `no-dedup` writes full dupes to disk instead of symlinking, since lots of windows software doesn't handle symlinks well
   * but the improved upload speed from `e2dsa` is not affected
 * `z` enables zeroconf, making the server available at `http://HOSTNAME.local/` from any other machine in the LAN
 * `p: 80,443` listens on the ports `80` and `443` instead of the default `3923`

docs/logo.svg

@@ -0,0 +1,210 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+  width="300mm"
+  height="207mm"
+  viewBox="0 0 300 207"
+  version="1.1"
+  id="svg1"
+  inkscape:version="1.3.2 (091e20ef0f, 2023-11-25)"
+  xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+  xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:svg="http://www.w3.org/2000/svg"
+  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+  xmlns:cc="http://creativecommons.org/ns#"
+  xmlns:dc="http://purl.org/dc/elements/1.1/">
+ <title
+   id="title1">copyparty_logo</title>
+ <defs
+   id="defs1">
+  <linearGradient
+    inkscape:collect="always"
+    id="linearGradient1">
+   <stop
+     style="stop-color:#ffcc55;stop-opacity:1"
+     offset="0"
+     id="stop1" />
+   <stop
+     style="stop-color:#ffcc00;stop-opacity:1"
+     offset="0.2"
+     id="stop2" />
+   <stop
+     style="stop-color:#ff8800;stop-opacity:1"
+     offset="1"
+     id="stop3" />
+  </linearGradient>
+  <linearGradient
+    inkscape:collect="always"
+    xlink:href="#linearGradient1"
+    id="linearGradient2"
+    x1="15"
+    y1="15"
+    x2="15"
+    y2="143"
+    gradientUnits="userSpaceOnUse" />
+ </defs>
+ <metadata
+   id="metadata5">
+  <rdf:RDF>
+   <cc:Work
+     rdf:about="">
+    <dc:format>image/svg+xml</dc:format>
+    <dc:type
+      rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+    <dc:title>copyparty_logo</dc:title>
+    <dc:source>github.com/9001/copyparty</dc:source>
+   </cc:Work>
+  </rdf:RDF>
+ </metadata>
+ <g
+   inkscape:groupmode="layer"
+   id="layer1"
+   inkscape:label="kassett">
+  <rect
+    style="fill:#333333"
+    id="rect1"
+    width="300"
+    height="205"
+    x="0"
+    y="0"
+    rx="12"
+    ry="12" />
+  <rect
+    style="fill:url(#linearGradient2)"
+    id="rect2"
+    width="270"
+    height="128"
+    x="15"
+    y="15"
+    rx="8"
+    ry="8" />
+  <rect
+    style="fill:#333333"
+    id="rect3"
+    width="172"
+    height="52"
+    x="64"
+    y="72"
+    rx="26"
+    ry="26" />
+  <circle
+    style="fill:#cccccc"
+    id="circle1"
+    cx="91"
+    cy="98"
+    r="18" />
+  <circle
+    style="fill:#cccccc"
+    id="circle2"
+    cx="209"
+    cy="98"
+    r="18" />
+  <path
+    style="fill:#737373;stroke-width:1px"
+    d="m 48,207 10,-39 c 1.79,-6.2 5.6,-7.8 12,-8 60,-1 100,-1 160,0 6.4,0.2 10,1.8 12,8 l 10,39 z"
+    id="path1"
+    sodipodi:nodetypes="ccccccc" />
+ </g>
+ <g
+   inkscape:groupmode="layer"
+   id="layer3"
+   inkscape:label="tekst"
+   style="display:none">
+  <text
+    xml:space="preserve"
+    style="font-size:38.8056px;line-height:1.25;font-family:Akbar;-inkscape-font-specification:Akbar;letter-spacing:3.70417px;word-spacing:0px;fill:#333333"
+    x="47.153069"
+    y="55.548954"
+    id="text1"><tspan
+     sodipodi:role="line"
+     id="tspan1"
+     x="47.153069"
+     y="55.548954"
+     style="-inkscape-font-specification:Akbar"
+     rotate="0 0">copyparty</tspan></text>
+ </g>
+ <g
+   inkscape:groupmode="layer"
+   id="layer4"
+   inkscape:label="stensatt">
+  <path
+    d="m 63.5,50.9 q -0.85,0.93 -4.73,2.3 -3.6,1.3 -4.4,1.3 -3.3,0 -5.1,-2.1 -1.75,-2 -1.75,-5.36 0,-4.6 3.76,-7.64 3.3,-2.7 7.3,-2.7 0.4,0 0.93,0.74 0.54,0.7 0.54,1.16 0,2.06 -2.2,2.7 -1.36,0.4 -4.04,1.16 -2.2,1.16 -2.2,4.4 0,3.2 2.9,3.2 0.85,0 0.85,0 0.54,0 1.44,-0.16 1.1,-0.23 2.9,-0.74 1.8,-0.54 2.13,-0.54 0.4,0 1.75,0.6 z"
+    style="fill:#333333"
+    id="path11" />
+  <path
+    d="m 87.6,45 q 0,4.2 -3.7,6.95 -3.2,2.3 -6.87,2.3 -3.4,0 -6,-2.6 -2.5,-2.6 -2.5,-6 0,-3.6 3.14,-6.64 3.2,-3 6.8,-3 3.5,0 6.3,2.76 2.83,2.76 2.83,6.25 z m -3.4,0.16 q 0,-2.25 -1.75,-3.7 -1.7,-1.5 -4,-1.5 -0.1,0 -1.6,1.6 -1.44,1.55 -2.44,1.55 -0.6,0 -0.8,-0.3 -1.16,2.3 -1.16,3 0,2.25 2.13,3.4 1.6,0.9 3.6,0.9 2,0 3.76,-1.1 2.25,-1.4 2.25,-3.84 z"
+    style="fill:#333333"
+    id="path12" />
+  <path
+    d="m 112.8,46.8 q 0,2.8 -1.9,4.4 -1.8,1.5 -4.7,1.5 -0.7,0 -2.7,-0.4 -1.9,-0.4 -2.6,-0.4 -2.1,0 -2.1,2.64 0,0.85 0.23,2.6 0.2,1.75 0.2,2.6 0,1.9 -0.77,2.83 -1.44,0 -3,-0.85 -1.46,-9.5 -1.46,-12 0,-3.65 1.75,-8.1 2.37,-6.05 6.45,-6.05 3.7,0 7.3,4.1 3.3,3.84 3.3,7.14 z m -3.8,0.2 q -0.6,-2.2 -2.6,-4.4 -2.3,-2.5 -4.3,-2.5 -1.3,0 -2.33,2.2 -0.9,1.8 -0.9,3.26 0,0.47 0.38,1.24 0.43,0.8 0.85,0.8 1.1,0 3.2,0.3 2.1,0.3 3.2,0.3 0.3,0 1.3,-0.4 1,-0.47 1.3,-0.74 z"
+    style="fill:#333333"
+    id="path13" />
+  <path
+    d="m 133,40 q -2.1,4.1 -3.2,7 -0.1,0.3 -1.6,4.5 -0.4,1.36 -1,4.2 -0.5,2.83 -1,4.2 -1,2.83 -2.3,2.64 -1.4,-0.2 -1.6,-1.6 0,-0.2 0,-0.5 0,-0.16 0.3,-1.5 1,-5.04 1,-6.44 0,-0.54 -0.1,-0.74 -1.4,-2.44 -4.1,-7.4 -2.7,-4.97 -2.4,-7.7 1.5,-1.36 2.1,-1.36 0.4,0 1.1,0.6 0.6,0.6 0.7,1.1 0.8,6.2 4.9,11.1 1,-1.8 1.8,-4.04 0.5,-1.4 1.6,-4.15 1.9,-4.46 3.4,-4.46 0.2,0 0.4,0.1 0.9,0.3 1.3,2.8 z"
+    style="fill:#333333"
+    id="path14" />
+  <path
+    d="m 157.5,48 q 0,2.8 -1.9,4.4 -1.8,1.5 -4.7,1.5 -0.7,0 -2.7,-0.4 -1.9,-0.4 -2.6,-0.4 -2,0 -2,2.64 0,0.85 0.2,2.6 0.2,1.75 0.2,2.6 0,1.9 -0.7,2.83 -1.5,0 -3,-0.85 -1.5,-9.5 -1.5,-11.95 0,-3.65 1.8,-8.1 2.3,-6.05 6.4,-6.05 3.7,0 7.2,4.1 3.3,3.84 3.3,7.14 z m -3.8,0.2 q -0.6,-2.2 -2.6,-4.4 -2.3,-2.5 -4.3,-2.5 -1.3,0 -2.3,2.2 -0.9,1.8 -0.9,3.26 0,0.47 0.4,1.24 0.4,0.8 0.8,0.8 1.1,0 3.2,0.3 2.1,0.3 3.2,0.3 0.3,0 1.3,-0.4 1,-0.47 1.3,-0.74 z"
+    style="fill:#333333"
+    id="path15" />
+  <path
+    d="m 182,53.3 q 0,0.9 -0.6,1.5 -0.6,0.6 -1.4,0.6 -1.6,0 -3,-0.9 -1.4,-0.93 -2.1,-2.3 -0.7,-0.1 -1.5,0.85 -0.9,1.16 -1.1,1.24 -1.2,0.54 -3.9,0.54 -2.2,0 -3.9,-2.44 -1.5,-2.13 -1.5,-4 0,-3.4 3.4,-6.4 3.2,-2.9 6.7,-2.9 0.9,0 1.7,0.6 0.8,0.6 0.8,1.44 0,0.54 -0.4,1.1 2.4,0.9 2.4,2.83 0,0.35 -0.1,1.05 -0.1,0.7 -0.1,1.05 0,0.4 0.1,0.6 0.5,1.3 2.5,3.4 1.9,1.9 1.9,2.2 z m -8.1,-10.1 q -0.4,0 -1.1,-0.1 -0.8,-0.16 -1.1,-0.16 -1.3,0 -3.2,1.94 -1.9,1.94 -1.9,3.3 0,0.8 0.7,1.8 0.9,1.3 2.2,1.3 2.6,0 3.5,-2.9 0.5,-2.6 1,-5.16 z"
+    style="fill:#333333"
+    id="path16" />
+  <path
+    d="m 203.8,42.4 q -0.4,0.4 -1.5,0.4 -0.9,0 -2.5,-0.3 -1.7,-0.3 -2.5,-0.3 -4.7,0 -5.5,6.9 -0.3,3.1 -0.4,3.3 -0.4,1 -1.7,2.3 h -1.1 q -0.7,-1.2 -1.3,-4.1 -0.6,-2.76 -0.6,-4.27 0,-1.16 0.1,-1.5 0.2,-0.54 1,-0.54 0.3,0 0.6,0.3 0.4,0.3 0.4,0.3 1.9,-3.53 3.1,-4.6 1.8,-1.7 5.1,-1.7 1.4,0 3.6,0.9 2.8,1.16 3.3,2.8 z"
+    style="fill:#333333"
+    id="path17" />
+  <path
+    d="m 229.5,37.16 q 0.3,0.8 0.3,1.44 0,1.86 -2.4,1.86 -1,0 -3.5,-0.5 -2.5,-0.54 -3.4,-0.54 -1.3,0 -1.5,0.1 -0.4,0.2 -0.4,1.2 0,2.2 0.6,6.9 0.7,5.86 1.6,6.13 -0.4,0.35 -0.4,1.1 -1.2,0.7 -2.6,0.7 -1.4,0 -2,-3.9 -0.2,-1.36 -0.5,-7.76 -0.2,-4.6 -0.8,-5.5 -0.3,-0.47 -4.3,-0.35 -1,0 -1.6,0.1 -0.5,0 -0.3,0 -0.8,0 -1.2,-0.7 -0.5,-1.3 -0.5,-1.4 0,-1.44 4.1,-2 1.6,-0.16 4.7,-0.5 0,-0.85 -0.1,-2.56 0,-1.75 0,-2.6 0,-4.35 2.1,-4.35 0.5,0 1.1,0.6 0.6,0.6 0.6,1.1 v 7.9 q 1.1,1.2 5,1.7 3.9,0.5 5.3,1.86 z"
+    style="fill:#333333"
+    id="path18" />
+  <path
+    d="m 251.2,40.2 q -2,4.1 -3.2,7 -0.1,0.3 -1.5,4.5 -0.5,1.36 -1,4.2 -0.5,2.83 -1,4.2 -1,2.83 -2.4,2.64 -1.4,-0.2 -1.5,-1.6 -0.1,-0.2 -0.1,-0.5 0,-0.16 0.3,-1.5 1.1,-5.04 1.1,-6.44 0,-0.54 -0.1,-0.74 -1.4,-2.44 -4.1,-7.4 -2.7,-4.97 -2.4,-7.7 1.4,-1.36 2.1,-1.36 0.4,0 1,0.6 0.6,0.6 0.7,1.1 0.9,6.2 4.9,11.1 1,-1.8 1.9,-4.04 0.5,-1.4 1.6,-4.15 1.8,-4.46 3.4,-4.46 0.2,0 0.4,0.1 0.8,0.3 1.2,2.8 z"
+    style="fill:#333333"
+    id="path19" />
+ </g>
+ <g
+   inkscape:groupmode="layer"
+   id="layer5"
+   inkscape:label="tagger">
+  <g
+    id="g1">
+   <path
+     id="path4"
+     style="fill:#333333"
+     d="m 111.4,83.335 -9.526,5.5 2.5,4.33 9.526,-5.5 z m -33.775,19.5 -9.526,5.5 2.5,4.33 9.526,-5.5 z"
+     sodipodi:nodetypes="cccccccccc" />
+   <path
+     id="path5"
+     style="fill:#333333"
+     d="M 88.5,73 V 84 h 5 V 73 Z m 0,39 v 11 h 5 V 112 Z"
+     sodipodi:nodetypes="cccccccccc" />
+   <path
+     id="path6"
+     style="fill:#333333"
+     d="m 68.1,87.665 9.526,5.5 2.5,-4.33 -9.526,-5.5 z m 33.775,19.5 9.527,5.5 2.5,-4.33 -9.527,-5.5 z"
+     sodipodi:nodetypes="cccccccccc" />
+  </g>
+  <g
+    id="g2"
+    transform="rotate(30,150,318.19)">
+   <path
+     id="path7"
+     style="fill:#333333"
+     d="m 111.4,83.335 -9.526,5.5 2.5,4.33 9.526,-5.5 z m -33.775,19.5 -9.526,5.5 2.5,4.33 9.526,-5.5 z"
+     sodipodi:nodetypes="cccccccccc" />
+   <path
+     id="path8"
+     style="fill:#333333"
+     d="M 88.5,73 V 84 h 5 V 73 Z m 0,39 v 11 h 5 V 112 Z"
+     sodipodi:nodetypes="cccccccccc" />
+   <path
+     id="path9"
+     style="fill:#333333"
+     d="m 68.1,87.665 9.526,5.5 2.5,-4.33 -9.526,-5.5 z m 33.775,19.5 9.527,5.5 2.5,-4.33 -9.527,-5.5 z"
+     sodipodi:nodetypes="cccccccccc" />
+  </g>
+ </g>
+</svg>

scripts/docker/Dockerfile.ac

@@ -8,7 +8,7 @@ LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
 ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk --no-cache add !pyc \
-            wget \
+            tzdata wget \
             py3-jinja2 py3-argon2-cffi py3-pillow \
             ffmpeg
 

scripts/docker/Dockerfile.dj

@@ -11,7 +11,7 @@ COPY    i/bin/mtag/install-deps.sh ./
 COPY    i/bin/mtag/audio-bpm.py /mtag/
 COPY    i/bin/mtag/audio-key.py /mtag/
 RUN     apk add -U !pyc \
-            wget \
+            tzdata wget \
             py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
             ffmpeg \
             vips-jxl vips-heif vips-poppler vips-magick \

scripts/docker/Dockerfile.im

@@ -8,7 +8,7 @@ LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
 ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk --no-cache add !pyc \
-            wget \
+            tzdata wget \
             py3-jinja2 py3-argon2-cffi py3-pillow py3-mutagen
 
 COPY    i/dist/copyparty-sfx.py innvikler.sh ./

scripts/docker/Dockerfile.iv

@@ -8,7 +8,7 @@ LABEL   org.opencontainers.image.url="https://github.com/9001/copyparty" \
 ENV     XDG_CONFIG_HOME=/cfg
 
 RUN     apk add -U !pyc \
-            wget \
+            tzdata wget \
             py3-jinja2 py3-argon2-cffi py3-pillow py3-pip py3-cffi \
             ffmpeg \
             vips-jxl vips-heif vips-poppler vips-magick \

scripts/docker/make.sh

@@ -100,11 +100,11 @@ filt=
             aa="$(printf '%11s' $a-$i)"
 
             # arm takes forever so make it top priority
-            [ ${a::3} == arm ] && nice= || nice=nice
+            [ ${a::3} == arm ] && nice= || nice=-n20
 
             # --pull=never does nothing at all btw
             (set -x
-            $nice podman build \
+            nice $nice podman build \
                 --squash \
                 --pull=never \
                 --from localhost/alpine-$a \

scripts/make-sfx.sh

@@ -3,6 +3,7 @@ set -e
 echo
 
 berr() { p=$(head -c 72 </dev/zero | tr '\0' =); printf '\n%s\n\n' $p; cat; printf '\n%s\n\n' $p; }
+aerr() { printf '%s\n' "$*" | berr; }
 
 help() { exec cat <<'EOF'
 
@@ -28,9 +29,11 @@ help() { exec cat <<'EOF'
 #
 # `no-tfp` saves ~10k by removing the tftp server, disabling --tftp
 #
+# `no-zm` saves ~7k by removing the zeroconf mDNS server
+#
 # `no-smb` saves ~3.5k by removing the smb / cifs server
 #
-# `no-zm` saves ~k by removing the zeroconf mDNS server
+# `no-pf` saves ~2.8k by removing the option to download partyfuse
 #
 # _____________________________________________________________________
 # web features:
@@ -52,10 +55,15 @@ help() { exec cat <<'EOF'
 #
 # `ign-wd` allows building an sfx without webdeps
 #
-# ---------------------------------------------------------------------
-#
+# _____________________________________________________________________
 # if you are on windows, you can use msys2:
 #   PATH=/c/Users/$USER/AppData/Local/Programs/Python/Python310:"$PATH" ./make-sfx.sh fast
+#
+# _____________________________________________________________________
+# some usage examples:
+#   ./scripts/make-sfx.sh lang eng no-cm no-hl no-dd no-fnt no-smb no-pf
+#   ./scripts/rls.sh sfx  lang eng no-cm no-hl no-dd no-fnt no-smb no-pf
+#   (reduces v1.14.2 from 700k to 495k)
 
 EOF
 }
@@ -112,6 +120,7 @@ while [ ! -z "$1" ]; do
 		no-tfp) no_tfp=1 ; ;;
 		no-smb) no_smb=1 ; ;;
 		no-zm)  no_zm=1  ; ;;
+		no-pf)  no_pf=1  ; ;;
 		no-fnt) no_fnt=1 ; ;;
 		no-hl)  no_hl=1  ; ;;
 		no-dd)  no_dd=1  ; ;;
@@ -119,7 +128,6 @@ while [ ! -z "$1" ]; do
 		dl-wd)  dl_wd=1  ; ;;
 		ign-wd) ign_wd=1 ; ;;
 		fast)   zopf=    ; ;;
-		ultra)  ultra=1  ; ;;
 		lang)   shift;langs="$1"; ;;
 		*)      help     ; ;;
 	esac
@@ -428,6 +436,9 @@ rm -f ftp/pyftpdlib/{__main__,prefork}.py
 [ $no_zm ] &&
 	rm -rf copyparty/mdns.py copyparty/stolen/dnslib
 
+[ $no_pf ] &&
+	rm -rf copyparty/web/a/partyfuse.py
+
 [ $no_cm ] && {
 	rm -rf copyparty/web/mde.* copyparty/web/deps/easymde*
 	echo h > copyparty/web/mde.html
@@ -451,11 +462,16 @@ rm -f ftp/pyftpdlib/{__main__,prefork}.py
 	ised 's/(cursor: ?)url\([^)]+\), ?(pointer)/\1\2/; s/[0-9]+% \{cursor:[^}]+\}//; s/animation: ?cursor[^};]+//' $f
 }
 
-[ $langs ] &&
+[ $langs ] && {
+	echo $langs | grep -q eng || {
+		langs="eng|$langs"
+		aerr "ERROR: removing english is not supported; will do this instead: $langs"
+	}
 	for f in copyparty/web/{browser.js,splash.js}; do
 		gzip -d "$f.gz" || true
-		iawk '/^\}/{l=0} !l; /^var Ls =/{l=1;next} o; /^\t["}]/{o=0} /^\t"'"$langs"'"/{o=1;print}' $f
+		iawk '/^\}/{l=0} !l; /^var Ls =/{l=1;next} !l{next} o; /^\t["}]/{o=0} /^\t"'"$langs"'"/{o=1;print}' $f
 	done
+}
 
 [ ! $repack ] && {
 	# uncomment
@@ -593,7 +609,7 @@ pc="bzip2 -"; pe=bz2
 [ $use_gzz ] && pc="pigz -11 -I$use_gzz" && pe=gz
 
 echo compressing tar
-for n in {2..9}; do cp tar t.$n; nice $pc$n t.$n & done; wait
+for n in {2..9}; do cp tar t.$n; nice -n20 $pc$n t.$n & done; wait
 minf=$(for f in t.*.$pe; do
 	s1=$(wc -c <$f)
 	s2=$(tr -d '\r\n\0' <$f | wc -c)

scripts/pyinstaller/deps.sha512

@@ -34,6 +34,6 @@ d1420c8417fad7888766dd26b9706a87c63e8f33dceeb8e26d0056d5127b0b3ed9272e44b4b76113
 8a6e2b13a2ec4ef914a5d62aad3db6464d45e525a82e07f6051ed10474eae959069e165dba011aefb8207cdfd55391d73d6f06362c7eb247b08763106709526e  mutagen-1.47.0-py3-none-any.whl
 0203ec2551c4836696cfab0b2c9fff603352f03fa36e7476e2e1ca7ec57a3a0c24bd791fcd92f342bf817f0887854d9f072e0271c643de4b313d8c9569ba8813  packaging-24.1-py3-none-any.whl
 2be320b4191f208cdd6af183c77ba2cf460ea52164ee45ac3ff17d6dfa57acd9deff016636c2dd42a21f4f6af977d5f72df7dacf599bebcf41757272354d14c1  pillow-10.4.0-cp312-cp312-win_amd64.whl
-776378f5414efd26ec8a1cb3228a7b5fdf6afca3fa335a0e9b071266d55d9d9e66ee157c25a468a05bfa70ccd33c48b101998523fc6ff6bcf5e82a1d81ed0af8  pyinstaller-6.9.0-py3-none-win_amd64.whl
-c0af77d2a57cb063ab038dc986ed3582bc5acc8c8bd91d726101935d6388f50854ddbca26bc846ed5d1022cdee4d96242938c66f0ddc4565c36b60d691064db8  pyinstaller_hooks_contrib-2024.7-py2.py3-none-any.whl
-0572c6345f6a4f7f3e5c2ff858e3ca7ca54ae4478f3d59d8e18cb0f596e61dcf12aef579db229e83d63b30f15d6684ee6bb3feaea9413e5e636a503933057678  python-3.12.5-amd64.exe
+896ddddbd4b85e86e0600cb65eb4c07fbc7f3802d47e7f660411e20b5500831469b97ed4770f25820f4e75cbfac40308da624fd86d4f62e578149d5c276a9cde  pyinstaller-6.10.0-py3-none-win_amd64.whl
+873781decaeef07f6a79b0ed8b9f35f3fa534a1ea0d866991e40278a10818fa5b60c70b0d5828971b045364f1099694cd1e5d5d60d480acb93fcfbfbced4a09e  pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
+37fa7250b10b0c03b87d800bf4f920589649309cb4fbd25864475084bb7873d62b809a4fdeabd06c79f03f33614218eb7e01a9bd796de29dd3b141f1906d588c  python-3.12.6-amd64.exe

scripts/pyinstaller/notes.txt

@@ -39,9 +39,9 @@ fns=(
   mutagen-1.47.0-py3-none-any.whl
   packaging-24.1-py3-none-any.whl
   pillow-10.4.0-cp312-cp312-win_amd64.whl
-  pyinstaller-6.9.0-py3-none-win_amd64.whl
-  pyinstaller_hooks_contrib-2024.7-py2.py3-none-any.whl
-  python-3.12.5-amd64.exe
+  pyinstaller-6.10.0-py3-none-win_amd64.whl
+  pyinstaller_hooks_contrib-2024.8-py3-none-any.whl
+  python-3.12.6-amd64.exe
 )
 [ $w7 ] && fns+=(  # u2c stuff
   certifi-2024.2.2-py3-none-any.whl

scripts/rls.sh

@@ -1,6 +1,8 @@
 #!/bin/bash
 set -e
 
+# if specified, keep the following sfx flags last: gz gzz fast
+
 parallel=1
 
 [ -e make-sfx.sh ] || cd scripts
@@ -35,6 +37,14 @@ f=../dist/copyparty-sfx
 
 $f$s.py --version >/dev/null
 
+while [ "$1" ]; do
+    case "$1" in
+        gz*) break;;
+        fast) break;;
+    esac
+    shift
+done
+
 [ $parallel -gt 1 ] && {
     printf '\033[%s' s 2r H "0;1;37;44mbruteforcing sfx size -- press enter to terminate" K u "7m $* " K $'27m\n'
     trap "rm -f .sfx-run; printf '\033[%s' s r u" INT TERM EXIT

scripts/tlcheck.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+set -e
+
+# usage: ./scripts/tlcheck.sh eng chi copyparty/web/browser.js 
+
+awk <"$3" -v lang1=\"$1\": -v lang2=\"$2\": '
+    /^\t\}/{fa=0;fb=0}
+    !/":/{next}
+    $0~lang1{fa=1}
+    $0~lang2{fb=1}
+    fa{a[ia++]=$0}
+    fb{b[ib++]=$0}
+    END{for (i=0;i<ia;i++) printf "%s\n%s\n\n",a[i],b[i]}
+' |
+awk -v apos=\' -v quot=\" '
+    # count special chars and prefix to line
+    function c(ch) {
+        m=$0;
+        gsub(ch,"",m);
+        t=t sprintf("%s%d ", ch, length($0)-length(m))
+    }
+    !$0 && t!=tp {
+        print "\n\033[1;37;41m====DIFF===="
+    }
+    !$0 { print; next; }
+    {
+        tp=t; t="";
+        c(quot);
+        c(apos);
+        c("<");
+        c(">");
+        c("{");
+        c("}");
+        c("&");
+        c("\\\$");
+        c("\\\\");
+        print t $0;
+    }
+' |
+sed -r $'
+    s/\\\\/\033[1;37;41m\\\\\033[0m/g;
+    s/\$N/\033[1;37;45m$N\033[0m/g;
+    s/([{}])/\033[34m\\1\033[0m/g;
+    s/"/\033[44m"\033[0m/g;
+    s/\'/\033[45m\'\033[0m/g;
+    s/&/\033[1;43;30m&\033[0m/g;
+    s/([<>])/\033[30;47m\\1\033[0m/g
+' |
+sed -r 's/\t+//' |
+less -R

scripts/toc.sh

@@ -20,7 +20,7 @@ cat $f | awk '
     o{next}
     /^#/{s=1;rs=0;pr()}
     /^#* *(nix package)/{rs=1}
-    /^#* *(install on android|dev env setup|just the sfx|complete release|optional gpl stuff|nixos module)|`$/{s=rs}
+    /^#* *(themes|install on android|dev env setup|just the sfx|complete release|optional gpl stuff|nixos module)|```/{s=rs}
     /^#/{
         lv=length($1);
         sub(/[^ ]+ /,"");

tests/test_dedup.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import json
+import os
+import shutil
+import tempfile
+import unittest
+from itertools import product
+
+from copyparty.authsrv import AuthSrv
+from copyparty.httpcli import HttpCli
+from tests import util as tu
+from tests.util import Cfg
+
+
+class TestDedup(unittest.TestCase):
+    def setUp(self):
+        self.td = tu.get_ramdisk()
+
+    def tearDown(self):
+        os.chdir(tempfile.gettempdir())
+        shutil.rmtree(self.td)
+
+    def reset(self):
+        td = os.path.join(self.td, "vfs")
+        if os.path.exists(td):
+            shutil.rmtree(td)
+        os.mkdir(td)
+        os.chdir(td)
+        return td
+
+    def cinit(self):
+        if self.conn:
+            self.fstab = self.conn.hsrv.hub.up2k.fstab
+            self.conn.hsrv.hub.up2k.shutdown()
+        self.asrv = AuthSrv(self.args, self.log)
+        self.conn = tu.VHttpConn(self.args, self.asrv, self.log, b"", True)
+        if self.fstab:
+            self.conn.hsrv.hub.up2k.fstab = self.fstab
+
+    def test(self):
+        quick = True  # sufficient for regular smoketests
+        # quick = False
+
+        dirnames = ["d1", "d2"]
+        filenames = ["f1", "f2"]
+        files = [
+            (
+                "one",
+                "BfcDQQeKz2oG1CPSFyD5ZD1flTYm2IoCY23DqeeVgq6w",
+                "XMbpLRqVdtGmgggqjUI6uSoNMTqZVX4K6zr74XA1BRKc",
+            ),
+            (
+                "two",
+                "ko1Q0eJNq3zKYs_oT83Pn8aVFgonj5G1wK8itwnYL4qj",
+                "fxvihWlnQIbVbUPr--TxyV41913kPLhXPD1ngXYxDfou",
+            ),
+        ]
+        # (data, chash, wark)
+
+        self.ctr = 336 if quick else 2016  # estimated total num uploads
+        self.conn = None
+        self.fstab = None
+        for e2d in [True, False]:
+            self.args = Cfg(v=[".::A"], a=[], e2d=e2d)
+            for cm1 in product(dirnames, filenames, files):
+                for cm2 in product(dirnames, filenames, files):
+                    if cm1 == cm2:
+                        continue
+                    for cm3 in product(dirnames, filenames, files):
+                        if cm3 in (cm1, cm2):
+                            continue
+
+                        f1 = cm1[2]
+                        f2 = cm2[2]
+                        f3 = cm3[2]
+                        if not e2d:
+                            rms = [-1]
+                        elif f1 == f2:
+                            if f1 == f3:
+                                rms = [0, 1, 2]
+                            else:
+                                rms = [0, 1]
+                        elif f1 == f3:
+                            rms = [0, 2]
+                        else:
+                            rms = [1, 2]
+
+                        for rm in rms:
+                            self.do_tc(cm1, cm2, cm3, rm)
+
+                        if quick:
+                            break
+
+    def do_tc(self, cm1, cm2, cm3, irm):
+        dn1, fn1, f1 = cm1
+        dn2, fn2, f2 = cm2
+        dn3, fn3, f3 = cm3
+
+        self.reset()
+        self.cinit()
+
+        fn1 = self.do_post(dn1, fn1, f1, True)
+        fn2 = self.do_post(dn2, fn2, f2, False)
+        fn3 = self.do_post(dn3, fn3, f3, False)
+
+        if irm < 0:
+            return
+
+        cms = [(dn1, fn1, f1), (dn2, fn2, f2), (dn3, fn3, f3)]
+        rm = cms[irm]
+        dn, fn, _ = rm
+        h, b = self.curl("%s/%s?delete" % (dn, fn), meth="POST")
+        self.assertIn(" 200 OK", h)
+        self.assertIn("deleted 1 files", b)
+        h, b = self.curl("%s/%s" % (dn, fn))
+        self.assertIn(" 404 Not Fo", h)
+        for cm in cms:
+            if cm == rm:
+                continue
+            dn, fn, f = cm
+            h, b = self.curl("%s/%s" % (dn, fn))
+            self.assertEqual(b, f[0])
+
+    def do_post(self, dn, fn, fi, first):
+        print("\n\n# do_post", self.ctr, repr((dn, fn, fi, first)))
+        self.ctr -= 1
+
+        data, chash, wark = fi
+        hs = self.handshake(dn, fn, fi)
+        self.assertEqual(hs["wark"], wark)
+
+        sfn = hs["name"]
+        if sfn == fn:
+            print("using original name " + fn)
+        else:
+            print(fn + " got renamed to " + sfn)
+            if first:
+                raise Exception("wait what")
+
+        if hs["hash"]:
+            self.assertEqual(hs["hash"][0], chash)
+            self.put_chunk(dn, wark, chash, data)
+        elif first:
+            raise Exception("found first; %r, %r" % ((dn, fn, fi), hs))
+
+        h, b = self.curl("%s/%s" % (dn, sfn))
+        self.assertEqual(b, data)
+        return sfn
+
+    def handshake(self, dn, fn, fi):
+        hdr = "POST /%s/ HTTP/1.1\r\nConnection: close\r\nContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n"
+        msg = {"name": fn, "size": 3, "lmod": 1234567890, "life": 0, "hash": [fi[1]]}
+        buf = json.dumps(msg).encode("utf-8")
+        buf = (hdr % (dn, len(buf))).encode("utf-8") + buf
+        print("HS -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        print("HS <--", ret)
+        return json.loads(ret[1])
+
+    def put_chunk(self, dn, wark, chash, data):
+        msg = [
+            "POST /%s/ HTTP/1.1" % (dn,),
+            "Connection: close",
+            "Content-Type: application/octet-stream",
+            "Content-Length: 3",
+            "X-Up2k-Hash: " + chash,
+            "X-Up2k-Wark: " + wark,
+            "",
+            data,
+        ]
+        buf = "\r\n".join(msg).encode("utf-8")
+        print("PUT -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        self.assertEqual(ret[1], "thank")
+
+    def curl(self, url, binary=False, meth=None):
+        h = "%s /%s HTTP/1.1\r\nConnection: close\r\n\r\n"
+        h = h % (meth or "GET", url)
+        HttpCli(self.conn.setbuf(h.encode("utf-8"))).run()
+        if binary:
+            h, b = self.conn.s._reply.split(b"\r\n\r\n", 1)
+            return [h.decode("utf-8"), b]
+
+        return self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def log(self, src, msg, c=0):
+        print(msg)

tests/test_dots.py

@@ -24,6 +24,10 @@ def hdr(query, uname):
 
 
 class TestDots(unittest.TestCase):
+    def __init__(self, *a, **ka):
+        super(TestDots, self).__init__(*a, **ka)
+        self.is_dut = True
+
     def setUp(self):
         self.td = tu.get_ramdisk()
 

tests/test_httpcli.py

@@ -86,6 +86,7 @@ class TestHttpCli(unittest.TestCase):
 
             self.args = Cfg(v=vcfg, a=["o:o", "x:x"])
             self.asrv = AuthSrv(self.args, self.log)
+            self.conn = tu.VHttpConn(self.args, self.asrv, self.log, b"")
             vfiles = [x for x in allfiles if x.startswith(top)]
             for fp in vfiles:
                 tctr += 1
@@ -204,14 +205,14 @@ class TestHttpCli(unittest.TestCase):
         buf = "PUT /{0} HTTP/1.1\r\nCookie: cppwd=o\r\nConnection: close\r\nContent-Length: {1}\r\n\r\nok {0}\n"
         buf = buf.format(url, len(url) + 4).encode("utf-8")
         print("PUT -->", buf)
-        conn = tu.VHttpConn(self.args, self.asrv, self.log, buf)
+        conn = self.conn.setbuf(buf)
         HttpCli(conn).run()
         ret = conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
         print("PUT <--", ret)
         return ret
 
     def curl(self, url, binary=False):
-        conn = tu.VHttpConn(self.args, self.asrv, self.log, hdr(url))
+        conn = self.conn.setbuf(hdr(url))
         HttpCli(conn).run()
         if binary:
             h, b = conn.s._reply.split(b"\r\n\r\n", 1)

tests/test_mv.py

@@ -0,0 +1,198 @@
+#!/usr/bin/env python3
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import json
+import os
+import shutil
+import tempfile
+import unittest
+from itertools import product
+
+from copyparty.__init__ import PY2
+from copyparty.authsrv import AuthSrv
+from copyparty.httpcli import HttpCli
+from tests import util as tu
+from tests.util import Cfg
+
+"""
+TODO inject tags into db and verify ls
+"""
+
+
+class TestDedup(unittest.TestCase):
+    def setUp(self):
+        self.td = tu.get_ramdisk()
+
+    def tearDown(self):
+        os.chdir(tempfile.gettempdir())
+        shutil.rmtree(self.td)
+
+    def reset(self):
+        td = os.path.join(self.td, "vfs")
+        if os.path.exists(td):
+            shutil.rmtree(td)
+        os.mkdir(td)
+        os.chdir(td)
+        return td
+
+    def cinit(self):
+        if self.conn:
+            self.fstab = self.conn.hsrv.hub.up2k.fstab
+            self.conn.hsrv.hub.up2k.shutdown()
+        self.asrv = AuthSrv(self.args, self.log)
+        self.conn = tu.VHttpConn(self.args, self.asrv, self.log, b"", True)
+        if self.fstab:
+            self.conn.hsrv.hub.up2k.fstab = self.fstab
+
+    def test(self):
+        if PY2:
+            raise unittest.SkipTest()
+
+        # tc_e2d = [True, False]  # maybe-TODO only known symlinks are translated
+        tc_e2d = [True]
+        tc_dedup = ["sym", "no", "sym-no"]
+        tc_vols = [["::A"], ["::A", "d1:d1:A"]]
+        dirs = ["d1", "d1/d2", "d1/d2/d3", "d1/d4"]
+        files = [
+            (
+                "one",
+                "BfcDQQeKz2oG1CPSFyD5ZD1flTYm2IoCY23DqeeVgq6w",
+                "XMbpLRqVdtGmgggqjUI6uSoNMTqZVX4K6zr74XA1BRKc",
+            )
+        ]
+        # (data, chash, wark)
+
+        self.conn = None
+        self.fstab = None
+        self.ctr = 0  # 2304
+        tcgen = product(tc_e2d, tc_dedup, tc_vols, dirs, ["d9", "../d9"])
+        for e2d, dedup, vols, mv_from, dst in tcgen:
+            if "/" not in mv_from and dst.startswith(".."):
+                continue  # would move past top of fs
+            if len(vols) > 1 and mv_from == "d1":
+                continue  # cannot move a vol
+
+            # print(e2d, dedup, vols, mv_from, dst)
+            ka = {"e2d": e2d}
+            if dedup == "hard":
+                ka["hardlink"] = True
+            elif dedup == "no":
+                ka["no_dedup"] = True
+            self.args = Cfg(v=vols[:], a=[], **ka)
+
+            for u1, u2, u3, u4 in product(dirs, dirs, dirs, dirs):
+                ups = (u1, u2, u3, u4)
+                if len(set(ups)) < 4:
+                    continue  # not unique
+
+                t = "e2d:%s dedup:%s vols:%d from:%s to:%s"
+                t = t % (e2d, dedup, len(vols), mv_from, dst)
+                print("\n\n\033[0;7m# files:", ups, t, "\033[0m")
+
+                self.reset()
+                self.cinit()
+
+                for up in [u1, u2, u3, u4]:
+                    self.do_post(up, "fn", files[0], up == u1)
+
+                restore_args = None
+                if dedup == "sym-no":
+                    restore_args = self.args
+                    ka = {"e2d": e2d, "no_dedup": True}
+                    self.args = Cfg(v=vols[:], a=[], **ka)
+                    self.cinit()
+
+                mv_to = mv_from
+                for _ in range(2 if dst.startswith("../") else 1):
+                    mv_to = mv_from.rsplit("/", 1)[0] if "/" in mv_from else ""
+                mv_to += "/" + dst.lstrip("./")
+
+                self.do_mv(mv_from, mv_to)
+
+                for dirpath in [u1, u2, u3, u4]:
+                    if dirpath == mv_from:
+                        dirpath = mv_to
+                    elif dirpath.startswith(mv_from):
+                        dirpath = mv_to + dirpath[len(mv_from) :]
+                    h, b = self.curl(dirpath + "/fn")
+                    self.assertEqual(b, "one")
+
+                if restore_args:
+                    self.args = restore_args
+
+    def do_mv(self, src, dst):
+        hdr = "POST /%s?move=/%s HTTP/1.1\r\nConnection: close\r\nContent-Length: 0\r\n\r\n"
+        buf = (hdr % (src, dst)).encode("utf-8")
+        print("MV [%s] => [%s]" % (src, dst))
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        print("MV <-- ", ret)
+        self.assertIn(" 201 Created", ret[0])
+        self.assertEqual("k\r\n", ret[1])
+        return ret
+
+    def do_post(self, dn, fn, fi, first):
+        print("\n# do_post", self.ctr, repr((dn, fn, fi, first)))
+        self.ctr -= 1
+
+        data, chash, wark = fi
+        hs = self.handshake(dn, fn, fi)
+        self.assertEqual(hs["wark"], wark)
+
+        sfn = hs["name"]
+        if sfn == fn:
+            print("using original name " + fn)
+        else:
+            print(fn + " got renamed to " + sfn)
+            if first:
+                raise Exception("wait what")
+
+        if hs["hash"]:
+            self.assertEqual(hs["hash"][0], chash)
+            self.put_chunk(dn, wark, chash, data)
+        elif first:
+            raise Exception("found first; %r, %r" % ((dn, fn, fi), hs))
+
+        h, b = self.curl("%s/%s" % (dn, sfn))
+        self.assertEqual(b, data)
+
+    def handshake(self, dn, fn, fi):
+        hdr = "POST /%s/ HTTP/1.1\r\nConnection: close\r\nContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n"
+        msg = {"name": fn, "size": 3, "lmod": 1234567890, "life": 0, "hash": [fi[1]]}
+        buf = json.dumps(msg).encode("utf-8")
+        buf = (hdr % (dn, len(buf))).encode("utf-8") + buf
+        print("HS -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        print("HS <--", ret)
+        return json.loads(ret[1])
+
+    def put_chunk(self, dn, wark, chash, data):
+        msg = [
+            "POST /%s/ HTTP/1.1" % (dn,),
+            "Connection: close",
+            "Content-Type: application/octet-stream",
+            "Content-Length: 3",
+            "X-Up2k-Hash: " + chash,
+            "X-Up2k-Wark: " + wark,
+            "",
+            data,
+        ]
+        buf = "\r\n".join(msg).encode("utf-8")
+        print("PUT -->", buf)
+        HttpCli(self.conn.setbuf(buf)).run()
+        ret = self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+        self.assertEqual(ret[1], "thank")
+
+    def curl(self, url, binary=False):
+        h = "GET /%s HTTP/1.1\r\nConnection: close\r\n\r\n"
+        HttpCli(self.conn.setbuf((h % (url,)).encode("utf-8"))).run()
+        if binary:
+            h, b = self.conn.s._reply.split(b"\r\n\r\n", 1)
+            return [h.decode("utf-8"), b]
+
+        return self.conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def log(self, src, msg, c=0):
+        print(msg)

tests/util.py

@@ -3,7 +3,6 @@
 from __future__ import print_function, unicode_literals
 
 import os
-import platform
 import re
 import shutil
 import socket
@@ -16,9 +15,7 @@ from argparse import Namespace
 
 import jinja2
 
-WINDOWS = platform.system() == "Windows"
-ANYWIN = WINDOWS or sys.platform in ["msys"]
-MACOS = platform.system() == "Darwin"
+from copyparty.__init__ import MACOS, WINDOWS, E
 
 J2_ENV = jinja2.Environment(loader=jinja2.BaseLoader)  # type: ignore
 J2_FILES = J2_ENV.from_string("{{ files|join('\n') }}\nJ2EOT")
@@ -42,10 +39,11 @@ if MACOS:
     # 25% faster; until any tests do symlink stuff
 
 
-from copyparty.__init__ import E
 from copyparty.__main__ import init_E
+from copyparty.broker_thr import BrokerThr
 from copyparty.ico import Ico
 from copyparty.u2idx import U2idx
+from copyparty.up2k import Up2k
 from copyparty.util import FHC, CachedDict, Garda, Unrecv
 
 init_E(E)
@@ -119,16 +117,16 @@ class Cfg(Namespace):
     def __init__(self, a=None, v=None, c=None, **ka0):
         ka = {}
 
-        ex = "chpw daw dav_auth dav_inf dav_mac dav_rt e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp early_ban ed emp exp force_js getmod grid gsel hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_lifetime no_logues no_mv no_pipe no_poll no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw og og_no_head og_s_title q rand smb srch_dbg stats uqe vague_403 vc ver write_uplog xdev xlink xvol"
+        ex = "chpw daw dav_auth dav_inf dav_mac dav_rt e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp early_ban ed emp exp force_js getmod grid gsel hardlink ih ihead magic hardlink_only nid nih no_acode no_athumb no_dav no_db_ip no_del no_dupe no_lifetime no_logues no_mv no_pipe no_poll no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw og og_no_head og_s_title q rand smb srch_dbg stats uqe vague_403 vc ver write_uplog xdev xlink xvol zs"
         ka.update(**{k: False for k in ex.split()})
 
-        ex = "dotpart dotsrch hook_v no_dhash no_fastboot no_rescan no_sendfile no_snap no_voldump re_dhash plain_ip"
+        ex = "dedup dotpart dotsrch hook_v no_dhash no_fastboot no_fpool no_htp no_rescan no_sendfile no_snap no_voldump re_dhash plain_ip"
         ka.update(**{k: True for k in ex.split()})
 
         ex = "ah_cli ah_gen css_browser hist js_browser js_other mime mimes no_forget no_hash no_idx nonsus_urls og_tpl og_ua"
         ka.update(**{k: None for k in ex.split()})
 
-        ex = "hash_mt srch_time u2abort u2j u2sz"
+        ex = "hash_mt safe_dedup srch_time u2abort u2j u2sz"
         ka.update(**{k: 1 for k in ex.split()})
 
         ex = "au_vol mtab_age reg_cap s_thead s_tbody th_convt"
@@ -137,9 +135,12 @@ class Cfg(Namespace):
         ex = "db_act k304 loris re_maxage rproxy rsp_jtr rsp_slp s_wr_slp snap_wri theme themes turbo"
         ka.update(**{k: 0 for k in ex.split()})
 
-        ex = "ah_alg bname chpw_db doctitle df exit favico idp_h_usr html_head lg_sbf log_fk md_sbf name og_desc og_site og_th og_title og_title_a og_title_v og_title_i shr tcolor textfiles unlist vname R RS SR"
+        ex = "ah_alg bname chpw_db doctitle df exit favico idp_h_usr ipa html_head lg_sbf log_fk md_sbf name og_desc og_site og_th og_title og_title_a og_title_v og_title_i shr tcolor textfiles unlist vname xff_src R RS SR"
         ka.update(**{k: "" for k in ex.split()})
 
+        ex = "ban_403 ban_404 ban_422 ban_pw ban_url"
+        ka.update(**{k: "no" for k in ex.split()})
+
         ex = "grp on403 on404 xad xar xau xban xbd xbr xbu xiu xm"
         ka.update(**{k: [] for k in ex.split()})
 
@@ -221,11 +222,29 @@ class VSock(object):
         pass
 
 
+class VHub(object):
+    def __init__(self, args, asrv, log):
+        self.args = args
+        self.asrv = asrv
+        self.log = log
+        self.is_dut = True
+        self.up2k = Up2k(self)
+
+
+class VBrokerThr(BrokerThr):
+    def __init__(self, hub):
+        self.hub = hub
+        self.log = hub.log
+        self.args = hub.args
+        self.asrv = hub.asrv
+
+
 class VHttpSrv(object):
     def __init__(self, args, asrv, log):
         self.args = args
         self.asrv = asrv
         self.log = log
+        self.hub = None
 
         self.broker = NullBroker(args, asrv)
         self.prism = None
@@ -252,18 +271,25 @@ class VHttpSrv(object):
         return self.u2idx
 
 
+class VHttpSrvUp2k(VHttpSrv):
+    def __init__(self, args, asrv, log):
+        super(VHttpSrvUp2k, self).__init__(args, asrv, log)
+        self.hub = VHub(args, asrv, log)
+        self.broker = VBrokerThr(self.hub)
+
+
 class VHttpConn(object):
-    def __init__(self, args, asrv, log, buf):
+    def __init__(self, args, asrv, log, buf, use_up2k=False):
         self.t0 = time.time()
-        self.s = VSock(buf)
-        self.sr = Unrecv(self.s, None)  # type: ignore
         self.aclose = {}
         self.addr = ("127.0.0.1", "42069")
         self.args = args
         self.asrv = asrv
         self.bans = {}
         self.freshen_pwd = 0.0
-        self.hsrv = VHttpSrv(args, asrv, log)
+
+        Ctor = VHttpSrvUp2k if use_up2k else VHttpSrv
+        self.hsrv = Ctor(args, asrv, log)
         self.ico = Ico(args)
         self.ipa_nm = None
         self.lf_url = None
@@ -279,6 +305,12 @@ class VHttpConn(object):
         self.u2fh = FHC()
 
         self.get_u2idx = self.hsrv.get_u2idx
+        self.setbuf(buf)
+
+    def setbuf(self, buf):
+        self.s = VSock(buf)
+        self.sr = Unrecv(self.s, None)  # type: ignore
+        return self
 
 
 if WINDOWS: