v1.9.17

webdav clients tend to upload and then immediately delete
files to test for write-access and available disk space,
so don't crash and burn when that happens

.vscode/launch.json

@@ -9,6 +9,10 @@
             "console": "integratedTerminal",
             "cwd": "${workspaceFolder}",
             "justMyCode": false,
+            "env": {
+                "PYDEVD_DISABLE_FILE_VALIDATION": "1",
+                "PYTHONWARNINGS": "always", //error
+            },
             "args": [
                 //"-nw",
                 "-ed",

.vscode/launch.py

@@ -41,7 +41,7 @@ if sfx:
     argv = [sys.executable, sfx] + argv
     sp.check_call(argv)
 elif re.search(" -j ?[0-9]", " ".join(argv)):
-    argv = [sys.executable, "-m", "copyparty"] + argv
+    argv = [sys.executable, "-Wa", "-m", "copyparty"] + argv
     sp.check_call(argv)
 else:
     sys.path.insert(0, os.getcwd())

.vscode/tasks.json

@@ -11,6 +11,7 @@
             "type": "shell",
             "command": "${config:python.pythonPath}",
             "args": [
+                "-Wa", //-We
                 ".vscode/launch.py"
             ]
         }

CONTRIBUTING.md

@@ -1,3 +1,43 @@
 * do something cool
 
-really tho, send a PR or an issue or whatever, all appreciated, anything goes, just behave aight
+really tho, send a PR or an issue or whatever, all appreciated, anything goes, just behave aight 👍👍
+
+but to be more specific,
+
+
+# contribution ideas
+
+
+## documentation
+
+I think we can agree that the documentation leaves a LOT to be desired. I've realized I'm not exactly qualified for this 😅 but maybe the [soon-to-come setup GUI](https://github.com/9001/copyparty/issues/57) will make this more manageable. The best documentation is the one that never had to be written, right? :> so I suppose we can give this a wait-and-see approach for a bit longer.
+
+
+## crazy ideas & features
+
+assuming they won't cause too much problems or side-effects :> 
+
+i think someone was working on a way to list directories over DNS for example...
+
+if you wanna have a go at coding it up yourself then maybe mention the idea on discord before you get too far, otherwise just go nuts 👍
+
+
+## others
+
+aside from documentation and ideas, some other things that would be cool to have some help with is:
+
+* **translations** -- the copyparty web-UI has translations for english and norwegian at the top of [browser.js](https://github.com/9001/copyparty/blob/hovudstraum/copyparty/web/browser.js); if you'd like to add a translation for another language then that'd be welcome! and if that language has a grammar that doesn't fit into the way the strings are assembled, then we'll fix that as we go :>
+
+* **UI ideas** -- at some point I was thinking of rewriting the UI in react/preact/something-not-vanilla-javascript, but I'll admit the comfiness of not having any build stage combined with raw performance has kinda convinced me otherwise :p but I'd be very open to ideas on how the UI could be improved, or be more intuitive.
+
+* **docker improvements** -- I don't really know what I'm doing when it comes to containers, so I'm sure there's a *huge* room for improvement here, mainly regarding how you're supposed to use the container with kubernetes / docker-compose / any of the other popular ways to do things. At some point I swear I'll start learning about docker so I can pick up clach04's [docker-compose draft](https://github.com/9001/copyparty/issues/38) and learn how that stuff ticks, unless someone beats me to it!
+
+* **packaging** for various linux distributions -- this could either be as simple as just plopping the sfx.py in the right place and calling that from systemd (the archlinux package [originally did this](https://github.com/9001/copyparty/pull/18)); maybe with a small config-file which would cause copyparty to load settings from `/etc/copyparty.d` (like the [archlinux package](https://github.com/9001/copyparty/tree/hovudstraum/contrib/package/arch) does with `copyparty.conf`), or it could be a proper installation of the copyparty python package into /usr/lib or similar (the archlinux package [eventually went for this approach](https://github.com/9001/copyparty/pull/26))
+
+  * [fpm](https://github.com/jordansissel/fpm) can probably help with the technical part of it, but someone needs to handle distro relations :-)
+
+* **software integration** -- I'm sure there's a lot of usecases where copyparty could complement something else, or the other way around, so any ideas or any work in this regard would be dope. This doesn't necessarily have to be code inside copyparty itself;
+
+  * [hooks](https://github.com/9001/copyparty/tree/hovudstraum/bin/hooks) -- these are small programs which are called by copyparty when certain things happen (files are uploaded, someone hits a 404, etc.), and could be a fun way to add support for more usecases
+
+  * [parser plugins](https://github.com/9001/copyparty/tree/hovudstraum/bin/mtag) -- if you want to have copyparty analyze and index metadata for some oddball file-formats, then additional plugins would be neat :>

README.md

@@ -20,9 +20,8 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [testimonials](#testimonials) - small collection of user feedback
 * [motivations](#motivations) - project goals / philosophy
     * [notes](#notes) - general notes
-* [bugs](#bugs)
-    * [general bugs](#general-bugs)
-    * [not my bugs](#not-my-bugs)
+* [bugs](#bugs) - roughly sorted by chance of encounter
+    * [not my bugs](#not-my-bugs) - same order here too
 * [breaking changes](#breaking-changes) - upgrade notes
 * [FAQ](#FAQ) - "frequently" asked questions
 * [accounts and volumes](#accounts-and-volumes) - per-folder, per-user permissions
@@ -40,7 +39,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [file manager](#file-manager) - cut/paste, rename, and delete files/folders (if you have permission)
     * [batch rename](#batch-rename) - select some files and press `F2` to bring up the rename UI
     * [media player](#media-player) - plays almost every audio format there is
-        * [audio equalizer](#audio-equalizer) - bass boosted
+        * [audio equalizer](#audio-equalizer) - and [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression)
         * [fix unreliable playback on android](#fix-unreliable-playback-on-android) - due to phone / app settings
     * [markdown viewer](#markdown-viewer) - and there are *two* editors
     * [other tricks](#other-tricks)
@@ -54,6 +53,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [webdav server](#webdav-server) - with read-write support
         * [connecting to webdav from windows](#connecting-to-webdav-from-windows) - using the GUI
     * [smb server](#smb-server) - unsafe, slow, not recommended for wan
+    * [browser ux](#browser-ux) - tweaking the ui
     * [file indexing](#file-indexing) - enables dedup and music search ++
         * [exclude-patterns](#exclude-patterns) - to save some time
         * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
@@ -263,20 +263,28 @@ server notes:
 
 # bugs
 
-* Windows: python 2.7 cannot index non-ascii filenames with `-e2d`
-* Windows: python 2.7 cannot handle filenames with mojibake
-* `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions (macos, some linux)
-* `--th-ff-swr` may fix audio thumbnails on some FFmpeg versions
+roughly sorted by chance of encounter
 
-## general bugs
+* general:
+  * `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions (macos, some linux)
+  * `--th-ff-swr` may fix audio thumbnails on some FFmpeg versions
+  * if the `up2k.db` (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
+    * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
+  * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
+  * probably more, pls let me know
 
-* Windows: if the `up2k.db` (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
-  * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
-* all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
-* probably more, pls let me know
+* python 3.4 and older (including 2.7):
+  * many rare and exciting edge-cases because [python didn't handle EINTR yet](https://peps.python.org/pep-0475/)
+    * downloads from copyparty may suddenly fail, but uploads *should* be fine
+
+* python 2.7 on Windows:
+  * cannot index non-ascii filenames with `-e2d`
+  * cannot handle filenames with mojibake
 
 ## not my bugs
 
+same order here too
+
 * [Chrome issue 1317069](https://bugs.chromium.org/p/chromium/issues/detail?id=1317069) -- if you try to upload a folder which contains symlinks by dragging it into the browser, the symlinked files will not get uploaded
 
 * [Chrome issue 1352210](https://bugs.chromium.org/p/chromium/issues/detail?id=1352210) -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive)
@@ -310,6 +318,8 @@ server notes:
 
 upgrade notes
 
+* `1.9.16` (2023-11-04):
+  * `--stats`/prometheus: `cpp_bans` renamed to `cpp_active_bans`, and that + `cpp_uptime` are gauges
 * `1.6.0` (2023-01-29):
   * http-api: delete/move is now `POST` instead of `GET`
   * everything other than `GET` and `HEAD` must pass [cors validation](#cors)
@@ -359,7 +369,7 @@ permissions:
 * `g` (get): only download files, cannot see folder contents or zip/tar
 * `G` (upget): same as `g` except uploaders get to see their own [filekeys](#filekeys) (see `fk` in examples below)
 * `h` (html): same as `g` except folders return their index.html, and filekeys are not necessary for index.html
-* `a` (admin): can see uploader IPs, config-reload
+* `a` (admin): can see upload time, uploader IPs, config-reload
 
 examples:
 * add accounts named u1, u2, u3 with passwords p1, p2, p3: `-a u1:p1 -a u2:p2 -a u3:p3`
@@ -736,7 +746,7 @@ open the `[🎺]` media-player-settings tab to configure it,
 
 ### audio equalizer
 
-bass boosted
+and [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression)
 
 can also boost the volume in general, or increase/decrease stereo width (like [crossfeed](https://www.foobar2000.org/components/view/foo_dsp_meiercf) just worse)
 
@@ -782,6 +792,8 @@ other notes,
 
 * files named `README.md` / `readme.md` will be rendered after directory listings unless `--no-readme` (but `.epilogue.html` takes precedence)
 
+* `README.md` and `*logue.html` can contain placeholder values which are replaced server-side before embedding into directory listings; see `--help-exp`
+
 
 ## searching
 
@@ -947,6 +959,16 @@ authenticate with one of the following:
 * username `$password`, password `k`
 
 
+## browser ux
+
+tweaking the ui
+
+* set default sort order globally with `--sort` or per-volume with the `sort` volflag; specify one or more comma-separated columns to sort by, and prefix the column name with `-` for reverse sort
+  * the column names you can use are visible as tooltips when hovering over the column headers in the directory listing, for example `href ext sz ts tags/.up_at tags/Cirle tags/.tn tags/Artist tags/Title`
+  * to sort in music order (album, track, artist, title) with filename as fallback, you could `--sort tags/Cirle,tags/.tn,tags/Artist,tags/Title,href`
+  * to sort by upload date, first enable showing the upload date in the listing with `-e2d -mte +.up_at` and then `--sort tags/.up_at`
+
+
 ## file indexing
 
 enables dedup and music search ++
@@ -973,6 +995,7 @@ the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`,
 * `-v ~/music::r:c,d2ts` same except only affecting tags
 
 note:
+* upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
 * deduplication is possible on windows if you run copyparty as administrator (not saying you should!)
@@ -1284,8 +1307,23 @@ scrape_configs:
 ```
 
 currently the following metrics are available,
-* `cpp_uptime_seconds`
-* `cpp_bans` number of banned IPs
+* `cpp_uptime_seconds` time since last copyparty restart
+* `cpp_boot_unixtime_seconds` same but as an absolute timestamp
+* `cpp_http_conns` number of open http(s) connections
+* `cpp_http_reqs` number of http(s) requests handled
+* `cpp_sus_reqs` number of 403/422/malicious requests
+* `cpp_active_bans` number of currently banned IPs
+* `cpp_total_bans` number of IPs banned since last restart
+
+these are available unless `--nos-vst` is specified:
+* `cpp_db_idle_seconds` time since last database activity (upload/rename/delete)
+* `cpp_db_act_seconds` same but as an absolute timestamp
+* `cpp_idle_vols` number of volumes which are idle / ready
+* `cpp_busy_vols` number of volumes which are busy / indexing
+* `cpp_offline_vols` number of volumes which are offline / unavailable
+* `cpp_hashing_files` number of files queued for hashing / indexing
+* `cpp_tagq_files` number of files queued for metadata scanning
+* `cpp_mtpq_files` number of files queued for plugin-based analysis
 
 and these are available per-volume only:
 * `cpp_disk_size_bytes` total HDD size
@@ -1304,9 +1342,12 @@ some of the metrics have additional requirements to function correctly,
 the following options are available to disable some of the metrics:
 * `--nos-hdd` disables `cpp_disk_*` which can prevent spinning up HDDs
 * `--nos-vol` disables `cpp_vol_*` which reduces server startup time
+* `--nos-vst` disables volume state, reducing the worst-case prometheus query time by 0.5 sec
 * `--nos-dup` disables `cpp_dupe_*` which reduces the server load caused by prometheus queries
 * `--nos-unf` disables `cpp_unf_*` for no particular purpose
 
+note: the following metrics are counted incorrectly if multiprocessing is enabled with `-j`: `cpp_http_conns`, `cpp_http_reqs`, `cpp_sus_reqs`, `cpp_active_bans`, `cpp_total_bans`
+
 
 # packages
 
@@ -1636,8 +1677,6 @@ safety profiles:
   * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
     * however note if you edit one file it will also affect the other copies
   * `--vague-403` returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
-  * `--ban-404=50,60,1440` ban client for 1440min (24h) if they hit 50 404's in 60min
-    * `--turbo=-1` to force-disable turbo-mode in the uploader which could otherwise hit the 404-ban
   * `--nih` removes the server hostname from directory listings
 
 * option `-sss` is a shortcut for the above plus:

bin/hooks/msg-log.py

@@ -7,7 +7,11 @@ import json
 import os
 import sys
 import time
-from datetime import datetime
+
+try:
+    from datetime import datetime, timezone
+except:
+    from datetime import datetime
 
 
 """
@@ -96,7 +100,11 @@ def main(argv=None):
     msg_info = json.loads(sys.argv[1])
     # print(msg_info)
 
-    dt = datetime.utcfromtimestamp(msg_info["at"])
+    try:
+        dt = datetime.fromtimestamp(msg_info["at"], timezone.utc)
+    except:
+        dt = datetime.utcfromtimestamp(msg_info["at"])
+
     msg_info["datetime"] = dt.strftime("%Y-%m-%d, %H:%M:%S")
 
     msg_text = TEMPLATE % msg_info

bin/hooks/notify2.py

@@ -4,7 +4,7 @@ import json
 import os
 import sys
 import subprocess as sp
-from datetime import datetime
+from datetime import datetime, timezone
 from plyer import notification
 
 
@@ -43,7 +43,8 @@ def main():
     fp = inf["ap"]
     sz = humansize(inf["sz"])
     dp, fn = os.path.split(fp)
-    mt = datetime.utcfromtimestamp(inf["mt"]).strftime("%Y-%m-%d %H:%M:%S")
+    dt = datetime.fromtimestamp(inf["mt"], timezone.utc)
+    mt = dt.strftime("%Y-%m-%d %H:%M:%S")
 
     msg = f"{fn} ({sz})\n📁 {dp}"
     title = "File received"

bin/hooks/xiu-sha.py

@@ -3,7 +3,7 @@
 import hashlib
 import json
 import sys
-from datetime import datetime
+from datetime import datetime, timezone
 
 
 _ = r"""
@@ -43,8 +43,11 @@ except:
         return p
 
 
+UTC = timezone.utc
+
+
 def humantime(ts):
-    return datetime.utcfromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
+    return datetime.fromtimestamp(ts, UTC).strftime("%Y-%m-%d %H:%M:%S")
 
 
 def find_files_root(inf):
@@ -96,7 +99,7 @@ def main():
 
     ret.append("# {} files, {} bytes total".format(len(inf), total_sz))
     ret.append("")
-    ftime = datetime.utcnow().strftime("%Y-%m%d-%H%M%S.%f")
+    ftime = datetime.now(UTC).strftime("%Y-%m%d-%H%M%S.%f")
     fp = "{}xfer-{}.sha512".format(inf[0]["ap"][:di], ftime)
     with open(fsenc(fp), "wb") as f:
         f.write("\n".join(ret).encode("utf-8", "replace"))

bin/partyfuse.py

@@ -46,12 +46,13 @@ import traceback
 import http.client  # py2: httplib
 import urllib.parse
 import calendar
-from datetime import datetime
+from datetime import datetime, timezone
 from urllib.parse import quote_from_bytes as quote
 from urllib.parse import unquote_to_bytes as unquote
 
 WINDOWS = sys.platform == "win32"
 MACOS = platform.system() == "Darwin"
+UTC = timezone.utc
 info = log = dbg = None
 
 
@@ -176,7 +177,7 @@ class RecentLog(object):
     def put(self, msg):
         msg = "{:10.6f} {} {}\n".format(time.time() % 900, rice_tid(), msg)
         if self.f:
-            fmsg = " ".join([datetime.utcnow().strftime("%H%M%S.%f"), str(msg)])
+            fmsg = " ".join([datetime.now(UTC).strftime("%H%M%S.%f"), str(msg)])
             self.f.write(fmsg.encode("utf-8"))
 
         with self.mtx:

bin/partyjournal.py

@@ -20,12 +20,13 @@ import sys
 import base64
 import sqlite3
 import argparse
-from datetime import datetime
+from datetime import datetime, timezone
 from urllib.parse import quote_from_bytes as quote
 from urllib.parse import unquote_to_bytes as unquote
 
 
 FS_ENCODING = sys.getfilesystemencoding()
+UTC = timezone.utc
 
 
 class APF(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
@@ -155,11 +156,10 @@ th {
                 link = txt.decode("utf-8")[4:]
 
         sz = "{:,}".format(sz)
+        dt = datetime.fromtimestamp(at if at > 0 else mt, UTC)
         v = [
             w[:16],
-            datetime.utcfromtimestamp(at if at > 0 else mt).strftime(
-                "%Y-%m-%d %H:%M:%S"
-            ),
+            dt.strftime("%Y-%m-%d %H:%M:%S"),
             sz,
             imap.get(ip, ip),
         ]

bin/u2c.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 from __future__ import print_function, unicode_literals
 
-S_VERSION = "1.10"
-S_BUILD_DT = "2023-08-15"
+S_VERSION = "1.11"
+S_BUILD_DT = "2023-11-11"
 
 """
 u2c.py: upload to copyparty
@@ -107,10 +107,12 @@ class File(object):
         self.ucids = []  # type: list[str]  # chunks which need to be uploaded
         self.wark = None  # type: str
         self.url = None  # type: str
+        self.nhs = 0
 
         # set by upload
         self.up_b = 0  # type: int
         self.up_c = 0  # type: int
+        self.cd = 0
 
         # t = "size({}) lmod({}) top({}) rel({}) abs({}) name({})\n"
         # eprint(t.format(self.size, self.lmod, self.top, self.rel, self.abs, self.name))
@@ -433,7 +435,7 @@ def walkdirs(err, tops, excl):
         za = [x.replace(b"/", b"\\") for x in za]
         tops = za
 
-    ptn = re.compile(excl.encode("utf-8") or b"\n")
+    ptn = re.compile(excl.encode("utf-8") or b"\n", re.I)
 
     for top in tops:
         isdir = os.path.isdir(top)
@@ -598,7 +600,7 @@ def handshake(ar, file, search):
                 raise
 
             eprint("handshake failed, retrying: {0}\n  {1}\n\n".format(file.name, em))
-            time.sleep(1)
+            time.sleep(ar.cd)
 
     try:
         r = r.json()
@@ -689,6 +691,7 @@ class Ctl(object):
 
     def __init__(self, ar, stats=None):
         self.ok = False
+        self.errs = 0
         self.ar = ar
         self.stats = stats or self._scan()
         if not self.stats:
@@ -736,7 +739,7 @@ class Ctl(object):
 
             self._fancy()
 
-        self.ok = True
+        self.ok = not self.errs
 
     def _safe(self):
         """minimal basic slow boring fallback codepath"""
@@ -961,13 +964,22 @@ class Ctl(object):
                 self.q_upload.put(None)
                 break
 
-            with self.mutex:
-                self.handshaker_busy += 1
-
             upath = file.abs.decode("utf-8", "replace")
             if not VT100:
                 upath = upath.lstrip("\\?")
 
+            file.nhs += 1
+            if file.nhs > 32:
+                print("ERROR: giving up on file %s" % (upath))
+                self.errs += 1
+                continue
+
+            with self.mutex:
+                self.handshaker_busy += 1
+
+            while time.time() < file.cd:
+                time.sleep(0.1)
+
             hs, sprs = handshake(self.ar, file, search)
             if search:
                 if hs:
@@ -1050,6 +1062,7 @@ class Ctl(object):
             except Exception as ex:
                 t = "upload failed, retrying: {0} #{1} ({2})\n"
                 eprint(t.format(file.name, cid[:8], ex))
+                file.cd = time.time() + self.ar.cd
                 # handshake will fix it
 
             with self.mutex:
@@ -1121,6 +1134,7 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("-J", type=int, metavar="THREADS", default=hcores, help="num cpu-cores to use for hashing; set 0 or 1 for single-core hashing")
     ap.add_argument("-nh", action="store_true", help="disable hashing while uploading")
     ap.add_argument("-ns", action="store_true", help="no status panel (for slow consoles and macos)")
+    ap.add_argument("--cd", type=float, metavar="SEC", default=5, help="delay before reattempting a failed handshake/upload")
     ap.add_argument("--safe", action="store_true", help="use simple fallback approach")
     ap.add_argument("-z", action="store_true", help="ZOOMIN' (skip uploading files if they exist at the destination with the ~same last-modified timestamp, so same as yolo / turbo with date-chk but even faster)")
 
@@ -1187,6 +1201,9 @@ source file/folder selection uses rsync syntax, meaning that:
         ar.z = True
         ctl = Ctl(ar, ctl.stats)
 
+    if ctl.errs:
+        print("WARNING: %d errors" % (ctl.errs))
+
     sys.exit(0 if ctl.ok else 1)
 
 

contrib/nginx/copyparty.conf

@@ -13,7 +13,7 @@
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 
 upstream cpp {
-	server 127.0.0.1:3923;
+	server 127.0.0.1:3923 fail_timeout=1s;
 	keepalive 1;
 }
 server {

contrib/package/arch/PKGBUILD

@@ -1,14 +1,15 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.9.11"
+pkgver="1.9.16"
 pkgrel=1
-pkgdesc="Portable file sharing hub"
+pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
 url="https://github.com/9001/${pkgname}"
 license=('MIT')
 depends=("python" "lsof" "python-jinja")
 makedepends=("python-wheel" "python-setuptools" "python-build" "python-installer" "make" "pigz")
 optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tags"
+            "cfssl: generate TLS certificates on startup (pointless when reverse-proxied)"
             "python-mutagen: music tags (alternative)" 
             "python-pillow: thumbnails for images" 
             "python-pyvips: thumbnails for images (higher quality, faster, uses more ram)" 
@@ -20,7 +21,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("67d08a0ca8426a068e6461243612dda347e42f056b6312348f5bfa109de2688f")
+sha256sums=("94c9d9991ca53cfbae628d09d4139a93c54b32a1776e9aaaee99a1327d0449cb")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/default.nix

@@ -3,6 +3,9 @@
 # use argon2id-hashed passwords in config files (sha2 is always available)
 withHashedPasswords ? true,
 
+# generate TLS certificates on startup (pointless when reverse-proxied)
+withCertgen ? false,
+
 # create thumbnails with Pillow; faster than FFmpeg / MediaProcessing
 withThumbnails ? true,
 
@@ -34,6 +37,7 @@ let
     ]
     ++ lib.optional withSMB impacket
     ++ lib.optional withFTPS pyopenssl
+    ++ lib.optional withCertgen cfssl
     ++ lib.optional withThumbnails pillow
     ++ lib.optional withFastThumbnails pyvips
     ++ lib.optional withMediaProcessing ffmpeg

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.9.11/copyparty-sfx.py",
-    "version": "1.9.11",
-    "hash": "sha256-A3nQ/b3KLDQVuioXYf3kgcrXSBOEt+KBCq2tnc5PDWg="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.9.16/copyparty-sfx.py",
+    "version": "1.9.16",
+    "hash": "sha256-eq9PKzW0GCWDgU+CmK5/biQxxPy8YGZvNd+ZlShSlqg="
 }

copyparty/__main__.py

@@ -27,6 +27,9 @@ from .authsrv import expand_config_file, re_vol, split_cfg_ln, upgrade_cfg_fmt
 from .cfg import flagcats, onedash
 from .svchub import SvcHub
 from .util import (
+    DEF_EXP,
+    DEF_MTE,
+    DEF_MTH,
     IMPLICATIONS,
     JINJA_VER,
     PYFTPD_VER,
@@ -184,7 +187,10 @@ def init_E(E: EnvParams) -> None:
 
         with open_binary("copyparty", "z.tar") as tgz:
             with tarfile.open(fileobj=tgz) as tf:
-                tf.extractall(tdn)  # nosec (archive is safe)
+                try:
+                    tf.extractall(tdn, filter="tar")
+                except TypeError:
+                    tf.extractall(tdn)  # nosec (archive is safe)
 
         return tdn
 
@@ -641,6 +647,47 @@ def get_sects():
             """
             ),
         ],
+        [
+            "exp",
+            "text expansion",
+            dedent(
+                """
+            specify --exp or the "exp" volflag to enable placeholder expansions
+            in README.md / .prologue.html / .epilogue.html
+
+            --exp-md (volflag exp_md) holds the list of placeholders which can be
+            expanded in READMEs, and --exp-lg (volflag exp_lg) likewise for logues;
+            any placeholder not given in those lists will be ignored and shown as-is
+
+            the default list will expand the following placeholders:
+            \033[36m{{self.ip}}     \033[35mclient ip
+            \033[36m{{self.ua}}     \033[35mclient user-agent
+            \033[36m{{self.uname}}  \033[35mclient username
+            \033[36m{{self.host}}   \033[35mthe "Host" header, or the server's external IP otherwise
+            \033[36m{{cfg.name}}    \033[35mthe --name global-config
+            \033[36m{{cfg.logout}}  \033[35mthe --logout global-config
+            \033[36m{{vf.scan}}     \033[35mthe "scan" volflag
+            \033[36m{{vf.thsize}}   \033[35mthumbnail size
+            \033[36m{{srv.itime}}   \033[35mserver time in seconds
+            \033[36m{{srv.htime}}   \033[35mserver time as YY-mm-dd, HH:MM:SS (UTC)
+            \033[36m{{hdr.cf_ipcountry}} \033[35mthe "CF-IPCountry" client header (probably blank)
+            \033[0m
+            so the following types of placeholders can be added to the lists:
+            * any client header can be accessed through {{hdr.*}}
+            * any variable in httpcli.py can be accessed through {{self.*}}
+            * any global server setting can be accessed through {{cfg.*}}
+            * any volflag can be accessed through {{vf.*}}
+
+            remove vf.scan from default list using --exp-md /vf.scan
+            add "accept" header to def. list using --exp-md +hdr.accept
+
+            for performance reasons, expansion only happens while embedding
+            documents into directory listings, and when accessing a ?doc=...
+            link, but never otherwise, so if you click a -txt- link you'll
+            have to refresh the page to apply expansion
+            """
+            ),
+        ],
         [
             "ls",
             "volume inspection",
@@ -771,8 +818,6 @@ def add_general(ap, nc, srvname):
     ap2.add_argument("-a", metavar="ACCT", type=u, action="append", help="add account, \033[33mUSER\033[0m:\033[33mPASS\033[0m; example [\033[32med:wark\033[0m]")
     ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m]")
     ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files")
-    ap2.add_argument("-emp", action="store_true", help="enable markdown plugins -- neat but dangerous, big XSS risk")
-    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
     ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see --help-urlform")
     ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="window title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
     ap2.add_argument("--name", metavar="TXT", type=u, default=srvname, help="server name (displayed topleft in browser and in mDNS)")
@@ -806,6 +851,8 @@ def add_upload(ap):
     ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead (volflag=copydupes)")
     ap2.add_argument("--no-dupe", action="store_true", help="reject duplicate files during upload; only matches within the same volume (volflag=nodupe)")
     ap2.add_argument("--no-snap", action="store_true", help="disable snapshots -- forget unfinished uploads on shutdown; don't create .hist/up2k.snap files -- abandoned/interrupted uploads must be cleaned up manually")
+    ap2.add_argument("--snap-wri", metavar="SEC", type=int, default=300, help="write upload state to ./hist/up2k.snap every SEC seconds; allows resuming incomplete uploads after a server crash")
+    ap2.add_argument("--snap-drop", metavar="MIN", type=float, default=1440, help="forget unfinished uploads after MIN minutes; impossible to resume them after that (360=6h, 1440=24h)")
     ap2.add_argument("--u2ts", metavar="TXT", type=u, default="c", help="how to timestamp uploaded files; [\033[32mc\033[0m]=client-last-modified, [\033[32mu\033[0m]=upload-time, [\033[32mfc\033[0m]=force-c, [\033[32mfu\033[0m]=force-u (volflag=u2ts)")
     ap2.add_argument("--rand", action="store_true", help="force randomized filenames, --nrand chars long (volflag=rand)")
     ap2.add_argument("--nrand", metavar="NUM", type=int, default=9, help="randomized filenames length (volflag=nrand)")
@@ -967,6 +1014,7 @@ def add_stats(ap):
     ap2.add_argument("--stats", action="store_true", help="enable openmetrics at /.cpr/metrics for admin accounts")
     ap2.add_argument("--nos-hdd", action="store_true", help="disable disk-space metrics (used/free space)")
     ap2.add_argument("--nos-vol", action="store_true", help="disable volume size metrics (num files, total bytes, vmaxb/vmaxn)")
+    ap2.add_argument("--nos-vst", action="store_true", help="disable volume state metrics (indexing, analyzing, activity)")
     ap2.add_argument("--nos-dup", action="store_true", help="disable dupe-files metrics (good idea; very slow)")
     ap2.add_argument("--nos-unf", action="store_true", help="disable unfinished-uploads metrics")
 
@@ -996,7 +1044,7 @@ def add_optouts(ap):
 def add_safety(ap):
     ap2 = ap.add_argument_group('safety options')
     ap2.add_argument("-s", action="count", default=0, help="increase safety: Disable thumbnails / potentially dangerous software (ffmpeg/pillow/vips), hide partial uploads, avoid crawlers.\n └─Alias of\033[32m --dotpart --no-thumb --no-mtag-ff --no-robots --force-js")
-    ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, webdav, 404 on 403, ban on excessive 404s.\n └─Alias of\033[32m -s --unpost=0 --no-del --no-mv --hardlink --vague-403 --ban-404=50,60,1440 --turbo=-1 -nih")
+    ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, webdav, 404 on 403, ban on excessive 404s.\n └─Alias of\033[32m -s --unpost=0 --no-del --no-mv --hardlink --vague-403 -nih")
     ap2.add_argument("-sss", action="store_true", help="further increase safety: Enable logging to disk, scan for dangerous symlinks.\n └─Alias of\033[32m -ss --no-dav --no-logues --no-readme -lo=cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz --ls=**,*,ln,p,r")
     ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="do a sanity/safety check of all volumes on startup; arguments \033[33mUSER\033[0m,\033[33mVOL\033[0m,\033[33mFLAGS\033[0m; example [\033[32m**,*,ln,p,r\033[0m]")
     ap2.add_argument("--xvol", action="store_true", help="never follow symlinks leaving the volume root, unless the link is into another volume where the user has similar access (volflag=xvol)")
@@ -1010,10 +1058,10 @@ def add_safety(ap):
     ap2.add_argument("--no-robots", action="store_true", help="adds http and html headers asking search engines to not index anything (volflag=norobots)")
     ap2.add_argument("--logout", metavar="H", type=float, default="8086", help="logout clients after H hours of inactivity; [\033[32m0.0028\033[0m]=10sec, [\033[32m0.1\033[0m]=6min, [\033[32m24\033[0m]=day, [\033[32m168\033[0m]=week, [\033[32m720\033[0m]=month, [\033[32m8760\033[0m]=year)")
     ap2.add_argument("--ban-pw", metavar="N,W,B", type=u, default="9,60,1440", help="more than \033[33mN\033[0m wrong passwords in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; disable with [\033[32mno\033[0m]")
-    ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="no", help="hitting more than \033[33mN\033[0m 404's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (disabled by default since turbo-up2k counts as 404s)")
+    ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="50,60,1440", help="hitting more than \033[33mN\033[0m 404's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; only affects users who cannot see directory listings because their access is either g/G/h")
     ap2.add_argument("--ban-403", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 403's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; [\033[32m1440\033[0m]=day, [\033[32m10080\033[0m]=week, [\033[32m43200\033[0m]=month")
     ap2.add_argument("--ban-422", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 422's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (422 is server fuzzing, invalid POSTs and so)")
-    ap2.add_argument("--ban-url", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m sus URL's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (decent replacement for --ban-404 if that can't be used)")
+    ap2.add_argument("--ban-url", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m sus URL's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; applies only to access g/G/h (decent replacement for --ban-404 if that can't be used)")
     ap2.add_argument("--sus-urls", metavar="R", type=u, default=r"\.php$|(^|/)wp-(admin|content|includes)/", help="URLs which are considered sus / eligible for banning; disable with blank or [\033[32mno\033[0m]")
     ap2.add_argument("--nonsus-urls", metavar="R", type=u, default=r"^(favicon\.ico|robots\.txt)$|^apple-touch-icon|^\.well-known", help="harmless URLs ignored from 404-bans; disable with blank or [\033[32mno\033[0m]")
     ap2.add_argument("--aclose", metavar="MIN", type=int, default=10, help="if a client maxes out the server connection limit, downgrade it from connection:keep-alive to connection:close for MIN minutes (and also kill its active connections) -- disable with 0")
@@ -1047,7 +1095,7 @@ def add_logging(ap):
     ap2.add_argument("--ansi", action="store_true", help="force colors; overrides environment-variable NO_COLOR")
     ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
     ap2.add_argument("--log-tdec", metavar="N", type=int, default=3, help="timestamp resolution / number of timestamp decimals")
-    ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log passphrase of failed login attempts: 0=terse, 1=plaintext, 2=hashed")
+    ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log failed login attempt passwords: 0=terse, 1=plaintext, 2=hashed")
     ap2.add_argument("--log-conn", action="store_true", help="debug: print tcp-server msgs")
     ap2.add_argument("--log-htp", action="store_true", help="debug: print http-server threadpool scaling")
     ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header")
@@ -1132,13 +1180,20 @@ def add_db_metadata(ap):
     ap2.add_argument("--mtag-v", action="store_true", help="verbose tag scanning; print errors from mtp subprocesses and such")
     ap2.add_argument("--mtag-vv", action="store_true", help="debug mtp settings and mutagen/ffprobe parsers")
     ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
-    ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
-        default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,fmt,res,.fps,ahash,vhash,up_ip,.up_at")
-    ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.)",
-        default=".vq,.aq,vc,ac,fmt,res,.fps")
+    ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.); either an entire replacement list, or add/remove stuff on the default-list with +foo or /bar", default=DEF_MTE)
+    ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.); assign/add/remove same as -mte", default=DEF_MTH)
     ap2.add_argument("-mtp", metavar="M=[f,]BIN", type=u, action="append", help="read tag M using program BIN to parse the file")
 
 
+def add_txt(ap):
+    ap2 = ap.add_argument_group('textfile options')
+    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="textfile editor checks for serverside changes every SEC seconds")
+    ap2.add_argument("-emp", action="store_true", help="enable markdown plugins -- neat but dangerous, big XSS risk")
+    ap2.add_argument("--exp", action="store_true", help="enable textfile expansion -- replace {{self.ip}} and such; see --help-exp (volflag=exp)")
+    ap2.add_argument("--exp-md", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in markdown files; add/remove stuff on the default list with +hdr_foo or /vf.scan (volflag=exp_md)")
+    ap2.add_argument("--exp-lg", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in prologue/epilogue files (volflag=exp_lg)")
+
+
 def add_ui(ap, retry):
     ap2 = ap.add_argument_group('ui options')
     ap2.add_argument("--grid", action="store_true", help="show grid/thumbnails by default (volflag=grid)")
@@ -1232,6 +1287,7 @@ def run_argparse(
     add_handlers(ap)
     add_hooks(ap)
     add_stats(ap)
+    add_txt(ap)
     add_ui(ap, retry)
     add_admin(ap)
     add_logging(ap)
@@ -1259,7 +1315,7 @@ def run_argparse(
     for k, h, t in sects:
         k2 = "help_" + k.replace("-", "_")
         if vars(ret)[k2]:
-            lprint("# {} help page".format(k))
+            lprint("# %s help page (%s)" % (k, h))
             lprint(t + "\033[0m")
             sys.exit(0)
 

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 9, 12)
+VERSION = (1, 9, 17)
 CODENAME = "prometheable"
-BUILD_DT = (2023, 10, 15)
+BUILD_DT = (2023, 11, 11)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -21,11 +21,14 @@ from .util import (
     META_NOBOTS,
     SQLITE_VER,
     UNPLICATIONS,
+    UTC,
+    ODict,
     Pebkac,
     absreal,
     afsenc,
     get_df,
     humansize,
+    odfusion,
     relchk,
     statdir,
     uncyg,
@@ -213,7 +216,7 @@ class Lim(object):
             if self.rot_re.search(path.replace("\\", "/")):
                 return path, ""
 
-            suf = datetime.utcnow().strftime(self.rotf)
+            suf = datetime.now(UTC).strftime(self.rotf)
             if path:
                 path += "/"
 
@@ -473,12 +476,10 @@ class VFS(object):
         err: int = 403,
     ) -> tuple["VFS", str]:
         """returns [vfsnode,fs_remainder] if user has the requested permissions"""
-        if ANYWIN:
-            mod = relchk(vpath)
-            if mod:
-                if self.log:
-                    self.log("vfs", "invalid relpath [{}]".format(vpath))
-                raise Pebkac(404)
+        if relchk(vpath):
+            if self.log:
+                self.log("vfs", "invalid relpath [{}]".format(vpath))
+            raise Pebkac(422)
 
         cvpath = undot(vpath)
         vn, rem = self._find(cvpath)
@@ -497,8 +498,8 @@ class VFS(object):
                     t = "{} has no {} in [{}] => [{}] => [{}]"
                     self.log("vfs", t.format(uname, msg, vpath, cvpath, ap), 6)
 
-                t = "you don't have {}-access for this location"
-                raise Pebkac(err, t.format(msg))
+                t = 'you don\'t have %s-access in "/%s"'
+                raise Pebkac(err, t % (msg, cvpath))
 
         return vn, rem
 
@@ -1426,12 +1427,12 @@ class AuthSrv(object):
 
             for ga, vf in [["no_hash", "nohash"], ["no_idx", "noidx"]]:
                 if vf in vol.flags:
-                    ptn = vol.flags.pop(vf)
+                    ptn = re.compile(vol.flags.pop(vf))
                 else:
                     ptn = getattr(self.args, ga)
 
                 if ptn:
-                    vol.flags[vf] = re.compile(ptn)
+                    vol.flags[vf] = ptn
 
             for ga, vf in vf_bmap().items():
                 if getattr(self.args, ga):
@@ -1476,14 +1477,11 @@ class AuthSrv(object):
                 raise Exception(t.format(dbd, dbds))
 
             # default tag cfgs if unset
-            if "mte" not in vol.flags:
-                vol.flags["mte"] = self.args.mte
-            elif vol.flags["mte"].startswith("+"):
-                vol.flags["mte"] = ",".join(
-                    x for x in [self.args.mte, vol.flags["mte"][1:]] if x
-                )
-            if "mth" not in vol.flags:
-                vol.flags["mth"] = self.args.mth
+            for k in ("mte", "mth", "exp_md", "exp_lg"):
+                if k not in vol.flags:
+                    vol.flags[k] = getattr(self.args, k).copy()
+                else:
+                    vol.flags[k] = odfusion(getattr(self.args, k), vol.flags[k])
 
             # append additive args from argv to volflags
             hooks = "xbu xau xiu xbr xar xbd xad xm xban".split()
@@ -1537,7 +1535,11 @@ class AuthSrv(object):
                 if vol.flags.get(grp, False):
                     continue
 
-                vol.flags = {k: v for k, v in vol.flags.items() if not k.startswith(rm)}
+                vol.flags = {
+                    k: v
+                    for k, v in vol.flags.items()
+                    if not k.startswith(rm) or k == "mte"
+                }
 
             for grp, rm in [["d2v", "e2v"]]:
                 if not vol.flags.get(grp, False):
@@ -1584,12 +1586,12 @@ class AuthSrv(object):
                 if local:
                     local_only_mtp[a] = True
 
-            local_mte = {}
-            for a in vol.flags.get("mte", "").split(","):
+            local_mte = ODict()
+            for a in vol.flags.get("mte", {}).keys():
                 local = True
                 all_mte[a] = True
                 local_mte[a] = True
-                for b in self.args.mte.split(","):
+                for b in self.args.mte.keys():
                     if not a or not b:
                         continue
 
@@ -1719,6 +1721,9 @@ class AuthSrv(object):
     def setup_pwhash(self, acct: dict[str, str]) -> None:
         self.ah = PWHash(self.args)
         if not self.ah.on:
+            if self.args.ah_cli or self.args.ah_gen:
+                t = "\n  BAD CONFIG:\n    cannot --ah-cli or --ah-gen without --ah-alg"
+                raise Exception(t)
             return
 
         if self.args.ah_cli:

copyparty/cert.py

@@ -132,7 +132,10 @@ def _gen_srv(log: "RootLogger", args, netdevs: dict[str, Netdev]):
 
     try:
         expiry, inf = _read_crt(args, "srv.pem")
-        expired = time.time() + args.crt_sdays * 60 * 60 * 24 * 0.1 > expiry
+        if "sans" not in inf:
+            raise Exception("no useable cert found")
+
+        expired = time.time() + args.crt_sdays * 60 * 60 * 24 * 0.5 > expiry
         cert_insec = os.path.join(args.E.mod, "res/insecure.pem")
         for n in names:
             if n not in inf["sans"]:

copyparty/cfg.py

@@ -13,18 +13,26 @@ def vf_bmap() -> dict[str, str]:
         "no_dedup": "copydupes",
         "no_dupe": "nodupe",
         "no_forget": "noforget",
+        "no_robots": "norobots",
+        "no_thumb": "dthumb",
+        "no_vthumb": "dvthumb",
+        "no_athumb": "dathumb",
         "th_no_crop": "nocrop",
         "dav_auth": "davauth",
         "dav_rt": "davrt",
     }
     for k in (
         "dotsrch",
+        "e2d",
+        "e2ds",
+        "e2dsa",
         "e2t",
         "e2ts",
         "e2tsr",
         "e2v",
         "e2vu",
         "e2vp",
+        "exp",
         "grid",
         "hardlink",
         "magic",
@@ -41,8 +49,22 @@ def vf_bmap() -> dict[str, str]:
 
 def vf_vmap() -> dict[str, str]:
     """argv-to-volflag: simple values"""
-    ret = {"th_convt": "convt", "th_size": "thsize"}
-    for k in ("dbd", "lg_sbf", "md_sbf", "nrand", "sort", "unlist", "u2ts"):
+    ret = {
+        "no_hash": "nohash",
+        "no_idx": "noidx",
+        "re_maxage": "scan",
+        "th_convt": "convt",
+        "th_size": "thsize",
+    }
+    for k in (
+        "dbd",
+        "lg_sbf",
+        "md_sbf",
+        "nrand",
+        "sort",
+        "unlist",
+        "u2ts",
+    ):
         ret[k] = k
     return ret
 
@@ -50,7 +72,23 @@ def vf_vmap() -> dict[str, str]:
 def vf_cmap() -> dict[str, str]:
     """argv-to-volflag: complex/lists"""
     ret = {}
-    for k in ("html_head", "mte", "mth"):
+    for k in (
+        "exp_lg",
+        "exp_md",
+        "html_head",
+        "mte",
+        "mth",
+        "mtp",
+        "xad",
+        "xar",
+        "xau",
+        "xban",
+        "xbd",
+        "xbr",
+        "xbu",
+        "xiu",
+        "xm",
+    ):
         ret[k] = k
     return ret
 

copyparty/ftpd.py

@@ -11,11 +11,6 @@ import time
 
 from .__init__ import ANYWIN, PY2, TYPE_CHECKING, E
 
-try:
-    import asynchat
-except:
-    sys.path.append(os.path.join(E.mod, "vend"))
-
 from pyftpdlib.authorizers import AuthenticationFailed, DummyAuthorizer
 from pyftpdlib.filesystems import AbstractedFS, FilesystemError
 from pyftpdlib.handlers import FTPHandler
@@ -92,6 +87,12 @@ class FtpAuth(DummyAuthorizer):
                 if bonk:
                     logging.warning("client banned: invalid passwords")
                     bans[ip] = bonk
+                    try:
+                        # only possible if multiprocessing disabled
+                        self.hub.broker.httpsrv.bans[ip] = bonk
+                        self.hub.broker.httpsrv.nban += 1
+                    except:
+                        pass
 
             raise AuthenticationFailed("Authentication failed.")
 
@@ -148,7 +149,7 @@ class FtpFs(AbstractedFS):
         try:
             vpath = vpath.replace("\\", "/").strip("/")
             rd, fn = os.path.split(vpath)
-            if ANYWIN and relchk(rd):
+            if relchk(rd):
                 logging.warning("malicious vpath: %s", vpath)
                 t = "Unsupported characters in [{}]"
                 raise FSE(t.format(vpath), 1)

copyparty/httpcli.py

@@ -39,7 +39,10 @@ from .szip import StreamZip
 from .util import (
     HTTPCODE,
     META_NOBOTS,
+    UTC,
+    Garda,
     MultipartParser,
+    ODict,
     Pebkac,
     UnrecvEOF,
     absreal,
@@ -73,6 +76,7 @@ from .util import (
     runhook,
     s3enc,
     sanitize_fn,
+    sanitize_vpath,
     sendfile_kern,
     sendfile_py,
     undot,
@@ -144,6 +148,7 @@ class HttpCli(object):
         self.rem = " "
         self.vpath = " "
         self.vpaths = " "
+        self.gctx = " "  # additional context for garda
         self.trailing_slash = True
         self.uname = " "
         self.pw = " "
@@ -252,8 +257,8 @@ class HttpCli(object):
                     k, zs = header_line.split(":", 1)
                     self.headers[k.lower()] = zs.strip()
             except:
-                msg = " ]\n#[ ".join(headerlines)
-                raise Pebkac(400, "bad headers:\n#[ " + msg + " ]")
+                msg = "#[ " + " ]\n#[ ".join(headerlines) + " ]"
+                raise Pebkac(400, "bad headers", log=msg)
 
         except Pebkac as ex:
             self.mode = "GET"
@@ -266,8 +271,14 @@ class HttpCli(object):
                 self.loud_reply(unicode(ex), status=ex.code, headers=h, volsan=True)
             except:
                 pass
+
+            if ex.log:
+                self.log("additional error context:\n" + ex.log, 6)
+
             return False
 
+        self.conn.hsrv.nreq += 1
+
         self.ua = self.headers.get("user-agent", "")
         self.is_rclone = self.ua.startswith("rclone/")
 
@@ -409,12 +420,9 @@ class HttpCli(object):
             self.vpath + "/" if self.trailing_slash and self.vpath else self.vpath
         )
 
-        ok = "\x00" not in self.vpath
-        if ANYWIN:
-            ok = ok and not relchk(self.vpath)
-
-        if not ok and (self.vpath != "*" or self.mode != "OPTIONS"):
+        if relchk(self.vpath) and (self.vpath != "*" or self.mode != "OPTIONS"):
             self.log("invalid relpath [{}]".format(self.vpath))
+            self.cbonk(self.conn.hsrv.g422, self.vpath, "bad_vp", "invalid relpaths")
             return self.tx_404() and self.keepalive
 
         zso = self.headers.get("authorization")
@@ -547,6 +555,9 @@ class HttpCli(object):
                 zb = b"<pre>" + html_escape(msg).encode("utf-8", "replace")
                 h = {"WWW-Authenticate": 'Basic realm="a"'} if pex.code == 401 else {}
                 self.reply(zb, status=pex.code, headers=h, volsan=True)
+                if pex.log:
+                    self.log("additional error context:\n" + pex.log, 6)
+
                 return self.keepalive
             except Pebkac:
                 return False
@@ -557,6 +568,36 @@ class HttpCli(object):
         else:
             return self.conn.iphash.s(self.ip)
 
+    def cbonk(self, g: Garda, v: str, reason: str, descr: str) -> bool:
+        self.conn.hsrv.nsus += 1
+        if not g.lim:
+            return False
+
+        bonk, ip = g.bonk(self.ip, v + self.gctx)
+        if not bonk:
+            return False
+
+        xban = self.vn.flags.get("xban")
+        if not xban or not runhook(
+            self.log,
+            xban,
+            self.vn.canonical(self.rem),
+            self.vpath,
+            self.host,
+            self.uname,
+            time.time(),
+            0,
+            self.ip,
+            time.time(),
+            reason,
+        ):
+            self.log("client banned: %s" % (descr,), 1)
+            self.conn.hsrv.bans[ip] = bonk
+            self.conn.hsrv.nban += 1
+            return True
+
+        return False
+
     def is_banned(self) -> bool:
         if not self.conn.bans:
             return False
@@ -654,6 +695,7 @@ class HttpCli(object):
                     and not body.startswith(b"<pre>source file busy")
                 )
             )
+            and (status != 404 or (self.can_get and not self.can_read))
         ):
             if status == 404:
                 g = self.conn.hsrv.g404
@@ -675,24 +717,7 @@ class HttpCli(object):
                 or not self.args.nonsus_urls
                 or not self.args.nonsus_urls.search(self.vpath)
             ):
-                bonk, ip = g.bonk(self.ip, self.vpath)
-                if bonk:
-                    xban = self.vn.flags.get("xban")
-                    if not xban or not runhook(
-                        self.log,
-                        xban,
-                        self.vn.canonical(self.rem),
-                        self.vpath,
-                        self.host,
-                        self.uname,
-                        time.time(),
-                        0,
-                        self.ip,
-                        time.time(),
-                        str(status),
-                    ):
-                        self.log("client banned: %ss" % (status,), 1)
-                        self.conn.hsrv.bans[ip] = bonk
+                self.cbonk(g, self.vpath, str(status), "%ss" % (status,))
 
         if volsan:
             vols = list(self.asrv.vfs.all_vols.values())
@@ -1971,8 +1996,7 @@ class HttpCli(object):
         self.log("q#: {} ({:.2f}s)".format(msg, idx.p_dur))
 
         order = []
-        cfg = self.args.mte.split(",")
-        for t in cfg:
+        for t in self.args.mte:
             if t in taglist:
                 order.append(t)
         for t in taglist:
@@ -2119,8 +2143,10 @@ class HttpCli(object):
         return True
 
     def get_pwd_cookie(self, pwd: str) -> str:
-        if self.asrv.ah.hash(pwd) in self.asrv.iacct:
-            msg = "login ok"
+        hpwd = self.asrv.ah.hash(pwd)
+        uname = self.asrv.iacct.get(hpwd)
+        if uname:
+            msg = "hi " + uname
             dur = int(60 * 60 * self.args.logout)
         else:
             logpwd = pwd
@@ -2131,27 +2157,7 @@ class HttpCli(object):
                 logpwd = "%" + base64.b64encode(zb[:12]).decode("utf-8")
 
             self.log("invalid password: {}".format(logpwd), 3)
-
-            g = self.conn.hsrv.gpwd
-            if g.lim:
-                bonk, ip = g.bonk(self.ip, pwd)
-                if bonk:
-                    xban = self.vn.flags.get("xban")
-                    if not xban or not runhook(
-                        self.log,
-                        xban,
-                        self.vn.canonical(self.rem),
-                        self.vpath,
-                        self.host,
-                        self.uname,
-                        time.time(),
-                        0,
-                        self.ip,
-                        time.time(),
-                        "pw",
-                    ):
-                        self.log("client banned: invalid passwords", 1)
-                        self.conn.hsrv.bans[ip] = bonk
+            self.cbonk(self.conn.hsrv.gpwd, pwd, "pw", "invalid passwords")
 
             msg = "naw dude"
             pwd = "x"  # nosec
@@ -2175,26 +2181,30 @@ class HttpCli(object):
         new_dir = self.parser.require("name", 512)
         self.parser.drop()
 
-        sanitized = sanitize_fn(new_dir, "", [])
-        return self._mkdir(vjoin(self.vpath, sanitized))
+        return self._mkdir(vjoin(self.vpath, new_dir))
 
     def _mkdir(self, vpath: str, dav: bool = False) -> bool:
         nullwrite = self.args.nw
+        self.gctx = vpath
+        vpath = undot(vpath)
         vfs, rem = self.asrv.vfs.get(vpath, self.uname, False, True)
-        self._assert_safe_rem(rem)
+        rem = sanitize_vpath(rem, "/", [])
         fn = vfs.canonical(rem)
+        if not fn.startswith(vfs.realpath):
+            self.log("invalid mkdir [%s] [%s]" % (self.gctx, vpath), 1)
+            raise Pebkac(422)
 
         if not nullwrite:
             fdir = os.path.dirname(fn)
 
-            if not bos.path.isdir(fdir):
+            if dav and not bos.path.isdir(fdir):
                 raise Pebkac(409, "parent folder does not exist")
 
             if bos.path.isdir(fn):
-                raise Pebkac(405, "that folder exists already")
+                raise Pebkac(405, 'folder "/%s" already exists' % (vpath,))
 
             try:
-                bos.mkdir(fn)
+                bos.makedirs(fn)
             except OSError as ex:
                 if ex.errno == errno.EACCES:
                     raise Pebkac(500, "the server OS denied write-access")
@@ -2203,7 +2213,7 @@ class HttpCli(object):
             except:
                 raise Pebkac(500, min_ex())
 
-        self.out_headers["X-New-Dir"] = quotep(vpath.split("/")[-1])
+        self.out_headers["X-New-Dir"] = quotep(vpath)
 
         if dav:
             self.reply(b"", 201)
@@ -2724,6 +2734,29 @@ class HttpCli(object):
 
         return file_lastmod, True
 
+    def _expand(self, txt: str, phs: list[str]) -> str:
+        for ph in phs:
+            if ph.startswith("hdr."):
+                sv = str(self.headers.get(ph[4:], ""))
+            elif ph.startswith("self."):
+                sv = str(getattr(self, ph[5:], ""))
+            elif ph.startswith("cfg."):
+                sv = str(getattr(self.args, ph[4:], ""))
+            elif ph.startswith("vf."):
+                sv = str(self.vn.flags.get(ph[3:]) or "")
+            elif ph == "srv.itime":
+                sv = str(int(time.time()))
+            elif ph == "srv.htime":
+                sv = datetime.now(UTC).strftime("%Y-%m-%d, %H:%M:%S")
+            else:
+                self.log("unknown placeholder in server config: [%s]" % (ph), 3)
+                continue
+
+            sv = self.conn.hsrv.ptn_hsafe.sub("_", sv)
+            txt = txt.replace("{{%s}}" % (ph,), sv)
+
+        return txt
+
     def tx_file(self, req_path: str) -> bool:
         status = 200
         logmsg = "{:4} {} ".format("", self.req)
@@ -3050,7 +3083,7 @@ class HttpCli(object):
         self.reply(ico, mime=mime, headers={"Last-Modified": lm})
         return True
 
-    def tx_md(self, fs_path: str) -> bool:
+    def tx_md(self, vn: VFS, fs_path: str) -> bool:
         logmsg = "     %s @%s " % (self.req, self.uname)
 
         if not self.can_write:
@@ -3067,9 +3100,16 @@ class HttpCli(object):
         st = bos.stat(html_path)
         ts_html = st.st_mtime
 
+        max_sz = 1024 * self.args.txt_max
         sz_md = 0
         lead = b""
+        fullfile = b""
         for buf in yieldfile(fs_path):
+            if sz_md < max_sz:
+                fullfile += buf
+            else:
+                fullfile = b""
+
             if not sz_md and b"\n" in buf[:2]:
                 lead = buf[: buf.find(b"\n") + 1]
                 sz_md += len(lead)
@@ -3078,6 +3118,21 @@ class HttpCli(object):
             for c, v in [(b"&", 4), (b"<", 3), (b">", 3)]:
                 sz_md += (len(buf) - len(buf.replace(c, b""))) * v
 
+        if (
+            fullfile
+            and "exp" in vn.flags
+            and "edit" not in self.uparam
+            and "edit2" not in self.uparam
+            and vn.flags.get("exp_md")
+        ):
+            fulltxt = fullfile.decode("utf-8", "replace")
+            fulltxt = self._expand(fulltxt, vn.flags.get("exp_md") or [])
+            fullfile = fulltxt.encode("utf-8", "replace")
+
+        if fullfile:
+            fullfile = html_bescape(fullfile)
+            sz_md = len(lead) + len(fullfile)
+
         file_ts = int(max(ts_md, ts_html, self.E.t0))
         file_lastmod, do_send = self._chk_lastmod(file_ts)
         self.out_headers["Last-Modified"] = file_lastmod
@@ -3119,8 +3174,11 @@ class HttpCli(object):
 
         try:
             self.s.sendall(html[0] + lead)
-            for buf in yieldfile(fs_path):
-                self.s.sendall(html_bescape(buf))
+            if fullfile:
+                self.s.sendall(fullfile)
+            else:
+                for buf in yieldfile(fs_path):
+                    self.s.sendall(html_bescape(buf))
 
             self.s.sendall(html[1])
 
@@ -3751,7 +3809,7 @@ class HttpCli(object):
                     or "edit2" in self.uparam
                 )
             ):
-                return self.tx_md(abspath)
+                return self.tx_md(vn, abspath)
 
             return self.tx_file(abspath)
 
@@ -3813,6 +3871,10 @@ class HttpCli(object):
                 if bos.path.exists(fn):
                     with open(fsenc(fn), "rb") as f:
                         logues[n] = f.read().decode("utf-8")
+                    if "exp" in vn.flags:
+                        logues[n] = self._expand(
+                            logues[n], vn.flags.get("exp_lg") or []
+                        )
 
         readme = ""
         if not self.args.no_readme and not logues[1]:
@@ -3822,6 +3884,8 @@ class HttpCli(object):
                     with open(fsenc(fn), "rb") as f:
                         readme = f.read().decode("utf-8")
                         break
+            if readme and "exp" in vn.flags:
+                readme = self._expand(readme, vn.flags.get("exp_md") or [])
 
         vf = vn.flags
         unlist = vf.get("unlist", "")
@@ -3991,7 +4055,7 @@ class HttpCli(object):
                 margin = "-"
 
             sz = inf.st_size
-            zd = datetime.utcfromtimestamp(linf.st_mtime)
+            zd = datetime.fromtimestamp(linf.st_mtime, UTC)
             dt = "%04d-%02d-%02d %02d:%02d:%02d" % (
                 zd.year,
                 zd.month,
@@ -4051,6 +4115,9 @@ class HttpCli(object):
                     ap = vn.canonical(rem)
                     return self.tx_file(ap)  # is no-cache
 
+        mte = vn.flags.get("mte", {})
+        add_up_at = ".up_at" in mte
+        is_admin = self.can_admin
         tagset: set[str] = set()
         for fe in files:
             fn = fe["name"]
@@ -4078,24 +4145,38 @@ class HttpCli(object):
                     self.log(t.format(rd, fn, min_ex()))
                     break
 
-            fe["tags"] = {k: v for k, v in r}
+            tags = {k: v for k, v in r}
 
-            if self.can_admin:
+            if is_admin:
                 q = "select ip, at from up where rd=? and fn=?"
                 try:
                     zs1, zs2 = icur.execute(q, erd_efn).fetchone()
-                    fe["tags"]["up_ip"] = zs1
-                    fe["tags"][".up_at"] = zs2
+                    if zs1:
+                        tags["up_ip"] = zs1
+                    if zs2:
+                        tags[".up_at"] = zs2
+                except:
+                    pass
+            elif add_up_at:
+                q = "select at from up where rd=? and fn=?"
+                try:
+                    (zs1,) = icur.execute(q, erd_efn).fetchone()
+                    if zs1:
+                        tags[".up_at"] = zs1
                 except:
                     pass
 
-            _ = [tagset.add(k) for k in fe["tags"]]
+            _ = [tagset.add(k) for k in tags]
+            fe["tags"] = tags
 
         if icur:
-            mte = vn.flags.get("mte") or "up_ip,.up_at"
-            taglist = [k for k in mte.split(",") if k in tagset]
+            lmte = list(mte)
+            if self.can_admin:
+                lmte += ["up_ip", ".up_at"]
+
+            taglist = [k for k in lmte if k in tagset]
             for fe in dirs:
-                fe["tags"] = {}
+                fe["tags"] = ODict()
         else:
             taglist = list(tagset)
 
@@ -4115,6 +4196,12 @@ class HttpCli(object):
                 if sz < 1024 * self.args.txt_max:
                     with open(fsenc(docpath), "rb") as f:
                         doctxt = f.read().decode("utf-8", "replace")
+
+                    if doc.lower().endswith(".md") and "exp" in vn.flags:
+                        doctxt = self._expand(doctxt, vn.flags.get("exp_md") or [])
+                else:
+                    self.log("doc 2big: [{}]".format(doc), c=6)
+                    doctxt = "( size of textfile exceeds serverside limit )"
             else:
                 self.log("doc 404: [{}]".format(doc), c=6)
                 doctxt = "( textfile not found )"
@@ -4142,7 +4229,7 @@ class HttpCli(object):
         j2a["txt_ext"] = self.args.textfiles.replace(",", " ")
 
         if "mth" in vn.flags:
-            j2a["def_hcols"] = vn.flags["mth"].split(",")
+            j2a["def_hcols"] = list(vn.flags["mth"])
 
         html = self.j2s(tpl, **j2a)
         self.reply(html.encode("utf-8", "replace"))

copyparty/httpsrv.py

@@ -128,6 +128,9 @@ class HttpSrv(object):
 
         self.u2fh = FHC()
         self.metrics = Metrics(self)
+        self.nreq = 0
+        self.nsus = 0
+        self.nban = 0
         self.srvs: list[socket.socket] = []
         self.ncli = 0  # exact
         self.clients: set[HttpConn] = set()  # laggy
@@ -149,6 +152,7 @@ class HttpSrv(object):
         self._build_statics()
 
         self.ptn_cc = re.compile(r"[\x00-\x1f]")
+        self.ptn_hsafe = re.compile(r"[\x00-\x1f<>\"'&]")
 
         self.mallow = "GET HEAD POST PUT DELETE OPTIONS".split()
         if not self.args.no_dav:

copyparty/metrics.py

@@ -34,14 +34,23 @@ class Metrics(object):
 
         ret: list[str] = []
 
-        def addc(k: str, unit: str, v: str, desc: str) -> None:
-            if unit:
-                k += "_" + unit
-                zs = "# TYPE %s counter\n# UNIT %s %s\n# HELP %s %s\n%s_created %s\n%s_total %s"
-                ret.append(zs % (k, k, unit, k, desc, k, int(self.hsrv.t0), k, v))
-            else:
-                zs = "# TYPE %s counter\n# HELP %s %s\n%s_created %s\n%s_total %s"
-                ret.append(zs % (k, k, desc, k, int(self.hsrv.t0), k, v))
+        def addc(k: str, v: str, desc: str) -> None:
+            zs = "# TYPE %s counter\n# HELP %s %s\n%s_created %s\n%s_total %s"
+            ret.append(zs % (k, k, desc, k, int(self.hsrv.t0), k, v))
+
+        def adduc(k: str, unit: str, v: str, desc: str) -> None:
+            k += "_" + unit
+            zs = "# TYPE %s counter\n# UNIT %s %s\n# HELP %s %s\n%s_created %s\n%s_total %s"
+            ret.append(zs % (k, k, unit, k, desc, k, int(self.hsrv.t0), k, v))
+
+        def addg(k: str, v: str, desc: str) -> None:
+            zs = "# TYPE %s gauge\n# HELP %s %s\n%s %s"
+            ret.append(zs % (k, k, desc, k, v))
+
+        def addug(k: str, unit: str, v: str, desc: str) -> None:
+            k += "_" + unit
+            zs = "# TYPE %s gauge\n# UNIT %s %s\n# HELP %s %s\n%s %s"
+            ret.append(zs % (k, k, unit, k, desc, k, v))
 
         def addh(k: str, typ: str, desc: str) -> None:
             zs = "# TYPE %s %s\n# HELP %s %s"
@@ -54,17 +63,75 @@ class Metrics(object):
         def addv(k: str, v: str) -> None:
             ret.append("%s %s" % (k, v))
 
+        t = "time since last copyparty restart"
         v = "{:.3f}".format(time.time() - self.hsrv.t0)
-        addc("cpp_uptime", "seconds", v, "time since last server restart")
+        addug("cpp_uptime", "seconds", v, t)
+
+        # timestamps are gauges because initial value is not zero
+        t = "unixtime of last copyparty restart"
+        v = "{:.3f}".format(self.hsrv.t0)
+        addug("cpp_boot_unixtime", "seconds", v, t)
+
+        t = "number of open http(s) client connections"
+        addg("cpp_http_conns", str(self.hsrv.ncli), t)
+
+        t = "number of http(s) requests since last restart"
+        addc("cpp_http_reqs", str(self.hsrv.nreq), t)
+
+        t = "number of 403/422/malicious reqs since restart"
+        addc("cpp_sus_reqs", str(self.hsrv.nsus), t)
 
         v = str(len(conn.bans or []))
-        addc("cpp_bans", "", v, "number of banned IPs")
+        addg("cpp_active_bans", v, "number of currently banned IPs")
+
+        t = "number of IPs banned since last restart"
+        addg("cpp_total_bans", str(self.hsrv.nban), t)
+
+        if not args.nos_vst:
+            x = self.hsrv.broker.ask("up2k.get_state")
+            vs = json.loads(x.get())
+
+            nvidle = 0
+            nvbusy = 0
+            nvoffline = 0
+            for v in vs["volstate"].values():
+                if v == "online, idle":
+                    nvidle += 1
+                elif "OFFLINE" in v:
+                    nvoffline += 1
+                else:
+                    nvbusy += 1
+
+            addg("cpp_idle_vols", str(nvidle), "number of idle/ready volumes")
+            addg("cpp_busy_vols", str(nvbusy), "number of busy/indexing volumes")
+            addg("cpp_offline_vols", str(nvoffline), "number of offline volumes")
+
+            t = "time since last database activity (upload/rename/delete)"
+            addug("cpp_db_idle", "seconds", str(vs["dbwt"]), t)
+
+            t = "unixtime of last database activity (upload/rename/delete)"
+            addug("cpp_db_act", "seconds", str(vs["dbwu"]), t)
+
+            t = "number of files queued for hashing/indexing"
+            addg("cpp_hashing_files", str(vs["hashq"]), t)
+
+            t = "number of files queued for metadata scanning"
+            addg("cpp_tagq_files", str(vs["tagq"]), t)
+
+            try:
+                t = "number of files queued for plugin-based analysis"
+                addg("cpp_mtpq_files", str(int(vs["mtpq"])), t)
+            except:
+                pass
 
         if not args.nos_hdd:
             addbh("cpp_disk_size_bytes", "total HDD size of volume")
             addbh("cpp_disk_free_bytes", "free HDD space in volume")
             for vpath, vol in allvols:
                 free, total = get_df(vol.realpath)
+                if free is None or total is None:
+                    continue
+
                 addv('cpp_disk_size_bytes{vol="/%s"}' % (vpath), str(total))
                 addv('cpp_disk_free_bytes{vol="/%s"}' % (vpath), str(free))
 
@@ -161,5 +228,6 @@ class Metrics(object):
         ret.append("# EOF")
 
         mime = "application/openmetrics-text; version=1.0.0; charset=utf-8"
+        mime = cli.uparam.get("mime") or mime
         cli.reply("\n".join(ret).encode("utf-8"), mime=mime)
         return True

copyparty/pwhash.py

@@ -136,8 +136,12 @@ class PWHash(object):
         import getpass
 
         while True:
-            p1 = getpass.getpass("password> ")
-            p2 = getpass.getpass("again or just hit ENTER> ")
+            try:
+                p1 = getpass.getpass("password> ")
+                p2 = getpass.getpass("again or just hit ENTER> ")
+            except EOFError:
+                return
+
             if p2 and p1 != p2:
                 print("\033[31minputs don't match; try again\033[0m", file=sys.stderr)
                 continue

copyparty/sutil.py

@@ -8,7 +8,7 @@ from datetime import datetime
 from .__init__ import CORES
 from .bos import bos
 from .th_cli import ThumbCli
-from .util import vjoin
+from .util import UTC, vjoin
 
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Generator, Optional
@@ -108,7 +108,7 @@ def errdesc(errors: list[tuple[str, str]]) -> tuple[dict[str, Any], list[str]]:
         tf_path = tf.name
         tf.write("\r\n".join(report).encode("utf-8", "replace"))
 
-    dt = datetime.utcnow().strftime("%Y-%m%d-%H%M%S")
+    dt = datetime.now(UTC).strftime("%Y-%m%d-%H%M%S")
 
     bos.chmod(tf_path, 0o444)
     return {

copyparty/svchub.py

@@ -36,16 +36,22 @@ from .tcpsrv import TcpSrv
 from .th_srv import HAVE_PIL, HAVE_VIPS, HAVE_WEBP, ThumbSrv
 from .up2k import Up2k
 from .util import (
+    DEF_EXP,
+    DEF_MTE,
+    DEF_MTH,
     FFMPEG_URL,
+    UTC,
     VERSIONS,
     Daemon,
     Garda,
     HLog,
     HMaccas,
+    ODict,
     alltrace,
     ansi_re,
     min_ex,
     mp,
+    odfusion,
     pybin,
     start_log_thrs,
     start_stackmon,
@@ -115,8 +121,6 @@ class SvcHub(object):
             args.no_mv = True
             args.hardlink = True
             args.vague_403 = True
-            args.ban_404 = "50,60,1440"
-            args.turbo = -1
             args.nih = True
 
         if args.s:
@@ -433,6 +437,21 @@ class SvcHub(object):
             zs = al.xff_src.replace(" ", "").replace(".", "\\.").replace(",", "|")
             al.xff_re = re.compile("^(?:" + zs + ")")
 
+        mte = ODict.fromkeys(DEF_MTE.split(","), True)
+        al.mte = odfusion(mte, al.mte)
+
+        mth = ODict.fromkeys(DEF_MTH.split(","), True)
+        al.mth = odfusion(mth, al.mth)
+
+        exp = ODict.fromkeys(DEF_EXP.split(" "), True)
+        al.exp_md = odfusion(exp, al.exp_md.replace(" ", ","))
+        al.exp_lg = odfusion(exp, al.exp_lg.replace(" ", ","))
+
+        for k in ["no_hash", "no_idx"]:
+            ptn = getattr(self.args, k)
+            if ptn:
+                setattr(self.args, k, re.compile(ptn))
+
         return True
 
     def _setlimits(self) -> None:
@@ -476,7 +495,7 @@ class SvcHub(object):
             self.args.nc = min(self.args.nc, soft // 2)
 
     def _logname(self) -> str:
-        dt = datetime.utcnow()
+        dt = datetime.now(UTC)
         fn = str(self.args.lo)
         for fs in "YmdHMS":
             fs = "%" + fs
@@ -725,7 +744,7 @@ class SvcHub(object):
             return
 
         with self.log_mutex:
-            zd = datetime.utcnow()
+            zd = datetime.now(UTC)
             ts = self.log_dfmt % (
                 zd.year,
                 zd.month * 100 + zd.day,
@@ -743,7 +762,7 @@ class SvcHub(object):
             self.logf.close()
             self._setup_logfile("")
 
-        dt = datetime.utcnow()
+        dt = datetime.now(UTC)
 
         # unix timestamp of next 00:00:00 (leap-seconds safe)
         day_now = dt.day
@@ -751,14 +770,20 @@ class SvcHub(object):
             dt += timedelta(hours=12)
 
         dt = dt.replace(hour=0, minute=0, second=0)
-        self.next_day = calendar.timegm(dt.utctimetuple())
+        try:
+            tt = dt.utctimetuple()
+        except:
+            # still makes me hella uncomfortable
+            tt = dt.timetuple()
+
+        self.next_day = calendar.timegm(tt)
 
     def _log_enabled(self, src: str, msg: str, c: Union[int, str] = 0) -> None:
         """handles logging from all components"""
         with self.log_mutex:
             now = time.time()
             if now >= self.next_day:
-                dt = datetime.utcfromtimestamp(now)
+                dt = datetime.fromtimestamp(now, UTC)
                 zs = "{}\n" if self.no_ansi else "\033[36m{}\033[0m\n"
                 zs = zs.format(dt.strftime("%Y-%m-%d"))
                 print(zs, end="")
@@ -781,7 +806,7 @@ class SvcHub(object):
                 else:
                     msg = "%s%s\033[0m" % (c, msg)
 
-            zd = datetime.utcfromtimestamp(now)
+            zd = datetime.fromtimestamp(now, UTC)
             ts = self.log_efmt % (
                 zd.hour,
                 zd.minute,

copyparty/u2idx.py

@@ -181,6 +181,11 @@ class U2idx(object):
                     is_date = True
                     have_up = True
 
+                elif v == "up_at":
+                    v = "up.at"
+                    is_date = True
+                    have_up = True
+
                 elif v == "path":
                     v = "trim(?||up.rd,'/')"
                     va.append("\nrd")

copyparty/up2k.py

@@ -24,7 +24,7 @@ from queue import Queue
 from .__init__ import ANYWIN, PY2, TYPE_CHECKING, WINDOWS
 from .authsrv import LEELOO_DALLAS, SSEELOG, VFS, AuthSrv
 from .bos import bos
-from .cfg import vf_bmap, vf_vmap
+from .cfg import vf_bmap, vf_cmap, vf_vmap
 from .fsutil import Fstab
 from .mtag import MParser, MTag
 from .util import (
@@ -65,6 +65,11 @@ from .util import (
     w8b64enc,
 )
 
+try:
+    from pathlib import Path
+except:
+    pass
+
 if HAVE_SQLITE3:
     import sqlite3
 
@@ -134,8 +139,6 @@ class Up2k(object):
         self.vol_act: dict[str, float] = {}
         self.busy_aps: set[str] = set()
         self.dupesched: dict[str, list[tuple[str, str, float]]] = {}
-        self.snap_persist_interval = 300  # persist unfinished index every 5 min
-        self.snap_discard_interval = 21600  # drop unfinished after 6 hours inactivity
         self.snap_prev: dict[str, Optional[tuple[int, float]]] = {}
 
         self.mtag: Optional[MTag] = None
@@ -263,6 +266,7 @@ class Up2k(object):
             "hashq": self.n_hashq,
             "tagq": self.n_tagq,
             "mtpq": mtpq,
+            "dbwu": "{:.2f}".format(self.db_act),
             "dbwt": "{:.2f}".format(
                 min(1000 * 24 * 60 * 60 - 1, time.time() - self.db_act)
             ),
@@ -642,10 +646,7 @@ class Up2k(object):
             if self.stop:
                 break
 
-            en: set[str] = set()
-            if "mte" in vol.flags:
-                en = set(vol.flags["mte"].split(","))
-
+            en = set(vol.flags.get("mte", {}))
             self.entags[vol.realpath] = en
 
             if "e2d" in vol.flags:
@@ -801,13 +802,25 @@ class Up2k(object):
         fv = "\033[0;36m{}:\033[90m{}"
         fx = set(("html_head",))
         fd = vf_bmap()
+        fd.update(vf_cmap())
         fd.update(vf_vmap())
         fd = {v: k for k, v in fd.items()}
         fl = {
             k: v
             for k, v in flags.items()
-            if k not in fd or v != getattr(self.args, fd[k])
+            if k not in fd
+            or (
+                v != getattr(self.args, fd[k])
+                and str(v) != str(getattr(self.args, fd[k]))
+            )
         }
+        for k1, k2 in vf_cmap().items():
+            if k1 not in fl or k1 in fx:
+                continue
+            if str(fl[k1]) == str(getattr(self.args, k2)):
+                del fl[k1]
+            else:
+                fl[k1] = ",".join(x for x in fl[k1])
         a = [
             (ft if v is True else ff if v is False else fv).format(k, str(v))
             for k, v in fl.items()
@@ -826,7 +839,7 @@ class Up2k(object):
                 vpath += "/"
 
             zs = " ".join(sorted(a))
-            zs = zs.replace("30mre.compile(", "30m(")  # nohash
+            zs = zs.replace("90mre.compile(", "90m(")  # nohash
             self.log("/{} {}".format(vpath, zs), "35")
 
         reg = {}
@@ -2135,7 +2148,7 @@ class Up2k(object):
 
             try:
                 nfiles = next(cur.execute("select count(w) from up"))[0]
-                self.log("OK: {} |{}|".format(db_path, nfiles))
+                self.log("  {} |{}|".format(db_path, nfiles), "90")
                 return cur
             except:
                 self.log("WARN: could not list files; DB corrupt?\n" + min_ex())
@@ -2716,7 +2729,18 @@ class Up2k(object):
                     raise Exception("symlink-fallback disabled in cfg")
 
             if not linked:
-                os.symlink(fsenc(lsrc), fsenc(ldst))
+                if ANYWIN:
+                    Path(ldst).symlink_to(lsrc)
+                    if not bos.path.exists(dst):
+                        try:
+                            bos.unlink(dst)
+                        except:
+                            pass
+                        t = "the created symlink [%s] did not resolve to [%s]"
+                        raise Exception(t % (ldst, lsrc))
+                else:
+                    os.symlink(fsenc(lsrc), fsenc(ldst))
+
                 linked = True
         except Exception as ex:
             self.log("cannot link; creating copy: " + repr(ex))
@@ -3748,13 +3772,16 @@ class Up2k(object):
             self._finish_upload(job["ptop"], job["wark"])
 
     def _snapshot(self) -> None:
-        slp = self.snap_persist_interval
+        slp = self.args.snap_wri
+        if not slp or self.args.no_snap:
+            return
+
         while True:
             time.sleep(slp)
             if self.pp:
                 slp = 5
             else:
-                slp = self.snap_persist_interval
+                slp = self.args.snap_wri
                 self.do_snapshot()
 
     def do_snapshot(self) -> None:
@@ -3768,11 +3795,8 @@ class Up2k(object):
         if not histpath:
             return
 
-        rm = [
-            x
-            for x in reg.values()
-            if x["need"] and now - x["poke"] > self.snap_discard_interval
-        ]
+        idrop = self.args.snap_drop * 60
+        rm = [x for x in reg.values() if x["need"] and now - x["poke"] >= idrop]
 
         if self.args.nw:
             lost = []
@@ -3897,45 +3921,58 @@ class Up2k(object):
                 self.n_hashq -= 1
             # self.log("hashq {}".format(self.n_hashq))
 
-            ptop, vtop, rd, fn, ip, at, usr, skip_xau = self.hashq.get()
-            # self.log("hashq {} pop {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
-            if "e2d" not in self.flags[ptop]:
-                continue
+            task = self.hashq.get()
+            if len(task) != 8:
+                raise Exception("invalid hash task")
 
-            abspath = djoin(ptop, rd, fn)
-            self.log("hashing " + abspath)
-            inf = bos.stat(abspath)
-            if not inf.st_size:
-                wark = up2k_wark_from_metadata(
-                    self.salt, inf.st_size, int(inf.st_mtime), rd, fn
-                )
-            else:
-                hashes = self._hashlist_from_file(abspath)
-                if not hashes:
+            try:
+                if not self._hash_t(task):
                     return
+            except Exception as ex:
+                self.log("failed to hash %s: %s" % (task, ex), 1)
 
-                wark = up2k_wark_from_hashlist(self.salt, inf.st_size, hashes)
+    def _hash_t(self, task: tuple[str, str, str, str, str, float, str, bool]) -> bool:
+        ptop, vtop, rd, fn, ip, at, usr, skip_xau = task
+        # self.log("hashq {} pop {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
+        if "e2d" not in self.flags[ptop]:
+            return True
 
-            with self.mutex:
-                self.idx_wark(
-                    self.flags[ptop],
-                    rd,
-                    fn,
-                    inf.st_mtime,
-                    inf.st_size,
-                    ptop,
-                    vtop,
-                    wark,
-                    "",
-                    usr,
-                    ip,
-                    at,
-                    skip_xau,
-                )
+        abspath = djoin(ptop, rd, fn)
+        self.log("hashing " + abspath)
+        inf = bos.stat(abspath)
+        if not inf.st_size:
+            wark = up2k_wark_from_metadata(
+                self.salt, inf.st_size, int(inf.st_mtime), rd, fn
+            )
+        else:
+            hashes = self._hashlist_from_file(abspath)
+            if not hashes:
+                return False
 
-            if at and time.time() - at > 30:
-                with self.rescan_cond:
-                    self.rescan_cond.notify_all()
+            wark = up2k_wark_from_hashlist(self.salt, inf.st_size, hashes)
+
+        with self.mutex:
+            self.idx_wark(
+                self.flags[ptop],
+                rd,
+                fn,
+                inf.st_mtime,
+                inf.st_size,
+                ptop,
+                vtop,
+                wark,
+                "",
+                usr,
+                ip,
+                at,
+                skip_xau,
+            )
+
+        if at and time.time() - at > 30:
+            with self.rescan_cond:
+                self.rescan_cond.notify_all()
+
+        return True
 
     def hash_file(
         self,

copyparty/util.py

@@ -25,7 +25,6 @@ import threading
 import time
 import traceback
 from collections import Counter
-from datetime import datetime
 from email.utils import formatdate
 
 from ipaddress import IPv4Address, IPv4Network, IPv6Address, IPv6Network
@@ -35,6 +34,35 @@ from .__init__ import ANYWIN, EXE, MACOS, PY2, TYPE_CHECKING, VT100, WINDOWS
 from .__version__ import S_BUILD_DT, S_VERSION
 from .stolen import surrogateescape
 
+try:
+    from datetime import datetime, timezone
+
+    UTC = timezone.utc
+except:
+    from datetime import datetime, timedelta, tzinfo
+
+    TD_ZERO = timedelta(0)
+
+    class _UTC(tzinfo):
+        def utcoffset(self, dt):
+            return TD_ZERO
+
+        def tzname(self, dt):
+            return "UTC"
+
+        def dst(self, dt):
+            return TD_ZERO
+
+    UTC = _UTC()
+
+
+if sys.version_info >= (3, 7) or (
+    sys.version_info >= (3, 6) and platform.python_implementation() == "CPython"
+):
+    ODict = dict
+else:
+    from collections import OrderedDict as ODict
+
 
 def _ens(want: str) -> tuple[int, ...]:
     ret: list[int] = []
@@ -261,6 +289,13 @@ EXTS["vnd.mozilla.apng"] = "png"
 MAGIC_MAP = {"jpeg": "jpg"}
 
 
+DEF_EXP = "self.ip self.ua self.uname self.host cfg.name cfg.logout vf.scan vf.thsize hdr.cf_ipcountry srv.itime srv.htime"
+
+DEF_MTE = "circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,fmt,res,.fps,ahash,vhash"
+
+DEF_MTH = ".vq,.aq,vc,ac,fmt,res,.fps"
+
+
 REKOBO_KEY = {
     v: ln.split(" ", 1)[0]
     for ln in """
@@ -1118,7 +1153,7 @@ def stackmon(fp: str, ival: float, suffix: str) -> None:
             buf = lzma.compress(buf, preset=0)
 
         if "%" in fp:
-            dt = datetime.utcnow()
+            dt = datetime.now(UTC)
             for fs in "YmdHMS":
                 fs = "%" + fs
                 if fs in fp:
@@ -1528,8 +1563,8 @@ def read_header(sr: Unrecv, t_idle: int, t_tot: int) -> list[str]:
 
             raise Pebkac(
                 400,
-                "protocol error while reading headers:\n"
-                + ret.decode("utf-8", "replace"),
+                "protocol error while reading headers",
+                log=ret.decode("utf-8", "replace"),
             )
 
         ofs = ret.find(b"\r\n\r\n")
@@ -1738,7 +1773,16 @@ def sanitize_fn(fn: str, ok: str, bad: list[str]) -> str:
     return fn.strip()
 
 
+def sanitize_vpath(vp: str, ok: str, bad: list[str]) -> str:
+    parts = vp.replace(os.sep, "/").split("/")
+    ret = [sanitize_fn(x, ok, bad) for x in parts]
+    return "/".join(ret)
+
+
 def relchk(rp: str) -> str:
+    if "\x00" in rp:
+        return "[nul]"
+
     if ANYWIN:
         if "\n" in rp or "\r" in rp:
             return "x\nx"
@@ -1774,6 +1818,24 @@ def exclude_dotfiles(filepaths: list[str]) -> list[str]:
     return [x for x in filepaths if not x.split("/")[-1].startswith(".")]
 
 
+def odfusion(base: ODict[str, bool], oth: str) -> ODict[str, bool]:
+    # merge an "ordered set" (just a dict really) with another list of keys
+    words0 = [x for x in oth.split(",") if x]
+    words1 = [x for x in oth[1:].split(",") if x]
+
+    ret = base.copy()
+    if oth.startswith("+"):
+        for k in words1:
+            ret[k] = True
+    elif oth[:1] in ("-", "/"):
+        for k in words1:
+            ret.pop(k, None)
+    else:
+        ret = ODict.fromkeys(words0, True)
+
+    return ret
+
+
 def ipnorm(ip: str) -> str:
     if ":" in ip:
         # assume /64 clients; drop 4 groups
@@ -2923,9 +2985,12 @@ def hidedir(dp) -> None:
 
 
 class Pebkac(Exception):
-    def __init__(self, code: int, msg: Optional[str] = None) -> None:
+    def __init__(
+        self, code: int, msg: Optional[str] = None, log: Optional[str] = None
+    ) -> None:
         super(Pebkac, self).__init__(msg or HTTPCODE[code])
         self.code = code
+        self.log = log
 
     def __repr__(self) -> str:
         return "Pebkac({}, {})".format(self.code, repr(self.args))

copyparty/web/browser.css

@@ -1414,14 +1414,17 @@ html.dz input {
 .opview input.i {
 	width: calc(100% - 16.2em);
 }
+input.drc_v,
 input.eq_gain {
 	width: 3em;
 	text-align: center;
 	margin: 0 .6em;
 }
+#audio_drc table,
 #audio_eq table {
 	border-collapse: collapse;
 }
+#audio_drc td,
 #audio_eq td {
 	text-align: center;
 }
@@ -1430,11 +1433,15 @@ input.eq_gain {
 	display: block;
 	padding: 0;
 }
+#au_drc,
 #au_eq {
 	display: block;
 	margin-top: .5em;
 	padding: 1.3em .3em;
 }
+#au_drc {
+	padding: .4em .3em;
+}
 #ico1 {
 	cursor: pointer;
 }
@@ -1475,6 +1482,8 @@ input.eq_gain {
 	width: calc(100% - 2em);
 	margin: .3em 0 0 1.4em;
 }
+@media (max-width: 130em) { #srch_form.tags #tq_raw { width: calc(100% - 34em) } }
+@media (max-width: 95em) { #srch_form.tags #tq_raw { width: calc(100% - 2em) } }
 #tq_raw td+td {
 	width: 100%;
 }
@@ -1882,6 +1891,10 @@ html.y #doc {
 	text-align: center;
 	padding: .5em;
 }
+#docul li.bn span {
+	font-weight: bold;
+	color: var(--fg-max);
+}
 #doc.prism {
 	padding-left: 3em;
 }

copyparty/web/browser.js

@@ -233,6 +233,7 @@ var Ls = {
 		"ml_tcode": "transcode",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer",
+		"ml_drc": "dynamic range compressor",
 
 		"mt_preload": "start loading the next song near the end for gapless playback\">preload",
 		"mt_fullpre": "try to preload the entire song;$N✅ enable on <b>unreliable</b> connections,$N❌ <b>disable</b> on slow connections probably\">full",
@@ -250,6 +251,7 @@ var Ls = {
 		"mt_coth": "convert all others (not mp3) to opus\">oth",
 		"mt_tint": "background level (0-100) on the seekbar$Nto make buffering less distracting",
 		"mt_eq": "enables the equalizer and gain control;$N$Nboost &lt;code&gt;0&lt;/code&gt; = standard 100% volume (unmodified)$N$Nwidth &lt;code&gt;1 &nbsp;&lt;/code&gt; = standard stereo (unmodified)$Nwidth &lt;code&gt;0.5&lt;/code&gt; = 50% left-right crossfeed$Nwidth &lt;code&gt;0 &nbsp;&lt;/code&gt; = mono$N$Nboost &lt;code&gt;-0.8&lt;/code&gt; &amp; width &lt;code&gt;10&lt;/code&gt; = vocal removal :^)$N$Nenabling the equalizer makes gapless albums fully gapless, so leave it on with all the values at zero (except width = 1) if you care about that",
+		"mt_drc": "enables the dynamic range compressor (volume flattener / brickwaller); will also enable EQ to balance the spaghetti, so set all EQ fields except for 'width' to 0 if you don't want it$N$Nlowers the volume of audio above THRESHOLD dB; for every RATIO dB past THRESHOLD there is 1 dB of output, so default values of tresh -24 and ratio 12 means it should never get louder than -22 dB and it is safe to increase the equalizer boost to 0.8, or even 1.8 with ATK 0 and a huge RLS like 90 (only works in firefox; RLS is max 1 in other browsers)$N$N(see wikipedia, they explain it much better)",
 
 		"mb_play": "play",
 		"mm_hashplay": "play this audio file?",
@@ -354,11 +356,14 @@ var Ls = {
 		"s_rd": "path",
 		"s_fn": "name",
 		"s_ta": "tags",
+		"s_ua": "up@",
 		"s_ad": "adv.",
 		"s_s1": "minimum MiB",
 		"s_s2": "maximum MiB",
 		"s_d1": "min. iso8601",
 		"s_d2": "max. iso8601",
+		"s_u1": "uploaded after",
+		"s_u2": "and/or before",
 		"s_r1": "path contains &nbsp; (space-separated)",
 		"s_f1": "name contains &nbsp; (negate with -nope)",
 		"s_t1": "tags contains &nbsp; (^=start, end=$)",
@@ -451,6 +456,8 @@ var Ls = {
 		"u_expl": "explain",
 		"u_tu": '<p class="warn">WARNING: turbo enabled, <span>&nbsp;client may not detect and resume incomplete uploads; see turbo-button tooltip</span></p>',
 		"u_ts": '<p class="warn">WARNING: turbo enabled, <span>&nbsp;search results can be incorrect; see turbo-button tooltip</span></p>',
+		"u_turbo_c": "turbo is disabled in server config",
+		"u_turbo_g": "disabling turbo because you don't have\ndirectory listing privileges within this volume",
 		"u_life_cfg": 'autodelete after <input id="lifem" p="60" /> min (or <input id="lifeh" p="3600" /> hours)',
 		"u_life_est": 'upload will be deleted <span id="lifew" tt="local time">---</span>',
 		"u_life_max": 'this folder enforces a\nmax lifetime of {0}',
@@ -705,6 +712,7 @@ var Ls = {
 		"ml_tcode": "konvertering",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer (tonejustering)",
+		"ml_drc": "compressor (volum-utjevning)",
 
 		"mt_preload": "hent ned litt av neste sang i forkant,$Nslik at pausen i overgangen blir mindre\">forles",
 		"mt_fullpre": "hent ned hele neste sang, ikke bare litt:$N✅ skru på hvis nettet ditt er <b>ustabilt</b>,$N❌ skru av hvis nettet ditt er <b>tregt</b>\">full",
@@ -722,6 +730,7 @@ var Ls = {
 		"mt_coth": "konverter alt annet (men ikke mp3) til opus\">andre",
 		"mt_tint": "nivå av bakgrunnsfarge på søkestripa (0-100),$Ngjør oppdateringer mindre distraherende",
 		"mt_eq": "aktiver tonekontroll og forsterker;$N$Nboost &lt;code&gt;0&lt;/code&gt; = normal volumskala$N$Nwidth &lt;code&gt;1 &nbsp;&lt;/code&gt; = normal stereo$Nwidth &lt;code&gt;0.5&lt;/code&gt; = 50% blanding venstre-høyre$Nwidth &lt;code&gt;0 &nbsp;&lt;/code&gt; = mono$N$Nboost &lt;code&gt;-0.8&lt;/code&gt; &amp; width &lt;code&gt;10&lt;/code&gt; = instrumental :^)$N$Nreduserer også dødtid imellom sangfiler",
+		"mt_drc": "aktiver volum-utjevning (dynamic range compressor); vil også aktivere tonejustering, så sett alle EQ-feltene bortsett fra 'width' til 0 hvis du ikke vil ha noe EQ$N$Nfilteret vil dempe volumet på alt som er høyere enn TRESH dB; for hver RATIO dB over grensen er det 1dB som treffer høyttalerne, så standardverdiene tresh -24 og ratio 12 skal bety at volumet ikke går høyere enn -22 dB, slik at man trygt kan øke boost-verdien i equalizer'n til rundt 0.8, eller 1.8 kombinert med ATK 0 og RLS 90 (bare mulig i firefox; andre nettlesere tar ikke høyere RLS enn 1)$N$Nwikipedia forklarer dette mye bedre forresten",
 
 		"mb_play": "lytt",
 		"mm_hashplay": "spill denne sangen?",
@@ -826,11 +835,14 @@ var Ls = {
 		"s_rd": "sti",
 		"s_fn": "navn",
 		"s_ta": "meta",
+		"s_ua": "up@",
 		"s_ad": "avns.",
 		"s_s1": "større enn ↓ MiB",
 		"s_s2": "mindre enn ↓ MiB",
 		"s_d1": "nyere enn &lt;dato&gt;",
 		"s_d2": "eldre enn",
+		"s_u1": "lastet opp etter",
+		"s_u2": "og/eller før",
 		"s_r1": "mappenavn inneholder",
 		"s_f1": "filnavn inneholder",
 		"s_t1": "sang-info inneholder",
@@ -923,6 +935,8 @@ var Ls = {
 		"u_expl": "forklar",
 		"u_tu": '<p class="warn">ADVARSEL: turbo er på, <span>&nbsp;avbrutte opplastninger vil muligens ikke oppdages og gjenopptas; hold musepekeren over turbo-knappen for mer info</span></p>',
 		"u_ts": '<p class="warn">ADVARSEL: turbo er på, <span>&nbsp;søkeresultater kan være feil; hold musepekeren over turbo-knappen for mer info</span></p>',
+		"u_turbo_c": "turbo er deaktivert i serverkonfigurasjonen",
+		"u_turbo_g": 'turbo ble deaktivert fordi du ikke har\ntilgang til å se mappeinnhold i dette volumet',
 		"u_life_cfg": 'slett opplastning etter <input id="lifem" p="60" /> min (eller <input id="lifeh" p="3600" /> timer)',
 		"u_life_est": 'opplastningen slettes <span id="lifew" tt="lokal tid">---</span>',
 		"u_life_max": 'denne mappen tillater ikke å \noppbevare filer i mer enn {0}',
@@ -1376,7 +1390,9 @@ var mpl = (function () {
 		'<input type="text" id="pb_tint" value="0" ' + NOAC + ' style="width:2.4em" tt="' + L.mt_tint + '" />' +
 		'</div></div>' +
 
-		'<div><h3>' + L.ml_eq + '</h3><div id="audio_eq"></div></div>');
+		'<div><h3 id="h_drc">' + L.ml_drc + '</h3><div id="audio_drc"></div></div>' +
+		'<div><h3>' + L.ml_eq + '</h3><div id="audio_eq"></div></div>' +
+		'');
 
 	var r = {
 		"pb_mode": (sread('pb_mode', ['loop', 'next']) || 'next').split('-')[0],
@@ -2482,8 +2498,13 @@ function start_actx() {
 var afilt = (function () {
 	var r = {
 		"eqen": false,
+		"drcen": false,
 		"bands": [31.25, 62.5, 125, 250, 500, 1000, 2000, 4000, 8000, 16000],
 		"gains": [4, 3, 2, 1, 0, 0, 1, 2, 3, 4],
+		"drcv": [-24, 30, 12, 0.01, 0.25],
+		"drch": ['tresh', 'knee', 'ratio', 'atk', 'rls'],
+		"drck": ['threshold', 'knee', 'ratio', 'attack', 'release'],
+		"drcn": null,
 		"filters": [],
 		"filterskip": [],
 		"plugs": [],
@@ -2493,16 +2514,18 @@ var afilt = (function () {
 		"acst": {}
 	};
 
-	if (!ACtx)
-		ebi('audio_eq').parentNode.style.display = 'none';
+	function setvis(vis) {
+		ebi('audio_eq').parentNode.style.display = ebi('audio_drc').parentNode.style.display = (vis ? '' : 'none');
+	}
+
+	setvis(ACtx);
 
 	r.init = function () {
 		start_actx();
 		if (r.cfg)
 			return;
 
-		if (!actx)
-			ebi('audio_eq').parentNode.style.display = 'none';
+		setvis(actx);
 
 		// some browsers have insane high-frequency boost
 		// (or rather the actual problem is Q but close enough)
@@ -2553,6 +2576,8 @@ var afilt = (function () {
 		var gains = jread('au_eq_gain', r.gains);
 		if (r.gains.length == gains.length)
 			r.gains = gains;
+
+		r.drcv = jread('au_drcv', r.drcv);
 	}
 	catch (ex) { }
 
@@ -2589,12 +2614,20 @@ var afilt = (function () {
 		mp.acs = mpo.acs = null;
 	};
 
-	r.apply = function () {
+	r.apply = function (v) {
 		r.init();
 		r.draw();
 
-		if (!actx)
-			bcfg_set('au_eq', false);
+		if (!actx) {
+			bcfg_set('au_eq', r.eqen = false);
+			bcfg_set('au_drc', r.drcen = false);
+		}
+		else if (v === true && r.drcen && !r.eqen)
+			bcfg_set('au_eq', r.eqen = true);
+		else if (v === false && !r.eqen)
+			bcfg_set('au_drc', r.drcen = false);
+
+		r.drcn = null;
 
 		var plug = false;
 		for (var a = 0; a < r.plugs.length; a++)
@@ -2654,6 +2687,28 @@ var afilt = (function () {
 		fi.gain.value = r.amp + 0.94;  // +.137 dB measured; now -.25 dB and almost bitperfect
 		r.filters.push(fi);
 
+		// wait nevermind, drc goes first
+		timer.rm(showdrc);
+		if (r.drcen) {
+			fi = r.drcn = actx.createDynamicsCompressor();
+			for (var a = 0; a < r.drcv.length; a++)
+				fi[r.drck[a]].value = r.drcv[a];
+
+			if (r.drcv[3] < 0.02) {
+				// avoid static at decode start
+				fi.attack.value = 0.02;
+				setTimeout(function () {
+					try {
+						fi.attack.value = r.drcv[3];
+					}
+					catch (ex) { }
+				}, 200);
+			}
+
+			r.filters.push(fi);
+			timer.add(showdrc);
+		}
+
 		if (Math.round(r.chw * 25) != 25) {
 			var split = actx.createChannelSplitter(2),
 				merge = actx.createChannelMerger(2),
@@ -2726,6 +2781,31 @@ var afilt = (function () {
 		clmod(that, 'err', err);
 	}
 
+	function adj_drc() {
+		var err = false;
+		try {
+			var n = this.getAttribute('k'),
+				ov = r.drcv[n],
+				vs = this.value,
+				v = parseFloat(vs);
+
+			if (!isNum(v) || v + '' != vs)
+				throw new Error('inval v');
+
+			if (v == ov)
+				return;
+
+			r.drcv[n] = v;
+			jwrite('au_drcv', r.drcv);
+			if (r.drcn)
+				r.drcn[r.drck[n]].value = v;
+		}
+		catch (ex) {
+			err = true;
+		}
+		clmod(this, 'err', err);
+	}
+
 	function eq_mod(e) {
 		ev(e);
 		adj_band(this, 0);
@@ -2737,6 +2817,13 @@ var afilt = (function () {
 			adj_band(this, step);
 	}
 
+	function showdrc() {
+		if (!r.drcn)
+			return timer.rm(showdrc);
+
+		ebi('h_drc').textContent = f2f(r.drcn.reduction, 1);
+	}
+
 	var html = ['<table><tr><td rowspan="4">',
 		'<a id="au_eq" class="tgl btn" href="#" tt="' + L.mt_eq + '">enable</a></td>'],
 		h2 = [], h3 = [], h4 = [];
@@ -2766,6 +2853,18 @@ var afilt = (function () {
 	html += h4.join('\n') + '</tr><table>';
 	ebi('audio_eq').innerHTML = html;
 
+	h2 = [];
+	html = ['<table><tr><td rowspan="2">',
+		'<a id="au_drc" class="tgl btn" href="#" tt="' + L.mt_drc + '">enable</a></td>'];
+
+	for (var a = 0; a < r.drch.length; a++) {
+		html.push('<td>' + r.drch[a] + '</td>');
+		h2.push('<td><input type="text" class="drc_v" ' + NOAC + ' k="' + a + '" value="' + r.drcv[a] + '" /></td>');
+	}
+	html = html.join('\n') + '</tr><tr>';
+	html += h2.join('\n') + '</tr><table>';
+	ebi('audio_drc').innerHTML = html;
+
 	var stp = QSA('a.eq_step');
 	for (var a = 0, aa = stp.length; a < aa; a++)
 		stp[a].onclick = eq_step;
@@ -2775,8 +2874,12 @@ var afilt = (function () {
 		txt[a].oninput = eq_mod;
 		txt[a].onkeydown = eq_keydown;
 	}
+	txt = QSA('input.drc_v');
+	for (var a = 0; a < txt.length; a++)
+		txt[a].oninput = txt[a].onkeydown = adj_drc;
 
 	bcfg_bind(r, 'eqen', 'au_eq', false, r.apply);
+	bcfg_bind(r, 'drcen', 'au_drc', false, r.apply);
 
 	r.draw();
 	return r;
@@ -3694,7 +3797,7 @@ var fileman = (function () {
 
 			function rename_cb() {
 				if (this.status !== 201) {
-					var msg = this.responseText;
+					var msg = unpre(this.responseText);
 					toast.err(9, L.fr_efail + msg);
 					return;
 				}
@@ -3743,7 +3846,7 @@ var fileman = (function () {
 		}
 		function delete_cb() {
 			if (this.status !== 200) {
-				var msg = this.responseText;
+				var msg = unpre(this.responseText);
 				toast.err(9, L.fd_err + msg);
 				return;
 			}
@@ -3864,7 +3967,7 @@ var fileman = (function () {
 		}
 		function paste_cb() {
 			if (this.status !== 201) {
-				var msg = this.responseText;
+				var msg = unpre(this.responseText);
 				toast.err(9, L.fp_err + msg);
 				return;
 			}
@@ -4197,7 +4300,7 @@ var showfile = (function () {
 	};
 
 	r.mktree = function () {
-		var html = ['<li class="bn">' + L.tv_lst + '<br />' + linksplit(get_vpath()).join('') + '</li>'];
+		var html = ['<li class="bn">' + L.tv_lst + '<br />' + linksplit(get_vpath()).join('<span>/</span>') + '</li>'];
 		for (var a = 0; a < r.files.length; a++) {
 			var file = r.files[a];
 			html.push('<li><a href="?doc=' +
@@ -4995,6 +5098,11 @@ document.onkeydown = function (e) {
 		[
 			L.s_ad,
 			["adv", "adv", L.s_a1, "30", "key>=1A  key<=2B  .bpm>165"]
+		],
+		[
+			L.s_ua,
+			["utl", "ut_min", L.s_u1, "14", "2007-04-08"],
+			["utu", "ut_max", L.s_u2, "14", "2038-01-19"]
 		]
 	];
 
@@ -5139,7 +5247,7 @@ document.onkeydown = function (e) {
 					}
 
 					if (k.length == 3) {
-						q += k.replace(/sz/, 'size').replace(/dt/, 'date').replace(/l$/, ' >= ').replace(/u$/, ' <= ') + tv;
+						q += k.replace(/l$/, ' >= ').replace(/u$/, ' <= ').replace(/^sz/, 'size').replace(/^dt/, 'date').replace(/^ut/, 'up_at') + tv;
 						continue;
 					}
 
@@ -5192,10 +5300,7 @@ document.onkeydown = function (e) {
 
 	function xhr_search_results() {
 		if (this.status !== 200) {
-			var msg = this.responseText;
-			if (msg.indexOf('<pre>') === 0)
-				msg = msg.slice(5);
-
+			var msg = unpre(this.responseText);
 			srch_msg(true, "http " + this.status + ": " + msg);
 			search_in_progress = 0;
 			return;
@@ -5234,7 +5339,7 @@ document.onkeydown = function (e) {
 			if (ext.length > 8)
 				ext = '%';
 
-			var links = linksplit(r.rp + '', id).join(''),
+			var links = linksplit(r.rp + '', id).join('<span>/</span>'),
 				nodes = ['<tr><td>-</td><td><div>' + links + '</div>', sz];
 
 			for (var b = 0; b < tagord.length; b++) {
@@ -5956,7 +6061,8 @@ var treectl = (function () {
 			}
 
 			if (tn.lead == '-')
-				tn.lead = '<a href="?doc=' + tn.href + '" class="doc' + (lang ? ' bri' : '') +
+				tn.lead = '<a href="?doc=' + bhref +
+					'" class="doc' + (lang ? ' bri' : '') +
 					'" hl="' + id + '" name="' + hname + '">-txt-</a>';
 
 			var ln = ['<tr><td>' + tn.lead + '</td><td><a href="' +
@@ -7059,16 +7165,17 @@ var msel = (function () {
 	form.onsubmit = function (e) {
 		ev(e);
 		clmod(sf, 'vis', 1);
-		sf.textContent = 'creating "' + tb.value + '"...';
+		var dn = tb.value;
+		sf.textContent = 'creating "' + dn + '"...';
 
 		var fd = new FormData();
 		fd.append("act", "mkdir");
-		fd.append("name", tb.value);
+		fd.append("name", dn);
 
 		var xhr = new XHR();
 		xhr.vp = get_evpath();
-		xhr.dn = tb.value;
-		xhr.open('POST', xhr.vp, true);
+		xhr.dn = dn;
+		xhr.open('POST', dn.startsWith('/') ? (SR || '/') : xhr.vp, true);
 		xhr.onload = xhr.onerror = cb;
 		xhr.responseType = 'text';
 		xhr.send(fd);
@@ -7085,7 +7192,7 @@ var msel = (function () {
 		xhrchk(this, L.fd_xe1, L.fd_xe2);
 
 		if (this.status !== 201) {
-			sf.textContent = 'error: ' + this.responseText;
+			sf.textContent = 'error: ' + unpre(this.responseText);
 			return;
 		}
 
@@ -7094,8 +7201,9 @@ var msel = (function () {
 		sf.textContent = '';
 
 		var dn = this.getResponseHeader('X-New-Dir');
-		dn = dn || uricom_enc(this.dn);
-		treectl.goto(this.vp + dn + '/', true);
+		dn = dn ? '/' + dn + '/' : uricom_enc(this.dn);
+		treectl.goto(dn, true);
+		tree_scrollto();
 	}
 })();
 
@@ -7132,7 +7240,7 @@ var msel = (function () {
 		xhrchk(this, L.fsm_xe1, L.fsm_xe2);
 
 		if (this.status < 200 || this.status > 201) {
-			sf.textContent = 'error: ' + this.responseText;
+			sf.textContent = 'error: ' + unpre(this.responseText);
 			return;
 		}
 
@@ -7477,7 +7585,7 @@ var unpost = (function () {
 					'<tr><td><a me="' + me + '" class="n' + a + '" href="#">' + L.un_del + '</a></td>' +
 					'<td>' + unix2iso(res[a].at) + '</td>' +
 					'<td>' + res[a].sz + '</td>' +
-					'<td>' + linksplit(res[a].vp).join(' ') + '</td></tr>');
+					'<td>' + linksplit(res[a].vp).join('<span> / </span>') + '</td></tr>');
 			}
 
 			html.push("</tbody></table>");
@@ -7510,7 +7618,7 @@ var unpost = (function () {
 
 	function unpost_delete_cb() {
 		if (this.status !== 200) {
-			var msg = this.responseText;
+			var msg = unpre(this.responseText);
 			toast.err(9, L.un_derr + msg);
 			return;
 		}

copyparty/web/svcs.html

@@ -10,6 +10,7 @@
 {{ html_head }}
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/splash.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
+	<style>ul{padding-left:1.3em}li{margin:.4em 0}</style>
 </head>
 
 <body>
@@ -48,9 +49,13 @@
                 rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud pacer_min_sleep=0.01ms{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-dav:{{ rvp }} <b>W:</b>
             </pre>
-            {% if s %}
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>--no-check-certificate</code> to the mount command</em><br />---</p>
-            {% endif %}
+            <ul>
+                {% if s %}
+                <li>running <code>rclone mount</code> on LAN (or just dont have valid certificates)? add <code>--no-check-certificate</code></li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             
             <p>if you want to use the native WebDAV client in windows instead (slow and buggy), first run <a href="{{ r }}/.cpr/a/webdav-cfg.bat">webdav-cfg.bat</a> to remove the 47 MiB filesize limit (also fixes latency and password login), then connect:</p>
             <pre>
@@ -73,10 +78,13 @@
                 rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud pacer_min_sleep=0.01ms{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-dav:{{ rvp }} <b>mp</b>
             </pre>
-            {% if s %}
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>--no-check-certificate</code> to the mount command</em><br />---</p>
-            {% endif %}
-            
+            <ul>
+                {% if s %}
+                <li>running <code>rclone mount</code> on LAN (or just dont have valid certificates)? add <code>--no-check-certificate</code></li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             <p>or the emergency alternative (gnome/gui-only):</p>
             <!-- gnome-bug: ignores vp -->
             <pre>
@@ -123,8 +131,14 @@
                 rclone config create {{ aname }}-ftps ftp host={{ rip }} port={{ args.ftps }} pass=k user={% if accs %}<b>{{ pw }}</b>{% else %}anonymous{% endif %} tls=false explicit_tls=true
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-ftps:{{ rvp }} <b>W:</b>
             </pre>
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>no_check_certificate=true</code> to the config command</em><br />---</p>
             {% endif %}
+            <ul>
+                {% if args.ftps %}
+                <li>running on LAN (or just dont have valid certificates)? add <code>no_check_certificate=true</code> to the config command</li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             <p>if you want to use the native FTP client in windows instead (please dont), press <code>win+R</code> and run this command:</p>
             <pre>
                 explorer {{ "ftp" if args.ftp else "ftps" }}://{% if accs %}<b>{{ pw }}</b>:k@{% endif %}{{ host }}:{{ args.ftp or args.ftps }}/{{ rvp }}
@@ -145,8 +159,14 @@
                 rclone config create {{ aname }}-ftps ftp host={{ rip }} port={{ args.ftps }} pass=k user={% if accs %}<b>{{ pw }}</b>{% else %}anonymous{% endif %} tls=false explicit_tls=true
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-ftps:{{ rvp }} <b>mp</b>
             </pre>
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>no_check_certificate=true</code> to the config command</em><br />---</p>
             {% endif %}
+            <ul>
+                {% if args.ftps %}
+                <li>running on LAN (or just dont have valid certificates)? add <code>no_check_certificate=true</code> to the config command</li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             <p>emergency alternative (gnome/gui-only):</p>
             <!-- gnome-bug: ignores vp -->
             <pre>
@@ -178,7 +198,7 @@
             partyfuse.py{% if accs %} -a <b>{{ pw }}</b>{% endif %} http{{ s }}://{{ ep }}/{{ rvp }} <b><span class="os win">W:</span><span class="os lin mac">mp</span></b>
         </pre>
         {% if s %}
-        <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>-td</code></em></p>
+        <ul><li>if you are on LAN (or just dont have valid certificates), add <code>-td</code></li></ul>
         {% endif %}
         <p>
             you can use <a href="{{ r }}/.cpr/a/u2c.py">u2c.py</a> to upload (sometimes faster than web-browsers)

copyparty/web/ui.css

@@ -1,3 +1,18 @@
+:root {
+	--fg: #ccc;
+	--fg-max: #fff;
+	--bg-u2: #2b2b2b;
+	--bg-u5: #444;
+}
+html.y {
+	--fg: #222;
+	--fg-max: #000;
+	--bg-u2: #f7f7f7;
+	--bg-u5: #ccc;
+}
+html.bz {
+	--bg-u2: #202231;
+}
 @font-face {
 	font-family: 'scp';
 	font-display: swap;
@@ -14,6 +29,7 @@ html {
 	max-width: min(34em, 90%);
 	max-width: min(34em, calc(100% - 7em));
 	color: #ddd;
+	color: var(--fg);
 	background: #333;
 	background: var(--bg-u2);
 	border: 0 solid #777;
@@ -171,24 +187,15 @@ html {
 	color: #f6a;
 }
 html.y #tt {
-	color: #333;
-	background: #fff;
 	border-color: #888 #000 #777 #000;
 }
 html.bz #tt {
-	background: #202231;
 	border-color: #3b3f58;
 }
 html.y #tt,
 html.y #toast {
 	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
 }
-html.y #tt code {
-	color: #fff;
-	color: var(--fg-max);
-	background: #060;
-	background: var(--bg-u5);
-}
 #modalc code {
 	color: #060;
 	background: transparent;
@@ -326,6 +333,9 @@ html.y .btn:focus {
 	box-shadow: 0 .1em .2em #037 inset;
 	outline: #037 solid .1em;
 }
+input[type="submit"] {
+	cursor: pointer;
+}
 input[type="text"]:focus,
 input:not([type]):focus,
 textarea:focus {

copyparty/web/up2k.js

@@ -1340,6 +1340,7 @@ function up2k_init(subtle) {
 
     function up_them(good_files) {
         start_actx();
+        draw_turbo();
         var evpath = get_evpath(),
             draw_each = good_files.length < 50;
 
@@ -1406,7 +1407,7 @@ function up2k_init(subtle) {
 
             pvis.addfile([
                 uc.fsearch ? esc(entry.name) : linksplit(
-                    entry.purl + uricom_enc(entry.name)).join(' '),
+                    entry.purl + uricom_enc(entry.name)).join(' / '),
                 '📐 ' + L.u_hashing,
                 ''
             ], entry.size, draw_each);
@@ -2283,7 +2284,7 @@ function up2k_init(subtle) {
                                 cdiff = (Math.abs(diff) <= 2) ? '3c0' : 'f0b',
                                 sdiff = '<span style="color:#' + cdiff + '">diff ' + diff;
 
-                            msg.push(linksplit(hit.rp).join('') + '<br /><small>' + tr + ' (srv), ' + tu + ' (You), ' + sdiff + '</small></span>');
+                            msg.push(linksplit(hit.rp).join(' / ') + '<br /><small>' + tr + ' (srv), ' + tu + ' (You), ' + sdiff + '</small></span>');
                         }
                         msg = msg.join('<br />\n');
                     }
@@ -2317,7 +2318,7 @@ function up2k_init(subtle) {
                         url += '?k=' + fk;
                     }
 
-                    pvis.seth(t.n, 0, linksplit(url).join(' '));
+                    pvis.seth(t.n, 0, linksplit(url).join(' / '));
                 }
 
                 var chunksize = get_chunksize(t.size),
@@ -2401,15 +2402,12 @@ function up2k_init(subtle) {
                 pvis.seth(t.n, 2, L.u_ehstmp, t);
 
                 var err = "",
-                    rsp = (xhr.responseText + ''),
+                    rsp = unpre(xhr.responseText),
                     ofs = rsp.lastIndexOf('\nURL: ');
 
                 if (ofs !== -1)
                     rsp = rsp.slice(0, ofs);
 
-                if (rsp.indexOf('<pre>') === 0)
-                    rsp = rsp.slice(5);
-
                 if (rsp.indexOf('rate-limit ') !== -1) {
                     var penalty = rsp.replace(/.*rate-limit /, "").split(' ')[0];
                     console.log("rate-limit: " + penalty);
@@ -2428,7 +2426,7 @@ function up2k_init(subtle) {
                     err = rsp;
                     ofs = err.indexOf('\n/');
                     if (ofs !== -1) {
-                        err = err.slice(0, ofs + 1) + linksplit(err.slice(ofs + 2).trimEnd()).join(' ');
+                        err = err.slice(0, ofs + 1) + linksplit(err.slice(ofs + 2).trimEnd()).join(' / ');
                     }
                     if (!t.rechecks && (err_pend || err_srcb)) {
                         t.rechecks = 0;
@@ -2535,7 +2533,7 @@ function up2k_init(subtle) {
             cdr = t.size;
 
         var orz = function (xhr) {
-            var txt = ((xhr.response && xhr.response.err) || xhr.responseText) + '';
+            var txt = unpre((xhr.response && xhr.response.err) || xhr.responseText);
             if (txt.indexOf('upload blocked by x') + 1) {
                 apop(st.busy.upload, upt);
                 apop(t.postlist, npart);
@@ -2711,7 +2709,12 @@ function up2k_init(subtle) {
     function draw_turbo() {
         if (turbolvl < 0 && uc.turbo) {
             bcfg_set('u2turbo', uc.turbo = false);
-            toast.err(10, "turbo is disabled in server config");
+            toast.err(10, L.u_turbo_c);
+        }
+
+        if (uc.turbo && !has(perms, 'read')) {
+            bcfg_set('u2turbo', uc.turbo = false);
+            toast.warn(30, L.u_turbo_g);
         }
 
         var msg = (turbolvl || !uc.turbo) ? null : uc.fsearch ? L.u_ts : L.u_tu,

copyparty/web/util.js

@@ -622,9 +622,8 @@ function linksplit(rp, id) {
         }
         var vlink = esc(uricom_dec(link));
 
-        if (link.indexOf('/') !== -1) {
-            vlink = vlink.slice(0, -1) + '<span>/</span>';
-        }
+        if (link.indexOf('/') !== -1)
+            vlink = vlink.slice(0, -1);
 
         if (!rp) {
             if (q)
@@ -1357,6 +1356,11 @@ function lf2br(txt) {
 }
 
 
+function unpre(txt) {
+    return ('' + txt).replace(/^<pre>/, '');
+}
+
+
 var toast = (function () {
     var r = {},
         te = null,

docs/changelog.md

@@ -1,3 +1,124 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1104-2158  `v1.9.16`  windedup
+
+## breaking changes
+* two of the prometheus metrics have changed slightly; see the [breaking changes readme section](https://github.com/9001/copyparty#breaking-changes)
+  * (i'm not familiar with prometheus so i'm not sure if this is a big deal)
+
+## new features
+* #58 versioned docker images! no longer just `latest`
+* browser: the mkdir feature now accepts `foo/bar/qux` and `../foo` and `/bar`
+* add 14 more prometheus metrics; see [readme](https://github.com/9001/copyparty#prometheus) for details
+  * connections, requests, malicious requests, volume state, file hashing/analyzation queues
+* catch some more malicious requests in the autoban filters
+  * some malicious requests are now answered with HTTP 422, so that they count against `--ban-422`
+
+## bugfixes
+* windows: fix symlink-based upload deduplication
+  * MS decided to make symlinks relative to working-directory rather than destination-path...
+* `--stats` would produce invalid metrics if a volume was offline
+* minor improvements to password hashing ux:
+  * properly warn if `--ah-cli` or `--ah-gen` is used without `--ah-alg`
+  * support `^D` during `--ah-cli`
+* browser-ux / cosmetics:
+  * fix toast/tooltip colors on splashpage
+  * easier to do partial text selection inside links (search results, breadcrumbs, uploads)
+  * more rclone-related hints on the connect-page
+
+## other changes
+* malformed http headers from clients are no longer included in the client error-message
+  * just in case there are deployments with a reverse-proxy inserting interesting stuff on the way in
+  * the serverlog still contains all the necessary info to debug your own clients
+* updated [example nginx config](https://github.com/9001/copyparty/blob/hovudstraum/contrib/nginx/copyparty.conf) to recover faster from brief server outages
+  * the default value of `fail_timeout` (10sec) makes nginx cache the outage for longer than necessary
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1024-1643  `v1.9.15`  expand placeholder
+
+[made it just in time!](https://a.ocv.me/pub/g/nerd-stuff/PXL_20231024_170348367.jpg) (EDIT: nevermind, three of the containers didn't finish uploading to ghcr before takeoff ;_; all up now)
+
+## new features
+* #56 placeholder variables in markdown documents and prologue/epilogue html files
+  * default-disabled; must be enabled globally with `--exp` or per-volume with volflag `exp`
+  * `{{self.ip}}` becomes the client IP; see [/srv/expand/README.md](https://github.com/9001/copyparty/blob/hovudstraum/srv/expand/README.md) for more examples
+* dynamic-range-compressor: reduced volume jumps between songs when enabled
+
+## bugfixes
+* v1.9.14 broke the `scan` volflag, causing volume rescans to happen every 10sec if enabled
+  * its global counterpart `--re-maxage` was not affected
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1021-1443  `v1.9.14`  uptime
+
+## new features
+* search for files by upload time
+* option to display upload time in directory listings
+  * enable globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at`
+  * has a ~17% performance impact on directory listings
+* [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression) in the audioplayer settings
+* `--ban-404` is now default-enabled
+  * the turbo-uploader will now un-turbo when necessary to avoid banning itself
+  * this only affects accounts with permissions `g`, `G`, or `h`
+    * accounts with read-access (which are able to see directory listings anyways) and accounts with write-only access are no longer affected by `--ban-404` or `--ban-url`
+
+## bugfixes
+* #55 clients could hit the `--url-ban` filter when uploading over webdav
+  * fixed by limiting `--ban-404` and `--ban-url` to accounts with permission `g`, `G`, or `h`
+* fixed 20% performance drop in python 3.12 due to utcfromtimestamp deprecation
+  * but 3.12.0 is still 5% slower than 3.11.6 for some reason
+* volume listing on startup would display some redundant info
+
+## other changes
+* timeout for unfinished uploads increased from 6 to 24 hours
+  * and is now configurable with `--snap-drop`
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1015-2006  `v1.9.12`  more buttons
+
+just adding requested features, nothing important
+
+## new features
+* button `📅` in the uploader (default-enabled) sends your local last-modified timestamps to the server
+  * when deselected, the files on the server will have the upload time as their timestamps instead
+  * `--u2ts` specifies the default setting, `c` client-last-modified or `u` upload-time, or `fc` and `fu` to force
+* button `full` in the gridview decides if thumbnails should be center-cropped or not
+  * `--no-crop` and the `nocrop` volflag now sets the default value of this instead of forcing the setting
+  * thumbnail cleanup is now more granular, cleaning full-jpg separately from cropped-webp for example
+* set default sort order with `--sort` or volflag `sort`
+  * one or more comma-separated values; `tags/Cirle,tags/.tn,tags/Artist,tags/Title,href`
+    * see the column header tooltips in the browser to know what names (`id`) to use
+  * prefix a column name with `-` for descending sort
+  * specifying a sort order in the client will override all server-defined ones
+* when visiting a read-only folder, the upload-or-filesearch toggle will remember its previous state and restore it when leaving the folder
+  * much more intuitive, if anything about this UI can be called that...
+
+## bugfixes
+* iPhone: rare javascript panic when switching between safari and another app
+* ie9: file-rename ui was borked
+
+## other changes
+* copyparty.exe: upgrade to pillow 10.1 (which adds a new font for thumbnails in chrome)
+  * still based on python 3.11.6 because 3.12 is currently slower than 3.11
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1009-0036  `v1.9.11`  bustin'
+
+okay, i swear this is the last version for weeks! probably
+
+## bugfixes
+* cachebuster didn't apply to dynamically loaded javascript files
+  * READMEs could fail to render with `ReferenceError: DOMPurify is not defined` after upgrading from a copyparty older than v1.9.2
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2023-1008-2051  `v1.9.10`  badpwd
 

docs/lics.txt

@@ -28,10 +28,6 @@ https://github.com/nayuki/QR-Code-generator/
 C: Project Nayuki
 L: MIT
 
-https://github.com/python/cpython/blob/3.10/Lib/asyncore.py
-C: 1996 Sam Rushing
-L: ISC
-
 https://github.com/ahupp/python-magic/
 C: 2001-2014 Adam Hupp
 L: MIT

scripts/docker/make.sh

@@ -141,12 +141,25 @@ filt=
 }
 
 [ $push ] && {
+    ver=$(
+        python3 ../../dist/copyparty-sfx.py --version 2>/dev/null |
+        awk '/^copyparty v/{sub(/-.*/,"");sub(/v/,"");print$2;exit}'
+    )
+    echo $ver | grep -E '[0-9]\.[0-9]' || {
+        echo no ver
+        exit 1
+    }
     for i in $dhub_order; do
+        printf '\ndockerhub %s\n' $i
+        podman manifest push --all copyparty-$i copyparty/$i:$ver
         podman manifest push --all copyparty-$i copyparty/$i:latest
-    done
+    done &
     for i in $ghcr_order; do
+        printf '\nghcr %s\n' $i
+        podman manifest push --all copyparty-$i ghcr.io/9001/copyparty-$i:$ver
         podman manifest push --all copyparty-$i ghcr.io/9001/copyparty-$i:latest
-    done
+    done &
+    wait
 }
 
 echo ok

scripts/make-sfx.sh

@@ -205,26 +205,22 @@ necho() {
 	mv {markupsafe,jinja2} j2/
 
 	necho collecting pyftpdlib
-	f="../build/pyftpdlib-1.5.8.tar.gz"
+	f="../build/pyftpdlib-1.5.9.tar.gz"
 	[ -e "$f" ] ||
-		(url=https://github.com/giampaolo/pyftpdlib/archive/refs/tags/release-1.5.8.tar.gz;
+		(url=https://github.com/giampaolo/pyftpdlib/archive/refs/tags/release-1.5.9.tar.gz;
 		wget -O$f "$url" || curl -L "$url" >$f)
 
 	tar -zxf $f
 	mv pyftpdlib-release-*/pyftpdlib .
 	rm -rf pyftpdlib-release-* pyftpdlib/test
+	for f in pyftpdlib/_async{hat,ore}.py; do
+		[ -e "$f" ] || continue;
+		iawk 'NR<4||NR>27||!/^#/;NR==4{print"# license: https://opensource.org/licenses/ISC\n"}' $f
+	done
 
 	mkdir ftp/
 	mv pyftpdlib ftp/
 
-	necho collecting asyncore, asynchat
-	for n in asyncore.py asynchat.py; do
-		f=../build/$n
-		[ -e "$f" ] ||
-			(url=https://raw.githubusercontent.com/python/cpython/c4d45ee670c09d4f6da709df072ec80cb7dfad22/Lib/$n;
-			wget -O$f "$url" || curl -L "$url" >$f)
-	done
-
 	necho collecting python-magic
 	v=0.4.27
 	f="../build/python-magic-$v.tar.gz"
@@ -293,12 +289,6 @@ necho() {
 		(cd "${x%/*}"; cp -p "../$(cat "${x##*/}")" ${x##*/})
 	done
 
-	# insert asynchat
-	mkdir copyparty/vend
-	for n in asyncore.py asynchat.py; do
-		awk 'NR<4||NR>27;NR==4{print"# license: https://opensource.org/licenses/ISC\n"}' ../build/$n >copyparty/vend/$n
-	done
-
 	rm -f copyparty/stolen/*/README.md
 
 	# remove type hints before build instead
@@ -419,7 +409,7 @@ iawk '/^ {0,4}[^ ]/{s=0}/^ {4}def (serve_forever|_loop)/{s=1}!s' ftp/pyftpdlib/s
 rm -f ftp/pyftpdlib/{__main__,prefork}.py
 
 [ $no_ftp ] &&
-	rm -rf copyparty/ftpd.py ftp asyncore.py asynchat.py &&
+	rm -rf copyparty/ftpd.py ftp &&
 	sed -ri '/\.ftp/d' copyparty/svchub.py
 
 [ $no_smb ] &&
@@ -576,8 +566,8 @@ nf=$(ls -1 "$zdir"/arc.* 2>/dev/null | wc -l)
 	cat ../$bdir/COPYING.txt) >> copyparty/res/COPYING.txt ||
 		echo "copying.txt 404 pls rebuild"
 
-	mv ftp/* j2/* copyparty/vend/* .
-	rm -rf ftp j2 py2 py37 copyparty/vend
+	mv ftp/* j2/* .
+	rm -rf ftp j2 py2 py37
 	(cd copyparty; tar -cvf z.tar $t; rm -rf $t)
 	cd ..
 	pyoxidizer build --release --target-triple $tgt

scripts/pyinstaller/loader.py

@@ -106,20 +106,19 @@ def meichk():
     if filt not in sys.executable:
         filt = os.path.basename(sys.executable)
 
-    pids = []
-    ptn = re.compile(r"^([^\s]+)\s+([0-9]+)")
+    hits = []
     try:
-        procs = sp.check_output("tasklist").decode("utf-8", "replace")
+        cmd = "tasklist /fo csv".split(" ")
+        procs = sp.check_output(cmd).decode("utf-8", "replace")
     except:
         procs = ""  # winpe
 
-    for ln in procs.splitlines():
-        m = ptn.match(ln)
-        if m and filt in m.group(1).lower():
-            pids.append(int(m.group(2)))
+    for ln in procs.split("\n"):
+        if filt in ln.split('"')[:2][-1]:
+            hits.append(ln)
 
     mod = os.path.dirname(os.path.realpath(__file__))
-    if os.path.basename(mod).startswith("_MEI") and len(pids) == 2:
+    if os.path.basename(mod).startswith("_MEI") and len(hits) == 2:
         meicln(mod)
 
 

scripts/sfx.ls

@@ -59,9 +59,6 @@ copyparty/th_srv.py,
 copyparty/u2idx.py,
 copyparty/up2k.py,
 copyparty/util.py,
-copyparty/vend,
-copyparty/vend/asynchat.py,
-copyparty/vend/asyncore.py,
 copyparty/web,
 copyparty/web/a,
 copyparty/web/a/__init__.py,

scripts/sfx.py

@@ -295,7 +295,10 @@ def unpack():
         # the only possible input is a single tar.bz2
         # which gets hardcoded into this script at build stage
         # skip 0
-        tf.extractall(mine)
+        try:
+            tf.extractall(mine, filter="tar")
+        except TypeError:
+            tf.extractall(mine)
 
     os.remove(tar)
 

scripts/test/smoketest.py

@@ -97,6 +97,7 @@ def tc1(vflags):
         ovid = f.read()
 
     args = [
+        "-q",
         "-p4321",
         "-e2dsa",
         "-e2tsr",

scripts/uncomment.py

@@ -16,16 +16,11 @@ def uncomment(fpath):
         orig = f.read().decode("utf-8")
 
     out = ""
-    for ln in orig.split("\n"):
-        if not ln.startswith("#"):
-            break
-
-        out += ln + "\n"
-
     io_obj = io.StringIO(orig)
     prev_toktype = tokenize.INDENT
     last_lineno = -1
     last_col = 0
+    code = False
     for tok in tokenize.generate_tokens(io_obj.readline):
         # print(repr(tok))
         token_type = tok[0]
@@ -53,7 +48,11 @@ def uncomment(fpath):
                 out += token_string
             else:
                 out += '"a"'
-        elif token_type != tokenize.COMMENT or is_legalese:
+        elif token_type != tokenize.COMMENT:
+            out += token_string
+            if not code and token_string.strip():
+                code = True
+        elif is_legalese or (not start_col and not code):
             out += token_string
         else:
             if out.rstrip(" ").endswith("\n"):

srv/expand/README.md

@@ -0,0 +1,26 @@
+## text expansion
+
+enable expansion of placeholder variables in `README.md` and prologue/epilogue files with `--exp` and customize the list of allowed placeholders to expand using `--exp-md` and `--exp-lg`
+
+| explanation          | placeholder          |
+| -------------------- | -------------------- |
+| your ip address      | {{self.ip}}          |
+| your user-agent      | {{self.ua}}          |
+| your username        | {{self.uname}}       |
+| the `Host` you see   | {{self.host}}        |
+| server unix time     | {{srv.itime}}        |
+| server datetime      | {{srv.htime}}        |
+| server name          | {{cfg.name}}         |
+| logout after         | {{cfg.logout}} hours |
+| vol reindex interval | {{vf.scan}}          |
+| thumbnail size       | {{vf.thsize}}        |
+| your country         | {{hdr.cf_ipcountry}} |
+
+placeholders starting with...
+* `self.` are grabbed from copyparty's internal state; anything in `httpcli.py` is fair game
+* `cfg.` are the global server settings
+* `vf.` are the volflags of the current volume
+* `hdr.` are grabbed from the client headers; any header is supported, just add it (in lowercase) to the allowlist
+* `srv.` are processed inside the `_expand` function in httpcli
+
+for example (bad example), `hdr_cf_ipcountry` maps to the header `CF-IPCountry` (which is generated by cloudflare before the request is passed on to your server / copyparty)

tests/util.py

@@ -109,13 +109,13 @@ class Cfg(Namespace):
     def __init__(self, a=None, v=None, c=None):
         ka = {}
 
-        ex = "daw dav_auth dav_inf dav_mac dav_rt dotsrch e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp ed emp force_js getmod grid hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_logues no_mv no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw rand smb th_no_crop vague_403 vc ver xdev xlink xvol"
+        ex = "daw dav_auth dav_inf dav_mac dav_rt dotsrch e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp ed emp exp force_js getmod grid hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_logues no_mv no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw rand smb th_no_crop vague_403 vc ver xdev xlink xvol"
         ka.update(**{k: False for k in ex.split()})
 
         ex = "dotpart no_rescan no_sendfile no_voldump plain_ip"
         ka.update(**{k: True for k in ex.split()})
 
-        ex = "css_browser hist js_browser no_forget no_hash no_idx nonsus_urls"
+        ex = "ah_cli ah_gen css_browser hist js_browser no_forget no_hash no_idx nonsus_urls"
         ka.update(**{k: None for k in ex.split()})
 
         ex = "s_thead s_tbody th_convt"
@@ -124,12 +124,15 @@ class Cfg(Namespace):
         ex = "df loris re_maxage rproxy rsp_jtr rsp_slp s_wr_slp theme themes turbo"
         ka.update(**{k: 0 for k in ex.split()})
 
-        ex = "ah_alg bname doctitle favico html_head lg_sbf log_fk md_sbf mth name textfiles unlist vname R RS SR"
+        ex = "ah_alg bname doctitle favico html_head lg_sbf log_fk md_sbf name textfiles unlist vname R RS SR"
         ka.update(**{k: "" for k in ex.split()})
 
         ex = "on403 on404 xad xar xau xban xbd xbr xbu xiu xm"
         ka.update(**{k: [] for k in ex.split()})
 
+        ex = "exp_lg exp_md"
+        ka.update(**{k: {} for k in ex.split()})
+
         super(Cfg, self).__init__(
             a=a or [],
             v=v or [],
@@ -141,8 +144,11 @@ class Cfg(Namespace):
             fk_salt="a" * 16,
             unpost=600,
             u2sort="s",
+            u2ts="c",
+            sort="href",
             mtp=[],
-            mte="a",
+            mte={"a": True},
+            mth={},
             lang="eng",
             logout=573,
             **ka
@@ -184,6 +190,7 @@ class VHttpSrv(object):
         self.broker = NullBroker()
         self.prism = None
         self.bans = {}
+        self.nreq = 0
 
         aliases = ["splash", "browser", "browser2", "msg", "md", "mde"]
         self.j2 = {x: J2_FILES for x in aliases}