v1.9.16

* cpp_uptime is now a gauge
* cpp_bans is now cpp_active_bans (and also a gauge)

and other related fixes:
* stop emitting invalid cpp_disk_size/free for offline volumes
* support overriding the spec-mandatory mimetype with ?mime=foo

.vscode/launch.json

@@ -9,6 +9,10 @@
             "console": "integratedTerminal",
             "cwd": "${workspaceFolder}",
             "justMyCode": false,
+            "env": {
+                "PYDEVD_DISABLE_FILE_VALIDATION": "1",
+                "PYTHONWARNINGS": "always", //error
+            },
             "args": [
                 //"-nw",
                 "-ed",

.vscode/launch.py

@@ -41,7 +41,7 @@ if sfx:
     argv = [sys.executable, sfx] + argv
     sp.check_call(argv)
 elif re.search(" -j ?[0-9]", " ".join(argv)):
-    argv = [sys.executable, "-m", "copyparty"] + argv
+    argv = [sys.executable, "-Wa", "-m", "copyparty"] + argv
     sp.check_call(argv)
 else:
     sys.path.insert(0, os.getcwd())

.vscode/tasks.json

@@ -11,6 +11,7 @@
             "type": "shell",
             "command": "${config:python.pythonPath}",
             "args": [
+                "-Wa", //-We
                 ".vscode/launch.py"
             ]
         }

README.md

@@ -20,9 +20,8 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [testimonials](#testimonials) - small collection of user feedback
 * [motivations](#motivations) - project goals / philosophy
     * [notes](#notes) - general notes
-* [bugs](#bugs)
-    * [general bugs](#general-bugs)
-    * [not my bugs](#not-my-bugs)
+* [bugs](#bugs) - roughly sorted by chance of encounter
+    * [not my bugs](#not-my-bugs) - same order here too
 * [breaking changes](#breaking-changes) - upgrade notes
 * [FAQ](#FAQ) - "frequently" asked questions
 * [accounts and volumes](#accounts-and-volumes) - per-folder, per-user permissions
@@ -40,7 +39,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [file manager](#file-manager) - cut/paste, rename, and delete files/folders (if you have permission)
     * [batch rename](#batch-rename) - select some files and press `F2` to bring up the rename UI
     * [media player](#media-player) - plays almost every audio format there is
-        * [audio equalizer](#audio-equalizer) - bass boosted
+        * [audio equalizer](#audio-equalizer) - and [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression)
         * [fix unreliable playback on android](#fix-unreliable-playback-on-android) - due to phone / app settings
     * [markdown viewer](#markdown-viewer) - and there are *two* editors
     * [other tricks](#other-tricks)
@@ -54,6 +53,7 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [webdav server](#webdav-server) - with read-write support
         * [connecting to webdav from windows](#connecting-to-webdav-from-windows) - using the GUI
     * [smb server](#smb-server) - unsafe, slow, not recommended for wan
+    * [browser ux](#browser-ux) - tweaking the ui
     * [file indexing](#file-indexing) - enables dedup and music search ++
         * [exclude-patterns](#exclude-patterns) - to save some time
         * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
@@ -263,20 +263,28 @@ server notes:
 
 # bugs
 
-* Windows: python 2.7 cannot index non-ascii filenames with `-e2d`
-* Windows: python 2.7 cannot handle filenames with mojibake
-* `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions (macos, some linux)
-* `--th-ff-swr` may fix audio thumbnails on some FFmpeg versions
+roughly sorted by chance of encounter
 
-## general bugs
+* general:
+  * `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions (macos, some linux)
+  * `--th-ff-swr` may fix audio thumbnails on some FFmpeg versions
+  * if the `up2k.db` (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
+    * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
+  * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
+  * probably more, pls let me know
 
-* Windows: if the `up2k.db` (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
-  * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
-* all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
-* probably more, pls let me know
+* python 3.4 and older (including 2.7):
+  * many rare and exciting edge-cases because [python didn't handle EINTR yet](https://peps.python.org/pep-0475/)
+    * downloads from copyparty may suddenly fail, but uploads *should* be fine
+
+* python 2.7 on Windows:
+  * cannot index non-ascii filenames with `-e2d`
+  * cannot handle filenames with mojibake
 
 ## not my bugs
 
+same order here too
+
 * [Chrome issue 1317069](https://bugs.chromium.org/p/chromium/issues/detail?id=1317069) -- if you try to upload a folder which contains symlinks by dragging it into the browser, the symlinked files will not get uploaded
 
 * [Chrome issue 1352210](https://bugs.chromium.org/p/chromium/issues/detail?id=1352210) -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive)
@@ -310,6 +318,8 @@ server notes:
 
 upgrade notes
 
+* `1.9.16` (2023-11-04):
+  * `--stats`/prometheus: `cpp_bans` renamed to `cpp_active_bans`, and that + `cpp_uptime` are gauges
 * `1.6.0` (2023-01-29):
   * http-api: delete/move is now `POST` instead of `GET`
   * everything other than `GET` and `HEAD` must pass [cors validation](#cors)
@@ -359,7 +369,7 @@ permissions:
 * `g` (get): only download files, cannot see folder contents or zip/tar
 * `G` (upget): same as `g` except uploaders get to see their own [filekeys](#filekeys) (see `fk` in examples below)
 * `h` (html): same as `g` except folders return their index.html, and filekeys are not necessary for index.html
-* `a` (admin): can see uploader IPs, config-reload
+* `a` (admin): can see upload time, uploader IPs, config-reload
 
 examples:
 * add accounts named u1, u2, u3 with passwords p1, p2, p3: `-a u1:p1 -a u2:p2 -a u3:p3`
@@ -580,7 +590,8 @@ the up2k UI is the epitome of polished inutitive experiences:
 * "parallel uploads" specifies how many chunks to upload at the same time
 * `[🏃]` analysis of other files should continue while one is uploading
 * `[🥔]` shows a simpler UI for faster uploads from slow devices
-* `[💭]` ask for confirmation before files are added to the queue
+* `[🎲]` generate random filenames during upload
+* `[📅]` preserve last-modified timestamps; server times will match yours
 * `[🔎]` switch between upload and [file-search](#file-search) mode
   * ignore `[🔎]` if you add files by dragging them into the browser
 
@@ -735,7 +746,7 @@ open the `[🎺]` media-player-settings tab to configure it,
 
 ### audio equalizer
 
-bass boosted
+and [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression)
 
 can also boost the volume in general, or increase/decrease stereo width (like [crossfeed](https://www.foobar2000.org/components/view/foo_dsp_meiercf) just worse)
 
@@ -781,6 +792,8 @@ other notes,
 
 * files named `README.md` / `readme.md` will be rendered after directory listings unless `--no-readme` (but `.epilogue.html` takes precedence)
 
+* `README.md` and `*logue.html` can contain placeholder values which are replaced server-side before embedding into directory listings; see `--help-exp`
+
 
 ## searching
 
@@ -946,6 +959,16 @@ authenticate with one of the following:
 * username `$password`, password `k`
 
 
+## browser ux
+
+tweaking the ui
+
+* set default sort order globally with `--sort` or per-volume with the `sort` volflag; specify one or more comma-separated columns to sort by, and prefix the column name with `-` for reverse sort
+  * the column names you can use are visible as tooltips when hovering over the column headers in the directory listing, for example `href ext sz ts tags/.up_at tags/Cirle tags/.tn tags/Artist tags/Title`
+  * to sort in music order (album, track, artist, title) with filename as fallback, you could `--sort tags/Cirle,tags/.tn,tags/Artist,tags/Title,href`
+  * to sort by upload date, first enable showing the upload date in the listing with `-e2d -mte +.up_at` and then `--sort tags/.up_at`
+
+
 ## file indexing
 
 enables dedup and music search ++
@@ -972,6 +995,7 @@ the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`,
 * `-v ~/music::r:c,d2ts` same except only affecting tags
 
 note:
+* upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
 * deduplication is possible on windows if you run copyparty as administrator (not saying you should!)
@@ -1283,8 +1307,23 @@ scrape_configs:
 ```
 
 currently the following metrics are available,
-* `cpp_uptime_seconds`
-* `cpp_bans` number of banned IPs
+* `cpp_uptime_seconds` time since last copyparty restart
+* `cpp_boot_unixtime_seconds` same but as an absolute timestamp
+* `cpp_http_conns` number of open http(s) connections
+* `cpp_http_reqs` number of http(s) requests handled
+* `cpp_sus_reqs` number of 403/422/malicious requests
+* `cpp_active_bans` number of currently banned IPs
+* `cpp_total_bans` number of IPs banned since last restart
+
+these are available unless `--nos-vst` is specified:
+* `cpp_db_idle_seconds` time since last database activity (upload/rename/delete)
+* `cpp_db_act_seconds` same but as an absolute timestamp
+* `cpp_idle_vols` number of volumes which are idle / ready
+* `cpp_busy_vols` number of volumes which are busy / indexing
+* `cpp_offline_vols` number of volumes which are offline / unavailable
+* `cpp_hashing_files` number of files queued for hashing / indexing
+* `cpp_tagq_files` number of files queued for metadata scanning
+* `cpp_mtpq_files` number of files queued for plugin-based analysis
 
 and these are available per-volume only:
 * `cpp_disk_size_bytes` total HDD size
@@ -1303,9 +1342,12 @@ some of the metrics have additional requirements to function correctly,
 the following options are available to disable some of the metrics:
 * `--nos-hdd` disables `cpp_disk_*` which can prevent spinning up HDDs
 * `--nos-vol` disables `cpp_vol_*` which reduces server startup time
+* `--nos-vst` disables volume state, reducing the worst-case prometheus query time by 0.5 sec
 * `--nos-dup` disables `cpp_dupe_*` which reduces the server load caused by prometheus queries
 * `--nos-unf` disables `cpp_unf_*` for no particular purpose
 
+note: the following metrics are counted incorrectly if multiprocessing is enabled with `-j`: `cpp_http_conns`, `cpp_http_reqs`, `cpp_sus_reqs`, `cpp_active_bans`, `cpp_total_bans`
+
 
 # packages
 
@@ -1635,8 +1677,6 @@ safety profiles:
   * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
     * however note if you edit one file it will also affect the other copies
   * `--vague-403` returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
-  * `--ban-404=50,60,1440` ban client for 1440min (24h) if they hit 50 404's in 60min
-    * `--turbo=-1` to force-disable turbo-mode in the uploader which could otherwise hit the 404-ban
   * `--nih` removes the server hostname from directory listings
 
 * option `-sss` is a shortcut for the above plus:

bin/hooks/msg-log.py

@@ -7,7 +7,11 @@ import json
 import os
 import sys
 import time
-from datetime import datetime
+
+try:
+    from datetime import datetime, timezone
+except:
+    from datetime import datetime
 
 
 """
@@ -96,7 +100,11 @@ def main(argv=None):
     msg_info = json.loads(sys.argv[1])
     # print(msg_info)
 
-    dt = datetime.utcfromtimestamp(msg_info["at"])
+    try:
+        dt = datetime.fromtimestamp(msg_info["at"], timezone.utc)
+    except:
+        dt = datetime.utcfromtimestamp(msg_info["at"])
+
     msg_info["datetime"] = dt.strftime("%Y-%m-%d, %H:%M:%S")
 
     msg_text = TEMPLATE % msg_info

bin/hooks/notify2.py

@@ -4,7 +4,7 @@ import json
 import os
 import sys
 import subprocess as sp
-from datetime import datetime
+from datetime import datetime, timezone
 from plyer import notification
 
 
@@ -43,7 +43,8 @@ def main():
     fp = inf["ap"]
     sz = humansize(inf["sz"])
     dp, fn = os.path.split(fp)
-    mt = datetime.utcfromtimestamp(inf["mt"]).strftime("%Y-%m-%d %H:%M:%S")
+    dt = datetime.fromtimestamp(inf["mt"], timezone.utc)
+    mt = dt.strftime("%Y-%m-%d %H:%M:%S")
 
     msg = f"{fn} ({sz})\n📁 {dp}"
     title = "File received"

bin/hooks/xiu-sha.py

@@ -3,7 +3,7 @@
 import hashlib
 import json
 import sys
-from datetime import datetime
+from datetime import datetime, timezone
 
 
 _ = r"""
@@ -43,8 +43,11 @@ except:
         return p
 
 
+UTC = timezone.utc
+
+
 def humantime(ts):
-    return datetime.utcfromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
+    return datetime.fromtimestamp(ts, UTC).strftime("%Y-%m-%d %H:%M:%S")
 
 
 def find_files_root(inf):
@@ -96,7 +99,7 @@ def main():
 
     ret.append("# {} files, {} bytes total".format(len(inf), total_sz))
     ret.append("")
-    ftime = datetime.utcnow().strftime("%Y-%m%d-%H%M%S.%f")
+    ftime = datetime.now(UTC).strftime("%Y-%m%d-%H%M%S.%f")
     fp = "{}xfer-{}.sha512".format(inf[0]["ap"][:di], ftime)
     with open(fsenc(fp), "wb") as f:
         f.write("\n".join(ret).encode("utf-8", "replace"))

bin/partyfuse.py

@@ -46,12 +46,13 @@ import traceback
 import http.client  # py2: httplib
 import urllib.parse
 import calendar
-from datetime import datetime
+from datetime import datetime, timezone
 from urllib.parse import quote_from_bytes as quote
 from urllib.parse import unquote_to_bytes as unquote
 
 WINDOWS = sys.platform == "win32"
 MACOS = platform.system() == "Darwin"
+UTC = timezone.utc
 info = log = dbg = None
 
 
@@ -176,7 +177,7 @@ class RecentLog(object):
     def put(self, msg):
         msg = "{:10.6f} {} {}\n".format(time.time() % 900, rice_tid(), msg)
         if self.f:
-            fmsg = " ".join([datetime.utcnow().strftime("%H%M%S.%f"), str(msg)])
+            fmsg = " ".join([datetime.now(UTC).strftime("%H%M%S.%f"), str(msg)])
             self.f.write(fmsg.encode("utf-8"))
 
         with self.mtx:

bin/partyjournal.py

@@ -20,12 +20,13 @@ import sys
 import base64
 import sqlite3
 import argparse
-from datetime import datetime
+from datetime import datetime, timezone
 from urllib.parse import quote_from_bytes as quote
 from urllib.parse import unquote_to_bytes as unquote
 
 
 FS_ENCODING = sys.getfilesystemencoding()
+UTC = timezone.utc
 
 
 class APF(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
@@ -155,11 +156,10 @@ th {
                 link = txt.decode("utf-8")[4:]
 
         sz = "{:,}".format(sz)
+        dt = datetime.fromtimestamp(at if at > 0 else mt, UTC)
         v = [
             w[:16],
-            datetime.utcfromtimestamp(at if at > 0 else mt).strftime(
-                "%Y-%m-%d %H:%M:%S"
-            ),
+            dt.strftime("%Y-%m-%d %H:%M:%S"),
             sz,
             imap.get(ip, ip),
         ]

contrib/nginx/copyparty.conf

@@ -13,7 +13,7 @@
 # on fedora/rhel, remember to setsebool -P httpd_can_network_connect 1
 
 upstream cpp {
-	server 127.0.0.1:3923;
+	server 127.0.0.1:3923 fail_timeout=1s;
 	keepalive 1;
 }
 server {

contrib/package/arch/PKGBUILD

@@ -1,8 +1,8 @@
 # Maintainer: icxes <dev.null@need.moe>
 pkgname=copyparty
-pkgver="1.9.7"
+pkgver="1.9.15"
 pkgrel=1
-pkgdesc="Portable file sharing hub"
+pkgdesc="File server with accelerated resumable uploads, dedup, WebDAV, FTP, zeroconf, media indexer, thumbnails++"
 arch=("any")
 url="https://github.com/9001/${pkgname}"
 license=('MIT')
@@ -20,7 +20,7 @@ optdepends=("ffmpeg: thumbnails for videos, images (slower) and audio, music tag
 )
 source=("https://github.com/9001/${pkgname}/releases/download/v${pkgver}/${pkgname}-${pkgver}.tar.gz")
 backup=("etc/${pkgname}.d/init" )
-sha256sums=("1d427392568ed2048b06b2049cb884f650ff1466e2f57779f0c263d3ae1db311")
+sha256sums=("ee569d664b22cb59ac0eb11850380648d9f8d42d1c26283d43dab350745c102e")
 
 build() {
     cd "${srcdir}/${pkgname}-${pkgver}"

contrib/package/nix/copyparty/pin.json

@@ -1,5 +1,5 @@
 {
-    "url": "https://github.com/9001/copyparty/releases/download/v1.9.7/copyparty-sfx.py",
-    "version": "1.9.7",
-    "hash": "sha256-VBvzdHNSTI6qPk1HESbpllzdP9mrKwfr4qbPlGTaabg="
+    "url": "https://github.com/9001/copyparty/releases/download/v1.9.15/copyparty-sfx.py",
+    "version": "1.9.15",
+    "hash": "sha256-EUenh567NYj1klMpjVOWKqiBSqZbdEA0ZGidzzzpnsY="
 }

contrib/rc/copyparty

@@ -10,7 +10,7 @@ name="copyparty"
 rcvar="copyparty_enable"
 copyparty_user="copyparty"
 copyparty_args="-e2dsa -v /storage:/storage:r" # change as you see fit
-copyparty_command="/usr/local/bin/python3.8 /usr/local/copyparty/copyparty-sfx.py ${copyparty_args}"
+copyparty_command="/usr/local/bin/python3.9 /usr/local/copyparty/copyparty-sfx.py ${copyparty_args}"
 pidfile="/var/run/copyparty/${name}.pid"
 command="/usr/sbin/daemon"
 command_args="-P ${pidfile} -r -f ${copyparty_command}"

copyparty/__main__.py

@@ -27,6 +27,9 @@ from .authsrv import expand_config_file, re_vol, split_cfg_ln, upgrade_cfg_fmt
 from .cfg import flagcats, onedash
 from .svchub import SvcHub
 from .util import (
+    DEF_EXP,
+    DEF_MTE,
+    DEF_MTH,
     IMPLICATIONS,
     JINJA_VER,
     PYFTPD_VER,
@@ -184,7 +187,10 @@ def init_E(E: EnvParams) -> None:
 
         with open_binary("copyparty", "z.tar") as tgz:
             with tarfile.open(fileobj=tgz) as tf:
-                tf.extractall(tdn)  # nosec (archive is safe)
+                try:
+                    tf.extractall(tdn, filter="tar")
+                except TypeError:
+                    tf.extractall(tdn)  # nosec (archive is safe)
 
         return tdn
 
@@ -641,6 +647,47 @@ def get_sects():
             """
             ),
         ],
+        [
+            "exp",
+            "text expansion",
+            dedent(
+                """
+            specify --exp or the "exp" volflag to enable placeholder expansions
+            in README.md / .prologue.html / .epilogue.html
+
+            --exp-md (volflag exp_md) holds the list of placeholders which can be
+            expanded in READMEs, and --exp-lg (volflag exp_lg) likewise for logues;
+            any placeholder not given in those lists will be ignored and shown as-is
+
+            the default list will expand the following placeholders:
+            \033[36m{{self.ip}}     \033[35mclient ip
+            \033[36m{{self.ua}}     \033[35mclient user-agent
+            \033[36m{{self.uname}}  \033[35mclient username
+            \033[36m{{self.host}}   \033[35mthe "Host" header, or the server's external IP otherwise
+            \033[36m{{cfg.name}}    \033[35mthe --name global-config
+            \033[36m{{cfg.logout}}  \033[35mthe --logout global-config
+            \033[36m{{vf.scan}}     \033[35mthe "scan" volflag
+            \033[36m{{vf.thsize}}   \033[35mthumbnail size
+            \033[36m{{srv.itime}}   \033[35mserver time in seconds
+            \033[36m{{srv.htime}}   \033[35mserver time as YY-mm-dd, HH:MM:SS (UTC)
+            \033[36m{{hdr.cf_ipcountry}} \033[35mthe "CF-IPCountry" client header (probably blank)
+            \033[0m
+            so the following types of placeholders can be added to the lists:
+            * any client header can be accessed through {{hdr.*}}
+            * any variable in httpcli.py can be accessed through {{self.*}}
+            * any global server setting can be accessed through {{cfg.*}}
+            * any volflag can be accessed through {{vf.*}}
+
+            remove vf.scan from default list using --exp-md /vf.scan
+            add "accept" header to def. list using --exp-md +hdr.accept
+
+            for performance reasons, expansion only happens while embedding
+            documents into directory listings, and when accessing a ?doc=...
+            link, but never otherwise, so if you click a -txt- link you'll
+            have to refresh the page to apply expansion
+            """
+            ),
+        ],
         [
             "ls",
             "volume inspection",
@@ -771,8 +818,6 @@ def add_general(ap, nc, srvname):
     ap2.add_argument("-a", metavar="ACCT", type=u, action="append", help="add account, \033[33mUSER\033[0m:\033[33mPASS\033[0m; example [\033[32med:wark\033[0m]")
     ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, \033[33mSRC\033[0m:\033[33mDST\033[0m:\033[33mFLAG\033[0m; examples [\033[32m.::r\033[0m], [\033[32m/mnt/nas/music:/music:r:aed\033[0m]")
     ap2.add_argument("-ed", action="store_true", help="enable the ?dots url parameter / client option which allows clients to see dotfiles / hidden files")
-    ap2.add_argument("-emp", action="store_true", help="enable markdown plugins -- neat but dangerous, big XSS risk")
-    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
     ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see --help-urlform")
     ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="window title, for example [\033[32m$ip-10.1.2.\033[0m] or [\033[32m$ip-]")
     ap2.add_argument("--name", metavar="TXT", type=u, default=srvname, help="server name (displayed topleft in browser and in mDNS)")
@@ -803,9 +848,12 @@ def add_upload(ap):
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
     ap2.add_argument("--hardlink", action="store_true", help="prefer hardlinks instead of symlinks when possible (within same filesystem) (volflag=hardlink)")
     ap2.add_argument("--never-symlink", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=neversymlink)")
-    ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead (volflag=copydupes")
+    ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead (volflag=copydupes)")
     ap2.add_argument("--no-dupe", action="store_true", help="reject duplicate files during upload; only matches within the same volume (volflag=nodupe)")
     ap2.add_argument("--no-snap", action="store_true", help="disable snapshots -- forget unfinished uploads on shutdown; don't create .hist/up2k.snap files -- abandoned/interrupted uploads must be cleaned up manually")
+    ap2.add_argument("--snap-wri", metavar="SEC", type=int, default=300, help="write upload state to ./hist/up2k.snap every SEC seconds; allows resuming incomplete uploads after a server crash")
+    ap2.add_argument("--snap-drop", metavar="MIN", type=float, default=1440, help="forget unfinished uploads after MIN minutes; impossible to resume them after that (360=6h, 1440=24h)")
+    ap2.add_argument("--u2ts", metavar="TXT", type=u, default="c", help="how to timestamp uploaded files; [\033[32mc\033[0m]=client-last-modified, [\033[32mu\033[0m]=upload-time, [\033[32mfc\033[0m]=force-c, [\033[32mfu\033[0m]=force-u (volflag=u2ts)")
     ap2.add_argument("--rand", action="store_true", help="force randomized filenames, --nrand chars long (volflag=rand)")
     ap2.add_argument("--nrand", metavar="NUM", type=int, default=9, help="randomized filenames length (volflag=nrand)")
     ap2.add_argument("--magic", action="store_true", help="enable filetype detection on nameless uploads (volflag=magic)")
@@ -966,6 +1014,7 @@ def add_stats(ap):
     ap2.add_argument("--stats", action="store_true", help="enable openmetrics at /.cpr/metrics for admin accounts")
     ap2.add_argument("--nos-hdd", action="store_true", help="disable disk-space metrics (used/free space)")
     ap2.add_argument("--nos-vol", action="store_true", help="disable volume size metrics (num files, total bytes, vmaxb/vmaxn)")
+    ap2.add_argument("--nos-vst", action="store_true", help="disable volume state metrics (indexing, analyzing, activity)")
     ap2.add_argument("--nos-dup", action="store_true", help="disable dupe-files metrics (good idea; very slow)")
     ap2.add_argument("--nos-unf", action="store_true", help="disable unfinished-uploads metrics")
 
@@ -995,7 +1044,7 @@ def add_optouts(ap):
 def add_safety(ap):
     ap2 = ap.add_argument_group('safety options')
     ap2.add_argument("-s", action="count", default=0, help="increase safety: Disable thumbnails / potentially dangerous software (ffmpeg/pillow/vips), hide partial uploads, avoid crawlers.\n └─Alias of\033[32m --dotpart --no-thumb --no-mtag-ff --no-robots --force-js")
-    ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, webdav, 404 on 403, ban on excessive 404s.\n └─Alias of\033[32m -s --unpost=0 --no-del --no-mv --hardlink --vague-403 --ban-404=50,60,1440 --turbo=-1 -nih")
+    ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, webdav, 404 on 403, ban on excessive 404s.\n └─Alias of\033[32m -s --unpost=0 --no-del --no-mv --hardlink --vague-403 -nih")
     ap2.add_argument("-sss", action="store_true", help="further increase safety: Enable logging to disk, scan for dangerous symlinks.\n └─Alias of\033[32m -ss --no-dav --no-logues --no-readme -lo=cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz --ls=**,*,ln,p,r")
     ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="do a sanity/safety check of all volumes on startup; arguments \033[33mUSER\033[0m,\033[33mVOL\033[0m,\033[33mFLAGS\033[0m; example [\033[32m**,*,ln,p,r\033[0m]")
     ap2.add_argument("--xvol", action="store_true", help="never follow symlinks leaving the volume root, unless the link is into another volume where the user has similar access (volflag=xvol)")
@@ -1009,10 +1058,10 @@ def add_safety(ap):
     ap2.add_argument("--no-robots", action="store_true", help="adds http and html headers asking search engines to not index anything (volflag=norobots)")
     ap2.add_argument("--logout", metavar="H", type=float, default="8086", help="logout clients after H hours of inactivity; [\033[32m0.0028\033[0m]=10sec, [\033[32m0.1\033[0m]=6min, [\033[32m24\033[0m]=day, [\033[32m168\033[0m]=week, [\033[32m720\033[0m]=month, [\033[32m8760\033[0m]=year)")
     ap2.add_argument("--ban-pw", metavar="N,W,B", type=u, default="9,60,1440", help="more than \033[33mN\033[0m wrong passwords in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; disable with [\033[32mno\033[0m]")
-    ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="no", help="hitting more than \033[33mN\033[0m 404's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (disabled by default since turbo-up2k counts as 404s)")
+    ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="50,60,1440", help="hitting more than \033[33mN\033[0m 404's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; only affects users who cannot see directory listings because their access is either g/G/h")
     ap2.add_argument("--ban-403", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 403's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; [\033[32m1440\033[0m]=day, [\033[32m10080\033[0m]=week, [\033[32m43200\033[0m]=month")
     ap2.add_argument("--ban-422", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m 422's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (422 is server fuzzing, invalid POSTs and so)")
-    ap2.add_argument("--ban-url", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m sus URL's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes (decent replacement for --ban-404 if that can't be used)")
+    ap2.add_argument("--ban-url", metavar="N,W,B", type=u, default="9,2,1440", help="hitting more than \033[33mN\033[0m sus URL's in \033[33mW\033[0m minutes = ban for \033[33mB\033[0m minutes; applies only to access g/G/h (decent replacement for --ban-404 if that can't be used)")
     ap2.add_argument("--sus-urls", metavar="R", type=u, default=r"\.php$|(^|/)wp-(admin|content|includes)/", help="URLs which are considered sus / eligible for banning; disable with blank or [\033[32mno\033[0m]")
     ap2.add_argument("--nonsus-urls", metavar="R", type=u, default=r"^(favicon\.ico|robots\.txt)$|^apple-touch-icon|^\.well-known", help="harmless URLs ignored from 404-bans; disable with blank or [\033[32mno\033[0m]")
     ap2.add_argument("--aclose", metavar="MIN", type=int, default=10, help="if a client maxes out the server connection limit, downgrade it from connection:keep-alive to connection:close for MIN minutes (and also kill its active connections) -- disable with 0")
@@ -1046,6 +1095,7 @@ def add_logging(ap):
     ap2.add_argument("--ansi", action="store_true", help="force colors; overrides environment-variable NO_COLOR")
     ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
     ap2.add_argument("--log-tdec", metavar="N", type=int, default=3, help="timestamp resolution / number of timestamp decimals")
+    ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log failed login attempt passwords: 0=terse, 1=plaintext, 2=hashed")
     ap2.add_argument("--log-conn", action="store_true", help="debug: print tcp-server msgs")
     ap2.add_argument("--log-htp", action="store_true", help="debug: print http-server threadpool scaling")
     ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header")
@@ -1067,7 +1117,7 @@ def add_thumbnail(ap):
     ap2.add_argument("--th-size", metavar="WxH", default="320x256", help="thumbnail res (volflag=thsize)")
     ap2.add_argument("--th-mt", metavar="CORES", type=int, default=CORES, help="num cpu cores to use for generating thumbnails")
     ap2.add_argument("--th-convt", metavar="SEC", type=float, default=60, help="conversion timeout in seconds (volflag=convt)")
-    ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image (volflag=nocrop)")
+    ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image by default (volflag=nocrop)")
     ap2.add_argument("--th-dec", metavar="LIBS", default="vips,pil,ff", help="image decoders, in order of preference")
     ap2.add_argument("--th-no-jpg", action="store_true", help="disable jpg output")
     ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output")
@@ -1080,9 +1130,9 @@ def add_thumbnail(ap):
     # https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
     # https://github.com/libvips/libvips
     # ffmpeg -hide_banner -demuxers | awk '/^ D  /{print$2}' | while IFS= read -r x; do ffmpeg -hide_banner -h demuxer=$x; done | grep -E '^Demuxer |extensions:'
-    ap2.add_argument("--th-r-pil", metavar="T,T", type=u, default="avif,avifs,blp,bmp,dcx,dds,dib,emf,eps,fits,flc,fli,fpx,gif,heic,heics,heif,heifs,icns,ico,im,j2p,j2k,jp2,jpeg,jpg,jpx,pbm,pcx,pgm,png,pnm,ppm,psd,sgi,spi,tga,tif,tiff,webp,wmf,xbm,xpm", help="image formats to decode using pillow")
+    ap2.add_argument("--th-r-pil", metavar="T,T", type=u, default="avif,avifs,blp,bmp,dcx,dds,dib,emf,eps,fits,flc,fli,fpx,gif,heic,heics,heif,heifs,icns,ico,im,j2p,j2k,jp2,jpeg,jpg,jpx,pbm,pcx,pgm,png,pnm,ppm,psd,qoi,sgi,spi,tga,tif,tiff,webp,wmf,xbm,xpm", help="image formats to decode using pillow")
     ap2.add_argument("--th-r-vips", metavar="T,T", type=u, default="avif,exr,fit,fits,fts,gif,hdr,heic,jp2,jpeg,jpg,jpx,jxl,nii,pfm,pgm,png,ppm,svg,tif,tiff,webp", help="image formats to decode using pyvips")
-    ap2.add_argument("--th-r-ffi", metavar="T,T", type=u, default="apng,avif,avifs,bmp,dds,dib,fit,fits,fts,gif,hdr,heic,heics,heif,heifs,icns,ico,jp2,jpeg,jpg,jpx,jxl,pbm,pcx,pfm,pgm,png,pnm,ppm,psd,sgi,tga,tif,tiff,webp,xbm,xpm", help="image formats to decode using ffmpeg")
+    ap2.add_argument("--th-r-ffi", metavar="T,T", type=u, default="apng,avif,avifs,bmp,dds,dib,fit,fits,fts,gif,hdr,heic,heics,heif,heifs,icns,ico,jp2,jpeg,jpg,jpx,jxl,pbm,pcx,pfm,pgm,png,pnm,ppm,psd,qoi,sgi,tga,tif,tiff,webp,xbm,xpm", help="image formats to decode using ffmpeg")
     ap2.add_argument("--th-r-ffv", metavar="T,T", type=u, default="3gp,asf,av1,avc,avi,flv,h264,h265,hevc,m4v,mjpeg,mjpg,mkv,mov,mp4,mpeg,mpeg2,mpegts,mpg,mpg2,mts,nut,ogm,ogv,rm,ts,vob,webm,wmv", help="video formats to decode using ffmpeg")
     ap2.add_argument("--th-r-ffa", metavar="T,T", type=u, default="aac,ac3,aif,aiff,alac,alaw,amr,apac,ape,au,bonk,dfpwm,dts,flac,gsm,ilbc,it,m4a,mo3,mod,mp2,mp3,mpc,mptm,mt2,mulaw,ogg,okt,opus,ra,s3m,tak,tta,ulaw,wav,wma,wv,xm,xpk", help="audio formats to decode using ffmpeg")
 
@@ -1130,19 +1180,27 @@ def add_db_metadata(ap):
     ap2.add_argument("--mtag-v", action="store_true", help="verbose tag scanning; print errors from mtp subprocesses and such")
     ap2.add_argument("--mtag-vv", action="store_true", help="debug mtp settings and mutagen/ffprobe parsers")
     ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
-    ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
-        default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,fmt,res,.fps,ahash,vhash,up_ip,.up_at")
-    ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.)",
-        default=".vq,.aq,vc,ac,fmt,res,.fps")
+    ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.); either an entire replacement list, or add/remove stuff on the default-list with +foo or /bar", default=DEF_MTE)
+    ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.); assign/add/remove same as -mte", default=DEF_MTH)
     ap2.add_argument("-mtp", metavar="M=[f,]BIN", type=u, action="append", help="read tag M using program BIN to parse the file")
 
 
+def add_txt(ap):
+    ap2 = ap.add_argument_group('textfile options')
+    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="textfile editor checks for serverside changes every SEC seconds")
+    ap2.add_argument("-emp", action="store_true", help="enable markdown plugins -- neat but dangerous, big XSS risk")
+    ap2.add_argument("--exp", action="store_true", help="enable textfile expansion -- replace {{self.ip}} and such; see --help-exp (volflag=exp)")
+    ap2.add_argument("--exp-md", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in markdown files; add/remove stuff on the default list with +hdr_foo or /vf.scan (volflag=exp_md)")
+    ap2.add_argument("--exp-lg", metavar="V,V,V", type=u, default=DEF_EXP, help="comma/space-separated list of placeholders to expand in prologue/epilogue files (volflag=exp_lg)")
+
+
 def add_ui(ap, retry):
     ap2 = ap.add_argument_group('ui options')
     ap2.add_argument("--grid", action="store_true", help="show grid/thumbnails by default (volflag=grid)")
     ap2.add_argument("--lang", metavar="LANG", type=u, default="eng", help="language; one of the following: eng nor")
     ap2.add_argument("--theme", metavar="NUM", type=int, default=0, help="default theme to use")
     ap2.add_argument("--themes", metavar="NUM", type=int, default=8, help="number of themes installed")
+    ap2.add_argument("--sort", metavar="C,C,C", type=u, default="href", help="default sort order, comma-separated column IDs (see header tooltips), prefix with '-' for descending. Examples: \033[32mhref -href ext sz ts tags/Album tags/.tn\033[0m (volflag=sort)")
     ap2.add_argument("--unlist", metavar="REGEX", type=u, default="", help="don't show files matching REGEX in file list. Purely cosmetic! Does not affect API calls, just the browser. Example: [\033[32m\\.(js|css)$\033[0m] (volflag=unlist)")
     ap2.add_argument("--favico", metavar="TXT", type=u, default="c 000 none" if retry else "🎉 000 none", help="\033[33mfavicon-text\033[0m [ \033[33mforeground\033[0m [ \033[33mbackground\033[0m ] ], set blank to disable")
     ap2.add_argument("--mpmc", metavar="URL", type=u, default="", help="change the mediaplayer-toggle mouse cursor; URL to a folder with {2..5}.png inside (or disable with [\033[32m.\033[0m])")
@@ -1229,6 +1287,7 @@ def run_argparse(
     add_handlers(ap)
     add_hooks(ap)
     add_stats(ap)
+    add_txt(ap)
     add_ui(ap, retry)
     add_admin(ap)
     add_logging(ap)
@@ -1256,7 +1315,7 @@ def run_argparse(
     for k, h, t in sects:
         k2 = "help_" + k.replace("-", "_")
         if vars(ret)[k2]:
-            lprint("# {} help page".format(k))
+            lprint("# %s help page (%s)" % (k, h))
             lprint(t + "\033[0m")
             sys.exit(0)
 

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 9, 8)
+VERSION = (1, 9, 16)
 CODENAME = "prometheable"
-BUILD_DT = (2023, 10, 6)
+BUILD_DT = (2023, 11, 4)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -12,7 +12,7 @@ import threading
 import time
 from datetime import datetime
 
-from .__init__ import ANYWIN, E, TYPE_CHECKING, WINDOWS
+from .__init__ import ANYWIN, TYPE_CHECKING, WINDOWS, E
 from .bos import bos
 from .cfg import flagdescs, permdescs, vf_bmap, vf_cmap, vf_vmap
 from .pwhash import PWHash
@@ -21,11 +21,14 @@ from .util import (
     META_NOBOTS,
     SQLITE_VER,
     UNPLICATIONS,
+    ODict,
     Pebkac,
+    UTC,
     absreal,
     afsenc,
     get_df,
     humansize,
+    odfusion,
     relchk,
     statdir,
     uncyg,
@@ -213,7 +216,7 @@ class Lim(object):
             if self.rot_re.search(path.replace("\\", "/")):
                 return path, ""
 
-            suf = datetime.utcnow().strftime(self.rotf)
+            suf = datetime.now(UTC).strftime(self.rotf)
             if path:
                 path += "/"
 
@@ -473,12 +476,10 @@ class VFS(object):
         err: int = 403,
     ) -> tuple["VFS", str]:
         """returns [vfsnode,fs_remainder] if user has the requested permissions"""
-        if ANYWIN:
-            mod = relchk(vpath)
-            if mod:
-                if self.log:
-                    self.log("vfs", "invalid relpath [{}]".format(vpath))
-                raise Pebkac(404)
+        if relchk(vpath):
+            if self.log:
+                self.log("vfs", "invalid relpath [{}]".format(vpath))
+            raise Pebkac(422)
 
         cvpath = undot(vpath)
         vn, rem = self._find(cvpath)
@@ -497,8 +498,8 @@ class VFS(object):
                     t = "{} has no {} in [{}] => [{}] => [{}]"
                     self.log("vfs", t.format(uname, msg, vpath, cvpath, ap), 6)
 
-                t = "you don't have {}-access for this location"
-                raise Pebkac(err, t.format(msg))
+                t = 'you don\'t have %s-access in "/%s"'
+                raise Pebkac(err, t % (msg, cvpath))
 
         return vn, rem
 
@@ -1426,12 +1427,12 @@ class AuthSrv(object):
 
             for ga, vf in [["no_hash", "nohash"], ["no_idx", "noidx"]]:
                 if vf in vol.flags:
-                    ptn = vol.flags.pop(vf)
+                    ptn = re.compile(vol.flags.pop(vf))
                 else:
                     ptn = getattr(self.args, ga)
 
                 if ptn:
-                    vol.flags[vf] = re.compile(ptn)
+                    vol.flags[vf] = ptn
 
             for ga, vf in vf_bmap().items():
                 if getattr(self.args, ga):
@@ -1476,14 +1477,11 @@ class AuthSrv(object):
                 raise Exception(t.format(dbd, dbds))
 
             # default tag cfgs if unset
-            if "mte" not in vol.flags:
-                vol.flags["mte"] = self.args.mte
-            elif vol.flags["mte"].startswith("+"):
-                vol.flags["mte"] = ",".join(
-                    x for x in [self.args.mte, vol.flags["mte"][1:]] if x
-                )
-            if "mth" not in vol.flags:
-                vol.flags["mth"] = self.args.mth
+            for k in ("mte", "mth", "exp_md", "exp_lg"):
+                if k not in vol.flags:
+                    vol.flags[k] = getattr(self.args, k).copy()
+                else:
+                    vol.flags[k] = odfusion(getattr(self.args, k), vol.flags[k])
 
             # append additive args from argv to volflags
             hooks = "xbu xau xiu xbr xar xbd xad xm xban".split()
@@ -1537,7 +1535,11 @@ class AuthSrv(object):
                 if vol.flags.get(grp, False):
                     continue
 
-                vol.flags = {k: v for k, v in vol.flags.items() if not k.startswith(rm)}
+                vol.flags = {
+                    k: v
+                    for k, v in vol.flags.items()
+                    if not k.startswith(rm) or k == "mte"
+                }
 
             for grp, rm in [["d2v", "e2v"]]:
                 if not vol.flags.get(grp, False):
@@ -1584,12 +1586,12 @@ class AuthSrv(object):
                 if local:
                     local_only_mtp[a] = True
 
-            local_mte = {}
-            for a in vol.flags.get("mte", "").split(","):
+            local_mte = ODict()
+            for a in vol.flags.get("mte", {}).keys():
                 local = True
                 all_mte[a] = True
                 local_mte[a] = True
-                for b in self.args.mte.split(","):
+                for b in self.args.mte.keys():
                     if not a or not b:
                         continue
 
@@ -1719,6 +1721,9 @@ class AuthSrv(object):
     def setup_pwhash(self, acct: dict[str, str]) -> None:
         self.ah = PWHash(self.args)
         if not self.ah.on:
+            if self.args.ah_cli or self.args.ah_gen:
+                t = "\n  BAD CONFIG:\n    cannot --ah-cli or --ah-gen without --ah-alg"
+                raise Exception(t)
             return
 
         if self.args.ah_cli:

copyparty/cfg.py

@@ -13,18 +13,26 @@ def vf_bmap() -> dict[str, str]:
         "no_dedup": "copydupes",
         "no_dupe": "nodupe",
         "no_forget": "noforget",
+        "no_robots": "norobots",
+        "no_thumb": "dthumb",
+        "no_vthumb": "dvthumb",
+        "no_athumb": "dathumb",
         "th_no_crop": "nocrop",
         "dav_auth": "davauth",
         "dav_rt": "davrt",
     }
     for k in (
         "dotsrch",
+        "e2d",
+        "e2ds",
+        "e2dsa",
         "e2t",
         "e2ts",
         "e2tsr",
         "e2v",
         "e2vu",
         "e2vp",
+        "exp",
         "grid",
         "hardlink",
         "magic",
@@ -41,8 +49,22 @@ def vf_bmap() -> dict[str, str]:
 
 def vf_vmap() -> dict[str, str]:
     """argv-to-volflag: simple values"""
-    ret = {"th_convt": "convt", "th_size": "thsize"}
-    for k in ("dbd", "lg_sbf", "md_sbf", "nrand", "unlist"):
+    ret = {
+        "no_hash": "nohash",
+        "no_idx": "noidx",
+        "re_maxage": "scan",
+        "th_convt": "convt",
+        "th_size": "thsize",
+    }
+    for k in (
+        "dbd",
+        "lg_sbf",
+        "md_sbf",
+        "nrand",
+        "sort",
+        "unlist",
+        "u2ts",
+    ):
         ret[k] = k
     return ret
 
@@ -50,7 +72,23 @@ def vf_vmap() -> dict[str, str]:
 def vf_cmap() -> dict[str, str]:
     """argv-to-volflag: complex/lists"""
     ret = {}
-    for k in ("html_head", "mte", "mth"):
+    for k in (
+        "exp_lg",
+        "exp_md",
+        "html_head",
+        "mte",
+        "mth",
+        "mtp",
+        "xad",
+        "xar",
+        "xau",
+        "xban",
+        "xbd",
+        "xbr",
+        "xbu",
+        "xiu",
+        "xm",
+    ):
         ret[k] = k
     return ret
 
@@ -86,6 +124,7 @@ flagcats = {
         "vmaxn=4k": "max 4096 files in volume (suffixes: b, k, m, g, t)",
         "rand": "force randomized filenames, 9 chars long by default",
         "nrand=N": "randomized filenames are N chars long",
+        "u2ts=fc": "[f]orce [c]lient-last-modified or [u]pload-time",
         "sz=1k-3m": "allow filesizes between 1 KiB and 3MiB",
         "df=1g": "ensure 1 GiB free disk space",
     },
@@ -129,7 +168,7 @@ flagcats = {
         "dathumb": "disables audio thumbnails (spectrograms)",
         "dithumb": "disables image thumbnails",
         "thsize": "thumbnail res; WxH",
-        "nocrop": "disable center-cropping",
+        "nocrop": "disable center-cropping by default",
         "convt": "conversion timeout in seconds",
     },
     "handlers\n(better explained in --help-handlers)": {
@@ -149,6 +188,7 @@ flagcats = {
     },
     "client and ux": {
         "grid": "show grid/thumbnails by default",
+        "sort": "default sort order",
         "unlist": "dont list files matching REGEX",
         "html_head=TXT": "includes TXT in the <head>",
         "robots": "allows indexing by search engines (default)",
@@ -162,7 +202,8 @@ flagcats = {
         "nohtml": "return html and markdown as text/html",
     },
     "others": {
-        "fk=8": 'generates per-file accesskeys,\nwhich will then be required at the "g" permission',
+        "fk=8": 'generates per-file accesskeys,\nwhich are then required at the "g" permission;\nkeys are invalidated if filesize or inode changes',
+        "fka=8": 'generates slightly weaker per-file accesskeys,\nwhich are then required at the "g" permission;\nnot affected by filesize or inode numbers',
         "davauth": "ask webdav clients to login for all folders",
         "davrt": "show lastmod time of symlink destination, not the link itself\n(note: this option is always enabled for recursive listings)",
     },

copyparty/ftpd.py

@@ -92,6 +92,12 @@ class FtpAuth(DummyAuthorizer):
                 if bonk:
                     logging.warning("client banned: invalid passwords")
                     bans[ip] = bonk
+                    try:
+                        # only possible if multiprocessing disabled
+                        self.hub.broker.httpsrv.bans[ip] = bonk
+                        self.hub.broker.httpsrv.nban += 1
+                    except:
+                        pass
 
             raise AuthenticationFailed("Authentication failed.")
 
@@ -148,7 +154,7 @@ class FtpFs(AbstractedFS):
         try:
             vpath = vpath.replace("\\", "/").strip("/")
             rd, fn = os.path.split(vpath)
-            if ANYWIN and relchk(rd):
+            if relchk(rd):
                 logging.warning("malicious vpath: %s", vpath)
                 t = "Unsupported characters in [{}]"
                 raise FSE(t.format(vpath), 1)

copyparty/httpcli.py

@@ -7,6 +7,7 @@ import calendar
 import copy
 import errno
 import gzip
+import hashlib
 import itertools
 import json
 import os
@@ -36,10 +37,13 @@ from .star import StreamTar
 from .sutil import StreamArc, gfilter
 from .szip import StreamZip
 from .util import (
+    Garda,
     HTTPCODE,
     META_NOBOTS,
     MultipartParser,
+    ODict,
     Pebkac,
+    UTC,
     UnrecvEOF,
     absreal,
     alltrace,
@@ -72,6 +76,7 @@ from .util import (
     runhook,
     s3enc,
     sanitize_fn,
+    sanitize_vpath,
     sendfile_kern,
     sendfile_py,
     undot,
@@ -131,6 +136,7 @@ class HttpCli(object):
         self.mode = " "
         self.req = " "
         self.http_ver = " "
+        self.hint = ""
         self.host = " "
         self.ua = " "
         self.is_rclone = False
@@ -142,6 +148,8 @@ class HttpCli(object):
         self.rem = " "
         self.vpath = " "
         self.vpaths = " "
+        self.gctx = " "  # additional context for garda
+        self.trailing_slash = True
         self.uname = " "
         self.pw = " "
         self.rvol = [" "]
@@ -159,22 +167,17 @@ class HttpCli(object):
         self.can_get = False
         self.can_upget = False
         self.can_admin = False
+        self.out_headerlist: list[tuple[str, str]] = []
+        self.out_headers: dict[str, str] = {}
+        self.html_head = " "
         # post
         self.parser: Optional[MultipartParser] = None
         # end placeholders
 
         self.bufsz = 1024 * 32
-        self.hint = ""
-        self.trailing_slash = True
-        self.out_headerlist: list[tuple[str, str]] = []
-        self.out_headers = {
-            "Vary": "Origin, PW, Cookie",
-            "Cache-Control": "no-store, max-age=0",
-        }
         h = self.args.html_head
         if self.args.no_robots:
             h = META_NOBOTS + (("\n" + h) if h else "")
-            self.out_headers["X-Robots-Tag"] = "noindex, nofollow"
         self.html_head = h
 
     def log(self, msg: str, c: Union[int, str] = 0) -> None:
@@ -223,10 +226,12 @@ class HttpCli(object):
 
     def run(self) -> bool:
         """returns true if connection can be reused"""
-        self.keepalive = False
-        self.is_https = False
-        self.headers = {}
-        self.hint = ""
+        self.out_headers = {
+            "Vary": "Origin, PW, Cookie",
+            "Cache-Control": "no-store, max-age=0",
+        }
+        if self.args.no_robots:
+            self.out_headers["X-Robots-Tag"] = "noindex, nofollow"
 
         if self.is_banned():
             return False
@@ -252,8 +257,8 @@ class HttpCli(object):
                     k, zs = header_line.split(":", 1)
                     self.headers[k.lower()] = zs.strip()
             except:
-                msg = " ]\n#[ ".join(headerlines)
-                raise Pebkac(400, "bad headers:\n#[ " + msg + " ]")
+                msg = "#[ " + " ]\n#[ ".join(headerlines) + " ]"
+                raise Pebkac(400, "bad headers", log=msg)
 
         except Pebkac as ex:
             self.mode = "GET"
@@ -264,9 +269,15 @@ class HttpCli(object):
             h = {"WWW-Authenticate": 'Basic realm="a"'} if ex.code == 401 else {}
             try:
                 self.loud_reply(unicode(ex), status=ex.code, headers=h, volsan=True)
-                return self.keepalive
             except:
-                return False
+                pass
+
+            if ex.log:
+                self.log("additional error context:\n" + ex.log, 6)
+
+            return False
+
+        self.conn.hsrv.nreq += 1
 
         self.ua = self.headers.get("user-agent", "")
         self.is_rclone = self.ua.startswith("rclone/")
@@ -409,12 +420,9 @@ class HttpCli(object):
             self.vpath + "/" if self.trailing_slash and self.vpath else self.vpath
         )
 
-        ok = "\x00" not in self.vpath
-        if ANYWIN:
-            ok = ok and not relchk(self.vpath)
-
-        if not ok and (self.vpath != "*" or self.mode != "OPTIONS"):
+        if relchk(self.vpath) and (self.vpath != "*" or self.mode != "OPTIONS"):
             self.log("invalid relpath [{}]".format(self.vpath))
+            self.cbonk(self.conn.hsrv.g422, self.vpath, "bad_vp", "invalid relpaths")
             return self.tx_404() and self.keepalive
 
         zso = self.headers.get("authorization")
@@ -547,6 +555,9 @@ class HttpCli(object):
                 zb = b"<pre>" + html_escape(msg).encode("utf-8", "replace")
                 h = {"WWW-Authenticate": 'Basic realm="a"'} if pex.code == 401 else {}
                 self.reply(zb, status=pex.code, headers=h, volsan=True)
+                if pex.log:
+                    self.log("additional error context:\n" + pex.log, 6)
+
                 return self.keepalive
             except Pebkac:
                 return False
@@ -557,6 +568,36 @@ class HttpCli(object):
         else:
             return self.conn.iphash.s(self.ip)
 
+    def cbonk(self, g: Garda, v: str, reason: str, descr: str) -> bool:
+        self.conn.hsrv.nsus += 1
+        if not g.lim:
+            return False
+
+        bonk, ip = g.bonk(self.ip, v + self.gctx)
+        if not bonk:
+            return False
+
+        xban = self.vn.flags.get("xban")
+        if not xban or not runhook(
+            self.log,
+            xban,
+            self.vn.canonical(self.rem),
+            self.vpath,
+            self.host,
+            self.uname,
+            time.time(),
+            0,
+            self.ip,
+            time.time(),
+            reason,
+        ):
+            self.log("client banned: %s" % (descr,), 1)
+            self.conn.hsrv.bans[ip] = bonk
+            self.conn.hsrv.nban += 1
+            return True
+
+        return False
+
     def is_banned(self) -> bool:
         if not self.conn.bans:
             return False
@@ -654,6 +695,7 @@ class HttpCli(object):
                     and not body.startswith(b"<pre>source file busy")
                 )
             )
+            and (status != 404 or (self.can_get and not self.can_read))
         ):
             if status == 404:
                 g = self.conn.hsrv.g404
@@ -675,24 +717,7 @@ class HttpCli(object):
                 or not self.args.nonsus_urls
                 or not self.args.nonsus_urls.search(self.vpath)
             ):
-                bonk, ip = g.bonk(self.ip, self.vpath)
-                if bonk:
-                    xban = self.vn.flags.get("xban")
-                    if not xban or not runhook(
-                        self.log,
-                        xban,
-                        self.vn.canonical(self.rem),
-                        self.vpath,
-                        self.host,
-                        self.uname,
-                        time.time(),
-                        0,
-                        self.ip,
-                        time.time(),
-                        str(status),
-                    ):
-                        self.log("client banned: %ss" % (status,), 1)
-                        self.conn.hsrv.bans[ip] = bonk
+                self.cbonk(g, self.vpath, str(status), "%ss" % (status,))
 
         if volsan:
             vols = list(self.asrv.vfs.all_vols.values())
@@ -1971,8 +1996,7 @@ class HttpCli(object):
         self.log("q#: {} ({:.2f}s)".format(msg, idx.p_dur))
 
         order = []
-        cfg = self.args.mte.split(",")
-        for t in cfg:
+        for t in self.args.mte:
             if t in taglist:
                 order.append(t)
         for t in taglist:
@@ -2119,31 +2143,21 @@ class HttpCli(object):
         return True
 
     def get_pwd_cookie(self, pwd: str) -> str:
-        if self.asrv.ah.hash(pwd) in self.asrv.iacct:
-            msg = "login ok"
+        hpwd = self.asrv.ah.hash(pwd)
+        uname = self.asrv.iacct.get(hpwd)
+        if uname:
+            msg = "hi " + uname
             dur = int(60 * 60 * self.args.logout)
         else:
-            self.log("invalid password: {}".format(pwd), 3)
-            g = self.conn.hsrv.gpwd
-            if g.lim:
-                bonk, ip = g.bonk(self.ip, pwd)
-                if bonk:
-                    xban = self.vn.flags.get("xban")
-                    if not xban or not runhook(
-                        self.log,
-                        xban,
-                        self.vn.canonical(self.rem),
-                        self.vpath,
-                        self.host,
-                        self.uname,
-                        time.time(),
-                        0,
-                        self.ip,
-                        time.time(),
-                        "pw",
-                    ):
-                        self.log("client banned: invalid passwords", 1)
-                        self.conn.hsrv.bans[ip] = bonk
+            logpwd = pwd
+            if self.args.log_badpwd == 0:
+                logpwd = ""
+            elif self.args.log_badpwd == 2:
+                zb = hashlib.sha512(pwd.encode("utf-8", "replace")).digest()
+                logpwd = "%" + base64.b64encode(zb[:12]).decode("utf-8")
+
+            self.log("invalid password: {}".format(logpwd), 3)
+            self.cbonk(self.conn.hsrv.gpwd, pwd, "pw", "invalid passwords")
 
             msg = "naw dude"
             pwd = "x"  # nosec
@@ -2167,26 +2181,30 @@ class HttpCli(object):
         new_dir = self.parser.require("name", 512)
         self.parser.drop()
 
-        sanitized = sanitize_fn(new_dir, "", [])
-        return self._mkdir(vjoin(self.vpath, sanitized))
+        return self._mkdir(vjoin(self.vpath, new_dir))
 
     def _mkdir(self, vpath: str, dav: bool = False) -> bool:
         nullwrite = self.args.nw
+        self.gctx = vpath
+        vpath = undot(vpath)
         vfs, rem = self.asrv.vfs.get(vpath, self.uname, False, True)
-        self._assert_safe_rem(rem)
+        rem = sanitize_vpath(rem, "/", [])
         fn = vfs.canonical(rem)
+        if not fn.startswith(vfs.realpath):
+            self.log("invalid mkdir [%s] [%s]" % (self.gctx, vpath), 1)
+            raise Pebkac(422)
 
         if not nullwrite:
             fdir = os.path.dirname(fn)
 
-            if not bos.path.isdir(fdir):
+            if dav and not bos.path.isdir(fdir):
                 raise Pebkac(409, "parent folder does not exist")
 
             if bos.path.isdir(fn):
-                raise Pebkac(405, "that folder exists already")
+                raise Pebkac(405, 'folder "/%s" already exists' % (vpath,))
 
             try:
-                bos.mkdir(fn)
+                bos.makedirs(fn)
             except OSError as ex:
                 if ex.errno == errno.EACCES:
                     raise Pebkac(500, "the server OS denied write-access")
@@ -2195,7 +2213,7 @@ class HttpCli(object):
             except:
                 raise Pebkac(500, min_ex())
 
-        self.out_headers["X-New-Dir"] = quotep(vpath.split("/")[-1])
+        self.out_headers["X-New-Dir"] = quotep(vpath)
 
         if dav:
             self.reply(b"", 201)
@@ -2716,6 +2734,29 @@ class HttpCli(object):
 
         return file_lastmod, True
 
+    def _expand(self, txt: str, phs: list[str]) -> str:
+        for ph in phs:
+            if ph.startswith("hdr."):
+                sv = str(self.headers.get(ph[4:], ""))
+            elif ph.startswith("self."):
+                sv = str(getattr(self, ph[5:], ""))
+            elif ph.startswith("cfg."):
+                sv = str(getattr(self.args, ph[4:], ""))
+            elif ph.startswith("vf."):
+                sv = str(self.vn.flags.get(ph[3:]) or "")
+            elif ph == "srv.itime":
+                sv = str(int(time.time()))
+            elif ph == "srv.htime":
+                sv = datetime.now(UTC).strftime("%Y-%m-%d, %H:%M:%S")
+            else:
+                self.log("unknown placeholder in server config: [%s]" % (ph), 3)
+                continue
+
+            sv = self.conn.hsrv.ptn_hsafe.sub("_", sv)
+            txt = txt.replace("{{%s}}" % (ph,), sv)
+
+        return txt
+
     def tx_file(self, req_path: str) -> bool:
         status = 200
         logmsg = "{:4} {} ".format("", self.req)
@@ -3042,7 +3083,7 @@ class HttpCli(object):
         self.reply(ico, mime=mime, headers={"Last-Modified": lm})
         return True
 
-    def tx_md(self, fs_path: str) -> bool:
+    def tx_md(self, vn: VFS, fs_path: str) -> bool:
         logmsg = "     %s @%s " % (self.req, self.uname)
 
         if not self.can_write:
@@ -3059,9 +3100,16 @@ class HttpCli(object):
         st = bos.stat(html_path)
         ts_html = st.st_mtime
 
+        max_sz = 1024 * self.args.txt_max
         sz_md = 0
         lead = b""
+        fullfile = b""
         for buf in yieldfile(fs_path):
+            if sz_md < max_sz:
+                fullfile += buf
+            else:
+                fullfile = b""
+
             if not sz_md and b"\n" in buf[:2]:
                 lead = buf[: buf.find(b"\n") + 1]
                 sz_md += len(lead)
@@ -3070,6 +3118,21 @@ class HttpCli(object):
             for c, v in [(b"&", 4), (b"<", 3), (b">", 3)]:
                 sz_md += (len(buf) - len(buf.replace(c, b""))) * v
 
+        if (
+            fullfile
+            and "exp" in vn.flags
+            and "edit" not in self.uparam
+            and "edit2" not in self.uparam
+            and vn.flags.get("exp_md")
+        ):
+            fulltxt = fullfile.decode("utf-8", "replace")
+            fulltxt = self._expand(fulltxt, vn.flags.get("exp_md") or [])
+            fullfile = fulltxt.encode("utf-8", "replace")
+
+        if fullfile:
+            fullfile = html_bescape(fullfile)
+            sz_md = len(lead) + len(fullfile)
+
         file_ts = int(max(ts_md, ts_html, self.E.t0))
         file_lastmod, do_send = self._chk_lastmod(file_ts)
         self.out_headers["Last-Modified"] = file_lastmod
@@ -3111,8 +3174,11 @@ class HttpCli(object):
 
         try:
             self.s.sendall(html[0] + lead)
-            for buf in yieldfile(fs_path):
-                self.s.sendall(html_bescape(buf))
+            if fullfile:
+                self.s.sendall(fullfile)
+            else:
+                for buf in yieldfile(fs_path):
+                    self.s.sendall(html_bescape(buf))
 
             self.s.sendall(html[1])
 
@@ -3743,7 +3809,7 @@ class HttpCli(object):
                     or "edit2" in self.uparam
                 )
             ):
-                return self.tx_md(abspath)
+                return self.tx_md(vn, abspath)
 
             return self.tx_file(abspath)
 
@@ -3805,6 +3871,10 @@ class HttpCli(object):
                 if bos.path.exists(fn):
                     with open(fsenc(fn), "rb") as f:
                         logues[n] = f.read().decode("utf-8")
+                    if "exp" in vn.flags:
+                        logues[n] = self._expand(
+                            logues[n], vn.flags.get("exp_lg") or []
+                        )
 
         readme = ""
         if not self.args.no_readme and not logues[1]:
@@ -3814,6 +3884,8 @@ class HttpCli(object):
                     with open(fsenc(fn), "rb") as f:
                         readme = f.read().decode("utf-8")
                         break
+            if readme and "exp" in vn.flags:
+                readme = self._expand(readme, vn.flags.get("exp_md") or [])
 
         vf = vn.flags
         unlist = vf.get("unlist", "")
@@ -3825,6 +3897,9 @@ class HttpCli(object):
             "acct": self.uname,
             "idx": e2d,
             "itag": e2t,
+            "dsort": vf["sort"],
+            "dfull": "nocrop" in vf,
+            "u2ts": vf["u2ts"],
             "lifetime": vn.flags.get("lifetime") or 0,
             "frand": bool(vn.flags.get("rand")),
             "unlist": unlist,
@@ -3832,41 +3907,46 @@ class HttpCli(object):
             "logues": logues,
             "readme": readme,
         }
-        j2a = {
-            "vdir": quotep(self.vpath),
-            "vpnodes": vpnodes,
-            "files": [],
+        cgv = {
             "ls0": None,
             "acct": self.uname,
-            "perms": json.dumps(perms),
+            "perms": perms,
+            "u2ts": vf["u2ts"],
             "lifetime": ls_ret["lifetime"],
             "frand": bool(vn.flags.get("rand")),
-            "taglist": [],
             "def_hcols": [],
             "have_emp": self.args.emp,
             "have_up2k_idx": e2d,
-            "have_tags_idx": e2t,
             "have_acode": (not self.args.no_acode),
             "have_mv": (not self.args.no_mv),
             "have_del": (not self.args.no_del),
             "have_zip": (not self.args.no_zip),
             "have_unpost": int(self.args.unpost),
-            "have_b_u": (self.can_write and self.uparam.get("b") == "u"),
             "sb_md": "" if "no_sb_md" in vf else (vf.get("md_sbf") or "y"),
-            "sb_lg": "" if "no_sb_lg" in vf else (vf.get("lg_sbf") or "y"),
-            "url_suf": url_suf,
-            "logues": logues,
             "readme": readme,
-            "title": html_escape("%s %s" % (self.args.bname, self.vpath), crlf=True),
-            "srv_info": srv_infot,
             "dgrid": "grid" in vf,
-            "unlist": unlist,
-            "dtheme": self.args.theme,
+            "dfull": "nocrop" in vf,
+            "dsort": vf["sort"],
             "themes": self.args.themes,
             "turbolvl": self.args.turbo,
             "idxh": int(self.args.ih),
             "u2sort": self.args.u2sort,
         }
+        j2a = {
+            "cgv": cgv,
+            "vpnodes": vpnodes,
+            "files": [],
+            "ls0": None,
+            "taglist": [],
+            "have_tags_idx": e2t,
+            "have_b_u": (self.can_write and self.uparam.get("b") == "u"),
+            "sb_lg": "" if "no_sb_lg" in vf else (vf.get("lg_sbf") or "y"),
+            "url_suf": url_suf,
+            "logues": logues,
+            "title": html_escape("%s %s" % (self.args.bname, self.vpath), crlf=True),
+            "srv_info": srv_infot,
+            "dtheme": self.args.theme,
+        }
 
         if self.args.js_browser:
             zs = self.args.js_browser
@@ -3975,7 +4055,7 @@ class HttpCli(object):
                 margin = "-"
 
             sz = inf.st_size
-            zd = datetime.utcfromtimestamp(linf.st_mtime)
+            zd = datetime.fromtimestamp(linf.st_mtime, UTC)
             dt = "%04d-%02d-%02d %02d:%02d:%02d" % (
                 zd.year,
                 zd.month,
@@ -4035,6 +4115,9 @@ class HttpCli(object):
                     ap = vn.canonical(rem)
                     return self.tx_file(ap)  # is no-cache
 
+        mte = vn.flags.get("mte", {})
+        add_up_at = ".up_at" in mte
+        is_admin = self.can_admin
         tagset: set[str] = set()
         for fe in files:
             fn = fe["name"]
@@ -4062,24 +4145,38 @@ class HttpCli(object):
                     self.log(t.format(rd, fn, min_ex()))
                     break
 
-            fe["tags"] = {k: v for k, v in r}
+            tags = {k: v for k, v in r}
 
-            if self.can_admin:
+            if is_admin:
                 q = "select ip, at from up where rd=? and fn=?"
                 try:
                     zs1, zs2 = icur.execute(q, erd_efn).fetchone()
-                    fe["tags"]["up_ip"] = zs1
-                    fe["tags"][".up_at"] = zs2
+                    if zs1:
+                        tags["up_ip"] = zs1
+                    if zs2:
+                        tags[".up_at"] = zs2
+                except:
+                    pass
+            elif add_up_at:
+                q = "select at from up where rd=? and fn=?"
+                try:
+                    (zs1,) = icur.execute(q, erd_efn).fetchone()
+                    if zs1:
+                        tags[".up_at"] = zs1
                 except:
                     pass
 
-            _ = [tagset.add(k) for k in fe["tags"]]
+            _ = [tagset.add(k) for k in tags]
+            fe["tags"] = tags
 
         if icur:
-            mte = vn.flags.get("mte") or "up_ip,.up_at"
-            taglist = [k for k in mte.split(",") if k in tagset]
+            lmte = list(mte)
+            if self.can_admin:
+                lmte += ["up_ip", ".up_at"]
+
+            taglist = [k for k in lmte if k in tagset]
             for fe in dirs:
-                fe["tags"] = {}
+                fe["tags"] = ODict()
         else:
             taglist = list(tagset)
 
@@ -4099,6 +4196,12 @@ class HttpCli(object):
                 if sz < 1024 * self.args.txt_max:
                     with open(fsenc(docpath), "rb") as f:
                         doctxt = f.read().decode("utf-8", "replace")
+
+                    if doc.lower().endswith(".md") and "exp" in vn.flags:
+                        doctxt = self._expand(doctxt, vn.flags.get("exp_md") or [])
+                else:
+                    self.log("doc 2big: [{}]".format(doc), c=6)
+                    doctxt = "( size of textfile exceeds serverside limit )"
             else:
                 self.log("doc 404: [{}]".format(doc), c=6)
                 doctxt = "( textfile not found )"
@@ -4112,7 +4215,7 @@ class HttpCli(object):
         dirs.sort(key=itemgetter("name"))
 
         if is_js:
-            j2a["ls0"] = {
+            j2a["ls0"] = cgv["ls0"] = {
                 "dirs": dirs,
                 "files": files,
                 "taglist": taglist,
@@ -4126,7 +4229,7 @@ class HttpCli(object):
         j2a["txt_ext"] = self.args.textfiles.replace(",", " ")
 
         if "mth" in vn.flags:
-            j2a["def_hcols"] = vn.flags["mth"].split(",")
+            j2a["def_hcols"] = list(vn.flags["mth"])
 
         html = self.j2s(tpl, **j2a)
         self.reply(html.encode("utf-8", "replace"))

copyparty/httpsrv.py

@@ -128,6 +128,9 @@ class HttpSrv(object):
 
         self.u2fh = FHC()
         self.metrics = Metrics(self)
+        self.nreq = 0
+        self.nsus = 0
+        self.nban = 0
         self.srvs: list[socket.socket] = []
         self.ncli = 0  # exact
         self.clients: set[HttpConn] = set()  # laggy
@@ -149,6 +152,7 @@ class HttpSrv(object):
         self._build_statics()
 
         self.ptn_cc = re.compile(r"[\x00-\x1f]")
+        self.ptn_hsafe = re.compile(r"[\x00-\x1f<>\"'&]")
 
         self.mallow = "GET HEAD POST PUT DELETE OPTIONS".split()
         if not self.args.no_dav:
@@ -170,7 +174,7 @@ class HttpSrv(object):
             if self.args.log_thrs:
                 start_log_thrs(self.log, self.args.log_thrs, nid)
 
-        self.th_cfg: dict[str, Any] = {}
+        self.th_cfg: dict[str, set[str]] = {}
         Daemon(self.post_init, "hsrv-init2")
 
     def post_init(self) -> None:

copyparty/ico.py

@@ -4,9 +4,10 @@ from __future__ import print_function, unicode_literals
 import argparse  # typechk
 import colorsys
 import hashlib
+import re
 
 from .__init__ import PY2
-from .th_srv import HAVE_PIL
+from .th_srv import HAVE_PIL, HAVE_PILF
 from .util import BytesIO
 
 
@@ -24,7 +25,7 @@ class Ico(object):
             zb = [ord(x) for x in zb]
 
         c1 = colorsys.hsv_to_rgb(zb[0] / 256.0, 1, 0.3)
-        c2 = colorsys.hsv_to_rgb(zb[0] / 256.0, 1, 1)
+        c2 = colorsys.hsv_to_rgb(zb[0] / 256.0, 0.8 if HAVE_PILF else 1, 1)
         ci = [int(x * 255) for x in list(c1) + list(c2)]
         c = "".join(["{:02x}".format(x) for x in ci])
 
@@ -37,6 +38,32 @@ class Ico(object):
 
         if chrome:
             # cannot handle more than ~2000 unique SVGs
+            if HAVE_PILF:
+                # pillow 10.1 made this the default font;
+                # svg: 3.7s, this: 36s
+                try:
+                    from PIL import Image, ImageDraw
+
+                    # [.lt] are hard to see lowercase / unspaced
+                    ext2 = re.sub("(.)", "\\1 ", ext).upper()
+
+                    h = int(128 * h / w)
+                    w = 128
+                    img = Image.new("RGB", (w, h), "#" + c[:6])
+                    pb = ImageDraw.Draw(img)
+                    _, _, tw, th = pb.textbbox((0, 0), ext2, font_size=16)
+                    xy = ((w - tw) // 2, (h - th) // 2)
+                    pb.text(xy, ext2, fill="#" + c[6:], font_size=16)
+
+                    img = img.resize((w * 2, h * 2), Image.NEAREST)
+
+                    buf = BytesIO()
+                    img.save(buf, format="PNG", compress_level=1)
+                    return "image/png", buf.getvalue()
+
+                except:
+                    pass
+
             if HAVE_PIL:
                 # svg: 3s, cache: 6s, this: 8s
                 from PIL import Image, ImageDraw

copyparty/metrics.py

@@ -34,14 +34,23 @@ class Metrics(object):
 
         ret: list[str] = []
 
-        def addc(k: str, unit: str, v: str, desc: str) -> None:
-            if unit:
-                k += "_" + unit
-                zs = "# TYPE %s counter\n# UNIT %s %s\n# HELP %s %s\n%s_created %s\n%s_total %s"
-                ret.append(zs % (k, k, unit, k, desc, k, int(self.hsrv.t0), k, v))
-            else:
-                zs = "# TYPE %s counter\n# HELP %s %s\n%s_created %s\n%s_total %s"
-                ret.append(zs % (k, k, desc, k, int(self.hsrv.t0), k, v))
+        def addc(k: str, v: str, desc: str) -> None:
+            zs = "# TYPE %s counter\n# HELP %s %s\n%s_created %s\n%s_total %s"
+            ret.append(zs % (k, k, desc, k, int(self.hsrv.t0), k, v))
+
+        def adduc(k: str, unit: str, v: str, desc: str) -> None:
+            k += "_" + unit
+            zs = "# TYPE %s counter\n# UNIT %s %s\n# HELP %s %s\n%s_created %s\n%s_total %s"
+            ret.append(zs % (k, k, unit, k, desc, k, int(self.hsrv.t0), k, v))
+
+        def addg(k: str, v: str, desc: str) -> None:
+            zs = "# TYPE %s gauge\n# HELP %s %s\n%s %s"
+            ret.append(zs % (k, k, desc, k, v))
+
+        def addug(k: str, unit: str, v: str, desc: str) -> None:
+            k += "_" + unit
+            zs = "# TYPE %s gauge\n# UNIT %s %s\n# HELP %s %s\n%s %s"
+            ret.append(zs % (k, k, unit, k, desc, k, v))
 
         def addh(k: str, typ: str, desc: str) -> None:
             zs = "# TYPE %s %s\n# HELP %s %s"
@@ -54,17 +63,75 @@ class Metrics(object):
         def addv(k: str, v: str) -> None:
             ret.append("%s %s" % (k, v))
 
+        t = "time since last copyparty restart"
         v = "{:.3f}".format(time.time() - self.hsrv.t0)
-        addc("cpp_uptime", "seconds", v, "time since last server restart")
+        addug("cpp_uptime", "seconds", v, t)
+
+        # timestamps are gauges because initial value is not zero
+        t = "unixtime of last copyparty restart"
+        v = "{:.3f}".format(self.hsrv.t0)
+        addug("cpp_boot_unixtime", "seconds", v, t)
+
+        t = "number of open http(s) client connections"
+        addg("cpp_http_conns", str(self.hsrv.ncli), t)
+
+        t = "number of http(s) requests since last restart"
+        addc("cpp_http_reqs", str(self.hsrv.nreq), t)
+
+        t = "number of 403/422/malicious reqs since restart"
+        addc("cpp_sus_reqs", str(self.hsrv.nsus), t)
 
         v = str(len(conn.bans or []))
-        addc("cpp_bans", "", v, "number of banned IPs")
+        addg("cpp_active_bans", v, "number of currently banned IPs")
+
+        t = "number of IPs banned since last restart"
+        addg("cpp_total_bans", str(self.hsrv.nban), t)
+
+        if not args.nos_vst:
+            x = self.hsrv.broker.ask("up2k.get_state")
+            vs = json.loads(x.get())
+
+            nvidle = 0
+            nvbusy = 0
+            nvoffline = 0
+            for v in vs["volstate"].values():
+                if v == "online, idle":
+                    nvidle += 1
+                elif "OFFLINE" in v:
+                    nvoffline += 1
+                else:
+                    nvbusy += 1
+
+            addg("cpp_idle_vols", str(nvidle), "number of idle/ready volumes")
+            addg("cpp_busy_vols", str(nvbusy), "number of busy/indexing volumes")
+            addg("cpp_offline_vols", str(nvoffline), "number of offline volumes")
+
+            t = "time since last database activity (upload/rename/delete)"
+            addug("cpp_db_idle", "seconds", str(vs["dbwt"]), t)
+
+            t = "unixtime of last database activity (upload/rename/delete)"
+            addug("cpp_db_act", "seconds", str(vs["dbwu"]), t)
+
+            t = "number of files queued for hashing/indexing"
+            addg("cpp_hashing_files", str(vs["hashq"]), t)
+
+            t = "number of files queued for metadata scanning"
+            addg("cpp_tagq_files", str(vs["tagq"]), t)
+
+            try:
+                t = "number of files queued for plugin-based analysis"
+                addg("cpp_mtpq_files", str(int(vs["mtpq"])), t)
+            except:
+                pass
 
         if not args.nos_hdd:
             addbh("cpp_disk_size_bytes", "total HDD size of volume")
             addbh("cpp_disk_free_bytes", "free HDD space in volume")
             for vpath, vol in allvols:
                 free, total = get_df(vol.realpath)
+                if free is None or total is None:
+                    continue
+
                 addv('cpp_disk_size_bytes{vol="/%s"}' % (vpath), str(total))
                 addv('cpp_disk_free_bytes{vol="/%s"}' % (vpath), str(free))
 
@@ -161,5 +228,6 @@ class Metrics(object):
         ret.append("# EOF")
 
         mime = "application/openmetrics-text; version=1.0.0; charset=utf-8"
+        mime = cli.uparam.get("mime") or mime
         cli.reply("\n".join(ret).encode("utf-8"), mime=mime)
         return True

copyparty/pwhash.py

@@ -136,8 +136,12 @@ class PWHash(object):
         import getpass
 
         while True:
-            p1 = getpass.getpass("password> ")
-            p2 = getpass.getpass("again or just hit ENTER> ")
+            try:
+                p1 = getpass.getpass("password> ")
+                p2 = getpass.getpass("again or just hit ENTER> ")
+            except EOFError:
+                return
+
             if p2 and p1 != p2:
                 print("\033[31minputs don't match; try again\033[0m", file=sys.stderr)
                 continue

copyparty/smbd.py

@@ -162,6 +162,7 @@ class SMB(object):
                 if "connData" in cl:
                     return cl["connData"]["partygoer"]
                 cf = cf.f_back
+            raise Exception()
         except:
             warning(
                 "nyoron... %s <<-- %s <<-- %s <<-- %s",

copyparty/sutil.py

@@ -8,7 +8,7 @@ from datetime import datetime
 from .__init__ import CORES
 from .bos import bos
 from .th_cli import ThumbCli
-from .util import vjoin
+from .util import UTC, vjoin
 
 if True:  # pylint: disable=using-constant-test
     from typing import Any, Generator, Optional
@@ -108,7 +108,7 @@ def errdesc(errors: list[tuple[str, str]]) -> tuple[dict[str, Any], list[str]]:
         tf_path = tf.name
         tf.write("\r\n".join(report).encode("utf-8", "replace"))
 
-    dt = datetime.utcnow().strftime("%Y-%m%d-%H%M%S")
+    dt = datetime.now(UTC).strftime("%Y-%m%d-%H%M%S")
 
     bos.chmod(tf_path, 0o444)
     return {

copyparty/svchub.py

@@ -39,13 +39,19 @@ from .util import (
     FFMPEG_URL,
     VERSIONS,
     Daemon,
+    DEF_EXP,
+    DEF_MTE,
+    DEF_MTH,
     Garda,
     HLog,
     HMaccas,
+    ODict,
+    UTC,
     alltrace,
     ansi_re,
     min_ex,
     mp,
+    odfusion,
     pybin,
     start_log_thrs,
     start_stackmon,
@@ -115,8 +121,6 @@ class SvcHub(object):
             args.no_mv = True
             args.hardlink = True
             args.vague_403 = True
-            args.ban_404 = "50,60,1440"
-            args.turbo = -1
             args.nih = True
 
         if args.s:
@@ -433,6 +437,21 @@ class SvcHub(object):
             zs = al.xff_src.replace(" ", "").replace(".", "\\.").replace(",", "|")
             al.xff_re = re.compile("^(?:" + zs + ")")
 
+        mte = ODict.fromkeys(DEF_MTE.split(","), True)
+        al.mte = odfusion(mte, al.mte)
+
+        mth = ODict.fromkeys(DEF_MTH.split(","), True)
+        al.mth = odfusion(mth, al.mth)
+
+        exp = ODict.fromkeys(DEF_EXP.split(" "), True)
+        al.exp_md = odfusion(exp, al.exp_md.replace(" ", ","))
+        al.exp_lg = odfusion(exp, al.exp_lg.replace(" ", ","))
+
+        for k in ["no_hash", "no_idx"]:
+            ptn = getattr(self.args, k)
+            if ptn:
+                setattr(self.args, k, re.compile(ptn))
+
         return True
 
     def _setlimits(self) -> None:
@@ -476,7 +495,7 @@ class SvcHub(object):
             self.args.nc = min(self.args.nc, soft // 2)
 
     def _logname(self) -> str:
-        dt = datetime.utcnow()
+        dt = datetime.now(UTC)
         fn = str(self.args.lo)
         for fs in "YmdHMS":
             fs = "%" + fs
@@ -725,7 +744,7 @@ class SvcHub(object):
             return
 
         with self.log_mutex:
-            zd = datetime.utcnow()
+            zd = datetime.now(UTC)
             ts = self.log_dfmt % (
                 zd.year,
                 zd.month * 100 + zd.day,
@@ -743,7 +762,7 @@ class SvcHub(object):
             self.logf.close()
             self._setup_logfile("")
 
-        dt = datetime.utcnow()
+        dt = datetime.now(UTC)
 
         # unix timestamp of next 00:00:00 (leap-seconds safe)
         day_now = dt.day
@@ -751,14 +770,20 @@ class SvcHub(object):
             dt += timedelta(hours=12)
 
         dt = dt.replace(hour=0, minute=0, second=0)
-        self.next_day = calendar.timegm(dt.utctimetuple())
+        try:
+            tt = dt.utctimetuple()
+        except:
+            # still makes me hella uncomfortable
+            tt = dt.timetuple()
+
+        self.next_day = calendar.timegm(tt)
 
     def _log_enabled(self, src: str, msg: str, c: Union[int, str] = 0) -> None:
         """handles logging from all components"""
         with self.log_mutex:
             now = time.time()
             if now >= self.next_day:
-                dt = datetime.utcfromtimestamp(now)
+                dt = datetime.fromtimestamp(now, UTC)
                 zs = "{}\n" if self.no_ansi else "\033[36m{}\033[0m\n"
                 zs = zs.format(dt.strftime("%Y-%m-%d"))
                 print(zs, end="")
@@ -781,7 +806,7 @@ class SvcHub(object):
                 else:
                     msg = "%s%s\033[0m" % (c, msg)
 
-            zd = datetime.utcfromtimestamp(now)
+            zd = datetime.fromtimestamp(now, UTC)
             ts = self.log_efmt % (
                 zd.hour,
                 zd.minute,

copyparty/th_cli.py

@@ -31,7 +31,7 @@ class ThumbCli(object):
             if not c:
                 raise Exception()
         except:
-            c = {k: {} for k in ["thumbable", "pil", "vips", "ffi", "ffv", "ffa"]}
+            c = {k: set() for k in ["thumbable", "pil", "vips", "ffi", "ffv", "ffa"]}
 
         self.thumbable = c["thumbable"]
         self.fmt_pil = c["pil"]
@@ -94,7 +94,7 @@ class ThumbCli(object):
             self.log("no histpath for [{}]".format(ptop))
             return None
 
-        tpath = thumb_path(histpath, rem, mtime, fmt)
+        tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
         tpaths = [tpath]
         if fmt == "w":
             # also check for jpg (maybe webp is unavailable)

copyparty/th_srv.py

@@ -37,14 +37,21 @@ if TYPE_CHECKING:
     from .svchub import SvcHub
 
 HAVE_PIL = False
+HAVE_PILF = False
 HAVE_HEIF = False
 HAVE_AVIF = False
 HAVE_WEBP = False
 
 try:
-    from PIL import ExifTags, Image, ImageOps
+    from PIL import ExifTags, Image, ImageFont, ImageOps
 
     HAVE_PIL = True
+    try:
+        ImageFont.load_default(size=16)
+        HAVE_PILF = True
+    except:
+        pass
+
     try:
         Image.new("RGB", (2, 2)).save(BytesIO(), format="webp")
         HAVE_WEBP = True
@@ -79,17 +86,23 @@ except:
     HAVE_VIPS = False
 
 
-def thumb_path(histpath: str, rem: str, mtime: float, fmt: str) -> str:
+def thumb_path(histpath: str, rem: str, mtime: float, fmt: str, ffa: set[str]) -> str:
     # base16 = 16 = 256
     # b64-lc = 38 = 1444
     # base64 = 64 = 4096
     rd, fn = vsplit(rem)
-    if rd:
-        h = hashlib.sha512(afsenc(rd)).digest()
-        b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
-        rd = "{}/{}/".format(b64[:2], b64[2:4]).lower() + b64
-    else:
-        rd = "top"
+    if not rd:
+        rd = "\ntop"
+
+    # spectrograms are never cropped; strip fullsize flag
+    ext = rem.split(".")[-1].lower()
+    if ext in ffa and fmt in ("wf", "jf"):
+        fmt = fmt[:1]
+
+    rd += "\n" + fmt
+    h = hashlib.sha512(afsenc(rd)).digest()
+    b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
+    rd = "{}/{}/".format(b64[:2], b64[2:4]).lower() + b64
 
     # could keep original filenames but this is safer re pathlen
     h = hashlib.sha512(afsenc(fn)).digest()
@@ -98,7 +111,8 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str) -> str:
     if fmt in ("opus", "caf"):
         cat = "ac"
     else:
-        fmt = "webp" if fmt == "w" else "png" if fmt == "p" else "jpg"
+        fc = fmt[:1]
+        fmt = "webp" if fc == "w" else "png" if fc == "p" else "jpg"
         cat = "th"
 
     return "{}/{}/{}/{}.{:x}.{}".format(histpath, cat, rd, fn, int(mtime), fmt)
@@ -118,7 +132,7 @@ class ThumbSrv(object):
         self.stopping = False
         self.nthr = max(1, self.args.th_mt)
 
-        self.q: Queue[Optional[tuple[str, str, VFS]]] = Queue(self.nthr * 4)
+        self.q: Queue[Optional[tuple[str, str, str, VFS]]] = Queue(self.nthr * 4)
         for n in range(self.nthr):
             Daemon(self.worker, "thumb-{}-{}".format(n, self.nthr))
 
@@ -193,7 +207,7 @@ class ThumbSrv(object):
             self.log("no histpath for [{}]".format(ptop))
             return None
 
-        tpath = thumb_path(histpath, rem, mtime, fmt)
+        tpath = thumb_path(histpath, rem, mtime, fmt, self.fmt_ffa)
         abspath = os.path.join(ptop, rem)
         cond = threading.Condition(self.mutex)
         do_conv = False
@@ -220,8 +234,8 @@ class ThumbSrv(object):
                 self.log("ptop [{}] not in {}".format(ptop, allvols), 3)
                 vn = self.asrv.vfs.all_aps[0][1]
 
-            self.q.put((abspath, tpath, vn))
-            self.log("conv {} \033[0m{}".format(tpath, abspath), c=6)
+            self.q.put((abspath, tpath, fmt, vn))
+            self.log("conv {} :{} \033[0m{}".format(tpath, fmt, abspath), c=6)
 
         while not self.stopping:
             with self.mutex:
@@ -257,7 +271,7 @@ class ThumbSrv(object):
             if not task:
                 break
 
-            abspath, tpath, vn = task
+            abspath, tpath, fmt, vn = task
             ext = abspath.split(".")[-1].lower()
             png_ok = False
             funs = []
@@ -290,7 +304,7 @@ class ThumbSrv(object):
 
             for fun in funs:
                 try:
-                    fun(abspath, ttpath, vn)
+                    fun(abspath, ttpath, fmt, vn)
                     break
                 except Exception as ex:
                     msg = "{} could not create thumbnail of {}\n{}"
@@ -324,7 +338,7 @@ class ThumbSrv(object):
         with self.mutex:
             self.nthr -= 1
 
-    def fancy_pillow(self, im: "Image.Image", vn: VFS) -> "Image.Image":
+    def fancy_pillow(self, im: "Image.Image", fmt: str, vn: VFS) -> "Image.Image":
         # exif_transpose is expensive (loads full image + unconditional copy)
         res = self.getres(vn)
         r = max(*res) * 2
@@ -341,7 +355,7 @@ class ThumbSrv(object):
         if rot in rots:
             im = im.transpose(rots[rot])
 
-        if "nocrop" in vn.flags:
+        if fmt.endswith("f"):
             im.thumbnail(res, resample=Image.LANCZOS)
         else:
             iw, ih = im.size
@@ -351,10 +365,10 @@ class ThumbSrv(object):
 
         return im
 
-    def conv_pil(self, abspath: str, tpath: str, vn: VFS) -> None:
+    def conv_pil(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         with Image.open(fsenc(abspath)) as im:
             try:
-                im = self.fancy_pillow(im, vn)
+                im = self.fancy_pillow(im, fmt, vn)
             except Exception as ex:
                 self.log("fancy_pillow {}".format(ex), "90")
                 im.thumbnail(self.getres(vn))
@@ -380,9 +394,9 @@ class ThumbSrv(object):
 
             im.save(tpath, **args)
 
-    def conv_vips(self, abspath: str, tpath: str, vn: VFS) -> None:
+    def conv_vips(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         crops = ["centre", "none"]
-        if "nocrop" in vn.flags:
+        if fmt.endswith("f"):
             crops = ["none"]
 
         w, h = self.getres(vn)
@@ -399,7 +413,7 @@ class ThumbSrv(object):
 
         img.write_to_file(tpath, Q=40)
 
-    def conv_ffmpeg(self, abspath: str, tpath: str, vn: VFS) -> None:
+    def conv_ffmpeg(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if not ret:
             return
@@ -412,7 +426,7 @@ class ThumbSrv(object):
             seek = [b"-ss", "{:.0f}".format(dur / 3).encode("utf-8")]
 
         scale = "scale={0}:{1}:force_original_aspect_ratio="
-        if "nocrop" in vn.flags:
+        if fmt.endswith("f"):
             scale += "decrease,setsar=1:1"
         else:
             scale += "increase,crop={0}:{1},setsar=1:1"
@@ -497,7 +511,7 @@ class ThumbSrv(object):
         self.log(t + txt, c=c)
         raise sp.CalledProcessError(ret, (cmd[0], b"...", cmd[-1]))
 
-    def conv_waves(self, abspath: str, tpath: str, vn: VFS) -> None:
+    def conv_waves(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in ret:
             raise Exception("not audio")
@@ -525,7 +539,7 @@ class ThumbSrv(object):
         cmd += [fsenc(tpath)]
         self._run_ff(cmd, vn)
 
-    def conv_spec(self, abspath: str, tpath: str, vn: VFS) -> None:
+    def conv_spec(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         ret, _ = ffprobe(abspath, int(vn.flags["convt"] / 2))
         if "ac" not in ret:
             raise Exception("not audio")
@@ -568,7 +582,7 @@ class ThumbSrv(object):
         cmd += [fsenc(tpath)]
         self._run_ff(cmd, vn)
 
-    def conv_opus(self, abspath: str, tpath: str, vn: VFS) -> None:
+    def conv_opus(self, abspath: str, tpath: str, fmt: str, vn: VFS) -> None:
         if self.args.no_acode:
             raise Exception("disabled in server config")
 

copyparty/u2idx.py

@@ -181,6 +181,11 @@ class U2idx(object):
                     is_date = True
                     have_up = True
 
+                elif v == "up_at":
+                    v = "up.at"
+                    is_date = True
+                    have_up = True
+
                 elif v == "path":
                     v = "trim(?||up.rd,'/')"
                     va.append("\nrd")

copyparty/up2k.py

@@ -24,7 +24,7 @@ from queue import Queue
 from .__init__ import ANYWIN, PY2, TYPE_CHECKING, WINDOWS
 from .authsrv import LEELOO_DALLAS, SSEELOG, VFS, AuthSrv
 from .bos import bos
-from .cfg import vf_bmap, vf_vmap
+from .cfg import vf_bmap, vf_cmap, vf_vmap
 from .fsutil import Fstab
 from .mtag import MParser, MTag
 from .util import (
@@ -65,6 +65,11 @@ from .util import (
     w8b64enc,
 )
 
+try:
+    from pathlib import Path
+except:
+    pass
+
 if HAVE_SQLITE3:
     import sqlite3
 
@@ -134,8 +139,6 @@ class Up2k(object):
         self.vol_act: dict[str, float] = {}
         self.busy_aps: set[str] = set()
         self.dupesched: dict[str, list[tuple[str, str, float]]] = {}
-        self.snap_persist_interval = 300  # persist unfinished index every 5 min
-        self.snap_discard_interval = 21600  # drop unfinished after 6 hours inactivity
         self.snap_prev: dict[str, Optional[tuple[int, float]]] = {}
 
         self.mtag: Optional[MTag] = None
@@ -263,6 +266,7 @@ class Up2k(object):
             "hashq": self.n_hashq,
             "tagq": self.n_tagq,
             "mtpq": mtpq,
+            "dbwu": "{:.2f}".format(self.db_act),
             "dbwt": "{:.2f}".format(
                 min(1000 * 24 * 60 * 60 - 1, time.time() - self.db_act)
             ),
@@ -642,10 +646,7 @@ class Up2k(object):
             if self.stop:
                 break
 
-            en: set[str] = set()
-            if "mte" in vol.flags:
-                en = set(vol.flags["mte"].split(","))
-
+            en = set(vol.flags.get("mte", {}))
             self.entags[vol.realpath] = en
 
             if "e2d" in vol.flags:
@@ -801,13 +802,25 @@ class Up2k(object):
         fv = "\033[0;36m{}:\033[90m{}"
         fx = set(("html_head",))
         fd = vf_bmap()
+        fd.update(vf_cmap())
         fd.update(vf_vmap())
         fd = {v: k for k, v in fd.items()}
         fl = {
             k: v
             for k, v in flags.items()
-            if k not in fd or v != getattr(self.args, fd[k])
+            if k not in fd
+            or (
+                v != getattr(self.args, fd[k])
+                and str(v) != str(getattr(self.args, fd[k]))
+            )
         }
+        for k1, k2 in vf_cmap().items():
+            if k1 not in fl or k1 in fx:
+                continue
+            if str(fl[k1]) == str(getattr(self.args, k2)):
+                del fl[k1]
+            else:
+                fl[k1] = ",".join(x for x in fl[k1])
         a = [
             (ft if v is True else ff if v is False else fv).format(k, str(v))
             for k, v in fl.items()
@@ -826,7 +839,7 @@ class Up2k(object):
                 vpath += "/"
 
             zs = " ".join(sorted(a))
-            zs = zs.replace("30mre.compile(", "30m(")  # nohash
+            zs = zs.replace("90mre.compile(", "90m(")  # nohash
             self.log("/{} {}".format(vpath, zs), "35")
 
         reg = {}
@@ -2135,7 +2148,7 @@ class Up2k(object):
 
             try:
                 nfiles = next(cur.execute("select count(w) from up"))[0]
-                self.log("OK: {} |{}|".format(db_path, nfiles))
+                self.log("  {} |{}|".format(db_path, nfiles), "90")
                 return cur
             except:
                 self.log("WARN: could not list files; DB corrupt?\n" + min_ex())
@@ -2339,6 +2352,9 @@ class Up2k(object):
             vols = [(ptop, jcur)] if jcur else []
             if vfs.flags.get("xlink"):
                 vols += [(k, v) for k, v in self.cur.items() if k != ptop]
+            if vfs.flags.get("up_ts", "") == "fu" or not cj["lmod"]:
+                # force upload time rather than last-modified
+                cj["lmod"] = int(time.time())
 
             alts: list[tuple[int, int, dict[str, Any]]] = []
             for ptop, cur in vols:
@@ -2680,7 +2696,7 @@ class Up2k(object):
             fs2 = bos.stat(os.path.dirname(dst)).st_dev
             if fs1 == 0 or fs2 == 0:
                 # py2 on winxp or other unsupported combination
-                raise OSError(38, "filesystem does not have st_dev")
+                raise OSError(errno.ENOSYS, "filesystem does not have st_dev")
             elif fs1 == fs2:
                 # same fs; make symlink as relative as possible
                 spl = r"[\\/]" if WINDOWS else "/"
@@ -2713,7 +2729,18 @@ class Up2k(object):
                     raise Exception("symlink-fallback disabled in cfg")
 
             if not linked:
-                os.symlink(fsenc(lsrc), fsenc(ldst))
+                if ANYWIN:
+                    Path(ldst).symlink_to(lsrc)
+                    if not bos.path.exists(dst):
+                        try:
+                            bos.unlink(dst)
+                        except:
+                            pass
+                        t = "the created symlink [%s] did not resolve to [%s]"
+                        raise Exception(t % (ldst, lsrc))
+                else:
+                    os.symlink(fsenc(lsrc), fsenc(ldst))
+
                 linked = True
         except Exception as ex:
             self.log("cannot link; creating copy: " + repr(ex))
@@ -3300,10 +3327,15 @@ class Up2k(object):
         if bos.path.exists(dabs):
             raise Pebkac(400, "mv2: target file exists")
 
+        stl = bos.lstat(sabs)
+        try:
+            st = bos.stat(sabs)
+        except:
+            st = stl
+
         xbr = svn.flags.get("xbr")
         xar = dvn.flags.get("xar")
         if xbr:
-            st = bos.stat(sabs)
             if not runhook(
                 self.log, xbr, sabs, svp, "", uname, st.st_mtime, st.st_size, "", 0, ""
             ):
@@ -3311,9 +3343,16 @@ class Up2k(object):
                 self.log(t, 1)
                 raise Pebkac(405, t)
 
+        is_xvol = svn.realpath != dvn.realpath
+        if stat.S_ISLNK(stl.st_mode):
+            is_dirlink = stat.S_ISDIR(st.st_mode)
+            is_link = True
+        else:
+            is_link = is_dirlink = False
+
         bos.makedirs(os.path.dirname(dabs))
 
-        if bos.path.islink(sabs):
+        if is_dirlink:
             dlabs = absreal(sabs)
             t = "moving symlink from [{}] to [{}], target [{}]"
             self.log(t.format(sabs, dabs, dlabs))
@@ -3336,36 +3375,22 @@ class Up2k(object):
         c2 = self.cur.get(dvn.realpath)
 
         if ftime_ is None:
-            st = bos.stat(sabs)
             ftime = st.st_mtime
             fsize = st.st_size
         else:
             ftime = ftime_
             fsize = fsize_ or 0
 
-        try:
-            atomic_move(sabs, dabs)
-        except OSError as ex:
-            if ex.errno != errno.EXDEV:
-                raise
-
-            self.log("cross-device move:\n  {}\n  {}".format(sabs, dabs))
-            b1, b2 = fsenc(sabs), fsenc(dabs)
-            try:
-                shutil.copy2(b1, b2)
-            except:
-                os.unlink(b2)
-                raise
-
-            os.unlink(b1)
-
+        has_dupes = False
         if w:
             assert c1
             if c2 and c2 != c1:
                 self._copy_tags(c1, c2, w)
 
-            self._forget_file(svn.realpath, srem, c1, w, c1 != c2, fsize)
-            self._relink(w, svn.realpath, srem, dabs)
+            has_dupes = self._forget_file(svn.realpath, srem, c1, w, is_xvol, fsize)
+            if not is_xvol:
+                has_dupes = self._relink(w, svn.realpath, srem, dabs)
+
             curs.add(c1)
 
             if c2:
@@ -3388,6 +3413,47 @@ class Up2k(object):
         else:
             self.log("not found in src db: [{}]".format(svp))
 
+        try:
+            if is_xvol and has_dupes:
+                raise OSError(errno.EXDEV, "src is symlink")
+
+            atomic_move(sabs, dabs)
+
+        except OSError as ex:
+            if ex.errno != errno.EXDEV:
+                raise
+
+            self.log("using copy+delete (%s):\n  %s\n  %s" % (ex.strerror, sabs, dabs))
+            b1, b2 = fsenc(sabs), fsenc(dabs)
+            is_link = os.path.islink(b1)  # due to _relink
+            try:
+                shutil.copy2(b1, b2)
+            except:
+                try:
+                    os.unlink(b2)
+                except:
+                    pass
+
+                if not is_link:
+                    raise
+
+                # broken symlink? keep it as-is
+                try:
+                    zb = os.readlink(b1)
+                    os.symlink(zb, b2)
+                except:
+                    os.unlink(b2)
+                    raise
+
+            if is_link:
+                try:
+                    times = (int(time.time()), int(stl.st_mtime))
+                    bos.utime(dabs, times, False)
+                except:
+                    pass
+
+            os.unlink(b1)
+
         if xar:
             runhook(self.log, xar, dabs, dvp, "", uname, 0, 0, "", 0, "")
 
@@ -3441,14 +3507,16 @@ class Up2k(object):
         wark: Optional[str],
         drop_tags: bool,
         sz: int,
-    ) -> None:
+    ) -> bool:
         """forgets file in db, fixes symlinks, does not delete"""
         srd, sfn = vsplit(vrem)
+        has_dupes = False
         self.log("forgetting {}".format(vrem))
         if wark and cur:
             self.log("found {} in db".format(wark))
             if drop_tags:
                 if self._relink(wark, ptop, vrem, ""):
+                    has_dupes = True
                     drop_tags = False
 
             if drop_tags:
@@ -3476,6 +3544,8 @@ class Up2k(object):
                 assert wark
                 del reg[wark]
 
+        return has_dupes
+
     def _relink(self, wark: str, sptop: str, srem: str, dabs: str) -> int:
         """
         update symlinks from file at svn/srem to dabs (rename),
@@ -3702,13 +3772,16 @@ class Up2k(object):
             self._finish_upload(job["ptop"], job["wark"])
 
     def _snapshot(self) -> None:
-        slp = self.snap_persist_interval
+        slp = self.args.snap_wri
+        if not slp or self.args.no_snap:
+            return
+
         while True:
             time.sleep(slp)
             if self.pp:
                 slp = 5
             else:
-                slp = self.snap_persist_interval
+                slp = self.args.snap_wri
                 self.do_snapshot()
 
     def do_snapshot(self) -> None:
@@ -3722,11 +3795,8 @@ class Up2k(object):
         if not histpath:
             return
 
-        rm = [
-            x
-            for x in reg.values()
-            if x["need"] and now - x["poke"] > self.snap_discard_interval
-        ]
+        idrop = self.args.snap_drop * 60
+        rm = [x for x in reg.values() if x["need"] and now - x["poke"] >= idrop]
 
         if self.args.nw:
             lost = []

copyparty/util.py

@@ -25,7 +25,6 @@ import threading
 import time
 import traceback
 from collections import Counter
-from datetime import datetime
 from email.utils import formatdate
 
 from ipaddress import IPv4Address, IPv4Network, IPv6Address, IPv6Network
@@ -35,6 +34,35 @@ from .__init__ import ANYWIN, EXE, MACOS, PY2, TYPE_CHECKING, VT100, WINDOWS
 from .__version__ import S_BUILD_DT, S_VERSION
 from .stolen import surrogateescape
 
+try:
+    from datetime import datetime, timezone
+
+    UTC = timezone.utc
+except:
+    from datetime import datetime, timedelta, tzinfo
+
+    TD_ZERO = timedelta(0)
+
+    class _UTC(tzinfo):
+        def utcoffset(self, dt):
+            return TD_ZERO
+
+        def tzname(self, dt):
+            return "UTC"
+
+        def dst(self, dt):
+            return TD_ZERO
+
+    UTC = _UTC()
+
+
+if sys.version_info >= (3, 7) or (
+    sys.version_info >= (3, 6) and platform.python_implementation() == "CPython"
+):
+    ODict = dict
+else:
+    from collections import OrderedDict as ODict
+
 
 def _ens(want: str) -> tuple[int, ...]:
     ret: list[int] = []
@@ -261,6 +289,13 @@ EXTS["vnd.mozilla.apng"] = "png"
 MAGIC_MAP = {"jpeg": "jpg"}
 
 
+DEF_EXP = "self.ip self.ua self.uname self.host cfg.name cfg.logout vf.scan vf.thsize hdr.cf_ipcountry srv.itime srv.htime"
+
+DEF_MTE = "circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,fmt,res,.fps,ahash,vhash"
+
+DEF_MTH = ".vq,.aq,vc,ac,fmt,res,.fps"
+
+
 REKOBO_KEY = {
     v: ln.split(" ", 1)[0]
     for ln in """
@@ -1118,7 +1153,7 @@ def stackmon(fp: str, ival: float, suffix: str) -> None:
             buf = lzma.compress(buf, preset=0)
 
         if "%" in fp:
-            dt = datetime.utcnow()
+            dt = datetime.now(UTC)
             for fs in "YmdHMS":
                 fs = "%" + fs
                 if fs in fp:
@@ -1528,8 +1563,8 @@ def read_header(sr: Unrecv, t_idle: int, t_tot: int) -> list[str]:
 
             raise Pebkac(
                 400,
-                "protocol error while reading headers:\n"
-                + ret.decode("utf-8", "replace"),
+                "protocol error while reading headers",
+                log=ret.decode("utf-8", "replace"),
             )
 
         ofs = ret.find(b"\r\n\r\n")
@@ -1738,7 +1773,16 @@ def sanitize_fn(fn: str, ok: str, bad: list[str]) -> str:
     return fn.strip()
 
 
+def sanitize_vpath(vp: str, ok: str, bad: list[str]) -> str:
+    parts = vp.replace(os.sep, "/").split("/")
+    ret = [sanitize_fn(x, ok, bad) for x in parts]
+    return "/".join(ret)
+
+
 def relchk(rp: str) -> str:
+    if "\x00" in rp:
+        return "[nul]"
+
     if ANYWIN:
         if "\n" in rp or "\r" in rp:
             return "x\nx"
@@ -1774,6 +1818,24 @@ def exclude_dotfiles(filepaths: list[str]) -> list[str]:
     return [x for x in filepaths if not x.split("/")[-1].startswith(".")]
 
 
+def odfusion(base: ODict[str, bool], oth: str) -> ODict[str, bool]:
+    # merge an "ordered set" (just a dict really) with another list of keys
+    words0 = [x for x in oth.split(",") if x]
+    words1 = [x for x in oth[1:].split(",") if x]
+
+    ret = base.copy()
+    if oth.startswith("+"):
+        for k in words1:
+            ret[k] = True
+    elif oth[:1] in ("-", "/"):
+        for k in words1:
+            ret.pop(k, None)
+    else:
+        ret = ODict.fromkeys(words0, True)
+
+    return ret
+
+
 def ipnorm(ip: str) -> str:
     if ":" in ip:
         # assume /64 clients; drop 4 groups
@@ -2923,9 +2985,12 @@ def hidedir(dp) -> None:
 
 
 class Pebkac(Exception):
-    def __init__(self, code: int, msg: Optional[str] = None) -> None:
+    def __init__(
+        self, code: int, msg: Optional[str] = None, log: Optional[str] = None
+    ) -> None:
         super(Pebkac, self).__init__(msg or HTTPCODE[code])
         self.code = code
+        self.log = log
 
     def __repr__(self) -> str:
         return "Pebkac({}, {})".format(self.code, repr(self.args))

copyparty/web/browser.css

@@ -1414,14 +1414,17 @@ html.dz input {
 .opview input.i {
 	width: calc(100% - 16.2em);
 }
+input.drc_v,
 input.eq_gain {
 	width: 3em;
 	text-align: center;
 	margin: 0 .6em;
 }
+#audio_drc table,
 #audio_eq table {
 	border-collapse: collapse;
 }
+#audio_drc td,
 #audio_eq td {
 	text-align: center;
 }
@@ -1430,11 +1433,15 @@ input.eq_gain {
 	display: block;
 	padding: 0;
 }
+#au_drc,
 #au_eq {
 	display: block;
 	margin-top: .5em;
 	padding: 1.3em .3em;
 }
+#au_drc {
+	padding: .4em .3em;
+}
 #ico1 {
 	cursor: pointer;
 }
@@ -1475,6 +1482,8 @@ input.eq_gain {
 	width: calc(100% - 2em);
 	margin: .3em 0 0 1.4em;
 }
+@media (max-width: 130em) { #srch_form.tags #tq_raw { width: calc(100% - 34em) } }
+@media (max-width: 95em) { #srch_form.tags #tq_raw { width: calc(100% - 2em) } }
 #tq_raw td+td {
 	width: 100%;
 }
@@ -1777,6 +1786,7 @@ html.y #tree.nowrap .ntree a+a:hover {
 	display: none;
 }
 .ghead {
+	background: #fff;
 	background: var(--bg-u2);
 	border-radius: .3em;
 	padding: .2em .5em;
@@ -1807,6 +1817,7 @@ html.y #tree.nowrap .ntree a+a:hover {
 	padding: 0;
 }
 #rui {
+	background: #fff;
 	background: var(--bg);
 	position: fixed;
 	top: 0;
@@ -1863,6 +1874,7 @@ html.y #tree.nowrap .ntree a+a:hover {
 }
 #doc {
 	overflow: visible;
+	background: #fff;
 	background: var(--bg);
 	margin: -1em 0 .5em 0;
 	padding: 1em 0 1em 0;
@@ -1879,6 +1891,10 @@ html.y #doc {
 	text-align: center;
 	padding: .5em;
 }
+#docul li.bn span {
+	font-weight: bold;
+	color: var(--fg-max);
+}
 #doc.prism {
 	padding-left: 3em;
 }
@@ -2499,14 +2515,14 @@ html.y #bbox-overlay figcaption a {
 	min-width: 24em;
 }
 #u2cards.w {
-	width: 44em;
+	width: 48em;
 	text-align: left;
 }
 #u2cards.ww {
 	display: inline-block;
 }
 #u2etaw.w {
-	width: 52em;
+	width: 55em;
 	text-align: right;
 	margin: 2em auto -2.7em auto;
 }
@@ -2551,10 +2567,10 @@ html.y #bbox-overlay figcaption a {
 	width: 30em;
 }
 #u2conf.w {
-	width: 48em;
+	width: 51em;
 }
 #u2conf.ww {
-	width: 78em;
+	width: 82em;
 }
 #u2conf.ww #u2c3w {
 	width: 29em;
@@ -3051,6 +3067,16 @@ html.d #treepar {
 
 
 
+@media (max-width: 32em) {
+	#u2conf {
+		font-size: .9em;
+	}
+}
+@media (max-width: 28em) {
+	#u2conf {
+		font-size: .8em;
+	}
+}
 @media (min-width: 70em) {
 	#barpos,
 	#barbuf {

copyparty/web/browser.html

@@ -135,35 +135,17 @@
 
 	<script>
 		var SR = {{ r|tojson }},
+			CGV = {{ cgv|tojson }},
 			TS = "{{ ts }}",
-			acct = "{{ acct }}",
-			perms = {{ perms }},
-			dgrid = {{ dgrid|tojson }},
-			themes = {{ themes }},
 			dtheme = "{{ dtheme }}",
 			srvinf = "{{ srv_info }}",
 			s_name = "{{ s_name }}",
 			lang = "{{ lang }}",
 			dfavico = "{{ favico }}",
-			def_hcols = {{ def_hcols|tojson }},
-			have_up2k_idx = {{ have_up2k_idx|tojson }},
 			have_tags_idx = {{ have_tags_idx|tojson }},
-			have_acode = {{ have_acode|tojson }},
-			have_mv = {{ have_mv|tojson }},
-			have_del = {{ have_del|tojson }},
-			have_unpost = {{ have_unpost }},
-			have_zip = {{ have_zip|tojson }},
-			sb_md = "{{ sb_md }}",
 			sb_lg = "{{ sb_lg }}",
-			lifetime = {{ lifetime }},
-			turbolvl = {{ turbolvl }},
-			idxh = {{ idxh }},
-			frand = {{ frand|tojson }},
-			u2sort = "{{ u2sort }}",
-			have_emp = {{ have_emp|tojson }},
 			txt_ext = "{{ txt_ext }}",
 			logues = {{ logues|tojson if sb_lg else "[]" }},
-			readme = {{ readme|tojson }},
 			ls0 = {{ ls0|tojson }};
 
 		document.documentElement.className = localStorage.cpp_thm || dtheme;

copyparty/web/browser.js

@@ -137,6 +137,7 @@ var Ls = {
 
 		"ul_par": "parallel uploads:",
 		"ut_rand": "randomize filenames",
+		"ut_u2ts": "copy the last-modified timestamp$Nfrom your filesystem to the server",
 		"ut_mt": "continue hashing other files while uploading$N$Nmaybe disable if your CPU or HDD is a bottleneck",
 		"ut_ask": "ask for confirmation before upload starts",
 		"ut_pot": "improve upload speed on slow devices$Nby making the UI less complex",
@@ -168,6 +169,7 @@ var Ls = {
 		"utl_prog": "progress",
 
 		"ul_flagblk": "the files were added to the queue</b><br>however there is a busy up2k in another browser tab,<br>so waiting for that to finish first",
+		"ul_btnlk": "the server configuration has locked this switch into this state",
 
 		"udt_up": "Upload",
 		"udt_srch": "Search",
@@ -231,6 +233,7 @@ var Ls = {
 		"ml_tcode": "transcode",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer",
+		"ml_drc": "dynamic range compressor",
 
 		"mt_preload": "start loading the next song near the end for gapless playback\">preload",
 		"mt_fullpre": "try to preload the entire song;$N✅ enable on <b>unreliable</b> connections,$N❌ <b>disable</b> on slow connections probably\">full",
@@ -248,6 +251,7 @@ var Ls = {
 		"mt_coth": "convert all others (not mp3) to opus\">oth",
 		"mt_tint": "background level (0-100) on the seekbar$Nto make buffering less distracting",
 		"mt_eq": "enables the equalizer and gain control;$N$Nboost &lt;code&gt;0&lt;/code&gt; = standard 100% volume (unmodified)$N$Nwidth &lt;code&gt;1 &nbsp;&lt;/code&gt; = standard stereo (unmodified)$Nwidth &lt;code&gt;0.5&lt;/code&gt; = 50% left-right crossfeed$Nwidth &lt;code&gt;0 &nbsp;&lt;/code&gt; = mono$N$Nboost &lt;code&gt;-0.8&lt;/code&gt; &amp; width &lt;code&gt;10&lt;/code&gt; = vocal removal :^)$N$Nenabling the equalizer makes gapless albums fully gapless, so leave it on with all the values at zero (except width = 1) if you care about that",
+		"mt_drc": "enables the dynamic range compressor (volume flattener / brickwaller); will also enable EQ to balance the spaghetti, so set all EQ fields except for 'width' to 0 if you don't want it$N$Nlowers the volume of audio above THRESHOLD dB; for every RATIO dB past THRESHOLD there is 1 dB of output, so default values of tresh -24 and ratio 12 means it should never get louder than -22 dB and it is safe to increase the equalizer boost to 0.8, or even 1.8 with ATK 0 and a huge RLS like 90 (only works in firefox; RLS is max 1 in other browsers)$N$N(see wikipedia, they explain it much better)",
 
 		"mb_play": "play",
 		"mm_hashplay": "play this audio file?",
@@ -331,6 +335,7 @@ var Ls = {
 		"tvt_edit": "open file in text editor$NHotkey: E\">✏️ edit",
 
 		"gt_msel": "enable file selection; ctrl-click a file to override$N$N&lt;em&gt;when active: doubleclick a file / folder to open it&lt;/em&gt;$N$NHotkey: S\">multiselect",
+		"gt_full": "show uncropped thumbnails\">full",
 		"gt_zoom": "zoom",
 		"gt_chop": "chop",
 		"gt_sort": "sort by",
@@ -351,11 +356,14 @@ var Ls = {
 		"s_rd": "path",
 		"s_fn": "name",
 		"s_ta": "tags",
+		"s_ua": "up@",
 		"s_ad": "adv.",
 		"s_s1": "minimum MiB",
 		"s_s2": "maximum MiB",
 		"s_d1": "min. iso8601",
 		"s_d2": "max. iso8601",
+		"s_u1": "uploaded after",
+		"s_u2": "and/or before",
 		"s_r1": "path contains &nbsp; (space-separated)",
 		"s_f1": "name contains &nbsp; (negate with -nope)",
 		"s_t1": "tags contains &nbsp; (^=start, end=$)",
@@ -448,6 +456,8 @@ var Ls = {
 		"u_expl": "explain",
 		"u_tu": '<p class="warn">WARNING: turbo enabled, <span>&nbsp;client may not detect and resume incomplete uploads; see turbo-button tooltip</span></p>',
 		"u_ts": '<p class="warn">WARNING: turbo enabled, <span>&nbsp;search results can be incorrect; see turbo-button tooltip</span></p>',
+		"u_turbo_c": "turbo is disabled in server config",
+		"u_turbo_g": "disabling turbo because you don't have\ndirectory listing privileges within this volume",
 		"u_life_cfg": 'autodelete after <input id="lifem" p="60" /> min (or <input id="lifeh" p="3600" /> hours)',
 		"u_life_est": 'upload will be deleted <span id="lifew" tt="local time">---</span>',
 		"u_life_max": 'this folder enforces a\nmax lifetime of {0}',
@@ -606,6 +616,7 @@ var Ls = {
 
 		"ul_par": "samtidige handl.:",
 		"ut_rand": "finn opp nye tilfeldige filnavn",
+		"ut_u2ts": "gi filen på serveren samme$Ntidsstempel som lokalt hos deg",
 		"ut_mt": "fortsett å befare køen mens opplastning foregår$N$Nskru denne av dersom du har en$Ntreg prosessor eller harddisk",
 		"ut_ask": "bekreft filutvalg før opplastning starter",
 		"ut_pot": "forbedre ytelsen på trege enheter ved å$Nforenkle brukergrensesnittet",
@@ -637,6 +648,7 @@ var Ls = {
 		"utl_prog": "fremdrift",
 
 		"ul_flagblk": "filene har blitt lagt i køen</b><br>men det er en annen nettleserfane som holder på med befaring eller opplastning akkurat nå,<br>så venter til den er ferdig først",
+		"ul_btnlk": "bryteren har blitt låst til denne tilstanden i serverens konfigurasjon",
 
 		"udt_up": "Last opp",
 		"udt_srch": "Søk",
@@ -700,6 +712,7 @@ var Ls = {
 		"ml_tcode": "konvertering",
 		"ml_tint": "tint",
 		"ml_eq": "audio equalizer (tonejustering)",
+		"ml_drc": "compressor (volum-utjevning)",
 
 		"mt_preload": "hent ned litt av neste sang i forkant,$Nslik at pausen i overgangen blir mindre\">forles",
 		"mt_fullpre": "hent ned hele neste sang, ikke bare litt:$N✅ skru på hvis nettet ditt er <b>ustabilt</b>,$N❌ skru av hvis nettet ditt er <b>tregt</b>\">full",
@@ -717,6 +730,7 @@ var Ls = {
 		"mt_coth": "konverter alt annet (men ikke mp3) til opus\">andre",
 		"mt_tint": "nivå av bakgrunnsfarge på søkestripa (0-100),$Ngjør oppdateringer mindre distraherende",
 		"mt_eq": "aktiver tonekontroll og forsterker;$N$Nboost &lt;code&gt;0&lt;/code&gt; = normal volumskala$N$Nwidth &lt;code&gt;1 &nbsp;&lt;/code&gt; = normal stereo$Nwidth &lt;code&gt;0.5&lt;/code&gt; = 50% blanding venstre-høyre$Nwidth &lt;code&gt;0 &nbsp;&lt;/code&gt; = mono$N$Nboost &lt;code&gt;-0.8&lt;/code&gt; &amp; width &lt;code&gt;10&lt;/code&gt; = instrumental :^)$N$Nreduserer også dødtid imellom sangfiler",
+		"mt_drc": "aktiver volum-utjevning (dynamic range compressor); vil også aktivere tonejustering, så sett alle EQ-feltene bortsett fra 'width' til 0 hvis du ikke vil ha noe EQ$N$Nfilteret vil dempe volumet på alt som er høyere enn TRESH dB; for hver RATIO dB over grensen er det 1dB som treffer høyttalerne, så standardverdiene tresh -24 og ratio 12 skal bety at volumet ikke går høyere enn -22 dB, slik at man trygt kan øke boost-verdien i equalizer'n til rundt 0.8, eller 1.8 kombinert med ATK 0 og RLS 90 (bare mulig i firefox; andre nettlesere tar ikke høyere RLS enn 1)$N$Nwikipedia forklarer dette mye bedre forresten",
 
 		"mb_play": "lytt",
 		"mm_hashplay": "spill denne sangen?",
@@ -800,6 +814,7 @@ var Ls = {
 		"tvt_edit": "redigér filen$NSnarvei: E\">✏️ endre",
 
 		"gt_msel": "markér filer istedenfor å åpne dem; ctrl-klikk filer for å overstyre$N$N&lt;em&gt;når aktiv: dobbelklikk en fil / mappe for å åpne&lt;/em&gt;$N$NSnarvei: S\">markering",
+		"gt_full": "ikke beskjær bildene\">full",
 		"gt_zoom": "zoom",
 		"gt_chop": "trim",
 		"gt_sort": "sorter",
@@ -820,11 +835,14 @@ var Ls = {
 		"s_rd": "sti",
 		"s_fn": "navn",
 		"s_ta": "meta",
+		"s_ua": "up@",
 		"s_ad": "avns.",
 		"s_s1": "større enn ↓ MiB",
 		"s_s2": "mindre enn ↓ MiB",
 		"s_d1": "nyere enn &lt;dato&gt;",
 		"s_d2": "eldre enn",
+		"s_u1": "lastet opp etter",
+		"s_u2": "og/eller før",
 		"s_r1": "mappenavn inneholder",
 		"s_f1": "filnavn inneholder",
 		"s_t1": "sang-info inneholder",
@@ -917,6 +935,8 @@ var Ls = {
 		"u_expl": "forklar",
 		"u_tu": '<p class="warn">ADVARSEL: turbo er på, <span>&nbsp;avbrutte opplastninger vil muligens ikke oppdages og gjenopptas; hold musepekeren over turbo-knappen for mer info</span></p>',
 		"u_ts": '<p class="warn">ADVARSEL: turbo er på, <span>&nbsp;søkeresultater kan være feil; hold musepekeren over turbo-knappen for mer info</span></p>',
+		"u_turbo_c": "turbo er deaktivert i serverkonfigurasjonen",
+		"u_turbo_g": 'turbo ble deaktivert fordi du ikke har\ntilgang til å se mappeinnhold i dette volumet',
 		"u_life_cfg": 'slett opplastning etter <input id="lifem" p="60" /> min (eller <input id="lifeh" p="3600" /> timer)',
 		"u_life_est": 'opplastningen slettes <span id="lifew" tt="lokal tid">---</span>',
 		"u_life_max": 'denne mappen tillater ikke å \noppbevare filer i mer enn {0}',
@@ -1042,6 +1062,10 @@ ebi('op_up2k').innerHTML = (
 	'			<input type="checkbox" id="u2rand" />\n' +
 	'			<label for="u2rand" tt="' + L.ut_rand + '">🎲</label>\n' +
 	'		</td>\n' +
+	'		<td class="c" rowspan="2">\n' +
+	'			<input type="checkbox" id="u2ts" />\n' +
+	'			<label for="u2ts" tt="' + L.ut_u2ts + '">📅</a>\n' +
+	'		</td>\n' +
 	'		<td class="c" data-perm="read" data-dep="idx" rowspan="2">\n' +
 	'			<input type="checkbox" id="fsearch" />\n' +
 	'			<label for="fsearch" tt="' + L.ut_srch + '">🔎</label>\n' +
@@ -1366,7 +1390,9 @@ var mpl = (function () {
 		'<input type="text" id="pb_tint" value="0" ' + NOAC + ' style="width:2.4em" tt="' + L.mt_tint + '" />' +
 		'</div></div>' +
 
-		'<div><h3>' + L.ml_eq + '</h3><div id="audio_eq"></div></div>');
+		'<div><h3 id="h_drc">' + L.ml_drc + '</h3><div id="audio_drc"></div></div>' +
+		'<div><h3>' + L.ml_eq + '</h3><div id="audio_eq"></div></div>' +
+		'');
 
 	var r = {
 		"pb_mode": (sread('pb_mode', ['loop', 'next']) || 'next').split('-')[0],
@@ -2472,8 +2498,13 @@ function start_actx() {
 var afilt = (function () {
 	var r = {
 		"eqen": false,
+		"drcen": false,
 		"bands": [31.25, 62.5, 125, 250, 500, 1000, 2000, 4000, 8000, 16000],
 		"gains": [4, 3, 2, 1, 0, 0, 1, 2, 3, 4],
+		"drcv": [-24, 30, 12, 0.01, 0.25],
+		"drch": ['tresh', 'knee', 'ratio', 'atk', 'rls'],
+		"drck": ['threshold', 'knee', 'ratio', 'attack', 'release'],
+		"drcn": null,
 		"filters": [],
 		"filterskip": [],
 		"plugs": [],
@@ -2483,16 +2514,18 @@ var afilt = (function () {
 		"acst": {}
 	};
 
-	if (!ACtx)
-		ebi('audio_eq').parentNode.style.display = 'none';
+	function setvis(vis) {
+		ebi('audio_eq').parentNode.style.display = ebi('audio_drc').parentNode.style.display = (vis ? '' : 'none');
+	}
+
+	setvis(ACtx);
 
 	r.init = function () {
 		start_actx();
 		if (r.cfg)
 			return;
 
-		if (!actx)
-			ebi('audio_eq').parentNode.style.display = 'none';
+		setvis(actx);
 
 		// some browsers have insane high-frequency boost
 		// (or rather the actual problem is Q but close enough)
@@ -2543,6 +2576,8 @@ var afilt = (function () {
 		var gains = jread('au_eq_gain', r.gains);
 		if (r.gains.length == gains.length)
 			r.gains = gains;
+
+		r.drcv = jread('au_drcv', r.drcv);
 	}
 	catch (ex) { }
 
@@ -2579,12 +2614,20 @@ var afilt = (function () {
 		mp.acs = mpo.acs = null;
 	};
 
-	r.apply = function () {
+	r.apply = function (v) {
 		r.init();
 		r.draw();
 
-		if (!actx)
-			bcfg_set('au_eq', false);
+		if (!actx) {
+			bcfg_set('au_eq', r.eqen = false);
+			bcfg_set('au_drc', r.drcen = false);
+		}
+		else if (v === true && r.drcen && !r.eqen)
+			bcfg_set('au_eq', r.eqen = true);
+		else if (v === false && !r.eqen)
+			bcfg_set('au_drc', r.drcen = false);
+
+		r.drcn = null;
 
 		var plug = false;
 		for (var a = 0; a < r.plugs.length; a++)
@@ -2644,6 +2687,28 @@ var afilt = (function () {
 		fi.gain.value = r.amp + 0.94;  // +.137 dB measured; now -.25 dB and almost bitperfect
 		r.filters.push(fi);
 
+		// wait nevermind, drc goes first
+		timer.rm(showdrc);
+		if (r.drcen) {
+			fi = r.drcn = actx.createDynamicsCompressor();
+			for (var a = 0; a < r.drcv.length; a++)
+				fi[r.drck[a]].value = r.drcv[a];
+
+			if (r.drcv[3] < 0.02) {
+				// avoid static at decode start
+				fi.attack.value = 0.02;
+				setTimeout(function () {
+					try {
+						fi.attack.value = r.drcv[3];
+					}
+					catch (ex) { }
+				}, 200);
+			}
+
+			r.filters.push(fi);
+			timer.add(showdrc);
+		}
+
 		if (Math.round(r.chw * 25) != 25) {
 			var split = actx.createChannelSplitter(2),
 				merge = actx.createChannelMerger(2),
@@ -2716,6 +2781,31 @@ var afilt = (function () {
 		clmod(that, 'err', err);
 	}
 
+	function adj_drc() {
+		var err = false;
+		try {
+			var n = this.getAttribute('k'),
+				ov = r.drcv[n],
+				vs = this.value,
+				v = parseFloat(vs);
+
+			if (!isNum(v) || v + '' != vs)
+				throw new Error('inval v');
+
+			if (v == ov)
+				return;
+
+			r.drcv[n] = v;
+			jwrite('au_drcv', r.drcv);
+			if (r.drcn)
+				r.drcn[r.drck[n]].value = v;
+		}
+		catch (ex) {
+			err = true;
+		}
+		clmod(this, 'err', err);
+	}
+
 	function eq_mod(e) {
 		ev(e);
 		adj_band(this, 0);
@@ -2727,6 +2817,13 @@ var afilt = (function () {
 			adj_band(this, step);
 	}
 
+	function showdrc() {
+		if (!r.drcn)
+			return timer.rm(showdrc);
+
+		ebi('h_drc').textContent = f2f(r.drcn.reduction, 1);
+	}
+
 	var html = ['<table><tr><td rowspan="4">',
 		'<a id="au_eq" class="tgl btn" href="#" tt="' + L.mt_eq + '">enable</a></td>'],
 		h2 = [], h3 = [], h4 = [];
@@ -2756,6 +2853,18 @@ var afilt = (function () {
 	html += h4.join('\n') + '</tr><table>';
 	ebi('audio_eq').innerHTML = html;
 
+	h2 = [];
+	html = ['<table><tr><td rowspan="2">',
+		'<a id="au_drc" class="tgl btn" href="#" tt="' + L.mt_drc + '">enable</a></td>'];
+
+	for (var a = 0; a < r.drch.length; a++) {
+		html.push('<td>' + r.drch[a] + '</td>');
+		h2.push('<td><input type="text" class="drc_v" ' + NOAC + ' k="' + a + '" value="' + r.drcv[a] + '" /></td>');
+	}
+	html = html.join('\n') + '</tr><tr>';
+	html += h2.join('\n') + '</tr><table>';
+	ebi('audio_drc').innerHTML = html;
+
 	var stp = QSA('a.eq_step');
 	for (var a = 0, aa = stp.length; a < aa; a++)
 		stp[a].onclick = eq_step;
@@ -2765,8 +2874,12 @@ var afilt = (function () {
 		txt[a].oninput = eq_mod;
 		txt[a].onkeydown = eq_keydown;
 	}
+	txt = QSA('input.drc_v');
+	for (var a = 0; a < txt.length; a++)
+		txt[a].oninput = txt[a].onkeydown = adj_drc;
 
 	bcfg_bind(r, 'eqen', 'au_eq', false, r.apply);
+	bcfg_bind(r, 'drcen', 'au_drc', false, r.apply);
 
 	r.draw();
 	return r;
@@ -3032,6 +3145,8 @@ function scan_hash(v) {
 
 
 function eval_hash() {
+	window.onpopstate = treectl.onpopfun;
+
 	var v = hash0;
 	hash0 = null;
 	if (!v)
@@ -3133,11 +3248,35 @@ function eval_hash() {
 })();
 
 
+function read_dsort(txt) {
+	try {
+		var zt = (('' + txt).trim() || 'href').split(/,+/g);
+		dsort = [];
+		for (var a = 0; a < zt.length; a++) {
+			var z = zt[a].trim(), n = 1, t = "";
+			if (z.startsWith("-")) {
+				z = z.slice(1);
+				n = -1;
+			}
+			if (z == "sz" || z.indexOf('/.') + 1)
+				t = "int";
+
+			dsort.push([z, n, t]);
+		}
+	}
+	catch (ex) {
+		toast.warn(10, 'failed to apply default sort order [' + txt + ']:\n' + ex);
+		dsort = [['href', 1, '']];
+	}
+}
+read_dsort(dsort);
+
+
 function sortfiles(nodes) {
 	if (!nodes.length)
 		return nodes;
 
-	var sopts = jread('fsort', [["href", 1, ""]]),
+	var sopts = jread('fsort', jcp(dsort)),
 		dir1st = sread('dir1st') !== '0';
 
 	try {
@@ -3658,7 +3797,7 @@ var fileman = (function () {
 
 			function rename_cb() {
 				if (this.status !== 201) {
-					var msg = this.responseText;
+					var msg = unpre(this.responseText);
 					toast.err(9, L.fr_efail + msg);
 					return;
 				}
@@ -3707,7 +3846,7 @@ var fileman = (function () {
 		}
 		function delete_cb() {
 			if (this.status !== 200) {
-				var msg = this.responseText;
+				var msg = unpre(this.responseText);
 				toast.err(9, L.fd_err + msg);
 				return;
 			}
@@ -3828,7 +3967,7 @@ var fileman = (function () {
 		}
 		function paste_cb() {
 			if (this.status !== 201) {
-				var msg = this.responseText;
+				var msg = unpre(this.responseText);
 				toast.err(9, L.fp_err + msg);
 				return;
 			}
@@ -4161,7 +4300,7 @@ var showfile = (function () {
 	};
 
 	r.mktree = function () {
-		var html = ['<li class="bn">' + L.tv_lst + '<br />' + linksplit(get_vpath()).join('') + '</li>'];
+		var html = ['<li class="bn">' + L.tv_lst + '<br />' + linksplit(get_vpath()).join('<span>/</span>') + '</li>'];
 		for (var a = 0; a < r.files.length; a++) {
 			var file = r.files[a];
 			html.push('<li><a href="?doc=' +
@@ -4234,7 +4373,8 @@ var thegrid = (function () {
 	gfiles.style.display = 'none';
 	gfiles.innerHTML = (
 		'<div id="ghead" class="ghead">' +
-		'<a href="#" class="tgl btn" id="gridsel" tt="' + L.gt_msel + '</a> <span>' + L.gt_zoom + ': ' +
+		'<a href="#" class="tgl btn" id="gridsel" tt="' + L.gt_msel + '</a> ' +
+		'<a href="#" class="tgl btn" id="gridfull" tt="' + L.gt_full + '</a> <span>' + L.gt_zoom + ': ' +
 		'<a href="#" class="btn" z="-1.2" tt="Hotkey: shift-A">&ndash;</a> ' +
 		'<a href="#" class="btn" z="1.2" tt="Hotkey: shift-D">+</a></span> <span>' + L.gt_chop + ': ' +
 		'<a href="#" class="btn" l="-1" tt="' + L.gt_c1 + '">&ndash;</a> ' +
@@ -4484,6 +4624,9 @@ var thegrid = (function () {
 		if (!r.dirty)
 			return r.loadsel();
 
+		if (dfull != r.full && !sread('gridfull'))
+			bcfg_upd_ui('gridfull', r.full = dfull);
+
 		var html = [],
 			svgs = new Set(),
 			max_svgs = CHROME ? 500 : 5000,
@@ -4501,6 +4644,8 @@ var thegrid = (function () {
 
 			if (r.thumbs) {
 				ihref += '?th=' + (have_webp ? 'w' : 'j');
+				if (r.full)
+					ihref += 'f'
 				if (href == "#")
 					ihref = SR + '/.cpr/ico/⏏️';
 			}
@@ -4587,6 +4732,7 @@ var thegrid = (function () {
 	};
 
 	bcfg_bind(r, 'thumbs', 'thumbs', true, r.setdirty);
+	bcfg_bind(r, 'full', 'gridfull', false, r.setdirty);
 	bcfg_bind(r, 'sel', 'gridsel', false, r.loadsel);
 	bcfg_bind(r, 'en', 'griden', dgrid, function (v) {
 		v ? loadgrid() : r.setvis(true);
@@ -4952,6 +5098,11 @@ document.onkeydown = function (e) {
 		[
 			L.s_ad,
 			["adv", "adv", L.s_a1, "30", "key>=1A  key<=2B  .bpm>165"]
+		],
+		[
+			L.s_ua,
+			["utl", "ut_min", L.s_u1, "14", "2007-04-08"],
+			["utu", "ut_max", L.s_u2, "14", "2038-01-19"]
 		]
 	];
 
@@ -5096,7 +5247,7 @@ document.onkeydown = function (e) {
 					}
 
 					if (k.length == 3) {
-						q += k.replace(/sz/, 'size').replace(/dt/, 'date').replace(/l$/, ' >= ').replace(/u$/, ' <= ') + tv;
+						q += k.replace(/l$/, ' >= ').replace(/u$/, ' <= ').replace(/^sz/, 'size').replace(/^dt/, 'date').replace(/^ut/, 'up_at') + tv;
 						continue;
 					}
 
@@ -5149,10 +5300,7 @@ document.onkeydown = function (e) {
 
 	function xhr_search_results() {
 		if (this.status !== 200) {
-			var msg = this.responseText;
-			if (msg.indexOf('<pre>') === 0)
-				msg = msg.slice(5);
-
+			var msg = unpre(this.responseText);
 			srch_msg(true, "http " + this.status + ": " + msg);
 			search_in_progress = 0;
 			return;
@@ -5191,7 +5339,7 @@ document.onkeydown = function (e) {
 			if (ext.length > 8)
 				ext = '%';
 
-			var links = linksplit(r.rp + '', id).join(''),
+			var links = linksplit(r.rp + '', id).join('<span>/</span>'),
 				nodes = ['<tr><td>-</td><td><div>' + links + '</div>', sz];
 
 			for (var b = 0; b < tagord.length; b++) {
@@ -5791,6 +5939,9 @@ var treectl = (function () {
 			if (res.files[a].tags === undefined)
 				res.files[a].tags = {};
 
+		read_dsort(res.dsort);
+		dfull = res.dfull;
+
 		srvinf = res.srvinf;
 		try {
 			ebi('srv_info').innerHTML = ebi('srv_info2').innerHTML = '<span>' + res.srvinf + '</span>';
@@ -5910,7 +6061,8 @@ var treectl = (function () {
 			}
 
 			if (tn.lead == '-')
-				tn.lead = '<a href="?doc=' + tn.href + '" class="doc' + (lang ? ' bri' : '') +
+				tn.lead = '<a href="?doc=' + bhref +
+					'" class="doc' + (lang ? ' bri' : '') +
 					'" hl="' + id + '" name="' + hname + '">-txt-</a>';
 
 			var ln = ['<tr><td>' + tn.lead + '</td><td><a href="' +
@@ -6127,7 +6279,7 @@ var treectl = (function () {
 	if (cs == 'tree' || (cs != 'na' && vw >= 60))
 		r.entree(null, true);
 
-	window.onpopstate = function (e) {
+	r.onpopfun = function (e) {
 		console.log("h-pop " + e.state);
 		if (!e.state)
 			return;
@@ -6274,6 +6426,7 @@ function apply_perms(res) {
 	if (res.frand)
 		ebi('u2rand').parentNode.style.display = 'none';
 
+	u2ts = res.u2ts;
 	if (up2k)
 		up2k.set_fsearch();
 
@@ -6354,6 +6507,7 @@ var filecols = (function () {
 				toh = ths[a].outerHTML, // !ff10
 				ttv = L.cols[ths[a].textContent];
 
+			ttv = (ttv ? ttv + '; ' : '') + 'id=<code>' + th.getAttribute('name') + '</code>';
 			if (!MOBILE && toh) {
 				th.innerHTML = '<div class="cfg"><a href="#">-</a></div>' + toh;
 				th.getElementsByTagName('a')[0].onclick = ev_row_tgl;
@@ -7011,16 +7165,17 @@ var msel = (function () {
 	form.onsubmit = function (e) {
 		ev(e);
 		clmod(sf, 'vis', 1);
-		sf.textContent = 'creating "' + tb.value + '"...';
+		var dn = tb.value;
+		sf.textContent = 'creating "' + dn + '"...';
 
 		var fd = new FormData();
 		fd.append("act", "mkdir");
-		fd.append("name", tb.value);
+		fd.append("name", dn);
 
 		var xhr = new XHR();
 		xhr.vp = get_evpath();
-		xhr.dn = tb.value;
-		xhr.open('POST', xhr.vp, true);
+		xhr.dn = dn;
+		xhr.open('POST', dn.startsWith('/') ? (SR || '/') : xhr.vp, true);
 		xhr.onload = xhr.onerror = cb;
 		xhr.responseType = 'text';
 		xhr.send(fd);
@@ -7037,7 +7192,7 @@ var msel = (function () {
 		xhrchk(this, L.fd_xe1, L.fd_xe2);
 
 		if (this.status !== 201) {
-			sf.textContent = 'error: ' + this.responseText;
+			sf.textContent = 'error: ' + unpre(this.responseText);
 			return;
 		}
 
@@ -7046,8 +7201,9 @@ var msel = (function () {
 		sf.textContent = '';
 
 		var dn = this.getResponseHeader('X-New-Dir');
-		dn = dn || uricom_enc(this.dn);
-		treectl.goto(this.vp + dn + '/', true);
+		dn = dn ? '/' + dn + '/' : uricom_enc(this.dn);
+		treectl.goto(dn, true);
+		tree_scrollto();
 	}
 })();
 
@@ -7084,7 +7240,7 @@ var msel = (function () {
 		xhrchk(this, L.fsm_xe1, L.fsm_xe2);
 
 		if (this.status < 200 || this.status > 201) {
-			sf.textContent = 'error: ' + this.responseText;
+			sf.textContent = 'error: ' + unpre(this.responseText);
 			return;
 		}
 
@@ -7284,7 +7440,7 @@ function sandbox(tgt, rules, cls, html) {
 	html = '<html class="iframe ' + document.documentElement.className + '"><head><style>' + globalcss() +
 		'</style><base target="_parent"></head><body id="b" class="logue ' + cls + '">' + html +
 		'<script>' + env + '</script>' + sandboxjs() +
-		'<script>var d=document.documentElement,' +
+		'<script>var d=document.documentElement,TS="' + TS + '",' +
 		'loc=new URL("' + location.href.split('?')[0] + '");' +
 		'function say(m){window.parent.postMessage(m,"*")};' +
 		'setTimeout(function(){var its=0,pih=-1,f=function(){' +
@@ -7429,7 +7585,7 @@ var unpost = (function () {
 					'<tr><td><a me="' + me + '" class="n' + a + '" href="#">' + L.un_del + '</a></td>' +
 					'<td>' + unix2iso(res[a].at) + '</td>' +
 					'<td>' + res[a].sz + '</td>' +
-					'<td>' + linksplit(res[a].vp).join(' ') + '</td></tr>');
+					'<td>' + linksplit(res[a].vp).join('<span> / </span>') + '</td></tr>');
 			}
 
 			html.push("</tbody></table>");
@@ -7462,7 +7618,7 @@ var unpost = (function () {
 
 	function unpost_delete_cb() {
 		if (this.status !== 200) {
-			var msg = this.responseText;
+			var msg = unpre(this.responseText);
 			toast.err(9, L.un_derr + msg);
 			return;
 		}

copyparty/web/svcs.html

@@ -10,6 +10,7 @@
 {{ html_head }}
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/splash.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="{{ r }}/.cpr/ui.css?_={{ ts }}">
+	<style>ul{padding-left:1.3em}li{margin:.4em 0}</style>
 </head>
 
 <body>
@@ -48,9 +49,13 @@
                 rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud pacer_min_sleep=0.01ms{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-dav:{{ rvp }} <b>W:</b>
             </pre>
-            {% if s %}
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>--no-check-certificate</code> to the mount command</em><br />---</p>
-            {% endif %}
+            <ul>
+                {% if s %}
+                <li>running <code>rclone mount</code> on LAN (or just dont have valid certificates)? add <code>--no-check-certificate</code></li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             
             <p>if you want to use the native WebDAV client in windows instead (slow and buggy), first run <a href="{{ r }}/.cpr/a/webdav-cfg.bat">webdav-cfg.bat</a> to remove the 47 MiB filesize limit (also fixes latency and password login), then connect:</p>
             <pre>
@@ -73,10 +78,13 @@
                 rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=owncloud pacer_min_sleep=0.01ms{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-dav:{{ rvp }} <b>mp</b>
             </pre>
-            {% if s %}
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>--no-check-certificate</code> to the mount command</em><br />---</p>
-            {% endif %}
-            
+            <ul>
+                {% if s %}
+                <li>running <code>rclone mount</code> on LAN (or just dont have valid certificates)? add <code>--no-check-certificate</code></li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             <p>or the emergency alternative (gnome/gui-only):</p>
             <!-- gnome-bug: ignores vp -->
             <pre>
@@ -123,8 +131,14 @@
                 rclone config create {{ aname }}-ftps ftp host={{ rip }} port={{ args.ftps }} pass=k user={% if accs %}<b>{{ pw }}</b>{% else %}anonymous{% endif %} tls=false explicit_tls=true
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-ftps:{{ rvp }} <b>W:</b>
             </pre>
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>no_check_certificate=true</code> to the config command</em><br />---</p>
             {% endif %}
+            <ul>
+                {% if args.ftps %}
+                <li>running on LAN (or just dont have valid certificates)? add <code>no_check_certificate=true</code> to the config command</li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             <p>if you want to use the native FTP client in windows instead (please dont), press <code>win+R</code> and run this command:</p>
             <pre>
                 explorer {{ "ftp" if args.ftp else "ftps" }}://{% if accs %}<b>{{ pw }}</b>:k@{% endif %}{{ host }}:{{ args.ftp or args.ftps }}/{{ rvp }}
@@ -145,8 +159,14 @@
                 rclone config create {{ aname }}-ftps ftp host={{ rip }} port={{ args.ftps }} pass=k user={% if accs %}<b>{{ pw }}</b>{% else %}anonymous{% endif %} tls=false explicit_tls=true
                 rclone mount --vfs-cache-mode writes --dir-cache-time 5s {{ aname }}-ftps:{{ rvp }} <b>mp</b>
             </pre>
-            <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>no_check_certificate=true</code> to the config command</em><br />---</p>
             {% endif %}
+            <ul>
+                {% if args.ftps %}
+                <li>running on LAN (or just dont have valid certificates)? add <code>no_check_certificate=true</code> to the config command</li>
+                {% endif %}
+                <li>running <code>rclone mount</code> as root? add <code>--allow-other</code></li>
+                <li>old version of rclone? replace all <code>=</code> with <code>&nbsp;</code> (space)</li>
+            </ul>
             <p>emergency alternative (gnome/gui-only):</p>
             <!-- gnome-bug: ignores vp -->
             <pre>
@@ -178,7 +198,7 @@
             partyfuse.py{% if accs %} -a <b>{{ pw }}</b>{% endif %} http{{ s }}://{{ ep }}/{{ rvp }} <b><span class="os win">W:</span><span class="os lin mac">mp</span></b>
         </pre>
         {% if s %}
-        <p><em>note: if you are on LAN (or just dont have valid certificates), add <code>-td</code></em></p>
+        <ul><li>if you are on LAN (or just dont have valid certificates), add <code>-td</code></li></ul>
         {% endif %}
         <p>
             you can use <a href="{{ r }}/.cpr/a/u2c.py">u2c.py</a> to upload (sometimes faster than web-browsers)

copyparty/web/ui.css

@@ -1,3 +1,18 @@
+:root {
+	--fg: #ccc;
+	--fg-max: #fff;
+	--bg-u2: #2b2b2b;
+	--bg-u5: #444;
+}
+html.y {
+	--fg: #222;
+	--fg-max: #000;
+	--bg-u2: #f7f7f7;
+	--bg-u5: #ccc;
+}
+html.bz {
+	--bg-u2: #202231;
+}
 @font-face {
 	font-family: 'scp';
 	font-display: swap;
@@ -14,7 +29,9 @@ html {
 	max-width: min(34em, 90%);
 	max-width: min(34em, calc(100% - 7em));
 	color: #ddd;
+	color: var(--fg);
 	background: #333;
+	background: var(--bg-u2);
 	border: 0 solid #777;
 	box-shadow: 0 .2em .5em #111;
 	border-radius: .4em;
@@ -159,9 +176,10 @@ html {
 #modalc code,
 #tt code {
 	color: #eee;
+	color: var(--fg-max);
 	background: #444;
+	background: var(--bg-u5);
 	padding: .1em .3em;
-	border-top: 1px solid #777;
 	border-radius: .3em;
 	line-height: 1.7em;
 }
@@ -169,22 +187,15 @@ html {
 	color: #f6a;
 }
 html.y #tt {
-	color: #333;
-	background: #fff;
 	border-color: #888 #000 #777 #000;
 }
 html.bz #tt {
-	background: #202231;
 	border-color: #3b3f58;
 }
 html.y #tt,
 html.y #toast {
 	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
 }
-html.y #tt code {
-	background: #060;
-	color: #fff;
-}
 #modalc code {
 	color: #060;
 	background: transparent;
@@ -322,6 +333,9 @@ html.y .btn:focus {
 	box-shadow: 0 .1em .2em #037 inset;
 	outline: #037 solid .1em;
 }
+input[type="submit"] {
+	cursor: pointer;
+}
 input[type="text"]:focus,
 input:not([type]):focus,
 textarea:focus {

copyparty/web/up2k.js

@@ -861,6 +861,7 @@ function up2k_init(subtle) {
     bcfg_bind(uc, 'multitask', 'multitask', true, null, false);
     bcfg_bind(uc, 'potato', 'potato', false, set_potato, false);
     bcfg_bind(uc, 'ask_up', 'ask_up', true, null, false);
+    bcfg_bind(uc, 'u2ts', 'u2ts', !u2ts.endsWith('u'), set_u2ts, false);
     bcfg_bind(uc, 'fsearch', 'fsearch', false, set_fsearch, false);
 
     bcfg_bind(uc, 'flag_en', 'flag_en', false, apply_flag_cfg);
@@ -1339,6 +1340,7 @@ function up2k_init(subtle) {
 
     function up_them(good_files) {
         start_actx();
+        draw_turbo();
         var evpath = get_evpath(),
             draw_each = good_files.length < 50;
 
@@ -1361,7 +1363,7 @@ function up2k_init(subtle) {
                 name = good_files[a][1],
                 fdir = evpath,
                 now = Date.now(),
-                lmod = fobj.lastModified || now,
+                lmod = uc.u2ts ? (fobj.lastModified || now) : 0,
                 ofs = name.lastIndexOf('/') + 1;
 
             if (ofs) {
@@ -1405,7 +1407,7 @@ function up2k_init(subtle) {
 
             pvis.addfile([
                 uc.fsearch ? esc(entry.name) : linksplit(
-                    entry.purl + uricom_enc(entry.name)).join(' '),
+                    entry.purl + uricom_enc(entry.name)).join(' / '),
                 '📐 ' + L.u_hashing,
                 ''
             ], entry.size, draw_each);
@@ -2282,7 +2284,7 @@ function up2k_init(subtle) {
                                 cdiff = (Math.abs(diff) <= 2) ? '3c0' : 'f0b',
                                 sdiff = '<span style="color:#' + cdiff + '">diff ' + diff;
 
-                            msg.push(linksplit(hit.rp).join('') + '<br /><small>' + tr + ' (srv), ' + tu + ' (You), ' + sdiff + '</small></span>');
+                            msg.push(linksplit(hit.rp).join(' / ') + '<br /><small>' + tr + ' (srv), ' + tu + ' (You), ' + sdiff + '</small></span>');
                         }
                         msg = msg.join('<br />\n');
                     }
@@ -2316,7 +2318,7 @@ function up2k_init(subtle) {
                         url += '?k=' + fk;
                     }
 
-                    pvis.seth(t.n, 0, linksplit(url).join(' '));
+                    pvis.seth(t.n, 0, linksplit(url).join(' / '));
                 }
 
                 var chunksize = get_chunksize(t.size),
@@ -2400,15 +2402,12 @@ function up2k_init(subtle) {
                 pvis.seth(t.n, 2, L.u_ehstmp, t);
 
                 var err = "",
-                    rsp = (xhr.responseText + ''),
+                    rsp = unpre(this.responseText),
                     ofs = rsp.lastIndexOf('\nURL: ');
 
                 if (ofs !== -1)
                     rsp = rsp.slice(0, ofs);
 
-                if (rsp.indexOf('<pre>') === 0)
-                    rsp = rsp.slice(5);
-
                 if (rsp.indexOf('rate-limit ') !== -1) {
                     var penalty = rsp.replace(/.*rate-limit /, "").split(' ')[0];
                     console.log("rate-limit: " + penalty);
@@ -2427,7 +2426,7 @@ function up2k_init(subtle) {
                     err = rsp;
                     ofs = err.indexOf('\n/');
                     if (ofs !== -1) {
-                        err = err.slice(0, ofs + 1) + linksplit(err.slice(ofs + 2).trimEnd()).join(' ');
+                        err = err.slice(0, ofs + 1) + linksplit(err.slice(ofs + 2).trimEnd()).join(' / ');
                     }
                     if (!t.rechecks && (err_pend || err_srcb)) {
                         t.rechecks = 0;
@@ -2534,7 +2533,7 @@ function up2k_init(subtle) {
             cdr = t.size;
 
         var orz = function (xhr) {
-            var txt = ((xhr.response && xhr.response.err) || xhr.responseText) + '';
+            var txt = unpre((xhr.response && xhr.response.err) || xhr.responseText);
             if (txt.indexOf('upload blocked by x') + 1) {
                 apop(st.busy.upload, upt);
                 apop(t.postlist, npart);
@@ -2620,7 +2619,7 @@ function up2k_init(subtle) {
             wpx = window.innerWidth,
             fpx = parseInt(getComputedStyle(bar)['font-size']),
             wem = wpx * 1.0 / fpx,
-            wide = wem > 54 ? 'w' : '',
+            wide = wem > 57 ? 'w' : '',
             parent = ebi(wide ? 'u2btn_cw' : 'u2btn_ct'),
             btn = ebi('u2btn');
 
@@ -2629,7 +2628,7 @@ function up2k_init(subtle) {
             ebi('u2conf').className = ebi('u2cards').className = ebi('u2etaw').className = wide;
         }
 
-        wide = wem > 82 ? 'ww' : wide;
+        wide = wem > 86 ? 'ww' : wide;
         parent = ebi(wide == 'ww' ? 'u2c3w' : 'u2c3t');
         var its = [ebi('u2etaw'), ebi('u2cards')];
         if (its[0].parentNode !== parent) {
@@ -2710,7 +2709,12 @@ function up2k_init(subtle) {
     function draw_turbo() {
         if (turbolvl < 0 && uc.turbo) {
             bcfg_set('u2turbo', uc.turbo = false);
-            toast.err(10, "turbo is disabled in server config");
+            toast.err(10, L.u_turbo_c);
+        }
+
+        if (uc.turbo && !has(perms, 'read')) {
+            bcfg_set('u2turbo', uc.turbo = false);
+            toast.warn(30, L.u_turbo_g);
         }
 
         var msg = (turbolvl || !uc.turbo) ? null : uc.fsearch ? L.u_ts : L.u_tu,
@@ -2811,6 +2815,8 @@ function up2k_init(subtle) {
 
     function set_fsearch(new_state) {
         var fixed = false,
+            persist = new_state !== undefined,
+            preferred = bcfg_get('fsearch', undefined),
             can_write = false;
 
         if (!ebi('fsearch')) {
@@ -2827,8 +2833,14 @@ function up2k_init(subtle) {
             }
         }
 
+        if (new_state === undefined)
+            new_state = preferred;
+
         if (new_state !== undefined)
-            bcfg_set('fsearch', uc.fsearch = new_state);
+            if (persist)
+                bcfg_set('fsearch', uc.fsearch = new_state);
+            else
+                bcfg_upd_ui('fsearch', uc.fsearch = new_state);
 
         try {
             clmod(ebi('u2c3w'), 's', !can_write);
@@ -2851,6 +2863,9 @@ function up2k_init(subtle) {
         ebi('u2cards').style.display = ebi('u2tab').style.display = potato ? 'none' : '';
         ebi('u2mu').style.display = potato ? '' : 'none';
 
+        if (u2ts.startsWith('f') || !sread('u2ts'))
+            uc.u2ts = bcfg_upd_ui('u2ts', !u2ts.endsWith('u'));
+
         draw_turbo();
         draw_life();
         onresize();
@@ -2875,12 +2890,24 @@ function up2k_init(subtle) {
         }
     }
 
-    function set_u2sort() {
+    function set_u2sort(en) {
         if (u2sort.indexOf('f') < 0)
             return;
 
-        bcfg_set('u2sort', uc.az = u2sort.indexOf('n') + 1);
-        localStorage.removeItem('u2sort');
+        var fen = uc.az = u2sort.indexOf('n') + 1;
+        bcfg_upd_ui('u2sort', fen);
+        if (en != fen)
+            toast.warn(10, L.ul_btnlk);
+    }
+
+    function set_u2ts(en) {
+        if (u2ts.indexOf('f') < 0)
+            return;
+
+        var fen = !u2ts.endsWith('u');
+        bcfg_upd_ui('u2ts', fen);
+        if (en != fen)
+            toast.warn(10, L.ul_btnlk);
     }
 
     function set_hashw() {
@@ -2978,7 +3005,7 @@ ebi('ico1').onclick = function () {
 if (QS('#op_up2k.act'))
     goto_up2k();
 
-apply_perms({ "perms": perms, "frand": frand });
+apply_perms({ "perms": perms, "frand": frand, "u2ts": u2ts });
 
 
 (function () {

copyparty/web/util.js

@@ -6,6 +6,11 @@ if (!window.console || !console.log)
     };
 
 
+if (window.CGV)
+    for (var k in CGV)
+        window[k] = CGV[k];
+
+
 var wah = '',
     NOAC = 'autocorrect="off" autocapitalize="off"',
     L, tt, treectl, thegrid, up2k, asmCrypto, hashwasm, vbar, marked,
@@ -364,7 +369,7 @@ function import_js(url, cb, ecb) {
     var head = document.head || document.getElementsByTagName('head')[0];
     var script = mknod('script');
     script.type = 'text/javascript';
-    script.src = url;
+    script.src = url + '?_=' + (window.TS || 'a');
     script.onload = cb;
     script.onerror = ecb || function () {
         var m = 'Failed to load module:\n' + url;
@@ -478,7 +483,7 @@ function yscroll() {
 
 function showsort(tab) {
     var v, vn, v1, v2, th = tab.tHead,
-        sopts = jread('fsort', [["href", 1, ""]]);
+        sopts = jread('fsort', jcp(dsort));
 
     th && (th = th.rows[0]) && (th = th.cells);
 
@@ -617,9 +622,8 @@ function linksplit(rp, id) {
         }
         var vlink = esc(uricom_dec(link));
 
-        if (link.indexOf('/') !== -1) {
-            vlink = vlink.slice(0, -1) + '<span>/</span>';
-        }
+        if (link.indexOf('/') !== -1)
+            vlink = vlink.slice(0, -1);
 
         if (!rp) {
             if (q)
@@ -977,13 +981,14 @@ function bcfg_set(name, val) {
 function bcfg_upd_ui(name, val) {
     var o = ebi(name);
     if (!o)
-        return;
+        return val;
 
     if (o.getAttribute('type') == 'checkbox')
         o.checked = val;
     else if (o) {
         clmod(o, 'on', val);
     }
+    return val;
 }
 
 function bcfg_bind(obj, oname, cname, defval, cb, un_ev) {
@@ -1351,6 +1356,11 @@ function lf2br(txt) {
 }
 
 
+function unpre(txt) {
+    return ('' + txt).replace(/^<pre>/, '');
+}
+
+
 var toast = (function () {
     var r = {},
         te = null,

docs/changelog.md

@@ -1,3 +1,129 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1024-1643  `v1.9.15`  expand placeholder
+
+[made it just in time!](https://a.ocv.me/pub/g/nerd-stuff/PXL_20231024_170348367.jpg) (EDIT: nevermind, three of the containers didn't finish uploading to ghcr before takeoff ;_; all up now)
+
+## new features
+* #56 placeholder variables in markdown documents and prologue/epilogue html files
+  * default-disabled; must be enabled globally with `--exp` or per-volume with volflag `exp`
+  * `{{self.ip}}` becomes the client IP; see [/srv/expand/README.md](https://github.com/9001/copyparty/blob/hovudstraum/srv/expand/README.md) for more examples
+* dynamic-range-compressor: reduced volume jumps between songs when enabled
+
+## bugfixes
+* v1.9.14 broke the `scan` volflag, causing volume rescans to happen every 10sec if enabled
+  * its global counterpart `--re-maxage` was not affected
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1021-1443  `v1.9.14`  uptime
+
+## new features
+* search for files by upload time
+* option to display upload time in directory listings
+  * enable globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at`
+  * has a ~17% performance impact on directory listings
+* [dynamic range compressor](https://en.wikipedia.org/wiki/Dynamic_range_compression) in the audioplayer settings
+* `--ban-404` is now default-enabled
+  * the turbo-uploader will now un-turbo when necessary to avoid banning itself
+  * this only affects accounts with permissions `g`, `G`, or `h`
+    * accounts with read-access (which are able to see directory listings anyways) and accounts with write-only access are no longer affected by `--ban-404` or `--ban-url`
+
+## bugfixes
+* #55 clients could hit the `--url-ban` filter when uploading over webdav
+  * fixed by limiting `--ban-404` and `--ban-url` to accounts with permission `g`, `G`, or `h`
+* fixed 20% performance drop in python 3.12 due to utcfromtimestamp deprecation
+  * but 3.12.0 is still 5% slower than 3.11.6 for some reason
+* volume listing on startup would display some redundant info
+
+## other changes
+* timeout for unfinished uploads increased from 6 to 24 hours
+  * and is now configurable with `--snap-drop`
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1015-2006  `v1.9.12`  more buttons
+
+just adding requested features, nothing important
+
+## new features
+* button `📅` in the uploader (default-enabled) sends your local last-modified timestamps to the server
+  * when deselected, the files on the server will have the upload time as their timestamps instead
+  * `--u2ts` specifies the default setting, `c` client-last-modified or `u` upload-time, or `fc` and `fu` to force
+* button `full` in the gridview decides if thumbnails should be center-cropped or not
+  * `--no-crop` and the `nocrop` volflag now sets the default value of this instead of forcing the setting
+  * thumbnail cleanup is now more granular, cleaning full-jpg separately from cropped-webp for example
+* set default sort order with `--sort` or volflag `sort`
+  * one or more comma-separated values; `tags/Cirle,tags/.tn,tags/Artist,tags/Title,href`
+    * see the column header tooltips in the browser to know what names (`id`) to use
+  * prefix a column name with `-` for descending sort
+  * specifying a sort order in the client will override all server-defined ones
+* when visiting a read-only folder, the upload-or-filesearch toggle will remember its previous state and restore it when leaving the folder
+  * much more intuitive, if anything about this UI can be called that...
+
+## bugfixes
+* iPhone: rare javascript panic when switching between safari and another app
+* ie9: file-rename ui was borked
+
+## other changes
+* copyparty.exe: upgrade to pillow 10.1 (which adds a new font for thumbnails in chrome)
+  * still based on python 3.11.6 because 3.12 is currently slower than 3.11
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1009-0036  `v1.9.11`  bustin'
+
+okay, i swear this is the last version for weeks! probably
+
+## bugfixes
+* cachebuster didn't apply to dynamically loaded javascript files
+  * READMEs could fail to render with `ReferenceError: DOMPurify is not defined` after upgrading from a copyparty older than v1.9.2
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1008-2051  `v1.9.10`  badpwd
+
+## new features
+* argument `--log-badpwd` specifies how to log invalid login attempts;
+  * `0` = just a warning with no further information
+  * `1` = log incorrect password in plaintext (default)
+  * `2` = log sha512 hash of the incorrect password
+  * `1` and `2` are convenient for stuff like setting up autoban triggers for common passwords using fail2ban or similar
+
+## bugfixes
+* none!
+  * the formerly mentioned caching-directives bug turned out to be unreachable... oh well, better safe than sorry
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1007-2229  `v1.9.9`  fix cross-volume dedup moves
+
+## bugfixes
+* v1.6.2 introduced a bug which, when moving files between volumes, could cause the move operation to abort when it encounters a deduplicated file
+
+
+
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2023-1006-1750  `v1.9.8`  static filekeys
+
+## new features
+* #52 add alternative filekey generator:
+  * volflag `fka` changes the calculation to ignore filesize and inode-number, only caring about the absolute-path on the filesystem and the `--fk-salt`
+  * good for linking to markdown files which might be edited, but reduces security a tiny bit
+* add warning on startup if `--fk-salt` is too weak (for example when it was upgraded from before [v1.7.6](https://github.com/9001/copyparty/releases/tag/v1.7.6))
+  * removed the filekey upgrade feaure to ensure a weak fk-salt is not selected; a new filekey will be generated from scratch on startup if necessary
+
+## other changes
+* pyftpdlib upgraded to 1.5.8
+* copyparty.exe built on python 3.11.6
+  * the exe in this release will be replaced with an 3.12.0 exe as soon as [pillow adds 3.12 support](https://github.com/python-pillow/Pillow/issues/6941)
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2023-0930-2332  `v1.9.7`  better column hider
 

scripts/make-sfx.sh

@@ -420,16 +420,13 @@ rm -f ftp/pyftpdlib/{__main__,prefork}.py
 
 [ $no_ftp ] &&
 	rm -rf copyparty/ftpd.py ftp asyncore.py asynchat.py &&
-	sed -ri '/add_argument\("--ftp/d' copyparty/__main__.py &&
 	sed -ri '/\.ftp/d' copyparty/svchub.py
 
 [ $no_smb ] &&
-	rm -f copyparty/smbd.py &&
-	sed -ri '/add_argument\("--smb/d' copyparty/__main__.py
+	rm -f copyparty/smbd.py
 
 [ $no_zm ] &&
-	rm -rf copyparty/mdns.py copyparty/stolen/dnslib &&
-	sed -ri '/add_argument\("--zm/d' copyparty/__main__.py
+	rm -rf copyparty/mdns.py copyparty/stolen/dnslib
 
 [ $no_cm ] && {
 	rm -rf copyparty/web/mde.* copyparty/web/deps/easymde*

scripts/pyinstaller/build.sh

@@ -9,10 +9,13 @@ tee build2.sh | cmp build.sh && rm build2.sh || {
     [[ $r =~ [yY] ]] && mv build{2,}.sh && exec ./build.sh
 }
 
-[ -e up2k.sh ] && [ ! "$1" ] && ./up2k.sh
+clean=--clean
+[ "$1" = f ] && clean= && shift
 
 uname -s | grep WOW64 && m=64 || m=32
 uname -s | grep NT-10 && w10=1 || w7=1
+[ $w7 ] && [ -e up2k.sh ] && [ ! "$1" ] && ./up2k.sh
+
 [ $w7 ] && pyv=37 || pyv=311
 esuf=
 [ $w7 ] && [ $m = 32 ] && esuf=32
@@ -65,12 +68,18 @@ sed -r 's/1,2,3,0/'$a,$b,$c,$d'/;s/1\.2\.3/'$a.$b.$c/ <loader.rc >loader.rc2
 sed -ri s/copyparty.exe/copyparty$esuf.exe/ loader.rc2
 
 excl=(
+    asyncio
     copyparty.broker_mp
     copyparty.broker_mpw
+    copyparty.smbd
     ctypes.macholib
     curses
+    email._header_value_parser
+    email.header
+    email.parser
     inspect
     multiprocessing
+    packaging
     pdb
     pickle
     PIL.EpsImagePlugin
@@ -85,6 +94,7 @@ excl=(
     PIL.ImageShow
     PIL.ImageTk
     PIL.ImageWin
+    PIL.PdfParser
 ) || excl+=(
     PIL
     PIL.ExifTags
@@ -95,7 +105,7 @@ excl=(
 excl=( "${excl[@]/#/--exclude-module }" )
 
 $APPDATA/python/python$pyv/scripts/pyinstaller \
-    -y --clean -p mods --upx-dir=. \
+    -y $clean -p mods --upx-dir=. \
     ${excl[*]} \
     --version-file loader.rc2 -i loader.ico -n copyparty -c -F loader.py \
     --add-data 'mods/copyparty/res;copyparty/res' \

scripts/pyinstaller/deps.sha512

@@ -1,22 +1,23 @@
-d5510a24cb5e15d6d30677335bbc7624c319b371c0513981843dc51d9b3a1e027661096dfcfc540634222bb2634be6db55bf95185b30133cb884f1e47652cf53  altgraph-0.17.3-py2.py3-none-any.whl
+f117016b1e6a7d7e745db30d3e67f1acf7957c443a0dd301b6c5e10b8368f2aa4db6be9782d2d3f84beadd139bfeef4982e40f21ca5d9065cb794eeb0e473e82  altgraph-0.17.4-py2.py3-none-any.whl
 eda6c38fc4d813fee897e969ff9ecc5acc613df755ae63df0392217bbd67408b5c1f6c676f2bf5497b772a3eb4e1a360e1245e1c16ee83f0af555f1ab82c3977  Git-2.39.1-32-bit.exe
 17ce52ba50692a9d964f57a23ac163fb74c77fdeb2ca988a6d439ae1fe91955ff43730c073af97a7b3223093ffea3479a996b9b50ee7fba0869247a56f74baa6  pefile-2023.2.7-py3-none-any.whl
 f298e34356b5590dde7477d7b3a88ad39c622a2bcf3fcd7c53870ce8384dd510f690af81b8f42e121a22d3968a767d2e07595036b2ed7049c8ef4d112bcf3a61  pyinstaller-5.13.2-py3-none-win32.whl
-ea73aa54cc6d5db20dfb127e54562dabf890e4cd6171a91b10a51af2bcfc76e1d64cbdce4546df2dcfe42b624724c85b1cd05934be2413425b1f880222727b4f  pyinstaller-5.13.2-py3-none-win_amd64.whl
-2f4e3927a38cf7757bc9a1c06370d79209669a285a80f1b09cf9917137825c7022a50a56b351807e6e687e2c3a7bd7b2c5cc6daeb4d90e11920284c1a04a1cc3  pyinstaller_hooks_contrib-2023.8-py2.py3-none-any.whl
+f23615c522ed58b9a05978ba4c69c06224590f3a6adbd8e89b31838b181a57160739ceff1fc2ba6f4239b8fee46f92ce02910b2debda2710558ed42cff1ce3f1  pyinstaller-6.1.0-py3-none-win_amd64.whl
+5747b3b119629c4cf956f0eaa85f29218bb3680d3a4a262fa6e976e56b35067302e153d2c0a001505f2cb642b1f78752567889b3b82e342d6cd29aac8b70e92e  pyinstaller_hooks_contrib-2023.10-py2.py3-none-any.whl
 749a473646c6d4c7939989649733d4c7699fd1c359c27046bf5bc9c070d1a4b8b986bbc65f60d7da725baf16dbfdd75a4c2f5bb8335f2cb5685073f5fee5c2d1  pywin32_ctypes-0.2.2-py3-none-any.whl
+6e0d854040baff861e1647d2bece7d090bc793b2bd9819c56105b94090df54881a6a9b43ebd82578cd7c76d47181571b671e60672afd9def389d03c9dae84fcf  setuptools-68.2.2-py3-none-any.whl
 3c5adf0a36516d284a2ede363051edc1bcc9df925c5a8a9fa2e03cab579dd8d847fdad42f7fd5ba35992e08234c97d2dbfec40a9d12eec61c8dc03758f2bd88e  typing_extensions-4.4.0-py3-none-any.whl
 8d16a967a0a7872a7575b1005cf66915deacda6ee8611fbb52f42fc3e3beb2f901a5140c942a5d146bd412b92bfa9cbadd82beeba83df6d70930c6dc26608a5b  upx-4.1.0-win32.zip
 # u2c (win7)
-a7d259277af4948bf960682bc9fb45a44b9ae9a19763c8a7c313cef4aa9ec2d447d843e4a7c409e9312c8c8f863a24487a8ee4ffa6891e9b1c4e111bb4723861  certifi-2022.12.7-py3-none-any.whl
-2822c0dae180b1c8cfb7a70c8c00bad62af9afdbb18b656236680def9d3f1fcdcb8ef5eb64fc3b4c934385cd175ad5992a2284bcba78a243130de75b2d1650db  charset_normalizer-3.1.0-cp37-cp37m-win32.whl
+4562b1065c6bce7084eb575b654985c990e26034bfcd8db54629312f43ac737e264db7a2b4d8b797e09919a485cbc6af3fd0931690b7ed79b62bcc0736aec9fc  certifi-2023.7.22-py3-none-any.whl
+904eb57b13bea80aea861de86987e618665d37fa9ea0856e0125a9ba767a53e5064de0b9c4735435a2ddf4f16f7f7d2c75a682e1de83d9f57922bdca8e29988c  charset_normalizer-3.3.0-cp37-cp37m-win32.whl
 ffdd45326f4e91c02714f7a944cbcc2fdd09299f709cfa8aec0892053eef0134fb80d9ba3790afd319538a86feb619037cbf533e2f5939cb56b35bb17f56c858  idna-3.4-py3-none-any.whl
-220e0e122d5851aaccf633224dd7fbd3ba8c8d2720944d8019d6a276ed818d83e3426fe21807f22d673b5428f19fcf9a6b4e645f69bbecd967c568bb6aeb7c8d  requests-2.28.2-py3-none-any.whl
-8770011f4ad1fe40a3062e6cdf1fda431530c59ee7de3fc5f8c57db54bfdb71c3aa220ca0e0bb1874fc6700e9ebb57defbae54ac84938bc9ad8f074910106681  urllib3-1.26.14-py2.py3-none-any.whl
+b795abb26ba2f04f1afcfb196f21f638014b26c8186f8f488f1c2d91e8e0220962fbd259dbc9c3875222eb47fc95c73fc0606aaa6602b9ebc524809c9ba3501f  requests-2.31.0-py3-none-any.whl
+5a25cb9b79bb6107f9055dc3e9f62ebc6d4d9ca2c730d824985c93cd82406b723c200d6300c5064e42ee9fc7a2853d6ec6661394f3ed7bac03750e1f2a6840d1  urllib3-1.26.17-py2.py3-none-any.whl
 # win7
 91c025f7d94bcdf93df838fab67053165a414fc84e8496f92ecbb910dd55f6b6af5e360bbd051444066880c5a6877e75157bd95e150ead46e5c605930dfc50f2  future-0.18.2.tar.gz
 c06b3295d1d0b0f0a6f9a6cd0be861b9b643b4a5ea37857f0bd41c45deaf27bb927b71922dab74e633e43d75d04a9bd0d1c4ad875569740b0f2a98dd2bfa5113  importlib_metadata-5.0.0-py3-none-any.whl
-4e71295da5d1a26c71a0baa8905fdccb522bb16d56bc964db636de68688c5bf703f3b2880cdeea07138789e0eb4506e06f9ccd0da906c89d2cb6d55ad64659ea  pip-22.3-py3-none-any.whl
+016a8cbd09384f1a9a44cb0e8274df75a8bcb2f3966bb5d708c62145289efaa5db98f75256c97e4f8046735ce2e529fbb076f284a46cdb716e89a75660200ad9  pip-23.2.1-py3-none-any.whl
 6bb73cc2db795c59c92f2115727f5c173cacc9465af7710db9ff2f2aec2d73130d0992d0f16dcb3fac222dc15c0916562d0813b2337401022020673a4461df3d  python-3.7.9-amd64.exe
 500747651c87f59f2436c5ab91207b5b657856e43d10083f3ce27efb196a2580fadd199a4209519b409920c562aaaa7dcbdfb83ed2072a43eaccae6e2d056f31  python-3.7.9.exe
 68e1b618d988be56aaae4e2eb92bc0093627a00441c1074ebe680c41aa98a6161e52733ad0c59888c643a33fe56884e4f935178b2557fbbdd105e92e0d993df6  windows6.1-kb2533623-x64.msu
@@ -26,5 +27,6 @@ ba91ab0518c61eff13e5612d9e6b532940813f6b56e6ed81ea6c7c4d45acee4d98136a383a250675
 00558cca2e0ac813d404252f6e5aeacb50546822ecb5d0570228b8ddd29d94e059fbeb6b90393dee5abcddaca1370aca784dc9b095cbb74e980b3c024767fb24  Jinja2-3.1.2-py3-none-any.whl
 7f8f4daa4f4f2dbf24cdd534b2952ee3fba6334eb42b37465ccda3aa1cccc3d6204aa6bfffb8a83bf42ec59c702b5b5247d4c8ee0d4df906334ae53072ef8c4c  MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl
 8a6e2b13a2ec4ef914a5d62aad3db6464d45e525a82e07f6051ed10474eae959069e165dba011aefb8207cdfd55391d73d6f06362c7eb247b08763106709526e  mutagen-1.47.0-py3-none-any.whl
-08a033202b5c51e50609b2700dd69cbae30edb367f34762fd1633aae08b35949b4f67f12c75f25868a5b62b4956190d0cc8d201b170758d9c04a523bc8442b9b  Pillow-10.0.1-cp311-cp311-win_amd64.whl
-c86bbeacad3ae3c7bde747f5b4f09c11eced841add14e79ec4a064e5e29ebca35460e543ba735b11bfb882837d5ff4371ce64492d28d096b4686233c9a8cda6d  python-3.11.5-amd64.exe
+656015f5cc2c04aa0653ee5609c39a7e5f0b6a58c84fe26b20bd070c52d20b4effb810132f7fb771168483e9fd975cc3302837dd7a1a687ee058b0460c857cc4  packaging-23.2-py3-none-any.whl
+6401616fdfdd720d1aaa9a0ed1398d00664b28b6d84517dff8d1f9c416452610c6afa64cfb012a78e61d1cf4f6d0784eca6e7610957859e511f15bc6f3b3bd53  Pillow-10.1.0-cp311-cp311-win_amd64.whl
+36442c017d8fc603745d33ca888b5b1194644103cbe1ff53e32d9b0355e290d5efac655fa1ae1b8e552ad8468878dc600d550c1158224260ca463991442e5264  python-3.11.6-amd64.exe

scripts/pyinstaller/loader.py

@@ -106,20 +106,19 @@ def meichk():
     if filt not in sys.executable:
         filt = os.path.basename(sys.executable)
 
-    pids = []
-    ptn = re.compile(r"^([^\s]+)\s+([0-9]+)")
+    hits = []
     try:
-        procs = sp.check_output("tasklist").decode("utf-8", "replace")
+        cmd = "tasklist /fo csv".split(" ")
+        procs = sp.check_output(cmd).decode("utf-8", "replace")
     except:
         procs = ""  # winpe
 
-    for ln in procs.splitlines():
-        m = ptn.match(ln)
-        if m and filt in m.group(1).lower():
-            pids.append(int(m.group(2)))
+    for ln in procs.split("\n"):
+        if filt in ln.split('"')[:2][-1]:
+            hits.append(ln)
 
     mod = os.path.dirname(os.path.realpath(__file__))
-    if os.path.basename(mod).startswith("_MEI") and len(pids) == 2:
+    if os.path.basename(mod).startswith("_MEI") and len(hits) == 2:
         meicln(mod)
 
 

scripts/pyinstaller/notes.txt

@@ -15,19 +15,24 @@ uname -s | grep NT-10 && w10=1 || {
   w7=1; uname -s | grep WOW64 && w7x64=1 || w7x32=1
 }
 fns=(
-  altgraph-0.17.3-py2.py3-none-any.whl
+  altgraph-0.17.4-py2.py3-none-any.whl
   pefile-2023.2.7-py3-none-any.whl
-  pyinstaller-5.13.2-py3-none-win_amd64.whl
-  pyinstaller_hooks_contrib-2023.7-py2.py3-none-any.whl
+  pyinstaller_hooks_contrib-2023.10-py2.py3-none-any.whl
   pywin32_ctypes-0.2.2-py3-none-any.whl
+  setuptools-68.2.2-py3-none-any.whl
   upx-4.1.0-win32.zip
 )
 [ $w10 ] && fns+=(
+  pyinstaller-6.1.0-py3-none-win_amd64.whl
+  Jinja2-3.1.2-py3-none-any.whl
+  MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl
   mutagen-1.47.0-py3-none-any.whl
-  Pillow-10.0.1-cp311-cp311-win_amd64.whl
-  python-3.11.3-amd64.exe
-}
+  packaging-23.2-py3-none-any.whl
+  Pillow-10.1.0-cp311-cp311-win_amd64.whl
+  python-3.11.6-amd64.exe
+)
 [ $w7 ] && fns+=(
+  pyinstaller-5.13.2-py3-none-win32.whl
   certifi-2022.12.7-py3-none-any.whl
   chardet-5.1.0-py3-none-any.whl
   idna-3.4-py3-none-any.whl
@@ -37,7 +42,7 @@ fns=(
 [ $w7 ] && fns+=(
   future-0.18.2.tar.gz
   importlib_metadata-5.0.0-py3-none-any.whl
-  pip-22.3-py3-none-any.whl
+  pip-23.2.1-py3-none-any.whl
   typing_extensions-4.4.0-py3-none-any.whl
   zipp-3.10.0-py3-none-any.whl
 )
@@ -67,31 +72,26 @@ uname -s | grep NT-10 && w10=1 || w7=1
 [ $w7 ] && pyv=37 || pyv=311
 appd=$(cygpath.exe "$APPDATA")
 cd ~/Downloads &&
-unzip upx-*-win32.zip &&
+yes | unzip upx-*-win32.zip &&
 mv upx-*/upx.exe . &&
 python -m ensurepip &&
-python -m pip install --user -U pip-*.whl &&
-{ [ $w7 ] || python -m pip install --user -U mutagen-*.whl Pillow-*.whl; } &&
+{ [ $w10 ] || python -m pip install --user -U pip-*.whl; } &&
+{ [ $w7 ] || python -m pip install --user -U {packaging,setuptools,mutagen,Pillow,Jinja2,MarkupSafe}-*.whl; } &&
 { [ $w10 ] || python -m pip install --user -U {requests,urllib3,charset_normalizer,certifi,idna}-*.whl; } &&
 { [ $w10 ] || python -m pip install --user -U future-*.tar.gz importlib_metadata-*.whl typing_extensions-*.whl zipp-*.whl; } &&
 python -m pip install --user -U pyinstaller-*.whl pefile-*.whl pywin32_ctypes-*.whl pyinstaller_hooks_contrib-*.whl altgraph-*.whl &&
 sed -ri 's/--lzma/--best/' $appd/Python/Python$pyv/site-packages/pyinstaller/building/utils.py &&
 curl -fkLO https://192.168.123.1:3923/cpp/scripts/uncomment.py &&
-python uncomment.py $(for d in $appd/Python/Python$pyv/site-packages/{requests,urllib3,charset_normalizer,certifi,idna}; do find $d -name \*.py; done) &&
+python uncomment.py 1 $(for d in $appd/Python/Python$pyv/site-packages/{requests,urllib3,charset_normalizer,certifi,idna,mutagen,PIL,jinja2,markupsafe}; do find $d -name \*.py; done) &&
 cd &&
 rm -f build.sh &&
 curl -fkLO https://192.168.123.1:3923/cpp/scripts/pyinstaller/build.sh &&
-curl -fkLO https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.sh &&
+{ [ $w10 ] || curl -fkLO https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.sh; } &&
 echo ok
 # python -m pip install --user -U Pillow-9.2.0-cp37-cp37m-win32.whl
 # sed -ri 's/, bestopt, /]+bestopt+[/' $APPDATA/Python/Python37/site-packages/pyinstaller/building/utils.py
 # sed -ri 's/(^\s+bestopt = ).*/\1["--best","--lzma","--ultra-brute"]/' $APPDATA/Python/Python37/site-packages/pyinstaller/building/utils.py
 
-===[ win10: copy-paste into git-bash ]=========================
-#for f in $appd/Python/Python311/site-packages/mutagen/*.py; do awk -i inplace '/^\s*def _?(save|write)/{sub(/d.*/,"    ");s=$0;ns=length(s)} ns&&/[^ ]/&&substr($0,0,ns)!=s{ns=0} !ns' "$f"; done &&
-python uncomment.py $appd/Python/Python311/site-packages/{mutagen,PIL,jinja2,markupsafe}/*.py &&
-echo ok
-
 
 ## ============================================================
 ## notes

scripts/sfx.py

@@ -295,7 +295,10 @@ def unpack():
         # the only possible input is a single tar.bz2
         # which gets hardcoded into this script at build stage
         # skip 0
-        tf.extractall(mine)
+        try:
+            tf.extractall(mine, filter="tar")
+        except TypeError:
+            tf.extractall(mine)
 
     os.remove(tar)
 

scripts/test/smoketest.py

@@ -97,6 +97,7 @@ def tc1(vflags):
         ovid = f.read()
 
     args = [
+        "-q",
         "-p4321",
         "-e2dsa",
         "-e2tsr",

srv/expand/README.md

@@ -0,0 +1,26 @@
+## text expansion
+
+enable expansion of placeholder variables in `README.md` and prologue/epilogue files with `--exp` and customize the list of allowed placeholders to expand using `--exp-md` and `--exp-lg`
+
+| explanation          | placeholder          |
+| -------------------- | -------------------- |
+| your ip address      | {{self.ip}}          |
+| your user-agent      | {{self.ua}}          |
+| your username        | {{self.uname}}       |
+| the `Host` you see   | {{self.host}}        |
+| server unix time     | {{srv.itime}}        |
+| server datetime      | {{srv.htime}}        |
+| server name          | {{cfg.name}}         |
+| logout after         | {{cfg.logout}} hours |
+| vol reindex interval | {{vf.scan}}          |
+| thumbnail size       | {{vf.thsize}}        |
+| your country         | {{hdr.cf_ipcountry}} |
+
+placeholders starting with...
+* `self.` are grabbed from copyparty's internal state; anything in `httpcli.py` is fair game
+* `cfg.` are the global server settings
+* `vf.` are the volflags of the current volume
+* `hdr.` are grabbed from the client headers; any header is supported, just add it (in lowercase) to the allowlist
+* `srv.` are processed inside the `_expand` function in httpcli
+
+for example (bad example), `hdr_cf_ipcountry` maps to the header `CF-IPCountry` (which is generated by cloudflare before the request is passed on to your server / copyparty)

tests/util.py

@@ -109,7 +109,7 @@ class Cfg(Namespace):
     def __init__(self, a=None, v=None, c=None):
         ka = {}
 
-        ex = "daw dav_auth dav_inf dav_mac dav_rt dotsrch e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp ed emp force_js getmod grid hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_logues no_mv no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw rand smb th_no_crop vague_403 vc ver xdev xlink xvol"
+        ex = "daw dav_auth dav_inf dav_mac dav_rt dotsrch e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp ed emp exp force_js getmod grid hardlink ih ihead magic never_symlink nid nih no_acode no_athumb no_dav no_dedup no_del no_dupe no_logues no_mv no_readme no_robots no_sb_md no_sb_lg no_scandir no_tarcmp no_thumb no_vthumb no_zip nrand nw rand smb th_no_crop vague_403 vc ver xdev xlink xvol"
         ka.update(**{k: False for k in ex.split()})
 
         ex = "dotpart no_rescan no_sendfile no_voldump plain_ip"
@@ -124,12 +124,15 @@ class Cfg(Namespace):
         ex = "df loris re_maxage rproxy rsp_jtr rsp_slp s_wr_slp theme themes turbo"
         ka.update(**{k: 0 for k in ex.split()})
 
-        ex = "ah_alg bname doctitle favico html_head lg_sbf log_fk md_sbf mth name textfiles unlist vname R RS SR"
+        ex = "ah_alg bname doctitle favico html_head lg_sbf log_fk md_sbf name textfiles unlist vname R RS SR"
         ka.update(**{k: "" for k in ex.split()})
 
         ex = "on403 on404 xad xar xau xban xbd xbr xbu xiu xm"
         ka.update(**{k: [] for k in ex.split()})
 
+        ex = "exp_lg exp_md"
+        ka.update(**{k: {} for k in ex.split()})
+
         super(Cfg, self).__init__(
             a=a or [],
             v=v or [],
@@ -141,8 +144,11 @@ class Cfg(Namespace):
             fk_salt="a" * 16,
             unpost=600,
             u2sort="s",
+            u2ts="c",
+            sort="href",
             mtp=[],
-            mte="a",
+            mte={"a": True},
+            mth={},
             lang="eng",
             logout=573,
             **ka