v1.1.0

.gitignore

@@ -9,6 +9,7 @@ buildenv/
 build/
 dist/
 sfx/
+py2/
 .venv/
 
 # ide

README.md

@@ -46,7 +46,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [markdown viewer](#markdown-viewer) - and there are *two* editors
     * [other tricks](#other-tricks)
     * [searching](#searching) - search by size, date, path/name, mp3-tags, ...
-* [server config](#server-config)
+* [server config](#server-config) - using arguments or config files, or a mix of both
     * [file indexing](#file-indexing)
     * [upload rules](#upload-rules) - set upload rules using volume flags
     * [compress uploads](#compress-uploads) - files can be autocompressed on upload
@@ -60,11 +60,17 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [up2k](#up2k) - quick outline of the up2k protocol, see [uploading](#uploading) for the web-client
     * [why chunk-hashes](#why-chunk-hashes) - a single sha512 would be better, right?
 * [performance](#performance) - defaults are usually fine - expect `8 GiB/s` download, `1 GiB/s` upload
+    * [client-side](#client-side) - when uploading files
 * [security](#security) - some notes on hardening
     * [gotchas](#gotchas) - behavior that might be unexpected
 * [recovering from crashes](#recovering-from-crashes)
     * [client crashes](#client-crashes)
         * [frefox wsod](#frefox-wsod) - firefox 87 can crash during uploads
+* [HTTP API](#HTTP-API)
+    * [read](#read)
+    * [write](#write)
+    * [admin](#admin)
+    * [general](#general)
 * [dependencies](#dependencies) - mandatory deps
     * [optional dependencies](#optional-dependencies) - install these to enable bonus features
     * [install recommended deps](#install-recommended-deps)
@@ -153,7 +159,7 @@ feature summary
 * browser
   * ☑ [navpane](#navpane) (directory tree sidebar)
   * ☑ file manager (cut/paste, delete, [batch-rename](#batch-rename))
-  * ☑ audio player (with OS media controls)
+  * ☑ audio player (with OS media controls and opus transcoding)
   * ☑ image gallery with webm player
   * ☑ [thumbnails](#thumbnails)
     * ☑ ...of images using Pillow
@@ -218,6 +224,7 @@ some improvement ideas
 * Windows: python 2.7 cannot index non-ascii filenames with `-e2d`
 * Windows: python 2.7 cannot handle filenames with mojibake
 * `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions (macos, some linux)
+* `--th-ff-swr` may fix audio thumbnails on some FFmpeg versions
 
 ## general bugs
 
@@ -304,6 +311,7 @@ the browser has the following hotkeys  (always qwerty)
 * `B` toggle breadcrumbs / [navpane](#navpane)
 * `I/K` prev/next folder
 * `M` parent folder (or unexpand current)
+* `V` toggle folders / textfiles in the navpane
 * `G` toggle list / [grid view](#thumbnails)
 * `T` toggle thumbnails / icons
 * `ctrl-X` cut selected files/folders
@@ -315,6 +323,10 @@ the browser has the following hotkeys  (always qwerty)
   * ctrl+`Up/Down` move cursor and scroll viewport
   * `Space` toggle file selection
   * `Ctrl-A` toggle select all
+* when a textfile is open:
+  * `I/K` prev/next textfile
+  * `S` toggle selection of open file
+  * `M` close textfile
 * when playing audio:
   * `J/L` prev/next song
   * `U/O` skip 10sec back/forward
@@ -366,6 +378,8 @@ press `g` to toggle grid-view instead of the file listing,  and `t` toggles icon
 
 it does static images with Pillow and uses FFmpeg for video files, so you may want to `--no-thumb` or maybe just `--no-vthumb` depending on how dangerous your users are
 
+audio files are covnerted into spectrograms using FFmpeg unless you `--no-athumb` (and some FFmpeg builds may need `--th-ff-swr`)
+
 images with the following names (see `--th-covers`) become the thumbnail of the folder they're in: `folder.png`, `folder.jpg`, `cover.png`, `cover.jpg`
 
 in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked
@@ -421,6 +435,8 @@ see [up2k](#up2k) for details on how it works
 
 **protip:** you can avoid scaring away users with [docs/minimal-up2k.html](docs/minimal-up2k.html) which makes it look [much simpler](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
 
+**protip:** if you enable `favicon` in the `[⚙️] settings` tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress
+
 the up2k UI is the epitome of polished inutitive experiences:
 * "parallel uploads" specifies how many chunks to upload at the same time
 * `[🏃]` analysis of other files should continue while one is uploading
@@ -546,6 +562,8 @@ and there are *two* editors
 
 * you can link a particular timestamp in an audio file by adding it to the URL, such as `&20` / `&20s` / `&1m20` / `&t=1:20` after the `.../#af-c8960dab`
 
+* get a plaintext file listing by adding `?ls=t` to a URL, or a compact colored one with `?ls=v` (for unix terminals)
+
 * if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider https://ocv.me/dev/?media-osd-bgone.ps1
 
 * click the bottom-left `π` to open a javascript prompt for debugging
@@ -574,6 +592,10 @@ add the argument `-e2ts` to also scan/index tags from music files, which brings
 
 # server config
 
+using arguments or config files, or a mix of both:
+* config files (`-c some.conf`) can set additional commandline arguments; see [./docs/example.conf](docs/example.conf)
+
+
 ## file indexing
 
 file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volume flags, or a mix of both.
@@ -716,7 +738,7 @@ that'll run the command `notify-send` with the path to the uploaded file as the
 
 note that it will only trigger on new unique files, not dupes
 
-and it will occupy the parsing threads, so fork anything expensive, or if you want to intentionally queue/singlethread you can combine it with `--no-mtag-mt`
+and it will occupy the parsing threads, so fork anything expensive, or if you want to intentionally queue/singlethread you can combine it with `--mtag-mt 1`
 
 if this becomes popular maybe there should be a less janky way to do it actually
 
@@ -747,7 +769,7 @@ TLDR: yes
 | zip selection   |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | file rename     |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | file cut/paste  |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
-| navpane         |  -  | `*2` | yep  | yep  |  yep  | yep  | yep | yep  |
+| navpane         |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | image viewer    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | video player    |  -  | yep  | yep  | yep  |  yep  | yep  | yep | yep  |
 | markdown editor |  -  |  -   | yep  | yep  |  yep  | yep  | yep | yep  |
@@ -759,7 +781,6 @@ TLDR: yes
 * internet explorer 6 to 8 behave the same
 * firefox 52 and chrome 49 are the final winxp versions
 * `*1` yes, but extremely slow (ie10: `1 MiB/s`, ie11: `270 KiB/s`)
-* `*2` causes a full-page refresh on each navigation
 * `*3` using a wasm decoder which consumes a bit more power
 
 quick summary of more eccentric web-browsers trying to view a directory index:
@@ -846,8 +867,6 @@ hashwasm would solve the streaming issue but reduces hashing speed for sha512 (x
 
 defaults are usually fine - expect `8 GiB/s` download, `1 GiB/s` upload
 
-you can ignore the `cannot efficiently use multiple CPU cores` message, very unlikely to be a problem
-
 below are some tweaks roughly ordered by usefulness:
 
 * `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
@@ -861,6 +880,21 @@ below are some tweaks roughly ordered by usefulness:
   ...however it adds an overhead to internal communication so it might be a net loss, see if it works 4 u
 
 
+## client-side
+
+when uploading files,
+
+* chrome is recommended, at least compared to firefox:
+  * up to 90% faster when hashing, especially on SSDs
+  * up to 40% faster when uploading over extremely fast internets
+  * but [up2k.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py) can be 40% faster than chrome again
+
+* if you're cpu-bottlenecked, or the browser is maxing a cpu core:
+  * up to 30% faster uploads if you hide the upload status list by switching away from the `[🚀]` up2k ui-tab (or closing it)
+    * switching to another browser-tab also works, the favicon will update every 10 seconds in that case
+  * unlikely to be a problem, but can happen when uploding many small files, or your internet is too fast, or PC too slow
+
+
 # security
 
 some notes on hardening
@@ -904,6 +938,83 @@ however you can hit `F12` in the up2k tab and use the devtools to see how far yo
   `await fetch('/inc', {method:'PUT', body:JSON.stringify(ng,null,1)})`
 
 
+# HTTP API
+
+* table-column `params` = URL parameters; `?foo=bar&qux=...`
+* table-column `body` = POST payload
+* method `jPOST` = json post
+* method `mPOST` = multipart post
+* method `uPOST` = url-encoded post
+* `FILE` = conventional HTTP file upload entry (rfc1867 et al, filename in `Content-Disposition`)
+
+authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
+
+## read
+
+| method | params | result |
+|--|--|--|
+| GET | `?ls` | list files/folders at URL as JSON |
+| GET | `?ls&dots` | list files/folders at URL as JSON, including dotfiles |
+| GET | `?ls=t` | list files/folders at URL as plaintext |
+| GET | `?ls=v` | list files/folders at URL, terminal-formatted |
+| GET | `?b` | list files/folders at URL as simplified HTML |
+| GET | `?tree=.` | list one level of subdirectories inside URL |
+| GET | `?tree` | list one level of subdirectories for each level until URL |
+| GET | `?tar` | download everything below URL as a tar file |
+| GET | `?zip=utf-8` | download everything below URL as a zip file |
+| GET | `?ups` | show recent uploads from your IP |
+| GET | `?ups&filter=f` | ...where URL contains `f` |
+| GET | `?mime=foo` | specify return mimetype `foo` |
+| GET | `?raw` | get markdown file at URL as plaintext |
+| GET | `?txt` | get file at URL as plaintext |
+| GET | `?txt=iso-8859-1` | ...with specific charset |
+| GET | `?th` | get image/video at URL as thumbnail |
+| GET | `?th=opus` | convert audio file to 128kbps opus |
+
+| method | body | result |
+|--|--|--|
+| jPOST | `{"q":"foo"}` | do a server-wide search; see the `[🔎]` search tab `raw` field for syntax |
+
+| method | params | body | result |
+|--|--|--|--|
+| jPOST | `?tar` | `["foo","bar"]` | download folders `foo` and `bar` inside URL as a tar file |
+
+## write
+
+| method | params | result |
+|--|--|--|
+| GET | `?move=/foo/bar` | move/rename the file/folder at URL to /foo/bar |
+
+| method | params | body | result |
+|--|--|--|--|
+| PUT | | (binary data) | upload into file at URL |
+| PUT | `?gz` | (binary data) | compress with gzip and write into file at URL |
+| PUT | `?xz` | (binary data) | compress with xz and write into file at URL |
+| mPOST | | `act=bput`, `f=FILE` | upload `FILE` into the folder at URL |
+| mPOST | `?j` | `act=bput`, `f=FILE` | ...and reply with json |
+| mPOST | | `act=mkdir`, `name=foo` | create directory `foo` at URL |
+| GET | `?delete` | | delete URL recursively |
+| jPOST | `?delete` | `["/foo","/bar"]` | delete `/foo` and `/bar` recursively |
+| uPOST | | `msg=foo` | send message `foo` into server log |
+| mPOST | | `act=tput`, `body=TEXT` | overwrite markdown document at URL |
+
+server behavior of `msg` can be reconfigured with `--urlform`
+
+## admin
+
+| method | params | result |
+|--|--|--|
+| GET | `?reload=cfg` | reload config files and rescan volumes |
+| GET | `?scan` | initiate a rescan of the volume which provides URL |
+| GET | `?stack` | show a stacktrace of all threads |
+
+## general
+
+| method | params | result |
+|--|--|--|
+| GET | `?pw=x` | logout |
+
+
 # dependencies
 
 mandatory deps:
@@ -920,7 +1031,7 @@ enable music tags:
 
 enable [thumbnails](#thumbnails) of...
 * **images:** `Pillow` (requires py2.7 or py3.5+)
-* **videos:** `ffmpeg` and `ffprobe` somewhere in `$PATH`
+* **videos/audio:** `ffmpeg` and `ffprobe` somewhere in `$PATH`
 * **HEIF pictures:** `pyheif-pillow-opener` (requires Linux or a C compiler)
 * **AVIF pictures:** `pillow-avif-plugin`
 
@@ -961,6 +1072,7 @@ if you don't need all the features, you can repack the sfx and save a bunch of s
 the features you can opt to drop are
 * `ogv`.js, the opus/vorbis decoder which is needed by apple devices to play foss audio files, saves ~192k
 * `cm`/easymde, the "fancy" markdown editor, saves ~92k
+* `hl`, prism, the syntax hilighter, saves ~41k
 * `fnt`, source-code-pro, the monospace font, saves ~9k
 * `dd`, the custom mouse cursor for the media player tray tab, saves ~2k
 

bin/up2k.py

@@ -3,7 +3,7 @@ from __future__ import print_function, unicode_literals
 
 """
 up2k.py: upload to copyparty
-2021-10-12, v0.9, ed <irc.rizon.net>, MIT-Licensed
+2021-10-31, v0.11, ed <irc.rizon.net>, MIT-Licensed
 https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py
 
 - dependencies: requests
@@ -250,9 +250,10 @@ def walkdirs(tops):
     """recursive statdir for a list of tops, yields [top, relpath, stat]"""
     sep = "{0}".format(os.sep).encode("ascii")
     for top in tops:
-        stop = top
         if top[-1:] == sep:
-            stop = os.path.dirname(top.rstrip(sep))
+            stop = top.rstrip(sep)
+        else:
+            stop = os.path.dirname(top)
 
         if os.path.isdir(top):
             for ap, inf in walkdir(top):
@@ -454,6 +455,7 @@ class Ctl(object):
         self.nbytes = nbytes
 
         if ar.td:
+            requests.packages.urllib3.disable_warnings()
             req_ses.verify = False
         if ar.te:
             req_ses.verify = ar.te

contrib/README.md

@@ -30,6 +30,7 @@ however if your copyparty is behind a reverse-proxy, you may want to use [`share
 # OS integration
 init-scripts to start copyparty as a service
 * [`systemd/copyparty.service`](systemd/copyparty.service) runs the sfx normally
+* [`rc/copyparty`](rc/copyparty) runs sfx normally on freebsd, create a `copyparty` user
 * [`systemd/prisonparty.service`](systemd/prisonparty.service) runs the sfx in a chroot
 * [`openrc/copyparty`](openrc/copyparty)
 

contrib/rc/copyparty

@@ -0,0 +1,31 @@
+#!/bin/sh
+#
+# PROVIDE: copyparty
+# REQUIRE: networking
+# KEYWORD:
+
+. /etc/rc.subr
+
+name="copyparty"
+rcvar="copyparty_enable"
+copyparty_user="copyparty"
+copyparty_args="-e2dsa -v /storage:/storage:r" # change as you see fit
+copyparty_command="/usr/local/bin/python3.8 /usr/local/copyparty/copyparty-sfx.py ${copyparty_args}"
+pidfile="/var/run/copyparty/${name}.pid"
+command="/usr/sbin/daemon"
+command_args="-P ${pidfile} -r -f ${copyparty_command}"
+
+stop_postcmd="copyparty_shutdown"
+
+copyparty_shutdown()
+{
+        if [ -e "${pidfile}" ]; then
+                echo "Stopping supervising daemon."
+                kill -s TERM `cat ${pidfile}`
+        fi
+}
+
+load_rc_config $name
+: ${copyparty_enable:=no}
+
+run_rc_command "$1"

contrib/systemd/copyparty.service

@@ -3,10 +3,16 @@
 #
 # installation:
 #   cp -pv copyparty.service /etc/systemd/system && systemctl enable --now copyparty
+#   restorecon -vr /etc/systemd/system/copyparty.service
+#   firewall-cmd --permanent --add-port={80,443,3923}/tcp
+#   firewall-cmd --reload
 #
 # you may want to:
-#   change '/usr/bin/python' to another interpreter
+#   change '/usr/bin/python3' to another interpreter
 #   change '/mnt::rw' to another location or permission-set
+#   remove '-p 80,443,3923' to only listen on port 3923
+#   add '-i 127.0.0.1' to only allow local connections
+#   add '--use-fpool' if uploading into nfs locations
 #
 # with `Type=notify`, copyparty will signal systemd when it is ready to
 #   accept connections; correctly delaying units depending on copyparty.
@@ -14,11 +20,8 @@
 #   python disabling line-buffering, so messages are out-of-order:
 #   https://user-images.githubusercontent.com/241032/126040249-cb535cc7-c599-4931-a796-a5d9af691bad.png
 #
-# enable line-buffering for realtime logging (slight performance cost):
+# if you remove -q to enable logging, you may also want to remove the
-#   modify ExecStart and prefix it with `/usr/bin/stdbuf -oL` like so:
+#   following line to enable buffering (slightly better performance):
-#   ExecStart=/usr/bin/stdbuf -oL /usr/bin/python3 [...]
-# but some systemd versions require this instead (higher performance cost):
-#   inside the [Service] block, add the following line:
 #   Environment=PYTHONUNBUFFERED=x
 
 [Unit]
@@ -27,8 +30,10 @@ Description=copyparty file server
 [Service]
 Type=notify
 SyslogIdentifier=copyparty
-ExecStart=/usr/bin/python3 /usr/local/bin/copyparty-sfx.py -q -v /mnt::rw
+Environment=PYTHONUNBUFFERED=x
+ExecReload=/bin/kill -s USR1 $MAINPID
 ExecStartPre=/bin/bash -c 'mkdir -p /run/tmpfiles.d/ && echo "x /tmp/pe-copyparty*" > /run/tmpfiles.d/copyparty.conf'
+ExecStart=/usr/bin/python3 /usr/local/bin/copyparty-sfx.py -q -p 80,443,3923 -v /mnt::rw
 
 [Install]
 WantedBy=multi-user.target

copyparty/__main__.py

@@ -20,7 +20,7 @@ import threading
 import traceback
 from textwrap import dedent
 
-from .__init__ import E, WINDOWS, VT100, PY2, unicode
+from .__init__ import E, WINDOWS, ANYWIN, VT100, PY2, unicode
 from .__version__ import S_VERSION, S_BUILD_DT, CODENAME
 from .svchub import SvcHub
 from .util import py_desc, align_tab, IMPLICATIONS, ansi_re
@@ -186,6 +186,32 @@ def configure_ssl_ciphers(al):
         sys.exit(0)
 
 
+def args_from_cfg(cfg_path):
+    ret = []
+    skip = False
+    with open(cfg_path, "rb") as f:
+        for ln in [x.decode("utf-8").strip() for x in f]:
+            if not ln:
+                skip = False
+                continue
+
+            if ln.startswith("#"):
+                continue
+
+            if not ln.startswith("-"):
+                continue
+
+            if skip:
+                continue
+
+            try:
+                ret.extend(ln.split(" ", 1))
+            except:
+                ret.append(ln)
+
+    return ret
+
+
 def sighandler(sig=None, frame=None):
     msg = [""] * 5
     for th in threading.enumerate():
@@ -208,6 +234,8 @@ def run_argparse(argv, formatter):
     except:
         fk_salt = "hunter2"
 
+    cores = os.cpu_count() if hasattr(os, "cpu_count") else 4
+
     sects = [
         [
             "accounts",
@@ -333,7 +361,7 @@ def run_argparse(argv, formatter):
     ap2 = ap.add_argument_group('general options')
     ap2.add_argument("-c", metavar="PATH", type=u, action="append", help="add config file")
     ap2.add_argument("-nc", metavar="NUM", type=int, default=64, help="max num clients")
-    ap2.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores")
+    ap2.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores, 0=all")
     ap2.add_argument("-a", metavar="ACCT", type=u, action="append", help="add account, USER:PASS; example [ed:wark")
     ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, SRC:DST:FLAG; example [.::r], [/mnt/nas/music:/music:r:aed")
     ap2.add_argument("-ed", action="store_true", help="enable ?dots")
@@ -395,23 +423,30 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--lf-url", metavar="RE", type=u, default=r"^/\.cpr/|\?th=[wj]$", help="dont log URLs matching")
 
     ap2 = ap.add_argument_group('admin panel options')
+    ap2.add_argument("--no-reload", action="store_true", help="disable ?reload=cfg (reload users/volumes/volflags from config file)")
     ap2.add_argument("--no-rescan", action="store_true", help="disable ?scan (volume reindexing)")
     ap2.add_argument("--no-stack", action="store_true", help="disable ?stack (list all stacks)")
 
     ap2 = ap.add_argument_group('thumbnail options')
     ap2.add_argument("--no-thumb", action="store_true", help="disable all thumbnails")
+    ap2.add_argument("--no-athumb", action="store_true", help="disable audio thumbnails (spectrograms)")
     ap2.add_argument("--no-vthumb", action="store_true", help="disable video thumbnails")
     ap2.add_argument("--th-size", metavar="WxH", default="320x256", help="thumbnail res")
-    ap2.add_argument("--th-mt", metavar="CORES", type=int, default=0, help="max num cpu cores to use, 0=all")
+    ap2.add_argument("--th-mt", metavar="CORES", type=int, default=cores, help="num cpu cores to use for generating thumbnails")
     ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image")
     ap2.add_argument("--th-no-jpg", action="store_true", help="disable jpg output")
     ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output")
     ap2.add_argument("--th-ff-jpg", action="store_true", help="force jpg for video thumbs")
+    ap2.add_argument("--th-ff-swr", action="store_true", help="use swresample instead of soxr for audio thumbs")
     ap2.add_argument("--th-poke", metavar="SEC", type=int, default=300, help="activity labeling cooldown")
     ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval; 0=disabled")
     ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age")
     ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat for")
 
+    ap2 = ap.add_argument_group('transcoding options')
+    ap2.add_argument("--no-acode", action="store_true", help="disable audio transcoding")
+    ap2.add_argument("--ac-maxage", metavar="SEC", type=int, default=86400, help="delete transcode output after SEC seconds")
+
     ap2 = ap.add_argument_group('general db options')
     ap2.add_argument("-e2d", action="store_true", help="enable up2k database")
     ap2.add_argument("-e2ds", action="store_true", help="enable up2k db-scanner, sets -e2d")
@@ -419,7 +454,6 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume data (db, thumbs)")
     ap2.add_argument("--no-hash", metavar="PTN", type=u, help="regex: disable hashing of matching paths during e2ds folder scans")
     ap2.add_argument("--no-idx", metavar="PTN", type=u, help="regex: disable indexing of matching paths during e2ds folder scans")
-    ap2.add_argument("--re-int", metavar="SEC", type=int, default=30, help="disk rescan check interval")
     ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval, 0=off, can be set per-volume with the 'scan' volflag")
     ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
     
@@ -428,8 +462,8 @@ def run_argparse(argv, formatter):
     ap2.add_argument("-e2ts", action="store_true", help="enable metadata scanner, sets -e2t")
     ap2.add_argument("-e2tsr", action="store_true", help="rescan all metadata, sets -e2ts")
     ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead")
-    ap2.add_argument("--no-mtag-mt", action="store_true", help="disable tag-read parallelism")
     ap2.add_argument("--no-mtag-ff", action="store_true", help="never use FFprobe as tag reader")
+    ap2.add_argument("--mtag-mt", metavar="CORES", type=int, default=cores, help="num cpu cores to use for tag scanning")
     ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
     ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
         default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,res,.fps,ahash,vhash")
@@ -437,8 +471,10 @@ def run_argparse(argv, formatter):
         default=".vq,.aq,vc,ac,res,.fps")
     ap2.add_argument("-mtp", metavar="M=[f,]bin", type=u, action="append", help="read tag M using bin")
 
-    ap2 = ap.add_argument_group('appearance options')
+    ap2 = ap.add_argument_group('ui options')
+    ap2.add_argument("--js-browser", metavar="L", type=u, help="URL to additional JS to include")
     ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include")
+    ap2.add_argument("--textfiles", metavar="CSV", type=u, default="txt,nfo,diz,cue,readme", help="file extensions to present as plaintext")
 
     ap2 = ap.add_argument_group('debug options')
     ap2.add_argument("--no-sendfile", action="store_true", help="disable sendfile")
@@ -481,7 +517,12 @@ def main(argv=None):
     if HAVE_SSL:
         ensure_cert()
 
-    deprecated = [["-e2s", "-e2ds"]]
+    for k, v in zip(argv, argv[1:]):
+        if k == "-c":
+            supp = args_from_cfg(v)
+            argv.extend(supp)
+
+    deprecated = []
     for dk, nk in deprecated:
         try:
             idx = argv.index(dk)
@@ -493,6 +534,12 @@ def main(argv=None):
         argv[idx] = nk
         time.sleep(2)
 
+    try:
+        if len(argv) == 1 and (ANYWIN or not os.geteuid()):
+            argv.extend(["-p80,443,3923", "--ign-ebind"])
+    except:
+        pass
+
     try:
         al = run_argparse(argv, RiceFormatter)
     except AssertionError:

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 0, 11)
+VERSION = (1, 1, 0)
-CODENAME = "sufficient"
+CODENAME = "opus"
-BUILD_DT = (2021, 10, 18)
+BUILD_DT = (2021, 11, 6)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -526,8 +526,27 @@ class AuthSrv(object):
 
         yield prev, True
 
+    def _map_volume(self, src, dst, mount, daxs, mflags):
+        if dst in mount:
+            m = "multiple filesystem-paths mounted at [/{}]:\n  [{}]\n  [{}]"
+            self.log(m.format(dst, mount[dst], src), c=1)
+            raise Exception("invalid config")
+
+        if src in mount.values():
+            m = "warning: filesystem-path [{}] mounted in multiple locations:"
+            m = m.format(src)
+            for v in [k for k, v in mount.items() if v == src] + [dst]:
+                m += "\n  /{}".format(v)
+
+            self.log(m, c=3)
+
+        mount[dst] = src
+        daxs[dst] = AXS()
+        mflags[dst] = {}
+
     def _parse_config_file(self, fd, acct, daxs, mflags, mount):
         # type: (any, str, dict[str, AXS], any, str) -> None
+        skip = False
         vol_src = None
         vol_dst = None
         self.line_ctr = 0
@@ -537,6 +556,11 @@ class AuthSrv(object):
                 vol_src = None
                 vol_dst = None
 
+            if skip:
+                if not ln:
+                    skip = False
+                continue
+
             if not ln or ln.startswith("#"):
                 continue
 
@@ -544,6 +568,8 @@ class AuthSrv(object):
                 if ln.startswith("u "):
                     u, p = ln[2:].split(":", 1)
                     acct[u] = p
+                elif ln.startswith("-"):
+                    skip = True  # argv
                 else:
                     vol_src = ln
                 continue
@@ -556,9 +582,7 @@ class AuthSrv(object):
                 # cfg files override arguments and previous files
                 vol_src = bos.path.abspath(vol_src)
                 vol_dst = vol_dst.strip("/")
-                mount[vol_dst] = vol_src
+                self._map_volume(vol_src, vol_dst, mount, daxs, mflags)
-                daxs[vol_dst] = AXS()
-                mflags[vol_dst] = {}
                 continue
 
             try:
@@ -597,7 +621,7 @@ class AuthSrv(object):
         if uname == "":
             uname = "*"
 
-        for un in uname.split(","):
+        for un in uname.replace(",", " ").strip().split():
             if "r" in lvl:
                 axs.uread[un] = 1
 
@@ -663,9 +687,7 @@ class AuthSrv(object):
                 # print("\n".join([src, dst, perms]))
                 src = bos.path.abspath(src)
                 dst = dst.strip("/")
-                mount[dst] = src
+                self._map_volume(src, dst, mount, daxs, mflags)
-                daxs[dst] = AXS()
-                mflags[dst] = {}
 
                 for x in perms.split(":"):
                     lvl, uname = x.split(",", 1) if "," in x else [x, ""]
@@ -986,7 +1008,7 @@ class AuthSrv(object):
             v, _ = vfs.get("/", "*", False, True)
             if self.warn_anonwrite and os.getcwd() == v.realpath:
                 self.warn_anonwrite = False
-                msg = "anyone can read/write the current directory: {}"
+                msg = "anyone can read/write the current directory: {}\n"
                 self.log(msg.format(v.realpath), c=1)
         except Pebkac:
             self.warn_anonwrite = True

copyparty/bos/path.py

@@ -21,6 +21,10 @@ def getsize(p):
     return os.path.getsize(fsenc(p))
 
 
+def isfile(p):
+    return os.path.isfile(fsenc(p))
+
+
 def isdir(p):
     return os.path.isdir(fsenc(p))
 

copyparty/broker_mp.py

@@ -62,6 +62,11 @@ class BrokerMp(object):
 
             procs.pop()
 
+    def reload(self):
+        self.log("broker", "reloading")
+        for _, proc in enumerate(self.procs):
+            proc.q_pend.put([0, "reload", []])
+
     def collector(self, proc):
         """receive message from hub in other process"""
         while True:

copyparty/broker_mpw.py

@@ -29,7 +29,7 @@ class MpWorker(object):
         # we inherited signal_handler from parent,
         # replace it with something harmless
         if not FAKE_MP:
-            for sig in [signal.SIGINT, signal.SIGTERM]:
+            for sig in [signal.SIGINT, signal.SIGTERM, signal.SIGUSR1]:
                 signal.signal(sig, self.signal_handler)
 
         # starting to look like a good idea
@@ -69,6 +69,11 @@ class MpWorker(object):
                 sys.exit(0)
                 return
 
+            elif dest == "reload":
+                self.logw("mpw.asrv reloading")
+                self.asrv.reload()
+                self.logw("mpw.asrv reloaded")
+
             elif dest == "listen":
                 self.httpsrv.listen(args[0], args[1])
 

copyparty/broker_thr.py

@@ -21,10 +21,13 @@ class BrokerThr(object):
 
         # instantiate all services here (TODO: inheritance?)
         self.httpsrv = HttpSrv(self, None)
+        self.reload = self.noop
 
     def shutdown(self):
         # self.log("broker", "shutting down")
         self.httpsrv.shutdown()
+
+    def noop(self):
         pass
 
     def put(self, want_retval, dest, *args):

copyparty/httpcli.py

@@ -274,6 +274,15 @@ class HttpCli(object):
             except Pebkac:
                 return False
 
+    def permit_caching(self):
+        cache = self.uparam.get("cache")
+        if cache is None:
+            self.out_headers.update(NO_CACHE)
+            return
+
+        n = "604800" if cache == "i" else cache or "69"
+        self.out_headers["Cache-Control"] = "max-age=" + n
+
     def send_headers(self, length, status=200, mime=None, headers=None):
         response = ["{} {} {}".format(self.http_ver, status, HTTPCODE[status])]
 
@@ -289,7 +298,7 @@ class HttpCli(object):
 
         # default to utf8 html if no content-type is set
         if not mime:
-            mime = self.out_headers.get("Content-Type", "text/html; charset=UTF-8")
+            mime = self.out_headers.get("Content-Type", "text/html; charset=utf-8")
 
         self.out_headers["Content-Type"] = mime
 
@@ -410,6 +419,9 @@ class HttpCli(object):
             return self.scanvol()
 
         if not self.vpath:
+            if "reload" in self.uparam:
+                return self.handle_reload()
+
             if "stack" in self.uparam:
                 return self.tx_stack()
 
@@ -1449,17 +1461,18 @@ class HttpCli(object):
 
         if is_compressed:
             self.out_headers["Cache-Control"] = "max-age=573"
-        elif "cache" in self.uparam:
-            self.out_headers["Cache-Control"] = "max-age=69"
         else:
-            self.out_headers.update(NO_CACHE)
+            self.permit_caching()
+
+        if "txt" in self.uparam:
+            mime = "text/plain; charset={}".format(self.uparam["txt"] or "utf-8")
+        elif "mime" in self.uparam:
+            mime = self.uparam.get("mime")
+        else:
+            mime = guess_mime(req_path)
 
         self.out_headers["Accept-Ranges"] = "bytes"
-        self.send_headers(
+        self.send_headers(length=upper - lower, status=status, mime=mime)
-            length=upper - lower,
-            status=status,
-            mime=guess_mime(req_path),
-        )
 
         logmsg += unicode(status) + logtail
 
@@ -1548,6 +1561,7 @@ class HttpCli(object):
         return True
 
     def tx_ico(self, ext, exact=False):
+        self.permit_caching()
         if ext.endswith("/"):
             ext = "folder"
             exact = True
@@ -1703,7 +1717,7 @@ class HttpCli(object):
 
         vn, _ = self.asrv.vfs.get(self.vpath, self.uname, True, True)
 
-        args = [self.asrv.vfs.all_vols, [vn.vpath]]
+        args = [self.asrv.vfs.all_vols, [vn.vpath], False]
 
         x = self.conn.hsrv.broker.put(True, "up2k.rescan", *args)
         x = x.get()
@@ -1713,6 +1727,20 @@ class HttpCli(object):
 
         raise Pebkac(500, x)
 
+    def handle_reload(self):
+        act = self.uparam.get("reload")
+        if act != "cfg":
+            raise Pebkac(400, "only config files ('cfg') can be reloaded rn")
+
+        if not [x for x in self.wvol if x in self.rvol]:
+            raise Pebkac(403, "not allowed for user " + self.uname)
+
+        if self.args.no_reload:
+            raise Pebkac(403, "the reload feature is disabled in server config")
+
+        x = self.conn.hsrv.broker.put(True, "reload")
+        return self.redirect("", "?h", x.get(), "return to", False)
+
     def tx_stack(self):
         if not [x for x in self.wvol if x in self.rvol]:
             raise Pebkac(403, "not allowed for user " + self.uname)
@@ -1848,6 +1876,64 @@ class HttpCli(object):
         )
         self.loud_reply(x.get())
 
+    def tx_ls(self, ls):
+        dirs = ls["dirs"]
+        files = ls["files"]
+        arg = self.uparam["ls"]
+        if arg in ["v", "t", "txt"]:
+            try:
+                biggest = max(ls["files"] + ls["dirs"], key=itemgetter("sz"))["sz"]
+            except:
+                biggest = 0
+
+            if arg == "v":
+                fmt = "\033[0;7;36m{{}} {{:>{}}}\033[0m {{}}"
+                nfmt = "{}"
+                biggest = 0
+                f2 = "".join(
+                    "{}{{}}".format(x)
+                    for x in [
+                        "\033[7m",
+                        "\033[27m",
+                        "",
+                        "\033[0;1m",
+                        "\033[0;36m",
+                        "\033[0m",
+                    ]
+                )
+                ctab = {"B": 6, "K": 5, "M": 1, "G": 3}
+                for lst in [dirs, files]:
+                    for x in lst:
+                        a = x["dt"].replace("-", " ").replace(":", " ").split(" ")
+                        x["dt"] = f2.format(*list(a))
+                        sz = humansize(x["sz"], True)
+                        x["sz"] = "\033[0;3{}m{:>5}".format(ctab.get(sz[-1:], 0), sz)
+            else:
+                fmt = "{{}}  {{:{},}}  {{}}"
+                nfmt = "{:,}"
+
+            fmt = fmt.format(len(nfmt.format(biggest)))
+            ret = [
+                "# {}: {}".format(x, ls[x])
+                for x in ["acct", "perms", "srvinf"]
+                if x in ls
+            ]
+            ret += [
+                fmt.format(x["dt"], x["sz"], x["name"])
+                for y in [dirs, files]
+                for x in y
+            ]
+            ret = "\n".join(ret)
+            mime = "text/plain; charset=utf-8"
+        else:
+            [x.pop(k) for k in ["name", "dt"] for y in [dirs, files] for x in y]
+
+            ret = json.dumps(ls)
+            mime = "application/json"
+
+        self.reply(ret.encode("utf-8", "replace") + b"\n", mime=mime)
+        return True
+
     def tx_browser(self):
         vpath = ""
         vpnodes = [["", "/"]]
@@ -2006,6 +2092,7 @@ class HttpCli(object):
             "def_hcols": [],
             "have_up2k_idx": ("e2d" in vn.flags),
             "have_tags_idx": ("e2t" in vn.flags),
+            "have_acode": (not self.args.no_acode),
             "have_mv": (not self.args.no_mv),
             "have_del": (not self.args.no_del),
             "have_zip": (not self.args.no_zip),
@@ -2019,9 +2106,7 @@ class HttpCli(object):
         }
         if not self.can_read:
             if is_ls:
-                ret = json.dumps(ls_ret)
+                return self.tx_ls(ls_ret)
-                self.reply(ret.encode("utf-8", "replace"), mime="application/json")
-                return True
 
             if not stat.S_ISDIR(st.st_mode):
                 return self.tx_404(True)
@@ -2118,6 +2203,8 @@ class HttpCli(object):
 
             try:
                 ext = "---" if is_dir else fn.rsplit(".", 1)[1]
+                if len(ext) > 16:
+                    ext = ext[:16]
             except:
                 ext = "%"
 
@@ -2196,13 +2283,26 @@ class HttpCli(object):
                 f["tags"] = {}
 
         if is_ls:
-            [x.pop(k) for k in ["name", "dt"] for y in [dirs, files] for x in y]
             ls_ret["dirs"] = dirs
             ls_ret["files"] = files
             ls_ret["taglist"] = taglist
-            ret = json.dumps(ls_ret)
+            return self.tx_ls(ls_ret)
-            self.reply(ret.encode("utf-8", "replace"), mime="application/json")
+
-            return True
+        doc = self.uparam.get("doc") if self.can_read else None
+        if doc:
+            doc = unquotep(doc.replace("+", " "))
+            j2a["docname"] = doc
+            if next((x for x in files if x["name"] == doc), None):
+                with open(os.path.join(abspath, doc), "rb") as f:
+                    doc = f.read().decode("utf-8", "replace")
+            else:
+                self.log("doc 404: [{}]".format(doc), c=6)
+                doc = "( textfile not found )"
+
+            j2a["doc"] = doc
+
+        if not self.conn.hsrv.prism:
+            j2a["no_prism"] = True
 
         for d in dirs:
             d["name"] += "/"
@@ -2212,10 +2312,14 @@ class HttpCli(object):
         j2a["files"] = dirs + files
         j2a["logues"] = logues
         j2a["taglist"] = taglist
+        j2a["txt_ext"] = self.args.textfiles.replace(",", " ")
 
         if "mth" in vn.flags:
             j2a["def_hcols"] = vn.flags["mth"].split(",")
 
+        if self.args.js_browser:
+            j2a["js"] = self.args.js_browser
+
         if self.args.css_browser:
             j2a["css"] = self.args.css_browser
 

copyparty/httpconn.py

@@ -39,7 +39,7 @@ class HttpConn(object):
         self.u2fh = hsrv.u2fh
 
         enth = HAVE_PIL and not self.args.no_thumb
-        self.thumbcli = ThumbCli(hsrv.broker) if enth else None
+        self.thumbcli = ThumbCli(hsrv) if enth else None
         self.ico = Ico(self.args)
 
         self.t0 = time.time()

copyparty/httpsrv.py

@@ -50,10 +50,9 @@ class HttpSrv(object):
         self.log = broker.log
         self.asrv = broker.asrv
 
-        nsuf = "-{}".format(nid) if nid else ""
+        nsuf = "-n{}-i{:x}".format(nid, os.getpid()) if nid else ""
-        nsuf2 = "-n{}-i{:x}".format(nid, os.getpid()) if nid else ""
 
-        self.name = "hsrv" + nsuf2
+        self.name = "hsrv" + nsuf
         self.mutex = threading.Lock()
         self.stopping = False
 
@@ -61,6 +60,7 @@ class HttpSrv(object):
         self.tp_ncli = 0  # fading
         self.tp_time = None  # latest worker collect
         self.tp_q = None if self.args.no_htp else queue.LifoQueue()
+        self.t_periodic = None
 
         self.u2fh = FHC()
         self.srvs = []
@@ -76,6 +76,7 @@ class HttpSrv(object):
             x: env.get_template(x + ".html")
             for x in ["splash", "browser", "browser2", "msg", "md", "mde"]
         }
+        self.prism = os.path.exists(os.path.join(E.mod, "web", "deps", "prism.js.gz"))
 
         cert_path = os.path.join(E.cfg, "cert.pem")
         if bos.path.exists(cert_path):
@@ -93,10 +94,6 @@ class HttpSrv(object):
             if self.args.log_thrs:
                 start_log_thrs(self.log, self.args.log_thrs, nid)
 
-        t = threading.Thread(target=self.periodic, name="hsrv-pt" + nsuf)
-        t.daemon = True
-        t.start()
-
     def start_threads(self, n):
         self.tp_nthr += n
         if self.args.log_htp:
@@ -120,7 +117,7 @@ class HttpSrv(object):
 
     def periodic(self):
         while True:
-            time.sleep(2 if self.tp_ncli else 10)
+            time.sleep(2 if self.tp_ncli or self.ncli else 10)
             with self.mutex:
                 self.u2fh.clean()
                 if self.tp_q:
@@ -128,6 +125,10 @@ class HttpSrv(object):
                     if self.tp_nthr > self.tp_ncli + 8:
                         self.stop_threads(4)
 
+                if not self.ncli and not self.u2fh.cache and self.tp_nthr <= 8:
+                    self.t_periodic = None
+                    return
+
     def listen(self, sck, nlisteners):
         ip, port = sck.getsockname()
         self.srvs.append(sck)
@@ -146,7 +147,12 @@ class HttpSrv(object):
         fno = srv_sck.fileno()
         msg = "subscribed @ {}:{}  f{}".format(ip, port, fno)
         self.log(self.name, msg)
+
+        def fun():
             self.broker.put(False, "cb_httpsrv_up")
+
+        threading.Thread(target=fun).start()
+
         while not self.stopping:
             if self.args.log_conn:
                 self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="1;30")
@@ -186,6 +192,16 @@ class HttpSrv(object):
 
         with self.mutex:
             self.ncli += 1
+            if not self.t_periodic:
+                name = "hsrv-pt"
+                if self.nid:
+                    name += "-{}".format(self.nid)
+
+                t = threading.Thread(target=self.periodic, name=name)
+                self.t_periodic = t
+                t.daemon = True
+                t.start()
+
             if self.tp_q:
                 self.tp_time = self.tp_time or now
                 self.tp_ncli = max(self.tp_ncli, self.ncli)

copyparty/mtag.py

@@ -413,6 +413,9 @@ class MTag(object):
         return r1
 
     def get_mutagen(self, abspath):
+        if not bos.path.isfile(abspath):
+            return {}
+
         import mutagen
 
         try:
@@ -458,10 +461,16 @@ class MTag(object):
         return self.normalize_tags(ret, md)
 
     def get_ffprobe(self, abspath):
+        if not bos.path.isfile(abspath):
+            return {}
+
         ret, md = ffprobe(abspath)
         return self.normalize_tags(ret, md)
 
     def get_bin(self, parsers, abspath):
+        if not bos.path.isfile(abspath):
+            return {}
+
         pypath = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
         pypath = [str(pypath)] + [str(x) for x in sys.path if x]
         pypath = str(os.pathsep.join(pypath))

copyparty/svchub.py

@@ -18,6 +18,7 @@ from .authsrv import AuthSrv
 from .tcpsrv import TcpSrv
 from .up2k import Up2k
 from .th_srv import ThumbSrv, HAVE_PIL, HAVE_WEBP
+from .mtag import HAVE_FFMPEG, HAVE_FFPROBE
 
 
 class SvcHub(object):
@@ -36,7 +37,9 @@ class SvcHub(object):
         self.argv = argv
         self.logf = None
         self.stop_req = False
+        self.reload_req = False
         self.stopping = False
+        self.reloading = False
         self.stop_cond = threading.Condition()
         self.retcode = 0
         self.httpsrv_up = 0
@@ -88,11 +91,22 @@ class SvcHub(object):
                     "thumb", msg.format(" " * 37, os.path.basename(sys.executable)), c=3
                 )
 
+        if not args.no_acode and args.no_thumb:
+            msg = "setting --no-acode because --no-thumb (sorry)"
+            self.log("thumb", msg, c=6)
+            args.no_acode = True
+
+        if not args.no_acode and (not HAVE_FFMPEG or not HAVE_FFPROBE):
+            msg = "setting --no-acode because either FFmpeg or FFprobe is not available"
+            self.log("thumb", msg, c=6)
+            args.no_acode = True
+
+        args.th_poke = min(args.th_poke, args.th_maxage, args.ac_maxage)
+
         # decide which worker impl to use
         if self.check_mp_enable():
             from .broker_mp import BrokerMp as Broker
         else:
-            self.log("root", "cannot efficiently use multiple CPU cores")
             from .broker_thr import BrokerThr as Broker
 
         self.broker = Broker(self)
@@ -104,16 +118,16 @@ class SvcHub(object):
         if not failed:
             return
 
-        m = "{}/{} workers failed to start"
-        m = m.format(failed, expected)
-        self.log("root", m, 1)
-
         if self.args.ign_ebind_all:
             return
 
         if self.args.ign_ebind and self.tcpsrv.srv:
             return
 
+        m = "{}/{} workers failed to start"
+        m = m.format(failed, expected)
+        self.log("root", m, 1)
+
         self.retcode = 1
         os.kill(os.getpid(), signal.SIGTERM)
 
@@ -122,6 +136,7 @@ class SvcHub(object):
         if self.httpsrv_up != self.broker.num_workers:
             return
 
+        time.sleep(0.1)  # purely cosmetic dw
         self.log("root", "workers OK\n")
         self.up2k.init_vols()
 
@@ -182,7 +197,11 @@ class SvcHub(object):
         thr.daemon = True
         thr.start()
 
-        for sig in [signal.SIGINT, signal.SIGTERM]:
+        sigs = [signal.SIGINT, signal.SIGTERM]
+        if not ANYWIN:
+            sigs.append(signal.SIGUSR1)
+
+        for sig in sigs:
             signal.signal(sig, self.signal_handler)
 
         # macos hangs after shutdown on sigterm with while-sleep,
@@ -206,18 +225,45 @@ class SvcHub(object):
         else:
             self.stop_thr()
 
+    def reload(self):
+        if self.reloading:
+            return "cannot reload; already in progress"
+
+        self.reloading = True
+        t = threading.Thread(target=self._reload)
+        t.daemon = True
+        t.start()
+        return "reload initiated"
+
+    def _reload(self):
+        self.log("root", "reload scheduled")
+        with self.up2k.mutex:
+            self.asrv.reload()
+            self.up2k.reload()
+            self.broker.reload()
+
+        self.reloading = False
+
     def stop_thr(self):
         while not self.stop_req:
             with self.stop_cond:
                 self.stop_cond.wait(9001)
 
+            if self.reload_req:
+                self.reload_req = False
+                self.reload()
+
         self.shutdown()
 
     def signal_handler(self, sig, frame):
         if self.stopping:
             return
 
+        if sig == signal.SIGUSR1:
+            self.reload_req = True
+        else:
             self.stop_req = True
+
         with self.stop_cond:
             self.stop_cond.notify_all()
 
@@ -349,10 +395,11 @@ class SvcHub(object):
 
     def check_mp_enable(self):
         if self.args.j == 1:
-            self.log("root", "multiprocessing disabled by argument -j 1;")
+            self.log("svchub", "multiprocessing disabled by argument -j 1")
             return False
 
         if mp.cpu_count() <= 1:
+            self.log("svchub", "only one CPU detected; multiprocessing disabled")
             return False
 
         try:
@@ -367,6 +414,7 @@ class SvcHub(object):
             return True
         else:
             self.log("svchub", err)
+            self.log("svchub", "cannot efficiently use multiple CPU cores")
             return False
 
     def sd_notify(self):

copyparty/tcpsrv.py

@@ -21,6 +21,29 @@ class TcpSrv(object):
 
         self.stopping = False
 
+        self.srv = []
+        self.nsrv = 0
+        ok = {}
+        for ip in self.args.i:
+            ok[ip] = []
+            for port in self.args.p:
+                self.nsrv += 1
+                try:
+                    self._listen(ip, port)
+                    ok[ip].append(port)
+                except Exception as ex:
+                    if self.args.ign_ebind or self.args.ign_ebind_all:
+                        m = "could not listen on {}:{}: {}"
+                        self.log("tcpsrv", m.format(ip, port, ex), c=3)
+                    else:
+                        raise
+
+        if not self.srv and not self.args.ign_ebind_all:
+            raise Exception("could not listen on any of the given interfaces")
+
+        if self.nsrv != len(self.srv):
+            self.log("tcpsrv", "")
+
         ip = "127.0.0.1"
         eps = {ip: "local only"}
         nonlocals = [x for x in self.args.i if x != ip]
@@ -34,6 +57,9 @@ class TcpSrv(object):
         m = "available @ http://{}:{}/  (\033[33m{}\033[0m)"
         for ip, desc in sorted(eps.items(), key=lambda x: x[1]):
             for port in sorted(self.args.p):
+                if port not in ok.get(ip, ok.get("0.0.0.0", [])):
+                    continue
+
                 msgs.append(m.format(ip, port, desc))
 
         if msgs:
@@ -41,23 +67,6 @@ class TcpSrv(object):
             for m in msgs:
                 self.log("tcpsrv", m)
 
-        self.srv = []
-        self.nsrv = 0
-        for ip in self.args.i:
-            for port in self.args.p:
-                self.nsrv += 1
-                try:
-                    self._listen(ip, port)
-                except Exception as ex:
-                    if self.args.ign_ebind or self.args.ign_ebind_all:
-                        m = "could not listen on {}:{}: {}"
-                        self.log("tcpsrv", m.format(ip, port, ex), c=1)
-                    else:
-                        raise
-
-        if not self.srv and not self.args.ign_ebind_all:
-            raise Exception("could not listen on any of the given interfaces")
-
     def _listen(self, ip, port):
         srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

copyparty/th_cli.py

@@ -4,28 +4,44 @@ from __future__ import print_function, unicode_literals
 import os
 
 from .util import Cooldown
-from .th_srv import thumb_path, THUMBABLE, FMT_FF
+from .th_srv import thumb_path, THUMBABLE, FMT_FFV, FMT_FFA
 from .bos import bos
 
 
 class ThumbCli(object):
-    def __init__(self, broker):
+    def __init__(self, hsrv):
-        self.broker = broker
+        self.broker = hsrv.broker
-        self.args = broker.args
+        self.log_func = hsrv.log
-        self.asrv = broker.asrv
+        self.args = hsrv.args
+        self.asrv = hsrv.asrv
 
         # cache on both sides for less broker spam
         self.cooldown = Cooldown(self.args.th_poke)
 
+    def log(self, msg, c=0):
+        self.log_func("thumbcli", msg, c)
+
     def get(self, ptop, rem, mtime, fmt):
         ext = rem.rsplit(".")[-1].lower()
         if ext not in THUMBABLE:
             return None
 
-        is_vid = ext in FMT_FF
+        is_vid = ext in FMT_FFV
         if is_vid and self.args.no_vthumb:
             return None
 
+        want_opus = fmt == "opus"
+        is_au = ext in FMT_FFA
+        if is_au:
+            if want_opus:
+                if self.args.no_acode:
+                    return None
+            else:
+                if self.args.no_athumb:
+                    return None
+        elif want_opus:
+            return None
+
         if rem.startswith(".hist/th/") and rem.split(".")[-1] in ["webp", "jpg"]:
             return os.path.join(ptop, rem)
 
@@ -33,10 +49,14 @@ class ThumbCli(object):
             fmt = "w"
 
         if fmt == "w":
-            if self.args.th_no_webp or (is_vid and self.args.th_ff_jpg):
+            if self.args.th_no_webp or ((is_vid or is_au) and self.args.th_ff_jpg):
                 fmt = "j"
 
-        histpath = self.asrv.vfs.histtab[ptop]
+        histpath = self.asrv.vfs.histtab.get(ptop)
+        if not histpath:
+            self.log("no histpath for [{}]".format(ptop))
+            return None
+
         tpath = thumb_path(histpath, rem, mtime, fmt)
         ret = None
         try:
@@ -53,6 +73,11 @@ class ThumbCli(object):
             if self.cooldown.poke(tdir):
                 self.broker.put(False, "thumbsrv.poke", tdir)
 
+            if want_opus:
+                # audio files expire individually
+                if self.cooldown.poke(tpath):
+                    self.broker.put(False, "thumbsrv.poke", tpath)
+
             return ret
 
         x = self.broker.put(True, "thumbsrv.get", ptop, rem, mtime, fmt)

copyparty/th_srv.py

@@ -10,7 +10,7 @@ import threading
 import subprocess as sp
 
 from .__init__ import PY2, unicode
-from .util import fsenc, vsplit, runcmd, Queue, Cooldown, BytesIO, min_ex
+from .util import fsenc, vsplit, statdir, runcmd, Queue, Cooldown, BytesIO, min_ex
 from .bos import bos
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe
 
@@ -50,7 +50,8 @@ except:
 # https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
 # ffmpeg -formats
 FMT_PIL = "bmp dib gif icns ico jpg jpeg jp2 jpx pcx png pbm pgm ppm pnm sgi tga tif tiff webp xbm dds xpm"
-FMT_FF = "av1 asf avi flv m4v mkv mjpeg mjpg mpg mpeg mpg2 mpeg2 h264 avc mts h265 hevc mov 3gp mp4 ts mpegts nut ogv ogm rm vob webm wmv"
+FMT_FFV = "av1 asf avi flv m4v mkv mjpeg mjpg mpg mpeg mpg2 mpeg2 h264 avc mts h265 hevc mov 3gp mp4 ts mpegts nut ogv ogm rm vob webm wmv"
+FMT_FFA = "aac m4a ogg opus flac alac mp3 mp2 ac3 dts wma ra wav aif aiff au alaw ulaw mulaw amr gsm ape tak tta wv"
 
 if HAVE_HEIF:
     FMT_PIL += " heif heifs heic heics"
@@ -58,7 +59,9 @@ if HAVE_HEIF:
 if HAVE_AVIF:
     FMT_PIL += " avif avifs"
 
-FMT_PIL, FMT_FF = [{x: True for x in y.split(" ") if x} for y in [FMT_PIL, FMT_FF]]
+FMT_PIL, FMT_FFV, FMT_FFA = [
+    {x: True for x in y.split(" ") if x} for y in [FMT_PIL, FMT_FFV, FMT_FFA]
+]
 
 
 THUMBABLE = {}
@@ -67,7 +70,8 @@ if HAVE_PIL:
     THUMBABLE.update(FMT_PIL)
 
 if HAVE_FFMPEG and HAVE_FFPROBE:
-    THUMBABLE.update(FMT_FF)
+    THUMBABLE.update(FMT_FFV)
+    THUMBABLE.update(FMT_FFA)
 
 
 def thumb_path(histpath, rem, mtime, fmt):
@@ -86,9 +90,13 @@ def thumb_path(histpath, rem, mtime, fmt):
     h = hashlib.sha512(fsenc(fn)).digest()
     fn = base64.urlsafe_b64encode(h).decode("ascii")[:24]
 
-    return "{}/th/{}/{}.{:x}.{}".format(
+    if fmt == "opus":
-        histpath, rd, fn, int(mtime), "webp" if fmt == "w" else "jpg"
+        cat = "ac"
-    )
+    else:
+        fmt = "webp" if fmt == "w" else "jpg"
+        cat = "th"
+
+    return "{}/{}/{}/{}.{:x}.{}".format(histpath, cat, rd, fn, int(mtime), fmt)
 
 
 class ThumbSrv(object):
@@ -105,9 +113,7 @@ class ThumbSrv(object):
         self.mutex = threading.Lock()
         self.busy = {}
         self.stopping = False
-        self.nthr = self.args.th_mt
+        self.nthr = max(1, self.args.th_mt)
-        if not self.nthr:
-            self.nthr = os.cpu_count() if hasattr(os, "cpu_count") else 4
 
         self.q = Queue(self.nthr * 4)
         for n in range(self.nthr):
@@ -117,7 +123,8 @@ class ThumbSrv(object):
             t.daemon = True
             t.start()
 
-        if not self.args.no_vthumb and (not HAVE_FFMPEG or not HAVE_FFPROBE):
+        want_ff = not self.args.no_vthumb or not self.args.no_athumb
+        if want_ff and (not HAVE_FFMPEG or not HAVE_FFPROBE):
             missing = []
             if not HAVE_FFMPEG:
                 missing.append("FFmpeg")
@@ -125,12 +132,12 @@ class ThumbSrv(object):
             if not HAVE_FFPROBE:
                 missing.append("FFprobe")
 
-            msg = "cannot create video thumbnails because some of the required programs are not available: "
+            msg = "cannot create audio/video thumbnails because some of the required programs are not available: "
             msg += ", ".join(missing)
             self.log(msg, c=3)
 
         if self.args.th_clean:
-            t = threading.Thread(target=self.cleaner, name="thumb-cleaner")
+            t = threading.Thread(target=self.cleaner, name="thumb.cln")
             t.daemon = True
             t.start()
 
@@ -147,7 +154,11 @@ class ThumbSrv(object):
             return not self.nthr
 
     def get(self, ptop, rem, mtime, fmt):
-        histpath = self.asrv.vfs.histtab[ptop]
+        histpath = self.asrv.vfs.histtab.get(ptop)
+        if not histpath:
+            self.log("no histpath for [{}]".format(ptop))
+            return None
+
         tpath = thumb_path(histpath, rem, mtime, fmt)
         abspath = os.path.join(ptop, rem)
         cond = threading.Condition(self.mutex)
@@ -183,6 +194,7 @@ class ThumbSrv(object):
         try:
             st = bos.stat(tpath)
             if st.st_size:
+                self.poke(tpath)
                 return tpath
         except:
             pass
@@ -201,8 +213,13 @@ class ThumbSrv(object):
             if not bos.path.exists(tpath):
                 if ext in FMT_PIL:
                     fun = self.conv_pil
-                elif ext in FMT_FF:
+                elif ext in FMT_FFV:
                     fun = self.conv_ffmpeg
+                elif ext in FMT_FFA:
+                    if tpath.endswith(".opus"):
+                        fun = self.conv_opus
+                    else:
+                        fun = self.conv_spec
 
             if fun:
                 try:
@@ -328,8 +345,10 @@ class ThumbSrv(object):
             ]
 
         cmd += [fsenc(tpath)]
-        # self.log((b" ".join(cmd)).decode("utf-8"))
+        self._run_ff(cmd)
 
+    def _run_ff(self, cmd):
+        # self.log((b" ".join(cmd)).decode("utf-8"))
         ret, sout, serr = runcmd(cmd)
         if ret != 0:
             m = "FFmpeg failed (probably a corrupt video file):\n"
@@ -337,16 +356,75 @@ class ThumbSrv(object):
             self.log(m, c="1;30")
             raise sp.CalledProcessError(ret, (cmd[0], b"...", cmd[-1]))
 
+    def conv_spec(self, abspath, tpath):
+        ret, _ = ffprobe(abspath)
+        if "ac" not in ret:
+            raise Exception("not audio")
+
+        fc = "[0:a:0]aresample=48000{},showspectrumpic=s=640x512,crop=780:544:70:50[o]"
+        fc = fc.format("" if self.args.th_ff_swr else ":resampler=soxr")
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-filter_complex", fc.encode("utf-8"),
+            b"-map", b"[o]"
+        ]
+        # fmt: on
+
+        if tpath.endswith(".jpg"):
+            cmd += [
+                b"-q:v",
+                b"6",  # default=??
+            ]
+        else:
+            cmd += [
+                b"-q:v",
+                b"50",  # default=75
+                b"-compression_level:v",
+                b"6",  # default=4, 0=fast, 6=max
+            ]
+
+        cmd += [fsenc(tpath)]
+        self._run_ff(cmd)
+
+    def conv_opus(self, abspath, tpath):
+        if self.args.no_acode:
+            raise Exception("disabled in server config")
+
+        ret, _ = ffprobe(abspath)
+        if "ac" not in ret:
+            raise Exception("not audio")
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-map", b"0:a:0",
+            b"-c:a", b"libopus",
+            b"-b:a", b"128k",
+            fsenc(tpath)
+        ]
+        # fmt: on
+
+        self._run_ff(cmd)
+
     def poke(self, tdir):
         if not self.poke_cd.poke(tdir):
             return
 
         ts = int(time.time())
         try:
-            p1 = os.path.dirname(tdir)
+            for _ in range(4):
-            p2 = os.path.dirname(p1)
+                bos.utime(tdir, (ts, ts))
-            for dp in [tdir, p1, p2]:
+                tdir = os.path.dirname(tdir)
-                bos.utime(dp, (ts, ts))
         except:
             pass
 
@@ -366,25 +444,36 @@ class ThumbSrv(object):
             self.log("\033[Jcln ok; rm {} dirs".format(ndirs))
 
     def clean(self, histpath):
-        thumbpath = os.path.join(histpath, "th")
+        ret = 0
+        for cat in ["th", "ac"]:
+            ret += self._clean(histpath, cat, None)
+
+        return ret
+
+    def _clean(self, histpath, cat, thumbpath):
+        if not thumbpath:
+            thumbpath = os.path.join(histpath, cat)
+
         # self.log("cln {}".format(thumbpath))
-        maxage = self.args.th_maxage
+        exts = ["jpg", "webp"] if cat == "th" else ["opus"]
+        maxage = getattr(self.args, cat + "_maxage")
         now = time.time()
         prev_b64 = None
         prev_fp = None
         try:
-            ents = bos.listdir(thumbpath)
+            ents = statdir(self.log, not self.args.no_scandir, False, thumbpath)
+            ents = sorted(list(ents))
         except:
             return 0
 
         ndirs = 0
-        for f in sorted(ents):
+        for f, inf in ents:
             fp = os.path.join(thumbpath, f)
             cmp = fp.lower().replace("\\", "/")
 
             # "top" or b64 prefix/full (a folder)
             if len(f) <= 3 or len(f) == 24:
-                age = now - bos.path.getmtime(fp)
+                age = now - inf.st_mtime
                 if age > maxage:
                     with self.mutex:
                         safe = True
@@ -398,16 +487,15 @@ class ThumbSrv(object):
                             self.log("rm -rf [{}]".format(fp))
                             shutil.rmtree(fp, ignore_errors=True)
                 else:
-                    ndirs += self.clean(fp)
+                    self._clean(histpath, cat, fp)
+
                 continue
 
             # thumb file
             try:
                 b64, ts, ext = f.split(".")
-                if len(b64) != 24 or len(ts) != 8 or ext not in ["jpg", "webp"]:
+                if len(b64) != 24 or len(ts) != 8 or ext not in exts:
                     raise Exception()
-
-                ts = int(ts, 16)
             except:
                 if f != "dir.txt":
                     self.log("foreign file in thumbs dir: [{}]".format(fp), 1)
@@ -418,6 +506,10 @@ class ThumbSrv(object):
                 self.log("rm replaced [{}]".format(fp))
                 bos.unlink(prev_fp)
 
+            if cat != "th" and inf.st_mtime + maxage < now:
+                self.log("rm expired [{}]".format(fp))
+                bos.unlink(fp)
+
             prev_b64 = b64
             prev_fp = fp
 

copyparty/u2idx.py

@@ -67,7 +67,11 @@ class U2idx(object):
         if cur:
             return cur
 
-        histpath = self.asrv.vfs.histtab[ptop]
+        histpath = self.asrv.vfs.histtab.get(ptop)
+        if not histpath:
+            self.log("no histpath for [{}]".format(ptop))
+            return None
+
         db_path = os.path.join(histpath, "up2k.db")
         if not bos.path.exists(db_path):
             return None

copyparty/up2k.py

@@ -63,10 +63,12 @@ class Up2k(object):
 
         # state
         self.mutex = threading.Lock()
+        self.rescan_cond = threading.Condition()
         self.hashq = Queue()
         self.tagq = Queue()
         self.n_hashq = 0
         self.n_tagq = 0
+        self.gid = 0
         self.volstate = {}
         self.need_rescan = {}
         self.dupesched = {}
@@ -113,6 +115,12 @@ class Up2k(object):
         t.daemon = True
         t.start()
 
+    def reload(self):
+        self.gid += 1
+        self.log("reload #{} initiated".format(self.gid))
+        all_vols = self.asrv.vfs.all_vols
+        self.rescan(all_vols, list(all_vols.keys()), True)
+
     def deferred_init(self):
         all_vols = self.asrv.vfs.all_vols
         have_e2d = self.init_indexes(all_vols)
@@ -131,7 +139,9 @@ class Up2k(object):
             thr.start()
 
             if self.mtag:
-                thr = threading.Thread(target=self._tagger, name="up2k-tagger")
+                for n in range(max(1, self.args.mtag_mt)):
+                    name = "tagger-{}".format(n)
+                    thr = threading.Thread(target=self._tagger, name=name)
                     thr.daemon = True
                     thr.start()
 
@@ -165,15 +175,15 @@ class Up2k(object):
         }
         return json.dumps(ret, indent=4)
 
-    def rescan(self, all_vols, scan_vols):
+    def rescan(self, all_vols, scan_vols, wait):
-        if hasattr(self, "pp"):
+        if not wait and hasattr(self, "pp"):
             return "cannot initiate; scan is already in progress"
 
         args = (all_vols, scan_vols)
         t = threading.Thread(
             target=self.init_indexes,
             args=args,
-            name="up2k-rescan-{}".format(scan_vols[0]),
+            name="up2k-rescan-{}".format(scan_vols[0] if scan_vols else "all"),
         )
         t.daemon = True
         t.start()
@@ -181,9 +191,23 @@ class Up2k(object):
 
     def _sched_rescan(self):
         volage = {}
+        cooldown = 0
+        timeout = time.time() + 3
         while True:
-            time.sleep(self.args.re_int)
+            timeout = max(timeout, cooldown)
+            wait = max(0.1, timeout + 0.1 - time.time())
+            with self.rescan_cond:
+                self.rescan_cond.wait(wait)
+
             now = time.time()
+            if now < cooldown:
+                continue
+
+            if hasattr(self, "pp"):
+                cooldown = now + 5
+                continue
+
+            timeout = now + 9001
             with self.mutex:
                 for vp, vol in sorted(self.asrv.vfs.all_vols.items()):
                     maxage = vol.flags.get("scan")
@@ -193,14 +217,18 @@ class Up2k(object):
                     if vp not in volage:
                         volage[vp] = now
 
-                    if now - volage[vp] >= maxage:
+                    deadline = volage[vp] + maxage
+                    if deadline <= now:
                         self.need_rescan[vp] = 1
 
+                    timeout = min(timeout, deadline)
+
                 vols = list(sorted(self.need_rescan.keys()))
                 self.need_rescan = {}
 
             if vols:
-                err = self.rescan(self.asrv.vfs.all_vols, vols)
+                cooldown = now + 10
+                err = self.rescan(self.asrv.vfs.all_vols, vols, False)
                 if err:
                     for v in vols:
                         self.need_rescan[v] = True
@@ -222,8 +250,11 @@ class Up2k(object):
                 if not cur:
                     continue
 
+                lifetime = int(lifetime)
+                timeout = min(timeout, now + lifetime)
+
                 nrm = 0
-                deadline = time.time() - int(lifetime)
+                deadline = time.time() - lifetime
                 q = "select rd, fn from up where at > 0 and at < ? limit 100"
                 while True:
                     with self.mutex:
@@ -240,12 +271,22 @@ class Up2k(object):
                         if vp:
                             fvp = "{}/{}".format(vp, fvp)
 
-                        self._handle_rm(LEELOO_DALLAS, None, fvp, True)
+                        self._handle_rm(LEELOO_DALLAS, None, fvp)
                         nrm += 1
 
                 if nrm:
                     self.log("{} files graduated in {}".format(nrm, vp))
 
+                if timeout < 10:
+                    continue
+
+                q = "select at from up where at > 0 order by at limit 1"
+                with self.mutex:
+                    hits = cur.execute(q).fetchone()
+
+                if hits:
+                    timeout = min(timeout, now + lifetime - (now - hits[0]))
+
     def _vis_job_progress(self, job):
         perc = 100 - (len(job["need"]) * 100.0 / len(job["hash"]))
         path = os.path.join(job["ptop"], job["prel"], job["name"])
@@ -269,6 +310,16 @@ class Up2k(object):
         return True, ret
 
     def init_indexes(self, all_vols, scan_vols=None):
+        gid = self.gid
+        while hasattr(self, "pp") and gid == self.gid:
+            time.sleep(0.1)
+
+        if gid != self.gid:
+            return
+
+        if gid:
+            self.log("reload #{} running".format(self.gid))
+
         self.pp = ProgressPrinter()
         vols = all_vols.values()
         t0 = time.time()
@@ -399,7 +450,11 @@ class Up2k(object):
         return have_e2d
 
     def register_vpath(self, ptop, flags):
-        histpath = self.asrv.vfs.histtab[ptop]
+        histpath = self.asrv.vfs.histtab.get(ptop)
+        if not histpath:
+            self.log("no histpath for [{}]".format(ptop))
+            return None
+
         db_path = os.path.join(histpath, "up2k.db")
         if ptop in self.registry:
             try:
@@ -530,7 +585,7 @@ class Up2k(object):
             else:
                 # self.log("file: {}".format(abspath))
                 seen_files[iname] = 1
-                rp = abspath[len(top) + 1 :]
+                rp = abspath[len(top) :].lstrip("/")
                 if WINDOWS:
                     rp = rp.replace("\\", "/").strip("/")
 
@@ -700,7 +755,7 @@ class Up2k(object):
                 return n_add, n_rm, False
 
             mpool = False
-            if self.mtag.prefer_mt and not self.args.no_mtag_mt:
+            if self.mtag.prefer_mt and self.args.mtag_mt > 1:
                 mpool = self._start_mpool()
 
             conn = sqlite3.connect(db_path, timeout=15)
@@ -767,10 +822,11 @@ class Up2k(object):
         return ret
 
     def _run_all_mtp(self):
+        gid = self.gid
         t0 = time.time()
         for ptop, flags in self.flags.items():
             if "mtp" in flags:
-                self._run_one_mtp(ptop)
+                self._run_one_mtp(ptop, gid)
 
         td = time.time() - t0
         msg = "mtp finished in {:.2f} sec ({})"
@@ -781,7 +837,10 @@ class Up2k(object):
             if "OFFLINE" not in self.volstate[k]:
                 self.volstate[k] = "online, idle"
 
-    def _run_one_mtp(self, ptop):
+    def _run_one_mtp(self, ptop, gid):
+        if gid != self.gid:
+            return
+
         entags = self.entags[ptop]
 
         parsers = {}
@@ -814,6 +873,9 @@ class Up2k(object):
         in_progress = {}
         while True:
             with self.mutex:
+                if gid != self.gid:
+                    break
+
                 q = "select w from mt where k = 't:mtp' limit ?"
                 warks = cur.execute(q, (batch_sz,)).fetchall()
                 warks = [x[0] for x in warks]
@@ -933,9 +995,7 @@ class Up2k(object):
     def _start_mpool(self):
         # mp.pool.ThreadPool and concurrent.futures.ThreadPoolExecutor
         # both do crazy runahead so lets reinvent another wheel
-        nw = os.cpu_count() if hasattr(os, "cpu_count") else 4
+        nw = max(1, self.args.mtag_mt)
-        if self.args.no_mtag_mt:
-            nw = 1
 
         if self.pending_tags is None:
             self.log("using {}x {}".format(nw, self.mtag.backend))
@@ -998,7 +1058,10 @@ class Up2k(object):
             except Exception as ex:
                 msg = "failed to read tags from {}:\n{}"
                 self.log(msg.format(abspath, ex), c=3)
-                return
+                return 0
+
+        if not bos.path.isfile(abspath):
+            return 0
 
         if entags:
             tags = {k: v for k, v in tags.items() if k in entags}
@@ -1501,7 +1564,7 @@ class Up2k(object):
         ok = {}
         ng = {}
         for vp in vpaths:
-            a, b, c = self._handle_rm(uname, ip, vp, False)
+            a, b, c = self._handle_rm(uname, ip, vp)
             n_files += a
             for k in b:
                 ok[k] = 1
@@ -1514,7 +1577,7 @@ class Up2k(object):
 
         return "deleted {} files (and {}/{} folders)".format(n_files, ok, ok + ng)
 
-    def _handle_rm(self, uname, ip, vpath, rm_topdir):
+    def _handle_rm(self, uname, ip, vpath):
         try:
             permsets = [[True, False, False, True]]
             vn, rem = self.asrv.vfs.get(vpath, uname, *permsets[0])
@@ -1585,7 +1648,7 @@ class Up2k(object):
 
                 bos.unlink(abspath)
 
-        rm = rmdirs(self.log_func, scandir, True, atop, 1 if rm_topdir else 0)
+        rm = rmdirs(self.log_func, scandir, True, atop, 1)
         return n_files, rm[0], rm[1]
 
     def handle_mv(self, uname, svp, dvp):
@@ -1664,6 +1727,9 @@ class Up2k(object):
             # folders are too scary, schedule rescan of both vols
             self.need_rescan[svn.vpath] = 1
             self.need_rescan[dvn.vpath] = 1
+            with self.rescan_cond:
+                self.rescan_cond.notify_all()
+
             return "k"
 
         c1, w, ftime, fsize, ip, at = self._find_from_vpath(svn.realpath, srem)
@@ -1926,6 +1992,7 @@ class Up2k(object):
         self.snap_prev = {}
         while True:
             time.sleep(self.snap_persist_interval)
+            if not hasattr(self, "pp"):
                 self.do_snapshot()
 
     def do_snapshot(self):
@@ -1935,7 +2002,10 @@ class Up2k(object):
 
     def _snap_reg(self, ptop, reg):
         now = time.time()
-        histpath = self.asrv.vfs.histtab[ptop]
+        histpath = self.asrv.vfs.histtab.get(ptop)
+        if not histpath:
+            return
+
         rm = [x for x in reg.values() if now - x["poke"] > self.snap_discard_interval]
         if rm:
             m = "dropping {} abandoned uploads in {}".format(len(rm), ptop)

copyparty/util.py

@@ -100,10 +100,24 @@ IMPLICATIONS = [
 
 
 MIMES = {
-    "md": "text/plain; charset=UTF-8",
+    "md": "text/plain",
+    "txt": "text/plain",
+    "js": "text/javascript",
     "opus": "audio/ogg; codecs=opus",
-    "webp": "image/webp",
+    "mp3": "audio/mpeg",
+    "m4a": "audio/mp4",
+    "jpg": "image/jpeg",
 }
+for ln in """text css html csv
+application json wasm xml pdf rtf zip
+image webp jpeg png gif bmp
+audio aac ogg wav
+video webm mp4 mpeg
+font woff woff2 otf ttf
+""".splitlines():
+    k, vs = ln.split(" ", 1)
+    for v in vs.strip().split():
+        MIMES[v] = "{}/{}".format(k, v)
 
 
 REKOBO_KEY = {
@@ -445,7 +459,7 @@ def log_thrs(log, ival, name):
         tv = [x.name for x in threading.enumerate()]
         tv = [
             x.split("-")[0]
-            if x.startswith("httpconn-") or x.startswith("thumb-")
+            if x.split("-")[0] in ["httpconn", "thumb", "tagger"]
             else "listen"
             if "-listen-" in x
             else x
@@ -1224,6 +1238,7 @@ def statdir(logger, scandir, lstat, top):
 def rmdirs(logger, scandir, lstat, top, depth):
     if not os.path.exists(fsenc(top)) or not os.path.isdir(fsenc(top)):
         top = os.path.dirname(top)
+        depth -= 1
 
     dirs = statdir(logger, scandir, lstat, top)
     dirs = [x[0] for x in dirs if stat.S_ISDIR(x[1].st_mode)]
@@ -1279,11 +1294,18 @@ def guess_mime(url, fallback="application/octet-stream"):
     except:
         return fallback
 
-    ret = MIMES.get(ext) or mimetypes.guess_type(url)[0] or fallback
+    ret = MIMES.get(ext)
+
+    if not ret:
+        x = mimetypes.guess_type(url)
+        ret = "application/{}".format(x[1]) if x[1] else x[0]
+
+    if not ret:
+        ret = fallback
 
     if ";" not in ret:
         if ret.startswith("text/") or ret.endswith("/javascript"):
-            ret += "; charset=UTF-8"
+            ret += "; charset=utf-8"
 
     return ret
 

copyparty/web/baguettebox.js

@@ -331,7 +331,7 @@ window.baguetteBox = (function () {
 
     function tglsel() {
         var thumb = currentGallery[currentIndex].imageElement,
-            name = vsplit(thumb.href)[1],
+            name = vsplit(thumb.href)[1].split('?')[0],
             files = msel.getall();
 
         for (var a = 0; a < files.length; a++)
@@ -345,7 +345,7 @@ window.baguetteBox = (function () {
     function selbg() {
         var img = vidimg(),
             thumb = currentGallery[currentIndex].imageElement,
-            name = vsplit(thumb.href)[1],
+            name = vsplit(thumb.href)[1].split('?')[0],
             files = msel.getsel(),
             sel = false;
 
@@ -530,9 +530,7 @@ window.baguetteBox = (function () {
             if (options.bodyClass && document.body.classList)
                 document.body.classList.remove(options.bodyClass);
 
-            var h = ebi('bbox-halp');
+            qsr('#bbox-halp');
-            if (h)
-                h.parentNode.removeChild(h);
 
             if (options.afterHide)
                 options.afterHide();
@@ -590,8 +588,7 @@ window.baguetteBox = (function () {
 
         image.addEventListener(is_vid ? 'loadedmetadata' : 'load', function () {
             // Remove loader element
-            var spinner = QS('#baguette-img-' + index + ' .bbox-spinner');
+            qsr('#baguette-img-' + index + ' .bbox-spinner');
-            figure.removeChild(spinner);
             if (!options.async && callback)
                 callback();
         });

copyparty/web/browser.css

@@ -16,7 +16,6 @@ html,body,tr,th,td,#files,a {
 }
 html {
 	color: #ccc;
-	background: #333;
 	font-family: sans-serif;
 	text-shadow: 1px 1px 0px #000;
 }
@@ -24,7 +23,7 @@ html, body {
 	margin: 0;
 	padding: 0;
 }
-pre, code, tt {
+pre, code, tt, #doc, #doc>code {
 	font-family: 'scp', monospace, monospace;
 }
 #path,
@@ -32,15 +31,12 @@ pre, code, tt {
 	font-size: 1em;
 }
 #path {
-	color: #aca;
+	color: #ccc;
 	text-shadow: 1px 1px 0 #000;
-	font-variant: small-caps;
 	font-weight: normal;
-	background: #4c4c4c;
 	display: inline-block;
 	padding: .35em .5em .2em .5em;
 	border-radius: 0 .3em .3em 0;
-	box-shadow: .1em .1em .4em #222;
 	margin: 1.3em 0 0 0;
 	font-size: 1.4em;
 }
@@ -71,7 +67,7 @@ a, #files tbody div a:last-child {
 }
 #files a:hover {
 	color: #fff;
-	background: #161616;
+	background: #111;
 	text-decoration: underline;
 }
 #files thead {
@@ -82,38 +78,23 @@ a, #files tbody div a:last-child {
 	color: #999;
 	font-weight: normal;
 }
-#files tr:hover td {
+#files tbody tr:hover td {
 	background: #1c1c1c;
 }
 #files thead th {
-	padding: .5em .3em .3em .3em;
+	padding: 0 .3em .3em .3em;
-	border-right: 2px solid #3c3c3c;
+	border-bottom: 1px solid #444;
-	border-bottom: 2px solid #444;
-	background: #333;
 	cursor: pointer;
 }
-#files thead th+th {
-	border-left: 2px solid #2a2a2a;
-}
-#files thead th:last-child {
-	border-right: none;
-}
-#files tbody {
-	background: #222;
-}
 #files td {
 	margin: 0;
-	padding: 0 .5em;
+	padding: .1em .5em;
-	border-bottom: 1px solid #111;
+	border-left: 1px solid #3c3c3c;
-	border-left: 1px solid #2c2c2c;
 }
 #files td+td+td {
 	max-width: 30em;
 	overflow: hidden;
 }
-#files tr+tr td {
-	border-top: 1px solid #383838;
-}
 #files tbody td:nth-child(3) {
 	font-family: 'scp', monospace, monospace;
 	text-align: right;
@@ -121,18 +102,15 @@ a, #files tbody div a:last-child {
 	white-space: nowrap;
 }
 #files tbody td:first-child {
-	padding-left: 1.5em;
 	color: #888;
-}
+	text-align: center;
-#files tbody tr:first-child td {
-	padding-top: .9em;
 }
 #files tbody tr:last-child td {
-	padding-bottom: 1.3em;
+	border-bottom: 1px solid #444;
-	border-bottom: .5em solid #444;
 }
 #files tbody tr td:last-child {
 	white-space: nowrap;
+	border-right: 1px solid #3c3c3c;
 }
 #files thead th[style] {
 	width: auto !important;
@@ -163,7 +141,7 @@ a, #files tbody div a:last-child {
 	background: linear-gradient(90deg, rgba(0,0,0,0), rgba(0,0,0,0.2), rgba(0,0,0,0));
 }
 .logue {
-	padding: .2em 1.5em;
+	padding: .2em 0;
 }
 .logue.hidden,
 .logue:empty {
@@ -175,6 +153,21 @@ a, #files tbody div a:last-child {
 #epi.logue {
 	margin: .8em 0;
 }
+#epi.logue.mdo:before {
+	content: 'README.md';
+	text-align: center;
+	display: block;
+	margin-top: -1.5em;
+}
+#epi.logue.mdo {
+	border-top: 1px solid #555;
+	margin-top: 2.5em;
+}
+.mdo>h1:first-child,
+.mdo>h2:first-child,
+.mdo>h3:first-child {
+	margin-top: 1.5rem;
+}
 .mdo {
 	max-width: 52em;
 }
@@ -184,7 +177,6 @@ a, #files tbody div a:last-child {
 }
 #srv_info {
 	color: #a73;
-	background: #333;
 	position: absolute;
 	font-size: .8em;
  	top: .5em;
@@ -212,6 +204,12 @@ a, #files tbody div a:last-child {
 #repl {
 	padding: .33em;
 }
+#files a.doc {
+	color: #666;
+}
+#files a.doc.bri {
+	color: #f5a;
+}
 #files tbody a.play {
 	color: #e70;
 	padding: .2em;
@@ -286,9 +284,6 @@ html.light #ggrid>a.sel {
 #files tr:focus td:first-child {
 	box-shadow: -.2em .2em 0 #fc0, -.2em -.2em 0 #fc0;
 }
-#files tr:focus+tr td {
-	border-top: 1px solid transparent;
-}
 #widget {
 	position: fixed;
 	font-size: 1.4em;
@@ -310,7 +305,6 @@ html.light #ggrid>a.sel {
 	z-index: 10;
 	width: 100%;
 	height: 100%;
-	background: #3c3c3c;
 }
 #wtgrid,
 #wtico {
@@ -351,7 +345,6 @@ html.light #ggrid>a.sel {
 	line-height: 1em;
 	text-align: center;
 	text-shadow: none;
-	background: #3c3c3c;
 	box-shadow: 0 0 .5em #222;
 	border-radius: .3em 0 0 0;
 	padding: 0 0 0 .1em;
@@ -363,7 +356,7 @@ html.light #ggrid>a.sel {
 #wzip, #wnp {
 	margin-right: .2em;
 	padding-right: .2em;
-	border: 1px solid #555;
+	border: 1px solid #444;
 	border-width: 0 .1em 0 0;
 }
 #wfm.act+#wzip,
@@ -519,36 +512,33 @@ html.light #wfm a:not(.en) {
 	box-shadow: 0 -.15em .2em #000 inset;
 	padding-bottom: .3em;
 }
-#ops,
+#ops a svg {
-.opbox,
+	width: 1.75em;
-#u2etas {
+	height: 1.75em;
-	border: 1px solid #3a3a3a;
+	margin: -.5em -.3em;
-	box-shadow: 0 0 1em #222 inset;
 }
 #ops {
-	background: #333;
 	margin: 1.7em 1.5em 0 1.5em;
 	padding: .3em .6em;
 	border-radius: .3em;
-	border-width: .15em 0;
+	border-width: 1px 0;
 	white-space: nowrap;
 }
 .opbox {
-	background: #2d2d2d;
 	margin: 1.5em 0 0 0;
 	padding: .5em;
-	border-radius: 0 1em 1em 0;
+	border-radius: 0 .3em .3em 0;
-	border-width: .15em .3em .3em 0;
+	border-width: 1px 1px 1px 0;
 	max-width: 41em;
+	max-width: min(41em, calc(100% - 2.6em));
 }
 .opbox input {
 	margin: .5em;
 }
 .opview input[type=text] {
-	background: #383838;
 	color: #fff;
 	border: none;
-	box-shadow: 0 0 .3em #222;
+	box-shadow: 0 0 .3em #181818;
 	border-bottom: 1px solid #fc5;
 	border-radius: .2em;
 	padding: .2em .3em;
@@ -565,14 +555,12 @@ html.light .opview input[type="text"].err {
 input[type="checkbox"]+label {
 	color: #f5a;
 }
+input[type="radio"]:checked+label,
 input[type="checkbox"]:checked+label {
 	color: #fc5;
 }
-input[type="radio"]:checked+label {
+.opview input.i {
-	color: #fc0;
+	width: calc(100% - 16.2em);
-}
-html.light input[type="radio"]:checked+label {
-	color: #07c;
 }
 input.eq_gain {
 	width: 3em;
@@ -602,11 +590,6 @@ input.eq_gain {
 
 
 #srch_form {
-	border: 1px solid #3a3a3a;
-	box-shadow: 0 0 1em #222 inset;
-	background: #2d2d2d;
-	border-radius: .4em;
-	margin: 1.4em;
 	margin-bottom: 0;
 	padding: 0 .5em .5em 0;
 }
@@ -663,8 +646,8 @@ input.eq_gain {
 	width: 100%;
 }
 #wrap {
-	margin-top: 2em;
+	margin: 1.8em 1.5em 0 1.5em;
-	min-height: 90vh;
+	min-height: 70vh;
 	padding-bottom: 5em;
 }
 #tree {
@@ -678,19 +661,34 @@ input.eq_gain {
 	-ms-scroll-chaining: none;
 	overscroll-behavior-y: none;
 	scrollbar-color: #eb0 #333;
+	border: 1px solid #333;
+	box-shadow: 0 0 1em #181818;
 }
 #treeh {
-	background: #333;
 	position: sticky;
 	z-index: 1;
 	top: 0;
 	height: 2.2em;
 	line-height: 2.2em;
-	border-bottom: 1px solid #555;
+	border-bottom: 1px solid #111;
 	overflow: hidden;
 }
-#thx_ff {
+#treepar {
-	padding: 5em 0;
+	z-index: 1;
+	position: fixed;
+	left: -.75em;
+	width: calc(var(--nav-sz) - 0.5em);
+	border-bottom: 1px solid #444;
+	overflow: hidden;
+}
+#treepar.off {
+	display: none;
+}
+#tree, #treeh {
+	border-radius: 0 .3em 0 0;
+}
+.np_open #thx_ff {
+	padding: 4.5em 0;
 	/* widget */
 }
 #tree::-webkit-scrollbar-track,
@@ -711,8 +709,6 @@ input.eq_gain {
 .btn {
 	padding: .2em .4em;
 	font-size: 1.2em;
-	background: #2a2a2a;
-	box-shadow: 0 .1em .2em #222 inset;
 	border-radius: .3em;
 	margin: .2em;
 	white-space: pre;
@@ -741,48 +737,57 @@ input.eq_gain {
 	margin: 0;
 }
 #tree ul {
-	border-left: .2em solid #555;
+	border-left: .2em solid #444;
 }
 #tree li {
 	margin-left: 1em;
 	list-style: none;
-	border-top: 1px solid #4c4c4c;
+	border-top: 1px solid #444;
-	border-bottom: 1px solid #222;
+	border-bottom: 1px solid #111;
 }
 #tree li:last-child {
 	border-bottom: none;
 }
-#treeul a.hl {
+#tree ul a.sel {
+	background: #111;
+	box-shadow: -.8em 0 0 #c37 inset;
+	color: #fff;
+}
+#tree ul a.hl {
 	color: #400;
 	background: #fc4;
 	text-shadow: none;
 }
-#treeul a {
+#tree ul a.par {
+	color: #fff;
+}
+#tree ul a {
 	border-radius: .3em;
 	display: inline-block;
 }
-#treeul a+a {
+.ntree a+a {
 	width: calc(100% - 2em);
 	line-height: 1em;
 }
-#tree.nowrap #treeul li {
+#tree.nowrap li {
 	min-height: 1.4em;
 	white-space: nowrap;
 }
-#tree.nowrap #treeul a+a:hover {
+#tree.nowrap .ntree a+a:hover {
-	background: rgba(34, 34, 34, 0.67);
+	background: rgba(16, 16, 16, 0.67);
 	min-width: calc(var(--nav-sz) - 2em);
 	width: auto;
 }
-html.light #tree.nowrap #treeul a+a:hover {
+html.light #tree.nowrap .ntree a+a:hover {
 	background: rgba(255, 255, 255, 0.67);
 	color: #000;
 }
-#treeul a+a:hover {
+#docul a:hover,
-	background: #222;
+#tree .ntree a+a:hover {
+	background: #181818;
 	color: #fff;
 }
-#treeul a:first-child {
+.ntree a:first-child {
 	font-family: 'scp', monospace, monospace;
 	font-size: 1.2em;
 	line-height: 0;
@@ -818,22 +823,23 @@ html.light #tree.nowrap #treeul a+a:hover {
 #files td:nth-child(2n) {
 	color: #f5a;
 }
-#files tr.play td,
+#files tbody tr.play td,
-#files tr.play div a {
+#files tbody tr.play div a {
 	background: #fc4;
 	border-color: transparent;
 	color: #400;
 	text-shadow: none;
 }
-#files tr.play a {
+#files tbody tr.play a {
 	color: inherit;
 }
-#files tr.play a:hover {
+#files tbody tr.play a:hover {
 	color: #300;
 	background: #fea;
 }
 .opwide,
-#op_unpost {
+#op_unpost,
+#srch_form {
 	max-width: none;
 	margin-right: 1.5em;
 }
@@ -876,31 +882,31 @@ html.light #tree.nowrap #treeul a+a:hover {
 #wraptree.on+#hovertree {
 	display: none;
 }
-#ghead {
+.ghead {
-	background: #3c3c3c;
-	border: 1px solid #444;
 	border-radius: .3em;
 	padding: .2em .5em;
 	line-height: 2.3em;
-	margin: 0 1.5em 1em .4em;
+	margin-bottom: 1.5em;
+}
+#ghead {
 	position: sticky;
 	top: -.3em;
 	z-index: 1;
 }
-html.light #ghead {
+html.light .ghead {
 	background: #f7f7f7;
 	border-color: #ddd;
 }
-#ghead .btn {
+.ghead .btn {
 	position: relative;
 	top: 0;
 }
-#ghead>span {
+.ghead>span {
 	white-space: pre;
 	padding-left: .3em;
 }
 #ggrid {
-	padding-top: .5em;
+	margin: -.2em -.5em;
 }
 #ggrid>a>span {
 	overflow: hidden;
@@ -916,17 +922,10 @@ html.light #ghead {
 	width: var(--grid-sz);
 	vertical-align: top;
 	overflow-wrap: break-word;
-	background: #383838;
-	border: 1px solid #444;
-	border-top: 1px solid #555;
-	box-shadow: 0 .1em .2em #222;
 	border-radius: .3em;
 	padding: .3em;
 	margin: .5em;
 }
-#ggrid>a[tt] {
-	background: linear-gradient(135deg, #383838 95%, #555 95%);
-}
 #ggrid>a img {
 	border-radius: .2em;
 	max-width: 10em;
@@ -949,25 +948,6 @@ html.light #ghead {
 	border-radius: .3em;
 	font-size: 2em;
 }
-#ggrid>a:hover {
-	background: #444;
-	border-color: #555;
-	color: #fd9;
-}
-html.light #ggrid>a {
-	background: #f7f7f7;
-	border-color: #ddd;
-	box-shadow: 0 .1em .2em #ddd;
-}
-html.light #ggrid>a[tt] {
-	background: linear-gradient(135deg, #f7f7f7 95%, #ccc 95%);
-}
-html.light #ggrid>a:hover {
-	background: #fff;
-	border-color: #ccc;
-	color: #015;
-	box-shadow: 0 .1em .5em #aaa;
-}
 #op_unpost {
 	padding: 1em;
 }
@@ -988,7 +968,6 @@ html.light #ggrid>a:hover {
 	max-height: calc(100% - 2em);
 	border-bottom: .5em solid #999;
 	box-shadow: 0 0 5em rgba(0,0,0,0.8);
-	background: #333;
 	padding: 1em;
 	z-index: 765;
 }
@@ -1036,6 +1015,53 @@ html.light #rui {
 	padding: 0;
 	font-size: 1.5em;
 }
+#doc {
+	background: none;
+	overflow: visible;
+	margin: -1em 0 .5em 0;
+	padding: 1em 0 1em 0;
+}
+#docul {
+	position: relative;
+}
+#docul li.bn {
+	text-align: center;
+	padding: .5em;
+}
+#doc.prism {
+	padding-left: 3em;
+}
+#doc>code {
+	background: none;
+	box-shadow: none;
+	z-index: 1;
+}
+#doc.mdo {
+	white-space: normal;
+	font-family: sans-serif;
+}
+#doc.prism * {
+	line-height: 1.5em;
+}
+#doc .line-highlight {
+	border-radius: .3em;
+	box-shadow: 0 0 .5em #333;
+	background: linear-gradient(90deg, #111, #222);
+}
+html.light #doc .line-highlight {
+	box-shadow: 0 0 .5em #ccc;
+	background: linear-gradient(90deg, #fff, #eee);
+}
+#docul li {
+	margin: 0;
+}
+#tree #docul a {
+	display: block;
+}
+#seldoc.sel {
+	color: #fff;
+	background: #925;
+}
 #pvol,
 #barbuf,
 #barpos,
@@ -1072,6 +1098,78 @@ a.btn,
 
 
 
+
+
+
+
+
+html,
+#rui,
+#files td,
+#files thead th,
+#bbox-halp,
+#u2notbtn,
+#srv_info {
+	background: #222;
+}
+#ops,
+.opbox,
+#path,
+#srch_form,
+.ghead {
+	background: #2b2b2b;
+	border: 1px solid #333;
+	box-shadow: 0 0 .3em #111;
+}
+#files tr:nth-child(2n+1) td {
+	background: #282828;
+}
+#tree,
+#treeh,
+#treepar {
+	background: #2b2b2b;
+}
+#wtoggle,
+#widgeti {
+	background: #333;
+}
+.btn,
+.opview input[type=text] {
+	background: #383838;
+}
+
+
+
+
+
+#ggrid>a {
+	background: #2c2c2c;
+	border: 1px solid #383838;
+	border-top: 1px solid #444;
+	box-shadow: 0 .1em .2em #181818;
+}
+#ggrid>a[tt] {
+	background: linear-gradient(135deg, #2c2c2c 95%, #444 95%);
+}
+#ggrid>a:hover {
+	background: #383838;
+	border-color: #555;
+	color: #fd9;
+}
+html.light #ggrid>a {
+	background: #f7f7f7;
+	border-color: #ddd;
+	box-shadow: 0 .1em .2em #ddd;
+}
+html.light #ggrid>a[tt] {
+	background: linear-gradient(135deg, #f7f7f7 95%, #ccc 95%);
+}
+html.light #ggrid>a:hover {
+	background: #fff;
+	border-color: #ccc;
+	color: #015;
+	box-shadow: 0 .1em .5em #aaa;
+}
 
 
 
@@ -1079,23 +1177,31 @@ a.btn,
 
 html.light {
 	color: #333;
-	background: #eee;
+	background: #eaeaea;
 	text-shadow: none;
 }
 html.light #ops,
 html.light .opbox,
+html.light #path,
 html.light #srch_form,
+html.light .ghead,
 html.light #u2etas {
 	background: #f7f7f7;
-	box-shadow: 0 0 .3em #ddd;
+	box-shadow: 0 0 .3em #ccc;
 	border-color: #f7f7f7;
 }
+html.light #wrap.doc {
+	background: #f7f7f7;
+}
 html.light #ops a.act {
 	box-shadow: 0 .2em .2em #ccc;
 	background: #fff;
 	border-color: #07a;
 	padding-top: .4em;
 }
+html.light #ops svg circle {
+	stroke: black;
+}
 html.light #op_cfg h3 {
 	border-color: #ccc;
 }
@@ -1125,21 +1231,25 @@ html.light #acc_info {
 html.light #srv_info span {
 	color: #777;
 }
-html.light #treeul a+a {
+html.light #tree .ntree a+a {
 	background: inherit;
 	color: #06a;
 }
-html.light #treeul a.hl {
+html.light #tree ul a.hl {
 	background: #07a;
 	color: #fff;
 }
-html.light #treeul a.hl:hover {
+html.light #tree ul a.par {
+	color: #000;
+}
+html.light #tree ul a.hl:hover {
 	background: #059;
 }
-html.light #tree li {
+html.light #tree li,
+html.light #tree #treepar {
 	border-color: #f7f7f7 #fff #ddd #fff;
 }
-html.light #treeul a:hover {
+html.light #tree ul a:hover {
 	background: #fff;
 }
 html.light #tree ul {
@@ -1150,25 +1260,19 @@ html.light #ops a,
 html.light #files tbody div a:last-child {
 	color: #06a;
 }
-html.light #files tbody {
+html.light #files thead th {
+	background: #eaeaea;
+	border-color: #ccc;
+}
+html.light #files tbody td {
+	background: #eee;
+	border-color: #ccc;
+}
+html.light #files tr:nth-child(2n+1) td {
 	background: #f7f7f7;
 }
-html.light #files {
-	box-shadow: 0 0 .3em #ccc;
-}
-html.light #files thead th {
-	background: #eee;
-	border: 1px solid #ccc;
-	border-top: none;
-}
-html.light #files thead th+th {
-	border-left: 1px solid #f7f7f7;
-}
-html.light #files td {
-	border-color: #fff #fff #ddd #ddd;
-}
 html.light #files tbody tr:last-child td {
-	border-bottom: .2em solid #ccc;
+	border-bottom: 1px solid #ccc;
 }
 html.light #files tr:focus td {
 	background: #fff;
@@ -1193,6 +1297,12 @@ html.light #files tbody a.play {
 html.light #files tbody a.play.act {
 	color: #90c;
 }
+html.light #files a.doc {
+	color: #bbb;
+}
+html.light #files a.doc.bri {
+	color: #d38;
+}
 html.light #files tr.play td {
 	background: #fc5;
 	border-color: #eb1;
@@ -1242,6 +1352,10 @@ html.light #files tr.sel a.play.act {
 html.light input[type="checkbox"] + label {
 	color: #333;
 }
+html.light input[type="radio"]:checked + label,
+html.light input[type="checkbox"]:checked + label {
+	color: #07c; 
+}
 html.light .opwide>div {
 	border-color: #ccc;
 }
@@ -1256,6 +1370,7 @@ html.light #files td div span {
 	color: #000;
 }
 html.light #path {
+	color: #777;
 	background: #f7f7f7;
 	text-shadow: none;
 	box-shadow: 0 0 .3em #bbb;
@@ -1273,24 +1388,30 @@ html.light #path a:hover {
 html.light #files tbody div a {
 	color: #d38;
 }
+html.light #docul a:hover,
 html.light #files a:hover,
 html.light #files tr.sel a:hover {
 	color: #000;
 	background: #fff;
+	text-decoration: underline;
 }
-html.light #treeh {
+html.light #treeh,
-	background: #eee;
+html.light #treepar {
+	background: #f7f7f7;
 	border-color: #ddd;
 }
 html.light #tree {
-	scrollbar-color: #a70 #ddd;
+	border-color: #ddd;
+	box-shadow: 0 0 1em #ddd;
+	background: #f7f7f7;
+	scrollbar-color: #490 #ddd;
 }
 html.light #tree::-webkit-scrollbar-track,
 html.light #tree::-webkit-scrollbar {
 	background: #ddd;
 }
-#tree::-webkit-scrollbar-thumb {
+html.light #tree::-webkit-scrollbar-thumb {
-	background: #da0;
+	background: #490;
 }
 
 
@@ -1369,7 +1490,7 @@ html.light #tree::-webkit-scrollbar {
 	box-shadow: 0 0 8px rgba(0, 0, 0, 0.6);
 }
 .full-image video {
-	background: #333;
+	background: #222;
 }
 .full-image figcaption {
 	display: block;
@@ -1465,7 +1586,6 @@ html.light #bbox-overlay figcaption a {
 }
 #bbox-halp {
 	color: #fff;
-	background: #333;
 	position: absolute;
 	top: 0;
 	left: 0;
@@ -1673,14 +1793,20 @@ html.light #u2err.err {
 	cursor: pointer;
 	box-shadow: .4em .4em 0 #111;
 }
+#u2conf.ww #u2btn {
+	line-height: 1em;
+	padding: .5em 0;
+	margin: -1.5em .5em -3em 0;
+}
 #op_up2k.srch #u2btn {
 	background: linear-gradient(to bottom, #ca3 0%, #fd8 50%, #fc6 51%, #b92 100%);
 	text-shadow: 1px 1px 1px #fc6;
 	color: #333;
 }
 #u2conf #u2btn {
-	margin: -2.4em 0;
+	padding: .6em 0;
-	padding: .8em 0;
+	margin: -2em 0;
+	font-size: 1.25em;
 	width: 100%;
 	max-width: 12em;
 	display: inline-block;
@@ -1691,7 +1817,6 @@ html.light #u2err.err {
 #u2notbtn {
 	display: none;
 	text-align: center;
-	background: #333;
 	padding-top: 1em;
 }
 #u2notbtn * {
@@ -1724,10 +1849,12 @@ html.light #u2err.err {
 	width: auto;
 }
 #u2tab tbody tr:hover td {
-	background: #222;
+	background: #333;
 }
 #u2etas {
-	background: #333;
+	background: #1c1c1c;
+	border: 1px solid #282828;
+	border-width: .1em 0;
 	padding: .2em .5em;
 	border-radius: .5em;
 	border-width: .25em 0;
@@ -1744,6 +1871,7 @@ html.light #u2err.err {
 	display: none;
 }
 #u2etas.o .o {
+	display: inherit;
 	display: unset;
 }
 #u2etaw {
@@ -1766,16 +1894,22 @@ html.light #u2err.err {
 	width: 44em;
 	text-align: left;
 }
+#u2cards.ww {
+	display: inline-block;
+}
 #u2etaw.w {
 	width: 52em;
 	text-align: right;
 	margin: 3em auto -2.7em auto;
 }
+#u2etaw.ww {
+	margin: 0 2em 1em 2em;
+}
 #u2cards a {
 	padding: .2em 1em;
 	border: 1px solid #777;
 	border-width: 0 0 1px 0;
-	background: linear-gradient(to bottom, #333, #222);
+	background: linear-gradient(to bottom, #222, #2b2b2b);
 }
 #u2cards a:first-child {
 	border-radius: .4em 0 0 0;
@@ -1788,23 +1922,35 @@ html.light #u2err.err {
 	border-width: 1px 1px .1em 1px;
 	border-radius: .3em .3em 0 0;
 	margin-left: -1px;
-	background: linear-gradient(to bottom, #464, #333 80%);
+	background: linear-gradient(to bottom, #353, #222 80%);
 	box-shadow: 0 -.17em .67em #280;
-	border-color: #7c5 #583 #333 #583;
+	border-color: #7c5 #583 #222 #583;
 	position: relative;
 	color: #fd7;
 }
 #u2cards span {
 	color: #fff;
 }
+#u2cards > a:nth-child(4) > span {
+	display: inline-block;
+	text-align: center;
+	min-width: 1.3em;
+}
 #u2conf {
 	margin: 1em auto;
 	width: 30em;
 }
-#u2conf.has_btn {
+#u2conf.w {
 	width: 48em;
 }
-#u2conf * {
+#u2conf.ww {
+	width: 74em;
+}
+#u2conf.ww #u2c3w {
+	width: 29em;
+}
+#u2conf .c,
+#u2conf .c * {
 	text-align: center;
 	line-height: 1em;
 	margin: 0;
@@ -1824,7 +1970,7 @@ html.light #u2err.err {
 #u2conf .txtbox.err {
 	background: #922;
 }
-#u2conf a {
+#u2conf a.b {
 	color: #fff;
 	background: #c38;
 	text-decoration: none;
@@ -1838,10 +1984,10 @@ html.light #u2err.err {
 	position: relative;
 	bottom: -0.08em;
 }
-#u2conf input+a {
+#u2conf input+a.b {
 	background: #d80;
 }
-#u2conf label {
+#u2conf .c label {
 	font-size: 1.6em;
 	width: 2em;
 	height: 1em;

copyparty/web/browser.html

@@ -18,9 +18,9 @@
 
 	<div id="op_search" class="opview">
 		{%- if have_tags_idx %}
-		<div id="srch_form" class="tags"></div>
+		<div id="srch_form" class="tags opbox"></div>
 		{%- else %}
-		<div id="srch_form"></div>
+		<div id="srch_form" class="opbox"></div>
 		{%- endif %}
 		<div id="srch_q"></div>
 	</div>
@@ -31,7 +31,7 @@
 		<div id="u2err"></div>
 		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
 			<input type="hidden" name="act" value="bput" />
-			<input type="file" name="f" multiple><br />
+			<input type="file" name="f" multiple /><br />
 			<input type="submit" value="start upload">
 		</form>
 	</div>
@@ -39,7 +39,7 @@
 	<div id="op_mkdir" class="opview opbox act">
 		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
 			<input type="hidden" name="act" value="mkdir" />
-			📂<input type="text" name="name" size="30">
+			📂<input type="text" name="name" class="i">
 			<input type="submit" value="make directory">
 		</form>
 	</div>
@@ -47,15 +47,15 @@
 	<div id="op_new_md" class="opview opbox">
 		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
 			<input type="hidden" name="act" value="new_md" />
-			📝<input type="text" name="name" size="30">
+			📝<input type="text" name="name" class="i">
 			<input type="submit" value="new markdown doc">
 		</form>
 	</div>
 
 	<div id="op_msg" class="opview opbox act">
 		<form method="post" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ url_suf }}">
-			📟<input type="text" name="msg" size="30">
+			📟<input type="text" name="msg" class="i">
-			<input type="submit" value="send msg to server log">
+			<input type="submit" value="send msg to srv log">
 		</form>
 	</div>
 
@@ -76,6 +76,12 @@
 
 <div id="wrap">
 
+	{%- if doc %}
+	<div id="bdoc"><pre>{{ doc|e }}</pre></div>
+	{%- else %}
+	<div id="bdoc"></div>
+	{%- endif %}
+
 	<div id="pro" class="logue">{{ logues[0] }}</div>
 
 	<table id="files">
@@ -130,10 +136,13 @@
 			def_hcols = {{ def_hcols|tojson }},
 			have_up2k_idx = {{ have_up2k_idx|tojson }},
 			have_tags_idx = {{ have_tags_idx|tojson }},
+			have_acode = {{ have_acode|tojson }},
 			have_mv = {{ have_mv|tojson }},
 			have_del = {{ have_del|tojson }},
 			have_unpost = {{ have_unpost|tojson }},
 			have_zip = {{ have_zip|tojson }},
+			txt_ext = "{{ txt_ext }}",
+			{% if no_prism %}no_prism = 1,{% endif %}
 			readme = {{ readme|tojson }};
 
 		document.documentElement.setAttribute("class", localStorage.lightmode == 1 ? "light" : "dark");
@@ -141,6 +150,9 @@
 	<script src="/.cpr/util.js?_={{ ts }}"></script>
 	<script src="/.cpr/browser.js?_={{ ts }}"></script>
 	<script src="/.cpr/up2k.js?_={{ ts }}"></script>
+	{%- if js %}
+	<script src="{{ js }}?_={{ ts }}"></script>
+	{%- endif %}
 </body>
 
 </html>

copyparty/web/md.js

@@ -164,10 +164,7 @@ function copydom(src, dst, lv) {
 
 
 function md_plug_err(ex, js) {
-    var errbox = ebi('md_errbox');
+    qsr('#md_errbox');
-    if (errbox)
-        errbox.parentNode.removeChild(errbox);
-
     if (!ex)
         return;
 
@@ -183,7 +180,7 @@ function md_plug_err(ex, js) {
             o.textContent = lns[ln - 1];
         }
     }
-    errbox = mknod('div');
+    var errbox = mknod('div');
     errbox.setAttribute('id', 'md_errbox');
     errbox.style.cssText = 'position:absolute;top:0;left:0;padding:1em .5em;background:#2b2b2b;color:#fc5'
     errbox.textContent = msg;
@@ -381,8 +378,7 @@ function convert_markdown(md_text, dest_dom) {
 
 
 function init_toc() {
-    var loader = ebi('ml');
+    qsr('#ml');
-    loader.parentNode.removeChild(loader);
 
     var anchors = [];  // list of toc entries, complex objects
     var anchor = null; // current toc node

copyparty/web/mde.js

@@ -65,8 +65,7 @@ var mde = (function () {
     mde.codemirror.on("change", function () {
         md_changed(mde);
     });
-    var loader = ebi('ml');
+    qsr('#ml');
-    loader.parentNode.removeChild(loader);
     return mde;
 })();
 

copyparty/web/splash.css

@@ -25,10 +25,22 @@ a {
 	color: #047;
 	background: #fff;
 	text-decoration: none;
-	border-bottom: 1px solid #aaa;
+	border-bottom: 1px solid #8ab;
 	border-radius: .2em;
 	padding: .2em .8em;
 }
+a+a {
+	margin-left: .5em;
+}
+.refresh,
+.logout {
+	float: right;
+	margin: -.2em 0 0 .5em;
+}
+.logout {
+	color: #c04;
+	border-color: #c7a;
+}
 #repl {
 	border: none;
 	background: none;
@@ -42,6 +54,7 @@ table {
 .vols th {
 	padding: .3em .6em;
 	text-align: left;
+	white-space: nowrap;
 }
 .num {
 	border-right: 1px solid #bbb;
@@ -81,6 +94,10 @@ html.dark a {
 	background: #057;
 	border-color: #37a;
 }
+html.dark .logout {
+	background: #804;
+	border-color: #c28;
+}
 html.dark input {
 	color: #fff;
 	background: #626;

copyparty/web/splash.html

@@ -12,9 +12,12 @@
 
 <body>
     <div id="wrap">
+        <a href="/?h" class="refresh">refresh</a>
+
         {%- if this.uname == '*' %}
             <p>howdy stranger &nbsp; <small>(you're not logged in)</small></p>
         {%- else %}
+            <a href="/?pw=x" class="logout">logout</a>
             <p>welcome back, <strong>{{ this.uname }}</strong></p>
         {%- endif %}
 
@@ -46,7 +49,8 @@
             </table>
         </td></tr></table>
         <div class="btns">
-            <a href="/?stack">dump stack</a>
+            <a href="/?stack" tt="shows the state of all active threads">dump stack</a>
+            <a href="/?reload=cfg" tt="reload config files (accounts/volumes/volflags),$Nand rescan all e2ds volumes">reload cfg</a>
         </div>
         {%- endif %}
 
@@ -85,5 +89,6 @@ if (localStorage.lightmode != 1)
 
 </script>
 <script src="/.cpr/util.js?_={{ ts }}"></script>
+<script>tt.init();</script>
 </body>
 </html>

copyparty/web/ui.css

@@ -11,9 +11,9 @@ html {
 	max-width: 34em;
 	max-width: min(34em, 90%);
 	max-width: min(34em, calc(100% - 7em));
-	background: #222;
+	background: #333;
 	border: 0 solid #777;
-	box-shadow: 0 .2em .5em #222;
+	box-shadow: 0 .2em .5em #111;
 	border-radius: .4em;
 	z-index: 9001;
 }
@@ -79,7 +79,8 @@ html {
 }
 #toast.vis {
 	right: 1.3em;
-	transform: unset;
+	transform: inherit;
+	transform: initial;
 }
 #toast.vis #toastc {
 	left: -2em;

copyparty/web/up2k.js

@@ -30,7 +30,10 @@ catch (ex) {
     try {
         up2k = up2k_init(false);
     }
-    catch (ex) { }
+    catch (ex) {
+        console.log('up2k init failed:', ex);
+        toast.err(10, 'could not initialze up2k\n\n' + basenames(ex));
+    }
 }
 treectl.onscroll();
 
@@ -210,14 +213,14 @@ function U2pvis(act, btns) {
     };
 
     r.setat = function (nfile, blocktab) {
-        r.tab[nfile].cb = blocktab;
+        var fo = r.tab[nfile], bd = 0;
 
-        var bd = 0;
         for (var a = 0; a < blocktab.length; a++)
             bd += blocktab[a];
 
-        r.tab[nfile].bd = bd;
+        fo.bd = bd;
-        r.tab[nfile].bd0 = bd;
+        fo.bd0 = bd;
+        fo.cb = blocktab;
     };
 
     r.perc = function (bd, bd0, sz, t0) {
@@ -246,7 +249,7 @@ function U2pvis(act, btns) {
 
         obj.innerHTML = fo.hp;
         obj.style.color = '#fff';
-        obj.style.background = 'linear-gradient(90deg, #025, #06a ' + o1 + '%, #09d ' + o2 + '%, #333 ' + o3 + '%, #333 99%, #777)';
+        obj.style.background = 'linear-gradient(90deg, #025, #06a ' + o1 + '%, #09d ' + o2 + '%, #222 ' + o3 + '%, #222 99%, #555)';
     };
 
     r.prog = function (fobj, nchunk, cbd) {
@@ -303,7 +306,7 @@ function U2pvis(act, btns) {
 
         obj.innerHTML = fo.hp;
         obj.style.color = '#fff';
-        obj.style.background = 'linear-gradient(90deg, #050, #270 ' + o1 + '%, #4b0 ' + o2 + '%, #333 ' + o3 + '%, #333 99%, #777)';
+        obj.style.background = 'linear-gradient(90deg, #050, #270 ' + o1 + '%, #4b0 ' + o2 + '%, #222 ' + o3 + '%, #222 99%, #555)';
     };
 
     r.move = function (nfile, newcat) {
@@ -329,8 +332,7 @@ function U2pvis(act, btns) {
                 r.head++;
 
             if (!bz_act) {
-                var tr = ebi("f" + nfile);
+                qsr("#f" + nfile);
-                tr.parentNode.removeChild(tr);
             }
         }
         else return;
@@ -349,9 +351,7 @@ function U2pvis(act, btns) {
         last = parseInt(last.getAttribute('id').slice(1));
 
         while (r.head - first > r.wsz) {
-            var obj = ebi('f' + (first++));
+            qsr('#f' + (first++));
-            if (obj)
-                obj.parentNode.removeChild(obj);
         }
         while (last - r.tail < r.wsz && last < r.tab.length - 2) {
             var obj = ebi('f' + (++last));
@@ -477,6 +477,86 @@ function U2pvis(act, btns) {
 }
 
 
+function Donut(uc, st) {
+    var r = this,
+        el = null,
+        psvg = null,
+        o = 20 * 2 * Math.PI,
+        optab = QS('#ops a[data-dest="up2k"]');
+
+    optab.setAttribute('ico', optab.textContent);
+
+    function svg(v) {
+        var ico = v !== undefined,
+            bg = ico ? '#333' : 'transparent',
+            fg = '#fff',
+            fsz = 52,
+            rc = 32;
+
+        if (r.eta && (r.eta > 99 || (uc.fsearch ? st.time.hashing : st.time.uploading) < 20))
+            r.eta = null;
+
+        if (r.eta) {
+            if (r.eta < 10) {
+                fg = '#fa0';
+                fsz = 72;
+            }
+            rc = 8;
+        }
+
+        return (
+            '<svg version="1.1" viewBox="0 0 64 64" xmlns="http://www.w3.org/2000/svg">\n' +
+            (ico ? '<rect width="100%" height="100%" rx="' + rc + '" fill="#333" />\n' :
+                '<circle stroke="white" stroke-width="6" r="3" cx="32" cy="32" />\n') +
+            (r.eta ? (
+                '<text x="55%" y="58%" dominant-baseline="middle" text-anchor="middle"' +
+                ' font-family="sans-serif" font-weight="bold" font-size="' + fsz + 'px"' +
+                ' fill="' + fg + '">' + r.eta + '</text></svg>'
+            ) : (
+                '<circle class="donut" stroke="white" fill="' + bg +
+                '" stroke-dashoffset="' + (ico ? v : o) + '" stroke-dasharray="' + o + ' ' + o +
+                '" transform="rotate(270 32 32)" stroke-width="12" r="20" cx="32" cy="32" /></svg>'
+            ))
+        );
+    }
+
+    function pos() {
+        return uc.fsearch ? Math.max(st.bytes.hashed, st.bytes.finished) : st.bytes.finished;
+    }
+
+    r.on = function (ya) {
+        r.fc = 99;
+        r.eta = null;
+        r.base = pos();
+        optab.innerHTML = ya ? svg() : optab.getAttribute('ico');
+        el = QS('#ops a .donut');
+        if (!ya)
+            favico.upd();
+    };
+    r.do = function () {
+        if (!el)
+            return;
+
+        var t = st.bytes.total - r.base,
+            v = pos() - r.base,
+            ofs = el.style.strokeDashoffset = o - o * v / t;
+
+        if (favico.txt) {
+            if (++r.fc < 10 && r.eta && r.eta > 99)
+                return;
+
+            var s = svg(ofs);
+            if (s == psvg || (r.eta === null && r.fc < 10))
+                return;
+
+            favico.upd('', s);
+            psvg = s;
+            r.fc = 0;
+        }
+    };
+}
+
+
 function fsearch_explain(n) {
     if (n)
         return toast.inf(60, 'your access to this folder is Read-Only\n\n' + (acct == '*' ? 'you are currently not logged in' : 'you are currently logged in as "' + acct + '"'));
@@ -623,7 +703,8 @@ function up2k_init(subtle) {
             });
     }
 
-    var pvis = new U2pvis("bz", '#u2cards');
+    var pvis = new U2pvis("bz", '#u2cards'),
+        donut = new Donut(uc, st);
 
     var bobslice = null;
     if (window.File)
@@ -942,6 +1023,7 @@ function up2k_init(subtle) {
                 "lmod": lmod / 1000,
                 "purl": fdir,
                 "done": false,
+                "bytes_uploaded": 0,
                 "hash": []
             },
                 key = entry.name + '\n' + entry.size;
@@ -990,11 +1072,9 @@ function up2k_init(subtle) {
         for (var a = 0; a < st.files.length; a++) {
             var t = st.files[a];
             if (t.done && t.name) {
-                var tr = ebi('f' + t.n);
+                if (!qsr('#f' + t.n))
-                if (!tr)
                     continue;
 
-                tr.parentNode.removeChild(tr);
                 t.name = undefined;
             }
         }
@@ -1066,6 +1146,7 @@ function up2k_init(subtle) {
                 continue;
             }
 
+            donut.eta = eta;
             if (etaskip)
                 continue;
 
@@ -1175,6 +1256,8 @@ function up2k_init(subtle) {
                     window[(is_busy ? "add" : "remove") +
                         "EventListener"]("beforeunload", warn_uploader_busy);
 
+                    donut.on(is_busy);
+
                     if (!is_busy) {
                         var k = uc.fsearch ? 'searches' : 'uploads',
                             ks = uc.fsearch ? 'Search' : 'Upload',
@@ -1196,9 +1279,11 @@ function up2k_init(subtle) {
                             toast.err(t, '{0} {1}'.format(ks, tng));
 
                         timer.rm(etafun);
+                        timer.rm(donut.do);
                         op_minh = 0;
                     }
                     else {
+                        timer.add(donut.do);
                         timer.add(etafun, false);
                         ebi('u2etas').style.textAlign = 'left';
                     }
@@ -1346,7 +1431,6 @@ function up2k_init(subtle) {
     function exec_hash() {
         var t = st.todo.hash.shift();
         st.busy.hash.push(t);
-        t.bytes_uploaded = 0;
 
         var bpend = 0,
             nchunk = 0,
@@ -1827,16 +1911,28 @@ function up2k_init(subtle) {
             wpx = window.innerWidth,
             fpx = parseInt(getComputedStyle(bar)['font-size']),
             wem = wpx * 1.0 / fpx,
-            wide = wem > 54,
+            wide = wem > 54 ? 'w' : '',
-            parent = ebi(wide && has(perms, 'write') ? 'u2btn_cw' : 'u2btn_ct'),
+            write = has(perms, 'write'),
+            parent = ebi(wide && write ? 'u2btn_cw' : 'u2btn_ct'),
             btn = ebi('u2btn');
 
         //console.log([wpx, fpx, wem]);
         if (btn.parentNode !== parent) {
             parent.appendChild(btn);
-            ebi('u2conf').setAttribute('class', wide ? 'has_btn' : '');
+            ebi('u2conf').setAttribute('class', wide);
-            ebi('u2cards').setAttribute('class', wide ? 'w' : '');
+            ebi('u2cards').setAttribute('class', wide);
-            ebi('u2etaw').setAttribute('class', wide ? 'w' : '');
+            ebi('u2etaw').setAttribute('class', wide);
+        }
+
+        wide = wem > 78 ? 'ww' : wide;
+        parent = ebi(wide == 'ww' && write ? 'u2c3w' : 'u2c3t');
+        var its = [ebi('u2etaw'), ebi('u2cards')];
+        if (its[0].parentNode !== parent) {
+            ebi('u2conf').setAttribute('class', wide);
+            for (var a = 0; a < 2; a++) {
+                parent.appendChild(its[a]);
+                its[a].setAttribute('class', wide);
+            }
         }
     }
     window.addEventListener('resize', onresize);
@@ -1849,7 +1945,7 @@ function up2k_init(subtle) {
         setTimeout(onresize, 500);
     }
 
-    var o = QSA('#u2conf *[tt]');
+    var o = QSA('#u2conf .c *[tt]');
     for (var a = o.length - 1; a >= 0; a--) {
         o[a].parentNode.getElementsByTagName('input')[0].setAttribute('tt', o[a].getAttribute('tt'));
     }

copyparty/web/util.js

@@ -18,6 +18,15 @@ var ebi = document.getElementById.bind(document),
     mknod = document.createElement.bind(document);
 
 
+function qsr(sel) {
+    var el = QS(sel);
+    if (el)
+        el.parentNode.removeChild(el);
+
+    return el;
+}
+
+
 // error handler for mobile devices
 function esc(txt) {
     return txt.replace(/[&"<>]/g, function (c) {
@@ -71,7 +80,7 @@ try {
 catch (ex) {
     if (console.stdlog)
         console.log = console.stdlog;
-    console.log(ex);
+    console.log('console capture failed', ex);
 }
 var crashed = false, ignexd = {};
 function vis_exh(msg, url, lineNo, columnNo, error) {
@@ -146,7 +155,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
 
             var s = mknod('style');
             s.innerHTML = (
-                '#exbox{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em;z-index:80386;position:fixed;top:0;left:0;right:0;bottom:0;width:100%;height:100%;overflow:auto;width:calc(100% - 2em)} ' +
+                '#exbox{background:#222;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em;z-index:80386;position:fixed;top:0;left:0;right:0;bottom:0;width:100%;height:100%;overflow:auto;width:calc(100% - 2em)} ' +
                 '#exbox,#exbox *{line-height:1.5em;overflow-wrap:break-word} ' +
                 '#exbox code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} ' +
                 '#exbox a{text-decoration:underline;color:#fc0} ' +
@@ -163,7 +172,6 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
     catch (e) {
         document.body.innerHTML = html.join('\n');
     }
-    throw 'fatal_err';
 }
 function ignex(all) {
     var o = ebi('exbox');
@@ -715,17 +723,19 @@ function scfg_bind(obj, oname, cname, defval, cb) {
 
 function hist_push(url) {
     console.log("h-push " + url);
+    if (window.history && history.pushState)
         history.pushState(url, url, url);
 }
 
 function hist_replace(url) {
     console.log("h-repl " + url);
+    if (window.history && history.replaceState)
         history.replaceState(url, url, url);
 }
 
 function sethash(hv) {
     if (window.history && history.replaceState) {
-        hist_replace(document.location.pathname + '#' + hv);
+        hist_replace(document.location.pathname + document.location.search + '#' + hv);
     }
     else {
         document.location.hash = hv;
@@ -1055,15 +1065,22 @@ var modal = (function () {
     }
 
     function onkey(e) {
-        if (e.code == 'Enter') {
+        var k = e.code,
-            var a = ebi('modal-ng');
+            eok = ebi('modal-ok'),
-            if (a && document.activeElement == a)
+            eng = ebi('modal-ng'),
+            ae = document.activeElement;
+
+        if (k == 'Space' && ae && (ae === eok || ae === eng))
+            k = 'Enter';
+
+        if (k == 'Enter') {
+            if (ae && ae == eng)
                 return ng();
 
             return ok();
         }
 
-        if (e.code == 'Escape')
+        if (k == 'Escape')
             return ng();
     }
 
@@ -1140,6 +1157,7 @@ function repl_load() {
         if (!ret.length)
             ret = [
                 'var v=Object.keys(localStorage); v.sort(); JSON.stringify(v)',
+                "for (var a of QSA('#files a[id]')) a.setAttribute('download','')",
                 'console.hist.slice(-10).join("\\n")'
             ];
 
@@ -1211,28 +1229,31 @@ if (ebi('repl'))
     ebi('repl').onclick = repl;
 
 
+var svg_decl = '<?xml version="1.0" encoding="UTF-8"?>\n';
+
+
 var favico = (function () {
     var r = {};
     r.en = true;
+    r.tag = null;
 
     function gx(txt) {
-        return (
+        return (svg_decl +
-            '<?xml version="1.0" encoding="UTF-8"?>\n' +
+            '<svg version="1.1" viewBox="0 0 64 64" xmlns="http://www.w3.org/2000/svg">\n' +
-            '<svg version="1.1" viewBox="0 0 64 64" xmlns="http://www.w3.org/2000/svg"><g>\n' +
             (r.bg ? '<rect width="100%" height="100%" rx="16" fill="#' + r.bg + '" />\n' : '') +
             '<text x="50%" y="55%" dominant-baseline="middle" text-anchor="middle"' +
             ' font-family="sans-serif" font-weight="bold" font-size="64px"' +
-            ' fill="#' + r.fg + '">' + txt + '</text></g></svg>'
+            ' fill="#' + r.fg + '">' + txt + '</text></svg>'
         );
     }
 
-    r.upd = function () {
+    r.upd = function (txt, svg) {
-        var i = QS('link[rel="icon"]'), b64;
         if (!r.txt)
             return;
 
+        var b64;
         try {
-            b64 = btoa(gx(r.txt));
+            b64 = btoa(svg ? svg_decl + svg : gx(r.txt));
         }
         catch (ex) {
             b64 = encodeURIComponent(r.txt).replace(/%([0-9A-F]{2})/g,
@@ -1241,19 +1262,19 @@ var favico = (function () {
             b64 = btoa(gx(unescape(encodeURIComponent(r.txt))));
         }
 
-        if (!i) {
+        if (!r.tag) {
-            i = mknod('link');
+            r.tag = mknod('link');
-            i.rel = 'icon';
+            r.tag.rel = 'icon';
-            document.head.appendChild(i);
+            document.head.appendChild(r.tag);
         }
-        i.href = 'data:image/svg+xml;base64,' + b64;
+        r.tag.href = 'data:image/svg+xml;base64,' + b64;
     };
 
     r.init = function () {
         clearTimeout(r.to);
         scfg_bind(r, 'txt', 'icot', '', r.upd);
         scfg_bind(r, 'fg', 'icof', 'fc5', r.upd);
-        scfg_bind(r, 'bg', 'icob', '333', r.upd);
+        scfg_bind(r, 'bg', 'icob', '222', r.upd);
         r.upd();
     };
 

docs/README.md

@@ -29,7 +29,7 @@ point `--css-browser` to one of these by URL:
 * notes on using rclone as a fuse client/server
 
 ## [`example.conf`](example.conf)
-* example config file for `-c` (supports accounts, volumes, and volume-flags)
+* example config file for `-c`
 
 
 

docs/browser.css

@@ -1,11 +1,11 @@
 html {
-    background: #333 url('/wp/wallhaven-mdjrqy.jpg') center / cover no-repeat fixed;
+    background: #222 url('/wp/wallhaven-mdjrqy.jpg') center / cover no-repeat fixed;
 }
 #files th {
     background: rgba(32, 32, 32, 0.9) !important;
 }
 #ops,
-#treeul,
+#tree,
 #files td {
     background: rgba(32, 32, 32, 0.3) !important;
 }
@@ -19,7 +19,7 @@ html.light #files th {
 }
 html.light .logue,
 html.light #ops,
-html.light #treeul,
+html.light #tree,
 html.light #files td {
     background: rgba(248, 248, 248, 0.8) !important;
 }

docs/example.conf

@@ -1,3 +1,10 @@
+# append some arguments to the commandline;
+# the first space in a line counts as a separator,
+# any additional spaces are part of the value
+-e2dsa
+-e2ts
+-i 127.0.0.1
+
 # create users:
 # u username:password
 u ed:123
@@ -24,7 +31,8 @@ rw ed
 r k
 rw ed
 
-# this does the same thing:
+# this does the same thing,
+# and will cause an error on startup since /priv is already taken:
 ./priv
 /priv
 r ed k

docs/minimal-up2k.html

@@ -27,7 +27,7 @@
     #u2conf #u2btn, #u2btn {padding:1.5em 0}
 
     /* adjust the button area a bit */
-    #u2conf.has_btn {width: 35em !important; margin: 5em auto}
+    #u2conf.w, #u2conf.ww {width: 35em !important; margin: 5em auto}
 
     /* a */
     #op_up2k {min-height: 0}

docs/notes.sh

@@ -38,6 +38,13 @@ para() { for s in 1 2 3 4 5 6 7 8 12 16 24 32 48 64; do echo $s; for r in {1..4}
 avg() { awk 'function pr(ncsz) {if (nsmp>0) {printf "%3s %s\n", csz, sum/nsmp} csz=$1;sum=0;nsmp=0} {sub(/\r$/,"")} /^[0-9]+$/ {pr($1);next} / MiB/ {sub(/ MiB.*/,"");sub(/.* /,"");sum+=$1;nsmp++} END {pr(0)}' "$1"; }
 
 
+##
+## time between first and last upload
+
+python3 -um copyparty -nw -v srv::rw -i 127.0.0.1 2>&1 | tee log 
+cat log | awk '!/"purl"/{next} {s=$1;sub(/[^m]+m/,"");gsub(/:/," ");t=60*(60*$1+$2)+$3} !a{a=t;sa=s} {b=t;sb=s} END {print b-a,sa,sb}'
+
+
 ##
 ## bad filenames
 

scripts/copyparty-repack.sh

@@ -10,14 +10,41 @@ set -e
 #  (and those are usually linux so bash is good inaff)
 #   (but that said this even has macos support)
 #
-# bundle will look like:
+# output summary (filesizes and contents):
-# -rwxr-xr-x  0 ed ed  183808 Nov 19 00:43 copyparty
+#
-# -rw-r--r--  0 ed ed  491318 Nov 19 00:40 copyparty-extras/copyparty-0.5.4.tar.gz
+# 535672  copyparty-extras/sfx-full/copyparty-sfx.sh
-# -rwxr-xr-x  0 ed ed   30254 Nov 17 23:58 copyparty-extras/copyparty-fuse.py
+# 550760  copyparty-extras/sfx-full/copyparty-sfx.py
-# -rwxr-xr-x  0 ed ed  481403 Nov 19 00:40 copyparty-extras/sfx-full/copyparty-sfx.sh
+#           `- original unmodified sfx from github
-# -rwxr-xr-x  0 ed ed  506043 Nov 19 00:40 copyparty-extras/sfx-full/copyparty-sfx.py
+#
-# -rwxr-xr-x  0 ed ed  167699 Nov 19 00:43 copyparty-extras/sfx-lite/copyparty-sfx.sh
+# 572923  copyparty-extras/sfx-full/copyparty-sfx-gz.py
-# -rwxr-xr-x  0 ed ed  183808 Nov 19 00:43 copyparty-extras/sfx-lite/copyparty-sfx.py
+#           `- unmodified but recompressed from bzip2 to gzip
+#
+# 341792  copyparty-extras/sfx-ent/copyparty-sfx.sh
+# 353975  copyparty-extras/sfx-ent/copyparty-sfx.py
+# 376934  copyparty-extras/sfx-ent/copyparty-sfx-gz.py
+#           `- removed iOS ogg/opus/vorbis audio decoder,
+#              removed the audio tray mouse cursor,
+#              "enterprise edition"
+#
+# 259288  copyparty-extras/sfx-lite/copyparty-sfx.sh
+# 270004  copyparty-extras/sfx-lite/copyparty-sfx.py
+# 293159  copyparty-extras/sfx-lite/copyparty-sfx-gz.py
+#           `- also removed the codemirror markdown editor
+#              and the text-viewer syntax hilighting,
+#              only essential features remaining
+#
+# 646297  copyparty-extras/copyparty-1.0.14.tar.gz
+#   4823  copyparty-extras/copyparty-repack.sh
+#           `- source files from github
+#
+#  23663  copyparty-extras/up2k.py
+#           `- standalone utility to upload or search for files
+#
+#  32280  copyparty-extras/copyparty-fuse.py
+#           `- standalone to mount a URL as a local read-only filesystem
+#
+# 270004  copyparty
+#           `- minimal binary, same as sfx-lite/copyparty-sfx.py
 
 
 command -v gnutar && tar() { gnutar "$@"; }
@@ -54,6 +81,7 @@ cache="$od/.copyparty-repack.cache"
 		# fallback to awk (sorry)
 		awk -F\" '/"browser_download_url".*(\.tar\.gz|-sfx\.)/ {print$4}'
 	) |
+	grep -E '(sfx\.(sh|py)|tar\.gz)$' |
 	tee /dev/stderr |
 	tr -d '\r' | tr '\n' '\0' |
 	xargs -0 bash -c 'dl_files "$@"' _
@@ -64,7 +92,7 @@ cache="$od/.copyparty-repack.cache"
 
 # move src into copyparty-extras/,
 # move sfx into copyparty-extras/sfx-full/
-mkdir -p copyparty-extras/sfx-{full,lite}
+mkdir -p copyparty-extras/sfx-{full,ent,lite}
 mv copyparty-sfx.* copyparty-extras/sfx-full/
 mv copyparty-*.tar.gz copyparty-extras/
 
@@ -112,14 +140,17 @@ repack() {
 }
 
 repack sfx-full "re gz no-sh"
-repack sfx-lite "re no-ogv no-cm"
+repack sfx-ent  "re no-dd no-ogv"
-repack sfx-lite "re no-ogv no-cm gz no-sh"
+repack sfx-ent  "re no-dd no-ogv gz no-sh"
+repack sfx-lite "re no-dd no-ogv no-cm no-hl"
+repack sfx-lite "re no-dd no-ogv no-cm no-hl gz no-sh"
 
 
-# move fuse client into copyparty-extras/,
+# move fuse and up2k clients into copyparty-extras/,
 # copy lite-sfx.py to ./copyparty,
 # delete extracted source code
 ( cd copyparty-extras/
+mv copyparty-*/bin/up2k.py .
 mv copyparty-*/bin/copyparty-fuse.py .
 cp -pv sfx-lite/copyparty-sfx.py ../copyparty
 rm -rf copyparty-{0..9}*.*.*{0..9}

scripts/deps-docker/Dockerfile

@@ -45,6 +45,12 @@ RUN     mkdir -p /z/dist/no-pk \
         && tar -xf zopfli.tgz
 
 
+# todo
+# https://cdn.jsdelivr.net/gh/highlightjs/cdn-release/build/highlight.min.js
+# https://cdn.jsdelivr.net/gh/highlightjs/cdn-release/build/styles/default.min.css
+# https://prismjs.com/download.html#themes=prism-funky&languages=markup+css+clike+javascript+autohotkey+bash+basic+batch+c+csharp+cpp+cmake+diff+docker+go+ini+java+json+kotlin+latex+less+lisp+lua+makefile+objectivec+perl+powershell+python+r+jsx+ruby+rust+sass+scss+sql+swift+systemd+toml+typescript+vbnet+verilog+vhdl+yaml&plugins=line-highlight+line-numbers+autolinker
+
+
 # build fonttools (which needs zopfli)
 RUN     tar -xf zopfli.tgz \
         && cd zopfli* \

scripts/make-sfx.sh

@@ -22,6 +22,8 @@ help() { exec cat <<'EOF'
 # `no-cm` saves ~92k by removing easymde/codemirror
 #   (the fancy markdown editor)
 #
+# `no-hl` saves ~41k by removing syntax hilighting in the text viewer
+#
 # `no-fnt` saves ~9k by removing the source-code-pro font
 #   (browsers will try to use 'Consolas' instead)
 #
@@ -75,6 +77,7 @@ while [ ! -z "$1" ]; do
 		gz)     use_gz=1 ; ;;
 		no-ogv) no_ogv=1 ; ;;
 		no-fnt) no_fnt=1 ; ;;
+		no-hl)  no_hl=1  ; ;;
 		no-dd)  no_dd=1  ; ;;
 		no-cm)  no_cm=1  ; ;;
 		no-sh)  do_sh=   ; ;;
@@ -226,6 +229,9 @@ rm have
 	tmv "$f"
 }
 
+[ $no_hl ] &&
+	rm -rf copyparty/web/deps/prism*
+
 [ $no_fnt ] && {
 	rm -f copyparty/web/deps/scp.woff2
 	f=copyparty/web/ui.css

scripts/sfx.ls

@@ -57,6 +57,9 @@ copyparty/web/deps/ogv-demuxer-ogg-wasm.js,
 copyparty/web/deps/ogv-demuxer-ogg-wasm.wasm,
 copyparty/web/deps/ogv-worker-audio.js,
 copyparty/web/deps/ogv.js,
+copyparty/web/deps/prism.js,
+copyparty/web/deps/prism.css,
+copyparty/web/deps/prismd.css,
 copyparty/web/deps/scp.woff2,
 copyparty/web/deps/sha512.ac.js,
 copyparty/web/deps/sha512.hw.js,

tests/test_httpcli.py

@@ -47,11 +47,13 @@ class Cfg(Namespace):
             mtp=[],
             mte="a",
             mth="",
+            textfiles="",
             hist=None,
             no_idx=None,
             no_hash=None,
+            js_browser=None,
             css_browser=None,
-            **{k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr".split()}
+            **{k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr no_acode".split()}
         )
 
 

tests/test_vfs.py

@@ -17,7 +17,8 @@ from copyparty import util
 
 class Cfg(Namespace):
     def __init__(self, a=None, v=None, c=None):
-        ex = {k: False for k in "nw e2d e2ds e2dsa e2t e2ts e2tsr".split()}
+        ex = "nw e2d e2ds e2dsa e2t e2ts e2tsr no_logues no_readme no_acode"
+        ex = {k: False for k in ex.split()}
         ex2 = {
             "mtp": [],
             "mte": "a",
@@ -25,10 +26,9 @@ class Cfg(Namespace):
             "hist": None,
             "no_idx": None,
             "no_hash": None,
+            "js_browser": None,
             "css_browser": None,
             "no_voldump": True,
-            "no_logues": False,
-            "no_readme": False,
             "re_maxage": 0,
             "rproxy": 0,
         }

tests/util.py

@@ -113,6 +113,7 @@ class VSock(object):
 class VHttpSrv(object):
     def __init__(self):
         self.broker = NullBroker()
+        self.prism = None
 
         aliases = ["splash", "browser", "browser2", "msg", "md", "mde"]
         self.j2 = {x: J2_FILES for x in aliases}