v1.4.2

* terse upload responser
* randomize filenames

.gitignore

@@ -5,13 +5,16 @@ __pycache__/
 MANIFEST.in
 MANIFEST
 copyparty.egg-info/
-buildenv/
-build/
-dist/
-sfx/
-py2/
 .venv/
 
+/buildenv/
+/build/
+/dist/
+/py2/
+/sfx*
+/unt/
+/log/
+
 # ide
 *.sublime-workspace
 

README.md

@@ -9,11 +9,12 @@
 turn your phone or raspi into a portable file server with resumable uploads/downloads using *any* web browser
 
 * server only needs `py2.7` or `py3.3+`, all dependencies optional
-* browse/upload with IE4 / netscape4.0 on win3.11 (heh)
-* *resumable* uploads need `firefox 34+` / `chrome 41+` / `safari 7+` for full speed
-* code standard: `black`
+* browse/upload with [IE4](#browser-support) / netscape4.0 on win3.11 (heh)
+* *resumable* uploads need `firefox 34+` / `chrome 41+` / `safari 7+`
 
-📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [unpost](#unpost) // [thumbnails](#thumbnails) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [md-viewer](#markdown-viewer) // [ie4](#browser-support)
+try the **[read-only demo server](https://a.ocv.me/pub/demo/)** 👀 running from a basement in finland
+
+📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [unpost](#unpost) // [thumbnails](#thumbnails) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [md-viewer](#markdown-viewer)
 
 
 ## get the app
@@ -43,11 +44,12 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [tabs](#tabs) - the main tabs in the ui
     * [hotkeys](#hotkeys) - the browser has the following hotkeys
     * [navpane](#navpane) - switching between breadcrumbs or navpane
-    * [thumbnails](#thumbnails) - press `g` to toggle grid-view instead of the file listing
+    * [thumbnails](#thumbnails) - press `g` or `田` to toggle grid-view instead of the file listing
     * [zip downloads](#zip-downloads) - download folders (or file selections) as `zip` or `tar` files
     * [uploading](#uploading) - drag files/folders into the web-browser to upload
         * [file-search](#file-search) - dropping files into the browser also lets you see if they exist on the server
         * [unpost](#unpost) - undo/delete accidental uploads
+        * [self-destruct](#self-destruct) - uploads can be given a lifetime
     * [file manager](#file-manager) - cut/paste, rename, and delete files/folders (if you have permission)
     * [batch rename](#batch-rename) - select some files and press `F2` to bring up the rename UI
     * [markdown viewer](#markdown-viewer) - and there are *two* editors
@@ -55,9 +57,13 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [searching](#searching) - search by size, date, path/name, mp3-tags, ...
 * [server config](#server-config) - using arguments or config files, or a mix of both
     * [ftp-server](#ftp-server) - an FTP server can be started using `--ftp 3921`
-    * [file indexing](#file-indexing)
-    * [upload rules](#upload-rules) - set upload rules using volume flags
+    * [file indexing](#file-indexing) - enables dedup and music search ++
+        * [exclude-patterns](#exclude-patterns) - to save some time
+        * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
+        * [periodic rescan](#periodic-rescan) - filesystem monitoring
+    * [upload rules](#upload-rules) - set upload rules using volflags
     * [compress uploads](#compress-uploads) - files can be autocompressed on upload
+    * [other flags](#other-flags)
     * [database location](#database-location) - in-volume (`.hist/up2k.db`, default) or somewhere else
     * [metadata from audio files](#metadata-from-audio-files) - set `-e2t` to index tags on upload
     * [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags, also see [./bin/mtag/README.md](./bin/mtag/README.md)
@@ -101,7 +107,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 
 download **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py)** and you're all set!
 
-running the sfx without arguments (for example doubleclicking it on Windows) will give everyone read/write access to the current folder; see `-h` for help if you want [accounts and volumes](#accounts-and-volumes) etc
+running the sfx without arguments (for example doubleclicking it on Windows) will give everyone read/write access to the current folder; you may want [accounts and volumes](#accounts-and-volumes)
 
 some recommended options:
 * `-e2dsa` enables general [file indexing](#file-indexing)
@@ -109,7 +115,7 @@ some recommended options:
 * `-v /mnt/music:/music:r:rw,foo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, and read-write for user `foo`, password `bar`
   * replace `:r:rw,foo` with `:r,foo` to only make the folder readable by `foo` and nobody else
   * see [accounts and volumes](#accounts-and-volumes) for the syntax and other permissions (`r`ead, `w`rite, `m`ove, `d`elete, `g`et)
-* `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access
+* `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access (see `--help-ls`)
 
 
 ### on servers
@@ -163,11 +169,12 @@ feature summary
   * ☑ [up2k](#uploading): js, resumable, multithreaded
   * ☑ stash: simple PUT filedropper
   * ☑ [unpost](#unpost): undo/delete accidental uploads
+  * ☑ [self-destruct](#self-destruct) (specified server-side or client-side)
   * ☑ symlink/discard existing files (content-matching)
 * download
   * ☑ single files in browser
   * ☑ [folders as zip / tar files](#zip-downloads)
-  * ☑ FUSE client (read-only)
+  * ☑ [FUSE client](https://github.com/9001/copyparty/tree/hovudstraum/bin#copyparty-fusepy) (read-only)
 * browser
   * ☑ [navpane](#navpane) (directory tree sidebar)
   * ☑ file manager (cut/paste, delete, [batch-rename](#batch-rename))
@@ -203,6 +210,7 @@ project goals / philosophy
 * inverse linux philosophy -- do all the things, and do an *okay* job
   * quick drop-in service to get a lot of features in a pinch
   * there are probably [better alternatives](https://github.com/awesome-selfhosted/awesome-selfhosted) if you have specific/long-term needs
+    * but the resumable multithreaded uploads are p slick ngl
 * run anywhere, support everything
   * as many web-browsers and python versions as possible
     * every browser should at least be able to browse, download, upload files
@@ -233,7 +241,6 @@ some improvement ideas
 
 # bugs
 
-* Windows: python 3.7 and older cannot read tags with FFprobe, so use Mutagen or upgrade
 * Windows: python 2.7 cannot index non-ascii filenames with `-e2d`
 * Windows: python 2.7 cannot handle filenames with mojibake
 * `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions (macos, some linux)
@@ -241,15 +248,27 @@ some improvement ideas
 
 ## general bugs
 
-* Windows: if the up2k db is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
+* Windows: if the `up2k.db` (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
   * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
 * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
+* [the database can get stuck](https://github.com/9001/copyparty/issues/10)
+  * has only happened once but that is once too many
+  * luckily not dangerous for file integrity and doesn't really stop uploads or anything like that
+  * but would really appreciate some logs if anyone ever runs into it again
 * probably more, pls let me know
 
 ## not my bugs
 
 * [Chrome issue 1317069](https://bugs.chromium.org/p/chromium/issues/detail?id=1317069) -- if you try to upload a folder which contains symlinks by dragging it into the browser, the symlinked files will not get uploaded
 
+* [Chrome issue 1354816](https://bugs.chromium.org/p/chromium/issues/detail?id=1354816) -- chrome may eat all RAM uploading over plaintext http with `mt` enabled
+
+  * more amusingly, [Chrome issue 1354800](https://bugs.chromium.org/p/chromium/issues/detail?id=1354800) -- chrome may eat all RAM uploading in general (altho you probably won't run into this one)
+
+* [Chrome issue 1352210](https://bugs.chromium.org/p/chromium/issues/detail?id=1352210) -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive and likely to run into the above gc bugs)
+
+* [Firefox issue 1790500](https://bugzilla.mozilla.org/show_bug.cgi?id=1790500) -- sometimes forgets to close filedescriptors during upload so the browser can crash after ~4000 files
+
 * iPhones: the volume control doesn't work because [apple doesn't want it to](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html#//apple_ref/doc/uid/TP40009523-CH5-SW11)
   * *future workaround:* enable the equalizer, make it all-zero, and set a negative boost to reduce the volume
     * "future" because `AudioContext` is broken in the current iOS version (15.1), maybe one day...
@@ -263,6 +282,9 @@ some improvement ideas
 * VirtualBox: sqlite throws `Disk I/O Error` when running in a VM and the up2k database is in a vboxsf
   * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db inside the vm instead
 
+* Ubuntu: dragging files from certain folders into firefox or chrome is impossible
+  * due to snap security policies -- see `snap connections firefox` for the allowlist, `removable-media` permits all of `/mnt` and `/media` apparently
+
 
 # FAQ
 
@@ -273,7 +295,7 @@ some improvement ideas
   * you can also do this with linux filesystem permissions; `chmod 111 music` will make it possible to access files and folders inside the `music` folder but not list the immediate contents -- also works with other software, not just copyparty
 
 * can I make copyparty download a file to my server if I give it a URL?
-  * not officially, but there is a [terrible hack](https://github.com/9001/copyparty/blob/hovudstraum/bin/mtag/wget.py) which makes it possible
+  * not really, but there is a [terrible hack](https://github.com/9001/copyparty/blob/hovudstraum/bin/mtag/wget.py) which makes it possible
 
 
 # accounts and volumes
@@ -281,6 +303,8 @@ some improvement ideas
 per-folder, per-user permissions  - if your setup is getting complex, consider making a [config file](./docs/example.conf) instead of using arguments
 * much easier to manage, and you can modify the config at runtime with `systemctl reload copyparty` or more conveniently using the `[reload cfg]` button in the control-panel (if logged in as admin)
 
+a quick summary can be seen using `--help-accounts`
+
 configuring accounts/volumes with arguments:
 * `-a usr:pwd` adds account `usr` with password `pwd`
 * `-v .::r` adds current-folder `.` as the webroot, `r`eadable by anyone
@@ -305,16 +329,18 @@ examples:
   * `u1` can open the `inc` folder, but cannot see the contents, only upload new files to it
   * `u2` can browse it and move files *from* `/inc` into any folder where `u2` has write-access
 * make folder `/mnt/ss` available at `/i`, read-write for u1, get-only for everyone else, and enable accesskeys: `-v /mnt/ss:i:rw,u1:g:c,fk=4`
-  * `c,fk=4` sets the `fk` volume-flag to 4, meaning each file gets a 4-character accesskey
+  * `c,fk=4` sets the `fk` volflag to 4, meaning each file gets a 4-character accesskey
   * `u1` can upload files, browse the folder, and see the generated accesskeys
   * other users cannot browse the folder, but can access the files if they have the full file URL with the accesskey
 
+anyone trying to bruteforce a password gets banned according to `--ban-pw`; default is 24h ban for 9 failed attempts in 1 hour
+
 
 # the browser
 
 accessing a copyparty server using a web-browser
 
-![copyparty-browser-fs8](https://user-images.githubusercontent.com/241032/129635359-d6dd9b07-8079-4020-ad77-2bfdb9ebd8d5.png)
+![copyparty-browser-fs8](https://user-images.githubusercontent.com/241032/192042695-522b3ec7-6845-494a-abdb-d1c0d0e23801.png)
 
 
 ## tabs
@@ -337,7 +363,7 @@ the browser has the following hotkeys  (always qwerty)
 * `I/K` prev/next folder
 * `M` parent folder (or unexpand current)
 * `V` toggle folders / textfiles in the navpane
-* `G` toggle list / [grid view](#thumbnails)
+* `G` toggle list / [grid view](#thumbnails) -- same as `田` bottom-right
 * `T` toggle thumbnails / icons
 * `ESC` close various things
 * `ctrl-X` cut selected files/folders
@@ -358,19 +384,24 @@ the browser has the following hotkeys  (always qwerty)
   * `U/O` skip 10sec back/forward
   * `0..9` jump to 0%..90%
   * `P` play/pause (also starts playing the folder)
+  * `Y` download file
 * when viewing images / playing videos:
   * `J/L, Left/Right` prev/next file
   * `Home/End` first/last file
+  * `F` toggle fullscreen
   * `S` toggle selection
   * `R` rotate clockwise (shift=ccw)
+  * `Y` download file
   * `Esc` close viewer
   * videos:
     * `U/O` skip 10sec back/forward
+    * `0..9` jump to 0%..90%
     * `P/K/Space` play/pause
-    * `F` fullscreen
-    * `C` continue playing next video
-    * `V` loop
     * `M` mute
+    * `C` continue playing next video
+    * `V` loop entire file
+    * `[` loop range (start)
+    * `]` loop range (end)
 * when the navpane is open:
   * `A/D` adjust tree width
 * in the [grid view](#thumbnails):
@@ -402,7 +433,7 @@ click the `🌲` or pressing the `B` hotkey to toggle between breadcrumbs path (
 
 ## thumbnails
 
-press `g` to toggle grid-view instead of the file listing,  and `t` toggles icons / thumbnails
+press `g` or `田` to toggle grid-view instead of the file listing  and `t` toggles icons / thumbnails
 
 ![copyparty-thumbs-fs8](https://user-images.githubusercontent.com/241032/129636211-abd20fa2-a953-4366-9423-1c88ebb96ba9.png)
 
@@ -444,13 +475,13 @@ you can also zip a selection of files or folders by clicking them in the browser
 
 ## uploading
 
-drag files/folders into the web-browser to upload
+drag files/folders into the web-browser to upload  (or use the [command-line uploader](https://github.com/9001/copyparty/tree/hovudstraum/bin#up2kpy))
 
 this initiates an upload using `up2k`; there are two uploaders available:
 * `[🎈] bup`, the basic uploader, supports almost every browser since netscape 4.0
-* `[🚀] up2k`, the fancy one
+* `[🚀] up2k`, the good / fancy one
 
-you can also undo/delete uploads by using `[🧯]` [unpost](#unpost)
+NB: you can undo/delete your own uploads with `[🧯]` [unpost](#unpost)
 
 up2k has several advantages:
 * you can drop folders into the browser (files are added recursively)
@@ -462,19 +493,19 @@ up2k has several advantages:
 * much higher speeds than ftp/scp/tarpipe on some internet connections (mainly american ones) thanks to parallel connections
 * the last-modified timestamp of the file is preserved
 
-see [up2k](#up2k) for details on how it works
+see [up2k](#up2k) for details on how it works, or watch a [demo video](https://a.ocv.me/pub/demo/pics-vids/#gf-0f6f5c0d)
 
 ![copyparty-upload-fs8](https://user-images.githubusercontent.com/241032/129635371-48fc54ca-fa91-48e3-9b1d-ba413e4b68cb.png)
 
 **protip:** you can avoid scaring away users with [contrib/plugins/minimal-up2k.html](contrib/plugins/minimal-up2k.html) which makes it look [much simpler](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
 
-**protip:** if you enable `favicon` in the `[⚙️] settings` tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress
+**protip:** if you enable `favicon` in the `[⚙️] settings` tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress -- also, the `[🔔]` and/or `[🔊]` switches enable visible and/or audible notifications on upload completion
 
 the up2k UI is the epitome of polished inutitive experiences:
 * "parallel uploads" specifies how many chunks to upload at the same time
 * `[🏃]` analysis of other files should continue while one is uploading
+* `[🥔]` shows a simpler UI for faster uploads from slow devices
 * `[💭]` ask for confirmation before files are added to the queue
-* `[💤]` sync uploading between other copyparty browser-tabs so only one is active
 * `[🔎]` switch between upload and [file-search](#file-search) mode
   * ignore `[🔎]` if you add files by dragging them into the browser
 
@@ -486,7 +517,7 @@ and then theres the tabs below it,
   * plus up to 3 entries each from `[done]` and `[que]` for context
 * `[que]` is all the files that are still queued
 
-note that since up2k has to read each file twice, `[🎈 bup]` can *theoretically* be up to 2x faster in some extreme cases (files bigger than your ram, combined with an internet connection faster than the read-speed of your HDD, or if you're uploading from a cuo2duo)
+note that since up2k has to read each file twice, `[🎈] bup` can *theoretically* be up to 2x faster in some extreme cases (files bigger than your ram, combined with an internet connection faster than the read-speed of your HDD, or if you're uploading from a cuo2duo)
 
 if you are resuming a massive upload and want to skip hashing the files which already finished, you can enable `turbo` in the `[⚙️] config` tab, but please read the tooltip on that button
 
@@ -516,6 +547,17 @@ undo/delete accidental uploads
 you can unpost even if you don't have regular move/delete access, however only for files uploaded within the past `--unpost` seconds (default 12 hours) and the server must be running with `-e2d`
 
 
+### self-destruct
+
+uploads can be given a lifetime,  afer which they expire / self-destruct
+
+the feature must be enabled per-volume with the `lifetime` [upload rule](#upload-rules) which sets the upper limit for how long a file gets to stay on the server
+
+clients can specify a shorter expiration time using the [up2k ui](#uploading) -- the relevant options become visible upon navigating into a folder with `lifetimes` enabled -- or by using the `life` [upload modifier](#write)
+
+specifying a custom expiration time client-side will affect the timespan in which unposts are permitted, so keep an eye on the estimates in the up2k ui
+
+
 ## file manager
 
 cut/paste, rename, and delete files/folders (if you have permission)
@@ -597,7 +639,7 @@ and there are *two* editors
 
 * get a plaintext file listing by adding `?ls=t` to a URL, or a compact colored one with `?ls=v` (for unix terminals)
 
-* if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider https://ocv.me/dev/?media-osd-bgone.ps1
+* if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider [./contrib/media-osd-bgone.ps1](contrib/#media-osd-bgoneps1)
 
 * click the bottom-left `π` to open a javascript prompt for debugging
 
@@ -620,7 +662,9 @@ path/name queries are space-separated, AND'ed together, and words are negated wi
 * path: `shibayan -bossa` finds all files where one of the folders contain `shibayan` but filters out any results where `bossa` exists somewhere in the path
 * name: `demetori styx` gives you [good stuff](https://www.youtube.com/watch?v=zGh0g14ZJ8I&list=PL3A147BD151EE5218&index=9)
 
-add the argument `-e2ts` to also scan/index tags from music files, which brings us over to:
+the `raw` field allows for more complex stuff such as `( tags like *nhato* or tags like *taishi* ) and ( not tags like *nhato* or not tags like *taishi* )` which finds all songs by either nhato or taishi, excluding collabs (terrible example, why would you do that)
+
+for the above example to work, add the commandline argument `-e2ts` to also scan/index tags from music files, which brings us over to:
 
 
 # server config
@@ -645,7 +689,9 @@ an FTP server can be started using `--ftp 3921`,  and/or `--ftps` for explicit T
 
 ## file indexing
 
-file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volume flags, or a mix of both.
+enables dedup and music search ++
+
+file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volflags, or a mix of both.
 
 through arguments:
 * `-e2d` enables file indexing on upload
@@ -654,8 +700,11 @@ through arguments:
 * `-e2t` enables metadata indexing on upload
 * `-e2ts` also scans for tags in all files that don't have tags yet
 * `-e2tsr` also deletes all existing tags, doing a full reindex
+* `-e2v` verfies file integrity at startup, comparing hashes from the db
+* `-e2vu` patches the database with the new hashes from the filesystem
+* `-e2vp` panics and kills copyparty instead
 
-the same arguments can be set as volume flags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts` for disabling:
+the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts`, `d2v` for disabling:
 * `-v ~/music::r:c,e2dsa,e2tsr` does a full reindex of everything on startup
 * `-v ~/music::r:c,d2d` disables **all** indexing, even if any `-e2*` are on
 * `-v ~/music::r:c,d2t` disables all `-e2t*` (tags), does not affect `-e2d*`
@@ -666,8 +715,11 @@ note:
 * the parser can finally handle `c,e2dsa,e2tsr` so you no longer have to `c,e2dsa:c,e2tsr`
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
+* deduplication is possible on windows if you run copyparty as administrator (not saying you should!)
 
-to save some time, you can provide a regex pattern for filepaths to only index by filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash \.iso$` or the volume-flag `:c,nohash=\.iso$`, this has the following consequences:
+### exclude-patterns
+
+to save some time,  you can provide a regex pattern for filepaths to only index by filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash \.iso$` or the volflag `:c,nohash=\.iso$`, this has the following consequences:
 * initial indexing is way faster, especially when the volume is on a network disk
 * makes it impossible to [file-search](#file-search)
 * if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected
@@ -676,12 +728,29 @@ similarly, you can fully ignore files/folders using `--no-idx [...]` and `:c,noi
 
 if you set `--no-hash [...]` globally, you can enable hashing for specific volumes using flag `:c,nohash=`
 
+### filesystem guards
+
+avoid traversing into other filesystems  using `--xdev` / volflag `:c,xdev`, skipping any symlinks or bind-mounts to another HDD for example
+
+and/or you can `--xvol` / `:c,xvol` to ignore all symlinks leaving the volume's top directory, but still allow bind-mounts pointing elsewhere
+
+**NB: only affects the indexer** -- users can still access anything inside a volume, unless shadowed by another volume
+
+### periodic rescan
+
+filesystem monitoring;  if copyparty is not the only software doing stuff on your filesystem, you may want to enable periodic rescans to keep the index up to date
+
+argument `--re-maxage 60` will rescan all volumes every 60 sec, same as volflag `:c,scan=60` to specify it per-volume
+
+uploads are disabled while a rescan is happening, so rescans will be delayed by `--db-act` (default 10 sec) when there is write-activity going on (uploads, renames, ...)
+
 
 ## upload rules
 
-set upload rules using volume flags,  some examples:
+set upload rules using volflags,  some examples:
 
 * `:c,sz=1k-3m` sets allowed filesize between 1 KiB and 3 MiB inclusive (suffixes: `b`, `k`, `m`, `g`)
+* `:c,df=4g` block uploads if there would be less than 4 GiB free disk space afterwards
 * `:c,nosub` disallow uploading into subdirectories; goes well with `rotn` and `rotf`:
 * `:c,rotn=1000,2` moves uploads into subfolders, up to 1000 files in each folder before making a new one, two levels deep (must be at least 1)
 * `:c,rotf=%Y/%m/%d/%H` enforces files to be uploaded into a structure of subfolders according to that date format
@@ -700,16 +769,16 @@ you can also set transaction limits which apply per-IP and per-volume, but these
 
 files can be autocompressed on upload,  either on user-request (if config allows) or forced by server-config
 
-* volume flag `gz` allows gz compression
-* volume flag `xz` allows lzma compression
-* volume flag `pk` **forces** compression on all files
+* volflag `gz` allows gz compression
+* volflag `xz` allows lzma compression
+* volflag `pk` **forces** compression on all files
 * url parameter `pk` requests compression with server-default algorithm
 * url parameter `gz` or `xz` requests compression with a specific algorithm
 * url parameter `xz` requests xz compression
 
 things to note,
 * the `gz` and `xz` arguments take a single optional argument, the compression level (range 0 to 9)
-* the `pk` volume flag takes the optional argument `ALGORITHM,LEVEL` which will then be forced for all uploads, for example `gz,9` or `xz,0`
+* the `pk` volflag takes the optional argument `ALGORITHM,LEVEL` which will then be forced for all uploads, for example `gz,9` or `xz,0`
 * default compression is gzip level 9
 * all upload methods except up2k are supported
 * the files will be indexed after compression, so dupe-detection and file-search will not work as expected
@@ -723,13 +792,18 @@ some examples,
   allows (but does not force) gz compression if client uploads to `/inc?pk` or `/inc?gz` or `/inc?gz=4`
 
 
+## other flags
+
+* `:c,magic` enables filetype detection for nameless uploads, same as `--magic`
+
+
 ## database location
 
 in-volume (`.hist/up2k.db`, default) or somewhere else
 
 copyparty creates a subfolder named `.hist` inside each volume where it stores the database, thumbnails, and some other stuff
 
-this can instead be kept in a single place using the `--hist` argument, or the `hist=` volume flag, or a mix of both:
+this can instead be kept in a single place using the `--hist` argument, or the `hist=` volflag, or a mix of both:
 * `--hist ~/.cache/copyparty -v ~/music::r:c,hist=-` sets `~/.cache/copyparty` as the default place to put volume info, but `~/music` gets the regular `.hist` subfolder (`-` restores default behavior)
 
 note:
@@ -762,21 +836,31 @@ see the beautiful mess of a dictionary in [mtag.py](https://github.com/9001/copy
 * avoids pulling any GPL code into copyparty
 * more importantly runs FFprobe on incoming files which is bad if your FFmpeg has a cve
 
+`--mtag-to` sets the tag-scan timeout; very high default (60 sec) to cater for zfs and other randomly-freezing filesystems. Lower values like 10 are usually safe, allowing for faster processing of tricky files
+
 
 ## file parser plugins
 
 provide custom parsers to index additional tags, also see [./bin/mtag/README.md](./bin/mtag/README.md)
 
-copyparty can invoke external programs to collect additional metadata for files using `mtp` (either as argument or volume flag), there is a default timeout of 30sec
+copyparty can invoke external programs to collect additional metadata for files using `mtp` (either as argument or volflag), there is a default timeout of 60sec, and only files which contain audio get analyzed by default (see ay/an/ad below)
 
 * `-mtp .bpm=~/bin/audio-bpm.py` will execute `~/bin/audio-bpm.py` with the audio file as argument 1 to provide the `.bpm` tag, if that does not exist in the audio metadata
 * `-mtp key=f,t5,~/bin/audio-key.py` uses `~/bin/audio-key.py` to get the `key` tag, replacing any existing metadata tag (`f,`), aborting if it takes longer than 5sec (`t5,`)
 * `-v ~/music::r:c,mtp=.bpm=~/bin/audio-bpm.py:c,mtp=key=f,t5,~/bin/audio-key.py` both as a per-volume config wow this is getting ugly
 
-*but wait, there's more!* `-mtp` can be used for non-audio files as well using the `a` flag: `ay` only do audio files, `an` only do non-audio files, or `ad` do all files (d as in dontcare) 
+*but wait, there's more!* `-mtp` can be used for non-audio files as well using the `a` flag: `ay` only do audio files (default), `an` only do non-audio files, or `ad` do all files (d as in dontcare)
 
+* "audio file" also means videos btw, as long as there is an audio stream
 * `-mtp ext=an,~/bin/file-ext.py` runs `~/bin/file-ext.py` to get the `ext` tag only if file is not audio (`an`)
 * `-mtp arch,built,ver,orig=an,eexe,edll,~/bin/exe.py` runs `~/bin/exe.py` to get properties about windows-binaries only if file is not audio (`an`) and file extension is exe or dll
+* if you want to daisychain parsers, use the `p` flag to set processing order
+  * `-mtp foo=p1,~/a.py` runs before `-mtp foo=p2,~/b.py` and will forward all the tags detected so far as json to the stdin of b.py
+* option `c0` disables capturing of stdout/stderr, so copyparty will not receive any tags from the process at all -- instead the invoked program is free to print whatever to the console, just using copyparty as a launcher
+  * `c1` captures stdout only, `c2` only stderr, and `c3` (default) captures both
+* you can control how the parser is killed if it times out with option `kt` killing the entire process tree (default), `km` just the main process, or `kn` let it continue running until copyparty is terminated
+
+if something doesn't work, try `--mtag-v` for verbose error messages
 
 
 ## upload events
@@ -784,16 +868,16 @@ copyparty can invoke external programs to collect additional metadata for files
 trigger a script/program on each upload  like so:
 
 ```
--v /mnt/inc:inc:w:c,mte=+a1:c,mtp=a1=ad,/usr/bin/notify-send
+-v /mnt/inc:inc:w:c,mte=+x1:c,mtp=x1=ad,kn,/usr/bin/notify-send
 ```
 
-so filesystem location `/mnt/inc` shared at `/inc`, write-only for everyone, appending `a1` to the list of tags to index, and using `/usr/bin/notify-send` to "provide" that tag
+so filesystem location `/mnt/inc` shared at `/inc`, write-only for everyone, appending `x1` to the list of tags to index (`mte`), and using `/usr/bin/notify-send` to "provide" tag `x1` for any filetype (`ad`) with kill-on-timeout disabled (`kn`)
 
 that'll run the command `notify-send` with the path to the uploaded file as the first and only argument (so on linux it'll show a notification on-screen)
 
 note that it will only trigger on new unique files, not dupes
 
-and it will occupy the parsing threads, so fork anything expensive, or if you want to intentionally queue/singlethread you can combine it with `--mtag-mt 1`
+and it will occupy the parsing threads, so fork anything expensive (or set `kn` to have copyparty fork it for you) -- otoh if you want to intentionally queue/singlethread you can combine it with `--mtag-mt 1`
 
 if this becomes popular maybe there should be a less janky way to do it actually
 
@@ -803,8 +887,8 @@ if this becomes popular maybe there should be a less janky way to do it actually
 tell search engines you dont wanna be indexed,  either using the good old [robots.txt](https://www.robotstxt.org/robotstxt.html) or through copyparty settings:
 
 * `--no-robots` adds HTTP (`X-Robots-Tag`) and HTML (`<meta>`) headers with `noindex, nofollow` globally
-* volume-flag `[...]:c,norobots` does the same thing for that single volume
-* volume-flag `[...]:c,robots` ALLOWS search-engine crawling for that volume, even if `--no-robots` is set globally
+* volflag `[...]:c,norobots` does the same thing for that single volume
+* volflag `[...]:c,robots` ALLOWS search-engine crawling for that volume, even if `--no-robots` is set globally
 
 also, `--force-js` disables the plain HTML folder listing, making things harder to parse for search engines
 
@@ -834,8 +918,17 @@ see the top of [./copyparty/web/browser.css](./copyparty/web/browser.css) where
 
 ## complete examples
 
-* read-only music server with bpm and key scanning  
-  `python copyparty-sfx.py -v /mnt/nas/music:/music:r -e2dsa -e2ts -mtp .bpm=f,audio-bpm.py -mtp key=f,audio-key.py`
+* read-only music server  
+  `python copyparty-sfx.py -v /mnt/nas/music:/music:r -e2dsa -e2ts --no-robots --force-js --theme 2`
+  
+  * ...with bpm and key scanning  
+    `-mtp .bpm=f,audio-bpm.py -mtp key=f,audio-key.py`
+  
+  * ...with a read-write folder for `kevin` whose password is `okgo`  
+    `-a kevin:okgo -v /mnt/nas/inc:/inc:rw,kevin`
+  
+  * ...with logging to disk  
+    `-lo log/cpp-%Y-%m%d-%H%M%S.txt.xz`
 
 
 # browser support
@@ -882,6 +975,7 @@ quick summary of more eccentric web-browsers trying to view a directory index:
 | **netsurf** (3.10/arch)   | is basically ie6 with much better css (javascript has almost no effect) | 
 | **opera** (11.60/winxp)   | OK: thumbnails, image-viewer, zip-selection, rename/cut/paste. NG: up2k, navpane, markdown, audio |
 | **ie4** and **netscape** 4.0  | can browse, upload with `?b=u`, auth with `&pw=wark` |
+| **ncsa mosaic** 2.7       | does not get a pass, [pic1](https://user-images.githubusercontent.com/241032/174189227-ae816026-cf6f-4be5-a26e-1b3b072c1b2f.png) - [pic2](https://user-images.githubusercontent.com/241032/174189225-5651c059-5152-46e9-ac26-7e98e497901b.png) |
 | **SerenityOS** (7e98457)  | hits a page fault, works with `?b=u`, file upload not-impl |
 
 
@@ -894,7 +988,9 @@ interact with copyparty using non-browser clients
   * `var xhr = new XMLHttpRequest(); xhr.open('POST', '//127.0.0.1:3923/msgs?raw'); xhr.send('foo');`
 
 * curl/wget: upload some files (post=file, chunk=stdin)
-  * `post(){ curl -b cppwd=wark -F act=bput -F f=@"$1" http://127.0.0.1:3923/;}`  
+  * `post(){ curl -F act=bput -F f=@"$1" http://127.0.0.1:3923/?pw=wark;}`  
+    `post movie.mkv`
+  * `post(){ curl -b cppwd=wark -H rand:8 -T "$1" http://127.0.0.1:3923/;}`  
     `post movie.mkv`
   * `post(){ wget --header='Cookie: cppwd=wark' --post-file="$1" -O- http://127.0.0.1:3923/?raw;}`  
     `post movie.mkv`
@@ -920,7 +1016,9 @@ copyparty returns a truncated sha512sum of your PUT/POST as base64; you can gene
     b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|tr '+/' '-_'|head -c44;}
     b512 <movie.mkv
 
-you can provide passwords using cookie `cppwd=hunter2`, as a url query `?pw=hunter2`, or with basic-authentication (either as the username or password)
+you can provide passwords using cookie `cppwd=hunter2`, as a url-param `?pw=hunter2`, or with basic-authentication (either as the username or password)
+
+NOTE: curl will not send the original filename if you use `-T` combined with url-params! Also, make sure to always leave a trailing slash in URLs unless you want to override the filename
 
 
 # up2k
@@ -938,19 +1036,33 @@ quick outline of the up2k protocol, see [uploading](#uploading) for the web-clie
   * server writes chunks into place based on the hash
 * client does another handshake with the hashlist; server replies with OK or a list of chunks to reupload
 
-up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well (thanks to tls also functioning as an integrity check)
+up2k has saved a few uploads from becoming corrupted in-transfer already;
+* caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well (thanks to tls also functioning as an integrity check)
+* also stopped someone from uploading because their ram was bad
+
+regarding the frequent server log message during uploads;  
+`6.0M 106M/s 2.77G 102.9M/s n948 thank 4/0/3/1 10042/7198 00:01:09`
+* this chunk was `6 MiB`, uploaded at `106 MiB/s`
+* on this http connection, `2.77 GiB` transferred, `102.9 MiB/s` average, `948` chunks handled
+* client says `4` uploads OK, `0` failed, `3` busy, `1` queued, `10042 MiB` total size, `7198 MiB` and `00:01:09` left
 
 
 ## why chunk-hashes
 
 a single sha512 would be better, right?
 
-this is due to `crypto.subtle` not providing a streaming api (or the option to seed the sha512 hasher with a starting hash)
+this is due to `crypto.subtle` [not yet](https://github.com/w3c/webcrypto/issues/73) providing a streaming api (or the option to seed the sha512 hasher with a starting hash)
 
 as a result, the hashes are much less useful than they could have been (search the server by sha512, provide the sha512 in the response http headers, ...)
 
+however it allows for hashing multiple chunks in parallel, greatly increasing upload speed from fast storage (NVMe, raid-0 and such)
+
+* both the [browser uploader](#uploading) and the [commandline one](https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py) does this now, allowing for fast uploading even from plaintext http
+
 hashwasm would solve the streaming issue but reduces hashing speed for sha512 (xxh128 does 6 GiB/s), and it would make old browsers and [iphones](https://bugs.webkit.org/show_bug.cgi?id=228552) unsupported
 
+* blake2 might be a better choice since xxh is non-cryptographic, but that gets ~15 MiB/s on slower androids
+
 
 # performance
 
@@ -980,6 +1092,7 @@ when uploading files,
 
 * if you're cpu-bottlenecked, or the browser is maxing a cpu core:
   * up to 30% faster uploads if you hide the upload status list by switching away from the `[🚀]` up2k ui-tab (or closing it)
+    * optionally you can switch to the lightweight potato ui by clicking the `[🥔]`
     * switching to another browser-tab also works, the favicon will update every 10 seconds in that case
   * unlikely to be a problem, but can happen when uploding many small files, or your internet is too fast, or PC too slow
 
@@ -988,16 +1101,30 @@ when uploading files,
 
 some notes on hardening
 
-on public copyparty instances with anonymous upload enabled:
+* option `-s` is a shortcut to set the following options:
+  * `--no-thumb` disables thumbnails and audio transcoding to stop copyparty from running `FFmpeg`/`Pillow`/`VIPS` on uploaded files, which is a [good idea](https://www.cvedetails.com/vulnerability-list.php?vendor_id=3611) if anonymous upload is enabled
+  * `--no-mtag-ff` uses `mutagen` to grab music tags instead of `FFmpeg`, which is safer and faster but less accurate
+  * `--dotpart` hides uploads from directory listings while they're still incoming
+  * `--no-robots` and `--force-js` makes life harder for crawlers, see [hiding from google](#hiding-from-google)
 
-* users can upload html/css/js which will evaluate for other visitors in a few ways,
-  * unless `--no-readme` is set: by uploading/modifying a file named `readme.md`
-  * if `move` access is granted AND none of `--no-logues`, `--no-dot-mv`, `--no-dot-ren` is set: by uploading some .html file and renaming it to `.epilogue.html` (uploading it directly is blocked)
+* option `-ss` is a shortcut for the above plus:
+  * `--no-logues` and `--no-readme` disables support for readme's and prologues / epilogues in directory listings, which otherwise lets people upload arbitrary `<script>` tags
+  * `--unpost 0`, `--no-del`, `--no-mv` disables all move/delete support
+  * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
+    * however note if you edit one file it will also affect the other copies
+  * `--vague-403` returns a "404 not found" instead of "403 forbidden" which is a common enterprise meme
+  * `--ban-404=50,60,1440` ban client for 1440min (24h) if they hit 50 404's in 60min
+    * **NB:** will ban anyone who enables up2k turbo
+  * `--nih` removes the server hostname from directory listings
 
-other misc:
+* option `-sss` is a shortcut for the above plus:
+  * `-lo cpp-%Y-%m%d-%H%M%S.txt.xz` enables logging to disk
+  * `-ls **,*,ln,p,r` does a scan on startup for any dangerous symlinks
+
+other misc notes:
 
 * you can disable directory listings by giving permission `g` instead of `r`, only accepting direct URLs to files
-  * combine this with volume-flag `c,fk` to generate per-file accesskeys; users which have full read-access will then see URLs with `?k=...` appended to the end, and `g` users must provide that URL including the correct key to avoid a 404
+  * combine this with volflag `c,fk` to generate per-file accesskeys; users which have full read-access will then see URLs with `?k=...` appended to the end, and `g` users must provide that URL including the correct key to avoid a 404
 
 
 ## gotchas
@@ -1088,7 +1215,17 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
 | uPOST | | `msg=foo` | send message `foo` into server log |
 | mPOST | | `act=tput`, `body=TEXT` | overwrite markdown document at URL |
 
-server behavior of `msg` can be reconfigured with `--urlform`
+upload modifiers:
+
+| http-header | url-param | effect |
+|--|--|--|
+| `Accept: url` | `want=url` | return just the file URL |
+| `Rand: 4` | `rand=4` | generate random filename with 4 characters |
+| `Life: 30` | `life=30` | delete file after 30 seconds |
+
+* `life` only has an effect if the volume has a lifetime, and the volume lifetime must be greater than the file's
+
+* server behavior of `msg` can be reconfigured with `--urlform`
 
 ## admin
 
@@ -1190,11 +1327,15 @@ if you want thumbnails, `apt -y install ffmpeg`
 
 ideas for context to include in bug reports
 
+in general, commandline arguments (and config file if any)
+
 if something broke during an upload (replacing FILENAME with a part of the filename that broke):
 ```
 journalctl -aS '48 hour ago' -u copyparty | grep -C10 FILENAME | tee bug.log
 ```
 
+if there's a wall of base64 in the log (thread stacks) then please include that, especially if you run into something freezing up or getting stuck, for example `OperationalError('database is locked')` -- alternatively you can visit `/?stack` to see the stacks live, so http://127.0.0.1:3923/?stack for example
+
 
 # building
 
@@ -1242,10 +1383,7 @@ also builds the sfx so skip the sfx section above
 in the `scripts` folder:
 
 * run `make -C deps-docker` to build all dependencies
-* `git tag v1.2.3 && git push origin --tags`
-* upload to pypi with `make-pypi-release.(sh|bat)`
-* create github release with `make-tgz-release.sh`
-* create sfx with `make-sfx.sh`
+* run `./rls.sh 1.2.3` which uploads to pypi + creates github release + sfx
 
 
 # todo
@@ -1272,7 +1410,7 @@ roughly sorted by priority
 * up2k partials ui
   * feels like there isn't much point
 * cache sha512 chunks on client
-  * too dangerous
+  * too dangerous -- overtaken by turbo mode
 * comment field
   * nah
 * look into android thumbnail cache file format

bin/mtag/README.md

@@ -42,7 +42,7 @@ run [`install-deps.sh`](install-deps.sh) to build/install most dependencies requ
 * `mtp` modules will not run if a file has existing tags in the db, so clear out the tags with `-e2tsr` the first time you launch with new `mtp` options
 
 
-## usage with volume-flags
+## usage with volflags
 
 instead of affecting all volumes, you can set the options for just one volume like so:
 

bin/mtag/cksum.py

@@ -17,7 +17,7 @@ except:
 
 """
 calculates various checksums for uploads,
-usage: -mtp crc32,md5,sha1,sha256b=bin/mtag/cksum.py
+usage: -mtp crc32,md5,sha1,sha256b=ad,bin/mtag/cksum.py
 """
 
 

bin/mtag/guestbook-read.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+
+"""
+fetch latest msg from guestbook and return as tag
+
+example copyparty config to use this:
+  --urlform save,get -vsrv/hello:hello:w:c,e2ts,mtp=guestbook=t10,ad,p,bin/mtag/guestbook-read.py:mte=+guestbook
+
+explained:
+  for realpath srv/hello (served at /hello), write-only for eveyrone,
+  enable file analysis on upload (e2ts),
+  use mtp plugin "bin/mtag/guestbook-read.py" to provide metadata tag "guestbook",
+  do this on all uploads regardless of extension,
+  t10 = 10 seconds timeout for each dwonload,
+  ad = parse file regardless if FFmpeg thinks it is audio or not
+  p = request upload info as json on stdin (need ip)
+  mte=+guestbook enabled indexing of that tag for this volume
+
+PS: this requires e2ts to be functional,
+  meaning you need to do at least one of these:
+   * apt install ffmpeg
+   * pip3 install mutagen
+"""
+
+
+import json
+import os
+import sqlite3
+import sys
+
+
+# set 0 to allow infinite msgs from one IP,
+# other values delete older messages to make space,
+# so 1 only keeps latest msg
+NUM_MSGS_TO_KEEP = 1
+
+
+def main():
+    fp = os.path.abspath(sys.argv[1])
+    fdir = os.path.dirname(fp)
+
+    zb = sys.stdin.buffer.read()
+    zs = zb.decode("utf-8", "replace")
+    md = json.loads(zs)
+
+    ip = md["up_ip"]
+
+    # can put the database inside `fdir` if you'd like,
+    # by default it saves to PWD:
+    # os.chdir(fdir)
+
+    db = sqlite3.connect("guestbook.db3")
+    with db:
+        t = "select msg from gb where ip = ? order by ts desc"
+        r = db.execute(t, (ip,)).fetchone()
+        if r:
+            print(r[0])
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/guestbook.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+
+"""
+store messages from users in an sqlite database
+which can be read from another mtp for example
+
+takes input from application/x-www-form-urlencoded POSTs,
+for example using the message/pager function on the website
+
+example copyparty config to use this:
+  --urlform save,get -vsrv/hello:hello:w:c,e2ts,mtp=xgb=ebin,t10,ad,p,bin/mtag/guestbook.py:mte=+xgb
+
+explained:
+  for realpath srv/hello (served at /hello),write-only for eveyrone,
+  enable file analysis on upload (e2ts),
+  use mtp plugin "bin/mtag/guestbook.py" to provide metadata tag "xgb",
+  do this on all uploads with the file extension "bin",
+  t300 = 300 seconds timeout for each dwonload,
+  ad = parse file regardless if FFmpeg thinks it is audio or not
+  p = request upload info as json on stdin
+  mte=+xgb enabled indexing of that tag for this volume
+
+PS: this requires e2ts to be functional,
+  meaning you need to do at least one of these:
+   * apt install ffmpeg
+   * pip3 install mutagen
+"""
+
+
+import json
+import os
+import sqlite3
+import sys
+from urllib.parse import unquote_to_bytes as unquote
+
+
+# set 0 to allow infinite msgs from one IP,
+# other values delete older messages to make space,
+# so 1 only keeps latest msg
+NUM_MSGS_TO_KEEP = 1
+
+
+def main():
+    fp = os.path.abspath(sys.argv[1])
+    fdir = os.path.dirname(fp)
+    fname = os.path.basename(fp)
+    if not fname.startswith("put-") or not fname.endswith(".bin"):
+        raise Exception("not a post file")
+
+    zb = sys.stdin.buffer.read()
+    zs = zb.decode("utf-8", "replace")
+    md = json.loads(zs)
+
+    buf = b""
+    with open(fp, "rb") as f:
+        while True:
+            b = f.read(4096)
+            buf += b
+            if len(buf) > 4096:
+                raise Exception("too big")
+
+            if not b:
+                break
+
+    if not buf:
+        raise Exception("file is empty")
+
+    buf = unquote(buf.replace(b"+", b" "))
+    txt = buf.decode("utf-8")
+
+    if not txt.startswith("msg="):
+        raise Exception("does not start with msg=")
+
+    ip = md["up_ip"]
+    ts = md["up_at"]
+    txt = txt[4:]
+
+    # can put the database inside `fdir` if you'd like,
+    # by default it saves to PWD:
+    # os.chdir(fdir)
+
+    db = sqlite3.connect("guestbook.db3")
+    try:
+        db.execute("select 1 from gb").fetchone()
+    except:
+        with db:
+            db.execute("create table gb (ip text, ts real, msg text)")
+            db.execute("create index gb_ip on gb(ip)")
+
+    with db:
+        if NUM_MSGS_TO_KEEP == 1:
+            t = "delete from gb where ip = ?"
+            db.execute(t, (ip,))
+
+        t = "insert into gb values (?,?,?)"
+        db.execute(t, (ip, ts, txt))
+
+        if NUM_MSGS_TO_KEEP > 1:
+            t = "select ts from gb where ip = ? order by ts desc"
+            hits = db.execute(t, (ip,)).fetchall()
+
+            if len(hits) > NUM_MSGS_TO_KEEP:
+                lim = hits[NUM_MSGS_TO_KEEP][0]
+                t = "delete from gb where ip = ? and ts <= ?"
+                db.execute(t, (ip, lim))
+
+    print(txt)
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/image-noexif.py

@@ -89,4 +89,7 @@ def main():
 
 
 if __name__ == "__main__":
-    main()
+    try:
+        main()
+    except:
+        pass

bin/mtag/mousepad.py

@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+import subprocess as sp
+
+
+"""
+mtp test -- opens a texteditor
+
+usage:
+  -vsrv/v1:v1:r:c,mte=+x1:c,mtp=x1=ad,p,bin/mtag/mousepad.py
+
+explained:
+  c,mte: list of tags to index in this volume
+  c,mtp: add new tag provider
+     x1: dummy tag to provide
+     ad: dontcare if audio or not
+      p: priority 1 (run after initial tag-scan with ffprobe or mutagen)
+"""
+
+
+def main():
+    env = os.environ.copy()
+    env["DISPLAY"] = ":0.0"
+
+    if False:
+        # open the uploaded file
+        fp = sys.argv[-1]
+    else:
+        # display stdin contents (`oth_tags`)
+        fp = "/dev/stdin"
+
+    p = sp.Popen(["/usr/bin/mousepad", fp])
+    p.communicate()
+
+
+main()

bin/mtag/rclone-upload.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python
+
+import json
+import os
+import subprocess as sp
+import sys
+import time
+
+try:
+    from copyparty.util import fsenc
+except:
+
+    def fsenc(p):
+        return p.encode("utf-8")
+
+
+_ = r"""
+first checks the tag "vidchk" which must be "ok" to continue,
+then uploads all files to some cloud storage (RCLONE_REMOTE)
+and DELETES THE ORIGINAL FILES if rclone returns 0 ("success")
+
+deps:
+  rclone
+
+usage:
+  -mtp x2=t43200,ay,p2,bin/mtag/rclone-upload.py
+
+explained:
+t43200: timeout 12h
+    ay: only process files which contain audio (including video with audio)
+    p2: set priority 2 (after vidchk's suggested priority of 1),
+          so the output of vidchk will be passed in here
+
+complete usage example as vflags along with vidchk:
+  -vsrv/vidchk:vidchk:r:rw,ed:c,e2dsa,e2ts,mtp=vidchk=t600,p,bin/mtag/vidchk.py:c,mtp=rupload=t43200,ay,p2,bin/mtag/rclone-upload.py:c,mte=+vidchk,rupload
+
+setup: see https://rclone.org/drive/
+
+if you wanna use this script standalone / separately from copyparty,
+either set CONDITIONAL_UPLOAD False or provide the following stdin:
+  {"vidchk":"ok"}
+"""
+
+
+RCLONE_REMOTE = "notmybox"
+CONDITIONAL_UPLOAD = True
+
+
+def main():
+    fp = sys.argv[1]
+    if CONDITIONAL_UPLOAD:
+        zb = sys.stdin.buffer.read()
+        zs = zb.decode("utf-8", "replace")
+        md = json.loads(zs)
+
+        chk = md.get("vidchk", None)
+        if chk != "ok":
+            print(f"vidchk={chk}", file=sys.stderr)
+            sys.exit(1)
+
+    dst = f"{RCLONE_REMOTE}:".encode("utf-8")
+    cmd = [b"rclone", b"copy", b"--", fsenc(fp), dst]
+
+    t0 = time.time()
+    try:
+        sp.check_call(cmd)
+    except:
+        print("rclone failed", file=sys.stderr)
+        sys.exit(1)
+
+    print(f"{time.time() - t0:.1f} sec")
+    os.unlink(fsenc(fp))
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/very-bad-idea.py

@@ -16,7 +16,7 @@ goes without saying, but this is HELLA DANGEROUS,
   GIVES RCE TO ANYONE WHO HAVE UPLOAD PERMISSIONS
 
 example copyparty config to use this:
-  --urlform save,get -v.::w:c,e2d,e2t,mte=+a1:c,mtp=a1=ad,bin/mtag/very-bad-idea.py
+  --urlform save,get -v.::w:c,e2d,e2t,mte=+a1:c,mtp=a1=ad,kn,c0,bin/mtag/very-bad-idea.py
 
 recommended deps:
   apt install xdotool libnotify-bin
@@ -63,8 +63,8 @@ set -e
 EOF
 chmod 755 /usr/local/bin/chromium-browser
 
-# start the server  (note: replace `-v.::rw:` with `-v.::r:` to disallow retrieving uploaded stuff)
-cd ~/Downloads; python3 copyparty-sfx.py --urlform save,get -v.::rw:c,e2d,e2t,mte=+a1:c,mtp=a1=ad,very-bad-idea.py
+# start the server  (note: replace `-v.::rw:` with `-v.::w:` to disallow retrieving uploaded stuff)
+cd ~/Downloads; python3 copyparty-sfx.py --urlform save,get -v.::rw:c,e2d,e2t,mte=+a1:c,mtp=a1=ad,kn,very-bad-idea.py
 
 """
 

bin/mtag/vidchk.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+
+import json
+import re
+import os
+import sys
+import subprocess as sp
+
+try:
+    from copyparty.util import fsenc
+except:
+
+    def fsenc(p):
+        return p.encode("utf-8")
+
+
+_ = r"""
+inspects video files for errors and such
+plus stores a bunch of metadata to filename.ff.json
+
+usage:
+  -mtp vidchk=t600,ay,p,bin/mtag/vidchk.py
+
+explained:
+t600: timeout 10min
+  ay: only process files which contain audio (including video with audio)
+   p: set priority 1 (lowest priority after initial ffprobe/mutagen for base tags),
+       makes copyparty feed base tags into this script as json
+
+if you wanna use this script standalone / separately from copyparty,
+provide the video resolution on stdin as json:  {"res":"1920x1080"}
+"""
+
+
+FAST = True  # parse entire file at container level
+# FAST = False  # fully decode audio and video streams
+
+
+# warnings to ignore
+harmless = re.compile(
+    r"Unsupported codec with id |Could not find codec parameters.*Attachment:|analyzeduration"
+    + r"|timescale not set"
+)
+
+
+def wfilter(lines):
+    return [x for x in lines if x.strip() and not harmless.search(x)]
+
+
+def errchk(so, se, rc, dbg):
+    if dbg:
+        with open(dbg, "wb") as f:
+            f.write(b"so:\n" + so + b"\nse:\n" + se + b"\n")
+
+    if rc:
+        err = (so + se).decode("utf-8", "replace").split("\n", 1)
+        err = wfilter(err) or err
+        return f"ERROR {rc}: {err[0]}"
+
+    if se:
+        err = se.decode("utf-8", "replace").split("\n", 1)
+        err = wfilter(err)
+        if err:
+            return f"Warning: {err[0]}"
+
+    return None
+
+
+def main():
+    fp = sys.argv[1]
+    zb = sys.stdin.buffer.read()
+    zs = zb.decode("utf-8", "replace")
+    md = json.loads(zs)
+
+    fdir = os.path.dirname(os.path.realpath(fp))
+    flag = os.path.join(fdir, ".processed")
+    if os.path.exists(flag):
+        return "already processed"
+
+    try:
+        w, h = [int(x) for x in md["res"].split("x")]
+        if not w + h:
+            raise Exception()
+    except:
+        return "could not determine resolution"
+
+    # grab streams/format metadata + 2 seconds of frames at the start and end
+    zs = "ffprobe -hide_banner -v warning -of json -show_streams -show_format -show_packets -show_data_hash crc32 -read_intervals %+2,999999%+2"
+    cmd = zs.encode("ascii").split(b" ") + [fsenc(fp)]
+    p = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
+    so, se = p.communicate()
+
+    # spaces to tabs, drops filesize from 69k to 48k
+    so = b"\n".join(
+        [
+            b"\t" * int((len(x) - len(x.lstrip())) / 4) + x.lstrip()
+            for x in (so or b"").split(b"\n")
+        ]
+    )
+    with open(fsenc(f"{fp}.ff.json"), "wb") as f:
+        f.write(so)
+
+    err = errchk(so, se, p.returncode, f"{fp}.vidchk")
+    if err:
+        return err
+
+    if max(w, h) < 1280 and min(w, h) < 720:
+        return "resolution too small"
+
+    zs = (
+        "ffmpeg -y -hide_banner -nostdin -v warning"
+        + " -err_detect +crccheck+bitstream+buffer+careful+compliant+aggressive+explode"
+        + " -xerror -i"
+    )
+
+    cmd = zs.encode("ascii").split(b" ") + [fsenc(fp)]
+
+    if FAST:
+        zs = "-c copy -f null -"
+    else:
+        zs = "-vcodec rawvideo -acodec pcm_s16le -f null -"
+
+    cmd += zs.encode("ascii").split(b" ")
+
+    p = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
+    so, se = p.communicate()
+    return errchk(so, se, p.returncode, f"{fp}.vidchk")
+
+
+if __name__ == "__main__":
+    print(main() or "ok")

bin/prisonparty.sh

@@ -11,13 +11,13 @@ sysdirs=( /bin /lib /lib32 /lib64 /sbin /usr )
 help() { cat <<'EOF'
 
 usage:
-  ./prisonparty.sh <ROOTDIR> <UID> <GID> [VOLDIR [VOLDIR...]] -- python3 copyparty-sfx.py [...]"
+  ./prisonparty.sh <ROOTDIR> <UID> <GID> [VOLDIR [VOLDIR...]] -- python3 copyparty-sfx.py [...]
 
 example:
-  ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- python3 copyparty-sfx.py -v /mnt/nas/music::rwmd"
+  ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- python3 copyparty-sfx.py -v /mnt/nas/music::rwmd
 
 example for running straight from source (instead of using an sfx):
-  PYTHONPATH=$PWD ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- python3 -um copyparty -v /mnt/nas/music::rwmd"
+  PYTHONPATH=$PWD ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- python3 -um copyparty -v /mnt/nas/music::rwmd
 
 note that if you have python modules installed as --user (such as bpm/key detectors),
   you should add /home/foo/.local as a VOLDIR

bin/unforget.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+
+"""
+unforget.py: rebuild db from logfiles
+2022-09-07, v0.1, ed <irc.rizon.net>, MIT-Licensed
+https://github.com/9001/copyparty/blob/hovudstraum/bin/unforget.py
+
+only makes sense if running copyparty with --no-forget
+(e.g. immediately shifting uploads to other storage)
+
+usage:
+  xz -d < log | ./unforget.py .hist/up2k.db
+
+"""
+
+import re
+import sys
+import json
+import base64
+import sqlite3
+import argparse
+
+
+FS_ENCODING = sys.getfilesystemencoding()
+
+
+class APF(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
+    pass
+
+
+mem_cur = sqlite3.connect(":memory:").cursor()
+mem_cur.execute(r"create table a (b text)")
+
+
+def s3enc(rd: str, fn: str) -> tuple[str, str]:
+    ret: list[str] = []
+    for v in [rd, fn]:
+        try:
+            mem_cur.execute("select * from a where b = ?", (v,))
+            ret.append(v)
+        except:
+            wtf8 = v.encode(FS_ENCODING, "surrogateescape")
+            ret.append("//" + base64.urlsafe_b64encode(wtf8).decode("ascii"))
+
+    return ret[0], ret[1]
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("db")
+    ar = ap.parse_args()
+
+    db = sqlite3.connect(ar.db).cursor()
+    ptn_times = re.compile(r"no more chunks, setting times \(([0-9]+)")
+    at = 0
+    ctr = 0
+
+    for ln in [x.decode("utf-8", "replace").rstrip() for x in sys.stdin.buffer]:
+        if "no more chunks, setting times (" in ln:
+            m = ptn_times.search(ln)
+            if m:
+                at = int(m.group(1))
+
+        if '"hash": []' in ln:
+            try:
+                ofs = ln.find("{")
+                j = json.loads(ln[ofs:])
+            except:
+                pass
+
+            w = j["wark"]
+            if db.execute("select w from up where w = ?", (w,)).fetchone():
+                continue
+
+            # PYTHONPATH=/home/ed/dev/copyparty/ python3 -m copyparty -e2dsa  -v foo:foo:rwmd,ed -aed:wark --no-forget
+            # 05:34:43.845 127.0.0.1 42496       no more chunks, setting times (1662528883, 1658001882)
+            # 05:34:43.863 127.0.0.1 42496       {"name": "f\"2", "purl": "/foo/bar/baz/", "size": 1674, "lmod": 1658001882, "sprs": true, "hash": [], "wark": "LKIWpp2jEAh9dH3fu-DobuURFGEKlODXDGTpZ1otMhUg"}
+            # |                      w                       |     mt     |  sz  |   rd    | fn  |    ip     |     at     |
+            # | LKIWpp2jEAh9dH3fu-DobuURFGEKlODXDGTpZ1otMhUg | 1658001882 | 1674 | bar/baz | f"2 | 127.0.0.1 | 1662528883 |
+
+            rd, fn = s3enc(j["purl"].strip("/"), j["name"])
+            ip = ln.split(" ")[1].split("m")[-1]
+
+            q = "insert into up values (?,?,?,?,?,?,?)"
+            v = (w, int(j["lmod"]), int(j["size"]), rd, fn, ip, at)
+            db.execute(q, v)
+            ctr += 1
+            if ctr % 1024 == 1023:
+                print(f"{ctr} commit...")
+                db.connection.commit()
+
+    if ctr:
+        db.connection.commit()
+
+    print(f"unforgot {ctr} files")
+
+
+if __name__ == "__main__":
+    main()

bin/up2k.py

@@ -3,7 +3,7 @@ from __future__ import print_function, unicode_literals
 
 """
 up2k.py: upload to copyparty
-2022-06-16, v0.15, ed <irc.rizon.net>, MIT-Licensed
+2022-09-05, v0.19, ed <irc.rizon.net>, MIT-Licensed
 https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py
 
 - dependencies: requests
@@ -22,12 +22,29 @@ import atexit
 import signal
 import base64
 import hashlib
-import argparse
 import platform
 import threading
 import datetime
 
-import requests
+try:
+    import argparse
+except:
+    m = "\n  ERROR: need 'argparse'; download it here:\n   https://github.com/ThomasWaldmann/argparse/raw/master/argparse.py\n"
+    print(m)
+    raise
+
+try:
+    import requests
+except ImportError:
+    if sys.version_info > (2, 7):
+        m = "\nERROR: need 'requests'; please run this command:\n {0} -m pip install --user requests\n"
+    else:
+        m = "requests/2.18.4 urllib3/1.23 chardet/3.0.4 certifi/2020.4.5.1 idna/2.7"
+        m = ["   https://pypi.org/project/" + x + "/#files" for x in m.split()]
+        m = "\n  ERROR: need these:\n" + "\n".join(m) + "\n"
+
+    print(m.format(sys.executable))
+    sys.exit(1)
 
 
 # from copyparty/__init__.py
@@ -126,6 +143,89 @@ class FileSlice(object):
         return ret
 
 
+class MTHash(object):
+    def __init__(self, cores):
+        self.f = None
+        self.sz = 0
+        self.csz = 0
+        self.omutex = threading.Lock()
+        self.imutex = threading.Lock()
+        self.work_q = Queue()
+        self.done_q = Queue()
+        self.thrs = []
+        for _ in range(cores):
+            t = threading.Thread(target=self.worker)
+            t.daemon = True
+            t.start()
+            self.thrs.append(t)
+
+    def hash(self, f, fsz, chunksz, pcb=None, pcb_opaque=None):
+        with self.omutex:
+            self.f = f
+            self.sz = fsz
+            self.csz = chunksz
+
+            chunks = {}
+            nchunks = int(math.ceil(fsz / chunksz))
+            for nch in range(nchunks):
+                self.work_q.put(nch)
+
+            ex = ""
+            for nch in range(nchunks):
+                qe = self.done_q.get()
+                try:
+                    nch, dig, ofs, csz = qe
+                    chunks[nch] = [dig, ofs, csz]
+                except:
+                    ex = ex or qe
+
+                if pcb:
+                    pcb(pcb_opaque, chunksz * nch)
+
+            if ex:
+                raise Exception(ex)
+
+            ret = []
+            for n in range(nchunks):
+                ret.append(chunks[n])
+
+            self.f = None
+            self.csz = 0
+            self.sz = 0
+            return ret
+
+    def worker(self):
+        while True:
+            ofs = self.work_q.get()
+            try:
+                v = self.hash_at(ofs)
+            except Exception as ex:
+                v = str(ex)
+
+            self.done_q.put(v)
+
+    def hash_at(self, nch):
+        f = self.f
+        ofs = ofs0 = nch * self.csz
+        hashobj = hashlib.sha512()
+        chunk_sz = chunk_rem = min(self.csz, self.sz - ofs)
+        while chunk_rem > 0:
+            with self.imutex:
+                f.seek(ofs)
+                buf = f.read(min(chunk_rem, 1024 * 1024 * 12))
+
+            if not buf:
+                raise Exception("EOF at " + str(ofs))
+
+            hashobj.update(buf)
+            chunk_rem -= len(buf)
+            ofs += len(buf)
+
+        digest = hashobj.digest()[:33]
+        digest = base64.urlsafe_b64encode(digest).decode("utf-8")
+        return nch, digest, ofs0, chunk_sz
+
+
 _print = print
 
 
@@ -230,8 +330,8 @@ def _scd(err, top):
             abspath = os.path.join(top, fh.name)
             try:
                 yield [abspath, fh.stat()]
-            except:
-                err.append(abspath)
+            except Exception as ex:
+                err.append((abspath, str(ex)))
 
 
 def _lsd(err, top):
@@ -240,25 +340,31 @@ def _lsd(err, top):
         abspath = os.path.join(top, name)
         try:
             yield [abspath, os.stat(abspath)]
-        except:
-            err.append(abspath)
+        except Exception as ex:
+            err.append((abspath, str(ex)))
 
 
-if hasattr(os, "scandir"):
+if hasattr(os, "scandir") and sys.version_info > (3, 6):
     statdir = _scd
 else:
     statdir = _lsd
 
 
-def walkdir(err, top):
+def walkdir(err, top, seen):
     """recursive statdir"""
+    atop = os.path.abspath(os.path.realpath(top))
+    if atop in seen:
+        err.append((top, "recursive-symlink"))
+        return
+
+    seen = seen[:] + [atop]
     for ap, inf in sorted(statdir(err, top)):
         if stat.S_ISDIR(inf.st_mode):
             try:
-                for x in walkdir(err, ap):
+                for x in walkdir(err, ap, seen):
                     yield x
-            except:
-                err.append(ap)
+            except Exception as ex:
+                err.append((ap, str(ex)))
         else:
             yield ap, inf
 
@@ -273,7 +379,7 @@ def walkdirs(err, tops):
             stop = os.path.dirname(top)
 
         if os.path.isdir(top):
-            for ap, inf in walkdir(err, top):
+            for ap, inf in walkdir(err, top, []):
                 yield stop, ap[len(stop) :].lstrip(sep), inf
         else:
             d, n = top.rsplit(sep, 1)
@@ -322,8 +428,8 @@ def up2k_chunksize(filesize):
 
 
 # mostly from copyparty/up2k.py
-def get_hashlist(file, pcb):
-    # type: (File, any) -> None
+def get_hashlist(file, pcb, mth):
+    # type: (File, any, any) -> None
     """generates the up2k hashlist from file contents, inserts it into `file`"""
 
     chunk_sz = up2k_chunksize(file.size)
@@ -331,7 +437,12 @@ def get_hashlist(file, pcb):
     file_ofs = 0
     ret = []
     with open(file.abs, "rb", 512 * 1024) as f:
+        if mth and file.size >= 1024 * 512:
+            ret = mth.hash(f, file.size, chunk_sz, pcb, file)
+            file_rem = 0
+
         while file_rem > 0:
+            # same as `hash_at` except for `imutex` / bufsz
             hashobj = hashlib.sha512()
             chunk_sz = chunk_rem = min(chunk_sz, file_rem)
             while chunk_rem > 0:
@@ -388,8 +499,9 @@ def handshake(req_ses, url, file, pw, search):
         try:
             r = req_ses.post(url, headers=headers, json=req)
             break
-        except:
-            eprint("handshake failed, retrying: {0}\n".format(file.name))
+        except Exception as ex:
+            em = str(ex).split("SSLError(")[-1]
+            eprint("handshake failed, retrying: {0}\n  {1}\n\n".format(file.name, em))
             time.sleep(1)
 
     try:
@@ -398,7 +510,7 @@ def handshake(req_ses, url, file, pw, search):
         raise Exception(r.text)
 
     if search:
-        return r["hits"]
+        return r["hits"], False
 
     try:
         pre, url = url.split("://")
@@ -470,12 +582,19 @@ class Ctl(object):
 
         if err:
             eprint("\n# failed to access {0} paths:\n".format(len(err)))
-            for x in err:
-                eprint(x.decode("utf-8", "replace") + "\n")
+            for ap, msg in err:
+                if ar.v:
+                    eprint("{0}\n `-{1}\n\n".format(ap.decode("utf-8", "replace"), msg))
+                else:
+                    eprint(ap.decode("utf-8", "replace") + "\n")
 
             eprint("^ failed to access those {0} paths ^\n\n".format(len(err)))
+
+            if not ar.v:
+                eprint("hint: set -v for detailed error messages\n")
+
             if not ar.ok:
-                eprint("aborting because --ok is not set\n")
+                eprint("hint: aborting because --ok is not set\n")
                 return
 
         eprint("found {0} files, {1}\n\n".format(nfiles, humansize(nbytes)))
@@ -516,6 +635,8 @@ class Ctl(object):
             self.st_hash = [None, "(idle, starting...)"]  # type: tuple[File, int]
             self.st_up = [None, "(idle, starting...)"]  # type: tuple[File, int]
 
+            self.mth = MTHash(ar.J) if ar.J > 1 else None
+
             self._fancy()
 
     def _safe(self):
@@ -526,7 +647,7 @@ class Ctl(object):
             upath = file.abs.decode("utf-8", "replace")
 
             print("{0} {1}\n  hash...".format(self.nfiles - nf, upath))
-            get_hashlist(file, None)
+            get_hashlist(file, None, None)
 
             burl = self.ar.url[:12] + self.ar.url[8:].split("/")[0] + "/"
             while True:
@@ -679,7 +800,7 @@ class Ctl(object):
 
                 time.sleep(0.05)
 
-            get_hashlist(file, self.cb_hasher)
+            get_hashlist(file, self.cb_hasher, self.mth)
             with self.mutex:
                 self.hash_f += 1
                 self.hash_c += len(file.cids)
@@ -808,6 +929,9 @@ def main():
     if not VT100:
         os.system("rem")  # enables colors
 
+    cores = (os.cpu_count() if hasattr(os, "cpu_count") else 0) or 2
+    hcores = min(cores, 3)  # 4% faster than 4+ on py3.9 @ r5-4500U
+
     # fmt: off
     ap = app = argparse.ArgumentParser(formatter_class=APF, epilog="""
 NOTE:
@@ -818,11 +942,13 @@ source file/folder selection uses rsync syntax, meaning that:
 
     ap.add_argument("url", type=unicode, help="server url, including destination folder")
     ap.add_argument("files", type=unicode, nargs="+", help="files and/or folders to process")
+    ap.add_argument("-v", action="store_true", help="verbose")
     ap.add_argument("-a", metavar="PASSWORD", help="password")
     ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
     ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
     ap = app.add_argument_group("performance tweaks")
     ap.add_argument("-j", type=int, metavar="THREADS", default=4, help="parallel connections")
+    ap.add_argument("-J", type=int, metavar="THREADS", default=hcores, help="num cpu-cores to use for hashing; set 0 or 1 for single-core hashing")
     ap.add_argument("-nh", action="store_true", help="disable hashing while uploading")
     ap.add_argument("--safe", action="store_true", help="use simple fallback approach")
     ap.add_argument("-z", action="store_true", help="ZOOMIN' (skip uploading files if they exist at the destination with the ~same last-modified timestamp, so same as yolo / turbo with date-chk but even faster)")

contrib/README.md

@@ -22,6 +22,9 @@ however if your copyparty is behind a reverse-proxy, you may want to use [`share
 * `URL`: full URL to the root folder (with trailing slash) followed by `$regex:1|1$`
 * `pw`: password (remove `Parameters` if anon-write)
 
+### [`media-osd-bgone.ps1`](media-osd-bgone.ps1)
+* disables the [windows OSD popup](https://user-images.githubusercontent.com/241032/122821375-0e08df80-d2dd-11eb-9fd9-184e8aacf1d0.png) (the thing on the left) which appears every time you hit media hotkeys to adjust volume or change song while playing music with the copyparty web-ui, or most other audio players really
+
 ### [`explorer-nothumbs-nofoldertypes.reg`](explorer-nothumbs-nofoldertypes.reg)
 * disables thumbnails and folder-type detection in windows explorer
 * makes it way faster (especially for slow/networked locations (such as copyparty-fuse))

contrib/media-osd-bgone.ps1

@@ -0,0 +1,104 @@
+# media-osd-bgone.ps1: disable media-control OSD on win10do
+# v1.1, 2021-06-25, ed <irc.rizon.net>, MIT-licensed
+# https://github.com/9001/copyparty/blob/hovudstraum/contrib/media-osd-bgone.ps1
+#
+# locates the first window that looks like the media OSD and minimizes it;
+# doing this once after each reboot should do the trick
+# (adjust the width/height filter if it doesn't work)
+#
+# ---------------------------------------------------------------------
+#
+# tip: save the following as "media-osd-bgone.bat" next to this script:
+#   start cmd /c "powershell -command ""set-executionpolicy -scope process bypass; .\media-osd-bgone.ps1"" & ping -n 2 127.1 >nul"
+#
+# then create a shortcut to that bat-file and move the shortcut here:
+#   %appdata%\Microsoft\Windows\Start Menu\Programs\Startup
+#
+# and now this will autorun on bootup
+
+
+Add-Type -TypeDefinition @"
+using System;
+using System.IO;
+using System.Threading;
+using System.Diagnostics;
+using System.Runtime.InteropServices;
+using System.Windows.Forms;
+
+namespace A {
+  public class B : Control {
+
+    [DllImport("user32.dll")]
+    static extern void keybd_event(byte bVk, byte bScan, uint dwFlags, int dwExtraInfo);
+
+    [DllImport("user32.dll", SetLastError = true)]
+    static extern IntPtr FindWindowEx(IntPtr hwndParent, IntPtr hwndChildAfter, string lpszClass, string lpszWindow);
+
+    [DllImport("user32.dll", SetLastError=true)]
+    static extern bool GetWindowRect(IntPtr hwnd, out RECT lpRect);
+
+    [DllImport("user32.dll")]
+    static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
+
+    [StructLayout(LayoutKind.Sequential)]
+    public struct RECT {
+      public int x;
+      public int y;
+      public int x2;
+      public int y2;
+    }
+    
+    bool fa() {
+      RECT r;
+      IntPtr it = IntPtr.Zero;
+      while ((it = FindWindowEx(IntPtr.Zero, it, "NativeHWNDHost", "")) != IntPtr.Zero) {
+        if (FindWindowEx(it, IntPtr.Zero, "DirectUIHWND", "") == IntPtr.Zero)
+          continue;
+        
+        if (!GetWindowRect(it, out r))
+          continue;
+
+        int w = r.x2 - r.x + 1;
+        int h = r.y2 - r.y + 1;
+
+        Console.WriteLine("[*] hwnd {0:x} @ {1}x{2} sz {3}x{4}", it, r.x, r.y, w, h);
+        if (h != 141)
+          continue;
+        
+        ShowWindow(it, 6);
+        Console.WriteLine("[+] poof");
+        return true;
+      }
+      return false;
+    }
+
+    void fb() {
+      keybd_event((byte)Keys.VolumeMute, 0, 0, 0);
+      keybd_event((byte)Keys.VolumeMute, 0, 2, 0);
+      Thread.Sleep(500);
+      keybd_event((byte)Keys.VolumeMute, 0, 0, 0);
+      keybd_event((byte)Keys.VolumeMute, 0, 2, 0);
+
+      while (true) {
+        if (fa()) {
+          break;
+        }
+        Console.WriteLine("[!] not found");
+        Thread.Sleep(1000);
+      }
+      this.Invoke((MethodInvoker)delegate {
+        Application.Exit();
+      });
+    }
+
+    public void Run() {
+      Console.WriteLine("[+] hi");
+      new Thread(new ThreadStart(fb)).Start();
+      Application.Run();
+      Console.WriteLine("[+] bye");
+    }
+  }
+}
+"@ -ReferencedAssemblies System.Windows.Forms
+
+(New-Object -TypeName A.B).Run()

contrib/plugins/README.md

@@ -11,6 +11,15 @@ save one of these as `.epilogue.html` inside a folder to customize it:
 
 
 
+## example browser-js
+point `--js-browser` to one of these by URL:
+
+* [`minimal-up2k.js`](minimal-up2k.js) is similar to the above `minimal-up2k.html` except it applies globally to all write-only folders
+* [`up2k-hooks.js`](up2k-hooks.js) lets you specify a ruleset for files to skip uploading
+  * [`up2k-hook-ytid.js`](up2k-hook-ytid.js) is a more specific example checking youtube-IDs against some API
+
+
+
 ## example browser-css
 point `--css-browser` to one of these by URL:
 

contrib/plugins/minimal-up2k.html

@@ -7,7 +7,7 @@
 
     /* make the up2k ui REALLY minimal by hiding a bunch of stuff: */
 
-    #ops, #tree, #path, #wrap>h2:last-child,  /* main tabs and navigators (tree/breadcrumbs) */
+    #ops, #tree, #path, #epi+h2,  /* main tabs and navigators (tree/breadcrumbs) */
 
     #u2conf tr:first-child>td[rowspan]:not(#u2btn_cw),  /* most of the config options */
 

contrib/plugins/minimal-up2k.js

@@ -0,0 +1,59 @@
+/*
+
+makes the up2k ui REALLY minimal by hiding a bunch of stuff
+
+almost the same as minimal-up2k.html except this one...:
+
+ -- applies to every write-only folder when used with --js-browser
+
+ -- only applies if javascript is enabled
+
+ -- doesn't hide the total upload ETA display
+
+ -- looks slightly better
+
+*/
+
+var u2min = `
+<style>
+
+#ops, #path, #tree, #files, #epi+div+h2,
+#u2conf td.c+.c, #u2cards, #srch_dz, #srch_zd {
+  display: none !important;
+}
+#u2conf {margin:5em auto 0 auto !important}
+#u2conf.ww {width:70em}
+#u2conf.w {width:50em}
+#u2conf.w .c,
+#u2conf.w #u2btn_cw {text-align:left}
+#u2conf.w #u2btn_cw {width:70%}
+#u2etaw {margin:3em auto}
+#u2etaw.w {
+  text-align: center;
+  margin: -3.5em auto 5em auto;
+}
+#u2etaw.w #u2etas {margin-right:-37em}
+#u2etaw.w #u2etas.o {margin-top:-2.2em}
+#u2etaw.ww {margin:-1em auto}
+#u2etaw.ww #u2etas {padding-left:4em}
+#u2etas {
+  background: none !important;
+  border: none !important;
+}
+#wrap {margin-left:2em !important}
+.logue {
+  border: none !important;
+  margin: 2em auto !important;
+}
+.logue:before {content:'' !important}
+
+</style>
+
+<a href="#" onclick="this.parentNode.innerHTML='';">show advanced options</a>
+`;
+
+if (!has(perms, 'read')) {
+  var e2 = mknod('div');
+  e2.innerHTML = u2min;
+  ebi('wrap').insertBefore(e2, QS('#epi+h2'));
+}

contrib/plugins/up2k-hook-ytid.js

@@ -0,0 +1,297 @@
+// way more specific example --
+// assumes all files dropped into the uploader have a youtube-id somewhere in the filename,
+// locates the youtube-ids and passes them to an API which returns a list of IDs which should be uploaded
+//
+// also tries to find the youtube-id in the embedded metadata
+//
+// assumes copyparty is behind nginx as /ytq is a standalone service which must be rproxied in place
+
+function up2k_namefilter(good_files, nil_files, bad_files, hooks) {
+    var passthru = up2k.uc.fsearch;
+    if (passthru)
+        return hooks[0](good_files, nil_files, bad_files, hooks.slice(1));
+
+    a_up2k_namefilter(good_files, nil_files, bad_files, hooks).then(() => { });
+}
+
+// ebi('op_up2k').appendChild(mknod('input','unick'));
+
+function bstrpos(buf, ptn) {
+    var ofs = 0,
+        ch0 = ptn[0],
+        sz = buf.byteLength;
+
+    while (true) {
+        ofs = buf.indexOf(ch0, ofs);
+        if (ofs < 0 || ofs >= sz)
+            return -1;
+
+        for (var a = 1; a < ptn.length; a++)
+            if (buf[ofs + a] !== ptn[a])
+                break;
+
+        if (a === ptn.length)
+            return ofs;
+
+        ++ofs;
+    }
+}
+
+async function a_up2k_namefilter(good_files, nil_files, bad_files, hooks) {
+    var t0 = Date.now(),
+        yt_ids = new Set(),
+        textdec = new TextDecoder('latin1'),
+        md_ptn = new TextEncoder().encode('youtube.com/watch?v='),
+        file_ids = [],  // all IDs found for each good_files
+        md_only = [],  // `${id} ${fn}` where ID was only found in metadata
+        mofs = 0,
+        mnchk = 0,
+        mfile = '',
+        myid = localStorage.getItem('ytid_t0');
+
+    if (!myid)
+        localStorage.setItem('ytid_t0', myid = Date.now());
+
+    for (var a = 0; a < good_files.length; a++) {
+        var [fobj, name] = good_files[a],
+            cname = name,  // will clobber
+            sz = fobj.size,
+            ids = [],
+            fn_ids = [],
+            md_ids = [],
+            id_ok = false,
+            m;
+
+        // all IDs found in this file
+        file_ids.push(ids);
+
+        // look for ID in filename; reduce the
+        // metadata-scan intensity if the id looks safe
+        m = /[\[(-]([\w-]{11})[\])]?\.(?:mp4|webm|mkv|flv|opus|ogg|mp3|m4a|aac)$/i.exec(name);
+        id_ok = !!m;
+
+        while (true) {
+            // fuzzy catch-all;
+            // some ytdl fork did %(title)-%(id).%(ext) ...
+            m = /(?:^|[^\w])([\w-]{11})(?:$|[^\w-])/.exec(cname);
+            if (!m)
+                break;
+
+            cname = cname.replace(m[1], '');
+            yt_ids.add(m[1]);
+            fn_ids.unshift(m[1]);
+        }
+
+        // look for IDs in video metadata,
+        if (/\.(mp4|webm|mkv|flv|opus|ogg|mp3|m4a|aac)$/i.exec(name)) {
+            toast.show('inf r', 0, `analyzing file ${a + 1} / ${good_files.length} :\n${name}\n\nhave analysed ${++mnchk} files in ${(Date.now() - t0) / 1000} seconds, ${humantime((good_files.length - (a + 1)) * (((Date.now() - t0) / 1000) / mnchk))} remaining,\n\nbiggest offset so far is ${mofs}, in this file:\n\n${mfile}`);
+
+            // check first and last 128 MiB;
+            // pWxOroN5WCo.mkv @  6edb98 (6.92M)
+            // Nf-nN1wF5Xo.mp4 @ 4a98034 (74.6M)
+            var chunksz = 1024 * 1024 * 2,  // byte
+                aspan = id_ok ? 128 : 512;  // MiB
+
+            aspan = parseInt(Math.min(sz / 2, aspan * 1024 * 1024) / chunksz) * chunksz;
+            if (!aspan)
+                aspan = Math.min(sz, chunksz);
+
+            for (var side = 0; side < 2; side++) {
+                var ofs = side ? Math.max(0, sz - aspan) : 0,
+                    nchunks = aspan / chunksz;
+
+                for (var chunk = 0; chunk < nchunks; chunk++) {
+                    var bchunk = await fobj.slice(ofs, ofs + chunksz + 16).arrayBuffer(),
+                        uchunk = new Uint8Array(bchunk, 0, bchunk.byteLength),
+                        bofs = bstrpos(uchunk, md_ptn),
+                        absofs = Math.min(ofs + bofs, (sz - ofs) + bofs),
+                        txt = bofs < 0 ? '' : textdec.decode(uchunk.subarray(bofs)),
+                        m;
+
+                    //console.log(`side ${ side }, chunk ${ chunk }, ofs ${ ofs }, bchunk ${ bchunk.byteLength }, txt ${ txt.length }`);
+                    while (true) {
+                        // mkv/webm have [a-z] immediately after url
+                        m = /(youtube\.com\/watch\?v=[\w-]{11})/.exec(txt);
+                        if (!m)
+                            break;
+
+                        txt = txt.replace(m[1], '');
+                        m = m[1].slice(-11);
+
+                        console.log(`found ${m} @${bofs}, ${name} `);
+                        yt_ids.add(m);
+                        if (!has(fn_ids, m) && !has(md_ids, m)) {
+                            md_ids.push(m);
+                            md_only.push(`${m} ${name}`);
+                        }
+                        else
+                            // id appears several times; make it preferred
+                            md_ids.unshift(m);
+
+                        // bail after next iteration
+                        chunk = nchunks - 1;
+                        side = 9;
+
+                        if (mofs < absofs) {
+                            mofs = absofs;
+                            mfile = name;
+                        }
+                    }
+                    ofs += chunksz;
+                    if (ofs >= sz)
+                        break;
+                }
+            }
+        }
+
+        for (var yi of md_ids)
+            ids.push(yi);
+
+        for (var yi of fn_ids)
+            if (!has(ids, yi))
+                ids.push(yi);
+    }
+
+    if (md_only.length)
+        console.log('recovered the following youtube-IDs by inspecting metadata:\n\n' + md_only.join('\n'));
+    else if (yt_ids.size)
+        console.log('did not discover any additional youtube-IDs by inspecting metadata; all the IDs also existed in the filenames');
+    else
+        console.log('failed to find any youtube-IDs at all, sorry');
+
+    if (false) {
+        var msg = `finished analysing ${mnchk} files in ${(Date.now() - t0) / 1000} seconds,\n\nbiggest offset was ${mofs} in this file:\n\n${mfile}`,
+            mfun = function () { toast.ok(0, msg); };
+
+        mfun();
+        setTimeout(mfun, 200);
+
+        return hooks[0]([], [], [], hooks.slice(1));
+    }
+
+    var el = ebi('unick'), unick = el ? el.value : '';
+    if (unick) {
+        console.log(`sending uploader nickname [${unick}]`);
+        fetch(document.location, {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8' },
+            body: 'msg=' + encodeURIComponent(unick)
+        });
+    }
+
+    toast.inf(5, `running query for ${yt_ids.size} youtube-IDs...`);
+
+    var xhr = new XHR();
+    xhr.open('POST', '/ytq', true);
+    xhr.setRequestHeader('Content-Type', 'text/plain');
+    xhr.onload = xhr.onerror = function () {
+        if (this.status != 200)
+            return toast.err(0, `sorry, database query failed ;_;\n\nplease let us know so we can look at it, thx!!\n\nerror ${this.status}: ${(this.response && this.response.err) || this.responseText}`);
+
+        process_id_list(this.responseText);
+    };
+    xhr.send(Array.from(yt_ids).join('\n'));
+
+    function process_id_list(txt) {
+        var wanted_ids = new Set(txt.trim().split('\n')),
+            name_id = {},
+            wanted_names = new Set(),  // basenames with a wanted ID -- not including relpath
+            wanted_names_scoped = {},  // basenames with a wanted ID -> list of dirs to search under
+            wanted_files = new Set();  // filedrops
+
+        for (var a = 0; a < good_files.length; a++) {
+            var name = good_files[a][1];
+            for (var b = 0; b < file_ids[a].length; b++)
+                if (wanted_ids.has(file_ids[a][b])) {
+                    // let the next stage handle this to prevent dupes
+                    //wanted_files.add(good_files[a]);
+
+                    var m = /(.*)\.(mp4|webm|mkv|flv|opus|ogg|mp3|m4a|aac)$/i.exec(name);
+                    if (!m)
+                        continue;
+
+                    var [rd, fn] = vsplit(m[1]);
+
+                    if (fn in wanted_names_scoped)
+                        wanted_names_scoped[fn].push(rd);
+                    else
+                        wanted_names_scoped[fn] = [rd];
+
+                    wanted_names.add(fn);
+                    name_id[m[1]] = file_ids[a][b];
+
+                    break;
+                }
+        }
+
+        // add all files with the same basename as each explicitly wanted file
+        // (infojson/chatlog/etc when ID was discovered from metadata)
+        for (var a = 0; a < good_files.length; a++) {
+            var [rd, name] = vsplit(good_files[a][1]);
+            for (var b = 0; b < 3; b++) {
+                name = name.replace(/\.[^\.]+$/, '');
+                if (!wanted_names.has(name))
+                    continue;
+
+                var vid_fp = false;
+                for (var c of wanted_names_scoped[name])
+                    if (rd.startsWith(c))
+                        vid_fp = c + name;
+
+                if (!vid_fp)
+                    continue;
+
+                var subdir = name_id[vid_fp];
+                subdir = `v${subdir.slice(0, 1)}/${subdir}-${myid}`;
+                var newpath = subdir + '/' + good_files[a][1].split(/\//g).pop();
+
+                // check if this file is a dupe
+                for (var c of good_files)
+                    if (c[1] == newpath)
+                        newpath = null;
+
+                if (!newpath)
+                    break;
+
+                good_files[a][1] = newpath;
+                wanted_files.add(good_files[a]);
+                break;
+            }
+        }
+
+        function upload_filtered() {
+            if (!wanted_files.size)
+                return modal.alert('Good news -- turns out we already have all those.\n\nBut thank you for checking in!');
+
+            hooks[0](Array.from(wanted_files), nil_files, bad_files, hooks.slice(1));
+        }
+
+        function upload_all() {
+            hooks[0](good_files, nil_files, bad_files, hooks.slice(1));
+        }
+
+        var n_skip = good_files.length - wanted_files.size,
+            msg = `you added ${good_files.length} files; ${good_files.length == n_skip ? 'all' : n_skip} of them were skipped --\neither because we already have them,\nor because there is no youtube-ID in your filenames.\n\n<code>OK</code> / <code>Enter</code> = continue uploading just the ${wanted_files.size} files we definitely need\n\n<code>Cancel</code> / <code>ESC</code> = override the filter; upload ALL the files you added`;
+
+        if (!n_skip)
+            upload_filtered();
+        else
+            modal.confirm(msg, upload_filtered, upload_all);
+    };
+}
+
+up2k_hooks.push(function () {
+    up2k.gotallfiles.unshift(up2k_namefilter);
+});
+
+// persist/restore nickname field if present
+setInterval(function () {
+    var o = ebi('unick');
+    if (!o || document.activeElement == o)
+        return;
+
+    o.oninput = function () {
+        localStorage.setItem('unick', o.value);
+    };
+    o.value = localStorage.getItem('unick') || '';
+}, 1000);

contrib/plugins/up2k-hooks.js

@@ -0,0 +1,45 @@
+// hooks into up2k
+
+function up2k_namefilter(good_files, nil_files, bad_files, hooks) {
+    // is called when stuff is dropped into the browser,
+    // after iterating through the directory tree and discovering all files,
+    // before the upload confirmation dialogue is shown
+
+    // good_files will successfully upload
+    // nil_files are empty files and will show an alert in the final hook
+    // bad_files are unreadable and cannot be uploaded
+    var file_lists = [good_files, nil_files, bad_files];
+
+    // build a list of filenames
+    var filenames = [];
+    for (var lst of file_lists)
+        for (var ent of lst)
+            filenames.push(ent[1]);
+
+    toast.inf(5, "running database query...");
+
+    // simulate delay while passing the list to some api for checking
+    setTimeout(function () {
+
+        // only keep webm files as an example
+        var new_lists = [];
+        for (var lst of file_lists) {
+            var keep = [];
+            new_lists.push(keep);
+
+            for (var ent of lst)
+                if (/\.webm$/.test(ent[1]))
+                    keep.push(ent);
+        }
+
+        // finally, call the next hook in the chain
+        [good_files, nil_files, bad_files] = new_lists;
+        hooks[0](good_files, nil_files, bad_files, hooks.slice(1));
+
+    }, 1000);
+}
+
+// register
+up2k_hooks.push(function () {
+    up2k.gotallfiles.unshift(up2k_namefilter);
+});

copyparty/__init__.py

@@ -7,8 +7,6 @@ import sys
 import time
 
 try:
-    from collections.abc import Callable
-
     from typing import TYPE_CHECKING, Any
 except:
     TYPE_CHECKING = False
@@ -33,57 +31,18 @@ ANYWIN = WINDOWS or sys.platform in ["msys", "cygwin"]
 
 MACOS = platform.system() == "Darwin"
 
-
-def get_unixdir() -> str:
-    paths: list[tuple[Callable[..., str], str]] = [
-        (os.environ.get, "XDG_CONFIG_HOME"),
-        (os.path.expanduser, "~/.config"),
-        (os.environ.get, "TMPDIR"),
-        (os.environ.get, "TEMP"),
-        (os.environ.get, "TMP"),
-        (unicode, "/tmp"),
-    ]
-    for chk in [os.listdir, os.mkdir]:
-        for pf, pa in paths:
-            try:
-                p = pf(pa)
-                # print(chk.__name__, p, pa)
-                if not p or p.startswith("~"):
-                    continue
-
-                p = os.path.normpath(p)
-                chk(p)  # type: ignore
-                p = os.path.join(p, "copyparty")
-                if not os.path.isdir(p):
-                    os.mkdir(p)
-
-                return p
-            except:
-                pass
-
-    raise Exception("could not find a writable path for config")
+try:
+    CORES = len(os.sched_getaffinity(0))
+except:
+    CORES = (os.cpu_count() if hasattr(os, "cpu_count") else 0) or 2
 
 
 class EnvParams(object):
     def __init__(self) -> None:
         self.t0 = time.time()
-        self.mod = os.path.dirname(os.path.realpath(__file__))
-        if self.mod.endswith("__init__"):
-            self.mod = os.path.dirname(self.mod)
-
-        if sys.platform == "win32":
-            self.cfg = os.path.normpath(os.environ["APPDATA"] + "/copyparty")
-        elif sys.platform == "darwin":
-            self.cfg = os.path.expanduser("~/Library/Preferences/copyparty")
-        else:
-            self.cfg = get_unixdir()
-
-        self.cfg = self.cfg.replace("\\", "/")
-        try:
-            os.makedirs(self.cfg)
-        except:
-            if not os.path.isdir(self.cfg):
-                raise
+        self.mod = None
+        self.cfg = None
+        self.ox = getattr(sys, "oxidized", None)
 
 
 E = EnvParams()

copyparty/__main__.py

@@ -20,13 +20,25 @@ import time
 import traceback
 from textwrap import dedent
 
-from .__init__ import ANYWIN, PY2, VT100, WINDOWS, E, unicode
+from .__init__ import ANYWIN, CORES, PY2, VT100, WINDOWS, E, EnvParams, unicode
 from .__version__ import CODENAME, S_BUILD_DT, S_VERSION
 from .authsrv import re_vol
 from .svchub import SvcHub
-from .util import IMPLICATIONS, align_tab, ansi_re, min_ex, py_desc, termsize, wrap
+from .util import (
+    IMPLICATIONS,
+    JINJA_VER,
+    PYFTPD_VER,
+    SQLITE_VER,
+    align_tab,
+    ansi_re,
+    min_ex,
+    py_desc,
+    termsize,
+    wrap,
+)
 
 try:
+    from collections.abc import Callable
     from types import FrameType
 
     from typing import Any, Optional
@@ -107,18 +119,92 @@ class BasicDodge11874(
 
 
 def lprint(*a: Any, **ka: Any) -> None:
-    txt: str = " ".join(unicode(x) for x in a) + ka.get("end", "\n")
+    eol = ka.pop("end", "\n")
+    txt: str = " ".join(unicode(x) for x in a) + eol
     printed.append(txt)
     if not VT100:
         txt = ansi_re.sub("", txt)
 
-    print(txt, **ka)
+    print(txt, end="", **ka)
 
 
 def warn(msg: str) -> None:
     lprint("\033[1mwarning:\033[0;33m {}\033[0m\n".format(msg))
 
 
+def init_E(E: EnvParams) -> None:
+    # __init__ runs 18 times when oxidized; do expensive stuff here
+
+    def get_unixdir() -> str:
+        paths: list[tuple[Callable[..., str], str]] = [
+            (os.environ.get, "XDG_CONFIG_HOME"),
+            (os.path.expanduser, "~/.config"),
+            (os.environ.get, "TMPDIR"),
+            (os.environ.get, "TEMP"),
+            (os.environ.get, "TMP"),
+            (unicode, "/tmp"),
+        ]
+        for chk in [os.listdir, os.mkdir]:
+            for pf, pa in paths:
+                try:
+                    p = pf(pa)
+                    # print(chk.__name__, p, pa)
+                    if not p or p.startswith("~"):
+                        continue
+
+                    p = os.path.normpath(p)
+                    chk(p)  # type: ignore
+                    p = os.path.join(p, "copyparty")
+                    if not os.path.isdir(p):
+                        os.mkdir(p)
+
+                    return p
+                except:
+                    pass
+
+        raise Exception("could not find a writable path for config")
+
+    def _unpack() -> str:
+        import atexit
+        import tarfile
+        import tempfile
+        from importlib.resources import open_binary
+
+        td = tempfile.TemporaryDirectory(prefix="")
+        atexit.register(td.cleanup)
+        tdn = td.name
+
+        with open_binary("copyparty", "z.tar") as tgz:
+            with tarfile.open(fileobj=tgz) as tf:
+                tf.extractall(tdn)
+
+        return tdn
+
+    try:
+        E.mod = os.path.dirname(os.path.realpath(__file__))
+        if E.mod.endswith("__init__"):
+            E.mod = os.path.dirname(E.mod)
+    except:
+        if not E.ox:
+            raise
+
+        E.mod = _unpack()
+
+    if sys.platform == "win32":
+        E.cfg = os.path.normpath(os.environ["APPDATA"] + "/copyparty")
+    elif sys.platform == "darwin":
+        E.cfg = os.path.expanduser("~/Library/Preferences/copyparty")
+    else:
+        E.cfg = get_unixdir()
+
+    E.cfg = E.cfg.replace("\\", "/")
+    try:
+        os.makedirs(E.cfg)
+    except:
+        if not os.path.isdir(E.cfg):
+            raise
+
+
 def ensure_locale() -> None:
     for x in [
         "en_US.UTF-8",
@@ -127,7 +213,7 @@ def ensure_locale() -> None:
     ]:
         try:
             locale.setlocale(locale.LC_ALL, x)
-            lprint("Locale:", x)
+            lprint("Locale: {}\n".format(x))
             break
         except:
             continue
@@ -275,7 +361,7 @@ def disable_quickedit() -> None:
                 raise ctypes.WinError(err)  # type: ignore
         return args
 
-    k32 = ctypes.WinDLL("kernel32", use_last_error=True)  # type: ignore
+    k32 = ctypes.WinDLL(str("kernel32"), use_last_error=True)  # type: ignore
     if PY2:
         wintypes.LPDWORD = ctypes.POINTER(wintypes.DWORD)
 
@@ -311,7 +397,17 @@ def disable_quickedit() -> None:
             cmode(True, mode | 4)
 
 
-def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
+def showlic() -> None:
+    p = os.path.join(E.mod, "res", "COPYING.txt")
+    if not os.path.exists(p):
+        print("no relevant license info to display")
+        return
+
+    with open(p, "rb") as f:
+        print(f.read().decode("utf-8", "replace"))
+
+
+def run_argparse(argv: list[str], formatter: Any, retry: bool) -> argparse.Namespace:
     ap = argparse.ArgumentParser(
         formatter_class=formatter,
         prog="copyparty",
@@ -323,7 +419,7 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     except:
         fk_salt = "hunter2"
 
-    cores = os.cpu_count() if hasattr(os, "cpu_count") else 4
+    hcores = min(CORES, 3)  # 4% faster than 4+ on py3.9 @ r5-4500U
 
     sects = [
         [
@@ -375,6 +471,7 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
             \033[0muploads, general:
               \033[36mnodupe\033[35m rejects existing files (instead of symlinking them)
               \033[36mnosub\033[35m forces all uploads into the top folder of the vfs
+              \033[36mmagic$\033[35m enables filetype detection for nameless uploads
               \033[36mgz\033[35m allows server-side gzip of uploads with ?gz (also c,xz)
               \033[36mpk\033[35m forces server-side compression, optional arg: xz,9
 
@@ -382,6 +479,7 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
               \033[36mmaxn=250,600\033[35m max 250 uploads over 15min
               \033[36mmaxb=1g,300\033[35m max 1 GiB over 5min (suffixes: b, k, m, g)
               \033[36msz=1k-3m\033[35m allow filesizes between 1 KiB and 3MiB
+              \033[36mdf=1g\033[35m ensure 1 GiB free disk space
 
             \033[0mupload rotation:
             (moves all uploads into the specified folder structure)
@@ -394,11 +492,15 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
               \033[36md2ts\033[35m disables metadata collection for existing files
               \033[36md2ds\033[35m disables onboot indexing, overrides -e2ds*
               \033[36md2t\033[35m disables metadata collection, overrides -e2t*
+              \033[36md2v\033[35m disables file verification, overrides -e2v*
               \033[36md2d\033[35m disables all database stuff, overrides -e2*
-              \033[36mnohash=\\.iso$\033[35m skips hashing file contents if path matches *.iso
-              \033[36mnoidx=\\.iso$\033[35m fully ignores the contents at paths matching *.iso
               \033[36mhist=/tmp/cdb\033[35m puts thumbnails and indexes at that location
               \033[36mscan=60\033[35m scan for new files every 60sec, same as --re-maxage
+              \033[36mnohash=\\.iso$\033[35m skips hashing file contents if path matches *.iso
+              \033[36mnoidx=\\.iso$\033[35m fully ignores the contents at paths matching *.iso
+              \033[36mnoforget$\033[35m don't forget files when deleted from disk
+              \033[36mxdev\033[35m do not descend into other filesystems
+              \033[36mxvol\033[35m skip symlinks leaving the volume root
 
             \033[0mdatabase, audio tags:
             "mte", "mth", "mtp", "mtm" all work the same as -mte, -mth, ...
@@ -471,18 +573,25 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
     ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-form POSTs; see --help-urlform")
     ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="window title, for example '$ip-10.1.2.' or '$ip-'")
+    ap2.add_argument("--license", action="store_true", help="show licenses and exit")
+    ap2.add_argument("--version", action="store_true", help="show versions and exit")
 
     ap2 = ap.add_argument_group('upload options')
     ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads, hiding them from clients unless -ed")
-    ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="windows-only: minimum size of incoming uploads through up2k before they are made into sparse files")
+    ap2.add_argument("--plain-ip", action="store_true", help="when avoiding filename collisions by appending the uploader's ip to the filename: append the plaintext ip instead of salting and hashing the ip")
     ap2.add_argument("--unpost", metavar="SEC", type=int, default=3600*12, help="grace period where uploads can be deleted by the uploader, even without delete permissions; 0=disabled")
+    ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without -e2d; roughly 1 MiB RAM per 600")
     ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload")
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even if copyparty thinks you're better off without -- probably useful on nfs and cow filesystems (zfs, btrfs)")
     ap2.add_argument("--hardlink", action="store_true", help="prefer hardlinks instead of symlinks when possible (within same filesystem)")
     ap2.add_argument("--never-symlink", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made")
     ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead")
-    ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without -e2d; roughly 1 MiB RAM per 600")
+    ap2.add_argument("--magic", action="store_true", help="enable filetype detection on nameless uploads")
+    ap2.add_argument("--df", metavar="GiB", type=float, default=0, help="ensure GiB free disk space by rejecting upload requests")
+    ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="windows-only: minimum size of incoming uploads through up2k before they are made into sparse files")
     ap2.add_argument("--turbo", metavar="LVL", type=int, default=0, help="configure turbo-mode in up2k client; 0 = off and warn if enabled, 1 = off, 2 = on, 3 = on and disable datecheck")
+    ap2.add_argument("--u2sort", metavar="TXT", type=u, default="s", help="upload order; s=smallest-first, n=alphabetical, fs=force-s, fn=force-n -- alphabetical is a bit slower on fiber/LAN but makes it easier to eyeball if everything went fine")
+    ap2.add_argument("--write-uplog", action="store_true", help="write POST reports to textfiles in working-directory")
 
     ap2 = ap.add_argument_group('network options')
     ap2.add_argument("-i", metavar="IP", type=u, default="0.0.0.0", help="ip to bind (comma-sep.)")
@@ -519,7 +628,7 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
 
     ap2 = ap.add_argument_group('safety options')
     ap2.add_argument("-s", action="count", default=0, help="increase safety: Disable thumbnails / potentially dangerous software (ffmpeg/pillow/vips), hide partial uploads, avoid crawlers.\n └─Alias of\033[32m --dotpart --no-thumb --no-mtag-ff --no-robots --force-js")
-    ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, 404 on 403.\n └─Alias of\033[32m -s --no-dot-mv --no-dot-ren --unpost=0 --no-del --no-mv --hardlink --vague-403 -nih")
+    ap2.add_argument("-ss", action="store_true", help="further increase safety: Prevent js-injection, accidental move/delete, broken symlinks, 404 on 403, ban on excessive 404s.\n └─Alias of\033[32m -s --no-dot-mv --no-dot-ren --unpost=0 --no-del --no-mv --hardlink --vague-403 --ban-404=50,60,1440 -nih")
     ap2.add_argument("-sss", action="store_true", help="further increase safety: Enable logging to disk, scan for dangerous symlinks.\n └─Alias of\033[32m -ss -lo=cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz --ls=**,*,ln,p,r")
     ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="do a sanity/safety check of all volumes on startup; arguments USER,VOL,FLAGS; example [**,*,ln,p,r]")
     ap2.add_argument("--salt", type=u, default="hunter2", help="up2k file-hash salt; used to generate unpredictable internal identifiers for uploads -- doesn't really matter")
@@ -532,10 +641,13 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     ap2.add_argument("--force-js", action="store_true", help="don't send folder listings as HTML, force clients to use the embedded json instead -- slight protection against misbehaving search engines which ignore --no-robots")
     ap2.add_argument("--no-robots", action="store_true", help="adds http and html headers asking search engines to not index anything")
     ap2.add_argument("--logout", metavar="H", type=float, default="8086", help="logout clients after H hours of inactivity (0.0028=10sec, 0.1=6min, 24=day, 168=week, 720=month, 8760=year)")
+    ap2.add_argument("--ban-pw", metavar="N,W,B", type=u, default="9,60,1440", help="more than N wrong passwords in W minutes = ban for B minutes (disable with \"no\")")
+    ap2.add_argument("--ban-404", metavar="N,W,B", type=u, default="no", help="hitting more than N 404's in W minutes = ban for B minutes (disabled by default since turbo-up2k counts as 404s)")
 
-    ap2 = ap.add_argument_group('yolo options')
+    ap2 = ap.add_argument_group('shutdown options')
     ap2.add_argument("--ign-ebind", action="store_true", help="continue running even if it's impossible to listen on some of the requested endpoints")
     ap2.add_argument("--ign-ebind-all", action="store_true", help="continue running even if it's impossible to receive connections at all")
+    ap2.add_argument("--exit", metavar="WHEN", type=u, default="", help="shutdown after WHEN has finished; for example 'idx' will do volume indexing + metadata analysis")
 
     ap2 = ap.add_argument_group('logging options')
     ap2.add_argument("-q", action="store_true", help="quiet")
@@ -556,7 +668,7 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     ap2.add_argument("--no-athumb", action="store_true", help="disable audio thumbnails (spectrograms)")
     ap2.add_argument("--no-vthumb", action="store_true", help="disable video thumbnails")
     ap2.add_argument("--th-size", metavar="WxH", default="320x256", help="thumbnail res")
-    ap2.add_argument("--th-mt", metavar="CORES", type=int, default=cores, help="num cpu cores to use for generating thumbnails")
+    ap2.add_argument("--th-mt", metavar="CORES", type=int, default=CORES, help="num cpu cores to use for generating thumbnails")
     ap2.add_argument("--th-convt", metavar="SEC", type=int, default=60, help="conversion timeout in seconds")
     ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image")
     ap2.add_argument("--th-dec", metavar="LIBS", default="vips,pil,ff", help="image decoders, in order of preference")
@@ -585,11 +697,20 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     ap2.add_argument("-e2d", action="store_true", help="enable up2k database, making files searchable + enables upload deduplocation")
     ap2.add_argument("-e2ds", action="store_true", help="scan writable folders for new files on startup; sets -e2d")
     ap2.add_argument("-e2dsa", action="store_true", help="scans all folders on startup; sets -e2ds")
+    ap2.add_argument("-e2v", action="store_true", help="verify file integrity; rehash all files and compare with db")
+    ap2.add_argument("-e2vu", action="store_true", help="on hash mismatch: update the database with the new hash")
+    ap2.add_argument("-e2vp", action="store_true", help="on hash mismatch: panic and quit copyparty")
     ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume data (db, thumbs)")
     ap2.add_argument("--no-hash", metavar="PTN", type=u, help="regex: disable hashing of matching paths during e2ds folder scans")
     ap2.add_argument("--no-idx", metavar="PTN", type=u, help="regex: disable indexing of matching paths during e2ds folder scans")
+    ap2.add_argument("--no-dhash", action="store_true", help="disable rescan acceleration; do full database integrity check -- makes the db ~5%% smaller and bootup/rescans 3~10x slower")
+    ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice")
+    ap2.add_argument("--xdev", action="store_true", help="do not descend into other filesystems (symlink or bind-mount to another HDD, ...)")
+    ap2.add_argument("--xvol", action="store_true", help="skip symlinks leaving the volume root")
+    ap2.add_argument("--hash-mt", metavar="CORES", type=int, default=hcores, help="num cpu cores to use for file hashing; set 0 or 1 for single-core hashing")
     ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval, 0=off, can be set per-volume with the 'scan' volflag")
-    ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline -- terminate searches running for more than SEC seconds")
+    ap2.add_argument("--db-act", metavar="SEC", type=float, default=10, help="defer any scheduled volume reindexing until SEC seconds after last db write (uploads, renames, ...)")
+    ap2.add_argument("--srch-time", metavar="SEC", type=int, default=45, help="search deadline -- terminate searches running for more than SEC seconds")
     ap2.add_argument("--srch-hits", metavar="N", type=int, default=7999, help="max search results to allow clients to fetch; 125 results will be shown initially")
 
     ap2 = ap.add_argument_group('metadata db options')
@@ -598,19 +719,22 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     ap2.add_argument("-e2tsr", action="store_true", help="delete all metadata from DB and do a full rescan; sets -e2ts")
     ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead; will catch more tags")
     ap2.add_argument("--no-mtag-ff", action="store_true", help="never use FFprobe as tag reader; is probably safer")
-    ap2.add_argument("--mtag-mt", metavar="CORES", type=int, default=cores, help="num cpu cores to use for tag scanning")
+    ap2.add_argument("--mtag-to", metavar="SEC", type=int, default=60, help="timeout for ffprobe tag-scan")
+    ap2.add_argument("--mtag-mt", metavar="CORES", type=int, default=CORES, help="num cpu cores to use for tag scanning")
     ap2.add_argument("--mtag-v", action="store_true", help="verbose tag scanning; print errors from mtp subprocesses and such")
+    ap2.add_argument("--mtag-vv", action="store_true", help="debug mtp settings")
     ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
     ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
-        default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,res,.fps,ahash,vhash")
+        default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,vc,ac,fmt,res,.fps,ahash,vhash")
     ap2.add_argument("-mth", metavar="M,M,M", type=u, help="tags to hide by default (comma-sep.)",
-        default=".vq,.aq,vc,ac,res,.fps")
+        default=".vq,.aq,vc,ac,fmt,res,.fps")
     ap2.add_argument("-mtp", metavar="M=[f,]BIN", type=u, action="append", help="read tag M using program BIN to parse the file")
 
     ap2 = ap.add_argument_group('ui options')
     ap2.add_argument("--lang", metavar="LANG", type=u, default="eng", help="language")
     ap2.add_argument("--theme", metavar="NUM", type=int, default=0, help="default theme to use")
     ap2.add_argument("--themes", metavar="NUM", type=int, default=8, help="number of themes installed")
+    ap2.add_argument("--favico", metavar="TXT", type=u, default="c 000 none" if retry else "🎉 000 none", help="favicon text [ foreground [ background ] ], set blank to disable")
     ap2.add_argument("--js-browser", metavar="L", type=u, help="URL to additional JS to include")
     ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include")
     ap2.add_argument("--html-head", metavar="TXT", type=u, default="", help="text to append to the <head> of all HTML pages")
@@ -623,8 +747,9 @@ def run_argparse(argv: list[str], formatter: Any) -> argparse.Namespace:
     ap2.add_argument("--no-scandir", action="store_true", help="disable scandir; instead using listdir + stat on each file")
     ap2.add_argument("--no-fastboot", action="store_true", help="wait for up2k indexing before starting the httpd")
     ap2.add_argument("--no-htp", action="store_true", help="disable httpserver threadpool, create threads as-needed instead")
-    ap2.add_argument("--stackmon", metavar="P,S", type=u, help="write stacktrace to Path every S second")
+    ap2.add_argument("--stackmon", metavar="P,S", type=u, help="write stacktrace to Path every S second, for example --stackmon=./st/%%Y-%%m/%%d/%%H%%M.xz,60")
     ap2.add_argument("--log-thrs", metavar="SEC", type=float, help="list active threads every SEC")
+    ap2.add_argument("--log-fk", metavar="REGEX", type=u, default="", help="log filekey params for files where path matches REGEX; '.' (a single dot) = all files")
     # fmt: on
 
     ap2 = ap.add_argument_group("help sections")
@@ -647,13 +772,28 @@ def main(argv: Optional[list[str]] = None) -> None:
     if WINDOWS:
         os.system("rem")  # enables colors
 
+    init_E(E)
     if argv is None:
         argv = sys.argv
 
-    desc = py_desc().replace("[", "\033[1;30m[")
+    f = '\033[36mcopyparty v{} "\033[35m{}\033[36m" ({})\n{}\033[0;36m\n   sqlite v{} | jinja2 v{} | pyftpd v{}\n\033[0m'
+    f = f.format(
+        S_VERSION,
+        CODENAME,
+        S_BUILD_DT,
+        py_desc().replace("[", "\033[1;30m["),
+        SQLITE_VER,
+        JINJA_VER,
+        PYFTPD_VER,
+    )
+    lprint(f)
 
-    f = '\033[36mcopyparty v{} "\033[35m{}\033[36m" ({})\n{}\033[0m\n'
-    lprint(f.format(S_VERSION, CODENAME, S_BUILD_DT, desc))
+    if "--version" in argv:
+        sys.exit(0)
+
+    if "--license" in argv:
+        showlic()
+        sys.exit(0)
 
     ensure_locale()
     if HAVE_SSL:
@@ -682,15 +822,18 @@ def main(argv: Optional[list[str]] = None) -> None:
     except:
         pass
 
-    for fmtr in [RiceFormatter, Dodge11874, BasicDodge11874]:
+    retry = False
+    for fmtr in [RiceFormatter, RiceFormatter, Dodge11874, BasicDodge11874]:
         try:
-            al = run_argparse(argv, fmtr)
+            al = run_argparse(argv, fmtr, retry)
         except SystemExit:
             raise
         except:
+            retry = True
             lprint("\n[ {} ]:\n{}\n".format(fmtr, min_ex()))
 
     assert al
+    al.E = E  # __init__ is not shared when oxidized
 
     if WINDOWS and not al.keep_qem:
         try:
@@ -752,6 +895,12 @@ def main(argv: Optional[list[str]] = None) -> None:
     except:
         raise Exception("invalid value for -p")
 
+    for arg, kname, okays in [["--u2sort", "u2sort", "s n fs fn"]]:
+        val = unicode(getattr(al, kname))
+        if val not in okays.split():
+            zs = "argument {} cannot be '{}'; try one of these: {}"
+            raise Exception(zs.format(arg, val, okays))
+
     if HAVE_SSL:
         if al.ssl_ver:
             configure_ssl_ver(al)

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 3, 1)
-CODENAME = "god dag"
-BUILD_DT = (2022, 6, 16)
+VERSION = (1, 4, 2)
+CODENAME = "mostly reliable"
+BUILD_DT = (2022, 9, 25)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -17,9 +17,12 @@ from .bos import bos
 from .util import (
     IMPLICATIONS,
     META_NOBOTS,
+    SQLITE_VER,
     Pebkac,
     absreal,
     fsenc,
+    get_df,
+    humansize,
     relchk,
     statdir,
     uncyg,
@@ -72,15 +75,23 @@ class AXS(object):
 
 
 class Lim(object):
-    def __init__(self) -> None:
+    def __init__(self, log_func: Optional["RootLogger"]) -> None:
+        self.log_func = log_func
+
+        self.reg: Optional[dict[str, dict[str, Any]]] = None  # up2k registry
+
         self.nups: dict[str, list[float]] = {}  # num tracker
         self.bups: dict[str, list[tuple[float, int]]] = {}  # byte tracker list
         self.bupc: dict[str, int] = {}  # byte tracker cache
 
         self.nosub = False  # disallow subdirectories
 
-        self.smin = -1  # filesize min
-        self.smax = -1  # filesize max
+        self.dfl = 0  # free disk space limit
+        self.dft = 0  # last-measured time
+        self.dfv = 0  # currently free
+
+        self.smin = 0  # filesize min
+        self.smax = 0  # filesize max
 
         self.bwin = 0  # bytes window
         self.bmax = 0  # bytes max
@@ -92,18 +103,34 @@ class Lim(object):
         self.rotf = ""  # rot datefmt
         self.rot_re = re.compile("")  # rotf check
 
+    def log(self, msg: str, c: Union[int, str] = 0) -> None:
+        if self.log_func:
+            self.log_func("up-lim", msg, c)
+
     def set_rotf(self, fmt: str) -> None:
         self.rotf = fmt
         r = re.escape(fmt).replace("%Y", "[0-9]{4}").replace("%j", "[0-9]{3}")
         r = re.sub("%[mdHMSWU]", "[0-9]{2}", r)
         self.rot_re = re.compile("(^|/)" + r + "$")
 
-    def all(self, ip: str, rem: str, sz: float, abspath: str) -> tuple[str, str]:
+    def all(
+        self,
+        ip: str,
+        rem: str,
+        sz: int,
+        abspath: str,
+        reg: Optional[dict[str, dict[str, Any]]] = None,
+    ) -> tuple[str, str]:
+        if reg is not None and self.reg is None:
+            self.reg = reg
+            self.dft = 0
+
         self.chk_nup(ip)
         self.chk_bup(ip)
         self.chk_rem(rem)
         if sz != -1:
             self.chk_sz(sz)
+            self.chk_df(abspath, sz)  # side effects; keep last-ish
 
         ap2, vp2 = self.rot(abspath)
         if abspath == ap2:
@@ -111,13 +138,33 @@ class Lim(object):
 
         return ap2, ("{}/{}".format(rem, vp2) if rem else vp2)
 
-    def chk_sz(self, sz: float) -> None:
-        if self.smin != -1 and sz < self.smin:
+    def chk_sz(self, sz: int) -> None:
+        if sz < self.smin:
             raise Pebkac(400, "file too small")
 
-        if self.smax != -1 and sz > self.smax:
+        if self.smax and sz > self.smax:
             raise Pebkac(400, "file too big")
 
+    def chk_df(self, abspath: str, sz: int, already_written: bool = False) -> None:
+        if not self.dfl:
+            return
+
+        if self.dft < time.time():
+            self.dft = int(time.time()) + 300
+            self.dfv = get_df(abspath)[0] or 0
+            for j in list(self.reg.values()) if self.reg else []:
+                self.dfv -= int(j["size"] / len(j["hash"]) * len(j["need"]))
+
+            if already_written:
+                sz = 0
+
+        if self.dfv - sz < self.dfl:
+            self.dft = min(self.dft, int(time.time()) + 10)
+            t = "server HDD is full; {} free, need {}"
+            raise Pebkac(500, t.format(humansize(self.dfv - self.dfl), humansize(sz)))
+
+        self.dfv -= int(sz)
+
     def chk_rem(self, rem: str) -> None:
         if self.nosub and rem:
             raise Pebkac(500, "no subdirectories allowed")
@@ -226,7 +273,7 @@ class VFS(object):
 
     def __init__(
         self,
-        log: Optional[RootLogger],
+        log: Optional["RootLogger"],
         realpath: str,
         vpath: str,
         axs: AXS,
@@ -569,7 +616,7 @@ class AuthSrv(object):
     def __init__(
         self,
         args: argparse.Namespace,
-        log_func: Optional[RootLogger],
+        log_func: Optional["RootLogger"],
         warn_anonwrite: bool = True,
     ) -> None:
         self.args = args
@@ -661,7 +708,7 @@ class AuthSrv(object):
                     raise Exception('invalid mountpoint "{}"'.format(vol_dst))
 
                 # cfg files override arguments and previous files
-                vol_src = bos.path.abspath(vol_src)
+                vol_src = absreal(vol_src)
                 vol_dst = vol_dst.strip("/")
                 self._map_volume(vol_src, vol_dst, mount, daxs, mflags)
                 continue
@@ -682,12 +729,12 @@ class AuthSrv(object):
         self, lvl: str, uname: str, axs: AXS, flags: dict[str, Any]
     ) -> None:
         if lvl.strip("crwmdg"):
-            raise Exception("invalid volume flag: {},{}".format(lvl, uname))
+            raise Exception("invalid volflag: {},{}".format(lvl, uname))
 
         if lvl == "c":
             cval: Union[bool, str] = True
             try:
-                # volume flag with arguments, possibly with a preceding list of bools
+                # volflag with arguments, possibly with a preceding list of bools
                 uname, cval = uname.split("=", 1)
             except:
                 # just one or more bools
@@ -772,7 +819,7 @@ class AuthSrv(object):
                     src = uncyg(src)
 
                 # print("\n".join([src, dst, perms]))
-                src = bos.path.abspath(src)
+                src = absreal(src)
                 dst = dst.strip("/")
                 self._map_volume(src, dst, mount, daxs, mflags)
 
@@ -801,7 +848,7 @@ class AuthSrv(object):
         if not mount:
             # -h says our defaults are CWD at root and read/write for everyone
             axs = AXS(["*"], ["*"], None, None)
-            vfs = VFS(self.log_func, bos.path.abspath("."), "", axs, {})
+            vfs = VFS(self.log_func, absreal("."), "", axs, {})
         elif "" not in mount:
             # there's volumes but no root; make root inaccessible
             vfs = VFS(self.log_func, "", "", AXS(), {})
@@ -917,13 +964,20 @@ class AuthSrv(object):
         vfs.histtab = {zv.realpath: zv.histpath for zv in vfs.all_vols.values()}
 
         for vol in vfs.all_vols.values():
-            lim = Lim()
+            lim = Lim(self.log_func)
             use = False
 
             if vol.flags.get("nosub"):
                 use = True
                 lim.nosub = True
 
+            zs = vol.flags.get("df") or (
+                "{}g".format(self.args.df) if self.args.df else ""
+            )
+            if zs:
+                use = True
+                lim.dfl = unhumanize(zs)
+
             zs = vol.flags.get("sz")
             if zs:
                 use = True
@@ -976,10 +1030,15 @@ class AuthSrv(object):
                 vol.flags["dathumb"] = True
                 vol.flags["dithumb"] = True
 
+        have_fk = False
         for vol in vfs.all_vols.values():
             fk = vol.flags.get("fk")
             if fk:
                 vol.flags["fk"] = int(fk) if fk is not True else 8
+                have_fk = True
+
+        if have_fk and re.match(r"^[0-9\.]+$", self.args.fk_salt):
+            self.log("filekey salt: {}".format(self.args.fk_salt))
 
         for vol in vfs.all_vols.values():
             if "pk" in vol.flags and "gz" not in vol.flags and "xz" not in vol.flags:
@@ -1008,10 +1067,14 @@ class AuthSrv(object):
                 if ptn:
                     vol.flags[vf] = re.compile(ptn)
 
-            for k in ["e2t", "e2ts", "e2tsr"]:
+            for k in ["e2t", "e2ts", "e2tsr", "e2v", "e2vu", "e2vp", "xdev", "xvol"]:
                 if getattr(self.args, k):
                     vol.flags[k] = True
 
+            for ga, vf in [["no_forget", "noforget"], ["magic", "magic"]]:
+                if getattr(self.args, ga):
+                    vol.flags[vf] = True
+
             for k1, k2 in IMPLICATIONS:
                 if k1 in vol.flags:
                     vol.flags[k2] = True
@@ -1026,11 +1089,11 @@ class AuthSrv(object):
             if "mth" not in vol.flags:
                 vol.flags["mth"] = self.args.mth
 
-            # append parsers from argv to volume-flags
+            # append parsers from argv to volflags
             self._read_volflag(vol.flags, "mtp", self.args.mtp, True)
 
             # d2d drops all database features for a volume
-            for grp, rm in [["d2d", "e2d"], ["d2t", "e2t"]]:
+            for grp, rm in [["d2d", "e2d"], ["d2t", "e2t"], ["d2d", "e2v"]]:
                 if not vol.flags.get(grp, False):
                     continue
 
@@ -1052,6 +1115,22 @@ class AuthSrv(object):
 
                 vol.flags = {k: v for k, v in vol.flags.items() if not k.startswith(rm)}
 
+            for grp, rm in [["d2v", "e2v"]]:
+                if not vol.flags.get(grp, False):
+                    continue
+
+                vol.flags = {k: v for k, v in vol.flags.items() if not k.startswith(rm)}
+
+            ints = ["lifetime"]
+            for k in list(vol.flags):
+                if k in ints:
+                    vol.flags[k] = int(vol.flags[k])
+
+            if "lifetime" in vol.flags and "e2d" not in vol.flags:
+                t = 'removing lifetime config from volume "/{}" because e2d is disabled'
+                self.log(t.format(vol.vpath), 1)
+                del vol.flags["lifetime"]
+
             # verify tags mentioned by -mt[mp] are used by -mte
             local_mtp = {}
             local_only_mtp = {}
@@ -1083,7 +1162,7 @@ class AuthSrv(object):
 
             for mtp in local_only_mtp:
                 if mtp not in local_mte:
-                    t = 'volume "/{}" defines metadata tag "{}", but doesnt use it in "-mte" (or with "cmte" in its volume-flags)'
+                    t = 'volume "/{}" defines metadata tag "{}", but doesnt use it in "-mte" (or with "cmte" in its volflags)'
                     self.log(t.format(vol.vpath, mtp), 1)
                     errors = True
 
@@ -1092,7 +1171,7 @@ class AuthSrv(object):
         tags = [y for x in tags for y in x.split(",")]
         for mtp in tags:
             if mtp not in all_mte:
-                t = 'metadata tag "{}" is defined by "-mtm" or "-mtp", but is not used by "-mte" (or by any "cmte" volume-flag)'
+                t = 'metadata tag "{}" is defined by "-mtm" or "-mtp", but is not used by "-mte" (or by any "cmte" volflag)'
                 self.log(t.format(mtp), 1)
                 errors = True
 
@@ -1101,6 +1180,7 @@ class AuthSrv(object):
 
         vfs.bubble_flags()
 
+        have_e2d = False
         t = "volumes and permissions:\n"
         for zv in vfs.all_vols.values():
             if not self.warn_anonwrite:
@@ -1118,17 +1198,28 @@ class AuthSrv(object):
                 u = ", ".join("\033[35meverybody\033[0m" if x == "*" else x for x in u)
                 u = u if u else "\033[36m--none--\033[0m"
                 t += "\n|  {}:  {}".format(txt, u)
+
+            if "e2d" in zv.flags:
+                have_e2d = True
+
             t += "\n"
 
-        if self.warn_anonwrite and not self.args.no_voldump:
-            self.log(t)
+        if self.warn_anonwrite:
+            if not self.args.no_voldump:
+                self.log(t)
+
+            if have_e2d:
+                t = self.chk_sqlite_threadsafe()
+                if t:
+                    self.log("\n\033[{}\033[0m\n".format(t))
 
         try:
             zv, _ = vfs.get("/", "*", False, True)
             if self.warn_anonwrite and os.getcwd() == zv.realpath:
-                self.warn_anonwrite = False
-                t = "anyone can read/write the current directory: {}\n"
+                t = "anyone can write to the current directory: {}\n"
                 self.log(t.format(zv.realpath), c=1)
+
+            self.warn_anonwrite = False
         except Pebkac:
             self.warn_anonwrite = True
 
@@ -1142,6 +1233,23 @@ class AuthSrv(object):
             if pwds:
                 self.re_pwd = re.compile("=(" + "|".join(pwds) + ")([]&; ]|$)")
 
+    def chk_sqlite_threadsafe(self) -> str:
+        v = SQLITE_VER[-1:]
+
+        if v == "1":
+            # threadsafe (linux, windows)
+            return ""
+
+        if v == "2":
+            # module safe, connections unsafe (macos)
+            return "33m  your sqlite3 was compiled with reduced thread-safety;\n   database features (-e2d, -e2t) SHOULD be fine\n    but MAY cause database-corruption and crashes"
+
+        if v == "0":
+            # everything unsafe
+            return "31m  your sqlite3 was compiled WITHOUT thread-safety!\n   database features (-e2d, -e2t) will PROBABLY cause crashes!"
+
+        return "36m  cannot verify sqlite3 thread-safety; strange but probably fine"
+
     def dbg_ls(self) -> None:
         users = self.args.ls
         vol = "*"

copyparty/broker_mp.py

@@ -6,7 +6,7 @@ import time
 
 import queue
 
-from .__init__ import TYPE_CHECKING
+from .__init__ import CORES, TYPE_CHECKING
 from .broker_mpw import MpWorker
 from .broker_util import try_exec
 from .util import mp
@@ -44,7 +44,7 @@ class BrokerMp(object):
         self.procs = []
         self.mutex = threading.Lock()
 
-        self.num_workers = self.args.j or mp.cpu_count()
+        self.num_workers = self.args.j or CORES
         self.log("broker", "booting {} subprocesses".format(self.num_workers))
         for n in range(1, self.num_workers + 1):
             q_pend: queue.Queue[tuple[int, str, list[Any]]] = mp.Queue(1)

copyparty/broker_mpw.py

@@ -2,6 +2,7 @@
 from __future__ import print_function, unicode_literals
 
 import argparse
+import os
 import signal
 import sys
 import threading
@@ -11,7 +12,7 @@ import queue
 from .authsrv import AuthSrv
 from .broker_util import BrokerCli, ExceptionalQueue
 from .httpsrv import HttpSrv
-from .util import FAKE_MP
+from .util import FAKE_MP, HMaccas
 
 try:
     from types import FrameType
@@ -54,6 +55,7 @@ class MpWorker(BrokerCli):
         self.asrv = AuthSrv(args, None, False)
 
         # instantiate all services here (TODO: inheritance?)
+        self.iphash = HMaccas(os.path.join(self.args.E.cfg, "iphash"), 8)
         self.httpsrv = HttpSrv(self, n)
 
         # on winxp and some other platforms,

copyparty/broker_thr.py

@@ -1,11 +1,13 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
+import os
 import threading
 
 from .__init__ import TYPE_CHECKING
 from .broker_util import BrokerCli, ExceptionalQueue, try_exec
 from .httpsrv import HttpSrv
+from .util import HMaccas
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -31,6 +33,7 @@ class BrokerThr(BrokerCli):
         self.num_workers = 1
 
         # instantiate all services here (TODO: inheritance?)
+        self.iphash = HMaccas(os.path.join(self.args.E.cfg, "iphash"), 8)
         self.httpsrv = HttpSrv(self, None)
         self.reload = self.noop
 

copyparty/broker_util.py

@@ -8,7 +8,7 @@ from queue import Queue
 
 from .__init__ import TYPE_CHECKING
 from .authsrv import AuthSrv
-from .util import Pebkac
+from .util import HMaccas, Pebkac
 
 try:
     from typing import Any, Optional, Union
@@ -42,10 +42,11 @@ class BrokerCli(object):
     """
 
     def __init__(self) -> None:
-        self.log: RootLogger = None
+        self.log: "RootLogger" = None
         self.args: argparse.Namespace = None
         self.asrv: AuthSrv = None
         self.httpsrv: "HttpSrv" = None
+        self.iphash: HMaccas = None
 
     def ask(self, dest: str, *args: Any) -> ExceptionalQueue:
         return ExceptionalQueue(1)

copyparty/fsutil.py

@@ -1,12 +1,13 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import ctypes
+import os
 import re
 import time
 
 from .__init__ import ANYWIN, MACOS
 from .authsrv import AXS, VFS
+from .bos import bos
 from .util import chkcmd, min_ex
 
 try:
@@ -18,28 +19,10 @@ except:
 
 
 class Fstab(object):
-    def __init__(self, log: RootLogger):
+    def __init__(self, log: "RootLogger"):
         self.log_func = log
 
-        self.no_sparse = set(
-            [
-                "fuse",  # termux-sdcard
-                "vfat",  # linux-efi
-                "fat32",
-                "fat16",
-                "fat12",
-                "fat-32",
-                "fat-16",
-                "fat-12",
-                "fat 32",
-                "fat 16",
-                "fat 12",
-                "exfat",
-                "ex-fat",
-                "ex fat",
-                "hpfs",  # macos
-            ]
-        )
+        self.trusted = False
         self.tab: Optional[VFS] = None
         self.cache: dict[str, str] = {}
         self.age = 0.0
@@ -47,8 +30,8 @@ class Fstab(object):
     def log(self, msg: str, c: Union[int, str] = 0) -> None:
         self.log_func("fstab", msg + "\033[K", c)
 
-    def get(self, path: str):
-        if time.time() - self.age > 600 or len(self.cache) > 9000:
+    def get(self, path: str) -> str:
+        if len(self.cache) > 9000:
             self.age = time.time()
             self.tab = None
             self.cache = {}
@@ -57,17 +40,14 @@ class Fstab(object):
         msg = "failed to determine filesystem at [{}]; assuming {}\n{}"
 
         if ANYWIN:
-            fs = "vfat"  # can smb do sparse files? gonna guess no
+            fs = "vfat"
             try:
-                # good enough
-                disk = path.split(":", 1)[0]
-                disk = "{}:\\".format(disk).lower()
-                assert len(disk) == 3
-                path = disk
+                path = self._winpath(path)
             except:
                 self.log(msg.format(path, fs, min_ex()), 3)
                 return fs
 
+        path = path.lstrip("/")
         try:
             return self.cache[path]
         except:
@@ -83,7 +63,20 @@ class Fstab(object):
         self.log("found {} at {}".format(fs, path))
         return fs
 
-    def build_tab(self):
+    def _winpath(self, path: str) -> str:
+        # try to combine volume-label + st_dev (vsn)
+        path = path.replace("/", "\\")
+        vid = path.split(":", 1)[0].strip("\\").split("\\", 1)[0]
+        try:
+            return "{}*{}".format(vid, bos.stat(path).st_dev)
+        except:
+            return vid
+
+    def build_fallback(self) -> None:
+        self.tab = VFS(self.log_func, "idk", "/", AXS(), {})
+        self.trusted = False
+
+    def build_tab(self) -> None:
         self.log("building tab")
 
         sptn = r"^.*? on (.*) type ([^ ]+) \(.*"
@@ -98,7 +91,8 @@ class Fstab(object):
             if not m:
                 continue
 
-            tab1.append(m.groups())
+            zs1, zs2 = m.groups()
+            tab1.append((str(zs1), str(zs2)))
 
         tab1.sort(key=lambda x: (len(x[0]), x[0]))
         path1, fs1 = tab1[0]
@@ -108,50 +102,53 @@ class Fstab(object):
 
         self.tab = tab
 
-    def get_unix(self, path: str):
+    def relabel(self, path: str, nval: str) -> None:
+        assert self.tab
+        self.cache = {}
+        if ANYWIN:
+            path = self._winpath(path)
+
+        path = path.lstrip("/")
+        ptn = re.compile(r"^[^\\/]*")
+        vn, rem = self.tab._find(path)
+        if not self.trusted:
+            # no mtab access; have to build as we go
+            if "/" in rem:
+                self.tab.add("idk", os.path.join(vn.vpath, rem.split("/")[0]))
+            if rem:
+                self.tab.add(nval, path)
+            else:
+                vn.realpath = nval
+
+            return
+
+        visit = [vn]
+        while visit:
+            vn = visit.pop()
+            vn.realpath = ptn.sub(nval, vn.realpath)
+            visit.extend(list(vn.nodes.values()))
+
+    def get_unix(self, path: str) -> str:
         if not self.tab:
-            self.build_tab()
+            try:
+                self.build_tab()
+                self.trusted = True
+            except:
+                # prisonparty or other restrictive environment
+                self.log("failed to build tab:\n{}".format(min_ex()), 3)
+                self.build_fallback()
 
-        return self.tab._find(path)[0].realpath.split("/")[0]
+        assert self.tab
+        ret = self.tab._find(path)[0]
+        if self.trusted or path == ret.vpath:
+            return ret.realpath.split("/")[0]
+        else:
+            return "idk"
 
-    def get_w32(self, path: str):
-        # list mountpoints: fsutil fsinfo drives
+    def get_w32(self, path: str) -> str:
+        if not self.tab:
+            self.build_fallback()
 
-        from ctypes.wintypes import BOOL, DWORD, LPCWSTR, LPDWORD, LPWSTR, MAX_PATH
-
-        def echk(rc, fun, args):
-            if not rc:
-                raise ctypes.WinError(ctypes.get_last_error())
-            return None
-
-        k32 = ctypes.WinDLL("kernel32", use_last_error=True)
-        k32.GetVolumeInformationW.errcheck = echk
-        k32.GetVolumeInformationW.restype = BOOL
-        k32.GetVolumeInformationW.argtypes = (
-            LPCWSTR,
-            LPWSTR,
-            DWORD,
-            LPDWORD,
-            LPDWORD,
-            LPDWORD,
-            LPWSTR,
-            DWORD,
-        )
-
-        bvolname = ctypes.create_unicode_buffer(MAX_PATH + 1)
-        bfstype = ctypes.create_unicode_buffer(MAX_PATH + 1)
-        serial = DWORD()
-        max_name_len = DWORD()
-        fs_flags = DWORD()
-
-        k32.GetVolumeInformationW(
-            path,
-            bvolname,
-            ctypes.sizeof(bvolname),
-            ctypes.byref(serial),
-            ctypes.byref(max_name_len),
-            ctypes.byref(fs_flags),
-            bfstype,
-            ctypes.sizeof(bfstype),
-        )
-        return bfstype.value
+        assert self.tab
+        ret = self.tab._find(path)[0]
+        return ret.realpath

copyparty/ftpd.py

@@ -56,7 +56,9 @@ class FtpAuth(DummyAuthorizer):
 
         handler.username = uname
 
-        if password and not uname:
+        if (password and not uname) or not (
+            asrv.vfs.aread.get(uname) or asrv.vfs.awrite.get(uname)
+        ):
             raise AuthenticationFailed("Authentication failed.")
 
     def get_home_dir(self, username: str) -> str:
@@ -356,7 +358,7 @@ class Ftpd(object):
                 print(t.format(sys.executable))
                 sys.exit(1)
 
-            h1.certfile = os.path.join(E.cfg, "cert.pem")
+            h1.certfile = os.path.join(self.args.E.cfg, "cert.pem")
             h1.tls_control_required = True
             h1.tls_data_required = True
 
@@ -391,7 +393,7 @@ class Ftpd(object):
             for h, lp in hs:
                 FTPServer((ip, int(lp)), h, ioloop)
 
-        thr = threading.Thread(target=ioloop.loop)
+        thr = threading.Thread(target=ioloop.loop, name="ftp")
         thr.daemon = True
         thr.start()
 

copyparty/httpcli.py

@@ -25,11 +25,11 @@ except:
     pass
 
 try:
-    import ctypes
+    from ipaddress import IPv6Address
 except:
     pass
 
-from .__init__ import ANYWIN, PY2, TYPE_CHECKING, WINDOWS, E, unicode
+from .__init__ import ANYWIN, PY2, TYPE_CHECKING, EnvParams, unicode
 from .authsrv import VFS  # typechk
 from .bos import bos
 from .star import StreamTar
@@ -47,7 +47,9 @@ from .util import (
     exclude_dotfiles,
     fsenc,
     gen_filekey,
+    gen_filekey_dbg,
     gencookie,
+    get_df,
     get_spd,
     guess_mime,
     gzip_orig_sz,
@@ -72,6 +74,7 @@ from .util import (
     unescape_cookie,
     unquote,
     unquotep,
+    vjoin,
     vol_san,
     vsplit,
     yieldfile,
@@ -106,12 +109,15 @@ class HttpCli(object):
         self.ip = conn.addr[0]
         self.addr: tuple[str, int] = conn.addr
         self.args = conn.args  # mypy404
+        self.E: EnvParams = self.args.E
         self.asrv = conn.asrv  # mypy404
         self.ico = conn.ico  # mypy404
         self.thumbcli = conn.thumbcli  # mypy404
         self.u2fh = conn.u2fh  # mypy404
         self.log_func = conn.log_func  # mypy404
         self.log_src = conn.log_src  # mypy404
+        self.bans = conn.hsrv.bans
+        self.gen_fk = self._gen_fk if self.args.log_fk else gen_filekey
         self.tls: bool = hasattr(self.s, "cipher")
 
         # placeholders; assigned by run()
@@ -124,7 +130,6 @@ class HttpCli(object):
         self.ua = " "
         self.is_rclone = False
         self.is_ancient = False
-        self.dip = " "
         self.ouparam: dict[str, str] = {}
         self.uparam: dict[str, str] = {}
         self.cookies: dict[str, str] = {}
@@ -181,9 +186,14 @@ class HttpCli(object):
         if rem.startswith("/") or rem.startswith("../") or "/../" in rem:
             raise Exception("that was close")
 
+    def _gen_fk(self, salt: str, fspath: str, fsize: int, inode: int) -> str:
+        return gen_filekey_dbg(salt, fspath, fsize, inode, self.log, self.args.log_fk)
+
     def j2s(self, name: str, **ka: Any) -> str:
         tpl = self.conn.hsrv.j2[name]
         ka["ts"] = self.conn.hsrv.cachebuster()
+        ka["lang"] = self.args.lang
+        ka["favico"] = self.args.favico
         ka["svcname"] = self.args.doctitle
         ka["html_head"] = self.html_head
         return tpl.render(**ka)  # type: ignore
@@ -260,7 +270,20 @@ class HttpCli(object):
 
                 self.log_src = self.conn.set_rproxy(self.ip)
 
-        self.dip = self.ip.replace(":", ".")
+        if self.bans:
+            ip = self.ip
+            if ":" in ip and not PY2:
+                ip = IPv6Address(ip).exploded[:-20]
+
+            if ip in self.bans:
+                ban = self.bans[ip] - time.time()
+                if ban < 0:
+                    self.log("client unbanned", 3)
+                    del self.bans[ip]
+                else:
+                    self.log("banned for {:.0f} sec".format(ban), 6)
+                    self.reply(b"thank you for playing", 403)
+                    return False
 
         if self.args.ihead:
             keys = self.args.ihead
@@ -378,26 +401,40 @@ class HttpCli(object):
                 if not self._check_nonfatal(pex, post):
                     self.keepalive = False
 
-                msg = str(ex) if pex == ex else min_ex()
-                self.log("{}\033[0m, {}".format(msg, self.vpath), 3)
+                em = str(ex)
+                msg = em if pex == ex else min_ex()
+                self.log(
+                    "{}\033[0m, {}".format(msg, self.vpath),
+                    6 if em.startswith("client d/c ") else 3,
+                )
 
-                msg = "{}\r\nURL: {}\r\n".format(str(ex), self.vpath)
+                msg = "{}\r\nURL: {}\r\n".format(em, self.vpath)
                 if self.hint:
                     msg += "hint: {}\r\n".format(self.hint)
 
+                if "database is locked" in em:
+                    self.conn.hsrv.broker.say("log_stacks")
+                    msg += "hint: important info in the server log\r\n"
+
                 msg = "<pre>" + html_escape(msg)
                 self.reply(msg.encode("utf-8", "replace"), status=pex.code, volsan=True)
                 return self.keepalive
             except Pebkac:
                 return False
 
+    def dip(self) -> str:
+        if self.args.plain_ip:
+            return self.ip.replace(":", ".")
+        else:
+            return self.conn.iphash.s(self.ip)
+
     def permit_caching(self) -> None:
         cache = self.uparam.get("cache")
         if cache is None:
             self.out_headers.update(NO_CACHE)
             return
 
-        n = "604800" if cache == "i" else cache or "69"
+        n = "604869" if cache == "i" else cache or "69"
         self.out_headers["Cache-Control"] = "max-age=" + n
 
     def k304(self) -> bool:
@@ -450,7 +487,13 @@ class HttpCli(object):
         headers: Optional[dict[str, str]] = None,
         volsan: bool = False,
     ) -> bytes:
-        # TODO something to reply with user-supplied values safely
+        if status == 404:
+            g = self.conn.hsrv.g404
+            if g.lim:
+                bonk, ip = g.bonk(self.ip, self.vpath)
+                if bonk:
+                    self.log("client banned: 404s", 1)
+                    self.conn.hsrv.bans[ip] = bonk
 
         if volsan:
             vols = list(self.asrv.vfs.all_vols.values())
@@ -542,9 +585,13 @@ class HttpCli(object):
             if self.vpath.startswith(".cpr/ico/"):
                 return self.tx_ico(self.vpath.split("/")[-1], exact=True)
 
-            static_path = os.path.join(E.mod, "web/", self.vpath[5:])
+            static_path = os.path.join(self.E.mod, "web/", self.vpath[5:])
             return self.tx_file(static_path)
 
+        if "cf_challenge" in self.uparam:
+            self.reply(self.j2s("cf").encode("utf-8", "replace"))
+            return True
+
         if not self.can_read and not self.can_write and not self.can_get:
             if self.vpath:
                 self.log("inaccessible: [{}]".format(self.vpath))
@@ -663,7 +710,13 @@ class HttpCli(object):
                 self.log("urlform: {} bytes, {}".format(post_sz, path))
             elif "print" in opt:
                 reader, _ = self.get_body_reader()
-                for buf in reader:
+                buf = b""
+                for rbuf in reader:
+                    buf += rbuf
+                    if not rbuf or len(buf) >= 32768:
+                        break
+
+                if buf:
                     orig = buf.decode("utf-8", "replace")
                     t = "urlform_raw {} @ {}\n  {}\n"
                     self.log(t.format(len(orig), self.vpath, orig))
@@ -700,8 +753,9 @@ class HttpCli(object):
         # post_sz, sha_hex, sha_b64, remains, path, url
         reader, remains = self.get_body_reader()
         vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
+        rnd, want_url, lifetime = self.upload_flags(vfs)
         lim = vfs.get_dbv(rem)[0].lim
-        fdir = os.path.join(vfs.realpath, rem)
+        fdir = vfs.canonical(rem)
         if lim:
             fdir, rem = lim.all(self.ip, rem, remains, fdir)
 
@@ -762,16 +816,22 @@ class HttpCli(object):
             else:
                 self.log("fallthrough? thats a bug", 1)
 
-        suffix = "-{:.6f}-{}".format(time.time(), self.dip)
+        suffix = "-{:.6f}-{}".format(time.time(), self.dip())
+        nameless = not fn
+        if nameless:
+            suffix += ".bin"
+            fn = "put" + suffix
+
         params = {"suffix": suffix, "fdir": fdir}
         if self.args.nw:
             params = {}
             fn = os.devnull
 
         params.update(open_ka)
+        assert fn
 
-        if not fn:
-            fn = "put" + suffix
+        if rnd and not self.args.nw:
+            fn = self.rand_name(fdir, fn, rnd)
 
         with ren_open(fn, *open_a, **params) as zfw:
             f, fn = zfw["orz"]
@@ -790,6 +850,28 @@ class HttpCli(object):
         if self.args.nw:
             return post_sz, sha_hex, sha_b64, remains, path, ""
 
+        if nameless and "magic" in vfs.flags:
+            try:
+                ext = self.conn.hsrv.magician.ext(path)
+            except Exception as ex:
+                self.log("filetype detection failed for [{}]: {}".format(path, ex), 6)
+                ext = None
+
+            if ext:
+                if rnd:
+                    fn2 = self.rand_name(fdir, "a." + ext, rnd)
+                else:
+                    fn2 = fn.rsplit(".", 1)[0] + "." + ext
+
+                params["suffix"] = suffix[:-4]
+                with ren_open(fn, *open_a, **params) as zfw:
+                    f, fn = zfw["orz"]
+
+                path2 = os.path.join(fdir, fn2)
+                atomic_move(path, path2)
+                fn = fn2
+                path = path2
+
         vfs, rem = vfs.get_dbv(rem)
         self.conn.hsrv.broker.say(
             "up2k.hash_file",
@@ -798,12 +880,12 @@ class HttpCli(object):
             rem,
             fn,
             self.ip,
-            time.time(),
+            time.time() - lifetime,
         )
 
         vsuf = ""
         if self.can_read and "fk" in vfs.flags:
-            vsuf = "?k=" + gen_filekey(
+            vsuf = "?k=" + self.gen_fk(
                 self.args.fk_salt,
                 path,
                 post_sz,
@@ -826,10 +908,39 @@ class HttpCli(object):
         spd = self._spd(post_sz)
         t = "{} wrote {}/{} bytes to {}  # {}"
         self.log(t.format(spd, post_sz, remains, path, sha_b64[:28]))  # 21
-        t = "{}\n{}\n{}\n{}\n".format(post_sz, sha_b64, sha_hex[:56], url)
+
+        ac = self.uparam.get(
+            "want", self.headers.get("accept", "").lower().split(";")[-1]
+        )
+        if ac == "url":
+            t = url
+        else:
+            t = "{}\n{}\n{}\n{}\n".format(post_sz, sha_b64, sha_hex[:56], url)
+
         self.reply(t.encode("utf-8"))
         return True
 
+    def rand_name(self, fdir: str, fn: str, rnd: int) -> str:
+        ok = False
+        try:
+            ext = "." + fn.rsplit(".", 1)[1]
+        except:
+            ext = ""
+
+        for extra in range(16):
+            for _ in range(16):
+                if ok:
+                    break
+
+                nc = rnd + extra
+                nb = int((6 + 6 * nc) / 8)
+                zb = os.urandom(nb)
+                zb = base64.urlsafe_b64encode(zb)
+                fn = zb[:nc].decode("utf-8") + ext
+                ok = not bos.path.exists(os.path.join(fdir, fn))
+
+        return fn
+
     def _spd(self, nbytes: int, add: bool = True) -> str:
         if add:
             self.conn.nbyte += nbytes
@@ -911,6 +1022,9 @@ class HttpCli(object):
         except:
             raise Pebkac(422, "you POSTed invalid json")
 
+        # self.reply(b"cloudflare", 503)
+        # return True
+
         if "srch" in self.uparam or "srch" in body:
             return self.handle_search(body)
 
@@ -937,7 +1051,7 @@ class HttpCli(object):
 
         if rem:
             try:
-                dst = os.path.join(vfs.realpath, rem)
+                dst = vfs.canonical(rem)
                 if not bos.path.isdir(dst):
                     bos.makedirs(dst)
             except OSError as ex:
@@ -1048,7 +1162,7 @@ class HttpCli(object):
             reader = read_socket(self.sr, remains)
 
             f = None
-            fpool = not self.args.no_fpool and (not ANYWIN or sprs)
+            fpool = not self.args.no_fpool and sprs
             if fpool:
                 with self.mutex:
                     try:
@@ -1118,8 +1232,10 @@ class HttpCli(object):
             except:
                 self.log("failed to utime ({}, {})".format(fin_path, times))
 
+        cinf = self.headers.get("x-up2k-stat", "")
+
         spd = self._spd(post_sz)
-        self.log("{} thank".format(spd))
+        self.log("{:70} thank {}".format(spd, cinf))
         self.reply(b"thank")
         return True
 
@@ -1148,6 +1264,14 @@ class HttpCli(object):
             msg = "login ok"
             dur = int(60 * 60 * self.args.logout)
         else:
+            self.log("invalid password: {}".format(pwd), 3)
+            g = self.conn.hsrv.gpwd
+            if g.lim:
+                bonk, ip = g.bonk(self.ip, pwd)
+                if bonk:
+                    self.log("client banned: invalid passwords", 1)
+                    self.conn.hsrv.bans[ip] = bonk
+
             msg = "naw dude"
             pwd = "x"  # nosec
             dur = None
@@ -1170,7 +1294,7 @@ class HttpCli(object):
         sanitized = sanitize_fn(new_dir, "", [])
 
         if not nullwrite:
-            fdir = os.path.join(vfs.realpath, rem)
+            fdir = vfs.canonical(rem)
             fn = os.path.join(fdir, sanitized)
 
             if not bos.path.isdir(fdir):
@@ -1209,7 +1333,7 @@ class HttpCli(object):
         sanitized = sanitize_fn(new_file, "", [])
 
         if not nullwrite:
-            fdir = os.path.join(vfs.realpath, rem)
+            fdir = vfs.canonical(rem)
             fn = os.path.join(fdir, sanitized)
 
             if bos.path.exists(fn):
@@ -1222,6 +1346,22 @@ class HttpCli(object):
         self.redirect(vpath, "?edit")
         return True
 
+    def upload_flags(self, vfs: VFS) -> tuple[int, bool, int]:
+        srnd = self.uparam.get("rand", self.headers.get("rand", ""))
+        rnd = int(srnd) if srnd and not self.args.nw else 0
+        ac = self.uparam.get(
+            "want", self.headers.get("accept", "").lower().split(";")[-1]
+        )
+        want_url = ac == "url"
+        zs = self.uparam.get("life", self.headers.get("life", ""))
+        if zs:
+            vlife = vfs.flags.get("lifetime") or 0
+            lifetime = max(0, int(vlife - int(zs)))
+        else:
+            lifetime = 0
+
+        return rnd, want_url, lifetime
+
     def handle_plain_upload(self) -> bool:
         assert self.parser
         nullwrite = self.args.nw
@@ -1230,16 +1370,19 @@ class HttpCli(object):
 
         upload_vpath = self.vpath
         lim = vfs.get_dbv(rem)[0].lim
-        fdir_base = os.path.join(vfs.realpath, rem)
+        fdir_base = vfs.canonical(rem)
         if lim:
             fdir_base, rem = lim.all(self.ip, rem, -1, fdir_base)
             upload_vpath = "{}/{}".format(vfs.vpath, rem).strip("/")
             if not nullwrite:
                 bos.makedirs(fdir_base)
 
+        rnd, want_url, lifetime = self.upload_flags(vfs)
+
         files: list[tuple[int, str, str, str, str, str]] = []
         # sz, sha_hex, sha_b64, p_file, fname, abspath
         errmsg = ""
+        dip = self.dip()
         t0 = time.time()
         try:
             assert self.parser.gen
@@ -1253,10 +1396,13 @@ class HttpCli(object):
                     p_file or "", "", [".prologue.html", ".epilogue.html"]
                 )
                 if p_file and not nullwrite:
+                    if rnd:
+                        fname = self.rand_name(fdir, fname, rnd)
+
                     if not bos.path.isdir(fdir):
                         raise Pebkac(404, "that folder does not exist")
 
-                    suffix = "-{:.6f}-{}".format(time.time(), self.dip)
+                    suffix = "-{:.6f}-{}".format(time.time(), dip)
                     open_args = {"fdir": fdir, "suffix": suffix}
 
                     # reserve destination filename
@@ -1271,14 +1417,19 @@ class HttpCli(object):
                 else:
                     open_args = {}
                     tnam = fname = os.devnull
-                    fdir = ""
+                    fdir = abspath = ""
 
                 if lim:
                     lim.chk_bup(self.ip)
                     lim.chk_nup(self.ip)
 
                 try:
-                    max_sz = lim.smax if lim else 0
+                    max_sz = 0
+                    if lim:
+                        v1 = lim.smax
+                        v2 = lim.dfv - lim.dfl
+                        max_sz = min(v1, v2) if v1 and v2 else v1 or v2
+
                     with ren_open(tnam, "wb", 512 * 1024, **open_args) as zfw:
                         f, tnam = zfw["orz"]
                         tabspath = os.path.join(fdir, tnam)
@@ -1293,16 +1444,20 @@ class HttpCli(object):
                         lim.nup(self.ip)
                         lim.bup(self.ip, sz)
                         try:
+                            lim.chk_df(tabspath, sz, True)
                             lim.chk_sz(sz)
                             lim.chk_bup(self.ip)
                             lim.chk_nup(self.ip)
                         except:
-                            bos.unlink(tabspath)
-                            bos.unlink(abspath)
+                            if not nullwrite:
+                                bos.unlink(tabspath)
+                                bos.unlink(abspath)
                             fname = os.devnull
                             raise
 
-                    atomic_move(tabspath, abspath)
+                    if not nullwrite:
+                        atomic_move(tabspath, abspath)
+
                     files.append(
                         (sz, sha_hex, sha_b64, p_file or "(discarded)", fname, abspath)
                     )
@@ -1314,7 +1469,7 @@ class HttpCli(object):
                         vrem,
                         fname,
                         self.ip,
-                        time.time(),
+                        time.time() - lifetime,
                     )
                     self.conn.nbyte += sz
 
@@ -1352,9 +1507,9 @@ class HttpCli(object):
         for sz, sha_hex, sha_b64, ofn, lfn, ap in files:
             vsuf = ""
             if self.can_read and "fk" in vfs.flags:
-                vsuf = "?k=" + gen_filekey(
+                vsuf = "?k=" + self.gen_fk(
                     self.args.fk_salt,
-                    abspath,
+                    ap,
                     sz,
                     0 if ANYWIN or not ap else bos.stat(ap).st_ino,
                 )[: vfs.flags["fk"]]
@@ -1390,21 +1545,31 @@ class HttpCli(object):
         vspd = self._spd(sz_total, False)
         self.log("{} {}".format(vspd, msg))
 
-        if not nullwrite:
-            log_fn = "up.{:.6f}.txt".format(t0)
-            with open(log_fn, "wb") as f:
-                ft = "{}:{}".format(self.ip, self.addr[1])
-                ft = "{}\n{}\n{}\n".format(ft, msg.rstrip(), errmsg)
-                f.write(ft.encode("utf-8"))
+        suf = ""
+        if not nullwrite and self.args.write_uplog:
+            try:
+                log_fn = "up.{:.6f}.txt".format(t0)
+                with open(log_fn, "wb") as f:
+                    ft = "{}:{}".format(self.ip, self.addr[1])
+                    ft = "{}\n{}\n{}\n".format(ft, msg.rstrip(), errmsg)
+                    f.write(ft.encode("utf-8"))
+            except Exception as ex:
+                suf = "\nfailed to write the upload report: {}".format(ex)
 
         sc = 400 if errmsg else 200
-        if "j" in self.uparam:
+        if want_url:
+            msg = "\n".join([x["url"] for x in jmsg["files"]])
+            if errmsg:
+                msg += "\n" + errmsg
+
+            self.reply(msg.encode("utf-8", "replace"), status=sc)
+        elif "j" in self.uparam:
             jtxt = json.dumps(jmsg, indent=2, sort_keys=True).encode("utf-8", "replace")
             self.reply(jtxt, mime="application/json", status=sc)
         else:
             self.redirect(
                 self.vpath,
-                msg=msg,
+                msg=msg + suf,
                 flavor="return to",
                 click=False,
                 status=sc,
@@ -1432,7 +1597,7 @@ class HttpCli(object):
             raise Pebkac(411)
 
         rp, fn = vsplit(rem)
-        fp = os.path.join(vfs.realpath, rp)
+        fp = vfs.canonical(rp)
         lim = vfs.get_dbv(rem)[0].lim
         if lim:
             fp, rp = lim.all(self.ip, rp, clen, fp)
@@ -1684,7 +1849,7 @@ class HttpCli(object):
         # send reply
 
         if is_compressed:
-            self.out_headers["Cache-Control"] = "max-age=573"
+            self.out_headers["Cache-Control"] = "max-age=604869"
         else:
             self.permit_caching()
 
@@ -1807,9 +1972,11 @@ class HttpCli(object):
         if len(ext) > 11:
             ext = "⋯" + ext[-9:]
 
-        mime, ico = self.ico.get(ext, not exact)
+        # chrome cannot handle more than ~2000 unique SVGs
+        chrome = " rv:" not in self.ua
+        mime, ico = self.ico.get(ext, not exact, chrome)
 
-        dt = datetime.utcfromtimestamp(E.t0)
+        dt = datetime.utcfromtimestamp(self.E.t0)
         lm = dt.strftime("%a, %d %b %Y %H:%M:%S GMT")
         self.reply(ico, mime=mime, headers={"Last-Modified": lm})
         return True
@@ -1822,7 +1989,7 @@ class HttpCli(object):
                 return self.tx_404(True)
 
         tpl = "mde" if "edit2" in self.uparam else "md"
-        html_path = os.path.join(E.mod, "web", "{}.html".format(tpl))
+        html_path = os.path.join(self.E.mod, "web", "{}.html".format(tpl))
         template = self.j2j(tpl)
 
         st = bos.stat(fs_path)
@@ -1837,7 +2004,7 @@ class HttpCli(object):
             for c, v in [(b"&", 4), (b"<", 3), (b">", 3)]:
                 sz_md += (len(buf) - len(buf.replace(c, b""))) * v
 
-        file_ts = max(ts_md, ts_html, E.t0)
+        file_ts = max(ts_md, ts_html, self.E.t0)
         file_lastmod, do_send = self._chk_lastmod(file_ts)
         self.out_headers["Last-Modified"] = file_lastmod
         self.out_headers.update(NO_CACHE)
@@ -1855,7 +2022,9 @@ class HttpCli(object):
             "edit": "edit" in self.uparam,
             "title": html_escape(self.vpath, crlf=True),
             "lastmod": int(ts_md * 1000),
-            "md_plug": "true" if self.args.emp else "false",
+            "lang": self.args.lang,
+            "favico": self.args.favico,
+            "have_emp": self.args.emp,
             "md_chk_rate": self.args.mcr,
             "md": boundary,
             "arg_base": arg_base,
@@ -1904,7 +2073,13 @@ class HttpCli(object):
             vstate = {("/" + k).rstrip("/") + "/": v for k, v in vs["volstate"].items()}
         else:
             vstate = {}
-            vs = {"scanning": None, "hashq": None, "tagq": None, "mtpq": None}
+            vs = {
+                "scanning": None,
+                "hashq": None,
+                "tagq": None,
+                "mtpq": None,
+                "dbwt": None,
+            }
 
         if self.uparam.get("ls") in ["v", "t", "txt"]:
             if self.uname == "*":
@@ -1914,7 +2089,7 @@ class HttpCli(object):
 
             if vstate:
                 txt += "\nstatus:"
-                for k in ["scanning", "hashq", "tagq", "mtpq"]:
+                for k in ["scanning", "hashq", "tagq", "mtpq", "dbwt"]:
                     txt += " {}({})".format(k, vs[k])
 
             if rvol:
@@ -1943,6 +2118,7 @@ class HttpCli(object):
             hashq=vs["hashq"],
             tagq=vs["tagq"],
             mtpq=vs["mtpq"],
+            dbwt=vs["dbwt"],
             url_suf=suf,
             k304=self.k304(),
         )
@@ -2218,7 +2394,8 @@ class HttpCli(object):
             ret = json.dumps(ls)
             mime = "application/json"
 
-        self.reply(ret.encode("utf-8", "replace") + b"\n", mime=mime)
+        ret += "\n\033[0m" if arg == "v" else "\n"
+        self.reply(ret.encode("utf-8", "replace"), mime=mime)
         return True
 
     def tx_browser(self) -> bool:
@@ -2279,8 +2456,9 @@ class HttpCli(object):
 
         if not is_dir and (self.can_read or self.can_get):
             if not self.can_read and "fk" in vn.flags:
-                correct = gen_filekey(
-                    self.args.fk_salt, abspath, st.st_size, 0 if ANYWIN else st.st_ino
+                vabs = vjoin(vn.realpath, rem)
+                correct = self.gen_fk(
+                    self.args.fk_salt, vabs, st.st_size, 0 if ANYWIN else st.st_ino
                 )[: vn.flags["fk"]]
                 got = self.uparam.get("k")
                 if got != correct:
@@ -2303,26 +2481,14 @@ class HttpCli(object):
         except:
             self.log("#wow #whoa")
 
-        try:
-            # some fuses misbehave
-            if not self.args.nid:
-                if WINDOWS:
-                    try:
-                        bfree = ctypes.c_ulonglong(0)
-                        ctypes.windll.kernel32.GetDiskFreeSpaceExW(  # type: ignore
-                            ctypes.c_wchar_p(abspath), None, None, ctypes.pointer(bfree)
-                        )
-                        srv_info.append(humansize(bfree.value) + " free")
-                    except:
-                        pass
-                else:
-                    sv = os.statvfs(fsenc(abspath))
-                    free = humansize(sv.f_frsize * sv.f_bfree, True)
-                    total = humansize(sv.f_frsize * sv.f_blocks, True)
-
-                    srv_info.append("{} free of {}".format(free, total))
-        except:
-            pass
+        if not self.args.nid:
+            free, total = get_df(abspath)
+            if total is not None:
+                h1 = humansize(free or 0)
+                h2 = humansize(total)
+                srv_info.append("{} free of {}".format(h1, h2))
+            elif free is not None:
+                srv_info.append(humansize(free, True) + " free")
 
         srv_infot = "</span> // <span>".join(srv_info)
 
@@ -2371,6 +2537,7 @@ class HttpCli(object):
             "srvinf": srv_infot,
             "acct": self.uname,
             "idx": ("e2d" in vn.flags),
+            "lifetime": vn.flags.get("lifetime") or 0,
             "perms": perms,
             "logues": logues,
             "readme": readme,
@@ -2382,26 +2549,38 @@ class HttpCli(object):
             "ls0": None,
             "acct": self.uname,
             "perms": json.dumps(perms),
+            "lifetime": ls_ret["lifetime"],
             "taglist": [],
             "def_hcols": [],
+            "have_emp": self.args.emp,
             "have_up2k_idx": ("e2d" in vn.flags),
             "have_tags_idx": ("e2t" in vn.flags),
             "have_acode": (not self.args.no_acode),
             "have_mv": (not self.args.no_mv),
             "have_del": (not self.args.no_del),
             "have_zip": (not self.args.no_zip),
-            "have_unpost": (self.args.unpost > 0),
+            "have_unpost": int(self.args.unpost),
             "have_b_u": (self.can_write and self.uparam.get("b") == "u"),
             "url_suf": url_suf,
             "logues": logues,
             "readme": readme,
             "title": html_escape(self.vpath, crlf=True),
             "srv_info": srv_infot,
-            "lang": self.args.lang,
             "dtheme": self.args.theme,
             "themes": self.args.themes,
             "turbolvl": self.args.turbo,
+            "u2sort": self.args.u2sort,
         }
+
+        if self.args.js_browser:
+            j2a["js"] = self.args.js_browser
+
+        if self.args.css_browser:
+            j2a["css"] = self.args.css_browser
+
+        if not self.conn.hsrv.prism:
+            j2a["no_prism"] = True
+
         if not self.can_read:
             if is_ls:
                 return self.tx_ls(ls_ret)
@@ -2504,7 +2683,7 @@ class HttpCli(object):
             if add_fk:
                 href = "{}?k={}".format(
                     quotep(href),
-                    gen_filekey(
+                    self.gen_fk(
                         self.args.fk_salt, fspath, sz, 0 if ANYWIN else inf.st_ino
                     )[:add_fk],
                 )
@@ -2532,43 +2711,28 @@ class HttpCli(object):
             rd = fe["rd"]
             del fe["rd"]
             if not icur:
-                break
+                continue
 
             if vn != dbv:
                 _, rd = vn.get_dbv(rd)
 
-            q = "select w from up where rd = ? and fn = ?"
-            r = None
+            q = "select mt.k, mt.v from up inner join mt on mt.w = substr(up.w,1,16) where up.rd = ? and up.fn = ? and +mt.k != 'x'"
             try:
-                r = icur.execute(q, (rd, fn)).fetchone()
+                r = icur.execute(q, (rd, fn))
             except Exception as ex:
                 if "database is locked" in str(ex):
                     break
 
                 try:
                     args = s3enc(idx.mem_cur, rd, fn)
-                    r = icur.execute(q, args).fetchone()
+                    r = icur.execute(q, args)
                 except:
-                    t = "tag list error, {}/{}\n{}"
+                    t = "tag read error, {}/{}\n{}"
                     self.log(t.format(rd, fn, min_ex()))
                     break
 
-            tags: dict[str, Any] = {}
-            fe["tags"] = tags
-
-            if not r:
-                continue
-
-            w = r[0][:16]
-            q = "select k, v from mt where w = ? and +k != 'x'"
-            try:
-                for k, v in icur.execute(q, (w,)):
-                    tagset.add(k)
-                    tags[k] = v
-            except:
-                t = "tag read error, {}/{} [{}]:\n{}"
-                self.log(t.format(rd, fn, w, min_ex()))
-                break
+            fe["tags"] = {k: v for k, v in r}
+            _ = [tagset.add(k) for k in fe["tags"]]
 
         if icur:
             taglist = [k for k in vn.flags.get("mte", "").split(",") if k in tagset]
@@ -2601,9 +2765,6 @@ class HttpCli(object):
             if doctxt is not None:
                 j2a["doc"] = doctxt
 
-        if not self.conn.hsrv.prism:
-            j2a["no_prism"] = True
-
         for d in dirs:
             d["name"] += "/"
 
@@ -2615,19 +2776,12 @@ class HttpCli(object):
         else:
             j2a["files"] = dirs + files
 
-        j2a["logues"] = logues
         j2a["taglist"] = taglist
         j2a["txt_ext"] = self.args.textfiles.replace(",", " ")
 
         if "mth" in vn.flags:
             j2a["def_hcols"] = vn.flags["mth"].split(",")
 
-        if self.args.js_browser:
-            j2a["js"] = self.args.js_browser
-
-        if self.args.css_browser:
-            j2a["css"] = self.args.css_browser
-
         html = self.j2s(tpl, **j2a)
         self.reply(html.encode("utf-8", "replace"))
         return True

copyparty/httpconn.py

@@ -15,7 +15,7 @@ except:
     HAVE_SSL = False
 
 from . import util as Util
-from .__init__ import TYPE_CHECKING, E
+from .__init__ import TYPE_CHECKING, EnvParams
 from .authsrv import AuthSrv  # typechk
 from .httpcli import HttpCli
 from .ico import Ico
@@ -23,6 +23,7 @@ from .mtag import HAVE_FFMPEG
 from .th_cli import ThumbCli
 from .th_srv import HAVE_PIL, HAVE_VIPS
 from .u2idx import U2idx
+from .util import HMaccas, shut_socket
 
 try:
     from typing import Optional, Pattern, Union
@@ -49,9 +50,11 @@ class HttpConn(object):
 
         self.mutex: threading.Lock = hsrv.mutex  # mypy404
         self.args: argparse.Namespace = hsrv.args  # mypy404
+        self.E: EnvParams = self.args.E
         self.asrv: AuthSrv = hsrv.asrv  # mypy404
         self.cert_path = hsrv.cert_path
         self.u2fh: Util.FHC = hsrv.u2fh  # mypy404
+        self.iphash: HMaccas = hsrv.broker.iphash
 
         enth = (HAVE_PIL or HAVE_VIPS or HAVE_FFMPEG) and not self.args.no_thumb
         self.thumbcli: Optional[ThumbCli] = ThumbCli(hsrv) if enth else None  # mypy404
@@ -62,7 +65,7 @@ class HttpConn(object):
         self.nreq: int = 0  # mypy404
         self.nbyte: int = 0  # mypy404
         self.u2idx: Optional[U2idx] = None
-        self.log_func: Util.RootLogger = hsrv.log  # mypy404
+        self.log_func: "Util.RootLogger" = hsrv.log  # mypy404
         self.log_src: str = "httpconn"  # mypy404
         self.lf_url: Optional[Pattern[str]] = (
             re.compile(self.args.lf_url) if self.args.lf_url else None
@@ -72,8 +75,7 @@ class HttpConn(object):
     def shutdown(self) -> None:
         self.stopping = True
         try:
-            self.s.shutdown(socket.SHUT_RDWR)
-            self.s.close()
+            shut_socket(self.log, self.s, 1)
         except:
             pass
 
@@ -91,7 +93,7 @@ class HttpConn(object):
         return self.log_src
 
     def respath(self, res_name: str) -> str:
-        return os.path.join(E.mod, "web", res_name)
+        return os.path.join(self.E.mod, "web", res_name)
 
     def log(self, msg: str, c: Union[int, str] = 0) -> None:
         self.log_func(self.log_src, msg, c)
@@ -189,11 +191,7 @@ class HttpConn(object):
             except Exception as ex:
                 em = str(ex)
 
-                if "ALERT_BAD_CERTIFICATE" in em:
-                    # firefox-linux if there is no exception yet
-                    self.log("client rejected our certificate (nice)")
-
-                elif "ALERT_CERTIFICATE_UNKNOWN" in em:
+                if "ALERT_CERTIFICATE_UNKNOWN" in em:
                     # android-chrome keeps doing this
                     pass
 

copyparty/httpsrv.py

@@ -28,10 +28,19 @@ except ImportError:
     )
     sys.exit(1)
 
-from .__init__ import MACOS, TYPE_CHECKING, E
+from .__init__ import MACOS, TYPE_CHECKING, EnvParams
 from .bos import bos
 from .httpconn import HttpConn
-from .util import FHC, min_ex, spack, start_log_thrs, start_stackmon
+from .util import (
+    FHC,
+    Garda,
+    Magician,
+    min_ex,
+    shut_socket,
+    spack,
+    start_log_thrs,
+    start_stackmon,
+)
 
 if TYPE_CHECKING:
     from .broker_util import BrokerCli
@@ -52,10 +61,18 @@ class HttpSrv(object):
         self.broker = broker
         self.nid = nid
         self.args = broker.args
+        self.E: EnvParams = self.args.E
         self.log = broker.log
         self.asrv = broker.asrv
 
+        # redefine in case of multiprocessing
+        socket.setdefaulttimeout(120)
+
         nsuf = "-n{}-i{:x}".format(nid, os.getpid()) if nid else ""
+        self.magician = Magician()
+        self.bans: dict[str, int] = {}
+        self.gpwd = Garda(self.args.ban_pw)
+        self.g404 = Garda(self.args.ban_404)
 
         self.name = "hsrv" + nsuf
         self.mutex = threading.Lock()
@@ -78,14 +95,15 @@ class HttpSrv(object):
         self.cb_v = ""
 
         env = jinja2.Environment()
-        env.loader = jinja2.FileSystemLoader(os.path.join(E.mod, "web"))
+        env.loader = jinja2.FileSystemLoader(os.path.join(self.E.mod, "web"))
         self.j2 = {
             x: env.get_template(x + ".html")
-            for x in ["splash", "browser", "browser2", "msg", "md", "mde"]
+            for x in ["splash", "browser", "browser2", "msg", "md", "mde", "cf"]
         }
-        self.prism = os.path.exists(os.path.join(E.mod, "web", "deps", "prism.js.gz"))
+        zs = os.path.join(self.E.mod, "web", "deps", "prism.js.gz")
+        self.prism = os.path.exists(zs)
 
-        cert_path = os.path.join(E.cfg, "cert.pem")
+        cert_path = os.path.join(self.E.cfg, "cert.pem")
         if bos.path.exists(cert_path):
             self.cert_path = cert_path
         else:
@@ -102,7 +120,7 @@ class HttpSrv(object):
                 start_log_thrs(self.log, self.args.log_thrs, nid)
 
         self.th_cfg: dict[str, Any] = {}
-        t = threading.Thread(target=self.post_init)
+        t = threading.Thread(target=self.post_init, name="hsrv-init2")
         t.daemon = True
         t.start()
 
@@ -150,6 +168,12 @@ class HttpSrv(object):
                     return
 
     def listen(self, sck: socket.socket, nlisteners: int) -> None:
+        if self.args.j != 1:
+            # lost in the pickle; redefine
+            sck.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            sck.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+            sck.settimeout(None)  # < does not inherit, ^ does
+
         ip, port = sck.getsockname()
         self.srvs.append(sck)
         self.nclimax = math.ceil(self.args.nc * 1.0 / nlisteners)
@@ -165,13 +189,13 @@ class HttpSrv(object):
         """listens on a shared tcp server"""
         ip, port = srv_sck.getsockname()
         fno = srv_sck.fileno()
-        msg = "subscribed @ {}:{}  f{}".format(ip, port, fno)
+        msg = "subscribed @ {}:{}  f{} p{}".format(ip, port, fno, os.getpid())
         self.log(self.name, msg)
 
         def fun() -> None:
             self.broker.say("cb_httpsrv_up")
 
-        threading.Thread(target=fun).start()
+        threading.Thread(target=fun, name="sig-hsrv-up1").start()
 
         while not self.stopping:
             if self.args.log_conn:
@@ -261,8 +285,11 @@ class HttpSrv(object):
                 )
                 self.thr_client(sck, addr)
                 me.name = self.name + "-poolw"
-            except:
-                self.log(self.name, "thr_client: " + min_ex(), 3)
+            except Exception as ex:
+                if str(ex).startswith("client d/c "):
+                    self.log(self.name, "thr_client: " + str(ex), 6)
+                else:
+                    self.log(self.name, "thr_client: " + min_ex(), 3)
 
     def shutdown(self) -> None:
         self.stopping = True
@@ -272,12 +299,12 @@ class HttpSrv(object):
             except:
                 pass
 
+        thrs = []
         clients = list(self.clients)
         for cli in clients:
-            try:
-                cli.shutdown()
-            except:
-                pass
+            t = threading.Thread(target=cli.shutdown)
+            thrs.append(t)
+            t.start()
 
         if self.tp_q:
             self.stop_threads(self.tp_nthr)
@@ -286,12 +313,13 @@ class HttpSrv(object):
                 if self.tp_q.empty():
                     break
 
+        for t in thrs:
+            t.join()
+
         self.log(self.name, "ok bye")
 
     def thr_client(self, sck: socket.socket, addr: tuple[str, int]) -> None:
         """thread managing one tcp client"""
-        sck.settimeout(120)
-
         cli = HttpConn(sck, addr, self)
         with self.mutex:
             self.clients.add(cli)
@@ -318,8 +346,7 @@ class HttpSrv(object):
 
             try:
                 fno = sck.fileno()
-                sck.shutdown(socket.SHUT_RDWR)
-                sck.close()
+                shut_socket(cli.log, sck)
             except (OSError, socket.error) as ex:
                 if not MACOS:
                     self.log(
@@ -348,9 +375,9 @@ class HttpSrv(object):
             if time.time() - self.cb_ts < 1:
                 return self.cb_v
 
-            v = E.t0
+            v = self.E.t0
             try:
-                with os.scandir(os.path.join(E.mod, "web")) as dh:
+                with os.scandir(os.path.join(self.E.mod, "web")) as dh:
                     for fh in dh:
                         inf = fh.stat()
                         v = max(v, inf.st_mtime)

copyparty/ico.py

@@ -6,16 +6,18 @@ import colorsys
 import hashlib
 
 from .__init__ import PY2
+from .th_srv import HAVE_PIL
+from .util import BytesIO
 
 
 class Ico(object):
     def __init__(self, args: argparse.Namespace) -> None:
         self.args = args
 
-    def get(self, ext: str, as_thumb: bool) -> tuple[str, bytes]:
+    def get(self, ext: str, as_thumb: bool, chrome: bool) -> tuple[str, bytes]:
         """placeholder to make thumbnails not break"""
 
-        zb = hashlib.md5(ext.encode("utf-8")).digest()[:2]
+        zb = hashlib.sha1(ext.encode("utf-8")).digest()[2:4]
         if PY2:
             zb = [ord(x) for x in zb]
 
@@ -24,10 +26,44 @@ class Ico(object):
         ci = [int(x * 255) for x in list(c1) + list(c2)]
         c = "".join(["{:02x}".format(x) for x in ci])
 
+        w = 100
         h = 30
         if not self.args.th_no_crop and as_thumb:
-            w, h = self.args.th_size.split("x")
-            h = int(100 / (float(w) / float(h)))
+            sw, sh = self.args.th_size.split("x")
+            h = int(100 / (float(sw) / float(sh)))
+            w = 100
+
+        if chrome and as_thumb:
+            # cannot handle more than ~2000 unique SVGs
+            if HAVE_PIL:
+                # svg: 3s, cache: 6s, this: 8s
+                from PIL import Image, ImageDraw
+
+                h = int(64 * h / w)
+                w = 64
+                img = Image.new("RGB", (w, h), "#" + c[:6])
+                pb = ImageDraw.Draw(img)
+                tw, th = pb.textsize(ext)
+                pb.text(((w - tw) // 2, (h - th) // 2), ext, fill="#" + c[6:])
+                img = img.resize((w * 3, h * 3), Image.NEAREST)
+
+                buf = BytesIO()
+                img.save(buf, format="PNG", compress_level=1)
+                return "image/png", buf.getvalue()
+
+            elif False:
+                # 48s, too slow
+                import pyvips
+
+                h = int(192 * h / w)
+                w = 192
+                img = pyvips.Image.text(
+                    ext, width=w, height=h, dpi=192, align=pyvips.Align.CENTRE
+                )
+                img = img.ifthenelse(ci[3:], ci[:3], blend=True)
+                # i = i.resize(3, kernel=pyvips.Kernel.NEAREST)
+                buf = img.write_to_buffer(".png[compression=1]")
+                return "image/png", buf
 
         svg = """\
 <?xml version="1.0" encoding="UTF-8"?>

copyparty/mtag.py

@@ -8,9 +8,9 @@ import shutil
 import subprocess as sp
 import sys
 
-from .__init__ import PY2, WINDOWS, unicode
+from .__init__ import PY2, WINDOWS, E, unicode
 from .bos import bos
-from .util import REKOBO_LKEY, fsenc, retchk, runcmd, uncyg
+from .util import REKOBO_LKEY, fsenc, min_ex, retchk, runcmd, uncyg
 
 try:
     from typing import Any, Union
@@ -42,9 +42,12 @@ class MParser(object):
         self.tag, args = cmdline.split("=", 1)
         self.tags = self.tag.split(",")
 
-        self.timeout = 30
+        self.timeout = 60
         self.force = False
+        self.kill = "t"  # tree; all children recursively
+        self.capture = 3  # outputs to consume
         self.audio = "y"
+        self.pri = 0  # priority; higher = later
         self.ext = []
 
         while True:
@@ -66,6 +69,14 @@ class MParser(object):
                 self.audio = arg[1:]  # [r]equire [n]ot [d]ontcare
                 continue
 
+            if arg.startswith("k"):
+                self.kill = arg[1:]  # [t]ree [m]ain [n]one
+                continue
+
+            if arg.startswith("c"):
+                self.capture = int(arg[1:])  # 0=none 1=stdout 2=stderr 3=both
+                continue
+
             if arg == "f":
                 self.force = True
                 continue
@@ -78,11 +89,15 @@ class MParser(object):
                 self.ext.append(arg[1:])
                 continue
 
+            if arg.startswith("p"):
+                self.pri = int(arg[1:] or "1")
+                continue
+
             raise Exception()
 
 
 def ffprobe(
-    abspath: str, timeout: int = 10
+    abspath: str, timeout: int = 60
 ) -> tuple[dict[str, tuple[int, Any]], dict[str, list[Any]]]:
     cmd = [
         b"ffprobe",
@@ -168,7 +183,7 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
             ]
 
         if typ == "format":
-            kvm = [["duration", ".dur"], ["bit_rate", ".q"]]
+            kvm = [["duration", ".dur"], ["bit_rate", ".q"], ["format_name", "fmt"]]
 
         for sk, rk in kvm:
             v1 = strm.get(sk)
@@ -213,7 +228,10 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
         fps = ret[".fps"]
         if "/" in fps:
             fa, fb = fps.split("/")
-            fps = int(fa) * 1.0 / int(fb)
+            try:
+                fps = int(fa) * 1.0 / int(fb)
+            except:
+                fps = 9001
 
         if fps < 1000 and fmt.get("format_name") not in ["image2", "png_pipe"]:
             ret[".fps"] = round(fps, 3)
@@ -226,6 +244,9 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
             if ".q" in ret:
                 del ret[".q"]
 
+    if "fmt" in ret:
+        ret["fmt"] = ret["fmt"].split(",")[0]
+
     if ".resw" in ret and ".resh" in ret:
         ret["res"] = "{}x{}".format(ret[".resw"], ret[".resh"])
 
@@ -235,17 +256,13 @@ def parse_ffprobe(txt: str) -> tuple[dict[str, tuple[int, Any]], dict[str, list[
 
 
 class MTag(object):
-    def __init__(self, log_func: RootLogger, args: argparse.Namespace) -> None:
+    def __init__(self, log_func: "RootLogger", args: argparse.Namespace) -> None:
         self.log_func = log_func
         self.args = args
         self.usable = True
         self.prefer_mt = not args.no_mtag_ff
         self.backend = "ffprobe" if args.no_mutagen else "mutagen"
-        self.can_ffprobe = (
-            HAVE_FFPROBE
-            and not args.no_mtag_ff
-            and (not WINDOWS or sys.version_info >= (3, 8))
-        )
+        self.can_ffprobe = HAVE_FFPROBE and not args.no_mtag_ff
         mappings = args.mtm
         or_ffprobe = " or FFprobe"
 
@@ -269,11 +286,6 @@ class MTag(object):
                 msg = "found FFprobe but it was disabled by --no-mtag-ff"
                 self.log(msg, c=3)
 
-            elif WINDOWS and sys.version_info < (3, 8):
-                or_ffprobe = " or python >= 3.8"
-                msg = "found FFprobe but your python is too old; need 3.8 or newer"
-                self.log(msg, c=1)
-
         if not self.usable:
             msg = "need Mutagen{} to read media tags so please run this:\n{}{} -m pip install --user mutagen\n"
             pybin = os.path.basename(sys.executable)
@@ -424,6 +436,8 @@ class MTag(object):
         return r1
 
     def get_mutagen(self, abspath: str) -> dict[str, Union[str, float]]:
+        ret: dict[str, tuple[int, Any]] = {}
+
         if not bos.path.isfile(abspath):
             return {}
 
@@ -437,7 +451,10 @@ class MTag(object):
             return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 
         sz = bos.path.getsize(abspath)
-        ret = {".q": (0, int((sz / md.info.length) / 128))}
+        try:
+            ret[".q"] = (0, int((sz / md.info.length) / 128))
+        except:
+            pass
 
         for attr, k, norm in [
             ["codec", "ac", unicode],
@@ -476,27 +493,43 @@ class MTag(object):
         if not bos.path.isfile(abspath):
             return {}
 
-        ret, md = ffprobe(abspath)
+        ret, md = ffprobe(abspath, self.args.mtag_to)
         return self.normalize_tags(ret, md)
 
-    def get_bin(self, parsers: dict[str, MParser], abspath: str) -> dict[str, Any]:
+    def get_bin(
+        self, parsers: dict[str, MParser], abspath: str, oth_tags: dict[str, Any]
+    ) -> dict[str, Any]:
         if not bos.path.isfile(abspath):
             return {}
 
-        pypath = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
-        zsl = [str(pypath)] + [str(x) for x in sys.path if x]
-        pypath = str(os.pathsep.join(zsl))
         env = os.environ.copy()
-        env["PYTHONPATH"] = pypath
+        try:
+            pypath = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
+            zsl = [str(pypath)] + [str(x) for x in sys.path if x]
+            pypath = str(os.pathsep.join(zsl))
+            env["PYTHONPATH"] = pypath
+        except:
+            if not E.ox:
+                raise
 
-        ret = {}
-        for tagname, parser in parsers.items():
+        ret: dict[str, Any] = {}
+        for tagname, parser in sorted(parsers.items(), key=lambda x: (x[1].pri, x[0])):
             try:
                 cmd = [parser.bin, abspath]
                 if parser.bin.endswith(".py"):
                     cmd = [sys.executable] + cmd
 
-                args = {"env": env, "timeout": parser.timeout}
+                args = {
+                    "env": env,
+                    "timeout": parser.timeout,
+                    "kill": parser.kill,
+                    "capture": parser.capture,
+                }
+
+                if parser.pri:
+                    zd = oth_tags.copy()
+                    zd.update(ret)
+                    args["sin"] = json.dumps(zd).encode("utf-8", "replace")
 
                 if WINDOWS:
                     args["creationflags"] = 0x4000
@@ -518,6 +551,8 @@ class MTag(object):
                         if tag and tag in zj:
                             ret[tag] = zj[tag]
             except:
-                pass
+                if self.args.mtag_v:
+                    t = "mtag error: tagname {}, parser {}, file {} => {}"
+                    self.log(t.format(tagname, parser.bin, abspath, min_ex()))
 
         return ret

copyparty/star.py

@@ -44,7 +44,7 @@ class StreamTar(StreamArc):
 
     def __init__(
         self,
-        log: NamedLogger,
+        log: "NamedLogger",
         fgen: Generator[dict[str, Any], None, None],
         **kwargs: Any
     ):
@@ -65,17 +65,19 @@ class StreamTar(StreamArc):
         w.start()
 
     def gen(self) -> Generator[Optional[bytes], None, None]:
-        while True:
-            buf = self.qfile.q.get()
-            if not buf:
-                break
+        try:
+            while True:
+                buf = self.qfile.q.get()
+                if not buf:
+                    break
 
-            self.co += len(buf)
-            yield buf
+                self.co += len(buf)
+                yield buf
 
-        yield None
-        if self.errf:
-            bos.unlink(self.errf["ap"])
+            yield None
+        finally:
+            if self.errf:
+                bos.unlink(self.errf["ap"])
 
     def ser(self, f: dict[str, Any]) -> None:
         name = f["vp"]

copyparty/sutil.py

@@ -17,7 +17,7 @@ except:
 class StreamArc(object):
     def __init__(
         self,
-        log: NamedLogger,
+        log: "NamedLogger",
         fgen: Generator[dict[str, Any], None, None],
         **kwargs: Any
     ):

copyparty/svchub.py

@@ -2,8 +2,11 @@
 from __future__ import print_function, unicode_literals
 
 import argparse
+import base64
 import calendar
+import gzip
 import os
+import re
 import shlex
 import signal
 import socket
@@ -17,17 +20,26 @@ try:
     from types import FrameType
 
     import typing
-    from typing import Optional, Union
+    from typing import Any, Optional, Union
 except:
     pass
 
-from .__init__ import ANYWIN, MACOS, PY2, VT100, WINDOWS, E, unicode
+from .__init__ import ANYWIN, MACOS, PY2, VT100, WINDOWS, EnvParams, unicode
 from .authsrv import AuthSrv
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE
 from .tcpsrv import TcpSrv
 from .th_srv import HAVE_PIL, HAVE_VIPS, HAVE_WEBP, ThumbSrv
 from .up2k import Up2k
-from .util import ansi_re, min_ex, mp, start_log_thrs, start_stackmon
+from .util import (
+    VERSIONS,
+    HMaccas,
+    alltrace,
+    ansi_re,
+    min_ex,
+    mp,
+    start_log_thrs,
+    start_stackmon,
+)
 
 
 class SvcHub(object):
@@ -44,18 +56,24 @@ class SvcHub(object):
     def __init__(self, args: argparse.Namespace, argv: list[str], printed: str) -> None:
         self.args = args
         self.argv = argv
+        self.E: EnvParams = args.E
         self.logf: Optional[typing.TextIO] = None
         self.logf_base_fn = ""
         self.stop_req = False
-        self.reload_req = False
         self.stopping = False
+        self.stopped = False
+        self.reload_req = False
         self.reloading = False
         self.stop_cond = threading.Condition()
+        self.nsigs = 3
         self.retcode = 0
         self.httpsrv_up = 0
 
         self.log_mutex = threading.Lock()
         self.next_day = 0
+        self.tstack = 0.0
+
+        self.iphash = HMaccas(os.path.join(self.E.cfg, "iphash"), 8)
 
         if args.sss or args.s >= 3:
             args.ss = True
@@ -64,13 +82,14 @@ class SvcHub(object):
 
         if args.ss or args.s >= 2:
             args.s = True
-            args.no_dot_mv = True
-            args.no_dot_ren = True
+            args.no_logues = True
+            args.no_readme = True
             args.unpost = 0
             args.no_del = True
             args.no_mv = True
             args.hardlink = True
             args.vague_403 = True
+            args.ban_404 = "50,60,1440"
             args.nih = True
 
         if args.s:
@@ -110,6 +129,9 @@ class SvcHub(object):
         if not args.hardlink and args.never_symlink:
             args.no_dedup = True
 
+        if args.log_fk:
+            args.log_fk = re.compile(args.log_fk)
+
         # initiate all services to manage
         self.asrv = AuthSrv(self.args, self.log)
         if args.ls:
@@ -129,8 +151,8 @@ class SvcHub(object):
         self.args.th_dec = list(decs.keys())
         self.thumbsrv = None
         if not args.no_thumb:
-            t = "decoder preference: {}".format(", ".join(self.args.th_dec))
-            self.log("thumb", t)
+            t = ", ".join(self.args.th_dec) or "(None available)"
+            self.log("thumb", "decoder preference: {}".format(t))
 
             if "pil" in self.args.th_dec and not HAVE_WEBP:
                 msg = "disabling webp thumbnails because either libwebp is not available or your Pillow is too old"
@@ -189,6 +211,9 @@ class SvcHub(object):
         self.log("root", t, 1)
 
         self.retcode = 1
+        self.sigterm()
+
+    def sigterm(self) -> None:
         os.kill(os.getpid(), signal.SIGTERM)
 
     def cb_httpsrv_up(self) -> None:
@@ -244,7 +269,7 @@ class SvcHub(object):
 
         msg = "[+] opened logfile [{}]\n".format(fn)
         printed += msg
-        lh.write("t0: {:.3f}\nargv: {}\n\n{}".format(E.t0, " ".join(argv), printed))
+        lh.write("t0: {:.3f}\nargv: {}\n\n{}".format(self.E.t0, " ".join(argv), printed))
         self.logf = lh
         self.logf_base_fn = base_fn
         print(msg, end="")
@@ -252,7 +277,7 @@ class SvcHub(object):
     def run(self) -> None:
         self.tcpsrv.run()
 
-        thr = threading.Thread(target=self.thr_httpsrv_up)
+        thr = threading.Thread(target=self.thr_httpsrv_up, name="sig-hsrv-up2")
         thr.daemon = True
         thr.start()
 
@@ -280,7 +305,9 @@ class SvcHub(object):
                 pass
 
             self.shutdown()
-            thr.join()
+            # cant join; eats signals on win10
+            while not self.stopped:
+                time.sleep(0.1)
         else:
             self.stop_thr()
 
@@ -289,7 +316,7 @@ class SvcHub(object):
             return "cannot reload; already in progress"
 
         self.reloading = True
-        t = threading.Thread(target=self._reload)
+        t = threading.Thread(target=self._reload, name="reloading")
         t.daemon = True
         t.start()
         return "reload initiated"
@@ -316,9 +343,22 @@ class SvcHub(object):
 
     def signal_handler(self, sig: int, frame: Optional[FrameType]) -> None:
         if self.stopping:
-            return
+            if self.nsigs <= 0:
+                try:
+                    threading.Thread(target=self.pr, args=("OMBO BREAKER",)).start()
+                    time.sleep(0.1)
+                except:
+                    pass
 
-        if sig == signal.SIGUSR1:
+                if ANYWIN:
+                    os.system("taskkill /f /pid {}".format(os.getpid()))
+                else:
+                    os.kill(os.getpid(), signal.SIGKILL)
+            else:
+                self.nsigs -= 1
+                return
+
+        if not ANYWIN and sig == signal.SIGUSR1:
             self.reload_req = True
         else:
             self.stop_req = True
@@ -339,9 +379,7 @@ class SvcHub(object):
 
         ret = 1
         try:
-            with self.log_mutex:
-                print("OPYTHAT")
-
+            self.pr("OPYTHAT")
             self.tcpsrv.shutdown()
             self.broker.shutdown()
             self.up2k.shutdown()
@@ -354,19 +392,23 @@ class SvcHub(object):
                         break
 
                     if n == 3:
-                        print("waiting for thumbsrv (10sec)...")
+                        self.pr("waiting for thumbsrv (10sec)...")
 
-            print("nailed it", end="")
+            self.pr("nailed it", end="")
             ret = self.retcode
+        except:
+            self.pr("\033[31m[ error during shutdown ]\n{}\033[0m".format(min_ex()))
+            raise
         finally:
             if self.args.wintitle:
                 print("\033]0;\033\\", file=sys.stderr, end="")
                 sys.stderr.flush()
 
-            print("\033[0m")
+            self.pr("\033[0m")
             if self.logf:
                 self.logf.close()
 
+            self.stopped = True
             sys.exit(ret)
 
     def _log_disabled(self, src: str, msg: str, c: Union[int, str] = 0) -> None:
@@ -433,6 +475,10 @@ class SvcHub(object):
             if self.logf:
                 self.logf.write(msg)
 
+    def pr(self, *a: Any, **ka: Any) -> None:
+        with self.log_mutex:
+            print(*a, **ka)
+
     def check_mp_support(self) -> str:
         vmin = sys.version_info[1]
         if WINDOWS:
@@ -460,7 +506,10 @@ class SvcHub(object):
         if self.args.j == 1:
             return False
 
-        if mp.cpu_count() <= 1:
+        try:
+            if mp.cpu_count() <= 1:
+                raise Exception()
+        except:
             self.log("svchub", "only one CPU detected; multiprocessing disabled")
             return False
 
@@ -497,3 +546,16 @@ class SvcHub(object):
             sck.sendall(b"READY=1")
         except:
             self.log("sd_notify", min_ex())
+
+    def log_stacks(self) -> None:
+        td = time.time() - self.tstack
+        if td < 300:
+            self.log("stacks", "cooldown {}".format(td))
+            return
+
+        self.tstack = time.time()
+        zs = "{}\n{}".format(VERSIONS, alltrace())
+        zb = zs.encode("utf-8", "replace")
+        zb = gzip.compress(zb)
+        zs = base64.b64encode(zb).decode("ascii")
+        self.log("stacks", zs)

copyparty/szip.py

@@ -218,7 +218,7 @@ def gen_ecdr64_loc(ecdr64_pos: int) -> bytes:
 class StreamZip(StreamArc):
     def __init__(
         self,
-        log: NamedLogger,
+        log: "NamedLogger",
         fgen: Generator[dict[str, Any], None, None],
         utf8: bool = False,
         pre_crc: bool = False,
@@ -272,41 +272,44 @@ class StreamZip(StreamArc):
 
     def gen(self) -> Generator[bytes, None, None]:
         errors = []
-        for f in self.fgen:
-            if "err" in f:
-                errors.append((f["vp"], f["err"]))
-                continue
+        try:
+            for f in self.fgen:
+                if "err" in f:
+                    errors.append((f["vp"], f["err"]))
+                    continue
 
-            try:
-                for x in self.ser(f):
+                try:
+                    for x in self.ser(f):
+                        yield x
+                except GeneratorExit:
+                    raise
+                except:
+                    ex = min_ex(5, True).replace("\n", "\n-- ")
+                    errors.append((f["vp"], ex))
+
+            if errors:
+                errf, txt = errdesc(errors)
+                self.log("\n".join(([repr(errf)] + txt[1:])))
+                for x in self.ser(errf):
                     yield x
-            except:
-                ex = min_ex(5, True).replace("\n", "\n-- ")
-                errors.append((f["vp"], ex))
 
-        if errors:
-            errf, txt = errdesc(errors)
-            self.log("\n".join(([repr(errf)] + txt[1:])))
-            for x in self.ser(errf):
-                yield x
+            cdir_pos = self.pos
+            for name, sz, ts, crc, h_pos in self.items:
+                buf = gen_hdr(h_pos, name, sz, ts, self.utf8, crc, self.pre_crc)
+                yield self._ct(buf)
+            cdir_end = self.pos
 
-        cdir_pos = self.pos
-        for name, sz, ts, crc, h_pos in self.items:
-            buf = gen_hdr(h_pos, name, sz, ts, self.utf8, crc, self.pre_crc)
-            yield self._ct(buf)
-        cdir_end = self.pos
+            _, need_64 = gen_ecdr(self.items, cdir_pos, cdir_end)
+            if need_64:
+                ecdir64_pos = self.pos
+                buf = gen_ecdr64(self.items, cdir_pos, cdir_end)
+                yield self._ct(buf)
 
-        _, need_64 = gen_ecdr(self.items, cdir_pos, cdir_end)
-        if need_64:
-            ecdir64_pos = self.pos
-            buf = gen_ecdr64(self.items, cdir_pos, cdir_end)
-            yield self._ct(buf)
+                buf = gen_ecdr64_loc(ecdir64_pos)
+                yield self._ct(buf)
 
-            buf = gen_ecdr64_loc(ecdir64_pos)
-            yield self._ct(buf)
-
-        ecdr, _ = gen_ecdr(self.items, cdir_pos, cdir_end)
-        yield self._ct(ecdr)
-
-        if errors:
-            bos.unlink(errf["ap"])
+            ecdr, _ = gen_ecdr(self.items, cdir_pos, cdir_end)
+            yield self._ct(ecdr)
+        finally:
+            if errors:
+                bos.unlink(errf["ap"])

copyparty/tcpsrv.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
+import os
 import re
 import socket
 import sys
@@ -23,8 +24,10 @@ class TcpSrv(object):
         self.args = hub.args
         self.log = hub.log
 
-        self.stopping = False
+        # mp-safe since issue6056
+        socket.setdefaulttimeout(120)
 
+        self.stopping = False
         self.srv: list[socket.socket] = []
         self.nsrv = 0
         ok: dict[str, list[int]] = {}
@@ -111,6 +114,7 @@ class TcpSrv(object):
         srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
         srv.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+        srv.settimeout(None)  # < does not inherit, ^ does
         try:
             srv.bind((ip, port))
             self.srv.append(srv)
@@ -128,7 +132,7 @@ class TcpSrv(object):
             srv.listen(self.args.nc)
             ip, port = srv.getsockname()
             fno = srv.fileno()
-            msg = "listening @ {}:{}  f{}".format(ip, port, fno)
+            msg = "listening @ {}:{}  f{} p{}".format(ip, port, fno, os.getpid())
             self.log("tcpsrv", msg)
             if self.args.q:
                 print(msg)

copyparty/th_cli.py

@@ -75,7 +75,7 @@ class ThumbCli(object):
 
         preferred = self.args.th_dec[0] if self.args.th_dec else ""
 
-        if rem.startswith(".hist/th/") and rem.split(".")[-1] in ["webp", "jpg"]:
+        if rem.startswith(".hist/th/") and rem.split(".")[-1] in ["webp", "jpg", "png"]:
             return os.path.join(ptop, rem)
 
         if fmt == "j" and self.args.th_no_jpg:
@@ -128,5 +128,8 @@ class ThumbCli(object):
         if abort:
             return None
 
+        if not bos.path.getsize(os.path.join(ptop, rem)):
+            return None
+
         x = self.broker.ask("thumbsrv.get", ptop, rem, mtime, fmt)
         return x.get()  # type: ignore

copyparty/th_srv.py

@@ -14,7 +14,7 @@ from queue import Queue
 from .__init__ import TYPE_CHECKING
 from .bos import bos
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe
-from .util import BytesIO, Cooldown, fsenc, min_ex, runcmd, statdir, vsplit
+from .util import BytesIO, Cooldown, Pebkac, fsenc, min_ex, runcmd, statdir, vsplit
 
 try:
     from typing import Optional, Union
@@ -82,7 +82,7 @@ def thumb_path(histpath: str, rem: str, mtime: float, fmt: str) -> str:
     if fmt in ("opus", "caf"):
         cat = "ac"
     else:
-        fmt = "webp" if fmt == "w" else "jpg"
+        fmt = "webp" if fmt == "w" else "png" if fmt == "p" else "jpg"
         cat = "th"
 
     return "{}/{}/{}/{}.{:x}.{}".format(histpath, cat, rd, fn, int(mtime), fmt)
@@ -239,6 +239,7 @@ class ThumbSrv(object):
 
             abspath, tpath = task
             ext = abspath.split(".")[-1].lower()
+            png_ok = False
             fun = None
             if not bos.path.exists(tpath):
                 for lib in self.args.th_dec:
@@ -253,19 +254,32 @@ class ThumbSrv(object):
                     elif lib == "ff" and ext in self.fmt_ffa:
                         if tpath.endswith(".opus") or tpath.endswith(".caf"):
                             fun = self.conv_opus
+                        elif tpath.endswith(".png"):
+                            fun = self.conv_waves
+                            png_ok = True
                         else:
                             fun = self.conv_spec
 
+            if not png_ok and tpath.endswith(".png"):
+                raise Pebkac(400, "png only allowed for waveforms")
+
             if fun:
                 try:
                     fun(abspath, tpath)
-                except:
+                except Exception as ex:
                     msg = "{} could not create thumbnail of {}\n{}"
                     msg = msg.format(fun.__name__, abspath, min_ex())
                     c: Union[str, int] = 1 if "<Signals.SIG" in msg else "1;30"
                     self.log(msg, c)
-                    with open(tpath, "wb") as _:
-                        pass
+                    if getattr(ex, "returncode", 0) != 321:
+                        with open(tpath, "wb") as _:
+                            pass
+                    else:
+                        # ffmpeg may spawn empty files on windows
+                        try:
+                            os.unlink(tpath)
+                        except:
+                            pass
 
             with self.mutex:
                 subs = self.busy[tpath]
@@ -352,7 +366,7 @@ class ThumbSrv(object):
         img.write_to_file(tpath, Q=40)
 
     def conv_ffmpeg(self, abspath: str, tpath: str) -> None:
-        ret, _ = ffprobe(abspath)
+        ret, _ = ffprobe(abspath, int(self.args.th_convt / 2))
         if not ret:
             return
 
@@ -411,21 +425,30 @@ class ThumbSrv(object):
 
         c: Union[str, int] = "1;30"
         t = "FFmpeg failed (probably a corrupt video file):\n"
-        if cmd[-1].lower().endswith(b".webp") and (
-            "Error selecting an encoder" in serr
-            or "Automatic encoder selection failed" in serr
-            or "Default encoder for format webp" in serr
-            or "Please choose an encoder manually" in serr
+        if (
+            (not self.args.th_ff_jpg or time.time() - int(self.args.th_ff_jpg) < 60)
+            and cmd[-1].lower().endswith(b".webp")
+            and (
+                "Error selecting an encoder" in serr
+                or "Automatic encoder selection failed" in serr
+                or "Default encoder for format webp" in serr
+                or "Please choose an encoder manually" in serr
+            )
         ):
-            self.args.th_ff_jpg = True
+            self.args.th_ff_jpg = time.time()
             t = "FFmpeg failed because it was compiled without libwebp; enabling --th-ff-jpg to force jpeg output:\n"
+            ret = 321
             c = 1
 
         if (
+            not self.args.th_ff_swr or time.time() - int(self.args.th_ff_swr) < 60
+        ) and (
             "Requested resampling engine is unavailable" in serr
             or "output pad on Parsed_aresample_" in serr
         ):
-            t = "FFmpeg failed because it was compiled without libsox; you must set --th-ff-swr to force swr resampling:\n"
+            self.args.th_ff_swr = time.time()
+            t = "FFmpeg failed because it was compiled without libsox; enabling --th-ff-swr to force swr resampling:\n"
+            ret = 321
             c = 1
 
         lines = serr.strip("\n").split("\n")
@@ -439,8 +462,36 @@ class ThumbSrv(object):
         self.log(t + txt, c=c)
         raise sp.CalledProcessError(ret, (cmd[0], b"...", cmd[-1]))
 
+    def conv_waves(self, abspath: str, tpath: str) -> None:
+        ret, _ = ffprobe(abspath, int(self.args.th_convt / 2))
+        if "ac" not in ret:
+            raise Exception("not audio")
+
+        flt = (
+            b"[0:a:0]"
+            b"compand=.3|.3:1|1:-90/-60|-60/-40|-40/-30|-20/-20:6:0:-90:0.2"
+            b",volume=2"
+            b",showwavespic=s=2048x64:colors=white"
+            b",convolution=1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1 1 1 1 1 1 1 1 1:1 -1 1 -1 5 -1 1 -1 1"  # idk what im doing but it looks ok
+        )
+
+        # fmt: off
+        cmd = [
+            b"ffmpeg",
+            b"-nostdin",
+            b"-v", b"error",
+            b"-hide_banner",
+            b"-i", fsenc(abspath),
+            b"-filter_complex", flt,
+            b"-frames:v", b"1",
+        ]
+        # fmt: on
+
+        cmd += [fsenc(tpath)]
+        self._run_ff(cmd)
+
     def conv_spec(self, abspath: str, tpath: str) -> None:
-        ret, _ = ffprobe(abspath)
+        ret, _ = ffprobe(abspath, int(self.args.th_convt / 2))
         if "ac" not in ret:
             raise Exception("not audio")
 
@@ -461,7 +512,8 @@ class ThumbSrv(object):
             b"-hide_banner",
             b"-i", fsenc(abspath),
             b"-filter_complex", fc.encode("utf-8"),
-            b"-map", b"[o]"
+            b"-map", b"[o]",
+            b"-frames:v", b"1",
         ]
         # fmt: on
 
@@ -485,7 +537,7 @@ class ThumbSrv(object):
         if self.args.no_acode:
             raise Exception("disabled in server config")
 
-        ret, _ = ffprobe(abspath)
+        ret, _ = ffprobe(abspath, int(self.args.th_convt / 2))
         if "ac" not in ret:
             raise Exception("not audio")
 
@@ -559,14 +611,15 @@ class ThumbSrv(object):
     def clean(self, histpath: str) -> int:
         ret = 0
         for cat in ["th", "ac"]:
-            ret += self._clean(histpath, cat, "")
+            top = os.path.join(histpath, cat)
+            if not bos.path.isdir(top):
+                continue
+
+            ret += self._clean(cat, top)
 
         return ret
 
-    def _clean(self, histpath: str, cat: str, thumbpath: str) -> int:
-        if not thumbpath:
-            thumbpath = os.path.join(histpath, cat)
-
+    def _clean(self, cat: str, thumbpath: str) -> int:
         # self.log("cln {}".format(thumbpath))
         exts = ["jpg", "webp"] if cat == "th" else ["opus", "caf"]
         maxage = getattr(self.args, cat + "_maxage")
@@ -600,7 +653,7 @@ class ThumbSrv(object):
                             self.log("rm -rf [{}]".format(fp))
                             shutil.rmtree(fp, ignore_errors=True)
                 else:
-                    self._clean(histpath, cat, fp)
+                    ndirs += self._clean(cat, fp)
 
                 continue
 

copyparty/util.py

@@ -4,6 +4,8 @@ from __future__ import print_function, unicode_literals
 import base64
 import contextlib
 import hashlib
+import hmac
+import math
 import mimetypes
 import os
 import platform
@@ -21,21 +23,42 @@ import traceback
 from collections import Counter
 from datetime import datetime
 
-from .__init__ import ANYWIN, PY2, TYPE_CHECKING, VT100, WINDOWS
+from queue import Queue
+
+from .__init__ import ANYWIN, MACOS, PY2, TYPE_CHECKING, VT100, WINDOWS
+from .__version__ import S_BUILD_DT, S_VERSION
 from .stolen import surrogateescape
 
+try:
+    import ctypes
+    import fcntl
+    import termios
+except:
+    pass
+
+try:
+    from ipaddress import IPv6Address
+except:
+    pass
+
 try:
     HAVE_SQLITE3 = True
     import sqlite3  # pylint: disable=unused-import  # typechk
 except:
     HAVE_SQLITE3 = False
 
+try:
+    HAVE_PSUTIL = True
+    import psutil
+except:
+    HAVE_PSUTIL = False
+
 try:
     import types
     from collections.abc import Callable, Iterable
 
     import typing
-    from typing import Any, Generator, Optional, Protocol, Union
+    from typing import Any, Generator, Optional, Pattern, Protocol, Union
 
     class RootLogger(Protocol):
         def __call__(self, src: str, msg: str, c: Union[int, str] = 0) -> None:
@@ -49,8 +72,9 @@ except:
     pass
 
 if TYPE_CHECKING:
-    from .authsrv import VFS
+    import magic
 
+    from .authsrv import VFS
 
 FAKE_MP = False
 
@@ -72,8 +96,6 @@ else:
     from urllib import quote  # pylint: disable=no-name-in-module
     from urllib import unquote  # pylint: disable=no-name-in-module
 
-_: Any = (mp, BytesIO, quote, unquote)
-__all__ = ["mp", "BytesIO", "quote", "unquote"]
 
 try:
     struct.unpack(b">i", b"idgi")
@@ -121,6 +143,7 @@ HTTPCODE = {
     429: "Too Many Requests",
     500: "Internal Server Error",
     501: "Not Implemented",
+    503: "Service Unavailable",
 }
 
 
@@ -130,26 +153,25 @@ IMPLICATIONS = [
     ["e2tsr", "e2ts"],
     ["e2ts", "e2t"],
     ["e2t", "e2d"],
+    ["e2vu", "e2v"],
+    ["e2vp", "e2v"],
+    ["e2v", "e2d"],
 ]
 
 
 MIMES = {
-    "md": "text/plain",
-    "txt": "text/plain",
-    "js": "text/javascript",
     "opus": "audio/ogg; codecs=opus",
-    "caf": "audio/x-caf",
-    "mp3": "audio/mpeg",
-    "m4a": "audio/mp4",
-    "jpg": "image/jpeg",
 }
 
 
 def _add_mimes() -> None:
+    # `mimetypes` is woefully unpopulated on windows
+    # but will be used as fallback on linux
+
     for ln in """text css html csv
-application json wasm xml pdf rtf zip
-image webp jpeg png gif bmp
-audio aac ogg wav
+application json wasm xml pdf rtf zip jar fits wasm
+image webp jpeg png gif bmp jxl jp2 jxs jxr tiff bpg heic heif avif
+audio aac ogg wav flac ape amr
 video webm mp4 mpeg
 font woff woff2 otf ttf
 """.splitlines():
@@ -157,10 +179,35 @@ font woff woff2 otf ttf
         for v in vs.strip().split():
             MIMES[v] = "{}/{}".format(k, v)
 
+    for ln in """text md=plain txt=plain js=javascript
+application 7z=x-7z-compressed tar=x-tar bz2=x-bzip2 gz=gzip rar=x-rar-compressed zst=zstd xz=x-xz lz=lzip cpio=x-cpio
+application msi=x-ms-installer cab=vnd.ms-cab-compressed rpm=x-rpm crx=x-chrome-extension
+application epub=epub+zip mobi=x-mobipocket-ebook lit=x-ms-reader rss=rss+xml atom=atom+xml torrent=x-bittorrent
+application p7s=pkcs7-signature dcm=dicom shx=vnd.shx shp=vnd.shp dbf=x-dbf gml=gml+xml gpx=gpx+xml amf=x-amf
+application swf=x-shockwave-flash m3u=vnd.apple.mpegurl db3=vnd.sqlite3 sqlite=vnd.sqlite3
+image jpg=jpeg xpm=x-xpixmap psd=vnd.adobe.photoshop jpf=jpx tif=tiff ico=x-icon djvu=vnd.djvu
+image heic=heic-sequence heif=heif-sequence hdr=vnd.radiance svg=svg+xml
+audio caf=x-caf mp3=mpeg m4a=mp4 mid=midi mpc=musepack aif=aiff au=basic qcp=qcelp
+video mkv=x-matroska mov=quicktime avi=x-msvideo m4v=x-m4v ts=mp2t
+video asf=x-ms-asf flv=x-flv 3gp=3gpp 3g2=3gpp2 rmvb=vnd.rn-realmedia-vbr
+font ttc=collection
+""".splitlines():
+        k, ems = ln.split(" ", 1)
+        for em in ems.strip().split():
+            ext, mime = em.split("=")
+            MIMES[ext] = "{}/{}".format(k, mime)
+
 
 _add_mimes()
 
 
+EXTS: dict[str, str] = {v: k for k, v in MIMES.items()}
+
+EXTS["vnd.mozilla.apng"] = "png"
+
+MAGIC_MAP = {"jpeg": "jpg"}
+
+
 REKOBO_KEY = {
     v: ln.split(" ", 1)[0]
     for ln in """
@@ -198,6 +245,72 @@ REKOBO_KEY = {
 REKOBO_LKEY = {k.lower(): v for k, v in REKOBO_KEY.items()}
 
 
+def py_desc() -> str:
+    interp = platform.python_implementation()
+    py_ver = ".".join([str(x) for x in sys.version_info])
+    ofs = py_ver.find(".final.")
+    if ofs > 0:
+        py_ver = py_ver[:ofs]
+
+    try:
+        bitness = struct.calcsize(b"P") * 8
+    except:
+        bitness = struct.calcsize("P") * 8
+
+    host_os = platform.system()
+    compiler = platform.python_compiler()
+
+    m = re.search(r"([0-9]+\.[0-9\.]+)", platform.version())
+    os_ver = m.group(1) if m else ""
+
+    return "{:>9} v{} on {}{} {} [{}]".format(
+        interp, py_ver, host_os, bitness, os_ver, compiler
+    )
+
+
+def _sqlite_ver() -> str:
+    try:
+        co = sqlite3.connect(":memory:")
+        cur = co.cursor()
+        try:
+            vs = cur.execute("select * from pragma_compile_options").fetchall()
+        except:
+            vs = cur.execute("pragma compile_options").fetchall()
+
+        v = next(x[0].split("=")[1] for x in vs if x[0].startswith("THREADSAFE="))
+        cur.close()
+        co.close()
+    except:
+        v = "W"
+
+    return "{}*{}".format(sqlite3.sqlite_version, v)
+
+
+try:
+    SQLITE_VER = _sqlite_ver()
+except:
+    SQLITE_VER = "(None)"
+
+try:
+    from jinja2 import __version__ as JINJA_VER
+except:
+    JINJA_VER = "(None)"
+
+try:
+    from pyftpdlib.__init__ import __ver__ as PYFTPD_VER
+except:
+    PYFTPD_VER = "(None)"
+
+
+VERSIONS = "copyparty v{} ({})\n{}\n   sqlite v{} | jinja v{} | pyftpd v{}".format(
+    S_VERSION, S_BUILD_DT, py_desc(), SQLITE_VER, JINJA_VER, PYFTPD_VER
+)
+
+
+_: Any = (mp, BytesIO, quote, unquote, SQLITE_VER, JINJA_VER, PYFTPD_VER)
+__all__ = ["mp", "BytesIO", "quote", "unquote", "SQLITE_VER", "JINJA_VER", "PYFTPD_VER"]
+
+
 class Cooldown(object):
     def __init__(self, maxage: float) -> None:
         self.maxage = maxage
@@ -233,7 +346,7 @@ class _Unrecv(object):
     undo any number of socket recv ops
     """
 
-    def __init__(self, s: socket.socket, log: Optional[NamedLogger]) -> None:
+    def __init__(self, s: socket.socket, log: Optional["NamedLogger"]) -> None:
         self.s = s
         self.log = log
         self.buf: bytes = b""
@@ -277,7 +390,7 @@ class _LUnrecv(object):
     with expensive debug logging
     """
 
-    def __init__(self, s: socket.socket, log: Optional[NamedLogger]) -> None:
+    def __init__(self, s: socket.socket, log: Optional["NamedLogger"]) -> None:
         self.s = s
         self.log = log
         self.buf = b""
@@ -414,6 +527,260 @@ class ProgressPrinter(threading.Thread):
         sys.stdout.flush()  # necessary on win10 even w/ stderr btw
 
 
+class MTHash(object):
+    def __init__(self, cores: int):
+        self.pp: Optional[ProgressPrinter] = None
+        self.f: Optional[typing.BinaryIO] = None
+        self.sz = 0
+        self.csz = 0
+        self.stop = False
+        self.omutex = threading.Lock()
+        self.imutex = threading.Lock()
+        self.work_q: Queue[int] = Queue()
+        self.done_q: Queue[tuple[int, str, int, int]] = Queue()
+        self.thrs = []
+        for n in range(cores):
+            t = threading.Thread(target=self.worker, name="mth-" + str(n))
+            t.daemon = True
+            t.start()
+            self.thrs.append(t)
+
+    def hash(
+        self,
+        f: typing.BinaryIO,
+        fsz: int,
+        chunksz: int,
+        pp: Optional[ProgressPrinter] = None,
+        prefix: str = "",
+        suffix: str = "",
+    ) -> list[tuple[str, int, int]]:
+        with self.omutex:
+            self.f = f
+            self.sz = fsz
+            self.csz = chunksz
+
+            chunks: dict[int, tuple[str, int, int]] = {}
+            nchunks = int(math.ceil(fsz / chunksz))
+            for nch in range(nchunks):
+                self.work_q.put(nch)
+
+            ex = ""
+            for nch in range(nchunks):
+                qe = self.done_q.get()
+                try:
+                    nch, dig, ofs, csz = qe
+                    chunks[nch] = (dig, ofs, csz)
+                except:
+                    ex = ex or str(qe)
+
+                if pp:
+                    mb = int((fsz - nch * chunksz) / 1024 / 1024)
+                    pp.msg = prefix + str(mb) + suffix
+
+            if ex:
+                raise Exception(ex)
+
+            ret = []
+            for n in range(nchunks):
+                ret.append(chunks[n])
+
+            self.f = None
+            self.csz = 0
+            self.sz = 0
+            return ret
+
+    def worker(self) -> None:
+        while True:
+            ofs = self.work_q.get()
+            try:
+                v = self.hash_at(ofs)
+            except Exception as ex:
+                v = str(ex)  # type: ignore
+
+            self.done_q.put(v)
+
+    def hash_at(self, nch: int) -> tuple[int, str, int, int]:
+        f = self.f
+        ofs = ofs0 = nch * self.csz
+        chunk_sz = chunk_rem = min(self.csz, self.sz - ofs)
+        if self.stop:
+            return nch, "", ofs0, chunk_sz
+
+        assert f
+        hashobj = hashlib.sha512()
+        while chunk_rem > 0:
+            with self.imutex:
+                f.seek(ofs)
+                buf = f.read(min(chunk_rem, 1024 * 1024 * 12))
+
+            if not buf:
+                raise Exception("EOF at " + str(ofs))
+
+            hashobj.update(buf)
+            chunk_rem -= len(buf)
+            ofs += len(buf)
+
+        bdig = hashobj.digest()[:33]
+        udig = base64.urlsafe_b64encode(bdig).decode("utf-8")
+        return nch, udig, ofs0, chunk_sz
+
+
+class HMaccas(object):
+    def __init__(self, keypath: str, retlen: int) -> None:
+        self.retlen = retlen
+        self.cache: dict[bytes, str] = {}
+        try:
+            with open(keypath, "rb") as f:
+                self.key = f.read()
+                if len(self.key) != 64:
+                    raise Exception()
+        except:
+            self.key = os.urandom(64)
+            with open(keypath, "wb") as f:
+                f.write(self.key)
+
+    def b(self, msg: bytes) -> str:
+        try:
+            return self.cache[msg]
+        except:
+            zb = hmac.new(self.key, msg, hashlib.sha512).digest()
+            zs = base64.urlsafe_b64encode(zb)[: self.retlen].decode("utf-8")
+            self.cache[msg] = zs
+            return zs
+
+    def s(self, msg: str) -> str:
+        return self.b(msg.encode("utf-8", "replace"))
+
+
+class Magician(object):
+    def __init__(self) -> None:
+        self.bad_magic = False
+        self.mutex = threading.Lock()
+        self.magic: Optional["magic.Magic"] = None
+
+    def ext(self, fpath: str) -> str:
+        import magic
+
+        try:
+            if self.bad_magic:
+                raise Exception()
+
+            if not self.magic:
+                try:
+                    with self.mutex:
+                        if not self.magic:
+                            self.magic = magic.Magic(uncompress=False, extension=True)
+                except:
+                    self.bad_magic = True
+                    raise
+
+            with self.mutex:
+                ret = self.magic.from_file(fpath)
+        except:
+            ret = "?"
+
+        ret = ret.split("/")[0]
+        ret = MAGIC_MAP.get(ret, ret)
+        if "?" not in ret:
+            return ret
+
+        mime = magic.from_file(fpath, mime=True)
+        mime = re.split("[; ]", mime, 1)[0]
+        try:
+            return EXTS[mime]
+        except:
+            pass
+
+        mg = mimetypes.guess_extension(mime)
+        if mg:
+            return mg[1:]
+        else:
+            raise Exception()
+
+
+class Garda(object):
+    """ban clients for repeated offenses"""
+
+    def __init__(self, cfg: str) -> None:
+        try:
+            a, b, c = cfg.strip().split(",")
+            self.lim = int(a)
+            self.win = int(b) * 60
+            self.pen = int(c) * 60
+        except:
+            self.lim = self.win = self.pen = 0
+
+        self.ct: dict[str, list[int]] = {}
+        self.prev: dict[str, str] = {}
+        self.last_cln = 0
+
+    def cln(self, ip: str) -> None:
+        n = 0
+        ok = int(time.time() - self.win)
+        for v in self.ct[ip]:
+            if v < ok:
+                n += 1
+            else:
+                break
+        if n:
+            te = self.ct[ip][n:]
+            if te:
+                self.ct[ip] = te
+            else:
+                del self.ct[ip]
+                try:
+                    del self.prev[ip]
+                except:
+                    pass
+
+    def allcln(self) -> None:
+        for k in list(self.ct):
+            self.cln(k)
+
+        self.last_cln = int(time.time())
+
+    def bonk(self, ip: str, prev: str) -> tuple[int, str]:
+        if not self.lim:
+            return 0, ip
+
+        if ":" in ip and not PY2:
+            # assume /64 clients; drop 4 groups
+            ip = IPv6Address(ip).exploded[:-20]
+
+        if prev:
+            if self.prev.get(ip) == prev:
+                return 0, ip
+
+            self.prev[ip] = prev
+
+        now = int(time.time())
+        try:
+            self.ct[ip].append(now)
+        except:
+            self.ct[ip] = [now]
+
+        if now - self.last_cln > 300:
+            self.allcln()
+        else:
+            self.cln(ip)
+
+        if len(self.ct[ip]) >= self.lim:
+            return now + self.pen, ip
+        else:
+            return 0, ip
+
+
+if WINDOWS and sys.version_info < (3, 8):
+    _popen = sp.Popen
+
+    def _spopen(c, *a, **ka):
+        enc = sys.getfilesystemencoding()
+        c = [x.decode(enc, "replace") if hasattr(x, "decode") else x for x in c]
+        return _popen(c, *a, **ka)
+
+    sp.Popen = _spopen
+
+
 def uprint(msg: str) -> None:
     try:
         print(msg, end="")
@@ -496,12 +863,43 @@ def start_stackmon(arg_str: str, nid: int) -> None:
 
 def stackmon(fp: str, ival: float, suffix: str) -> None:
     ctr = 0
+    fp0 = fp
     while True:
         ctr += 1
+        fp = fp0
         time.sleep(ival)
         st = "{}, {}\n{}".format(ctr, time.time(), alltrace())
+        buf = st.encode("utf-8", "replace")
+
+        if fp.endswith(".gz"):
+            import gzip
+
+            # 2459b 2304b 2241b 2202b 2194b 2191b lv3..8
+            # 0.06s 0.08s 0.11s 0.13s 0.16s 0.19s
+            buf = gzip.compress(buf, compresslevel=6)
+
+        elif fp.endswith(".xz"):
+            import lzma
+
+            # 2276b 2216b 2200b 2192b 2168b lv0..4
+            # 0.04s 0.10s 0.22s 0.41s 0.70s
+            buf = lzma.compress(buf, preset=0)
+
+        if "%" in fp:
+            dt = datetime.utcnow()
+            for fs in "YmdHMS":
+                fs = "%" + fs
+                if fs in fp:
+                    fp = fp.replace(fs, dt.strftime(fs))
+
+        if "/" in fp:
+            try:
+                os.makedirs(fp.rsplit("/", 1)[0])
+            except:
+                pass
+
         with open(fp + suffix, "wb") as f:
-            f.write(st.encode("utf-8", "replace"))
+            f.write(buf)
 
 
 def start_log_thrs(
@@ -589,6 +987,7 @@ def ren_open(
         ext = bname[ofs:] + ext
         bname = bname[:ofs]
 
+    asciified = False
     b64 = ""
     while True:
         try:
@@ -614,11 +1013,20 @@ def ren_open(
 
         except OSError as ex_:
             ex = ex_
-            if ex.errno not in [36, 63] and (not WINDOWS or ex.errno != 22):
+
+            if ex.errno == 22 and not asciified:
+                asciified = True
+                bname, fname = [
+                    zs.encode("ascii", "replace").decode("ascii").replace("?", "_")
+                    for zs in [bname, fname]
+                ]
+                continue
+
+            if ex.errno not in [36, 63, 95] and (not WINDOWS or ex.errno != 22):
                 raise
 
         if not b64:
-            zs = (bname + ext).encode("utf-8", "replace")
+            zs = (orig_name + "\n" + suffix).encode("utf-8", "replace")
             zs = hashlib.sha512(zs).digest()[:12]
             b64 = base64.urlsafe_b64encode(zs).decode("utf-8")
 
@@ -642,7 +1050,9 @@ def ren_open(
 
 
 class MultipartParser(object):
-    def __init__(self, log_func: NamedLogger, sr: Unrecv, http_headers: dict[str, str]):
+    def __init__(
+        self, log_func: "NamedLogger", sr: Unrecv, http_headers: dict[str, str]
+    ):
         self.sr = sr
         self.log = log_func
         self.headers = http_headers
@@ -905,6 +1315,24 @@ def gen_filekey(salt: str, fspath: str, fsize: int, inode: int) -> str:
     ).decode("ascii")
 
 
+def gen_filekey_dbg(
+    salt: str,
+    fspath: str,
+    fsize: int,
+    inode: int,
+    log: "NamedLogger",
+    log_ptn: Optional[Pattern[str]],
+) -> str:
+    ret = gen_filekey(salt, fspath, fsize, inode)
+
+    assert log_ptn
+    if log_ptn.search(fspath):
+        t = "fk({}) salt({}) size({}) inode({}) fspath({})"
+        log(t.format(ret[:8], salt, fsize, inode, fspath))
+
+    return ret
+
+
 def gencookie(k: str, v: str, dur: Optional[int]) -> str:
     v = v.replace(";", "")
     if dur:
@@ -962,6 +1390,11 @@ def s2hms(s: float, optional_h: bool = False) -> str:
     return "{}:{:02}:{:02}".format(h, m, s)
 
 
+def djoin(*paths: str) -> str:
+    """joins without adding a trailing slash on blank args"""
+    return os.path.join(*[x for x in paths if x])
+
+
 def uncyg(path: str) -> str:
     if len(path) < 2 or not path.startswith("/"):
         return path
@@ -1111,6 +1544,10 @@ def vsplit(vpath: str) -> tuple[str, str]:
     return vpath.rsplit("/", 1)  # type: ignore
 
 
+def vjoin(rd: str, fn: str) -> str:
+    return rd + "/" + fn if rd else fn
+
+
 def w8dec(txt: bytes) -> str:
     """decodes filesystem-bytes to wtf8"""
     if PY2:
@@ -1164,15 +1601,30 @@ def s3enc(mem_cur: "sqlite3.Cursor", rd: str, fn: str) -> tuple[str, str]:
 
 
 def s3dec(rd: str, fn: str) -> tuple[str, str]:
-    ret = []
-    for v in [rd, fn]:
-        if v.startswith("//"):
-            ret.append(w8b64dec(v[2:]))
-            # self.log("mojide [{}] {}".format(ret[-1], v[2:]))
-        else:
-            ret.append(v)
+    return (
+        w8b64dec(rd[2:]) if rd.startswith("//") else rd,
+        w8b64dec(fn[2:]) if fn.startswith("//") else fn,
+    )
 
-    return ret[0], ret[1]
+
+def db_ex_chk(log: "NamedLogger", ex: Exception, db_path: str) -> bool:
+    if str(ex) != "database is locked":
+        return False
+
+    thr = threading.Thread(target=lsof, args=(log, db_path), name="dbex")
+    thr.daemon = True
+    thr.start()
+
+    return True
+
+
+def lsof(log: "NamedLogger", abspath: str) -> None:
+    try:
+        rc, so, se = runcmd([b"lsof", b"-R", fsenc(abspath)], timeout=45)
+        zs = (so.strip() + "\n" + se.strip()).strip()
+        log("lsof {} = {}\n{}".format(abspath, rc, zs), 3)
+    except:
+        log("lsof failed; " + min_ex(), 3)
 
 
 def atomic_move(usrc: str, udst: str) -> None:
@@ -1187,6 +1639,73 @@ def atomic_move(usrc: str, udst: str) -> None:
         os.rename(src, dst)
 
 
+def get_df(abspath: str) -> tuple[Optional[int], Optional[int]]:
+    try:
+        # some fuses misbehave
+        if ANYWIN:
+            bfree = ctypes.c_ulonglong(0)
+            ctypes.windll.kernel32.GetDiskFreeSpaceExW(  # type: ignore
+                ctypes.c_wchar_p(abspath), None, None, ctypes.pointer(bfree)
+            )
+            return (bfree.value, None)
+        else:
+            sv = os.statvfs(fsenc(abspath))
+            free = sv.f_frsize * sv.f_bfree
+            total = sv.f_frsize * sv.f_blocks
+            return (free, total)
+    except:
+        return (None, None)
+
+
+if not ANYWIN and not MACOS:
+
+    def siocoutq(sck: socket.socket) -> int:
+        # SIOCOUTQ^sockios.h == TIOCOUTQ^ioctl.h
+        try:
+            zb = fcntl.ioctl(sck.fileno(), termios.TIOCOUTQ, b"AAAA")
+            return sunpack(b"I", zb)[0]  # type: ignore
+        except:
+            return 1
+
+else:
+    # macos: getsockopt(fd, SOL_SOCKET, SO_NWRITE, ...)
+    # windows: TcpConnectionEstatsSendBuff
+
+    def siocoutq(sck: socket.socket) -> int:
+        return 1
+
+
+def shut_socket(log: "NamedLogger", sck: socket.socket, timeout: int = 3) -> None:
+    t0 = time.time()
+    fd = sck.fileno()
+    if fd == -1:
+        sck.close()
+        return
+
+    try:
+        sck.settimeout(timeout)
+        sck.shutdown(socket.SHUT_WR)
+        try:
+            while time.time() - t0 < timeout:
+                if not siocoutq(sck):
+                    # kernel says tx queue empty, we good
+                    break
+
+                # on windows in particular, drain rx until client shuts
+                if not sck.recv(32 * 1024):
+                    break
+
+            sck.shutdown(socket.SHUT_RDWR)
+        except:
+            pass
+    finally:
+        td = time.time() - t0
+        if td >= 1:
+            log("shut({}) in {:.3f} sec".format(fd, td), "1;30")
+
+        sck.close()
+
+
 def read_socket(sr: Unrecv, total_size: int) -> Generator[bytes, None, None]:
     remains = total_size
     while remains > 0:
@@ -1213,7 +1732,7 @@ def read_socket_unbounded(sr: Unrecv) -> Generator[bytes, None, None]:
 
 
 def read_socket_chunked(
-    sr: Unrecv, log: Optional[NamedLogger] = None
+    sr: Unrecv, log: Optional["NamedLogger"] = None
 ) -> Generator[bytes, None, None]:
     err = "upload aborted: expected chunk length, got [{}] |{}| instead"
     while True:
@@ -1291,7 +1810,7 @@ def hashcopy(
 
 
 def sendfile_py(
-    log: NamedLogger,
+    log: "NamedLogger",
     lower: int,
     upper: int,
     f: typing.BinaryIO,
@@ -1319,7 +1838,7 @@ def sendfile_py(
 
 
 def sendfile_kern(
-    log: NamedLogger,
+    log: "NamedLogger",
     lower: int,
     upper: int,
     f: typing.BinaryIO,
@@ -1360,7 +1879,7 @@ def sendfile_kern(
 
 
 def statdir(
-    logger: Optional[RootLogger], scandir: bool, lstat: bool, top: str
+    logger: Optional["RootLogger"], scandir: bool, lstat: bool, top: str
 ) -> Generator[tuple[str, os.stat_result], None, None]:
     if lstat and ANYWIN:
         lstat = False
@@ -1403,9 +1922,10 @@ def statdir(
 
 
 def rmdirs(
-    logger: RootLogger, scandir: bool, lstat: bool, top: str, depth: int
+    logger: "RootLogger", scandir: bool, lstat: bool, top: str, depth: int
 ) -> tuple[list[str], list[str]]:
-    if not os.path.exists(fsenc(top)) or not os.path.isdir(fsenc(top)):
+    """rmdir all descendants, then self"""
+    if not os.path.isdir(fsenc(top)):
         top = os.path.dirname(top)
         depth -= 1
 
@@ -1429,6 +1949,21 @@ def rmdirs(
     return ok, ng
 
 
+def rmdirs_up(top: str) -> tuple[list[str], list[str]]:
+    """rmdir on self, then all parents"""
+    try:
+        os.rmdir(fsenc(top))
+    except:
+        return [], [top]
+
+    par = os.path.dirname(top)
+    if not par:
+        return [top], []
+
+    ok, ng = rmdirs_up(par)
+    return [top] + ok, ng
+
+
 def unescape_cookie(orig: str) -> str:
     # mw=idk; doot=qwe%2Crty%3Basd+fgh%2Bjkl%25zxc%26vbn  # qwe,rty;asd fgh+jkl%zxc&vbn
     ret = ""
@@ -1479,22 +2014,113 @@ def guess_mime(url: str, fallback: str = "application/octet-stream") -> str:
     return ret
 
 
+def getalive(pids: list[int], pgid: int) -> list[int]:
+    alive = []
+    for pid in pids:
+        try:
+            if pgid:
+                # check if still one of ours
+                if os.getpgid(pid) == pgid:
+                    alive.append(pid)
+            else:
+                # windows doesn't have pgroups; assume
+                psutil.Process(pid)
+                alive.append(pid)
+        except:
+            pass
+
+    return alive
+
+
+def killtree(root: int) -> None:
+    """still racy but i tried"""
+    try:
+        # limit the damage where possible (unixes)
+        pgid = os.getpgid(os.getpid())
+    except:
+        pgid = 0
+
+    if HAVE_PSUTIL:
+        pids = [root]
+        parent = psutil.Process(root)
+        for child in parent.children(recursive=True):
+            pids.append(child.pid)
+            child.terminate()
+        parent.terminate()
+        parent = None
+    elif pgid:
+        # linux-only
+        pids = []
+        chk = [root]
+        while chk:
+            pid = chk[0]
+            chk = chk[1:]
+            pids.append(pid)
+            _, t, _ = runcmd(["pgrep", "-P", str(pid)])
+            chk += [int(x) for x in t.strip().split("\n") if x]
+
+        pids = getalive(pids, pgid)  # filter to our pgroup
+        for pid in pids:
+            os.kill(pid, signal.SIGTERM)
+    else:
+        # windows gets minimal effort sorry
+        os.kill(pid, signal.SIGTERM)
+        return
+
+    for n in range(10):
+        time.sleep(0.1)
+        pids = getalive(pids, pgid)
+        if not pids or n > 3 and pids == [root]:
+            break
+
+    for pid in pids:
+        try:
+            os.kill(pid, signal.SIGKILL)
+        except:
+            pass
+
+
 def runcmd(
     argv: Union[list[bytes], list[str]], timeout: Optional[int] = None, **ka: Any
 ) -> tuple[int, str, str]:
-    p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE, **ka)
+    kill = ka.pop("kill", "t")  # [t]ree [m]ain [n]one
+    capture = ka.pop("capture", 3)  # 0=none 1=stdout 2=stderr 3=both
+
+    sin = ka.pop("sin", None)
+    if sin:
+        ka["stdin"] = sp.PIPE
+
+    cout = sp.PIPE if capture in [1, 3] else None
+    cerr = sp.PIPE if capture in [2, 3] else None
+
+    p = sp.Popen(argv, stdout=cout, stderr=cerr, **ka)
     if not timeout or PY2:
-        stdout, stderr = p.communicate()
+        stdout, stderr = p.communicate(sin)
     else:
         try:
-            stdout, stderr = p.communicate(timeout=timeout)
+            stdout, stderr = p.communicate(sin, timeout=timeout)
         except sp.TimeoutExpired:
-            p.kill()
-            stdout, stderr = p.communicate()
+            if kill == "n":
+                return -18, "", ""  # SIGCONT; leave it be
+            elif kill == "m":
+                p.kill()
+            else:
+                killtree(p.pid)
 
-    stdout = stdout.decode("utf-8", "replace")
-    stderr = stderr.decode("utf-8", "replace")
-    return p.returncode, stdout, stderr
+            try:
+                stdout, stderr = p.communicate(timeout=1)
+            except:
+                stdout = b""
+                stderr = b""
+
+    stdout = stdout.decode("utf-8", "replace") if cout else b""
+    stderr = stderr.decode("utf-8", "replace") if cerr else b""
+
+    rc = p.returncode
+    if rc is None:
+        rc = -14  # SIGALRM; failed to kill
+
+    return rc, stdout, stderr
 
 
 def chkcmd(argv: Union[list[bytes], list[str]], **ka: Any) -> tuple[str, str]:
@@ -1521,7 +2147,7 @@ def retchk(
     rc: int,
     cmd: Union[list[bytes], list[str]],
     serr: str,
-    logger: Optional[NamedLogger] = None,
+    logger: Optional["NamedLogger"] = None,
     color: Union[int, str] = 0,
     verbose: bool = False,
 ) -> None:
@@ -1573,29 +2199,6 @@ def gzip_orig_sz(fn: str) -> int:
         return sunpack(b"I", rv)[0]  # type: ignore
 
 
-def py_desc() -> str:
-    interp = platform.python_implementation()
-    py_ver = ".".join([str(x) for x in sys.version_info])
-    ofs = py_ver.find(".final.")
-    if ofs > 0:
-        py_ver = py_ver[:ofs]
-
-    try:
-        bitness = struct.calcsize(b"P") * 8
-    except:
-        bitness = struct.calcsize("P") * 8
-
-    host_os = platform.system()
-    compiler = platform.python_compiler()
-
-    m = re.search(r"([0-9]+\.[0-9\.]+)", platform.version())
-    os_ver = m.group(1) if m else ""
-
-    return "{:>9} v{} on {}{} {} [{}]".format(
-        interp, py_ver, host_os, bitness, os_ver, compiler
-    )
-
-
 def align_tab(lines: list[str]) -> list[str]:
     rows = []
     ncols = 0
@@ -1695,16 +2298,12 @@ def termsize() -> tuple[int, int]:
     # from hashwalk
     env = os.environ
 
-    def ioctl_GWINSZ(fd):
+    def ioctl_GWINSZ(fd: int) -> Optional[tuple[int, int]]:
         try:
-            import fcntl
-            import struct
-            import termios
-
-            cr = struct.unpack("hh", fcntl.ioctl(fd, termios.TIOCGWINSZ, "1234"))
+            cr = sunpack(b"hh", fcntl.ioctl(fd, termios.TIOCGWINSZ, b"AAAA"))
+            return int(cr[1]), int(cr[0])
         except:
-            return
-        return cr
+            return None
 
     cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
     if not cr:
@@ -1715,13 +2314,13 @@ def termsize() -> tuple[int, int]:
         except:
             pass
 
-    if not cr:
-        try:
-            cr = (env["LINES"], env["COLUMNS"])
-        except:
-            cr = (25, 80)
+    if cr:
+        return cr
 
-    return int(cr[1]), int(cr[0])
+    try:
+        return int(env["COLUMNS"]), int(env["LINES"])
+    except:
+        return 80, 25
 
 
 class Pebkac(Exception):

copyparty/web/baguettebox.js

@@ -21,7 +21,7 @@ window.baguetteBox = (function () {
             afterHide: null,
             onChange: null,
         },
-        overlay, slider, btnPrev, btnNext, btnHelp, btnAnim, btnRotL, btnRotR, btnSel, btnVmode, btnClose,
+        overlay, slider, btnPrev, btnNext, btnHelp, btnAnim, btnRotL, btnRotR, btnSel, btnFull, btnVmode, btnClose,
         currentGallery = [],
         currentIndex = 0,
         isOverlayVisible = false,
@@ -37,6 +37,9 @@ window.baguetteBox = (function () {
         vmute = false,
         vloop = sread('vmode') == 'L',
         vnext = sread('vmode') == 'C',
+        loopA = null,
+        loopB = null,
+        url_ts = null,
         resume_mp = false;
 
     var onFSC = function (e) {
@@ -182,6 +185,7 @@ window.baguetteBox = (function () {
                 '<button id="bbox-rotl" type="button">↶</button>' +
                 '<button id="bbox-rotr" type="button">↷</button>' +
                 '<button id="bbox-tsel" type="button">sel</button>' +
+                '<button id="bbox-full" type="button">⛶</button>' +
                 '<button id="bbox-vmode" type="button" tt="a"></button>' +
                 '<button id="bbox-close" type="button" aria-label="Close">X</button>' +
                 '</div></div>'
@@ -198,9 +202,9 @@ window.baguetteBox = (function () {
         btnRotL = ebi('bbox-rotl');
         btnRotR = ebi('bbox-rotr');
         btnSel = ebi('bbox-tsel');
+        btnFull = ebi('bbox-full');
         btnVmode = ebi('bbox-vmode');
         btnClose = ebi('bbox-close');
-        bindEvents();
     }
 
     function halp() {
@@ -215,23 +219,26 @@ window.baguetteBox = (function () {
             ['home', 'first file'],
             ['end', 'last file'],
             ['R', 'rotate (shift=ccw)'],
+            ['F', 'toggle fullscreen'],
             ['S', 'toggle file selection'],
             ['space, P, K', 'video: play / pause'],
             ['U', 'video: seek 10sec back'],
             ['P', 'video: seek 10sec ahead'],
+            ['0..9', 'video: seek 0%..90%'],
             ['M', 'video: toggle mute'],
             ['V', 'video: toggle loop'],
             ['C', 'video: toggle auto-next'],
-            ['F', 'video: toggle fullscreen'],
+            ['<code>[</code>, <code>]</code>', 'video: loop start / end'],
         ],
-            d = mknod('table'),
+            d = mknod('table', 'bbox-halp'),
             html = ['<tbody>'];
 
         for (var a = 0; a < list.length; a++)
             html.push('<tr><td>' + list[a][0] + '</td><td>' + list[a][1] + '</td></tr>');
 
+        html.push('<tr><td colspan="2">tap middle of img to hide btns</td></tr>');
+        html.push('<tr><td colspan="2">tap left/right sides for prev/next</td></tr>');
         d.innerHTML = html.join('\n') + '</tbody>';
-        d.setAttribute('id', 'bbox-halp');
         d.onclick = function () {
             overlay.removeChild(d);
         };
@@ -242,7 +249,7 @@ window.baguetteBox = (function () {
         if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing || modal.busy)
             return;
 
-        var k = e.code + '', v = vid();
+        var k = e.code + '', v = vid(), pos = -1;
 
         if (k == "ArrowLeft" || k == "KeyJ")
             showPreviousImage();
@@ -258,6 +265,8 @@ window.baguetteBox = (function () {
             playpause();
         else if (k == "KeyU" || k == "KeyO")
             relseek(k == "KeyU" ? -10 : 10);
+        else if (k.indexOf('Digit') === 0)
+            vid().currentTime = vid().duration * parseInt(k.slice(-1)) * 0.1;
         else if (k == "KeyM" && v) {
             v.muted = vmute = !vmute;
             mp_ctl();
@@ -273,19 +282,17 @@ window.baguetteBox = (function () {
             setVmode();
         }
         else if (k == "KeyF")
-            try {
-                if (isFullscreen)
-                    document.exitFullscreen();
-                else
-                    v.requestFullscreen();
-            }
-            catch (ex) { }
+            tglfull();
         else if (k == "KeyS")
             tglsel();
         else if (k == "KeyR")
             rotn(e.shiftKey ? -1 : 1);
         else if (k == "KeyY")
             dlpic();
+        else if (k == "BracketLeft")
+            setloop(1);
+        else if (k == "BracketRight")
+            setloop(2);
     }
 
     function anim() {
@@ -354,6 +361,16 @@ window.baguetteBox = (function () {
                 return [name, a, files, ebi(files[a].id)];
     }
 
+    function tglfull() {
+        try {
+            if (isFullscreen)
+                document.exitFullscreen();
+            else
+                (vid() || ebi('bbox-overlay')).requestFullscreen();
+        }
+        catch (ex) { alert(ex); }
+    }
+
     function tglsel() {
         var o = findfile()[3];
         clmod(o.closest('tr'), 'sel', 't');
@@ -416,6 +433,9 @@ window.baguetteBox = (function () {
     var nonPassiveEvent = passiveSupp ? { passive: true } : null;
 
     function bindEvents() {
+        bind(document, 'keydown', keyDownHandler);
+        bind(document, 'keyup', keyUpHandler);
+        bind(document, 'fullscreenchange', onFSC);
         bind(overlay, 'click', overlayClickHandler);
         bind(btnPrev, 'click', showPreviousImage);
         bind(btnNext, 'click', showNextImage);
@@ -426,6 +446,7 @@ window.baguetteBox = (function () {
         bind(btnRotL, 'click', rotl);
         bind(btnRotR, 'click', rotr);
         bind(btnSel, 'click', tglsel);
+        bind(btnFull, 'click', tglfull);
         bind(slider, 'contextmenu', contextmenuHandler);
         bind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
         bind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
@@ -434,6 +455,9 @@ window.baguetteBox = (function () {
     }
 
     function unbindEvents() {
+        unbind(document, 'keydown', keyDownHandler);
+        unbind(document, 'keyup', keyUpHandler);
+        unbind(document, 'fullscreenchange', onFSC);
         unbind(overlay, 'click', overlayClickHandler);
         unbind(btnPrev, 'click', showPreviousImage);
         unbind(btnNext, 'click', showNextImage);
@@ -444,6 +468,7 @@ window.baguetteBox = (function () {
         unbind(btnRotL, 'click', rotl);
         unbind(btnRotR, 'click', rotr);
         unbind(btnSel, 'click', tglsel);
+        unbind(btnFull, 'click', tglfull);
         unbind(slider, 'contextmenu', contextmenuHandler);
         unbind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
         unbind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
@@ -464,9 +489,8 @@ window.baguetteBox = (function () {
         var imagesFiguresIds = [];
         var imagesCaptionsIds = [];
         for (var i = 0, fullImage; i < gallery.length; i++) {
-            fullImage = mknod('div');
+            fullImage = mknod('div', 'baguette-img-' + i);
             fullImage.className = 'full-image';
-            fullImage.id = 'baguette-img-' + i;
             imagesElements.push(fullImage);
 
             imagesFiguresIds.push('bbox-figure-' + i);
@@ -508,9 +532,7 @@ window.baguetteBox = (function () {
         if (overlay.style.display === 'block')
             return;
 
-        bind(document, 'keydown', keyDownHandler);
-        bind(document, 'keyup', keyUpHandler);
-        bind(document, 'fullscreenchange', onFSC);
+        bindEvents();
         currentIndex = chosenImageIndex;
         touch = {
             count: 0,
@@ -522,6 +544,10 @@ window.baguetteBox = (function () {
             preloadPrev(currentIndex);
         });
 
+        clmod(ebi('bbox-btns'), 'off');
+        clmod(btnPrev, 'off');
+        clmod(btnNext, 'off');
+
         updateOffset();
         overlay.style.display = 'block';
         // Fade in overlay
@@ -534,9 +560,10 @@ window.baguetteBox = (function () {
                 options.afterShow();
         }, 50);
 
-        if (options.onChange)
+        if (options.onChange && !url_ts)
             options.onChange(currentIndex, imagesElements.length);
 
+        url_ts = null;
         documentLastFocus = document.activeElement;
         btnClose.focus();
         isOverlayVisible = true;
@@ -553,9 +580,13 @@ window.baguetteBox = (function () {
             return;
 
         sethash('');
-        unbind(document, 'keydown', keyDownHandler);
-        unbind(document, 'keyup', keyUpHandler);
-        unbind(document, 'fullscreenchange', onFSC);
+        unbindEvents();
+        try {
+            document.exitFullscreen();
+            isFullscreen = false;
+        }
+        catch (ex) { }
+
         // Fade out and hide the overlay
         overlay.className = '';
         setTimeout(function () {
@@ -601,16 +632,14 @@ window.baguetteBox = (function () {
         if (is_vid && index != currentIndex)
             return;  // no preload
 
-        var figure = mknod('figure');
-        figure.id = 'bbox-figure-' + index;
+        var figure = mknod('figure', 'bbox-figure-' + index);
         figure.innerHTML = '<div class="bbox-spinner">' +
             '<div class="bbox-double-bounce1"></div>' +
             '<div class="bbox-double-bounce2"></div>' +
             '</div>';
 
         if (options.captions && imageCaption) {
-            var figcaption = mknod('figcaption');
-            figcaption.id = 'bbox-figcaption-' + index;
+            var figcaption = mknod('figcaption', 'bbox-figcaption-' + index);
             figcaption.innerHTML = imageCaption;
             figure.appendChild(figcaption);
         }
@@ -670,18 +699,12 @@ window.baguetteBox = (function () {
             showOverlay(index);
             return true;
         }
-        if (index < 0) {
-            if (options.animation)
-                bounceAnimation('left');
 
-            return false;
-        }
-        if (index >= imagesElements.length) {
-            if (options.animation)
-                bounceAnimation('right');
+        if (index < 0)
+            return bounceAnimation('left');
 
-            return false;
-        }
+        if (index >= imagesElements.length)
+            return bounceAnimation('right');
 
         var v = vid();
         if (v) {
@@ -789,8 +812,18 @@ window.baguetteBox = (function () {
     }
 
     function playvid(play) {
-        if (vid())
-            vid()[play ? 'play' : 'pause']();
+        if (!play) {
+            timer.rm(loopchk);
+            loopA = loopB = null;
+        }
+
+        var v = vid();
+        if (!v)
+            return;
+
+        v[play ? 'play' : 'pause']();
+        if (play && loopA !== null && v.currentTime < loopA)
+            v.currentTime = loopA;
     }
 
     function playpause() {
@@ -809,6 +842,38 @@ window.baguetteBox = (function () {
             showNextImage();
     }
 
+    function setloop(side) {
+        var v = vid();
+        if (!v)
+            return;
+
+        var t = v.currentTime;
+        if (side == 1) loopA = t;
+        if (side == 2) loopB = t;
+        if (side)
+            toast.inf(5, 'Loop' + (side == 1 ? 'A' : 'B') + ': ' + f2f(t, 2));
+
+        if (loopB !== null) {
+            timer.add(loopchk);
+            sethash(window.location.hash.slice(1).split('&')[0] + '&t=' + (loopA || 0) + '-' + loopB);
+        }
+    }
+
+    function loopchk() {
+        if (loopB === null)
+            return;
+
+        var v = vid();
+        if (!v || v.paused || v.currentTime < loopB)
+            return;
+
+        v.currentTime = loopA || 0;
+    }
+
+    function urltime(txt) {
+        url_ts = txt;
+    }
+
     function mp_ctl() {
         var v = vid();
         if (!vmute && v && mp.au && !mp.au.paused) {
@@ -822,10 +887,11 @@ window.baguetteBox = (function () {
     }
 
     function bounceAnimation(direction) {
-        slider.className = 'bounce-from-' + direction;
+        slider.className = options.animation == 'slideIn' ? 'bounce-from-' + direction : 'eog';
         setTimeout(function () {
             slider.className = '';
-        }, 400);
+        }, 300);
+        return false;
     }
 
     function updateOffset() {
@@ -851,6 +917,15 @@ window.baguetteBox = (function () {
             playvid(true);
             v.muted = vmute;
             v.loop = vloop;
+            if (url_ts) {
+                var seek = ('' + url_ts).split('-');
+                v.currentTime = seek[0];
+                if (seek.length > 1) {
+                    loopA = parseFloat(seek[0]);
+                    loopB = parseFloat(seek[1]);
+                    setloop();
+                }
+            }
         }
         selbg();
         mp_ctl();
@@ -862,6 +937,28 @@ window.baguetteBox = (function () {
         else
             timer.rm(rotn);
 
+        var ctime = 0;
+        el.onclick = v ? null : function (e) {
+            var rc = e.target.getBoundingClientRect(),
+                x = e.clientX - rc.left,
+                fx = x / (rc.right - rc.left);
+
+            if (fx < 0.3)
+                return showPreviousImage();
+
+            if (fx > 0.7)
+                return showNextImage();
+
+            clmod(ebi('bbox-btns'), 'off', 't');
+            clmod(btnPrev, 'off', 't');
+            clmod(btnNext, 'off', 't');
+
+            if (Date.now() - ctime <= 500)
+                tglfull();
+
+            ctime = Date.now();
+        };
+
         var prev = QS('.full-image.vis');
         if (prev)
             clmod(prev, 'vis');
@@ -898,8 +995,6 @@ window.baguetteBox = (function () {
     function destroyPlugin() {
         unbindEvents();
         clearCachedData();
-        unbind(document, 'keydown', keyDownHandler);
-        unbind(document, 'keyup', keyUpHandler);
         document.getElementsByTagName('body')[0].removeChild(ebi('bbox-overlay'));
         data = {};
         currentGallery = [];
@@ -912,6 +1007,7 @@ window.baguetteBox = (function () {
         showNext: showNextImage,
         showPrevious: showPreviousImage,
         relseek: relseek,
+        urltime: urltime,
         playpause: playpause,
         hide: hideOverlay,
         destroy: destroyPlugin

copyparty/web/browser.css

@@ -88,10 +88,10 @@
 
 	--g-sel-fg: #fff;
 	--g-sel-bg: #925;
-	--g-sel-b1: #c37;
+	--g-sel-b1: #e39;
 	--g-sel-sh: #b36;
 	--g-fsel-bg: #d39;
-	--g-fsel-b1: #d48;
+	--g-fsel-b1: #f4a;
 	--g-fsel-ts: #804;
 	--g-fg: var(--a-hil);
 	--g-bg: var(--bg-u2);
@@ -238,6 +238,7 @@ html.b {
 	--u2-txt-bg: transparent;
 	--u2-tab-1-sh: var(--bg);
 	--u2-b1-bg: rgba(128,128,128,0.15);
+	--u2-b2-bg: var(--u2-b1-bg);
 
 	--u2-o-bg: var(--btn-bg);
 	--u2-o-h-bg: var(--btn-h-bg);
@@ -245,6 +246,7 @@ html.b {
 	--u2-o-1h-bg: var(--a-hil);
 
 	--f-sh1: 0.1;
+	--mp-b-bg: transparent;
 }
 html.bz {
 	--fg: #cce;
@@ -258,7 +260,7 @@ html.bz {
 	--bg-d2: #34384e;
 	--bg-d3: #34384e;
 
-	--row-alt: rgba(139, 150, 205, 0.06);
+	--row-alt: #181a27;
 
 	--btn-bg: #202231;
 	--btn-h-bg: #2d2f45;
@@ -277,6 +279,7 @@ html.bz {
 
 	--f-h-b1: #34384e;
 	--mp-sh: #11121d;
+	/*--mp-b-bg: #2c3044;*/
 }
 html.by {
 	--bg: #f2f2f2;
@@ -308,7 +311,7 @@ html.c {
 	--a-gray: #0ae;
 
 	--tab-alt: #6ef;
-	--row-alt: rgba(180,0,255,0.3);
+	--row-alt: #47237d;
 	--scroll: #ff0;
 
 	--btn-fg: #fff;
@@ -320,6 +323,7 @@ html.c {
 	--u2-o-1-bg: #4cf;
 
 	--srv-1: #ea0;
+	--mp-b-bg: transparent;
 }
 html.cz {
 	--bgg: var(--bg-u2);
@@ -352,6 +356,8 @@ html.cy {
 	--srv-1: #f00;
 	--op-aa-bg: #fff;
 
+	--u2-b1-bg: #f00;
+	--u2-b2-bg: #f00;
 	--u2-o-bg: #ff0;
 	--u2-o-1-bg: #f00;
 }
@@ -371,7 +377,7 @@ html.dz {
 	--bg: #010;
 	--bgg: var(--bg);
 	--bg-d1: #000;
-	--bg-d2: #000;
+	--bg-d2: #020;
 	--bg-d3: #000;
 	--bg-max: #000;
 
@@ -381,8 +387,8 @@ html.dz {
 	--scroll: #0f0;
 
 	--a: #9f9;
-	--a-b: #fff;
-	--a-hil: #fff;
+	--a-b: #cfc;
+	--a-hil: #cfc;
 	--a-dark: #afa;
 	--a-gray: #2a2;
 
@@ -412,8 +418,8 @@ html.dz {
 	--u2-tab-1-b2: #583;
 	--u2-tab-1-sh: #280;
 	--u2-b-fg: #fff;
-	--u2-b1-bg: #c38;
-	--u2-b2-bg: #d80;
+	--u2-b1-bg: #3a3;
+	--u2-b2-bg: #3a3;
 	--u2-inf-bg: #07a;
 	--u2-inf-b1: #0be;
 	--u2-ok-bg: #380;
@@ -458,7 +464,7 @@ html.dz {
 	--f-sh1: 0.33;
 	--f-sh2: 0.02;
 	--f-sh3: 0.2;
-	--f-h-b1: rgba(128,128,128,0.7);
+	--f-h-b1: #3b3;
 
 	--f-play-bg: #fc5;
 	--f-play-fg: #000;
@@ -467,7 +473,6 @@ html.dz {
 
 	--fm-off: #f6c;
 	--mp-sh: var(--bg-d3);
-	--mp-b-bg: rgba(0,0,0,0.2);
 
 	--err-fg: #fff;
 	--err-bg: #a20;
@@ -501,7 +506,7 @@ html.dy {
 	--a: #000;
 	--a-b: #000;
 	--a-hil: #000;
-	--a-gray: #000;
+	--a-gray: #bbb;
 	--a-dark: #000;
 
 	--btn-fg: #000;
@@ -522,6 +527,13 @@ html.dy {
 	--u2-tab-1-b2: a;
 	--u2-tab-1-fg: a;
 	--u2-tab-1-bg: a;
+	--u2-b1-bg: #000;
+	--u2-b2-bg: #000;
+	--u2-o-h-bg: #999;
+	--u2-o-1h-bg: #999;
+	--u2-o-bg: #eee;
+	--u2-o-1-bg: #000;
+
 	--ud-b1: a;
 
 	--sort-1: a;
@@ -534,6 +546,9 @@ html.dy {
 
 	--tree-bg: #fff;
 
+	--g-sel-bg: #000;
+	--g-fsel-bg: #444;
+	--g-fsel-ts: #000;
 	--g-fg: a;
 	--g-bg: a;
 	--g-b1: a;
@@ -651,7 +666,7 @@ a:hover {
 	background: var(--bg-d3);
 	text-decoration: underline;
 }
-#files thead {
+#files thead th {
 	position: sticky;
 	top: -1px;
 }
@@ -697,6 +712,7 @@ html.y #files thead th {
 #files td {
 	margin: 0;
 	padding: .3em .5em;
+	background: var(--bg);
 }
 #files tr:nth-child(2n) td {
 	background: var(--row-alt);
@@ -713,6 +729,7 @@ html.y #files thead th {
 }
 #files td:first-child {
 	border-radius: .25em 0 0 .25em;
+	white-space: nowrap;
 }
 #files td:last-child {
 	border-radius: 0 .25em .25em 0;
@@ -1246,8 +1263,12 @@ html.y #ops svg circle {
 	max-width: min(41em, calc(100% - 2.6em));
 }
 .opbox input {
+	position: relative;
 	margin: .5em;
 }
+#op_cfg input[type=text] {
+	top: -.3em;
+}
 .opview input[type=text] {
 	color: var(--fg);
 	background: var(--txt-bg);
@@ -1439,7 +1460,7 @@ html {
 	margin: .2em;
 	white-space: pre;
 	position: relative;
-	top: -.2em;
+	top: -.12em;
 }
 html.c .btn,
 html.a .btn {
@@ -1566,11 +1587,27 @@ html.y #tree.nowrap .ntree a+a:hover {
 }
 #files>thead>tr>th.min,
 #files td.min {
-		display: none;
+	display: none;
 }
 #files td:nth-child(2n) {
 	color: var(--tab-alt);
 }
+#plazy {
+	width: 1px;
+	height: 1px;
+	overflow: hidden;
+	white-space: nowrap;
+}
+#blazy {
+	text-align: center;
+	font-size: 1.2em;
+	margin: 1em 0;
+}
+#blazy code,
+#blazy a {
+	font-size: 1.1em;
+	padding: 0 .2em;
+}
 .opwide,
 #op_unpost,
 #srch_form {
@@ -1584,9 +1621,6 @@ html.y #tree.nowrap .ntree a+a:hover {
 	margin: .7em 0 .7em .5em;
 	padding-left: .5em;
 }
-.opwide>div.fill {
-	display: block;
-}
 .opwide>div>div>a {
 	line-height: 2em;
 }
@@ -1841,6 +1875,7 @@ a.btn,
 .full-image img,
 .full-image video {
 	display: inline-block;
+	outline: none;
 	width: auto;
 	height: auto;
 	max-width: 100%;
@@ -1896,10 +1931,13 @@ html.y #bbox-overlay figcaption a {
 	transition: left .2s ease, transform .2s ease;
 }
 .bounce-from-right {
-	animation: bounceFromRight .4s ease-out;
+	animation: bounceFromRight .3s ease-out;
 }
 .bounce-from-left {
-	animation: bounceFromLeft .4s ease-out;
+	animation: bounceFromLeft .3s ease-out;
+}
+.eog {
+	animation: eog .2s;
 }
 @keyframes bounceFromRight {
 	0% {margin-left: 0}
@@ -1911,6 +1949,9 @@ html.y #bbox-overlay figcaption a {
 	50% {margin-left: 30px}
 	100% {margin-left: 0}
 }
+@keyframes eog {
+	0% {filter: brightness(1.5)}
+}
 #bbox-next,
 #bbox-prev {
 	top: 50%;
@@ -1921,6 +1962,15 @@ html.y #bbox-overlay figcaption a {
 .bbox-btn {
 	position: fixed;
 }
+.bbox-btn,
+#bbox-btns {
+    opacity: 1;
+	animation: opacity .2s infinite ease-in-out;
+}
+.bbox-btn.off,
+#bbox-btns.off {
+    opacity: 0;
+}
 #bbox-overlay button {
 	cursor: pointer;
 	outline: none;
@@ -1965,7 +2015,7 @@ html.y #bbox-overlay figcaption a {
 #bbox-halp td {
 	padding: .2em .5em;
 }
-#bbox-halp td:first-child {
+#bbox-halp td:first-child:not([colspan]) {
 	text-align: right;
 }
 .bbox-spinner {
@@ -2200,12 +2250,20 @@ html.y #bbox-overlay figcaption a {
 #u2notbtn * {
 	line-height: 1.3em;
 }
+#u2mu div {
+	height: 1.2em;
+	overflow: hidden;
+}
 #u2tabw {
 	min-height: 0;
 	transition: min-height .2s;
 	margin: 2em 0;
 }
+#u2tabw.na>table {
+	display: none;
+}
 #u2tab {
+	table-layout: fixed;
 	border-collapse: collapse;
 	width: calc(100% - 2em);
 	max-width: 100em;
@@ -2215,6 +2273,7 @@ html.y #bbox-overlay figcaption a {
 	max-width: none;
 }
 #u2tab td {
+	word-wrap: break-word;
 	border: 1px solid rgba(128,128,128,0.8);
 	border-width: 0 0px 1px 0;
 	padding: .2em .3em;
@@ -2229,7 +2288,19 @@ html.y #bbox-overlay figcaption a {
 #u2tab.up.ok td:nth-child(3),
 #u2tab.up.bz td:nth-child(3),
 #u2tab.up.q  td:nth-child(3) {
-	width: 19em;
+	width: 18em;
+}
+@media (max-width: 65em) {
+	#u2tab {
+		font-size: .9em;
+	}
+}
+@media (max-width: 50em) {
+	#u2tab.up.ok td:nth-child(3),
+	#u2tab.up.bz td:nth-child(3),
+	#u2tab.up.q  td:nth-child(3) {
+		width: 16em;
+	}
 }
 #op_up2k.srch td.prog {
 	font-family: sans-serif;
@@ -2329,11 +2400,14 @@ html.y #bbox-overlay figcaption a {
 	width: 48em;
 }
 #u2conf.ww {
-	width: 74em;
+	width: 78em;
 }
 #u2conf.ww #u2c3w {
 	width: 29em;
 }
+#u2conf.ww #u2c3w.s {
+	width: 39em;
+}
 #u2conf .c,
 #u2conf .c * {
 	text-align: center;
@@ -2369,6 +2443,9 @@ html.y #bbox-overlay figcaption a {
 	position: relative;
 	bottom: -0.08em;
 }
+#u2conf input+a.b {
+	background: var(--u2-b2-bg);
+}
 html.b #u2conf a.b:hover {
 	background: var(--btn-h-bg);
 }
@@ -2423,15 +2500,36 @@ html.b #u2conf a.b:hover {
 	text-align: center;
 	border: .2em dashed rgba(128, 128, 128, 0.3);
 }
-#u2foot {
+#u2foot,
+#u2life {
 	color: var(--fg-max);
-	font-style: italic;
 	text-align: center;
-	font-size: .9em;
-	margin: 1em 0;
+	margin: .8em 0;
+}
+#u2life {
+	margin: 2.5em 0;
+	line-height: 1.5em;
+}
+#u2life div {
+	display: inline-block;
+	white-space: nowrap;
+	margin: 0 2em;
+}
+#u2life div:first-child {
+	margin-bottom: .2em;
+}
+#u2life small {
+	opacity: .6;
+}
+#lifew {
+	border-bottom: 1px dotted var(--fg-max);
+}
+#u2foot {
+	font-size: 1.2em;
+	font-style: italic;
 }
 #u2foot .warn {
-	font-size: 1.3em;
+	font-size: 1.2em;
 	padding: .5em .8em;
 	margin: 1em -.6em;
 	border-width: .1em 0;
@@ -2445,6 +2543,13 @@ html.b #u2conf a.b:hover {
 	font-size: .9em;
 	font-weight: normal;
 }
+#u2foot>*+* {
+	margin-top: 1.5em;
+}
+#u2life input {
+	width: 4em;
+	text-align: right;
+}
 .prog {
 	font-family: 'scp', monospace, monospace;
 }
@@ -2598,9 +2703,6 @@ html.c #u2cards,
 html.a #u2cards {
 	margin: 0 auto -1em auto;
 }
-html.a #u2conf input+a.b {
-	background: var(--u2-b2-bg);
-}
 html.c #u2foot:empty,
 html.a #u2foot:empty {
 	margin-bottom: -1em;
@@ -2668,7 +2770,6 @@ html.b #barpos,
 html.b #barbuf,
 html.b #pvol {
 	border-radius: .2em;
-	background: none;
 }
 html.b #barpos {
 	box-shadow: 0 0 0 1px rgba(0,0,0,0.4);
@@ -2710,6 +2811,7 @@ html.b #tree li {
 html.b #tree li {
 	margin-left: .8em;
 }
+html.b #docul a,
 html.b .ntree a {
 	padding: .6em .2em;
 }
@@ -2778,6 +2880,17 @@ html.cy #files tbody div a:last-child {
 
 
 
+
+html.dz * {
+	border-radius: 0 !important;
+}
+html.d #treepar {
+	border-bottom: .2em solid var(--f-h-b1);
+}
+
+
+
+
 @media (min-width: 70em) {
 	#barpos,
 	#barbuf {
@@ -2830,3 +2943,28 @@ html.cy #files tbody div a:last-child {
 		margin-top: 1.7em;
 	}
 }
+@supports (display: grid) {
+	#ggrid {
+		display: grid;
+		margin: 0em 0.25em;
+		padding: unset;
+		grid-template-columns: repeat(auto-fit,var(--grid-sz));
+		justify-content: center;
+		gap: 1em;
+	}
+
+	html.b #ggrid {
+		padding: 0 2em 2em 0;
+		gap: .5em 3em;
+	}
+
+	#ggrid > a {
+		margin: unset;
+		padding: unset;
+  	}
+
+  	#ggrid>a>span {
+		text-align: center;
+		padding: 0.2em;
+	}
+}

copyparty/web/browser.html

@@ -3,7 +3,7 @@
 
 <head>
 	<meta charset="utf-8">
-	<title>⇆🎉 {{ title }}</title>
+	<title>{{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
 {{ html_head }}
@@ -139,15 +139,19 @@
 			dtheme = "{{ dtheme }}",
 			srvinf = "{{ srv_info }}",
 			lang = "{{ lang }}",
+			dfavico = "{{ favico }}",
 			def_hcols = {{ def_hcols|tojson }},
 			have_up2k_idx = {{ have_up2k_idx|tojson }},
 			have_tags_idx = {{ have_tags_idx|tojson }},
 			have_acode = {{ have_acode|tojson }},
 			have_mv = {{ have_mv|tojson }},
 			have_del = {{ have_del|tojson }},
-			have_unpost = {{ have_unpost|tojson }},
+			have_unpost = {{ have_unpost }},
 			have_zip = {{ have_zip|tojson }},
-			turbolvl = {{ turbolvl|tojson }},
+			lifetime = {{ lifetime }},
+			turbolvl = {{ turbolvl }},
+			u2sort = "{{ u2sort }}",
+			have_emp = {{ have_emp|tojson }},
 			txt_ext = "{{ txt_ext }}",
 			{% if no_prism %}no_prism = 1,{% endif %}
 			readme = {{ readme|tojson }},

copyparty/web/cf.html

@@ -0,0 +1,27 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+	<meta charset="utf-8">
+	<title>{{ svcname }}</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+</head>
+
+<body>
+	<div id="box" style="opacity: 0; font-family: sans-serif">
+		<h3>please press F5 to reload the page</h3>
+		<p>sorry for the inconvenience</p>
+	</div>
+
+	<script>
+		setTimeout(function() {
+			document.getElementById('box').style.opacity = 1;
+		}, 500);
+
+		parent.toast.ok(30, parent.L.cf_ok);
+		parent.qsr('#cf_frame');
+	</script>
+</body>
+
+</html>

copyparty/web/dbg-audio.js

@@ -13,8 +13,7 @@ audio_eq.apply = function () {
 
     var can = ebi('fft_can');
     if (!can) {
-        can = mknod('canvas');
-        can.setAttribute('id', 'fft_can');
+        can = mknod('canvas', 'fft_can');
         can.style.cssText = 'position:absolute;left:0;bottom:5em;width:' + w + 'px;height:' + h + 'px;z-index:9001';
         document.body.appendChild(can);
         can.width = w;

copyparty/web/md.html

@@ -1,6 +1,6 @@
 <!DOCTYPE html><html><head>
 	<meta charset="utf-8">
-	<title>📝🎉 {{ title }}</title>
+	<title>📝 {{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
 {{ html_head }}
@@ -127,10 +127,12 @@ write markdown (most html is 🙆 too)
 	
 	<script>
 
-var last_modified = {{ lastmod }};
+var last_modified = {{ lastmod }},
+	have_emp = {{ have_emp|tojson }},
+	dfavico = "{{ favico }}";
+
 var md_opt = {
 	link_md_as_html: false,
-	allow_plugins: {{ md_plug }},
 	modpoll_freq: {{ md_chk_rate }}
 };
 

copyparty/web/md.js

@@ -20,10 +20,6 @@ var dbg = function () { };
 // dbg = console.log
 
 
-// plugins
-var md_plug = {};
-
-
 // dodge browser issues
 (function () {
     var ua = navigator.userAgent;
@@ -44,7 +40,7 @@ var md_plug = {};
         link += parts[a] + (a < aa ? '/' : '');
         o = mknod('a');
         o.setAttribute('href', link);
-        o.textContent = uricom_dec(parts[a])[0] || 'top';
+        o.textContent = uricom_dec(parts[a]) || 'top';
         dom_nav.appendChild(o);
     }
 })();
@@ -160,7 +156,7 @@ function copydom(src, dst, lv) {
 }
 
 
-function md_plug_err(ex, js) {
+md_plug_err = function (ex, js) {
     qsr('#md_errbox');
     if (!ex)
         return;
@@ -177,8 +173,7 @@ function md_plug_err(ex, js) {
             o.textContent = lns[ln - 1];
         }
     }
-    var errbox = mknod('div');
-    errbox.setAttribute('id', 'md_errbox');
+    var errbox = mknod('div', 'md_errbox');
     errbox.style.cssText = 'position:absolute;top:0;left:0;padding:1em .5em;background:#2b2b2b;color:#fc5'
     errbox.textContent = msg;
     errbox.onclick = function () {
@@ -197,50 +192,12 @@ function md_plug_err(ex, js) {
 }
 
 
-function load_plug(md_text, plug_type) {
-    if (!md_opt.allow_plugins)
-        return md_text;
-
-    var find = '\n```copyparty_' + plug_type + '\n';
-    var ofs = md_text.indexOf(find);
-    if (ofs === -1)
-        return md_text;
-
-    var ofs2 = md_text.indexOf('\n```', ofs + 1);
-    if (ofs2 == -1)
-        return md_text;
-
-    var js = md_text.slice(ofs + find.length, ofs2 + 1);
-    var md = md_text.slice(0, ofs + 1) + md_text.slice(ofs2 + 4);
-
-    var old_plug = md_plug[plug_type];
-    if (!old_plug || old_plug[1] != js) {
-        js = 'const x = { ' + js + ' }; x;';
-        try {
-            var x = eval(js);
-        }
-        catch (ex) {
-            md_plug[plug_type] = null;
-            md_plug_err(ex, js);
-            return md;
-        }
-        if (x['ctor']) {
-            x['ctor']();
-            delete x['ctor'];
-        }
-        md_plug[plug_type] = [x, js];
-    }
-
-    return md;
-}
-
-
 function convert_markdown(md_text, dest_dom) {
     md_text = md_text.replace(/\r/g, '');
 
     md_plug_err(null);
-    md_text = load_plug(md_text, 'pre');
-    md_text = load_plug(md_text, 'post');
+    md_text = load_md_plug(md_text, 'pre');
+    md_text = load_md_plug(md_text, 'post');
 
     var marked_opts = {
         //headerPrefix: 'h-',
@@ -248,7 +205,7 @@ function convert_markdown(md_text, dest_dom) {
         gfm: true
     };
 
-    var ext = md_plug['pre'];
+    var ext = md_plug.pre;
     if (ext)
         Object.assign(marked_opts, ext[0]);
 
@@ -349,7 +306,7 @@ function convert_markdown(md_text, dest_dom) {
         el.innerHTML = '<a href="#' + id + '">' + el.innerHTML + '</a>';
     }
 
-    ext = md_plug['post'];
+    ext = md_plug.post;
     if (ext && ext[0].render)
         try {
             ext[0].render(md_dom);
@@ -438,7 +395,7 @@ function init_toc() {
 
     // collect vertical position of all toc items (headers in document)
     function freshen_offsets() {
-        var top = window.pageYOffset || document.documentElement.scrollTop;
+        var top = yscroll();
         for (var a = anchors.length - 1; a >= 0; a--) {
             var y = top + anchors[a].elm.getBoundingClientRect().top;
             y = Math.round(y * 10.0) / 10;
@@ -454,7 +411,7 @@ function init_toc() {
         if (anchors.length == 0)
             return;
 
-        var ptop = window.pageYOffset || document.documentElement.scrollTop;
+        var ptop = yscroll();
         var hit = anchors.length - 1;
         for (var a = 0; a < anchors.length; a++) {
             if (anchors[a].y >= ptop - 8) {  //???

copyparty/web/md2.css

@@ -36,6 +36,11 @@
 	width: 55em;
 	width: min(55em, calc(100% - 2em));
 }
+#mtw.single.editor,
+#mw.single.editor {
+	width: calc(100% - 1em);
+	left: .5em;
+}
 
 
 #mp {

copyparty/web/md2.js

@@ -16,8 +16,7 @@ var dom_sbs = ebi('sbs');
 var dom_nsbs = ebi('nsbs');
 var dom_tbox = ebi('toolsbox');
 var dom_ref = (function () {
-    var d = mknod('div');
-    d.setAttribute('id', 'mtr');
+    var d = mknod('div', 'mtr');
     dom_swrap.appendChild(d);
     d = ebi('mtr');
     // hide behind the textarea (offsetTop is not computed if display:none)
@@ -509,6 +508,20 @@ function setsel(s) {
 }
 
 
+// cut/copy current line
+function md_cut(cut) {
+    var s = linebounds();
+    if (s.car != s.cdr)
+        return;
+
+    dom_src.setSelectionRange(s.n1, s.n2 + 1, 'forward');
+    setTimeout(function () {
+        var i = cut ? s.n1 : s.car;
+        dom_src.setSelectionRange(i, i, 'forward');
+    }, 1);
+}
+
+
 // indent/dedent
 function md_indent(dedent) {
     var s = getsel(),
@@ -955,6 +968,10 @@ var set_lno = (function () {
                 md_p_jump(dn);
                 return false;
             }
+            if (ev.code == "KeyX" || ev.code == "KeyC") {
+                md_cut(ev.code == "KeyX");
+                return true; //sic
+            }
         }
         else {
             if (ev.code == "Tab" || kc == 9) {

copyparty/web/mde.html

@@ -1,6 +1,6 @@
 <!DOCTYPE html><html><head>
 	<meta charset="utf-8">
-	<title>📝🎉 {{ title }}</title>
+	<title>📝 {{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
 {{ html_head }}
@@ -25,10 +25,12 @@
 	<a href="#" id="repl">π</a>
 	<script>
 
-var last_modified = {{ lastmod }};
+var last_modified = {{ lastmod }},
+	have_emp = {{ have_emp|tojson }},
+	dfavico = "{{ favico }}";
+
 var md_opt = {
 	link_md_as_html: false,
-	allow_plugins: {{ md_plug }},
 	modpoll_freq: {{ md_chk_rate }}
 };
 

copyparty/web/mde.js

@@ -15,7 +15,7 @@ var dom_md = ebi('mt');
         if (a > 0)
             loc.push(n[a]);
 
-        var dec = uricom_dec(n[a])[0].replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
+        var dec = uricom_dec(n[a]).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
 
         nav.push('<a href="/' + loc.join('/') + '">' + dec + '</a>');
     }

copyparty/web/splash.html

@@ -36,6 +36,7 @@
 				<tr><td>hash-q</td><td>{{ hashq }}</td></tr>
 				<tr><td>tag-q</td><td>{{ tagq }}</td></tr>
 				<tr><td>mtp-q</td><td>{{ mtpq }}</td></tr>
+				<tr><td>db-act</td><td id="u">{{ dbwt }}</td></tr>
 			</table>
 		</td><td>
 			<table class="vols">
@@ -50,8 +51,8 @@
 			</table>
 		</td></tr></table>
 		<div class="btns">
-			<a id="d" href="/?stack" tt="shows the state of all active threads">dump stack</a>
-			<a id="e" href="/?reload=cfg" tt="reload config files (accounts/volumes/volflags),$Nand rescan all e2ds volumes">reload cfg</a>
+			<a id="d" href="/?stack">dump stack</a>
+			<a id="e" href="/?reload=cfg">reload cfg</a>
 		</div>
 		{%- endif %}
 
@@ -97,7 +98,9 @@
 	<a href="#" id="repl">π</a>
 	<script>
 
-var lang="{{ this.args.lang }}";
+var lang="{{ lang }}",
+	dfavico="{{ favico }}";
+
 document.documentElement.className=localStorage.theme||"{{ this.args.theme }}";
 
 </script>

copyparty/web/splash.js

@@ -23,6 +23,12 @@ var Ls = {
 		"r1": "gå hjem",
 		".s1": "kartlegg",
 		"t1": "handling",
+		"u2": "tid siden noen sist skrev til serveren$N( opplastning / navneendring / ... )$N$N17d = 17 dager$N1h23 = 1 time 23 minutter$N4m56 = 4 minuter 56 sekunder",
+	},
+	"eng": {
+		"d2": "shows the state of all active threads",
+		"e2": "reload config files (accounts/volumes/volflags),$Nand rescan all e2ds volumes",
+		"u2": "time since the last server write$N( upload / rename / ... )$N$N17d = 17 days$N1h23 = 1 hour 23 minutes$N4m56 = 4 minutes 56 seconds",
 	}
 },
 	d = Ls[sread("lang") || lang];
@@ -40,5 +46,10 @@ for (var k in (d || {})) {
 }
 
 tt.init();
-if (!ebi('c'))
-	QS('input[name="cppwd"]').focus();
+var o = QS('input[name="cppwd"]');
+if (!ebi('c') && o.offsetTop + o.offsetHeight < window.innerHeight)
+	o.focus();
+
+o = ebi('u');
+if (o && /[0-9]+$/.exec(o.innerHTML))
+	o.innerHTML = shumantime(o.innerHTML);

copyparty/web/ui.css

@@ -190,6 +190,18 @@ html.y #tth {
 	color: #000;
 	background: #fff;
 }
+#cf_frame {
+	position: fixed;
+	z-index: 573;
+	top: 3em;
+	left: 50%;
+	width: 40em;
+	height: 30em;
+	margin-left: -20.2em;
+	border-radius: .4em;
+	border: .4em solid var(--fg);
+	box-shadow: 0 2em 4em 1em var(--bg-max);
+}
 #modal {
 	position: fixed;
     overflow: auto;
@@ -224,6 +236,10 @@ html.y #tth {
 	max-height: 30em;
 	overflow: auto;
 }
+#modalc td {
+	text-align: unset;
+	padding: .2em;
+}
 @media (min-width: 40em) {
     #modalc {
         min-width: 30em;
@@ -239,6 +255,9 @@ html.y #tth {
 	padding: .3em;
 	text-align: center;
 }
+#modalc a {
+	color: #07b;
+}
 #modalb {
 	position: sticky;
 	text-align: right;
@@ -380,11 +399,13 @@ html.y textarea:focus {
 	padding-left: 2em;
 	border-left: .3em solid #ddd;
 }
-.mdo ul>li,
-.mdo ol>li {
+.mdo ul>li {
 	margin: .7em 0;
 	list-style-type: disc;
 }
+.mdo ol>li {
+	margin: .7em 0 .7em 2em;
+}
 .mdo strong {
 	color: #000;
 }

copyparty/web/util.js

@@ -6,19 +6,47 @@ if (!window['console'])
     };
 
 
-var is_touch = 'ontouchstart' in window,
-    is_https = (window.location + '').indexOf('https:') === 0,
-    IPHONE = is_touch && /iPhone|iPad|iPod/i.test(navigator.userAgent),
+var wah = '',
+    CB = '?_=' + Date.now(),
+    HALFMAX = 8192 * 8192 * 8192 * 8192,
+    HTTPS = (window.location + '').indexOf('https:') === 0,
+    TOUCH = 'ontouchstart' in window,
+    MOBILE = TOUCH,
+    CHROME = !!window.chrome,
+    FIREFOX = ('netscape' in window) && / rv:/.test(navigator.userAgent),
+    IPHONE = TOUCH && /iPhone|iPad|iPod/i.test(navigator.userAgent),
     WINDOWS = navigator.platform ? navigator.platform == 'Win32' : /Windows/.test(navigator.userAgent);
 
+try {
+    CB = '?' + document.currentScript.src.split('?').pop();
+
+    if (navigator.userAgentData.mobile)
+        MOBILE = true;
+
+    if (navigator.userAgentData.platform == 'Windows')
+        WINDOWS = true;
+
+    if (navigator.userAgentData.brands.some(function (d) { return d.brand == 'Chromium' }))
+        CHROME = true;
+}
+catch (ex) { }
+
 
 var ebi = document.getElementById.bind(document),
     QS = document.querySelector.bind(document),
     QSA = document.querySelectorAll.bind(document),
-    mknod = document.createElement.bind(document),
     XHR = XMLHttpRequest;
 
 
+function mknod(et, eid) {
+    var ret = document.createElement(et);
+    if (eid)
+        ret.id = eid;
+
+    return ret;
+}
+
+
 function qsr(sel) {
     var el = QS(sel);
     if (el)
@@ -83,17 +111,20 @@ catch (ex) {
         console.log = console.stdlog;
     console.log('console capture failed', ex);
 }
-var crashed = false, ignexd = {};
+var crashed = false, ignexd = {}, evalex_fatal = false;
 function vis_exh(msg, url, lineNo, columnNo, error) {
-    if ((msg + '').indexOf('ResizeObserver') !== -1)
+    if ((msg + '').indexOf('ResizeObserver') + 1)
         return;  // chrome issue 809574 (benign, from <video>)
 
-    if ((msg + '').indexOf('l2d.js') !== -1)
+    if ((msg + '').indexOf('l2d.js') + 1)
         return;  // `t` undefined in tapEvent -> hitTestSimpleCustom
 
     if (!/\.js($|\?)/.exec('' + url))
         return;  // chrome debugger
 
+    if ((url + '').indexOf(' > eval') + 1 && !evalex_fatal)
+        return;  // md timer
+
     var ekey = url + '\n' + lineNo + '\n' + msg;
     if (ignexd[ekey] || crashed)
         return;
@@ -104,7 +135,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
         '<h1>you hit a bug!</h1>',
         '<p style="font-size:1.3em;margin:0">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a></p>',
         '<p style="color:#fff">please send me a screenshot arigathanks gozaimuch: <a href="<ghi>" target="_blank">github issue</a> or <code>ed#2644</code></p>',
-        '<p class="b">' + esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(String(msg)) + '</p>',
+        '<p class="b">' + esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(String(msg)).replace(/\n/g, '<br />') + '</p>',
         '<p><b>UA:</b> ' + esc(navigator.userAgent + '')
     ];
 
@@ -156,8 +187,7 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
     try {
         var exbox = ebi('exbox');
         if (!exbox) {
-            exbox = mknod('div');
-            exbox.setAttribute('id', 'exbox');
+            exbox = mknod('div', 'exbox');
             document.body.appendChild(exbox);
 
             var s = mknod('style');
@@ -205,7 +235,7 @@ function ev(e) {
         return;
 
     if (e.preventDefault)
-        e.preventDefault()
+        e.preventDefault();
 
     if (e.stopPropagation)
         e.stopPropagation();
@@ -218,6 +248,11 @@ function ev(e) {
 }
 
 
+function noope(e) {
+    ev(e);
+}
+
+
 // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith
 if (!String.prototype.endsWith)
     String.prototype.endsWith = function (search, this_len) {
@@ -355,6 +390,21 @@ if (window.matchMedia) {
 }
 
 
+function yscroll() {
+    if (document.documentElement.scrollTop) {
+        return (window.yscroll = function () {
+            return document.documentElement.scrollTop;
+        })();
+    }
+    if (window.pageYOffset) {
+        return (window.yscroll = function () {
+            return window.pageYOffset;
+        })();
+    }
+    return 0;
+}
+
+
 function showsort(tab) {
     var v, vn, v1, v2, th = tab.tHead,
         sopts = jread('fsort', [["href", 1, ""]]);
@@ -443,6 +493,16 @@ function sortTable(table, col, cb) {
         }
         return reverse * (a.localeCompare(b));
     });
+    if (sread('dir1st') !== '0') {
+        var r1 = [], r2 = [];
+        for (var i = 0; i < tr.length; i++) {
+            var cell = tr[vl[i][1]].cells[1],
+                href = cell.getAttribute('sortv') || cell.textContent.trim();
+
+            (href.split('?')[0].slice(-1) == '/' ? r1 : r2).push(vl[i]);
+        }
+        vl = r1.concat(r2);
+    }
     for (i = 0; i < tr.length; ++i) tb.appendChild(tr[vl[i][1]]);
     if (cb) cb();
 }
@@ -484,7 +544,7 @@ function linksplit(rp, id) {
             link = rp.slice(0, ofs + 1);
             rp = rp.slice(ofs + 1);
         }
-        var vlink = esc(uricom_dec(link)[0]);
+        var vlink = esc(uricom_dec(link));
 
         if (link.indexOf('/') !== -1) {
             vlink = vlink.slice(0, -1) + '<span>/</span>';
@@ -542,6 +602,17 @@ function url_enc(txt) {
 
 
 function uricom_dec(txt) {
+    try {
+        return decodeURIComponent(txt);
+    }
+    catch (ex) {
+        console.log("ucd-err [" + txt + "]");
+        return txt;
+    }
+}
+
+
+function uricom_sdec(txt) {
     try {
         return [decodeURIComponent(txt), true];
     }
@@ -555,7 +626,7 @@ function uricom_dec(txt) {
 function uricom_adec(arr, li) {
     var ret = [];
     for (var a = 0; a < arr.length; a++) {
-        var txt = uricom_dec(arr[a])[0];
+        var txt = uricom_dec(arr[a]);
         ret.push(li ? '<li>' + esc(txt) + '</li>' : txt);
     }
 
@@ -577,7 +648,7 @@ function get_evpath() {
 
 
 function get_vpath() {
-    return uricom_dec(get_evpath())[0];
+    return uricom_dec(get_evpath());
 }
 
 
@@ -607,6 +678,14 @@ function s2ms(s) {
 }
 
 
+var isNum = function (v) {
+    var n = parseFloat(v);
+    return !isNaN(v - n) && n === v;
+};
+if (window.Number && Number.isFinite)
+    isNum = Number.isFinite;
+
+
 function f2f(val, nd) {
     // 10.toFixed(1) returns 10.00 for certain values of 10
     val = (val * Math.pow(10, nd)).toFixed(0).split('.')[0];
@@ -615,18 +694,19 @@ function f2f(val, nd) {
 
 
 function humansize(b, terse) {
-    var i = 0, u = terse ? ['B', 'K', 'M', 'G'] : ['B', 'KB', 'MB', 'GB'];
-    while (b >= 1000 && i < u.length) {
+    var i = 0, u = ['B', 'KB', 'MB', 'GB', 'TB', 'PB'];
+    while (b >= 1000 && i < u.length - 1) {
         b /= 1024;
         i += 1;
     }
-    return f2f(b, b >= 100 ? 0 : b >= 10 ? 1 : 2) + ' ' + u[i];
+    return (f2f(b, b >= 100 ? 0 : b >= 10 ? 1 : 2) +
+        ' ' + (terse ? u[i].charAt(0) : u[i]));
 }
 
 
 function humantime(v) {
     if (v >= 60 * 60 * 24)
-        return v;
+        return shumantime(v);
 
     try {
         return /.*(..:..:..).*/.exec(new Date(v * 1000).toUTCString())[1];
@@ -637,12 +717,55 @@ function humantime(v) {
 }
 
 
+function shumantime(v, long) {
+    if (v < 10)
+        return f2f(v, 2) + 's';
+    if (v < 60)
+        return f2f(v, 1) + 's';
+
+    v = parseInt(v);
+    var st = [[60 * 60 * 24, 60 * 60, 'd'], [60 * 60, 60, 'h'], [60, 1, 'm']];
+
+    for (var a = 0; a < st.length; a++) {
+        var m1 = st[a][0],
+            m2 = st[a][1],
+            ch = st[a][2];
+
+        if (v < m1)
+            continue;
+
+        var v1 = parseInt(v / m1),
+            v2 = ('0' + parseInt((v % m1) / m2)).slice(-2);
+
+        return v1 + ch + (v1 >= 10 || v2 == '00' ? '' : v2 + (
+            long && a < st.length - 1 ? st[a + 1][2] : ''));
+    }
+}
+
+
+function lhumantime(v) {
+    var t = shumantime(v, 1),
+        tp = t.replace(/([a-z])/g, " $1 ").split(/ /g).slice(0, -1);
+
+    if (!window.L || tp.length < 2 || tp[1].indexOf('$') + 1)
+        return t;
+
+    var ret = '';
+    for (var a = 0; a < tp.length; a += 2)
+        ret += tp[a] + ' ' + L['ht_' + tp[a + 1]].replace(tp[a] == 1 ? /!.*/ : /!/, '') + L.ht_and;
+
+    return ret.slice(0, -L.ht_and.length);
+}
+
+
 function clamp(v, a, b) {
     return Math.min(Math.max(v, a), b);
 }
 
 
 function has(haystack, needle) {
+    try { return haystack.includes(needle); } catch (ex) { }
+
     for (var a = 0; a < haystack.length; a++)
         if (haystack[a] == needle)
             return true;
@@ -705,7 +828,7 @@ function fcfg_get(name, defval) {
     var o = ebi(name),
         val = parseFloat(sread(name));
 
-    if (isNaN(val))
+    if (!isNum(val))
         return parseFloat(o ? o.value : defval);
 
     if (o)
@@ -796,14 +919,18 @@ function scfg_bind(obj, oname, cname, defval, cb) {
 
 function hist_push(url) {
     console.log("h-push " + url);
-    if (window.history && history.pushState)
+    try {
         history.pushState(url, url, url);
+    }
+    catch (ex) { }
 }
 
 function hist_replace(url) {
     console.log("h-repl " + url);
-    if (window.history && history.replaceState)
+    try {
         history.replaceState(url, url, url);
+    }
+    catch (ex) { }  // ff "The operation is insecure." on rapid switches
 }
 
 function sethash(hv) {
@@ -862,8 +989,8 @@ var timer = (function () {
 
 var tt = (function () {
     var r = {
-        "tt": mknod("div"),
-        "th": mknod("div"),
+        "tt": mknod("div", 'tt'),
+        "th": mknod("div", 'tth'),
         "en": true,
         "el": null,
         "skip": false,
@@ -871,8 +998,6 @@ var tt = (function () {
     };
 
     r.th.innerHTML = '?';
-    r.tt.setAttribute('id', 'tt');
-    r.th.setAttribute('id', 'tth');
     document.body.appendChild(r.tt);
     document.body.appendChild(r.th);
 
@@ -884,7 +1009,7 @@ var tt = (function () {
         prev = this;
     };
 
-    var tev;
+    var tev, vh;
     r.dshow = function (e) {
         clearTimeout(tev);
         if (!r.getmsg(this))
@@ -894,9 +1019,10 @@ var tt = (function () {
             return r.show.bind(this)();
 
         tev = setTimeout(r.show.bind(this), 800);
-        if (is_touch)
+        if (TOUCH)
             return;
 
+        vh = window.innerHeight;
         this.addEventListener('mousemove', r.move);
         clmod(r.th, 'act', 1);
         r.move(e);
@@ -958,7 +1084,7 @@ var tt = (function () {
     };
 
     r.hide = function (e) {
-        ev(e);
+        //ev(e);  // eats checkbox-label clicks
         clearTimeout(tev);
         window.removeEventListener('scroll', r.hide);
 
@@ -975,8 +1101,9 @@ var tt = (function () {
     };
 
     r.move = function (e) {
+        var sy = e.clientY + 128 > vh ? -1 : 1;
         r.th.style.left = (e.pageX + 12) + 'px';
-        r.th.style.top = (e.pageY + 12) + 'px';
+        r.th.style.top = (e.pageY + 12 * sy) + 'px';
     };
 
     if (IPHONE) {
@@ -1041,12 +1168,12 @@ var toast = (function () {
     var r = {},
         te = null,
         scrolling = false,
-        obj = mknod('div');
+        obj = mknod('div', 'toast');
 
-    obj.setAttribute('id', 'toast');
     document.body.appendChild(obj);
     r.visible = false;
     r.txt = null;
+    r.tag = obj;  // filler value (null is scary)
 
     function scrollchk() {
         if (scrolling)
@@ -1075,9 +1202,10 @@ var toast = (function () {
         clearTimeout(te);
         clmod(obj, 'vis');
         r.visible = false;
+        r.tag = obj;
     };
 
-    r.show = function (cl, sec, txt) {
+    r.show = function (cl, sec, txt, tag) {
         clearTimeout(te);
         if (sec)
             te = setTimeout(r.hide, sec * 1000);
@@ -1093,19 +1221,20 @@ var toast = (function () {
         timer.add(scrollchk);
         r.visible = true;
         r.txt = txt;
+        r.tag = tag;
     };
 
-    r.ok = function (sec, txt) {
-        r.show('ok', sec, txt);
+    r.ok = function (sec, txt, tag) {
+        r.show('ok', sec, txt, tag);
     };
-    r.inf = function (sec, txt) {
-        r.show('inf', sec, txt);
+    r.inf = function (sec, txt, tag) {
+        r.show('inf', sec, txt, tag);
     };
-    r.warn = function (sec, txt) {
-        r.show('warn', sec, txt);
+    r.warn = function (sec, txt, tag) {
+        r.show('warn', sec, txt, tag);
     };
-    r.err = function (sec, txt) {
-        r.show('err', sec, txt);
+    r.err = function (sec, txt, tag) {
+        r.show('err', sec, txt, tag);
     };
 
     return r;
@@ -1119,15 +1248,21 @@ var modal = (function () {
         cb_up = null,
         cb_ok = null,
         cb_ng = null,
-        prim = '<a href="#" id="modal-ok">OK</a>',
-        sec = '<a href="#" id="modal-ng">Cancel</a>',
+        tok, tng, prim, sec, ok_cancel;
+
+    r.load = function () {
+        tok = (window.L && L.m_ok) || 'OK';
+        tng = (window.L && L.m_ng) || 'Cancel';
+        prim = '<a href="#" id="modal-ok">' + tok + '</a>';
+        sec = '<a href="#" id="modal-ng">' + tng + '</a>';
         ok_cancel = WINDOWS ? prim + sec : sec + prim;
+    };
+    r.load();
 
     r.busy = false;
 
     r.show = function (html) {
-        o = mknod('div');
-        o.setAttribute('id', 'modal');
+        o = mknod('div', 'modal');
         o.innerHTML = '<table><tr><td><div id="modalc">' + html + '</div></td></tr></table>';
         document.body.appendChild(o);
         document.addEventListener('keydown', onkey);
@@ -1182,7 +1317,8 @@ var modal = (function () {
             return;
 
         setTimeout(function () {
-            ebi('modal-ok').focus();
+            if (ctr = ebi('modal-ok'))
+                ctr.focus();
         }, 20);
         ev(e);
     }
@@ -1228,17 +1364,17 @@ var modal = (function () {
         r.show(html);
     }
 
-    r.confirm = function (html, cok, cng, fun) {
+    r.confirm = function (html, cok, cng, fun, btns) {
         q.push(function () {
-            _confirm(lf2br(html), cok, cng, fun);
+            _confirm(lf2br(html), cok, cng, fun, btns);
         });
         next();
     }
-    function _confirm(html, cok, cng, fun) {
+    function _confirm(html, cok, cng, fun, btns) {
         cb_ok = cok;
         cb_ng = cng === undefined ? cok : cng;
         cb_up = fun;
-        html += '<div id="modalb">' + ok_cancel + '</div>';
+        html += '<div id="modalb">' + (btns || ok_cancel) + '</div>';
         r.show(html);
     }
 
@@ -1340,8 +1476,10 @@ function repl(e) {
         if (!cmd)
             return toast.inf(3, 'eval aborted');
 
-        if (cmd.startsWith(','))
-            return modal.alert(esc(eval(cmd.slice(1)) + ''))
+        if (cmd.startsWith(',')) {
+            evalex_fatal = true;
+            return modal.alert(esc(eval(cmd.slice(1)) + ''));
+        }
 
         try {
             modal.alert(esc(eval(cmd) + ''));
@@ -1355,6 +1493,49 @@ if (ebi('repl'))
     ebi('repl').onclick = repl;
 
 
+var md_plug = {};
+var md_plug_err = function (ex, js) {
+    if (ex)
+        console.log(ex, js);
+};
+function load_md_plug(md_text, plug_type) {
+    if (!have_emp)
+        return md_text;
+
+    var find = '\n```copyparty_' + plug_type + '\n';
+    var ofs = md_text.indexOf(find);
+    if (ofs === -1)
+        return md_text;
+
+    var ofs2 = md_text.indexOf('\n```', ofs + 1);
+    if (ofs2 == -1)
+        return md_text;
+
+    var js = md_text.slice(ofs + find.length, ofs2 + 1);
+    var md = md_text.slice(0, ofs + 1) + md_text.slice(ofs2 + 4);
+
+    var old_plug = md_plug[plug_type];
+    if (!old_plug || old_plug[1] != js) {
+        js = 'const x = { ' + js + ' }; x;';
+        try {
+            var x = eval(js);
+            if (x['ctor']) {
+                x['ctor']();
+                delete x['ctor'];
+            }
+        }
+        catch (ex) {
+            md_plug[plug_type] = null;
+            md_plug_err(ex, js);
+            return md;
+        }
+        md_plug[plug_type] = [x, js];
+    }
+
+    return md;
+}
+
+
 var svg_decl = '<?xml version="1.0" encoding="UTF-8"?>\n';
 
 
@@ -1380,12 +1561,24 @@ var favico = (function () {
         var b64;
         try {
             b64 = btoa(svg ? svg_decl + svg : gx(r.txt));
+            //console.log('f1');
         }
-        catch (ex) {
-            b64 = encodeURIComponent(r.txt).replace(/%([0-9A-F]{2})/g,
-                function x(m, v) { return String.fromCharCode('0x' + v); });
-
-            b64 = btoa(gx(unescape(encodeURIComponent(r.txt))));
+        catch (e1) {
+            try {
+                b64 = btoa(gx(encodeURIComponent(r.txt).replace(/%([0-9A-F]{2})/g,
+                    function x(m, v) { return String.fromCharCode('0x' + v); })));
+                //console.log('f2');
+            }
+            catch (e2) {
+                try {
+                    b64 = btoa(gx(unescape(encodeURIComponent(r.txt))));
+                    //console.log('f3');
+                }
+                catch (e3) {
+                    //console.log('fe');
+                    return;
+                }
+            }
         }
 
         if (!r.tag) {
@@ -1398,9 +1591,13 @@ var favico = (function () {
 
     r.init = function () {
         clearTimeout(r.to);
-        scfg_bind(r, 'txt', 'icot', '', r.upd);
-        scfg_bind(r, 'fg', 'icof', 'fc5', r.upd);
-        scfg_bind(r, 'bg', 'icob', '222', r.upd);
+        var dv = (window.dfavico || '').trim().split(/ +/),
+            fg = dv.length < 2 ? 'fc5' : dv[1].toLowerCase() == 'none' ? '' : dv[1],
+            bg = dv.length < 3 ? '222' : dv[2].toLowerCase() == 'none' ? '' : dv[2];
+
+        scfg_bind(r, 'txt', 'icot', dv[0], r.upd);
+        scfg_bind(r, 'fg', 'icof', fg, r.upd);
+        scfg_bind(r, 'bg', 'icob', bg, r.upd);
         r.upd();
     };
 
@@ -1409,16 +1606,34 @@ var favico = (function () {
 })();
 
 
-function xhrchk(xhr, prefix, e404) {
+var cf_cha_t = 0;
+function xhrchk(xhr, prefix, e404, lvl, tag) {
     if (xhr.status < 400 && xhr.status >= 200)
         return true;
 
     if (xhr.status == 403)
-        return toast.err(0, prefix + (window.L && L.xhr403 || "403: access denied\n\ntry pressing F5, maybe you got logged out"));
+        return toast.err(0, prefix + (window.L && L.xhr403 || "403: access denied\n\ntry pressing F5, maybe you got logged out"), tag);
 
     if (xhr.status == 404)
-        return toast.err(0, prefix + e404);
+        return toast.err(0, prefix + e404, tag);
 
-    return toast.err(0, prefix + xhr.status + ": " + (
-        (xhr.response && xhr.response.err) || xhr.responseText));
+    var errtxt = (xhr.response && xhr.response.err) || xhr.responseText,
+        fun = toast[lvl || 'err'];
+
+    if (xhr.status == 503 && /[Cc]loud[f]lare|>Just a mo[m]ent|#cf-b[u]bbles|Chec[k]ing your br[o]wser/.test(errtxt)) {
+        var now = Date.now(), td = now - cf_cha_t;
+        if (td < 15000)
+            return;
+
+        cf_cha_t = now;
+        errtxt = 'Clou' + wah + 'dflare protection kicked in\n\n<strong>trying to fix it...</strong>';
+        fun = toast.warn;
+
+        qsr('#cf_frame');
+        var fr = mknod('iframe', 'cf_frame');
+        fr.src = '/?cf_challenge';
+        document.body.appendChild(fr);
+    }
+
+    return fun(0, prefix + xhr.status + ": " + errtxt, tag);
 }

copyparty/web/w.hash.js

@@ -0,0 +1,106 @@
+"use strict";
+
+
+function hex2u8(txt) {
+    return new Uint8Array(txt.match(/.{2}/g).map(function (b) { return parseInt(b, 16); }));
+}
+
+
+var subtle = null;
+try {
+    subtle = crypto.subtle;
+    subtle.digest('SHA-512', new Uint8Array(1)).then(
+        function (x) { },
+        function (x) { load_fb(); }
+    );
+}
+catch (ex) {
+    load_fb();
+}
+function load_fb() {
+    subtle = null;
+    importScripts('/.cpr/deps/sha512.hw.js');
+}
+
+
+var reader = null,
+    gc1, gc2, gc3,
+    busy = false;
+
+
+onmessage = (d) => {
+    if (busy)
+        return postMessage(["panic", 'worker got another task while busy']);
+
+    if (!reader)
+        reader = new FileReader();
+
+    var [nchunk, fobj, car, cdr] = d.data,
+        t0 = Date.now();
+
+    reader.onload = function (e) {
+        try {
+            // chrome gc forgets the filereader output; remind it
+            // (for some chromes, also necessary for subtle)
+            gc1 = e.target.result;
+            gc2 = new Uint8Array(gc1, 0, 1);
+            gc3 = new Uint8Array(gc1, gc1.byteLength - 1);
+
+            //console.log('[ w] %d HASH bgin', nchunk);
+            postMessage(["read", nchunk, cdr - car, Date.now() - t0]);
+            hash_calc(gc1);
+        }
+        catch (ex) {
+            busy = false;
+            postMessage(["panic", ex + '']);
+        }
+    };
+    reader.onerror = function () {
+        busy = false;
+        var err = reader.error + '';
+
+        if (err.indexOf('NotReadableError') !== -1 || // win10-chrome defender
+            err.indexOf('NotFoundError') !== -1  // macos-firefox permissions
+        )
+            return postMessage(["fail", 'OS-error', err + ' @ ' + car]);
+
+        postMessage(["ferr", err]);
+    };
+    //console.log('[ w] %d read bgin', nchunk);
+    busy = true;
+    reader.readAsArrayBuffer(fobj.slice(car, cdr));
+
+
+    var hash_calc = function (buf) {
+        var hash_done = function (hashbuf) {
+            // stop gc from attempting to free early
+            if (!gc1 || !gc2 || !gc3)
+                return console.log('torch went out');
+
+            gc1 = gc2 = gc3 = null;
+            busy = false;
+            try {
+                var hslice = new Uint8Array(hashbuf).subarray(0, 33);
+                //console.log('[ w] %d HASH DONE', nchunk);
+                postMessage(["done", nchunk, hslice, cdr - car]);
+            }
+            catch (ex) {
+                postMessage(["panic", ex + '']);
+            }
+        };
+
+        // stop gc from attempting to free early
+        if (!gc1 || !gc2 || !gc3)
+            console.log('torch went out');
+
+        if (subtle)
+            subtle.digest('SHA-512', buf).then(hash_done);
+        else {
+            // note: lifting u8buf counterproductive for the chrome gc bug
+            var u8buf = new Uint8Array(buf);
+            hashwasm.sha512(u8buf).then(function (v) {
+                hash_done(hex2u8(v))
+            });
+        }
+    };
+}

docs/README.md

@@ -13,6 +13,9 @@
 
 # other stuff
 
+## [`changelog.md`](changelog.md)
+* occasionally grabbed from github release notes
+
 ## [`rclone.md`](rclone.md)
 * notes on using rclone as a fuse client/server
 

docs/lics.txt

@@ -0,0 +1,63 @@
+--- server-side --- software ---
+
+https://github.com/9001/copyparty/
+C: 2019 ed
+L: MIT
+
+https://github.com/pallets/jinja/
+C: 2007 Pallets
+L: BSD 3-Clause 
+
+https://github.com/pallets/markupsafe/
+C: 2010 Pallets
+L: BSD 3-Clause 
+
+https://github.com/giampaolo/pyftpdlib/
+C: 2007 Giampaolo Rodola'
+L: MIT
+
+https://github.com/python/cpython/blob/3.10/Lib/asyncore.py
+C: 1996 Sam Rushing
+L: ISC
+
+https://github.com/ahupp/python-magic/
+C: 2001-2014 Adam Hupp
+L: MIT
+
+--- client-side --- software ---
+
+https://github.com/Daninet/hash-wasm/
+C: 2020 Dani Biró
+L: MIT
+
+https://github.com/openpgpjs/asmcrypto.js/
+C: 2013 Artem S Vybornov
+L: MIT
+
+https://github.com/feimosi/baguetteBox.js/
+C: 2017 Marek Grzybek
+L: MIT
+
+https://github.com/markedjs/marked/
+C: 2018+, MarkedJS
+C: 2011-2018, Christopher Jeffrey (https://github.com/chjj/)
+L: MIT
+
+https://github.com/codemirror/codemirror5/
+C: 2017 Marijn Haverbeke <marijnh@gmail.com> and others
+L: MIT
+
+https://github.com/Ionaru/easy-markdown-editor/
+C: 2015 Sparksuite, Inc.
+C: 2017 Jeroen Akkerman.
+L: MIT
+
+--- client-side --- fonts ---
+
+https://github.com/adobe-fonts/source-code-pro/
+C: 2010-2019 Adobe
+L: SIL OFL 1.1
+
+https://github.com/FortAwesome/Font-Awesome/
+C: 2022 Fonticons, Inc.
+L: SIL OFL 1.1

docs/notes.md

@@ -0,0 +1,10 @@
+# up2k.js
+
+## potato detection
+
+* tsk 0.25/8.4/31.5 bzw 1.27/22.9/18 = 77% (38.4s, 49.7s)
+  * 4c locale #1313, ff-102,deb-11 @ ryzen4500u wifi -> win10
+  * profiling shows 2sec heavy gc every 2sec
+
+* tsk 0.41/4.1/10 bzw 1.41/9.9/7 = 73% (13.3s, 18.2s)
+  * 4c locale #1313, ch-103,deb-11 @ ryzen4500u wifi -> win10

docs/notes.sh

@@ -48,7 +48,15 @@ avg() { awk 'function pr(ncsz) {if (nsmp>0) {printf "%3s %s\n", csz, sum/nsmp} c
 ## time between first and last upload
 
 python3 -um copyparty -nw -v srv::rw -i 127.0.0.1 2>&1 | tee log 
-cat log | awk '!/"purl"/{next} {s=$1;sub(/[^m]+m/,"");gsub(/:/," ");t=60*(60*$1+$2)+$3} !a{a=t;sa=s} {b=t;sb=s} END {print b-a,sa,sb}'
+cat log | awk '!/"purl"/{next} {s=$1;sub(/[^m]+m/,"");gsub(/:/," ");t=60*(60*$1+$2)+$3} t<p{t+=86400} !a{a=t;sa=s} {b=t;sb=s} END {print b-a,sa,sb}'
+
+# or if the client youre measuring dies for ~15sec every once ina while and you wanna filter those out,
+cat log | awk '!/"purl"/{next} {s=$1;sub(/[^m]+m/,"");gsub(/:/," ");t=60*(60*$1+$2)+$3} t<p{t+=86400} !p{a=t;p=t;r=0;next} t-p>1{printf "%.3f += %.3f - %.3f  (%.3f)  # %.3f -> %.3f\n",r,p,a,p-a,p,t;r+=p-a;a=t} {p=t} END {print r+p-a}'
+
+
+##
+## find uploads blocked by slow i/o or maybe deadlocks
+awk '/^.\+. opened logfile/{print;next} {sub(/.$/,"")} !/^..36m[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3} /{next} !/0m(POST|writing) /{next} {c=0;p=$3} /0mPOST/{c=1} {s=$1;sub(/[^m]+m/,"");gsub(/:/," ");s=60*(60*$1+$2)+$3} c{t[p]=s;next} {d=s-t[p]} d>10{print $0 "  # " d}'
 
 
 ##
@@ -135,6 +143,31 @@ sqlite3 -readonly up2k.db.key-full 'select w, v from mt where k = "key" order by
 sqlite3 -readonly up2k.db.key-full 'select w, v from mt where k = "key" order by w' > k1; sqlite3 -readonly up2k.db 'select mt.w, mt.v, up.rd, up.fn from mt inner join up on mt.w = substr(up.w,1,16) where mt.k = "key" order by up.rd, up.fn' > k2; ok=0; ng=0; while IFS='|' read w k2 path; do k1="$(grep -E "^$w" k1 | sed -r 's/.*\|//')"; [ "$k1" = "$k2" ] && ok=$((ok+1)) || { ng=$((ng+1)); printf '%3s %3s  %s\n' "$k1" "$k2" "$path"; }; done < <(cat k2); echo "match $ok   diff $ng"
 
 
+##
+## tracking bitflips
+
+l=log.tmux-1662316902  # your logfile (tmux-capture or decompressed -lo)
+
+# grab handshakes to a smaller logfile
+tr -d '\r' <$l | awk '/^.\[36m....-..-...\[0m.?$/{d=substr($0,6,10)} !d{next} /"purl": "/{t=substr($1,6);sub(/[^ ]+ /,"");sub(/ .\[34m[0-9]+ /," ");printf("%s %s %s %s\n",d,t,ip,$0)}' | while read d t ip f; do u=$(date +%s --date="${d}T${t}Z"); printf '%s\n' "$u $ip $f"; done > handshakes
+
+# quick list of affected files
+grep 'your chunk got corrupted somehow' -A1 $l | tr -d '\r' | grep -E '^[a-zA-Z0-9_-]{44}$' | sort | uniq | while IFS= read -r x; do grep -F "$x" handshakes | head -c 200; echo; done | sed -r 's/.*"name": "//' | sort | uniq -cw20
+
+# find all cases of corrupt chunks and print their respective handshakes (if any),
+# timestamps are when the corrupted chunk was received (and also the order they are displayed),
+# first checksum is the expected value from the handshake, second is what got uploaded
+awk <$l '/^.\[36m....-..-...\[0m.?$/{d=substr($0,6,10)} /your chunk got corrupted somehow/{n=2;t=substr($1,6);next} !n{next} {n--;sub(/\r$/,"")} n{a=$0;next} {sub(/.\[0m,.*/,"");printf "%s %s %s %s\n",d,t,a,$0}' |
+while read d t h1 h2; do printf '%s %s\n' $d $t; (
+printf '  %s [%s]\n' $h1 "$(grep -F $h1 <handshakes | head -n 1)"
+printf '  %s [%s]\n' $h2 "$(grep -F $h2 <handshakes | head -n 1)"
+) | sed 's/, "sprs":.*//'; done | less -R
+
+# notes; TODO clean up and put in the readme maybe --
+# quickest way to drop the bad files (if a client generated bad hashes for the initial handshake) is shutting down copyparty and moving aside the unfinished file (both the .PARTIAL and the empty placeholder)
+# BUT the clients will immediately re-handshake the upload with the same bitflipped hashes, so the uploaders have to refresh their browsers before you do that,
+# so maybe just ask them to refresh and do nothing for 6 hours so the timeout kicks in, which deletes the placeholders/name-reservations and you can then manually delete the .PARTIALs at some point later
+
 ##
 ## media
 
@@ -182,7 +215,7 @@ brew install python@2
 pip install virtualenv
 
 # readme toc
-cat README.md | awk 'function pr() { if (!h) {return}; if (/^ *[*!#|]/||!s) {printf "%s\n",h;h=0;return}; if (/.../) {printf "%s - %s\n",h,$0;h=0}; };    /^#/{s=1;pr()} /^#* *(file indexing|install on android|dev env setup|just the sfx|complete release|optional gpl stuff)|`$/{s=0} /^#/{lv=length($1);sub(/[^ ]+ /,"");bab=$0;gsub(/ /,"-",bab); h=sprintf("%" ((lv-1)*4+1) "s [%s](#%s)", "*",$0,bab);next} !h{next} {sub(/  .*/,"");sub(/[:,]$/,"")} {pr()}' > toc; grep -E '^## readme toc' -B1000 -A2 <README.md >p1; grep -E '^## quickstart' -B2 -A999999 <README.md >p2; (cat p1; grep quickstart -A1000 <toc; cat p2) >README.md; rm p1 p2 toc
+cat README.md | awk 'function pr() { if (!h) {return}; if (/^ *[*!#|]/||!s) {printf "%s\n",h;h=0;return}; if (/.../) {printf "%s - %s\n",h,$0;h=0}; };    /^#/{s=1;pr()} /^#* *(install on android|dev env setup|just the sfx|complete release|optional gpl stuff)|`$/{s=0} /^#/{lv=length($1);sub(/[^ ]+ /,"");bab=$0;gsub(/ /,"-",bab); h=sprintf("%" ((lv-1)*4+1) "s [%s](#%s)", "*",$0,bab);next} !h{next} {sub(/  .*/,"");sub(/[:;,]$/,"")} {pr()}' > toc; grep -E '^## readme toc' -B1000 -A2 <README.md >p1; grep -E '^## quickstart' -B2 -A999999 <README.md >p2; (cat p1; grep quickstart -A1000 <toc; cat p2) >README.md; rm p1 p2 toc
 
 # fix firefox phantom breakpoints,
 # suggestions from bugtracker, doesnt work (debugger is not attachable)
@@ -200,6 +233,9 @@ git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --
 # download all sfx versions
 curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | .tag_name + " " + .name' | tr -d '\r' | while read v t; do fn="$(printf '%s\n' "copyparty $v $t.py" | tr / -)"; [ -e "$fn" ] || curl https://github.com/9001/copyparty/releases/download/$v/copyparty-sfx.py -Lo "$fn"; done
 
+# convert releasenotes to changelog
+curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | "▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  \n# \(.created_at)  `\(.tag_name)`  \(.name)\n\n\(.body)\n\n\n"' | sed -r 's/^# ([0-9]{4}-)([0-9]{2})-([0-9]{2})T([0-9]{2}):([0-9]{2}):([0-9]{2})Z /# \1\2\3-\4\5 /' > changelog.md
+
 # push to multiple git remotes
 git config -l | grep '^remote'
 git remote add all git@github.com:9001/copyparty.git

docs/pyoxidizer.txt

@@ -0,0 +1,52 @@
+pyoxidizer doesn't crosscompile yet so need to build in a windows vm,
+luckily possible to do mostly airgapped (https-proxy for crates)
+
+none of this is version-specific but doing absolute links just in case
+(only exception is py3.8 which is the final win7 ver)
+
+# deps (download on linux host):
+https://www.python.org/ftp/python/3.10.7/python-3.10.7-amd64.exe
+https://github.com/indygreg/PyOxidizer/releases/download/pyoxidizer%2F0.22.0/pyoxidizer-0.22.0-x86_64-pc-windows-msvc.zip
+https://github.com/upx/upx/releases/download/v3.96/upx-3.96-win64.zip
+https://static.rust-lang.org/dist/rust-1.61.0-x86_64-pc-windows-msvc.msi
+https://github.com/indygreg/python-build-standalone/releases/download/20220528/cpython-3.8.13%2B20220528-i686-pc-windows-msvc-static-noopt-full.tar.zst
+
+# need cl.exe, prefer 2017 -- download on linux host:
+https://visualstudio.microsoft.com/downloads/?q=build+tools
+https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-history#release-dates-and-build-numbers
+https://aka.ms/vs/15/release/vs_buildtools.exe  # 2017
+https://aka.ms/vs/16/release/vs_buildtools.exe  # 2019
+https://aka.ms/vs/17/release/vs_buildtools.exe  # 2022
+https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2017
+
+# use disposable w10 vm to prep offline installer; xfer to linux host with firefox to copyparty
+vs_buildtools-2017.exe --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK.17763 --layout c:\msbt2017 --lang en-us
+
+# need two proxies on host; s5s or ssh for msys2(socks5), and tinyproxy for rust(http)
+UP=- python3 socks5server.py 192.168.123.1 4321
+ssh -vND 192.168.123.1:4321 localhost
+git clone https://github.com/tinyproxy/tinyproxy.git
+  ./autogen.sh
+  ./configure --prefix=/home/ed/pe/tinyproxy
+  make -j24 install
+  printf '%s\n' >cfg "Port 4380" "Listen 192.168.123.1"
+  ./tinyproxy -dccfg
+
+https://github.com/msys2/msys2-installer/releases/download/2022-09-04/msys2-x86_64-20220904.exe
+export all_proxy=socks5h://192.168.123.1:4321
+# if chat dies after auth (2 messages) it probably failed dns, note the h in socks5h to tunnel dns
+pacman -Syuu
+pacman -S git patch mingw64/mingw-w64-x86_64-zopfli
+cd /c && curl -k https://192.168.123.1:3923/ro/ox/msbt2017/?tar | tar -xv
+
+first install certs from msbt/certificates then admin-cmd `vs_buildtools.exe --noweb`,
+default selection (vc++2017-v15.9-v14.16, vc++redist, vc++bt-core) += win10sdk (for io.h)
+
+install rust without documentation, python 3.10, put upx and pyoxidizer into ~/bin,
+[cmd.exe] python -m pip install --user -U wheel-0.37.1.tar.gz strip-hints-0.1.10.tar.gz
+p=192.168.123.1:4380; export https_proxy=$p; export http_proxy=$p
+
+# and with all of the one-time-setup out of the way,
+mkdir /c/d; cd /c/d && curl -k https://192.168.123.1:3923/cpp/gb?pw=wark > gb && git clone gb copyparty
+cd /c/d/copyparty/ && curl -k https://192.168.123.1:3923/cpp/patch?pw=wark | patch -p1
+cd /c/d/copyparty/scripts && CARGO_HTTP_CHECK_REVOKE=false PATH=/c/Users/$USER/AppData/Local/Programs/Python/Python310:/c/Users/$USER/bin:"$(cygpath "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\Hostx86\x86"):$PATH"  ./make-sfx.sh ox ultra

pyoxidizer.bzl

@@ -0,0 +1,48 @@
+# builds win7-i386 exe on win10-ltsc-1809(17763.316)
+# see docs/pyoxidizer.txt
+
+def make_exe():
+    dist = default_python_distribution(flavor="standalone_static", python_version="3.8")
+    policy = dist.make_python_packaging_policy()
+    policy.allow_files = True
+    policy.allow_in_memory_shared_library_loading = True
+    #policy.bytecode_optimize_level_zero = True
+    #policy.include_distribution_sources = False  # error instantiating embedded Python interpreter: during initializing Python main: init_fs_encoding: failed to get the Python codec of the filesystem encoding
+    policy.include_distribution_resources = False
+    policy.include_non_distribution_sources = False
+    policy.include_test = False
+    python_config = dist.make_python_interpreter_config()
+    #python_config.module_search_paths = ["$ORIGIN/lib"]
+
+    python_config.run_module = "copyparty"
+    exe = dist.to_python_executable(
+        name="copyparty",
+        config=python_config,
+        packaging_policy=policy,
+    )
+    exe.windows_runtime_dlls_mode = "never"
+    exe.windows_subsystem = "console"
+    exe.add_python_resources(exe.read_package_root(
+        path="sfx",
+        packages=[
+            "copyparty",
+            "jinja2",
+            "markupsafe",
+            "pyftpdlib",
+            "python-magic",
+        ]
+    ))
+    return exe
+
+def make_embedded_resources(exe):
+    return exe.to_embedded_resources()
+
+def make_install(exe):
+    files = FileManifest()
+    files.add_python_resource("copyparty", exe)
+    return files
+
+register_target("exe", make_exe)
+register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
+register_target("install", make_install, depends=["exe"], default=True)
+resolve_targets()

scripts/deps-docker/Dockerfile

@@ -1,10 +1,10 @@
-FROM    alpine:3.16
+FROM    alpine:3
 WORKDIR /z
 ENV     ver_asmcrypto=5b994303a9d3e27e0915f72a10b6c2c51535a4dc \
         ver_hashwasm=4.9.0 \
-        ver_marked=4.0.17 \
-        ver_mde=2.16.1 \
-        ver_codemirror=5.65.5 \
+        ver_marked=4.0.18 \
+        ver_mde=2.18.0 \
+        ver_codemirror=5.65.9 \
         ver_fontawesome=5.13.0 \
         ver_zopfli=1.0.3
 
@@ -17,7 +17,7 @@ RUN     mkdir -p /z/dist/no-pk \
         && wget https://github.com/openpgpjs/asmcrypto.js/archive/$ver_asmcrypto.tar.gz -O asmcrypto.tgz \
         && wget https://github.com/markedjs/marked/archive/v$ver_marked.tar.gz -O marked.tgz \
         && wget https://github.com/Ionaru/easy-markdown-editor/archive/$ver_mde.tar.gz -O mde.tgz \
-        && wget https://github.com/codemirror/CodeMirror/archive/$ver_codemirror.tar.gz -O codemirror.tgz \
+        && wget https://github.com/codemirror/codemirror5/archive/$ver_codemirror.tar.gz -O codemirror.tgz \
         && wget https://github.com/FortAwesome/Font-Awesome/releases/download/$ver_fontawesome/fontawesome-free-$ver_fontawesome-web.zip -O fontawesome.zip \
         && wget https://github.com/google/zopfli/archive/zopfli-$ver_zopfli.tar.gz -O zopfli.tgz \
         && wget https://github.com/Daninet/hash-wasm/releases/download/v$ver_hashwasm/hash-wasm@$ver_hashwasm.zip -O hash-wasm.zip \

scripts/make-pypi-release.sh

@@ -14,10 +14,6 @@ gtar=$(command -v gtar || command -v gnutar) || true
 		realpath() { grealpath "$@"; }
 }
 
-which md5sum 2>/dev/null >/dev/null &&
-	md5sum=md5sum ||
-	md5sum="md5 -r"
-
 mode="$1"
 
 [ -z "$mode" ] &&

scripts/make-sfx.sh

@@ -12,6 +12,8 @@ help() { exec cat <<'EOF'
 # `re` does a repack of an sfx which you already executed once
 #   (grabs files from the sfx-created tempdir), overrides `clean`
 #
+# `ox` builds a pyoxidizer exe instead of py
+#
 # `gz` creates a gzip-compressed python sfx instead of bzip2
 #
 # `lang` limits which languages/translations to include,
@@ -26,6 +28,11 @@ help() { exec cat <<'EOF'
 #   (browsers will try to use 'Consolas' instead)
 #
 # `no-dd` saves ~2k by removing the mouse cursor
+#
+# ---------------------------------------------------------------------
+#
+# if you are on windows, you can use msys2:
+#   PATH=/c/Users/$USER/AppData/Local/Programs/Python/Python310:"$PATH" ./make-sfx.sh fast
 
 EOF
 }
@@ -51,6 +58,10 @@ gtar=$(command -v gtar || command -v gnutar) || true
 gawk=$(command -v gawk || command -v gnuawk || command -v awk)
 awk() { $gawk "$@"; }
 
+targs=(--owner=1000 --group=1000)
+[ "$OSTYPE" = msys ] &&
+	targs=()
+
 pybin=$(command -v python3 || command -v python) || {
 	echo need python
 	exit 1
@@ -64,6 +75,9 @@ pybin=$(command -v python3 || command -v python) || {
 	exit 1
 }
 
+[ $CSN ] ||
+	CSN=sfx
+
 langs=
 use_gz=
 zopf=2560
@@ -71,12 +85,14 @@ while [ ! -z "$1" ]; do
 	case $1 in
 		clean)  clean=1  ; ;;
 		re)     repack=1 ; ;;
+		ox)     use_ox=1 ; ;;
 		gz)     use_gz=1 ; ;;
 		no-fnt) no_fnt=1 ; ;;
 		no-hl)  no_hl=1  ; ;;
 		no-dd)  no_dd=1  ; ;;
 		no-cm)  no_cm=1  ; ;;
 		fast)   zopf=    ; ;;
+		ultra)  ultra=1  ; ;;
 		lang)   shift;langs="$1"; ;;
 		*)      help     ; ;;
 	esac
@@ -94,9 +110,9 @@ stamp=$(
 	done | sort | tail -n 1 | sha1sum | cut -c-16
 )
 
-rm -rf sfx/*
-mkdir -p sfx build
-cd sfx
+rm -rf $CSN/*
+mkdir -p $CSN build
+cd $CSN
 
 tmpdir="$(
 	printf '%s\n' "$TMPDIR" /tmp |
@@ -104,9 +120,9 @@ tmpdir="$(
 )"
 
 [ $repack ] && {
-	old="$tmpdir/pe-copyparty"
+	old="$tmpdir/pe-copyparty.$(id -u)"
 	echo "repack of files in $old"
-	cp -pR "$old/"*{j2,ftp,copyparty} .
+	cp -pR "$old/"*{py2,j2,ftp,copyparty} .
 }
 
 [ $repack ] || {
@@ -154,8 +170,25 @@ tmpdir="$(
 			wget -O$f "$url" || curl -L "$url" >$f)
 	done
 
-    # enable this to dynamically remove type hints at startup,
-    # in case a future python version can use them for performance
+	echo collecting python-magic
+	v=0.4.27
+	f="../build/python-magic-$v.tar.gz"
+	[ -e "$f" ] ||
+		(url=https://files.pythonhosted.org/packages/da/db/0b3e28ac047452d079d375ec6798bf76a036a08182dbb39ed38116a49130/python-magic-0.4.27.tar.gz;
+		wget -O$f "$url" || curl -L "$url" >$f)
+
+	tar -zxf $f
+	mkdir magic
+	mv python-magic-*/magic .
+	rm -rf python-magic-*
+	rm magic/compat.py
+	f=magic/__init__.py
+	awk '/^def _add_compat/{o=1} !o; /^_add_compat/{o=0}' <$f >t
+	tmv "$f"
+	mv magic ftp/  # doesn't provide a version label anyways
+
+	# enable this to dynamically remove type hints at startup,
+	# in case a future python version can use them for performance
 	true || (
 		echo collecting strip-hints
 		f=../build/strip-hints-0.1.10.tar.gz
@@ -190,9 +223,49 @@ tmpdir="$(
 	done
 
 	# remove type hints before build instead
-	(cd copyparty; python3 ../../scripts/strip_hints/a.py; rm uh)
+	(cd copyparty; "$pybin" ../../scripts/strip_hints/a.py; rm uh)
 }
 
+f=../build/mit.txt
+[ -e $f ] ||
+	curl https://opensource.org/licenses/MIT |
+	awk '/div>/{o=0}o>1;o{o++}/;COPYRIGHT HOLDER/{o=1}' |
+	awk '{gsub(/<[^>]+>/,"")};1' >$f
+
+f=../build/isc.txt
+[ -e $f ] ||
+	curl https://opensource.org/licenses/ISC |
+	awk '/div>/{o=0}o>2;o{o++}/;OWNER/{o=1}' |
+	awk '{gsub(/<[^>]+>/,"")};/./{b=0}!/./{b++}b>1{next}1' >$f
+
+f=../build/3bsd.txt
+[ -e $f ] ||
+	curl https://opensource.org/licenses/BSD-3-Clause |
+	awk '/div>/{o=0}o>1;o{o++}/HOLDER/{o=1}' |
+	awk '{gsub(/<[^>]+>/,"")};1' >$f
+
+f=../build/ofl.txt
+[ -e $f ] ||
+	curl https://opensource.org/licenses/OFL-1.1 |
+	awk '/PREAMBLE/{o=1}/sil\.org/{o=0}!o{next}/./{printf "%s ",$0;next}{print"\n"}' |
+	awk '{gsub(/<[^>]+>/,"");gsub(/^\s+/,"");gsub(/&amp;/,"\\&")}/./{b=0}!/./{b++}b>1{next}1' >$f
+
+(sed -r 's/^L: /License: /;s/^C: /Copyright (c) /' <../docs/lics.txt
+printf '\n\n--- MIT License ---\n\n'; cat ../build/mit.txt
+printf '\n\n--- ISC License ---\n\n'; cat ../build/isc.txt
+printf '\n\n--- BSD 3-Clause License ---\n\n'; cat ../build/3bsd.txt
+printf '\n\n--- SIL Open Font License v1.1 ---\n\n'; cat ../build/ofl.txt
+) |
+while IFS= read -r x; do
+	[ "${x:0:4}" = "--- " ] || {
+		printf '%s\n' "$x"
+		continue
+	}
+	n=${#x}
+	p=$(( (80-n)/2 ))
+	printf "%${p}s\033[07m%s\033[0m\n" "" "$x"
+done > copyparty/res/COPYING.txt
+
 ver=
 [ -z "$repack" ] &&
 git describe --tags >/dev/null 2>/dev/null && {
@@ -232,7 +305,7 @@ ts=$(date -u +%s)
 hts=$(date -u +%Y-%m%d-%H%M%S) # --date=@$ts (thx osx)
 
 mkdir -p ../dist
-sfx_out=../dist/copyparty-sfx
+sfx_out=../dist/copyparty-$CSN
 
 echo cleanup
 find -name '*.pyc' -delete
@@ -295,8 +368,8 @@ rm have
 		tmv "$f"
 	done
 
-[ $repack ] || {
-	# uncomment
+[ ! $repack ] && [ ! $use_ox ] && {
+	# uncomment; oxidized drops 45 KiB but becomes undebuggable
 	find | grep -E '\.py$' |
 		grep -vE '__version__' |
 		tr '\n' '\0' |
@@ -310,6 +383,7 @@ rm have
 f=j2/jinja2/constants.py
 awk '/^LOREM_IPSUM_WORDS/{o=1;print "LOREM_IPSUM_WORDS = u\"a\"";next} !o; /"""/{o=0}' <$f >t
 tmv "$f"
+rm -f j2/jinja2/async*
 
 grep -rLE '^#[^a-z]*coding: utf-8' j2 |
 while IFS= read -r f; do
@@ -340,9 +414,9 @@ find | grep -E '\.(js|html)$' | while IFS= read -r f; do
 done
 
 gzres() {
-	command -v pigz && [ $zopf ] &&
-		pk="pigz -11 -I $zopf" ||
-		pk='gzip'
+	[ $zopf ] && command -v zopfli && pk="zopfli --i$zopf"
+	[ $zopf ] && command -v pigz && pk="pigz -11 -I $zopf"
+	[ -z "$pk" ] && pk='gzip'
 
 	np=$(nproc)
 	echo "$pk #$np"
@@ -366,7 +440,7 @@ gzres() {
 }
 
 
-zdir="$tmpdir/cpp-mksfx"
+zdir="$tmpdir/cpp-mk$CSN"
 [ -e "$zdir/$stamp" ] || rm -rf "$zdir"
 mkdir -p "$zdir"
 echo a > "$zdir/$stamp"
@@ -391,14 +465,41 @@ nf=$(ls -1 "$zdir"/arc.* | wc -l)
 }
 
 
+[ $use_ox ] && {
+	tgt=x86_64-pc-windows-msvc
+	tgt=i686-pc-windows-msvc  # 2M smaller (770k after upx)
+	bdir=build/$tgt/release/install/copyparty
+
+	t="res web"
+	(printf "\n\n\nBUT WAIT! THERE'S MORE!!\n\n";
+	cat ../$bdir/COPYING.txt) >> copyparty/res/COPYING.txt ||
+		echo "copying.txt 404 pls rebuild"
+
+	mv ftp/* j2/* copyparty/vend/* .
+	rm -rf ftp j2 py2 copyparty/vend
+	(cd copyparty; tar -cvf z.tar $t; rm -rf $t)
+	cd ..
+	pyoxidizer build --release --target-triple $tgt
+	mv $bdir/copyparty.exe dist/
+	cp -pv "$(for d in '/c/Program Files (x86)/Microsoft Visual Studio/'*'/BuildTools/VC/Redist/MSVC'; do
+		find "$d" -name vcruntime140.dll; done | sort | grep -vE '/x64/|/onecore/'ƒ | head -n 1)" dist/
+	dist/copyparty.exe --version
+	cp -pv dist/copyparty{,.orig}.exe
+	[ $ultra ] && a="--best --lzma" || a=-1
+	/bin/time -f %es upx $a dist/copyparty.exe >/dev/null
+	ls -al dist/copyparty{,.orig}.exe
+	exit 0
+}
+
+
 echo gen tarlist
 for d in copyparty j2 ftp py2; do find $d -type f; done |  # strip_hints
 sed -r 's/(.*)\.(.*)/\2 \1/' | LC_ALL=C sort |
 sed -r 's/([^ ]*) (.*)/\2.\1/' | grep -vE '/list1?$' > list1
 
 for n in {1..50}; do
-	(grep -vE '\.(gz|br)$' list1; grep -E '\.(gz|br)$' list1 | shuf) >list || true
-	s=$(md5sum list | cut -c-16)
+	(grep -vE '\.(gz|br)$' list1; grep -E '\.(gz|br)$' list1 | (shuf||gshuf) ) >list || true
+	s=$( (sha1sum||shasum) < list | cut -c-16)
 	grep -q $s "$zdir/h" && continue
 	echo $s >> "$zdir/h"
 	break
@@ -406,11 +507,7 @@ done
 [ $n -eq 50 ] && exit
 
 echo creating tar
-args=(--owner=1000 --group=1000)
-[ "$OSTYPE" = msys ] &&
-	args=()
-
-tar -cf tar "${args[@]}" --numeric-owner -T list
+tar -cf tar "${targs[@]}" --numeric-owner -T list
 
 pc=bzip2
 pe=bz2
@@ -418,7 +515,7 @@ pe=bz2
 
 echo compressing tar
 # detect best level; bzip2 -7 is usually better than -9
-for n in {2..9}; do cp tar t.$n; $pc  -$n t.$n & done; wait; mv -v $(ls -1S t.*.$pe | tail -n 1) tar.bz2
+for n in {2..9}; do cp tar t.$n; nice $pc  -$n t.$n & done; wait; mv -v $(ls -1S t.*.$pe | tail -n 1) tar.bz2
 rm t.* || true
 exts=()
 

scripts/profile.py

@@ -8,7 +8,7 @@ cmd = sys.argv[1]
 if cmd == "cpp":
     from copyparty.__main__ import main
 
-    argv = ["__main__", "-v", "srv::r", "-v", "../../yt:yt:r"]
+    argv = ["__main__", "-vsrv::r:c,e2ds,e2ts"]
     main(argv=argv)
 
 elif cmd == "test":
@@ -29,6 +29,6 @@ else:
 #
 # python -m vmprof -o prof --lines ./scripts/profile.py test
 
-# linux: ~/.local/bin/vmprofshow prof tree | grep -vF '[1m  0.'
-# macos: ~/Library/Python/3.9/bin/vmprofshow prof tree | grep -vF '[1m  0.'
+# linux: ~/.local/bin/vmprofshow prof tree | awk '$2>1{n=5} !n{next} 1;{n--} !n{print""}'
+# macos: ~/Library/Python/3.9/bin/vmprofshow prof tree
 #   win: %appdata%\..\Roaming\Python\Python39\Scripts\vmprofshow.exe prof tree

scripts/py2/queue/__init__.py

@@ -0,0 +1,4 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+from Queue import Queue, LifoQueue, PriorityQueue, Empty, Full

scripts/pyinstaller/README.md

@@ -0,0 +1,12 @@
+builds a fully standalone copyparty.exe compatible with 32bit win7-sp1 and later
+
+requires a win7 vm which has never been connected to the internet and a host-only network with the linux host at 192.168.123.1
+
+first-time setup steps in notes.txt
+
+run build.sh in the vm to fetch src + compile + push a new exe to the linux host for manual publishing
+
+
+## ffmpeg
+
+built with [ffmpeg-windows-build-helpers](https://github.com/rdp/ffmpeg-windows-build-helpers) and [this patch](./ffmpeg.patch) using [these steps](./ffmpeg.txt)

scripts/pyinstaller/build.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+set -e
+
+curl http://192.168.123.1:3923/cpp/scripts/pyinstaller/build.sh |
+tee build2.sh | cmp build.sh && rm build2.sh || {
+    echo "new build script; upgrade y/n:"
+    while true; do read -u1 -n1 -r r; [[ $r =~ [yYnN] ]] && break; done
+    [[ $r =~ [yY] ]] && mv build{2,}.sh && exec ./build.sh
+}
+
+dl() { curl -fkLO "$1"; }
+
+cd ~/Downloads
+
+dl https://192.168.123.1:3923/cpp/dist/copyparty-sfx.py
+dl https://192.168.123.1:3923/cpp/scripts/pyinstaller/loader.ico
+dl https://192.168.123.1:3923/cpp/scripts/pyinstaller/loader.py
+dl https://192.168.123.1:3923/cpp/scripts/pyinstaller/loader.rc
+
+rm -rf $TEMP/pe-copyparty*
+python copyparty-sfx.py --version
+
+rm -rf mods; mkdir mods
+cp -pR $TEMP/pe-copyparty/copyparty/ $TEMP/pe-copyparty/{ftp,j2}/* mods/
+
+af() { awk "$1" <$2 >tf; mv tf "$2"; }
+
+rm -rf mods/magic/
+
+sed -ri /pickle/d mods/jinja2/_compat.py
+sed -ri '/(bccache|PackageLoader)/d' mods/jinja2/__init__.py
+af '/^class/{s=0}/^class PackageLoader/{s=1}!s' mods/jinja2/loaders.py
+
+sed -ri /fork_process/d mods/pyftpdlib/servers.py
+af '/^class _Base/{s=1}!s' mods/pyftpdlib/authorizers.py
+
+read a b c d _ < <(
+    grep -E '^VERSION =' mods/copyparty/__version__.py |
+    tail -n 1 |
+    sed -r 's/[^0-9]+//;s/[" )]//g;s/[-,]/ /g;s/$/ 0/'
+)
+sed -r 's/1,2,3,0/'$a,$b,$c,$d'/;s/1\.2\.3/'$a.$b.$c/ <loader.rc >loader.rc2
+
+$APPDATA/python/python37/scripts/pyinstaller \
+    -y --clean -p mods --upx-dir=. \
+    --exclude-module copyparty.broker_mp \
+    --exclude-module copyparty.broker_mpw \
+    --exclude-module curses \
+    --exclude-module ctypes.macholib \
+    --exclude-module multiprocessing \
+    --exclude-module pdb \
+    --exclude-module pickle \
+    --exclude-module pyftpdlib.prefork \
+    --exclude-module urllib.request \
+    --exclude-module urllib.response \
+    --exclude-module urllib.robotparser \
+    --exclude-module zipfile \
+    --version-file loader.rc2 -i loader.ico -n copyparty -c -F loader.py \
+    --add-data 'mods/copyparty/res;copyparty/res' \
+    --add-data 'mods/copyparty/web;copyparty/web'
+
+# ./upx.exe --best --ultra-brute --lzma -k dist/copyparty.exe
+
+curl -fkT dist/copyparty.exe -b cppwd=wark https://192.168.123.1:3923/

scripts/pyinstaller/ffmpeg.patch

@@ -0,0 +1,228 @@
+diff --git a/cross_compile_ffmpeg.sh b/cross_compile_ffmpeg.sh
+index 45c4ef8..f9bc83a 100755
+--- a/cross_compile_ffmpeg.sh
++++ b/cross_compile_ffmpeg.sh
+@@ -2287,15 +2287,8 @@ build_ffmpeg() {
+   else
+     local output_dir=$3
+   fi
+-  if [[ "$non_free" = "y" ]]; then
+-    output_dir+="_with_fdk_aac"
+-  fi
+-  if [[ $build_intel_qsv == "n" ]]; then
+-    output_dir+="_xp_compat"
+-  fi
+-  if [[ $enable_gpl == 'n' ]]; then
+-    output_dir+="_lgpl"
+-  fi
++  output_dir+="_xp_compat"
++  output_dir+="_lgpl"
+ 
+   if [[ ! -z $ffmpeg_git_checkout_version ]]; then
+     local output_branch_sanitized=$(echo ${ffmpeg_git_checkout_version} | sed "s/\//_/g") # release/4.3 to release_4.3
+@@ -2354,9 +2347,9 @@ build_ffmpeg() {
+       init_options+=" --disable-schannel"
+       # Fix WinXP incompatibility by disabling Microsoft's Secure Channel, because Windows XP doesn't support TLS 1.1 and 1.2, but with GnuTLS or OpenSSL it does.  XP compat!
+     fi
+-    config_options="$init_options --enable-libcaca --enable-gray --enable-libtesseract --enable-fontconfig --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libbs2b --enable-libflite --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libvorbis --enable-libwebp --enable-libzimg --enable-libzvbi --enable-libmysofa --enable-libopenjpeg  --enable-libopenh264  --enable-libvmaf --enable-libsrt --enable-libxml2 --enable-opengl --enable-libdav1d --enable-cuda-llvm"
++    config_options="$init_options --enable-gray --enable-libopus --enable-libvorbis --enable-libwebp --enable-libopenjpeg"
+ 
+-    if [[ $build_svt = y ]]; then
++    if [[ '' ]]; then
+       if [ "$bits_target" != "32" ]; then
+ 
+         # SVT-VP9 see comments below
+@@ -2379,40 +2372,13 @@ build_ffmpeg() {
+         config_options+=" --enable-libvpx"
+       fi # else doesn't work/matter with 32 bit
+     fi
+-    config_options+=" --enable-libaom"
+-
+-    if [[ $compiler_flavors != "native" ]]; then
+-      config_options+=" --enable-nvenc --enable-nvdec" # don't work OS X
+-    fi
+ 
+-    config_options+=" --extra-libs=-lharfbuzz" #  grr...needed for pre x264 build???
+     config_options+=" --extra-libs=-lm" # libflite seemed to need this linux native...and have no .pc file huh?
+     config_options+=" --extra-libs=-lshlwapi" # lame needed this, no .pc file?
+-    config_options+=" --extra-libs=-lmpg123" # ditto
+     config_options+=" --extra-libs=-lpthread" # for some reason various and sundry needed this linux native
+ 
+-    config_options+=" --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DCACA_STATIC" # if we ever do a git pull then it nukes changes, which overrides manual changes to configure, so just use these for now :|
+-    if [[ $build_amd_amf = n ]]; then
+-      config_options+=" --disable-amf" # Since its autodetected we have to disable it if we do not want it. #unless we define no autodetection but.. we don't.
+-    else
+-      config_options+=" --enable-amf" # This is actually autodetected but for consistency.. we might as well set it.
+-    fi
+-
+-    if [[ $build_intel_qsv = y && $compiler_flavors != "native" ]]; then # Broken for native builds right now: https://github.com/lu-zero/mfx_dispatch/issues/71
+-      config_options+=" --enable-libmfx"
+-    else
+-      config_options+=" --disable-libmfx"
+-    fi
+-    if [[ $enable_gpl == 'y' ]]; then
+-      config_options+=" --enable-gpl --enable-frei0r --enable-librubberband --enable-libvidstab --enable-libx264 --enable-libx265 --enable-avisynth --enable-libaribb24"
+-      config_options+=" --enable-libxvid --enable-libdavs2"
+-      if [[ $host_target != 'i686-w64-mingw32' ]]; then
+-        config_options+=" --enable-libxavs2"
+-      fi
+-      if [[ $compiler_flavors != "native" ]]; then
+-        config_options+=" --enable-libxavs" # don't compile OS X
+-      fi
+-    fi
++    config_options+=" --disable-amf" # Since its autodetected we have to disable it if we do not want it. #unless we define no autodetection but.. we don't.
++    config_options+=" --disable-libmfx"
+     local licensed_gpl=n # lgpl build with libx264 included for those with "commercial" license :)
+     if [[ $licensed_gpl == 'y' ]]; then
+       apply_patch file://$patch_dir/x264_non_gpl.diff -p1
+@@ -2427,7 +2393,7 @@ build_ffmpeg() {
+ 
+     config_options+=" $postpend_configure_opts"
+ 
+-    if [[ "$non_free" = "y" ]]; then
++    if [[ '' ]]; then
+       config_options+=" --enable-nonfree --enable-libfdk-aac"
+ 
+       if [[ $compiler_flavors != "native" ]]; then
+@@ -2436,6 +2402,17 @@ build_ffmpeg() {
+       # other possible options: --enable-openssl [unneeded since we already use gnutls]
+     fi
+ 
++    config_options+=" --disable-indevs --disable-outdevs --disable-protocols --disable-hwaccels --disable-schannel --disable-mediafoundation"  # 8032256
++    config_options+=" --disable-muxers --enable-muxer=image2 --enable-muxer=mjpeg --enable-muxer=opus --enable-muxer=webp"  # 7927296
++    config_options+=" --disable-encoders --enable-encoder=libopus --enable-encoder=libopenjpeg --enable-encoder=libwebp --enable-encoder=ljpeg --enable-encoder=png"  # 6776320
++    config_options+=" --enable-small"  # 5409792
++    #config_options+=" --disable-runtime-cpudetect"  # 5416448
++    config_options+=" --disable-bsfs --disable-filters --enable-filter=scale --enable-filter=compand --enable-filter=volume --enable-filter=showwavespic --enable-filter=convolution --enable-filter=aresample --enable-filter=showspectrumpic --enable-filter=crop"  # 4647424
++    config_options+=" --disable-network"  # 4585984
++    #config_options+=" --disable-pthreads --disable-w32threads"  # kills ffmpeg
++    config_options+=" --enable-protocol=cache --enable-protocol=file --enable-protocol=pipe"
++
++
+     do_debug_build=n # if you need one for backtraces/examining segfaults using gdb.exe ... change this to y :) XXXX make it affect x264 too...and make it real param :)
+     if [[ "$do_debug_build" = "y" ]]; then
+       # not sure how many of these are actually needed/useful...possibly none LOL
+@@ -2561,36 +2538,16 @@ build_ffmpeg_dependencies() {
+   build_meson_cross
+   build_mingw_std_threads
+   build_zlib # Zlib in FFmpeg is autodetected.
+-  build_libcaca # Uses zlib and dlfcn (on windows).
+   build_bzip2 # Bzlib (bzip2) in FFmpeg is autodetected.
+   build_liblzma # Lzma in FFmpeg is autodetected. Uses dlfcn.
+   build_iconv # Iconv in FFmpeg is autodetected. Uses dlfcn.
+-  build_sdl2 # Sdl2 in FFmpeg is autodetected. Needed to build FFPlay. Uses iconv and dlfcn.
+-  if [[ $build_amd_amf = y ]]; then
+-    build_amd_amf_headers
+-  fi
+-  if [[ $build_intel_qsv = y && $compiler_flavors != "native" ]]; then # Broken for native builds right now: https://github.com/lu-zero/mfx_dispatch/issues/71
+-    build_intel_quicksync_mfx
+-  fi
+-  build_nv_headers
+   build_libzimg # Uses dlfcn.
+   build_libopenjpeg
+-  build_glew
+-  build_glfw
+   #build_libjpeg_turbo # mplayer can use this, VLC qt might need it? [replaces libjpeg] (ffmpeg seems to not need it so commented out here)
+   build_libpng # Needs zlib >= 1.0.4. Uses dlfcn.
+   build_libwebp # Uses dlfcn.
+-  build_harfbuzz
+   # harf does now include build_freetype # Uses zlib, bzip2, and libpng.
+-  build_libxml2 # Uses zlib, liblzma, iconv and dlfcn.
+-  build_libvmaf
+-  build_fontconfig # Needs freetype and libxml >= 2.6. Uses iconv and dlfcn.
+-  build_gmp # For rtmp support configure FFmpeg with '--enable-gmp'. Uses dlfcn.
+   #build_librtmfp # mainline ffmpeg doesn't use it yet
+-  build_libnettle # Needs gmp >= 3.0. Uses dlfcn.
+-  build_unistring
+-  build_libidn2 # needs iconv and unistring
+-  build_gnutls # Needs nettle >= 3.1, hogweed (nettle) >= 3.1. Uses libidn2, unistring, zlib, and dlfcn.
+   #if [[ "$non_free" = "y" ]]; then
+   #  build_openssl-1.0.2 # Nonfree alternative to GnuTLS. 'build_openssl-1.0.2 "dllonly"' to build shared libraries only.
+   #  build_openssl-1.1.1 # Nonfree alternative to GnuTLS. Can't be used with LibRTMP. 'build_openssl-1.1.1 "dllonly"' to build shared libraries only.
+@@ -2598,86 +2555,13 @@ build_ffmpeg_dependencies() {
+   build_libogg # Uses dlfcn.
+   build_libvorbis # Needs libogg >= 1.0. Uses dlfcn.
+   build_libopus # Uses dlfcn.
+-  build_libspeexdsp # Needs libogg for examples. Uses dlfcn.
+-  build_libspeex # Uses libspeexdsp and dlfcn.
+-  build_libtheora # Needs libogg >= 1.1. Needs libvorbis >= 1.0.1, sdl and libpng for test, programs and examples [disabled]. Uses dlfcn.
+-  build_libsndfile "install-libgsm" # Needs libogg >= 1.1.3 and libvorbis >= 1.2.3 for external support [disabled]. Uses dlfcn. 'build_libsndfile "install-libgsm"' to install the included LibGSM 6.10.
+-  build_mpg123
+-  build_lame # Uses dlfcn, mpg123
+-  build_twolame # Uses libsndfile >= 1.0.0 and dlfcn.
+-  build_libopencore # Uses dlfcn.
+-  build_libilbc # Uses dlfcn.
+-  build_libmodplug # Uses dlfcn.
+-  build_libgme
+-  build_libbluray # Needs libxml >= 2.6, freetype, fontconfig. Uses dlfcn.
+-  build_libbs2b # Needs libsndfile. Uses dlfcn.
+-  build_libsoxr
+-  build_libflite
+-  build_libsnappy # Uses zlib (only for unittests [disabled]) and dlfcn.
+-  build_vamp_plugin # Needs libsndfile for 'vamp-simple-host.exe' [disabled].
+   build_fftw # Uses dlfcn.
+-  build_libsamplerate # Needs libsndfile >= 1.0.6 and fftw >= 0.15.0 for tests. Uses dlfcn.
+-  build_librubberband # Needs libsamplerate, libsndfile, fftw and vamp_plugin. 'configure' will fail otherwise. Eventhough librubberband doesn't necessarily need them (libsndfile only for 'rubberband.exe' and vamp_plugin only for "Vamp audio analysis plugin"). How to use the bundled libraries '-DUSE_SPEEX' and '-DUSE_KISSFFT'?
+-  build_frei0r # Needs dlfcn. could use opencv...
+-  if [[ "$bits_target" != "32" && $build_svt = "y" ]]; then
+-    build_svt-hevc
+-    build_svt-av1
+-    build_svt-vp9
+-  fi
+-  build_vidstab
+-  #build_facebooktransform360 # needs modified ffmpeg to use it so not typically useful
+-  build_libmysofa # Needed for FFmpeg's SOFAlizer filter (https://ffmpeg.org/ffmpeg-filters.html#sofalizer). Uses dlfcn.
+-  if [[ "$non_free" = "y" ]]; then
+-    build_fdk-aac # Uses dlfcn.
+-    if [[ $compiler_flavors != "native" ]]; then
+-      build_libdecklink # Error finding rpc.h in native builds even if it's available
+-    fi
+-  fi
+-  build_zvbi # Uses iconv, libpng and dlfcn.
+-  build_fribidi # Uses dlfcn.
+-  build_libass # Needs freetype >= 9.10.3 (see https://bugs.launchpad.net/ubuntu/+source/freetype1/+bug/78573 o_O) and fribidi >= 0.19.0. Uses fontconfig >= 2.10.92, iconv and dlfcn.
+-
+-  build_libxvid # FFmpeg now has native support, but libxvid still provides a better image.
+-  build_libsrt # requires gnutls, mingw-std-threads
+-  build_libaribb24
+-  build_libtesseract
+-  build_lensfun  # requires png, zlib, iconv
+-  # build_libtensorflow # broken
+-  build_libvpx
+-  build_libx265
+-  build_libopenh264
+-  build_libaom
+-  build_dav1d
+-  build_avisynth
+-  build_libx264 # at bottom as it might internally build a copy of ffmpeg (which needs all the above deps...
+  }
+ 
+ build_apps() {
+-  if [[ $build_dvbtee = "y" ]]; then
+-    build_dvbtee_app
+-  fi
+-  # now the things that use the dependencies...
+-  if [[ $build_libmxf = "y" ]]; then
+-    build_libMXF
+-  fi
+-  if [[ $build_mp4box = "y" ]]; then
+-    build_mp4box
+-  fi
+-  if [[ $build_mplayer = "y" ]]; then
+-    build_mplayer
+-  fi
+   if [[ $build_ffmpeg_static = "y" ]]; then
+     build_ffmpeg static
+   fi
+-  if [[ $build_ffmpeg_shared = "y" ]]; then
+-    build_ffmpeg shared
+-  fi
+-  if [[ $build_vlc = "y" ]]; then
+-    build_vlc
+-  fi
+-  if [[ $build_lsw = "y" ]]; then
+-    build_lsw
+-  fi
+ }
+ 
+ # set some parameters initial values

scripts/pyinstaller/ffmpeg.txt

@@ -0,0 +1,13 @@
+apt install subversion ragel curl texinfo ed bison flex cvs yasm automake libtool cmake git make pkg-config pax nasm gperf autogen bzip2 autoconf-archive p7zip-full meson clang libtool-bin ed python-is-python3
+
+git clone https://github.com/rdp/ffmpeg-windows-build-helpers
+# commit 3d88e2b6aedfbb5b8fed19dd24621e5dd7fc5519 (HEAD -> master, origin/master, origin/HEAD)
+# Merge: b0bd70c 9905dd7
+# Author: Roger Pack <rogerpack2005@gmail.com>
+# Date:   Fri Aug 19 23:36:35 2022 -0600
+
+cd ffmpeg-windows-build-helpers/
+vim cross_compile_ffmpeg.sh
+(cd ./sandbox/win32/ffmpeg_git_xp_compat_lgpl/ ; git reset --hard ; git clean -fx )
+./cross_compile_ffmpeg.sh
+for f in sandbox/win32/ffmpeg_git_xp_compat_lgpl/ff{mpeg,probe}.exe; do upx --best --ultra-brute -k $f; mv $f ~/dev; done 

scripts/pyinstaller/icon.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+set -e
+
+# imagemagick png compression is broken, use pillow instead
+convert ~/AndroidStudioProjects/PartyUP/metadata/en-US/images/icon.png a.bmp
+
+#convert a.bmp -trim -resize '48x48!' -strip a.png
+python3 <<'EOF'
+from PIL import Image
+i = Image.open('a.bmp')
+i = i.crop(i.getbbox())
+i = i.resize((48,48), Image.BICUBIC)
+i = Image.alpha_composite(i,i)
+i.save('a.png')
+EOF
+
+pngquant --strip --quality 30 a.png
+mv a-*.png a.png
+
+python3 <<'EOF'
+from PIL import Image
+Image.open('a.png').save('loader.ico',sizes=[(48,48)])
+EOF
+
+rm a.{bmp,png}
+ls -al
+exit 0

scripts/pyinstaller/loader.py

@@ -0,0 +1,94 @@
+# coding: utf-8
+
+v = r"""
+
+this is the EXE edition of copyparty, compatible with Windows7-SP1
+and later. To make this possible, the EXE was compiled with Python
+3.7.9, which is EOL and does not receive security patches anymore.
+
+if possible, for performance and security reasons, please use this instead:
+https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py
+"""
+
+print(v.replace("\n", "\n▒▌ ")[1:] + "\n")
+
+
+import re
+import os
+import sys
+import shutil
+import traceback
+import subprocess as sp
+
+
+def confirm(rv):
+    print()
+    print("retcode", rv if rv else traceback.format_exc())
+    print("*** hit enter to exit ***")
+    try:
+        input()
+    except:
+        pass
+
+    sys.exit(rv or 1)
+
+
+def meicln(mod):
+    pdir, mine = os.path.split(mod)
+    dirs = os.listdir(pdir)
+    dirs = [x for x in dirs if x.startswith("_MEI") and x != mine]
+    dirs = [os.path.join(pdir, x) for x in dirs]
+    rm = []
+    for d in dirs:
+        if os.path.isdir(os.path.join(d, "copyparty", "web")):
+            rm.append(d)
+
+    if not rm:
+        return
+
+    print("deleting abandoned SFX dirs:")
+    for d in rm:
+        print(d)
+        for _ in range(9):
+            try:
+                shutil.rmtree(d)
+                break
+            except:
+                pass
+
+    print()
+
+
+def meichk():
+    filt = "copyparty"
+    if filt not in sys.executable:
+        filt = os.path.basename(sys.executable)
+
+    pids = []
+    ptn = re.compile(r"^([^\s]+)\s+([0-9]+)")
+    procs = sp.check_output("tasklist").decode("utf-8", "replace")
+    for ln in procs.splitlines():
+        m = ptn.match(ln)
+        if m and filt in m.group(1).lower():
+            pids.append(int(m.group(2)))
+
+    mod = os.path.dirname(os.path.realpath(__file__))
+    if os.path.basename(mod).startswith("_MEI") and len(pids) == 2:
+        meicln(mod)
+
+
+meichk()
+
+
+from copyparty.__main__ import main
+
+try:
+    main()
+except SystemExit as ex:
+    c = ex.code
+    if c not in [0, -15]:
+        confirm(ex.code)
+except KeyboardInterrupt:
+    pass
+except:
+    confirm(0)

scripts/pyinstaller/loader.rc

@@ -0,0 +1,29 @@
+# UTF-8
+VSVersionInfo(
+  ffi=FixedFileInfo(
+    filevers=(1,2,3,0),
+    prodvers=(1,2,3,0),
+    mask=0x3f,
+    flags=0x0,
+    OS=0x4,
+    fileType=0x1,
+    subtype=0x0,
+    date=(0, 0)
+  ),
+  kids=[
+    StringFileInfo(
+      [
+      StringTable(
+        '000004b0',
+        [StringStruct('CompanyName', 'ocv.me'),
+        StringStruct('FileDescription', 'copyparty'),
+        StringStruct('FileVersion', '1.2.3'),
+        StringStruct('InternalName', 'copyparty'),
+        StringStruct('LegalCopyright', '2019, ed'),
+        StringStruct('OriginalFilename', 'copyparty.exe'),
+        StringStruct('ProductName', 'copyparty'),
+        StringStruct('ProductVersion', '1.2.3')])
+      ]), 
+    VarFileInfo([VarStruct('Translation', [0, 1200])])
+  ]
+)

scripts/pyinstaller/notes.txt

@@ -0,0 +1,58 @@
+run ./build.sh in git-bash to build + upload the exe
+
+
+## ============================================================
+## first-time setup on a stock win7x32sp1 vm:
+## 
+
+download + install git:
+http://192.168.123.1:3923/ro/pyi/Git-2.37.3-32-bit.exe
+
+<git-bash>
+dl() { curl -fkLOC- "$1"; }
+cd ~/Downloads &&
+dl https://192.168.123.1:3923/ro/pyi/upx-3.96-win32.zip &&
+dl https://192.168.123.1:3923/ro/pyi/KB2533623/Windows6.1-KB2533623-x86.msu &&
+dl https://192.168.123.1:3923/ro/pyi/python-3.7.9.exe &&
+dl https://192.168.123.1:3923/ro/pyi/pip-22.2.2-py3-none-any.whl &&
+dl https://192.168.123.1:3923/ro/pyi/altgraph-0.17.2-py2.py3-none-any.whl &&
+dl https://192.168.123.1:3923/ro/pyi/future-0.18.2.tar.gz &&
+dl https://192.168.123.1:3923/ro/pyi/importlib_metadata-4.12.0-py3-none-any.whl &&
+dl https://192.168.123.1:3923/ro/pyi/pefile-2022.5.30.tar.gz &&
+dl https://192.168.123.1:3923/ro/pyi/pyinstaller-5.4.1-py3-none-win32.whl &&
+dl https://192.168.123.1:3923/ro/pyi/pyinstaller_hooks_contrib-2022.10-py2.py3-none-any.whl &&
+dl https://192.168.123.1:3923/ro/pyi/pywin32_ctypes-0.2.0-py2.py3-none-any.whl &&
+dl https://192.168.123.1:3923/ro/pyi/typing_extensions-4.3.0-py3-none-any.whl &&
+dl https://192.168.123.1:3923/ro/pyi/zipp-3.8.1-py3-none-any.whl &&
+echo ok
+
+manually install:
+  windows6.1-kb2533623-x86.msu + reboot
+  python-3.7.9.exe
+
+<git-bash>
+cd ~/Downloads &&
+unzip -j upx-3.96-win32.zip upx-3.96-win32/upx.exe &&
+python -m ensurepip &&
+python -m pip install --user -U pip-22.2.2-py3-none-any.whl &&
+python -m pip install --user -U pyinstaller-5.4.1-py3-none-win32.whl pefile-2022.5.30.tar.gz pywin32_ctypes-0.2.0-py2.py3-none-any.whl pyinstaller_hooks_contrib-2022.10-py2.py3-none-any.whl altgraph-0.17.2-py2.py3-none-any.whl future-0.18.2.tar.gz importlib_metadata-4.12.0-py3-none-any.whl typing_extensions-4.3.0-py3-none-any.whl zipp-3.8.1-py3-none-any.whl &&
+echo ok
+# python -m pip install --user -U Pillow-9.2.0-cp37-cp37m-win32.whl
+# sed -ri 's/, bestopt, /]+bestopt+[/' $APPDATA/Python/Python37/site-packages/pyinstaller/building/utils.py
+# sed -ri 's/(^\s+bestopt = ).*/\1["--best","--lzma","--ultra-brute"]/' $APPDATA/Python/Python37/site-packages/pyinstaller/building/utils.py
+
+
+## ============================================================
+## notes
+##
+
+size   t-unpack  virustotal                     cmnt
+8059k  0m0.375s  5/70 generic-only, sandbox-ok  no-upx
+7095k  0m0.563s  4/70 generic-only, sandbox-ok  standard-upx
+6958k  0m0.578s  7/70 generic-only, sandbox-ok  upx+upx
+
+use python 3.7 since 3.8 onwards requires KB2533623 on target
+
+generate loader.rc template:
+%appdata%\python\python37\scripts\pyi-grab_version C:\Users\ed\AppData\Local\Programs\Python\Python37\python.exe
+

scripts/rls.sh

@@ -1,7 +1,13 @@
 #!/bin/bash
 set -e
 
-cd ~/dev/copyparty/scripts
+parallel=2
+
+[ -e make-sfx.sh ] || cd scripts
+[ -e make-sfx.sh ] && [ -e deps-docker ] || {
+    echo cd into the scripts folder first
+    exit 1
+}
 
 v=$1
 
@@ -21,14 +27,36 @@ v=$1
     ./make-tgz-release.sh $v
 }
 
-rm -f ../dist/copyparty-sfx.*
-f=../dist/copyparty-sfx.py
-./make-sfx.sh
-$f -h
+rm -f ../dist/copyparty-sfx*
+shift
+./make-sfx.sh "$@"
+f=../dist/copyparty-sfx
+[ -e $f.py ] || 
+    f=../dist/copyparty-sfx-gz
+
+$f.py -h >/dev/null
+
+[ $parallel -gt 1 ] && {
+    printf '\033[%s' s 2r H "0;1;37;44mbruteforcing sfx size -- press enter to terminate" K u "7m $* " K $'27m\n'
+    trap "rm -f .sfx-run; printf '\033[%s' s r u" INT TERM EXIT
+    touch .sfx-run
+    min=99999999
+    for ((a=0; a<$parallel; a++)); do
+        while [ -e .sfx-run ]; do
+            CSN=sfx$a ./make-sfx.sh re "$@"
+            sz=$(wc -c <$f$a.py | awk '{print$1}')
+            [ $sz -ge $min ] && continue
+            mv $f$a.py $f.py.$sz
+            min=$sz
+        done &
+    done
+    read
+    exit
+}
 
 while true; do
-    mv $f $f.$(wc -c <$f | awk '{print$1}')
-    ./make-sfx.sh re $ar
+    mv $f.py $f.$(wc -c <$f.py | awk '{print$1}').py
+    ./make-sfx.sh re "$@"
 done
 
 # git tag -d v$v; git push --delete origin v$v

scripts/run-tests.sh

@@ -17,8 +17,10 @@ for py in python{2,3}; do
     pids+=($!)
 done
 
-python3 ../scripts/test/smoketest.py &
-pids+=($!)
+[ "$1" ] || {
+    python3 ../scripts/test/smoketest.py &
+    pids+=($!)
+}
 
 for pid in ${pids[@]}; do
     wait $pid

scripts/sfx.ls

@@ -19,6 +19,7 @@ copyparty/httpsrv.py,
 copyparty/ico.py,
 copyparty/mtag.py,
 copyparty/res,
+copyparty/res/COPYING.txt,
 copyparty/res/insecure.pem,
 copyparty/star.py,
 copyparty/stolen,
@@ -43,6 +44,7 @@ copyparty/web/browser.html,
 copyparty/web/browser.js,
 copyparty/web/browser2.html,
 copyparty/web/copyparty.gif,
+copyparty/web/cf.html,
 copyparty/web/dd,
 copyparty/web/dd/2.png,
 copyparty/web/dd/3.png,
@@ -76,3 +78,4 @@ copyparty/web/splash.js,
 copyparty/web/ui.css,
 copyparty/web/up2k.js,
 copyparty/web/util.js,
+copyparty/web/w.hash.js,

scripts/sfx.py

@@ -213,22 +213,31 @@ def yieldfile(fn):
 
 
 def hashfile(fn):
-    h = hashlib.md5()
+    h = hashlib.sha1()
     for block in yieldfile(fn):
         h.update(block)
 
-    return h.hexdigest()
+    return h.hexdigest()[:24]
 
 
 def unpack():
     """unpacks the tar yielded by `data`"""
-    name = "pe-copyparty"
+    name = "pe-copyparty."
+    try:
+        name += str(os.geteuid())
+    except:
+        pass
+
     tag = "v" + str(STAMP)
-    withpid = "{}.{}".format(name, os.getpid())
     top = tempfile.gettempdir()
     opj = os.path.join
     final = opj(top, name)
-    mine = opj(top, withpid)
+    for suf in range(0, 9001):
+        withpid = "{}.{}.{}".format(name, os.getpid(), suf)
+        mine = opj(top, withpid)
+        if not os.path.exists(mine):
+            break
+
     tar = opj(mine, "tar")
 
     try:
@@ -360,11 +369,12 @@ def utime(top):
 def confirm(rv):
     msg()
     msg("retcode", rv if rv else traceback.format_exc())
-    msg("*** hit enter to exit ***")
-    try:
-        raw_input() if PY2 else input()
-    except:
-        pass
+    if WINDOWS:
+        msg("*** hit enter to exit ***")
+        try:
+            raw_input() if PY2 else input()
+        except:
+            pass
 
     sys.exit(rv or 1)
 

tests/test_httpcli.py

@@ -10,9 +10,10 @@ import pprint
 import tarfile
 import tempfile
 import unittest
-from argparse import Namespace
 
 from tests import util as tu
+from tests.util import Cfg
+
 from copyparty.authsrv import AuthSrv
 from copyparty.httpcli import HttpCli
 
@@ -22,56 +23,6 @@ def hdr(query):
     return h.format(query).encode("utf-8")
 
 
-class Cfg(Namespace):
-    def __init__(self, a=None, v=None, c=None):
-        super(Cfg, self).__init__(
-            a=a or [],
-            v=v or [],
-            c=c,
-            rproxy=0,
-            rsp_slp=0,
-            s_wr_slp=0,
-            s_wr_sz=512 * 1024,
-            ed=False,
-            nw=False,
-            unpost=600,
-            no_mv=False,
-            no_del=False,
-            no_zip=False,
-            no_thumb=False,
-            no_athumb=False,
-            no_vthumb=False,
-            no_voldump=True,
-            no_scandir=False,
-            no_sendfile=True,
-            no_rescan=True,
-            no_logues=False,
-            no_readme=False,
-            re_maxage=0,
-            ihead=False,
-            nih=True,
-            mtp=[],
-            mte="a",
-            mth="",
-            textfiles="",
-            doctitle="",
-            html_head="",
-            lang="eng",
-            theme=0,
-            themes=0,
-            turbo=0,
-            logout=573,
-            hist=None,
-            no_idx=None,
-            no_hash=None,
-            force_js=False,
-            no_robots=False,
-            js_browser=None,
-            css_browser=None,
-            **{k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr no_acode".split()}
-        )
-
-
 class TestHttpCli(unittest.TestCase):
     def setUp(self):
         self.td = tu.get_ramdisk()

tests/test_vfs.py

@@ -8,44 +8,14 @@ import shutil
 import tempfile
 import unittest
 from textwrap import dedent
-from argparse import Namespace
 
 from tests import util as tu
+from tests.util import Cfg
+
 from copyparty.authsrv import AuthSrv, VFS
 from copyparty import util
 
 
-class Cfg(Namespace):
-    def __init__(self, a=None, v=None, c=None):
-        ex = "nw e2d e2ds e2dsa e2t e2ts e2tsr no_logues no_readme no_acode force_js no_robots no_thumb no_athumb no_vthumb"
-        ex = {k: False for k in ex.split()}
-        ex2 = {
-            "mtp": [],
-            "mte": "a",
-            "mth": "",
-            "doctitle": "",
-            "html_head": "",
-            "hist": None,
-            "no_idx": None,
-            "no_hash": None,
-            "js_browser": None,
-            "css_browser": None,
-            "no_voldump": True,
-            "re_maxage": 0,
-            "rproxy": 0,
-            "rsp_slp": 0,
-            "s_wr_slp": 0,
-            "s_wr_sz": 512 * 1024,
-            "lang": "eng",
-            "theme": 0,
-            "themes": 0,
-            "turbo": 0,
-            "logout": 573,
-        }
-        ex.update(ex2)
-        super(Cfg, self).__init__(a=a or [], v=v or [], c=c, **ex)
-
-
 class TestVFS(unittest.TestCase):
     def setUp(self):
         self.td = tu.get_ramdisk()

tests/util.py

@@ -7,6 +7,7 @@ import threading
 import tempfile
 import platform
 import subprocess as sp
+from argparse import Namespace
 
 
 WINDOWS = platform.system() == "Windows"
@@ -29,8 +30,12 @@ if MACOS:
     # 25% faster; until any tests do symlink stuff
 
 
+from copyparty.__init__ import E
+from copyparty.__main__ import init_E
 from copyparty.util import Unrecv, FHC
 
+init_E(E)
+
 
 def runcmd(argv):
     p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
@@ -89,6 +94,41 @@ def get_ramdisk():
         return subdir(ret)
 
 
+class Cfg(Namespace):
+    def __init__(self, a=None, v=None, c=None):
+        ka = {}
+
+        ex = "e2d e2ds e2dsa e2t e2ts e2tsr e2v e2vu e2vp xdev xvol ed emp force_js ihead magic no_acode no_athumb no_del no_logues no_mv no_readme no_robots no_scandir no_thumb no_vthumb no_zip nid nih nw"
+        ka.update(**{k: False for k in ex.split()})
+
+        ex = "no_rescan no_sendfile no_voldump plain_ip"
+        ka.update(**{k: True for k in ex.split()})
+
+        ex = "css_browser hist js_browser no_hash no_idx no_forget"
+        ka.update(**{k: None for k in ex.split()})
+
+        ex = "re_maxage rproxy rsp_slp s_wr_slp theme themes turbo df"
+        ka.update(**{k: 0 for k in ex.split()})
+
+        ex = "doctitle favico html_head mth textfiles log_fk"
+        ka.update(**{k: "" for k in ex.split()})
+
+        super(Cfg, self).__init__(
+            a=a or [],
+            v=v or [],
+            c=c,
+            E=E,
+            s_wr_sz=512 * 1024,
+            unpost=600,
+            u2sort="s",
+            mtp=[],
+            mte="a",
+            lang="eng",
+            logout=573,
+            **ka
+        )
+
+
 class NullBroker(object):
     def say(*args):
         pass
@@ -120,6 +160,7 @@ class VHttpSrv(object):
     def __init__(self):
         self.broker = NullBroker()
         self.prism = None
+        self.bans = {}
 
         aliases = ["splash", "browser", "browser2", "msg", "md", "mde"]
         self.j2 = {x: J2_FILES for x in aliases}