v1.2.3

README.md

@@ -16,6 +16,13 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [unpost](#unpost) // [thumbnails](#thumbnails) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [md-viewer](#markdown-viewer) // [ie4](#browser-support)
 
 
+## get the app
+
+<a href="https://f-droid.org/packages/me.ocv.partyup/"><img src="https://ocv.me/fdroid.png" alt="Get it on F-Droid" height="50" /> '' <img src="https://img.shields.io/f-droid/v/me.ocv.partyup.svg" alt="f-droid version info" /></a> '' <a href="https://github.com/9001/party-up"><img src="https://img.shields.io/github/release/9001/party-up.svg?logo=github" alt="github version info" /></a>
+
+(the app is **NOT** the full copyparty server! just a basic upload client, nothing fancy yet)
+
+
 ## readme toc
 
 * top
@@ -47,13 +54,15 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [other tricks](#other-tricks)
     * [searching](#searching) - search by size, date, path/name, mp3-tags, ...
 * [server config](#server-config) - using arguments or config files, or a mix of both
+    * [ftp-server](#ftp-server) - an FTP server can be started using `--ftp 3921`
     * [file indexing](#file-indexing)
     * [upload rules](#upload-rules) - set upload rules using volume flags
     * [compress uploads](#compress-uploads) - files can be autocompressed on upload
     * [database location](#database-location) - in-volume (`.hist/up2k.db`, default) or somewhere else
     * [metadata from audio files](#metadata-from-audio-files) - set `-e2t` to index tags on upload
-    * [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags
+    * [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags, also see [./bin/mtag/README.md](./bin/mtag/README.md)
     * [upload events](#upload-events) - trigger a script/program on each upload
+    * [hiding from google](#hiding-from-google) - tell search engines you dont wanna be indexed
     * [complete examples](#complete-examples)
 * [browser support](#browser-support) - TLDR: yes
 * [client examples](#client-examples) - interact with copyparty using non-browser clients
@@ -75,7 +84,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [optional dependencies](#optional-dependencies) - install these to enable bonus features
     * [install recommended deps](#install-recommended-deps)
     * [optional gpl stuff](#optional-gpl-stuff)
-* [sfx](#sfx) - there are two self-contained "binaries"
+* [sfx](#sfx) - the self-contained "binary"
     * [sfx repack](#sfx-repack) - reduce the size of an sfx by removing features
 * [install on android](#install-on-android)
 * [reporting bugs](#reporting-bugs) - ideas for context to include in bug reports
@@ -147,6 +156,7 @@ feature summary
   * ☑ multiprocessing (actual multithreading)
   * ☑ volumes (mountpoints)
   * ☑ [accounts](#accounts-and-volumes)
+  * ☑ [ftp-server](#ftp-server)
 * upload
   * ☑ basic: plain multipart, ie6 support
   * ☑ [up2k](#uploading): js, resumable, multithreaded
@@ -169,7 +179,6 @@ feature summary
     * ☑ ...of audio (spectrograms) using FFmpeg
     * ☑ cache eviction (max-age; maybe max-size eventually)
   * ☑ SPA (browse while uploading)
-    * if you use the navpane to navigate, not folders in the file list
 * server indexing
   * ☑ [locate files by contents](#file-search)
   * ☑ search by name/path/date/size
@@ -231,6 +240,8 @@ some improvement ideas
 
 ## general bugs
 
+* Windows: if the up2k db is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
+  * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
 * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
 * probably more, pls let me know
 
@@ -325,6 +336,7 @@ the browser has the following hotkeys  (always qwerty)
 * `V` toggle folders / textfiles in the navpane
 * `G` toggle list / [grid view](#thumbnails)
 * `T` toggle thumbnails / icons
+* `ESC` close various things
 * `ctrl-X` cut selected files/folders
 * `ctrl-V` paste
 * `F2` [rename](#batch-rename) selected file/folder
@@ -376,9 +388,13 @@ switching between breadcrumbs or navpane
 
 click the `🌲` or pressing the `B` hotkey to toggle between breadcrumbs path (default), or a navpane (tree-browser sidebar thing)
 
-* `[-]` and `[+]` (or hotkeys `A`/`D`) adjust the size
-* `[v]` jumps to the currently open folder
+* `[+]` and `[-]` (or hotkeys `A`/`D`) adjust the size
+* `[🎯]` jumps to the currently open folder
+* `[📃]` toggles between showing folders and textfiles
+* `[📌]` shows the name of all parent folders in a docked panel
 * `[a]` toggles automatic widening as you go deeper
+* `[↵]` toggles wordwrap
+* `[👀]` show full name on hover (if wordwrap is off)
 
 
 ## thumbnails
@@ -394,6 +410,7 @@ audio files are covnerted into spectrograms using FFmpeg unless you `--no-athumb
 images with the following names (see `--th-covers`) become the thumbnail of the folder they're in: `folder.png`, `folder.jpg`, `cover.png`, `cover.jpg`
 
 in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked
+* indicated by the audio files having the ▶ icon instead of 💾
 
 
 ## zip downloads
@@ -444,7 +461,7 @@ see [up2k](#up2k) for details on how it works
 
 ![copyparty-upload-fs8](https://user-images.githubusercontent.com/241032/129635371-48fc54ca-fa91-48e3-9b1d-ba413e4b68cb.png)
 
-**protip:** you can avoid scaring away users with [docs/minimal-up2k.html](docs/minimal-up2k.html) which makes it look [much simpler](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
+**protip:** you can avoid scaring away users with [contrib/plugins/minimal-up2k.html](contrib/plugins/minimal-up2k.html) which makes it look [much simpler](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
 
 **protip:** if you enable `favicon` in the `[⚙️] settings` tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress
 
@@ -571,6 +588,8 @@ and there are *two* editors
 
 * you can link a particular timestamp in an audio file by adding it to the URL, such as `&20` / `&20s` / `&1m20` / `&t=1:20` after the `.../#af-c8960dab`
 
+* enabling the audio equalizer can help make gapless albums fully gapless in some browsers (chrome), so consider leaving it on with all the values at zero
+
 * get a plaintext file listing by adding `?ls=t` to a URL, or a compact colored one with `?ls=v` (for unix terminals)
 
 * if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider https://ocv.me/dev/?media-osd-bgone.ps1
@@ -607,6 +626,18 @@ using arguments or config files, or a mix of both:
   * or click the `[reload cfg]` button in the control-panel when logged in as admin 
 
 
+## ftp-server
+
+an FTP server can be started using `--ftp 3921`,  and/or `--ftps` for explicit TLS (ftpes)
+
+* based on [pyftpdlib](https://github.com/giampaolo/pyftpdlib)
+* needs a dedicated port (cannot share with the HTTP/HTTPS API)
+* uploads are not resumable -- delete and restart if necessary
+* runs in active mode by default, you probably want `--ftp-pr 12000-13000`
+  * if you enable both `ftp` and `ftps`, the port-range will be divided in half
+  * some older software (filezilla on debian-stable) cannot passive-mode with TLS
+
+
 ## file indexing
 
 file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volume flags, or a mix of both.
@@ -619,10 +650,12 @@ through arguments:
 * `-e2ts` also scans for tags in all files that don't have tags yet
 * `-e2tsr` also deletes all existing tags, doing a full reindex
 
-the same arguments can be set as volume flags, in addition to `d2d` and `d2t` for disabling:
+the same arguments can be set as volume flags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts` for disabling:
 * `-v ~/music::r:c,e2dsa,e2tsr` does a full reindex of everything on startup
 * `-v ~/music::r:c,d2d` disables **all** indexing, even if any `-e2*` are on
 * `-v ~/music::r:c,d2t` disables all `-e2t*` (tags), does not affect `-e2d*`
+* `-v ~/music::r:c,d2ds` disables on-boot scans; only index new uploads
+* `-v ~/music::r:c,d2ts` same except only affecting tags
 
 note:
 * the parser can finally handle `c,e2dsa,e2tsr` so you no longer have to `c,e2dsa:c,e2tsr`
@@ -677,6 +710,12 @@ things to note,
 * the files will be indexed after compression, so dupe-detection and file-search will not work as expected
 
 some examples,
+* `-v inc:inc:w:c,pk=xz,0`  
+  folder named inc, shared at inc, write-only for everyone, forces xz compression at level 0
+* `-v inc:inc:w:c,pk`  
+  same write-only inc, but forces gz compression (default) instead of xz
+* `-v inc:inc:w:c,gz`  
+  allows (but does not force) gz compression if client uploads to `/inc?pk` or `/inc?gz` or `/inc?gz=4`
 
 
 ## database location
@@ -721,7 +760,7 @@ see the beautiful mess of a dictionary in [mtag.py](https://github.com/9001/copy
 
 ## file parser plugins
 
-provide custom parsers to index additional tags
+provide custom parsers to index additional tags, also see [./bin/mtag/README.md](./bin/mtag/README.md)
 
 copyparty can invoke external programs to collect additional metadata for files using `mtp` (either as argument or volume flag), there is a default timeout of 30sec
 
@@ -754,6 +793,17 @@ and it will occupy the parsing threads, so fork anything expensive, or if you wa
 if this becomes popular maybe there should be a less janky way to do it actually
 
 
+## hiding from google
+
+tell search engines you dont wanna be indexed,  either using the good old [robots.txt](https://www.robotstxt.org/robotstxt.html) or through copyparty settings:
+
+* `--no-robots` adds HTTP (`X-Robots-Tag`) and HTML (`<meta>`) headers with `noindex, nofollow` globally
+* volume-flag `[...]:c,norobots` does the same thing for that single volume
+* volume-flag `[...]:c,robots` ALLOWS search-engine crawling for that volume, even if `--no-robots` is set globally
+
+also, `--force-js` disables the plain HTML folder listing, making things harder to parse for search engines
+
+
 ## complete examples
 
 * read-only music server with bpm and key scanning  
@@ -842,7 +892,7 @@ copyparty returns a truncated sha512sum of your PUT/POST as base64; you can gene
     b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|tr '+/' '-_'|head -c44;}
     b512 <movie.mkv
 
-you can provide passwords using cookie 'cppwd=hunter2', as a url query `?pw=hunter2`, or with basic-authentication (either as the username or password)
+you can provide passwords using cookie `cppwd=hunter2`, as a url query `?pw=hunter2`, or with basic-authentication (either as the username or password)
 
 
 # up2k
@@ -1037,6 +1087,10 @@ mandatory deps:
 
 install these to enable bonus features
 
+enable ftp-server:
+* for just plaintext FTP, `pyftpdlib` (is built into the SFX)
+* with TLS encryption, `pyftpdlib pyopenssl`
+
 enable music tags:
 * either `mutagen` (fast, pure-python, skips a few tags, makes copyparty GPL? idk)
 * or `ffprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users)
@@ -1063,13 +1117,7 @@ these are standalone programs and will never be imported / evaluated by copypart
 
 # sfx
 
-there are two self-contained "binaries":
-* [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) -- pure python, works everywhere, **recommended**
-* [copyparty-sfx.sh](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.sh) -- smaller, but only for linux and macos, kinda deprecated
-
-launch either of them (**use sfx.py on systemd**) and it'll unpack and run copyparty, assuming you have python installed of course
-
-pls note that `copyparty-sfx.sh` will fail if you rename `copyparty-sfx.py` to `copyparty.py` and keep it in the same folder because `sys.path` is funky
+the self-contained "binary"  [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) will unpack itself and run copyparty, assuming you have python installed of course
 
 
 ## sfx repack
@@ -1144,8 +1192,8 @@ mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/deps/
 then build the sfx using any of the following examples:
 
 ```sh
-./scripts/make-sfx.sh  # both python and sh editions
-./scripts/make-sfx.sh no-sh gz  # just python with gzip
+./scripts/make-sfx.sh           # regular edition
+./scripts/make-sfx.sh gz no-cm  # gzip-compressed + no fancy markdown editor
 ```
 
 

bin/README.md

@@ -5,6 +5,11 @@
 * if something breaks just restart it
 
 
+# [`partyjournal.py`](partyjournal.py)
+produces a chronological list of all uploads by collecting info from up2k databases and the filesystem
+* outputs a standalone html file
+* optional mapping from IP-addresses to nicknames
+
 
 # [`copyparty-fuse.py`](copyparty-fuse.py)
 * mount a copyparty server as a local filesystem (read-only)

bin/copyparty-fuseb.py

@@ -11,14 +11,18 @@ import re
 import os
 import sys
 import time
+import json
 import stat
 import errno
 import struct
+import codecs
+import platform
 import threading
 import http.client  # py2: httplib
 import urllib.parse
 from datetime import datetime
 from urllib.parse import quote_from_bytes as quote
+from urllib.parse import unquote_to_bytes as unquote
 
 try:
     import fuse
@@ -38,7 +42,7 @@ except:
 mount a copyparty server (local or remote) as a filesystem
 
 usage:
-  python ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,url=http://192.168.1.69:3923 /mnt/nas
+  python ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,pw=wark,url=http://192.168.1.69:3923 /mnt/nas
 
 dependencies:
   sudo apk add fuse-dev python3-dev
@@ -50,6 +54,10 @@ fork of copyparty-fuse.py based on fuse-python which
 """
 
 
+WINDOWS = sys.platform == "win32"
+MACOS = platform.system() == "Darwin"
+
+
 def threadless_log(msg):
     print(msg + "\n", end="")
 
@@ -93,6 +101,41 @@ def html_dec(txt):
     )
 
 
+def register_wtf8():
+    def wtf8_enc(text):
+        return str(text).encode("utf-8", "surrogateescape"), len(text)
+
+    def wtf8_dec(binary):
+        return bytes(binary).decode("utf-8", "surrogateescape"), len(binary)
+
+    def wtf8_search(encoding_name):
+        return codecs.CodecInfo(wtf8_enc, wtf8_dec, name="wtf-8")
+
+    codecs.register(wtf8_search)
+
+
+bad_good = {}
+good_bad = {}
+
+
+def enwin(txt):
+    return "".join([bad_good.get(x, x) for x in txt])
+
+    for bad, good in bad_good.items():
+        txt = txt.replace(bad, good)
+
+    return txt
+
+
+def dewin(txt):
+    return "".join([good_bad.get(x, x) for x in txt])
+
+    for bad, good in bad_good.items():
+        txt = txt.replace(good, bad)
+
+    return txt
+
+
 class CacheNode(object):
     def __init__(self, tag, data):
         self.tag = tag
@@ -115,8 +158,9 @@ class Stat(fuse.Stat):
 
 
 class Gateway(object):
-    def __init__(self, base_url):
+    def __init__(self, base_url, pw):
         self.base_url = base_url
+        self.pw = pw
 
         ui = urllib.parse.urlparse(base_url)
         self.web_root = ui.path.strip("/")
@@ -135,8 +179,7 @@ class Gateway(object):
         self.conns = {}
 
     def quotep(self, path):
-        # TODO: mojibake support
-        path = path.encode("utf-8", "ignore")
+        path = path.encode("wtf-8")
         return quote(path, safe="/")
 
     def getconn(self, tid=None):
@@ -159,20 +202,29 @@ class Gateway(object):
         except:
             pass
 
-    def sendreq(self, *args, **kwargs):
+    def sendreq(self, *args, **ka):
         tid = get_tid()
+        if self.pw:
+            ck = "cppwd=" + self.pw
+            try:
+                ka["headers"]["Cookie"] = ck
+            except:
+                ka["headers"] = {"Cookie": ck}
         try:
             c = self.getconn(tid)
-            c.request(*list(args), **kwargs)
+            c.request(*list(args), **ka)
             return c.getresponse()
         except:
             self.closeconn(tid)
             c = self.getconn(tid)
-            c.request(*list(args), **kwargs)
+            c.request(*list(args), **ka)
             return c.getresponse()
 
     def listdir(self, path):
-        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
+        if bad_good:
+            path = dewin(path)
+
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots&ls"
         r = self.sendreq("GET", web_path)
         if r.status != 200:
             self.closeconn()
@@ -182,9 +234,12 @@ class Gateway(object):
                 )
             )
 
-        return self.parse_html(r)
+        return self.parse_jls(r)
 
     def download_file_range(self, path, ofs1, ofs2):
+        if bad_good:
+            path = dewin(path)
+
         web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?raw"
         hdr_range = "bytes={}-{}".format(ofs1, ofs2 - 1)
         log("downloading {}".format(hdr_range))
@@ -200,40 +255,27 @@ class Gateway(object):
 
         return r.read()
 
-    def parse_html(self, datasrc):
-        ret = []
-        remainder = b""
-        ptn = re.compile(
-            r"^<tr><td>(-|DIR)</td><td><a [^>]+>([^<]+)</a></td><td>([^<]+)</td><td>([^<]+)</td></tr>$"
-        )
-
+    def parse_jls(self, datasrc):
+        rsp = b""
         while True:
-            buf = remainder + datasrc.read(4096)
-            # print('[{}]'.format(buf.decode('utf-8')))
+            buf = datasrc.read(1024 * 32)
             if not buf:
                 break
 
-            remainder = b""
-            endpos = buf.rfind(b"\n")
-            if endpos >= 0:
-                remainder = buf[endpos + 1 :]
-                buf = buf[:endpos]
+            rsp += buf
 
-            lines = buf.decode("utf-8").split("\n")
-            for line in lines:
-                m = ptn.match(line)
-                if not m:
-                    # print(line)
-                    continue
+        rsp = json.loads(rsp.decode("utf-8"))
+        ret = []
+        for statfun, nodes in [
+            [self.stat_dir, rsp["dirs"]],
+            [self.stat_file, rsp["files"]],
+        ]:
+            for n in nodes:
+                fname = unquote(n["href"].split("?")[0]).rstrip(b"/").decode("wtf-8")
+                if bad_good:
+                    fname = enwin(fname)
 
-                ftype, fname, fsize, fdate = m.groups()
-                fname = html_dec(fname)
-                ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
-                sz = int(fsize)
-                if ftype == "-":
-                    ret.append([fname, self.stat_file(ts, sz), 0])
-                else:
-                    ret.append([fname, self.stat_dir(ts, sz), 0])
+                ret.append([fname, statfun(n["ts"], n["sz"]), 0])
 
         return ret
 
@@ -262,6 +304,7 @@ class CPPF(Fuse):
         Fuse.__init__(self, *args, **kwargs)
 
         self.url = None
+        self.pw = None
 
         self.dircache = []
         self.dircache_mtx = threading.Lock()
@@ -271,7 +314,7 @@ class CPPF(Fuse):
 
     def init2(self):
         # TODO figure out how python-fuse wanted this to go
-        self.gw = Gateway(self.url)  # .decode('utf-8'))
+        self.gw = Gateway(self.url, self.pw)  # .decode('utf-8'))
         info("up")
 
     def clean_dircache(self):
@@ -536,6 +579,8 @@ class CPPF(Fuse):
 
     def getattr(self, path):
         log("getattr [{}]".format(path))
+        if WINDOWS:
+            path = enwin(path)  # windows occasionally decodes f0xx to xx
 
         path = path.strip("/")
         try:
@@ -568,9 +613,25 @@ class CPPF(Fuse):
 
 def main():
     time.strptime("19970815", "%Y%m%d")  # python#7980
+    register_wtf8()
+    if WINDOWS:
+        os.system("rem")
+
+        for ch in '<>:"\\|?*':
+            # microsoft maps illegal characters to f0xx
+            # (e000 to f8ff is basic-plane private-use)
+            bad_good[ch] = chr(ord(ch) + 0xF000)
+
+        for n in range(0, 0x100):
+            # map surrogateescape to another private-use area
+            bad_good[chr(n + 0xDC00)] = chr(n + 0xF100)
+
+        for k, v in bad_good.items():
+            good_bad[v] = k
 
     server = CPPF()
     server.parser.add_option(mountopt="url", metavar="BASE_URL", default=None)
+    server.parser.add_option(mountopt="pw", metavar="PASSWORD", default=None)
     server.parse(values=server, errex=1)
     if not server.url or not str(server.url).startswith("http"):
         print("\nerror:")
@@ -578,7 +639,7 @@ def main():
         print("  need argument: mount-path")
         print("example:")
         print(
-            "  ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,url=http://192.168.1.69:3923 /mnt/nas"
+            "  ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,pw=wark,url=http://192.168.1.69:3923 /mnt/nas"
         )
         sys.exit(1)
 

bin/mtag/README.md

@@ -6,9 +6,13 @@ some of these rely on libraries which are not MIT-compatible
 
 * [audio-bpm.py](./audio-bpm.py) detects the BPM of music using the BeatRoot Vamp Plugin; imports GPL2
 * [audio-key.py](./audio-key.py) detects the melodic key of music using the Mixxx fork of keyfinder; imports GPL3
-* [media-hash.py](./media-hash.py) generates checksums for audio and video streams; uses FFmpeg (LGPL or GPL)
 
-these do not have any problematic dependencies:
+these invoke standalone programs which are GPL or similar, so is legally fine for most purposes:
+
+* [media-hash.py](./media-hash.py) generates checksums for audio and video streams; uses FFmpeg (LGPL or GPL)
+* [image-noexif.py](./image-noexif.py) removes exif tags from images; uses exiftool (GPLv1 or artistic-license)
+
+these do not have any problematic dependencies at all:
 
 * [cksum.py](./cksum.py) computes various checksums
 * [exe.py](./exe.py) grabs metadata from .exe and .dll files (example for retrieving multiple tags with one parser)

bin/mtag/audio-bpm.py

@@ -19,18 +19,18 @@ dep: ffmpeg
 def det(tf):
     # fmt: off
     sp.check_call([
-        "ffmpeg",
-        "-nostdin",
-        "-hide_banner",
-        "-v", "fatal",
-        "-ss", "13",
-        "-y", "-i", fsenc(sys.argv[1]),
-        "-map", "0:a:0",
-        "-ac", "1",
-        "-ar", "22050",
-        "-t", "300",
-        "-f", "f32le",
-        tf
+        b"ffmpeg",
+        b"-nostdin",
+        b"-hide_banner",
+        b"-v", b"fatal",
+        b"-ss", b"13",
+        b"-y", b"-i", fsenc(sys.argv[1]),
+        b"-map", b"0:a:0",
+        b"-ac", b"1",
+        b"-ar", b"22050",
+        b"-t", b"300",
+        b"-f", b"f32le",
+        fsenc(tf)
     ])
     # fmt: on
 

bin/mtag/audio-key.py

@@ -23,15 +23,15 @@ dep: ffmpeg
 def det(tf):
     # fmt: off
     sp.check_call([
-        "ffmpeg",
-        "-nostdin",
-        "-hide_banner",
-        "-v", "fatal",
-        "-y", "-i", fsenc(sys.argv[1]),
-        "-map", "0:a:0",
-        "-t", "300",
-        "-sample_fmt", "s16",
-        tf
+        b"ffmpeg",
+        b"-nostdin",
+        b"-hide_banner",
+        b"-v", b"fatal",
+        b"-y", b"-i", fsenc(sys.argv[1]),
+        b"-map", b"0:a:0",
+        b"-t", b"300",
+        b"-sample_fmt", b"s16",
+        fsenc(tf)
     ])
     # fmt: on
 

bin/mtag/image-noexif.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+
+"""
+remove exif tags from uploaded images
+
+dependencies:
+  exiftool
+
+about:
+  creates a "noexif" subfolder and puts exif-stripped copies of each image there,
+  the reason for the subfolder is to avoid issues with the up2k.db / deduplication:
+
+  if the original image is modified in-place, then copyparty will keep the original
+  hash in up2k.db for a while (until the next volume rescan), so if the image is
+  reuploaded after a rescan then the upload will be renamed and kept as a dupe
+
+  alternatively you could switch the logic around, making a copy of the original
+  image into a subfolder named "exif" and modify the original in-place, but then
+  up2k.db will be out of sync until the next rescan, so any additional uploads
+  of the same image will get symlinked (deduplicated) to the modified copy
+  instead of the original in "exif"
+
+  or maybe delete the original image after processing, that would kinda work too
+
+example copyparty config to use this:
+  -v/mnt/nas/pics:pics:rwmd,ed:c,e2ts,mte=+noexif:c,mtp=noexif=ejpg,ejpeg,ad,bin/mtag/image-noexif.py
+
+explained:
+  for realpath /mnt/nas/pics (served at /pics) with read-write-modify-delete for ed,
+  enable file analysis on upload (e2ts),
+  append "noexif" to the list of known tags (mtp),
+  and use mtp plugin "bin/mtag/image-noexif.py" to provide that tag,
+  do this on all uploads with the file extension "jpg" or "jpeg",
+  ad = parse file regardless if FFmpeg thinks it is audio or not
+
+PS: this requires e2ts to be functional,
+  meaning you need to do at least one of these:
+   * apt install ffmpeg
+   * pip3 install mutagen
+  and your python must have sqlite3 support compiled in
+"""
+
+
+import os
+import sys
+import time
+import filecmp
+import subprocess as sp
+
+try:
+    from copyparty.util import fsenc
+except:
+
+    def fsenc(p):
+        return p.encode("utf-8")
+
+
+def main():
+    cwd, fn = os.path.split(sys.argv[1])
+    if os.path.basename(cwd) == "noexif":
+        return
+
+    os.chdir(cwd)
+    f1 = fsenc(fn)
+    f2 = os.path.join(b"noexif", f1)
+    cmd = [
+        b"exiftool",
+        b"-exif:all=",
+        b"-iptc:all=",
+        b"-xmp:all=",
+        b"-P",
+        b"-o",
+        b"noexif/",
+        b"--",
+        f1,
+    ]
+    sp.check_output(cmd)
+    if not os.path.exists(f2):
+        print("failed")
+        return
+
+    if filecmp.cmp(f1, f2, shallow=False):
+        print("clean")
+    else:
+        print("exif")
+
+    # lastmod = os.path.getmtime(f1)
+    # times = (int(time.time()), int(lastmod))
+    # os.utime(f2, times)
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/install-deps.sh

@@ -4,8 +4,8 @@ set -e
 
 # install dependencies for audio-*.py
 #
-# linux/alpine: requires {python3,ffmpeg,fftw}-dev py3-{wheel,pip} py3-numpy{,-dev} vamp-sdk-dev patchelf cmake
-# linux/debian: requires libav{codec,device,filter,format,resample,util}-dev {libfftw3,python3}-dev python3-{numpy,pip} vamp-{plugin-sdk,examples} patchelf cmake
+# linux/alpine: requires gcc g++ make cmake patchelf {python3,ffmpeg,fftw,libsndfile}-dev py3-{wheel,pip} py3-numpy{,-dev}
+# linux/debian: requires libav{codec,device,filter,format,resample,util}-dev {libfftw3,python3,libsndfile1}-dev python3-{numpy,pip} vamp-{plugin-sdk,examples} patchelf cmake
 # win64: requires msys2-mingw64 environment
 # macos: requires macports
 #
@@ -101,8 +101,11 @@ export -f dl_files
 
 
 github_tarball() {
+	rm -rf g
+	mkdir g
+	cd g
 	dl_text "$1" |
-	tee json |
+	tee ../json |
 	(
 		# prefer jq if available
 		jq -r '.tarball_url' ||
@@ -111,8 +114,11 @@ github_tarball() {
 		awk -F\" '/"tarball_url": "/ {print$4}'
 	) |
 	tee /dev/stderr |
+	head -n 1 |
 	tr -d '\r' | tr '\n' '\0' |
 	xargs -0 bash -c 'dl_files "$@"' _
+	mv * ../tgz
+	cd ..
 }
 
 
@@ -127,6 +133,7 @@ gitlab_tarball() {
 		tr \" '\n' | grep -E '\.tar\.gz$' | head -n 1
 	) |
 	tee /dev/stderr |
+	head -n 1 |
 	tr -d '\r' | tr '\n' '\0' |
 	tee links |
 	xargs -0 bash -c 'dl_files "$@"' _
@@ -138,10 +145,17 @@ install_keyfinder() {
 	#   use msys2 in mingw-w64 mode
 	#   pacman -S --needed mingw-w64-x86_64-{ffmpeg,python}
 	
-	github_tarball https://api.github.com/repos/mixxxdj/libkeyfinder/releases/latest
+	[ -e $HOME/pe/keyfinder ] && {
+		echo found a keyfinder build in ~/pe, skipping
+		return
+	}
 
-	tar -xf mixxxdj-libkeyfinder-*
-	rm -- *.tar.gz
+	cd "$td"
+	github_tarball https://api.github.com/repos/mixxxdj/libkeyfinder/releases/latest
+	ls -al
+
+	tar -xf tgz
+	rm tgz
 	cd mixxxdj-libkeyfinder*
 	
 	h="$HOME"
@@ -208,6 +222,22 @@ install_vamp() {
 	
 	$pybin -m pip install --user vamp
 
+	cd "$td"
+	echo '#include <vamp-sdk/Plugin.h>' | gcc -x c -c -o /dev/null - || [ -e ~/pe/vamp-sdk ] || {
+		printf '\033[33mcould not find the vamp-sdk, building from source\033[0m\n'
+		(dl_files yolo https://code.soundsoftware.ac.uk/attachments/download/2588/vamp-plugin-sdk-2.9.0.tar.gz)
+		sha512sum -c <(
+			echo "7ef7f837d19a08048b059e0da408373a7964ced452b290fae40b85d6d70ca9000bcfb3302cd0b4dc76cf2a848528456f78c1ce1ee0c402228d812bd347b6983b  -"
+		) <vamp-plugin-sdk-2.9.0.tar.gz
+		tar -xf vamp-plugin-sdk-2.9.0.tar.gz
+		rm -- *.tar.gz
+		ls -al
+		cd vamp-plugin-sdk-*
+		./configure --prefix=$HOME/pe/vamp-sdk
+		make -j1 install
+	}
+
+	cd "$td"
 	have_beatroot || {
 		printf '\033[33mcould not find the vamp beatroot plugin, building from source\033[0m\n'
 		(dl_files yolo https://code.soundsoftware.ac.uk/attachments/download/885/beatroot-vamp-v1.0.tar.gz)
@@ -215,8 +245,11 @@ install_vamp() {
 			echo "1f444d1d58ccf565c0adfe99f1a1aa62789e19f5071e46857e2adfbc9d453037bc1c4dcb039b02c16240e9b97f444aaff3afb625c86aa2470233e711f55b6874  -"
 		) <beatroot-vamp-v1.0.tar.gz
 		tar -xf beatroot-vamp-v1.0.tar.gz 
+		rm -- *.tar.gz
 		cd beatroot-vamp-v1.0
-		make -f Makefile.linux -j4
+		[ -e ~/pe/vamp-sdk ] &&
+			sed -ri 's`^(CFLAGS :=.*)`\1 -I'$HOME'/pe/vamp-sdk/include`' Makefile.linux
+		make -f Makefile.linux -j4 LDFLAGS=-L$HOME/pe/vamp-sdk/lib
 		# /home/ed/vamp /home/ed/.vamp /usr/local/lib/vamp
 		mkdir ~/vamp
 		cp -pv beatroot-vamp.* ~/vamp/
@@ -230,6 +263,7 @@ install_vamp() {
 
 # not in use because it kinda segfaults, also no windows support
 install_soundtouch() {
+	cd "$td"
 	gitlab_tarball https://gitlab.com/api/v4/projects/soundtouch%2Fsoundtouch/releases
 	
 	tar -xvf soundtouch-*

bin/mtag/media-hash.py

@@ -13,7 +13,7 @@ try:
 except:
 
     def fsenc(p):
-        return p
+        return p.encode("utf-8")
 
 
 """
@@ -24,13 +24,13 @@ dep: ffmpeg
 def det():
     # fmt: off
     cmd = [
-        "ffmpeg",
-        "-nostdin",
-        "-hide_banner",
-        "-v", "fatal",
-        "-i", fsenc(sys.argv[1]),
-        "-f", "framemd5",
-        "-"
+        b"ffmpeg",
+        b"-nostdin",
+        b"-hide_banner",
+        b"-v", b"fatal",
+        b"-i", fsenc(sys.argv[1]),
+        b"-f", b"framemd5",
+        b"-"
     ]
     # fmt: on
 

bin/mtag/res/twitter-unmute.user.js

@@ -0,0 +1,21 @@
+// ==UserScript==
+// @name         twitter-unmute
+// @namespace    http://ocv.me/
+// @version      0.1
+// @description  memes
+// @author       ed <irc.rizon.net>
+// @match        https://twitter.com/*
+// @icon         https://www.google.com/s2/favicons?domain=twitter.com
+// @grant        GM_addStyle
+// ==/UserScript==
+
+function grunnur() {
+    setInterval(function () {
+        //document.querySelector('div[aria-label="Unmute"]').click();
+        document.querySelector('video').muted = false;
+    }, 200);
+}
+
+var scr = document.createElement('script');
+scr.textContent = '(' + grunnur.toString() + ')();';
+(document.head || document.getElementsByTagName('head')[0]).appendChild(scr);

bin/mtag/very-bad-idea.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+
+"""
+use copyparty as a chromecast replacement:
+  * post a URL and it will open in the default browser
+  * upload a file and it will open in the default application
+  * the `key` command simulates keyboard input
+  * the `x` command executes other xdotool commands
+  * the `c` command executes arbitrary unix commands
+
+the android app makes it a breeze to post pics and links:
+  https://github.com/9001/party-up/releases
+  (iOS devices have to rely on the web-UI)
+
+goes without saying, but this is HELLA DANGEROUS,
+  GIVES RCE TO ANYONE WHO HAVE UPLOAD PERMISSIONS
+
+example copyparty config to use this:
+  --urlform save,get -v.::w:c,e2d,e2t,mte=+a1:c,mtp=a1=ad,bin/mtag/very-bad-idea.py
+
+recommended deps:
+  apt install xdotool libnotify-bin
+  https://github.com/9001/copyparty/blob/hovudstraum/contrib/plugins/meadup.js
+
+and you probably want `twitter-unmute.user.js` from the res folder
+
+
+-----------------------------------------------------------------------
+-- startup script:
+-----------------------------------------------------------------------
+
+#!/bin/bash
+set -e
+
+# create qr code
+ip=$(ip r | awk '/^default/{print$(NF-2)}'); echo http://$ip:3923/ | qrencode -o - -s 4 >/dev/shm/cpp-qr.png
+/usr/bin/feh -x /dev/shm/cpp-qr.png &
+
+# reposition and make topmost (with janky raspbian support)
+( sleep 0.5
+xdotool search --name cpp-qr.png windowactivate --sync windowmove 1780 0
+wmctrl -r :ACTIVE: -b toggle,above || true
+
+ps aux | grep -E 'sleep[ ]7\.27' ||
+while true; do
+  w=$(xdotool getactivewindow)
+  xdotool search --name cpp-qr.png windowactivate windowraise windowfocus
+  xdotool windowactivate $w
+  xdotool windowfocus $w
+  sleep 7.27 || break
+done &
+xeyes  # distraction window to prevent ^w from closing the qr-code
+) &
+
+# bail if copyparty is already running
+ps aux | grep -E '[3] copy[p]arty' && exit 0
+
+# dumb chrome wrapper to allow autoplay
+cat >/usr/local/bin/chromium-browser <<'EOF'
+#!/bin/bash
+set -e
+/usr/bin/chromium-browser --autoplay-policy=no-user-gesture-required "$@"
+EOF
+chmod 755 /usr/local/bin/chromium-browser
+
+# start the server  (note: replace `-v.::rw:` with `-v.::r:` to disallow retrieving uploaded stuff)
+cd ~/Downloads; python3 copyparty-sfx.py --urlform save,get -v.::rw:c,e2d,e2t,mte=+a1:c,mtp=a1=ad,very-bad-idea.py
+
+"""
+
+
+import os
+import sys
+import time
+import subprocess as sp
+from urllib.parse import unquote_to_bytes as unquote
+
+
+def main():
+    fp = os.path.abspath(sys.argv[1])
+    with open(fp, "rb") as f:
+        txt = f.read(4096)
+
+    if txt.startswith(b"msg="):
+        open_post(txt)
+    else:
+        open_url(fp)
+
+
+def open_post(txt):
+    txt = unquote(txt.replace(b"+", b" ")).decode("utf-8")[4:]
+    try:
+        k, v = txt.split(" ", 1)
+    except:
+        open_url(txt)
+
+    if k == "key":
+        sp.call(["xdotool", "key"] + v.split(" "))
+    elif k == "x":
+        sp.call(["xdotool"] + v.split(" "))
+    elif k == "c":
+        env = os.environ.copy()
+        while " " in v:
+            v1, v2 = v.split(" ", 1)
+            if "=" not in v1:
+                break
+
+            ek, ev = v1.split("=", 1)
+            env[ek] = ev
+            v = v2
+
+        sp.call(v.split(" "), env=env)
+    else:
+        open_url(txt)
+
+
+def open_url(txt):
+    ext = txt.rsplit(".")[-1].lower()
+    sp.call(["notify-send", "--", txt])
+    if ext not in ["jpg", "jpeg", "png", "gif", "webp"]:
+        # sp.call(["wmctrl", "-c", ":ACTIVE:"])  # closes the active window correctly
+        sp.call(["killall", "vlc"])
+        sp.call(["killall", "mpv"])
+        sp.call(["killall", "feh"])
+        time.sleep(0.5)
+        for _ in range(20):
+            sp.call(["xdotool", "key", "ctrl+w"])  # closes the open tab correctly
+    # else:
+    #    sp.call(["xdotool", "getactivewindow", "windowminimize"])  # minimizes the focused windo
+
+    # close any error messages:
+    sp.call(["xdotool", "search", "--name", "Error", "windowclose"])
+    # sp.call(["xdotool", "key", "ctrl+alt+d"])  # doesnt work at all
+    # sp.call(["xdotool", "keydown", "--delay", "100", "ctrl+alt+d"])
+    # sp.call(["xdotool", "keyup", "ctrl+alt+d"])
+    sp.call(["xdg-open", txt])
+
+
+main()

bin/partyjournal.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+
+"""
+partyjournal.py: chronological history of uploads
+2021-12-31, v0.1, ed <irc.rizon.net>, MIT-Licensed
+https://github.com/9001/copyparty/blob/hovudstraum/bin/partyjournal.py
+
+produces a chronological list of all uploads,
+by collecting info from up2k databases and the filesystem
+
+specify subnet `192.168.1.*` with argument `.=192.168.1.`,
+affecting all successive mappings
+
+usage:
+  ./partyjournal.py > partyjournal.html .=192.168.1. cart=125 steen=114 steen=131 sleepy=121 fscarlet=144 ed=101 ed=123
+
+"""
+
+import sys
+import base64
+import sqlite3
+import argparse
+from datetime import datetime
+from urllib.parse import quote_from_bytes as quote
+from urllib.parse import unquote_to_bytes as unquote
+
+
+FS_ENCODING = sys.getfilesystemencoding()
+
+
+class APF(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
+    pass
+
+
+##
+## snibbed from copyparty
+
+
+def s3dec(v):
+    if not v.startswith("//"):
+        return v
+
+    v = base64.urlsafe_b64decode(v.encode("ascii")[2:])
+    return v.decode(FS_ENCODING, "replace")
+
+
+def quotep(txt):
+    btxt = txt.encode("utf-8", "replace")
+    quot1 = quote(btxt, safe=b"/")
+    quot1 = quot1.encode("ascii")
+    quot2 = quot1.replace(b" ", b"+")
+    return quot2.decode("utf-8", "replace")
+
+
+def html_escape(s, quote=False, crlf=False):
+    """html.escape but also newlines"""
+    s = s.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
+    if quote:
+        s = s.replace('"', "&quot;").replace("'", "&#x27;")
+    if crlf:
+        s = s.replace("\r", "&#13;").replace("\n", "&#10;")
+
+    return s
+
+
+## end snibs
+##
+
+
+def main():
+    ap = argparse.ArgumentParser(formatter_class=APF)
+    ap.add_argument("who", nargs="*")
+    ar = ap.parse_args()
+
+    imap = {}
+    subnet = ""
+    for v in ar.who:
+        if "=" not in v:
+            raise Exception("bad who: " + v)
+
+        k, v = v.split("=")
+        if k == ".":
+            subnet = v
+            continue
+
+        imap["{}{}".format(subnet, v)] = k
+
+    print(repr(imap), file=sys.stderr)
+
+    print(
+        """\
+<!DOCTYPE html>
+<html lang="en">
+<head><meta charset="utf-8"><style>
+
+html, body {
+    color: #ccc;
+    background: #222;
+    font-family: sans-serif;
+}
+a {
+    color: #fc5;
+}
+td, th {
+    padding: .2em .5em;
+    border: 1px solid #999;
+    border-width: 0 1px 1px 0;
+    white-space: nowrap;
+}
+td:nth-child(1),
+td:nth-child(2),
+td:nth-child(3) {
+    font-family: monospace, monospace;
+    text-align: right;
+}
+tr:first-child {
+    position: sticky;
+    top: -1px;
+}
+th {
+    background: #222;
+    text-align: left;
+}
+
+</style></head><body><table><tr>
+    <th>wark</th>
+    <th>time</th>
+    <th>size</th>
+    <th>who</th>
+    <th>link</th>
+</tr>"""
+    )
+
+    db_path = ".hist/up2k.db"
+    conn = sqlite3.connect(db_path)
+    q = r"pragma table_info(up)"
+    inf = conn.execute(q).fetchall()
+    cols = [x[1] for x in inf]
+    print("<!-- " + str(cols) + " -->")
+    # ['w', 'mt', 'sz', 'rd', 'fn', 'ip', 'at']
+
+    q = r"select * from up order by case when at > 0 then at else mt end"
+    for w, mt, sz, rd, fn, ip, at in conn.execute(q):
+        link = "/".join([s3dec(x) for x in [rd, fn] if x])
+        if fn.startswith("put-") and sz < 4096:
+            try:
+                with open(link, "rb") as f:
+                    txt = f.read().decode("utf-8", "replace")
+            except:
+                continue
+
+            if txt.startswith("msg="):
+                txt = txt.encode("utf-8", "replace")
+                txt = unquote(txt.replace(b"+", b" "))
+                link = txt.decode("utf-8")[4:]
+
+        sz = "{:,}".format(sz)
+        v = [
+            w[:16],
+            datetime.utcfromtimestamp(at if at > 0 else mt).strftime(
+                "%Y-%m-%d %H:%M:%S"
+            ),
+            sz,
+            imap.get(ip, ip),
+        ]
+
+        row = "<tr>\n  "
+        row += "\n  ".join(["<td>{}</th>".format(x) for x in v])
+        row += '\n  <td><a href="{}">{}</a></td>'.format(link, html_escape(link))
+        row += "\n</tr>"
+        print(row)
+
+    print("</table></body></html>")
+
+
+if __name__ == "__main__":
+    main()

bin/prisonparty.sh

@@ -11,10 +11,16 @@ sysdirs=( /bin /lib /lib32 /lib64 /sbin /usr )
 help() { cat <<'EOF'
 
 usage:
-  ./prisonparty.sh <ROOTDIR> <UID> <GID> [VOLDIR [VOLDIR...]] -- copyparty-sfx.py [...]"
+  ./prisonparty.sh <ROOTDIR> <UID> <GID> [VOLDIR [VOLDIR...]] -- python3 copyparty-sfx.py [...]"
 
 example:
-  ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- copyparty-sfx.py -v /mnt/nas/music::rwmd"
+  ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- python3 copyparty-sfx.py -v /mnt/nas/music::rwmd"
+
+example for running straight from source (instead of using an sfx):
+  PYTHONPATH=$PWD ./prisonparty.sh /var/lib/copyparty-jail 1000 1000 /mnt/nas/music -- python3 -um copyparty -v /mnt/nas/music::rwmd"
+
+note that if you have python modules installed as --user (such as bpm/key detectors),
+  you should add /home/foo/.local as a VOLDIR
 
 EOF
 exit 1
@@ -35,10 +41,20 @@ while true; do
 	vols+=( "$(realpath "$v")" )
 done
 pybin="$1"; shift
-pybin="$(realpath "$pybin")"
+pybin="$(command -v "$pybin")"
+pyarg=
+while true; do
+	v="$1"
+	[ "${v:0:1}" = - ] || break
+	pyarg="$pyarg $v"
+	shift
+done
 cpp="$1"; shift
-cpp="$(realpath "$cpp")"
-cppdir="$(dirname "$cpp")"
+[ -d "$cpp" ] && cppdir="$PWD" || {
+	# sfx, not module
+	cpp="$(realpath "$cpp")"
+	cppdir="$(dirname "$cpp")"
+}
 trap - EXIT
 
 
@@ -60,11 +76,10 @@ echo
 
 # remove any trailing slashes
 jail="${jail%/}"
-cppdir="${cppdir%/}"
 
 
 # bind-mount system directories and volumes
-printf '%s\n' "${sysdirs[@]}" "${vols[@]}" | LC_ALL=C sort |
+printf '%s\n' "${sysdirs[@]}" "${vols[@]}" | sed -r 's`/$``' | LC_ALL=C sort | uniq |
 while IFS= read -r v; do
 	[ -e "$v" ] || {
 		# printf '\033[1;31mfolder does not exist:\033[0m %s\n' "/$v"
@@ -72,6 +87,7 @@ while IFS= read -r v; do
 	}
 	i1=$(stat -c%D.%i "$v"      2>/dev/null || echo a)
 	i2=$(stat -c%D.%i "$jail$v" 2>/dev/null || echo b)
+	# echo "v [$v] i1 [$i1] i2 [$i2]"
 	[ $i1 = $i2 ] && continue
 	
 	mkdir -p "$jail$v"
@@ -79,21 +95,34 @@ while IFS= read -r v; do
 done
 
 
+cln() {
+	rv=$?
+	# cleanup if not in use
+	lsof "$jail" | grep -qF "$jail" &&
+		echo "chroot is in use, will not cleanup" ||
+	{
+		mount | grep -F " on $jail" |
+		awk '{sub(/ type .*/,"");sub(/.* on /,"");print}' |
+		LC_ALL=C sort -r  | tee /dev/stderr | tr '\n' '\0' | xargs -r0 umount
+	}
+	exit $rv
+}
+trap cln EXIT
+
+
 # create a tmp
 mkdir -p "$jail/tmp"
 chmod 777 "$jail/tmp"
 
 
 # run copyparty
-/sbin/chroot --userspec=$uid:$gid "$jail" "$pybin" "$cpp" "$@" && rv=0 || rv=$?
-
-
-# cleanup if not in use
-lsof "$jail" | grep -qF "$jail" &&
-	echo "chroot is in use, will not cleanup" ||
-{
-	mount | grep -qF " on $jail" |
-	awk '{sub(/ type .*/,"");sub(/.* on /,"");print}' |
-	LC_ALL=C sort -r  | tee /dev/stderr | tr '\n' '\0' | xargs -r0 umount
-}
-exit $rv
+export HOME=$(getent passwd $uid | cut -d: -f6)
+export USER=$(getent passwd $uid | cut -d: -f1)
+export LOGNAME="$USER"
+#echo "pybin [$pybin]"
+#echo "pyarg [$pyarg]"
+#echo "cpp [$cpp]"
+chroot --userspec=$uid:$gid "$jail" "$pybin" $pyarg "$cpp" "$@" &
+p=$!
+trap 'kill $p' INT TERM
+wait

bin/up2k.py

@@ -3,7 +3,7 @@ from __future__ import print_function, unicode_literals
 
 """
 up2k.py: upload to copyparty
-2021-11-16, v0.12, ed <irc.rizon.net>, MIT-Licensed
+2021-11-28, v0.13, ed <irc.rizon.net>, MIT-Licensed
 https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py
 
 - dependencies: requests
@@ -390,7 +390,7 @@ def handshake(req_ses, url, file, pw, search):
             r = req_ses.post(url, headers=headers, json=req)
             break
         except:
-            eprint("handshake failed, retry...\n")
+            eprint("handshake failed, retrying: {0}\n".format(file.name))
             time.sleep(1)
 
     try:
@@ -470,11 +470,11 @@ class Ctl(object):
             nbytes += inf.st_size
 
         if err:
-            eprint("\n# failed to access {} paths:\n".format(len(err)))
+            eprint("\n# failed to access {0} paths:\n".format(len(err)))
             for x in err:
                 eprint(x.decode("utf-8", "replace") + "\n")
 
-            eprint("^ failed to access those {} paths ^\n\n".format(len(err)))
+            eprint("^ failed to access those {0} paths ^\n\n".format(len(err)))
             if not ar.ok:
                 eprint("aborting because --ok is not set\n")
                 return
@@ -505,7 +505,7 @@ class Ctl(object):
             print("{0} {1}\n  hash...".format(self.nfiles - nf, upath))
             get_hashlist(file, None)
 
-            burl = self.ar.url[:8] + self.ar.url[8:].split("/")[0] + "/"
+            burl = self.ar.url[:12] + self.ar.url[8:].split("/")[0] + "/"
             while True:
                 print("  hs...")
                 hs = handshake(req_ses, self.ar.url, file, self.ar.a, search)
@@ -773,7 +773,7 @@ class Ctl(object):
             try:
                 upload(req_ses, file, cid, self.ar.a)
             except:
-                eprint("upload failed, retry...\n")
+                eprint("upload failed, retrying: {0} #{1}\n".format(file.name, cid[:8]))
                 pass  # handshake will fix it
 
             with self.mutex:

contrib/README.md

@@ -1,3 +1,6 @@
+### [`plugins/`](plugins/)
+* example extensions
+
 ### [`copyparty.bat`](copyparty.bat)
 * launches copyparty with no arguments (anon read+write within same folder)
 * intended for windows machines with no python.exe in PATH

contrib/nginx/copyparty.conf

@@ -13,7 +13,7 @@
 
 upstream cpp {
 	server 127.0.0.1:3923;
-	keepalive 120;
+	keepalive 1;
 }
 server {
 	listen 443 ssl;

contrib/plugins/README.md

@@ -0,0 +1,25 @@
+# example resource files
+
+can be provided to copyparty to tweak things
+
+
+
+## example `.epilogue.html`
+save one of these as `.epilogue.html` inside a folder to customize it:
+
+* [`minimal-up2k.html`](minimal-up2k.html) will [simplify the upload ui](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
+
+
+
+## example browser-css
+point `--css-browser` to one of these by URL:
+
+* [`browser.css`](browser.css) changes the background
+* [`browser-icons.css`](browser-icons.css) adds filetype icons
+
+
+
+## meadup.js
+
+* turns copyparty into chromecast just more flexible (and probably way more buggy)
+* usage: put the js somewhere in the webroot and `--js-browser /memes/meadup.js`

contrib/plugins/meadup.js

@@ -0,0 +1,506 @@
+// USAGE:
+//   place this file somewhere in the webroot and then
+//   python3 -m copyparty --js-browser /memes/meadup.js
+//
+// FEATURES:
+// * adds an onscreen keyboard for operating a media center remotely,
+//    relies on https://github.com/9001/copyparty/blob/hovudstraum/bin/mtag/very-bad-idea.py
+// * adds an interactive anime girl (if you can find the dependencies)
+
+var hambagas = [
+    "https://www.youtube.com/watch?v=pFA3KGp4GuU"
+];
+
+// keybaord,
+//   onscreen keyboard by @steinuil
+function initKeybaord(BASE_URL, HAMBAGA, consoleLog, consoleError) {
+    document.querySelector('.keybaord-container').innerHTML = `
+      <div class="keybaord-body">
+        <div class="keybaord-row keybaord-row-1">
+          <div class="keybaord-key" data-keybaord-key="Escape">
+            esc
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F1">
+            F1
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F2">
+            F2
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F3">
+            F3
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F4">
+            F4
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F5">
+            F5
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F6">
+            F6
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F7">
+            F7
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F8">
+            F8
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F9">
+            F9
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F10">
+            F10
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F11">
+            F11
+          </div>
+          <div class="keybaord-key" data-keybaord-key="F12">
+            F12
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Insert">
+            ins
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Delete">
+            del
+          </div>
+        </div>
+        <div class="keybaord-row keybaord-row-2">
+          <div class="keybaord-key" data-keybaord-key="\`">
+            \`
+          </div>
+          <div class="keybaord-key" data-keybaord-key="1">
+            1
+          </div>
+          <div class="keybaord-key" data-keybaord-key="2">
+            2
+          </div>
+          <div class="keybaord-key" data-keybaord-key="3">
+            3
+          </div>
+          <div class="keybaord-key" data-keybaord-key="4">
+            4
+          </div>
+          <div class="keybaord-key" data-keybaord-key="5">
+            5
+          </div>
+          <div class="keybaord-key" data-keybaord-key="6">
+            6
+          </div>
+          <div class="keybaord-key" data-keybaord-key="7">
+            7
+          </div>
+          <div class="keybaord-key" data-keybaord-key="8">
+            8
+          </div>
+          <div class="keybaord-key" data-keybaord-key="9">
+            9
+          </div>
+          <div class="keybaord-key" data-keybaord-key="0">
+            0
+          </div>
+          <div class="keybaord-key" data-keybaord-key="-">
+            -
+          </div>
+          <div class="keybaord-key" data-keybaord-key="=">
+            =
+          </div>
+          <div class="keybaord-key keybaord-backspace" data-keybaord-key="BackSpace">
+            backspace
+          </div>
+        </div>
+        <div class="keybaord-row keybaord-row-3">
+          <div class="keybaord-key keybaord-tab" data-keybaord-key="Tab">
+            tab
+          </div>
+          <div class="keybaord-key" data-keybaord-key="q">
+            q
+          </div>
+          <div class="keybaord-key" data-keybaord-key="w">
+            w
+          </div>
+          <div class="keybaord-key" data-keybaord-key="e">
+            e
+          </div>
+          <div class="keybaord-key" data-keybaord-key="r">
+            r
+          </div>
+          <div class="keybaord-key" data-keybaord-key="t">
+            t
+          </div>
+          <div class="keybaord-key" data-keybaord-key="y">
+            y
+          </div>
+          <div class="keybaord-key" data-keybaord-key="u">
+            u
+          </div>
+          <div class="keybaord-key" data-keybaord-key="i">
+            i
+          </div>
+          <div class="keybaord-key" data-keybaord-key="o">
+            o
+          </div>
+          <div class="keybaord-key" data-keybaord-key="p">
+            p
+          </div>
+          <div class="keybaord-key" data-keybaord-key="[">
+            [
+          </div>
+          <div class="keybaord-key" data-keybaord-key="]">
+            ]
+          </div>
+          <div class="keybaord-key keybaord-enter" data-keybaord-key="Return">
+            enter
+          </div>
+        </div>
+        <div class="keybaord-row keybaord-row-4">
+          <div class="keybaord-key keybaord-capslock" data-keybaord-key="HAMBAGA">
+            🍔
+          </div>
+          <div class="keybaord-key" data-keybaord-key="a">
+            a
+          </div>
+          <div class="keybaord-key" data-keybaord-key="s">
+            s
+          </div>
+          <div class="keybaord-key" data-keybaord-key="d">
+            d
+          </div>
+          <div class="keybaord-key" data-keybaord-key="f">
+            f
+          </div>
+          <div class="keybaord-key" data-keybaord-key="g">
+            g
+          </div>
+          <div class="keybaord-key" data-keybaord-key="h">
+            h
+          </div>
+          <div class="keybaord-key" data-keybaord-key="j">
+            j
+          </div>
+          <div class="keybaord-key" data-keybaord-key="k">
+            k
+          </div>
+          <div class="keybaord-key" data-keybaord-key="l">
+            l
+          </div>
+          <div class="keybaord-key" data-keybaord-key=";">
+            ;
+          </div>
+          <div class="keybaord-key" data-keybaord-key="'">
+            '
+          </div>
+          <div class="keybaord-key keybaord-backslash" data-keybaord-key="\\">
+            \\
+          </div>
+        </div>
+        <div class="keybaord-row keybaord-row-5">
+          <div class="keybaord-key keybaord-lshift" data-keybaord-key="Shift_L">
+            shift
+          </div>
+          <div class="keybaord-key" data-keybaord-key="\\">
+            \\
+          </div>
+          <div class="keybaord-key" data-keybaord-key="z">
+            z
+          </div>
+          <div class="keybaord-key" data-keybaord-key="x">
+            x
+          </div>
+          <div class="keybaord-key" data-keybaord-key="c">
+            c
+          </div>
+          <div class="keybaord-key" data-keybaord-key="v">
+            v
+          </div>
+          <div class="keybaord-key" data-keybaord-key="b">
+            b
+          </div>
+          <div class="keybaord-key" data-keybaord-key="n">
+            n
+          </div>
+          <div class="keybaord-key" data-keybaord-key="m">
+            m
+          </div>
+          <div class="keybaord-key" data-keybaord-key=",">
+            ,
+          </div>
+          <div class="keybaord-key" data-keybaord-key=".">
+            .
+          </div>
+          <div class="keybaord-key" data-keybaord-key="/">
+            /
+          </div>
+          <div class="keybaord-key keybaord-rshift" data-keybaord-key="Shift_R">
+            shift
+          </div>
+        </div>
+        <div class="keybaord-row keybaord-row-6">
+          <div class="keybaord-key keybaord-lctrl" data-keybaord-key="Control_L">
+            ctrl
+          </div>
+          <div class="keybaord-key keybaord-super" data-keybaord-key="Meta_L">
+            win
+          </div>
+          <div class="keybaord-key keybaord-alt" data-keybaord-key="Alt_L">
+            alt
+          </div>
+          <div class="keybaord-key keybaord-spacebar" data-keybaord-key="space">
+            space
+          </div>
+          <div class="keybaord-key keybaord-altgr" data-keybaord-key="Alt_R">
+            altgr
+          </div>
+          <div class="keybaord-key keybaord-what" data-keybaord-key="Menu">
+            menu
+          </div>
+          <div class="keybaord-key keybaord-rctrl" data-keybaord-key="Control_R">
+            ctrl
+          </div>
+        </div>
+        <div class="keybaord-row">
+          <div class="keybaord-key" data-keybaord-key="XF86AudioLowerVolume">
+            🔉
+          </div>
+          <div class="keybaord-key" data-keybaord-key="XF86AudioRaiseVolume">
+            🔊
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Left">
+            ⬅️
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Down">
+            ⬇️
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Up">
+            ⬆️
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Right">
+            ➡️
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Page_Up">
+            PgUp
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Page_Down">
+            PgDn
+          </div>
+          <div class="keybaord-key" data-keybaord-key="Home">
+            🏠
+          </div>
+          <div class="keybaord-key" data-keybaord-key="End">
+            End
+          </div>
+        </div>
+      <div>
+    `;
+
+    function arraySample(array) {
+        return array[Math.floor(Math.random() * array.length)];
+    }
+
+    function sendMessage(msg) {
+        return fetch(BASE_URL, {
+            method: "POST",
+            headers: {
+                "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
+            },
+            body: "msg=" + encodeURIComponent(msg),
+        }).then(
+            (r) => r.text(), // so the response body shows up in network tab
+            (err) => consoleError(err)
+        );
+    }
+    const MODIFIER_ON_CLASS = "keybaord-modifier-on";
+    const KEY_DATASET = "data-keybaord-key";
+    const KEY_CLASS = "keybaord-key";
+
+    const modifiers = new Set()
+
+    function toggleModifier(button, key) {
+        button.classList.toggle(MODIFIER_ON_CLASS);
+        if (modifiers.has(key)) {
+            modifiers.delete(key);
+        } else {
+            modifiers.add(key);
+        }
+    }
+
+    function popModifiers() {
+        let modifierString = "";
+
+        modifiers.forEach((mod) => {
+            document.querySelector("[" + KEY_DATASET + "='" + mod + "']")
+                .classList.remove(MODIFIER_ON_CLASS);
+
+            modifierString += mod + "+";
+        });
+
+        modifiers.clear();
+
+        return modifierString;
+    }
+
+    Array.from(document.querySelectorAll("." + KEY_CLASS)).forEach((button) => {
+        const key = button.dataset.keybaordKey;
+
+        button.addEventListener("click", (ev) => {
+            switch (key) {
+                case "HAMBAGA":
+                    sendMessage(arraySample(HAMBAGA));
+                    break;
+
+                case "Shift_L":
+                case "Shift_R":
+
+                case "Control_L":
+                case "Control_R":
+
+                case "Meta_L":
+
+                case "Alt_L":
+                case "Alt_R":
+                    toggleModifier(button, key);
+                    break;
+
+                default: {
+                    const keyWithModifiers = popModifiers() + key;
+
+                    consoleLog(keyWithModifiers);
+
+                    sendMessage("key " + keyWithModifiers)
+                        .then(() => consoleLog(keyWithModifiers + " OK"));
+                }
+            }
+        });
+    });
+}
+
+
+// keybaord integration
+(function () {
+    var o = mknod('div');
+    clmod(o, 'keybaord-container', 1);
+    ebi('op_msg').appendChild(o);
+
+    o = mknod('style');
+    o.innerHTML = `
+.keybaord-body {
+	display: flex;
+	flex-flow: column nowrap;
+    margin: .6em 0;
+}
+
+.keybaord-row {
+	display: flex;
+}
+
+.keybaord-key {
+	border: 1px solid rgba(128,128,128,0.2);
+	width: 41px;
+	height: 40px;
+
+	display: flex;
+	justify-content: center;
+	align-items: center;
+}
+
+.keybaord-key:active {
+	background-color: lightgrey;
+}
+
+.keybaord-key.keybaord-modifier-on {
+	background-color: lightblue;
+}
+
+.keybaord-key.keybaord-backspace {
+	width: 82px;
+}
+
+.keybaord-key.keybaord-tab {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-enter {
+	width: 69px;
+}
+
+.keybaord-key.keybaord-capslock {
+	width: 80px;
+}
+
+.keybaord-key.keybaord-backslash {
+	width: 88px;
+}
+
+.keybaord-key.keybaord-lshift {
+	width: 65px;
+}
+
+.keybaord-key.keybaord-rshift {
+	width: 103px;
+}
+
+.keybaord-key.keybaord-lctrl {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-super {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-alt {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-altgr {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-what {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-rctrl {
+	width: 55px;
+}
+
+.keybaord-key.keybaord-spacebar {
+	width: 302px;
+}
+`;
+    document.head.appendChild(o);
+
+    initKeybaord('/', hambagas,
+        (msg) => { toast.inf(2, msg.toString()) },
+        (msg) => { toast.err(30, msg.toString()) });
+})();
+
+
+// live2d (dumb pointless meme)
+//   dependencies for this part are not tracked in git
+//   so delete this section if you wanna use this file
+//   (or supply your own l2d model and js)
+(function () {
+    var o = mknod('link');
+    o.setAttribute('rel', 'stylesheet');
+    o.setAttribute('href', "/bad-memes/pio.css");
+    document.head.appendChild(o);
+
+    o = mknod('style');
+    o.innerHTML = '.pio-container{text-shadow:none;z-index:1}';
+    document.head.appendChild(o);
+
+    o = mknod('div');
+    clmod(o, 'pio-container', 1);
+    o.innerHTML = '<div class="pio-action"></div><canvas id="pio" width="280" height="500"></canvas>';
+    document.body.appendChild(o);
+
+    var remaining = 3;
+    for (var a of ['pio', 'l2d', 'fireworks']) {
+        import_js(`/bad-memes/${a}.js`, function () {
+            if (remaining --> 1)
+                return;
+
+            o = mknod('script');
+            o.innerHTML = 'var pio = new Paul_Pio({"selector":[],"mode":"fixed","hidden":false,"content":{"close":"ok bye"},"model":["/bad-memes/sagiri/model.json"]});';
+            document.body.appendChild(o);
+        });
+    }
+})();

copyparty/__init__.py

@@ -25,26 +25,34 @@ ANYWIN = WINDOWS or sys.platform in ["msys"]
 MACOS = platform.system() == "Darwin"
 
 
-def get_unix_home():
-    try:
-        v = os.environ["XDG_CONFIG_HOME"]
-        if not v:
-            raise Exception()
-        ret = os.path.normpath(v)
-        os.listdir(ret)
-        return ret
-    except:
-        pass
+def get_unixdir():
+    paths = [
+        (os.environ.get, "XDG_CONFIG_HOME"),
+        (os.path.expanduser, "~/.config"),
+        (os.environ.get, "TMPDIR"),
+        (os.environ.get, "TEMP"),
+        (os.environ.get, "TMP"),
+        (unicode, "/tmp"),
+    ]
+    for chk in [os.listdir, os.mkdir]:
+        for pf, pa in paths:
+            try:
+                p = pf(pa)
+                # print(chk.__name__, p, pa)
+                if not p or p.startswith("~"):
+                    continue
 
-    try:
-        v = os.path.expanduser("~/.config")
-        if v.startswith("~"):
-            raise Exception()
-        ret = os.path.normpath(v)
-        os.listdir(ret)
-        return ret
-    except:
-        return "/tmp"
+                p = os.path.normpath(p)
+                chk(p)
+                p = os.path.join(p, "copyparty")
+                if not os.path.isdir(p):
+                    os.mkdir(p)
+
+                return p
+            except:
+                pass
+
+    raise Exception("could not find a writable path for config")
 
 
 class EnvParams(object):
@@ -59,7 +67,7 @@ class EnvParams(object):
         elif sys.platform == "darwin":
             self.cfg = os.path.expanduser("~/Library/Preferences/copyparty")
         else:
-            self.cfg = get_unix_home() + "/copyparty"
+            self.cfg = get_unixdir()
 
         self.cfg = self.cfg.replace("\\", "/")
         try:

copyparty/__main__.py

@@ -23,7 +23,7 @@ from textwrap import dedent
 from .__init__ import E, WINDOWS, ANYWIN, VT100, PY2, unicode
 from .__version__ import S_VERSION, S_BUILD_DT, CODENAME
 from .svchub import SvcHub
-from .util import py_desc, align_tab, IMPLICATIONS, ansi_re
+from .util import py_desc, align_tab, IMPLICATIONS, ansi_re, min_ex
 from .authsrv import re_vol
 
 HAVE_SSL = True
@@ -222,6 +222,54 @@ def sighandler(sig=None, frame=None):
     print("\n".join(msg))
 
 
+def disable_quickedit():
+    import ctypes
+    import atexit
+    from ctypes import wintypes
+
+    def ecb(ok, fun, args):
+        if not ok:
+            err = ctypes.get_last_error()
+            if err:
+                raise ctypes.WinError(err)
+        return args
+
+    k32 = ctypes.WinDLL("kernel32", use_last_error=True)
+    if PY2:
+        wintypes.LPDWORD = ctypes.POINTER(wintypes.DWORD)
+
+    k32.GetStdHandle.errcheck = ecb
+    k32.GetConsoleMode.errcheck = ecb
+    k32.SetConsoleMode.errcheck = ecb
+    k32.GetConsoleMode.argtypes = (wintypes.HANDLE, wintypes.LPDWORD)
+    k32.SetConsoleMode.argtypes = (wintypes.HANDLE, wintypes.DWORD)
+
+    def cmode(out, mode=None):
+        h = k32.GetStdHandle(-11 if out else -10)
+        if mode:
+            return k32.SetConsoleMode(h, mode)
+
+        mode = wintypes.DWORD()
+        k32.GetConsoleMode(h, ctypes.byref(mode))
+        return mode.value
+
+    # disable quickedit
+    mode = orig_in = cmode(False)
+    quickedit = 0x40
+    extended = 0x80
+    mask = quickedit + extended
+    if mode & mask != extended:
+        atexit.register(cmode, False, orig_in)
+        cmode(False, mode & ~mask | extended)
+
+    # enable colors in case the os.system("rem") trick ever stops working
+    if VT100:
+        mode = orig_out = cmode(True)
+        if mode & 4 != 4:
+            atexit.register(cmode, True, orig_out)
+            cmode(True, mode | 4)
+
+
 def run_argparse(argv, formatter):
     ap = argparse.ArgumentParser(
         formatter_class=formatter,
@@ -302,6 +350,8 @@ def run_argparse(argv, formatter):
             
             \033[0mdatabase, general:
               \033[36me2d\033[35m sets -e2d (all -e2* args can be set using ce2* volflags)
+              \033[36md2ts\033[35m disables metadata collection for existing files
+              \033[36md2ds\033[35m disables onboot indexing, overrides -e2ds*
               \033[36md2t\033[35m disables metadata collection, overrides -e2t*
               \033[36md2d\033[35m disables all database stuff, overrides -e2*
               \033[36mnohash=\\.iso$\033[35m skips hashing file contents if path matches *.iso
@@ -368,6 +418,7 @@ def run_argparse(argv, formatter):
     ap2.add_argument("-emp", action="store_true", help="enable markdown plugins")
     ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
     ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
+    ap2.add_argument("--wintitle", metavar="TXT", type=u, default="cpp @ $pub", help="window title, for example '$ip-10.1.2.' or '$ip-'")
 
     ap2 = ap.add_argument_group('upload options')
     ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads")
@@ -394,8 +445,16 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--ssl-dbg", action="store_true", help="dump some tls info")
     ap2.add_argument("--ssl-log", metavar="PATH", type=u, help="log master secrets")
 
+    ap2 = ap.add_argument_group('FTP options')
+    ap2.add_argument("--ftp", metavar="PORT", type=int, help="enable FTP server on PORT, for example 3921")
+    ap2.add_argument("--ftps", metavar="PORT", type=int, help="enable FTPS server on PORT, for example 3990")
+    ap2.add_argument("--ftp-dbg", action="store_true", help="enable debug logging")
+    ap2.add_argument("--ftp-nat", metavar="ADDR", type=u, help="the NAT address to use for passive connections")
+    ap2.add_argument("--ftp-pr", metavar="P-P", type=u, help="the range of TCP ports to use for passive connections, for example 12000-13000")
+
     ap2 = ap.add_argument_group('opt-outs')
     ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
+    ap2.add_argument("--keep-qem", action="store_true", help="do not disable quick-edit-mode on windows")
     ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
     ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
     ap2.add_argument("-nih", action="store_true", help="no info hostname")
@@ -412,6 +471,8 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--no-logues", action="store_true", help="disable rendering .prologue/.epilogue.html into directory listings")
     ap2.add_argument("--no-readme", action="store_true", help="disable rendering readme.md into directory listings")
     ap2.add_argument("--vague-403", action="store_true", help="send 404 instead of 403 (security through ambiguity, very enterprise)")
+    ap2.add_argument("--force-js", action="store_true", help="don't send HTML folder listings, force clients to use the embedded json instead")
+    ap2.add_argument("--no-robots", action="store_true", help="adds http and html headers asking search engines to not index anything")
 
     ap2 = ap.add_argument_group('yolo options')
     ap2.add_argument("--ign-ebind", action="store_true", help="continue running even if it's impossible to listen on some of the requested endpoints")
@@ -461,6 +522,7 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--no-idx", metavar="PTN", type=u, help="regex: disable indexing of matching paths during e2ds folder scans")
     ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval, 0=off, can be set per-volume with the 'scan' volflag")
     ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
+    ap2.add_argument("--srch-hits", metavar="N", type=int, default=1000, help="max search results")
     
     ap2 = ap.add_argument_group('metadata db options')
     ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing")
@@ -479,7 +541,9 @@ def run_argparse(argv, formatter):
     ap2 = ap.add_argument_group('ui options')
     ap2.add_argument("--js-browser", metavar="L", type=u, help="URL to additional JS to include")
     ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include")
+    ap2.add_argument("--html-head", metavar="TXT", type=u, default="", help="text to append to the <head> of all HTML pages")
     ap2.add_argument("--textfiles", metavar="CSV", type=u, default="txt,nfo,diz,cue,readme", help="file extensions to present as plaintext")
+    ap2.add_argument("--doctitle", metavar="TXT", type=u, default="copyparty", help="title / service-name to show in html documents")
 
     ap2 = ap.add_argument_group('debug options')
     ap2.add_argument("--no-sendfile", action="store_true", help="disable sendfile")
@@ -550,6 +614,15 @@ def main(argv=None):
     except AssertionError:
         al = run_argparse(argv, Dodge11874)
 
+    if WINDOWS and not al.keep_qem:
+        try:
+            disable_quickedit()
+        except:
+            print("\nfailed to disable quick-edit-mode:\n" + min_ex() + "\n")
+
+    if not VT100:
+        al.wintitle = ""
+
     nstrs = []
     anymod = False
     for ostr in al.v or []:

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 1, 3)
-CODENAME = "opus"
-BUILD_DT = (2021, 11, 20)
+VERSION = (1, 2, 3)
+CODENAME = "ftp btw"
+BUILD_DT = (2022, 3, 24)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -14,6 +14,7 @@ from datetime import datetime
 from .__init__ import WINDOWS
 from .util import (
     IMPLICATIONS,
+    META_NOBOTS,
     uncyg,
     undot,
     unhumanize,
@@ -394,6 +395,13 @@ class VFS(object):
                 if ok:
                     virt_vis[name] = vn2
 
+        if ".hist" in abspath:
+            p = abspath.replace("\\", "/") if WINDOWS else abspath
+            if p.endswith("/.hist"):
+                real = [x for x in real if not x[0].startswith("up2k.")]
+            elif "/.hist/th/" in p:
+                real = [x for x in real if not x[0].endswith("dir.txt")]
+
         return [abspath, real, virt_vis]
 
     def walk(self, rel, rem, seen, uname, permsets, dots, scandir, lstat):
@@ -444,10 +452,6 @@ class VFS(object):
         if flt:
             flt = {k: True for k in flt}
 
-        f1 = "{0}.hist{0}up2k.".format(os.sep)
-        f2a = os.sep + "dir.txt"
-        f2b = "{0}.hist{0}".format(os.sep)
-
         # if multiselect: add all items to archive root
         # if single folder: the folder itself is the top-level item
         folder = "" if flt else (vrem.split("/")[-1] or "top")
@@ -483,13 +487,6 @@ class VFS(object):
                 for x in rm:
                     del vd[x]
 
-            # up2k filetring based on actual abspath
-            files = [
-                x
-                for x in files
-                if f1 not in x[1] and (not x[1].endswith(f2a) or f2b not in x[1])
-            ]
-
             for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in files]:
                 yield f
 
@@ -865,6 +862,19 @@ class AuthSrv(object):
             if use:
                 vol.lim = lim
 
+        if self.args.no_robots:
+            for vol in vfs.all_vols.values():
+                # volflag "robots" overrides global "norobots", allowing indexing by search engines for this vol
+                if not vol.flags.get("robots"):
+                    vol.flags["norobots"] = True
+
+        for vol in vfs.all_vols.values():
+            h = [vol.flags.get("html_head", self.args.html_head)]
+            if vol.flags.get("norobots"):
+                h.insert(0, META_NOBOTS)
+
+            vol.flags["html_head"] = "\n".join([x for x in h if x])
+
         for vol in vfs.all_vols.values():
             fk = vol.flags.get("fk")
             if fk:
@@ -926,6 +936,14 @@ class AuthSrv(object):
                 vol.flags["d2t"] = True
                 vol.flags = {k: v for k, v in vol.flags.items() if not k.startswith(rm)}
 
+            # d2ds drops all onboot scans for a volume
+            for grp, rm in [["d2ds", "e2ds"], ["d2ts", "e2ts"]]:
+                if not vol.flags.get(grp, False):
+                    continue
+
+                vol.flags["d2ts"] = True
+                vol.flags = {k: v for k, v in vol.flags.items() if not k.startswith(rm)}
+
             # mt* needs e2t so drop those too
             for grp, rm in [["e2t", "mt"]]:
                 if vol.flags.get(grp, False):

copyparty/bos/bos.py

@@ -2,7 +2,7 @@
 from __future__ import print_function, unicode_literals
 
 import os
-from ..util import fsenc, fsdec
+from ..util import fsenc, fsdec, SYMTIME
 from . import path
 
 
@@ -18,10 +18,6 @@ def listdir(p="."):
     return [fsdec(x) for x in os.listdir(fsenc(p))]
 
 
-def lstat(p):
-    return os.lstat(fsenc(p))
-
-
 def makedirs(name, mode=0o755, exist_ok=True):
     bname = fsenc(name)
     try:
@@ -55,5 +51,17 @@ def unlink(p):
     return os.unlink(fsenc(p))
 
 
-def utime(p, times=None):
-    return os.utime(fsenc(p), times)
+def utime(p, times=None, follow_symlinks=True):
+    if SYMTIME:
+        return os.utime(fsenc(p), times, follow_symlinks=follow_symlinks)
+    else:
+        return os.utime(fsenc(p), times)
+
+
+if hasattr(os, "lstat"):
+
+    def lstat(p):
+        return os.lstat(fsenc(p))
+
+else:
+    lstat = stat

copyparty/bos/path.py

@@ -2,7 +2,7 @@
 from __future__ import print_function, unicode_literals
 
 import os
-from ..util import fsenc, fsdec
+from ..util import fsenc, fsdec, SYMTIME
 
 
 def abspath(p):
@@ -13,8 +13,11 @@ def exists(p):
     return os.path.exists(fsenc(p))
 
 
-def getmtime(p):
-    return os.path.getmtime(fsenc(p))
+def getmtime(p, follow_symlinks=True):
+    if not follow_symlinks and SYMTIME:
+        return os.lstat(fsenc(p)).st_mtime
+    else:
+        return os.path.getmtime(fsenc(p))
 
 
 def getsize(p):
@@ -33,5 +36,9 @@ def islink(p):
     return os.path.islink(fsenc(p))
 
 
+def lexists(p):
+    return os.path.lexists(fsenc(p))
+
+
 def realpath(p):
     return fsdec(os.path.realpath(fsenc(p)))

copyparty/ftpd.py

@@ -0,0 +1,374 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import os
+import sys
+import stat
+import time
+import logging
+import threading
+
+from .__init__ import E, PY2
+from .util import Pebkac, fsenc, exclude_dotfiles
+from .bos import bos
+
+try:
+    from pyftpdlib.ioloop import IOLoop
+except ImportError:
+    p = os.path.join(E.mod, "vend")
+    print("loading asynchat from " + p)
+    sys.path.append(p)
+    from pyftpdlib.ioloop import IOLoop
+
+from pyftpdlib.authorizers import DummyAuthorizer, AuthenticationFailed
+from pyftpdlib.filesystems import AbstractedFS, FilesystemError
+from pyftpdlib.handlers import FTPHandler
+from pyftpdlib.servers import FTPServer
+from pyftpdlib.log import config_logging
+
+
+try:
+    from typing import TYPE_CHECKING
+
+    if TYPE_CHECKING:
+        from .svchub import SvcHub
+except ImportError:
+    pass
+
+
+class FtpAuth(DummyAuthorizer):
+    def __init__(self):
+        super(FtpAuth, self).__init__()
+        self.hub = None  # type: SvcHub
+
+    def validate_authentication(self, username, password, handler):
+        asrv = self.hub.asrv
+        if username == "anonymous":
+            password = ""
+
+        uname = "*"
+        if password:
+            uname = asrv.iacct.get(password, None)
+
+        handler.username = uname
+
+        if password and not uname:
+            raise AuthenticationFailed("Authentication failed.")
+
+    def get_home_dir(self, username):
+        return "/"
+
+    def has_user(self, username):
+        asrv = self.hub.asrv
+        return username in asrv.acct
+
+    def has_perm(self, username, perm, path=None):
+        return True  # handled at filesystem layer
+
+    def get_perms(self, username):
+        return "elradfmwMT"
+
+    def get_msg_login(self, username):
+        return "sup {}".format(username)
+
+    def get_msg_quit(self, username):
+        return "cya"
+
+
+class FtpFs(AbstractedFS):
+    def __init__(self, root, cmd_channel):
+        self.h = self.cmd_channel = cmd_channel  # type: FTPHandler
+        self.hub = cmd_channel.hub  # type: SvcHub
+        self.args = cmd_channel.args
+
+        self.uname = self.hub.asrv.iacct.get(cmd_channel.password, "*")
+
+        self.cwd = "/"  # pyftpdlib convention of leading slash
+        self.root = "/var/lib/empty"
+
+        self.listdirinfo = self.listdir
+        self.chdir(".")
+
+    def v2a(self, vpath, r=False, w=False, m=False, d=False):
+        try:
+            vpath = vpath.replace("\\", "/").lstrip("/")
+            vfs, rem = self.hub.asrv.vfs.get(vpath, self.uname, r, w, m, d)
+            if not vfs.realpath:
+                raise FilesystemError("no filesystem mounted at this path")
+
+            return os.path.join(vfs.realpath, rem)
+        except Pebkac as ex:
+            raise FilesystemError(str(ex))
+
+    def rv2a(self, vpath, r=False, w=False, m=False, d=False):
+        return self.v2a(os.path.join(self.cwd, vpath), r, w, m, d)
+
+    def ftp2fs(self, ftppath):
+        # return self.v2a(ftppath)
+        return ftppath  # self.cwd must be vpath
+
+    def fs2ftp(self, fspath):
+        # raise NotImplementedError()
+        return fspath
+
+    def validpath(self, path):
+        if "/.hist/" in path:
+            if "/up2k." in path or path.endswith("/dir.txt"):
+                raise FilesystemError("access to this file is forbidden")
+
+        return True
+
+    def open(self, filename, mode):
+        r = "r" in mode
+        w = "w" in mode or "a" in mode or "+" in mode
+
+        ap = self.rv2a(filename, r, w)
+        if w and bos.path.exists(ap):
+            raise FilesystemError("cannot open existing file for writing")
+
+        self.validpath(ap)
+        return open(fsenc(ap), mode)
+
+    def chdir(self, path):
+        self.cwd = join(self.cwd, path)
+        x = self.hub.asrv.vfs.can_access(self.cwd.lstrip("/"), self.h.username)
+        self.can_read, self.can_write, self.can_move, self.can_delete, self.can_get = x
+
+    def mkdir(self, path):
+        ap = self.rv2a(path, w=True)
+        bos.mkdir(ap)
+
+    def listdir(self, path):
+        vpath = join(self.cwd, path).lstrip("/")
+        try:
+            vfs, rem = self.hub.asrv.vfs.get(vpath, self.uname, True, False)
+
+            fsroot, vfs_ls, vfs_virt = vfs.ls(
+                rem, self.uname, not self.args.no_scandir, [[True], [False, True]]
+            )
+            vfs_ls = [x[0] for x in vfs_ls]
+            vfs_ls.extend(vfs_virt.keys())
+
+            if not self.args.ed:
+                vfs_ls = exclude_dotfiles(vfs_ls)
+
+            vfs_ls.sort()
+            return vfs_ls
+        except Exception as ex:
+            if vpath:
+                # display write-only folders as empty
+                return []
+
+            # return list of volumes
+            r = {x.split("/")[0]: 1 for x in self.hub.asrv.vfs.all_vols.keys()}
+            return list(sorted(list(r.keys())))
+
+    def rmdir(self, path):
+        ap = self.rv2a(path, d=True)
+        bos.rmdir(ap)
+
+    def remove(self, path):
+        if self.args.no_del:
+            raise FilesystemError("the delete feature is disabled in server config")
+
+        vp = join(self.cwd, path).lstrip("/")
+        x = self.hub.broker.put(
+            True, "up2k.handle_rm", self.uname, self.h.remote_ip, [vp]
+        )
+
+        try:
+            x.get()
+        except Exception as ex:
+            raise FilesystemError(str(ex))
+
+    def rename(self, src, dst):
+        if not self.can_move:
+            raise FilesystemError("not allowed for user " + self.h.username)
+
+        if self.args.no_mv:
+            m = "the rename/move feature is disabled in server config"
+            raise FilesystemError(m)
+
+        svp = join(self.cwd, src).lstrip("/")
+        dvp = join(self.cwd, dst).lstrip("/")
+        x = self.hub.broker.put(True, "up2k.handle_mv", self.uname, svp, dvp)
+        try:
+            x.get()
+        except Exception as ex:
+            raise FilesystemError(str(ex))
+
+    def chmod(self, path, mode):
+        pass
+
+    def stat(self, path):
+        try:
+            ap = self.rv2a(path, r=True)
+            return bos.stat(ap)
+        except:
+            ap = self.rv2a(path)
+            st = bos.stat(ap)
+            if not stat.S_ISDIR(st.st_mode):
+                raise
+
+            return st
+
+    def utime(self, path, timeval):
+        ap = self.rv2a(path, w=True)
+        return bos.utime(ap, (timeval, timeval))
+
+    def lstat(self, path):
+        ap = self.rv2a(path)
+        return bos.lstat(ap)
+
+    def isfile(self, path):
+        st = self.stat(path)
+        return stat.S_ISREG(st.st_mode)
+
+    def islink(self, path):
+        ap = self.rv2a(path)
+        return bos.path.islink(ap)
+
+    def isdir(self, path):
+        try:
+            st = self.stat(path)
+            return stat.S_ISDIR(st.st_mode)
+        except:
+            return True
+
+    def getsize(self, path):
+        ap = self.rv2a(path)
+        return bos.path.getsize(ap)
+
+    def getmtime(self, path):
+        ap = self.rv2a(path)
+        return bos.path.getmtime(ap)
+
+    def realpath(self, path):
+        return path
+
+    def lexists(self, path):
+        ap = self.rv2a(path)
+        return bos.path.lexists(ap)
+
+    def get_user_by_uid(self, uid):
+        return "root"
+
+    def get_group_by_uid(self, gid):
+        return "root"
+
+
+class FtpHandler(FTPHandler):
+    abstracted_fs = FtpFs
+
+    def __init__(self, conn, server, ioloop=None):
+        if PY2:
+            FTPHandler.__init__(self, conn, server, ioloop)
+        else:
+            super(FtpHandler, self).__init__(conn, server, ioloop)
+
+        # abspath->vpath mapping to resolve log_transfer paths
+        self.vfs_map = {}
+
+    def ftp_STOR(self, file, mode="w"):
+        vp = join(self.fs.cwd, file).lstrip("/")
+        ap = self.fs.v2a(vp)
+        self.vfs_map[ap] = vp
+        # print("ftp_STOR: {} {} => {}".format(vp, mode, ap))
+        ret = FTPHandler.ftp_STOR(self, file, mode)
+        # print("ftp_STOR: {} {} OK".format(vp, mode))
+        return ret
+
+    def log_transfer(self, cmd, filename, receive, completed, elapsed, bytes):
+        ap = filename.decode("utf-8", "replace")
+        vp = self.vfs_map.pop(ap, None)
+        # print("xfer_end: {} => {}".format(ap, vp))
+        if vp:
+            vp, fn = os.path.split(vp)
+            vfs, rem = self.hub.asrv.vfs.get(vp, self.username, False, True)
+            vfs, rem = vfs.get_dbv(rem)
+            self.hub.broker.put(
+                False,
+                "up2k.hash_file",
+                vfs.realpath,
+                vfs.flags,
+                rem,
+                fn,
+                self.remote_ip,
+                time.time(),
+            )
+
+        return FTPHandler.log_transfer(
+            self, cmd, filename, receive, completed, elapsed, bytes
+        )
+
+
+try:
+    from pyftpdlib.handlers import TLS_FTPHandler
+
+    class SftpHandler(FtpHandler, TLS_FTPHandler):
+        pass
+
+except:
+    pass
+
+
+class Ftpd(object):
+    def __init__(self, hub):
+        self.hub = hub
+        self.args = hub.args
+
+        hs = []
+        if self.args.ftp:
+            hs.append([FtpHandler, self.args.ftp])
+        if self.args.ftps:
+            try:
+                h = SftpHandler
+            except:
+                m = "\nftps requires pyopenssl;\nplease run the following:\n\n  {} -m pip install --user pyopenssl\n"
+                print(m.format(sys.executable))
+                sys.exit(1)
+
+            h.certfile = os.path.join(E.cfg, "cert.pem")
+            h.tls_control_required = True
+            h.tls_data_required = True
+
+            hs.append([h, self.args.ftps])
+
+        for h in hs:
+            h, lp = h
+            h.hub = hub
+            h.args = hub.args
+            h.authorizer = FtpAuth()
+            h.authorizer.hub = hub
+
+            if self.args.ftp_pr:
+                p1, p2 = [int(x) for x in self.args.ftp_pr.split("-")]
+                if self.args.ftp and self.args.ftps:
+                    # divide port range in half
+                    d = int((p2 - p1) / 2)
+                    if lp == self.args.ftp:
+                        p2 = p1 + d
+                    else:
+                        p1 += d + 1
+
+                h.passive_ports = list(range(p1, p2 + 1))
+
+            if self.args.ftp_nat:
+                h.masquerade_address = self.args.ftp_nat
+
+        if self.args.ftp_dbg:
+            config_logging(level=logging.DEBUG)
+
+        ioloop = IOLoop()
+        for ip in self.args.i:
+            for h, lp in hs:
+                FTPServer((ip, int(lp)), h, ioloop)
+
+        t = threading.Thread(target=ioloop.loop)
+        t.daemon = True
+        t.start()
+
+
+def join(p1, p2):
+    w = os.path.join(p1, p2.replace("\\", "/"))
+    return os.path.normpath(w).replace("\\", "/")

copyparty/httpcli.py

@@ -60,10 +60,16 @@ class HttpCli(object):
         self.bufsz = 1024 * 32
         self.hint = None
         self.trailing_slash = True
+        self.out_headerlist = []
         self.out_headers = {
             "Access-Control-Allow-Origin": "*",
             "Cache-Control": "no-store; max-age=0",
         }
+        h = self.args.html_head
+        if self.args.no_robots:
+            h = META_NOBOTS + (("\n" + h) if h else "")
+            self.out_headers["X-Robots-Tag"] = "noindex, nofollow"
+        self.html_head = h
 
     def log(self, msg, c=0):
         ptn = self.asrv.re_pwd
@@ -91,6 +97,8 @@ class HttpCli(object):
         tpl = self.conn.hsrv.j2[name]
         if ka:
             ka["ts"] = self.conn.hsrv.cachebuster()
+            ka["svcname"] = self.args.doctitle
+            ka["html_head"] = self.html_head
             return tpl.render(**ka)
 
         return tpl
@@ -226,7 +234,7 @@ class HttpCli(object):
         self.gvol = self.asrv.vfs.aget[self.uname]
 
         if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"):
-            self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0]
+            self.out_headerlist.append(("Set-Cookie", self.get_pwd_cookie(pwd)[0]))
 
         self.ua = self.headers.get("user-agent", "")
         self.is_rclone = self.ua.startswith("rclone/")
@@ -285,7 +293,7 @@ class HttpCli(object):
         self.out_headers["Cache-Control"] = "max-age=" + n
 
     def k304(self):
-        k304 = self.cookies.get("k304", "")
+        k304 = self.cookies.get("k304")
         return k304 == "y" or ("; Trident/" in self.ua and not k304)
 
     def send_headers(self, length, status=200, mime=None, headers=None):
@@ -310,7 +318,7 @@ class HttpCli(object):
 
         self.out_headers["Content-Type"] = mime
 
-        for k, v in self.out_headers.items():
+        for k, v in list(self.out_headers.items()) + self.out_headerlist:
             response.append("{}: {}".format(k, v))
 
         try:
@@ -439,6 +447,12 @@ class HttpCli(object):
             if "k304" in self.uparam:
                 return self.set_k304()
 
+            if "am_js" in self.uparam:
+                return self.set_am_js()
+
+            if "reset" in self.uparam:
+                return self.set_cfg_reset()
+
             if "h" in self.uparam:
                 return self.tx_mounts()
 
@@ -516,7 +530,7 @@ class HttpCli(object):
                 return self.handle_stash()
 
             if "save" in opt:
-                post_sz, _, _, _, path = self.dump_to_file()
+                post_sz, _, _, _, path, _ = self.dump_to_file()
                 self.log("urlform: {} bytes, {}".format(post_sz, path))
             elif "print" in opt:
                 reader, _ = self.get_body_reader()
@@ -601,8 +615,8 @@ class HttpCli(object):
                 alg = alg or "gz"  # def.pk
                 try:
                     # config-forced opts
-                    alg, lv = pk.split(",")
-                    lv[alg] = int(lv)
+                    alg, nlv = pk.split(",")
+                    lv[alg] = int(nlv)
                 except:
                     pass
 
@@ -632,7 +646,7 @@ class HttpCli(object):
         with ren_open(fn, *open_a, **params) as f:
             f, fn = f["orz"]
             path = os.path.join(fdir, fn)
-            post_sz, sha_hex, sha_b64 = hashcopy(reader, f)
+            post_sz, sha_hex, sha_b64 = hashcopy(reader, f, self.args.s_wr_slp)
 
         if lim:
             lim.nup(self.ip)
@@ -643,26 +657,47 @@ class HttpCli(object):
                 bos.unlink(path)
                 raise
 
-        if not self.args.nw:
-            vfs, vrem = vfs.get_dbv(rem)
-            self.conn.hsrv.broker.put(
-                False,
-                "up2k.hash_file",
-                vfs.realpath,
-                vfs.flags,
-                vrem,
-                fn,
-                self.ip,
-                time.time(),
-            )
+        if self.args.nw:
+            return post_sz, sha_hex, sha_b64, remains, path, ""
 
-        return post_sz, sha_hex, sha_b64, remains, path
+        vfs, rem = vfs.get_dbv(rem)
+        self.conn.hsrv.broker.put(
+            False,
+            "up2k.hash_file",
+            vfs.realpath,
+            vfs.flags,
+            rem,
+            fn,
+            self.ip,
+            time.time(),
+        )
+
+        vsuf = ""
+        if self.can_read and "fk" in vfs.flags:
+            vsuf = "?k=" + gen_filekey(
+                self.args.fk_salt,
+                path,
+                post_sz,
+                0 if ANYWIN else bos.stat(path).st_ino,
+            )[: vfs.flags["fk"]]
+
+        vpath = "/".join([x for x in [vfs.vpath, rem, fn] if x])
+        vpath = quotep(vpath)
+
+        url = "{}://{}/{}".format(
+            "https" if self.is_https else "http",
+            self.headers.get("host") or "{}:{}".format(*list(self.s.getsockname())),
+            vpath + vsuf,
+        )
+
+        return post_sz, sha_hex, sha_b64, remains, path, url
 
     def handle_stash(self):
-        post_sz, sha_hex, sha_b64, remains, path = self.dump_to_file()
+        post_sz, sha_hex, sha_b64, remains, path, url = self.dump_to_file()
         spd = self._spd(post_sz)
-        self.log("{} wrote {}/{} bytes to {}".format(spd, post_sz, remains, path))
-        m = "{}\n{}\n{}\n".format(post_sz, sha_b64, sha_hex[:56])
+        m = "{} wrote {}/{} bytes to {}  # {}"
+        self.log(m.format(spd, post_sz, remains, path, sha_b64[:28]))  # 21
+        m = "{}\n{}\n{}\n{}\n".format(post_sz, sha_b64, sha_hex[:56], url)
         self.reply(m.encode("utf-8"))
         return True
 
@@ -893,7 +928,7 @@ class HttpCli(object):
 
             try:
                 f.seek(cstart[0])
-                post_sz, _, sha_b64 = hashcopy(reader, f)
+                post_sz, _, sha_b64 = hashcopy(reader, f, self.args.s_wr_slp)
 
                 if sha_b64 != chash:
                     m = "your chunk got corrupted somehow (received {} bytes); expected vs received hash:\n{}\n{}"
@@ -1086,7 +1121,7 @@ class HttpCli(object):
                         f, fname = f["orz"]
                         abspath = os.path.join(fdir, fname)
                         self.log("writing to {}".format(abspath))
-                        sz, sha_hex, sha_b64 = hashcopy(p_data, f)
+                        sz, sha_hex, sha_b64 = hashcopy(p_data, f, self.args.s_wr_slp)
                         if sz == 0:
                             raise Pebkac(400, "empty files in post")
 
@@ -1162,11 +1197,12 @@ class HttpCli(object):
                 )[: vfs.flags["fk"]]
 
             vpath = "{}/{}".format(upload_vpath, lfn).strip("/")
+            rel_url = quotep(vpath) + vsuf
             msg += 'sha512: {} // {} // {} bytes // <a href="/{}">{}</a> {}\n'.format(
                 sha_hex[:56],
                 sha_b64,
                 sz,
-                quotep(vpath) + vsuf,
+                rel_url,
                 html_escape(ofn, crlf=True),
                 vsuf,
             )
@@ -1175,15 +1211,16 @@ class HttpCli(object):
             jpart = {
                 "url": "{}://{}/{}".format(
                     "https" if self.is_https else "http",
-                    self.headers.get("host", "copyparty"),
-                    vpath + vsuf,
+                    self.headers.get("host")
+                    or "{}:{}".format(*list(self.s.getsockname())),
+                    rel_url,
                 ),
                 "sha512": sha_hex[:56],
                 "sha_b64": sha_b64,
                 "sz": sz,
                 "fn": lfn,
                 "fn_orig": ofn,
-                "path": vpath + vsuf,
+                "path": rel_url,
             }
             jmsg["files"].append(jpart)
 
@@ -1302,7 +1339,7 @@ class HttpCli(object):
             raise Pebkac(400, "expected body, got {}".format(p_field))
 
         with open(fsenc(fp), "wb", 512 * 1024) as f:
-            sz, sha512, _ = hashcopy(p_data, f)
+            sz, sha512, _ = hashcopy(p_data, f, self.args.s_wr_slp)
 
         if lim:
             lim.nup(self.ip)
@@ -1359,6 +1396,9 @@ class HttpCli(object):
             try:
                 fs_path = req_path + ext
                 st = bos.stat(fs_path)
+                if stat.S_ISDIR(st.st_mode):
+                    continue
+
                 file_ts = max(file_ts, st.st_mtime)
                 editions[ext or "plain"] = [fs_path, st.st_size]
             except:
@@ -1504,11 +1544,12 @@ class HttpCli(object):
         with open_func(*open_args) as f:
             sendfun = sendfile_kern if use_sendfile else sendfile_py
             remains = sendfun(
-                lower, upper, f, self.s, self.args.s_wr_sz, self.args.s_wr_slp
+                self.log, lower, upper, f, self.s, self.args.s_wr_sz, self.args.s_wr_slp
             )
 
         if remains > 0:
             logmsg += " \033[31m" + unicode(upper - remains) + "\033[0m"
+            self.keepalive = False
 
         spd = self._spd((upper - lower) - remains)
         if self.do_log:
@@ -1642,13 +1683,15 @@ class HttpCli(object):
 
         boundary = "\roll\tide"
         targs = {
+            "ts": self.conn.hsrv.cachebuster(),
+            "svcname": self.args.doctitle,
+            "html_head": self.html_head,
             "edit": "edit" in self.uparam,
             "title": html_escape(self.vpath, crlf=True),
             "lastmod": int(ts_md * 1000),
             "md_plug": "true" if self.args.emp else "false",
             "md_chk_rate": self.args.mcr,
             "md": boundary,
-            "ts": self.conn.hsrv.cachebuster(),
             "arg_base": arg_base,
         }
         html = template.render(**targs).encode("utf-8", "replace")
@@ -1697,6 +1740,31 @@ class HttpCli(object):
             vstate = {}
             vs = {"scanning": None, "hashq": None, "tagq": None, "mtpq": None}
 
+        if self.uparam.get("ls") in ["v", "t", "txt"]:
+            if self.uname == "*":
+                txt = "howdy stranger (you're not logged in)"
+            else:
+                txt = "welcome back {}".format(self.uname)
+
+            if vstate:
+                txt += "\nstatus:"
+                for k in ["scanning", "hashq", "tagq", "mtpq"]:
+                    txt += " {}({})".format(k, vs[k])
+
+            if rvol:
+                txt += "\nyou can browse:"
+                for v in rvol:
+                    txt += "\n  " + v
+
+            if wvol:
+                txt += "\nyou can upload to:"
+                for v in wvol:
+                    txt += "\n  " + v
+
+            txt = txt.encode("utf-8", "replace") + b"\n"
+            self.reply(txt, mime="text/plain; charset=utf-8")
+            return True
+
         html = self.j2(
             "splash",
             this=self,
@@ -1717,7 +1785,19 @@ class HttpCli(object):
 
     def set_k304(self):
         ck = gencookie("k304", self.uparam["k304"], 60 * 60 * 24 * 365)
-        self.out_headers["Set-Cookie"] = ck
+        self.out_headerlist.append(("Set-Cookie", ck))
+        self.redirect("", "?h#cc")
+
+    def set_am_js(self):
+        v = "n" if self.uparam["am_js"] == "n" else "y"
+        ck = gencookie("js", v, 60 * 60 * 24 * 365)
+        self.out_headerlist.append(("Set-Cookie", ck))
+        self.reply(b"promoted\n")
+
+    def set_cfg_reset(self):
+        for k in ("k304", "js", "cppwd"):
+            self.out_headerlist.append(("Set-Cookie", gencookie(k, "x", None)))
+
         self.redirect("", "?h#cc")
 
     def tx_404(self, is_403=False):
@@ -1939,6 +2019,13 @@ class HttpCli(object):
                 fmt = "{{}}  {{:{},}}  {{}}"
                 nfmt = "{:,}"
 
+            for x in dirs:
+                n = x["name"] + "/"
+                if arg == "v":
+                    n = "\033[94m" + n
+
+                x["name"] = n
+
             fmt = fmt.format(len(nfmt.format(biggest)))
             ret = [
                 "# {}: {}".format(x, ls[x])
@@ -1987,6 +2074,12 @@ class HttpCli(object):
         ):
             raise Pebkac(403)
 
+        self.html_head = vn.flags.get("html_head", "")
+        if vn.flags.get("norobots"):
+            self.out_headers["X-Robots-Tag"] = "noindex, nofollow"
+        else:
+            self.out_headers.pop("X-Robots-Tag", None)
+
         is_dir = stat.S_ISDIR(st.st_mode)
         if self.can_read:
             th_fmt = self.uparam.get("th")
@@ -2077,10 +2170,12 @@ class HttpCli(object):
 
         url_suf = self.urlq({}, [])
         is_ls = "ls" in self.uparam
+        is_js = self.args.force_js or self.cookies.get("js") == "y"
 
         tpl = "browser"
         if "b" in self.uparam:
             tpl = "browser2"
+            is_js = False
 
         logues = ["", ""]
         if not self.args.no_logues:
@@ -2105,6 +2200,7 @@ class HttpCli(object):
             "taglist": [],
             "srvinf": srv_info,
             "acct": self.uname,
+            "idx": ("e2d" in vn.flags),
             "perms": perms,
             "logues": logues,
             "readme": readme,
@@ -2113,6 +2209,7 @@ class HttpCli(object):
             "vdir": quotep(self.vpath),
             "vpnodes": vpnodes,
             "files": [],
+            "ls0": None,
             "acct": self.uname,
             "perms": json.dumps(perms),
             "taglist": [],
@@ -2177,10 +2274,6 @@ class HttpCli(object):
         if not self.args.ed or "dots" not in self.uparam:
             vfs_ls = exclude_dotfiles(vfs_ls)
 
-        hidden = []
-        if rem == ".hist":
-            hidden = ["up2k."]
-
         icur = None
         if "e2t" in vn.flags:
             idx = self.conn.get_u2idx()
@@ -2193,14 +2286,12 @@ class HttpCli(object):
         for fn in vfs_ls:
             base = ""
             href = fn
-            if not is_ls and not self.trailing_slash and vpath:
+            if not is_ls and not is_js and not self.trailing_slash and vpath:
                 base = "/" + vpath + "/"
                 href = base + fn
 
             if fn in vfs_virt:
                 fspath = vfs_virt[fn].realpath
-            elif hidden and any(fn.startswith(x) for x in hidden):
-                continue
             else:
                 fspath = fsroot + "/" + fn
 
@@ -2317,7 +2408,7 @@ class HttpCli(object):
 
         doc = self.uparam.get("doc") if self.can_read else None
         if doc:
-            doc = unquotep(doc.replace("+", " "))
+            doc = unquotep(doc.replace("+", " ").split("?")[0])
             j2a["docname"] = doc
             if next((x for x in files if x["name"] == doc), None):
                 with open(os.path.join(abspath, doc), "rb") as f:
@@ -2336,7 +2427,12 @@ class HttpCli(object):
 
         dirs.sort(key=itemgetter("name"))
 
-        j2a["files"] = dirs + files
+        if is_js:
+            j2a["ls0"] = {"dirs": dirs, "files": files, "taglist": taglist}
+            j2a["files"] = []
+        else:
+            j2a["files"] = dirs + files
+
         j2a["logues"] = logues
         j2a["taglist"] = taglist
         j2a["txt_ext"] = self.args.textfiles.replace(",", " ")

copyparty/mtag.py

@@ -418,7 +418,8 @@ class MTag(object):
 
         try:
             md = mutagen.File(fsenc(abspath), easy=True)
-            x = md.info.length
+            if not md.info.length and not md.info.codec:
+                raise Exception()
         except Exception as ex:
             return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 

copyparty/star.py

@@ -5,7 +5,7 @@ import tarfile
 import threading
 
 from .sutil import errdesc
-from .util import Queue, fsenc
+from .util import Queue, fsenc, min_ex
 from .bos import bos
 
 
@@ -88,8 +88,9 @@ class StreamTar(object):
 
             try:
                 self.ser(f)
-            except Exception as ex:
-                errors.append([f["vp"], repr(ex)])
+            except Exception:
+                ex = min_ex(5, True).replace("\n", "\n-- ")
+                errors.append([f["vp"], ex])
 
         if errors:
             self.errf, txt = errdesc(errors)

copyparty/svchub.py

@@ -105,6 +105,11 @@ class SvcHub(object):
 
         args.th_poke = min(args.th_poke, args.th_maxage, args.ac_maxage)
 
+        if args.ftp or args.ftps:
+            from .ftpd import Ftpd
+
+            self.ftpd = Ftpd(self)
+
         # decide which worker impl to use
         if self.check_mp_enable():
             from .broker_mp import BrokerMp as Broker
@@ -302,6 +307,10 @@ class SvcHub(object):
             print("nailed it", end="")
             ret = self.retcode
         finally:
+            if self.args.wintitle:
+                print("\033]0;\033\\", file=sys.stderr, end="")
+                sys.stderr.flush()
+
             print("\033[0m")
             if self.logf:
                 self.logf.close()
@@ -353,7 +362,7 @@ class SvcHub(object):
                     src = ansi_re.sub("", src)
             elif c:
                 if isinstance(c, int):
-                    msg = "\033[3{}m{}".format(c, msg)
+                    msg = "\033[3{}m{}\033[0m".format(c, msg)
                 elif "\033" not in c:
                     msg = "\033[{}m{}\033[0m".format(c, msg)
                 else:

copyparty/szip.py

@@ -1,13 +1,12 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import os
 import time
 import zlib
 from datetime import datetime
 
 from .sutil import errdesc
-from .util import yieldfile, sanitize_fn, spack, sunpack
+from .util import yieldfile, sanitize_fn, spack, sunpack, min_ex
 from .bos import bos
 
 
@@ -36,7 +35,10 @@ def unixtime2dos(ts):
 
     bd = ((dy - 1980) << 9) + (dm << 5) + dd
     bt = (th << 11) + (tm << 5) + ts // 2
-    return spack(b"<HH", bt, bd)
+    try:
+        return spack(b"<HH", bt, bd)
+    except:
+        return b"\x00\x00\x21\x00"
 
 
 def gen_fdesc(sz, crc32, z64):
@@ -244,8 +246,9 @@ class StreamZip(object):
             try:
                 for x in self.ser(f):
                     yield x
-            except Exception as ex:
-                errors.append([f["vp"], repr(ex)])
+            except Exception:
+                ex = min_ex(5, True).replace("\n", "\n-- ")
+                errors.append([f["vp"], ex])
 
         if errors:
             errf, txt = errdesc(errors)

copyparty/tcpsrv.py

@@ -2,9 +2,10 @@
 from __future__ import print_function, unicode_literals
 
 import re
+import sys
 import socket
 
-from .__init__ import MACOS, ANYWIN
+from .__init__ import MACOS, ANYWIN, unicode
 from .util import chkcmd
 
 
@@ -54,19 +55,55 @@ class TcpSrv(object):
                     eps[x] = "external"
 
         msgs = []
-        m = "available @ http://{}:{}/  (\033[33m{}\033[0m)"
+        title_tab = {}
+        title_vars = [x[1:] for x in self.args.wintitle.split(" ") if x.startswith("$")]
+        m = "available @ {}://{}:{}/  (\033[33m{}\033[0m)"
         for ip, desc in sorted(eps.items(), key=lambda x: x[1]):
             for port in sorted(self.args.p):
                 if port not in ok.get(ip, ok.get("0.0.0.0", [])):
                     continue
 
-                msgs.append(m.format(ip, port, desc))
+                proto = " http"
+                if self.args.http_only:
+                    pass
+                elif self.args.https_only or port == 443:
+                    proto = "https"
+
+                msgs.append(m.format(proto, ip, port, desc))
+
+                if not self.args.wintitle:
+                    continue
+
+                if port in [80, 443]:
+                    ep = ip
+                else:
+                    ep = "{}:{}".format(ip, port)
+
+                hits = []
+                if "pub" in title_vars and "external" in unicode(desc):
+                    hits.append(("pub", ep))
+
+                if "pub" in title_vars or "all" in title_vars:
+                    hits.append(("all", ep))
+
+                for var in title_vars:
+                    if var.startswith("ip-") and ep.startswith(var[3:]):
+                        hits.append((var, ep))
+
+                for tk, tv in hits:
+                    try:
+                        title_tab[tk][tv] = 1
+                    except:
+                        title_tab[tk] = {tv: 1}
 
         if msgs:
             msgs[-1] += "\n"
             for m in msgs:
                 self.log("tcpsrv", m)
 
+        if self.args.wintitle:
+            self._set_wintitle(title_tab)
+
     def _listen(self, ip, port):
         srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
@@ -113,10 +150,15 @@ class TcpSrv(object):
             return eps
 
         r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
+        ri = re.compile(r"^\s*[0-9]+\s*:.*")
+        up = False
         for ln in txt.split("\n"):
+            if ri.match(ln):
+                up = "UP" in re.split("[>,< ]", ln)
+
             try:
                 ip, dev = r.match(ln.rstrip()).groups()
-                eps[ip] = dev
+                eps[ip] = dev + ("" if up else ", \033[31mLINK-DOWN")
             except:
                 pass
 
@@ -146,6 +188,7 @@ class TcpSrv(object):
 
     def ips_windows_ipconfig(self):
         eps = {}
+        offs = {}
         try:
             txt, _ = chkcmd(["ipconfig"])
         except:
@@ -153,18 +196,29 @@ class TcpSrv(object):
 
         rdev = re.compile(r"(^[^ ].*):$")
         rip = re.compile(r"^ +IPv?4? [^:]+: *([0-9\.]{7,15})$")
+        roff = re.compile(r".*: Media disconnected$")
         dev = None
         for ln in txt.replace("\r", "").split("\n"):
             m = rdev.match(ln)
             if m:
+                if dev and dev not in eps.values():
+                    offs[dev] = 1
+
                 dev = m.group(1).split(" adapter ", 1)[-1]
 
+            if dev and roff.match(ln):
+                offs[dev] = 1
+                dev = None
+
             m = rip.match(ln)
             if m and dev:
                 eps[m.group(1)] = dev
                 dev = None
 
-        return eps
+        if dev and dev not in eps.values():
+            offs[dev] = 1
+
+        return eps, offs
 
     def ips_windows_netsh(self):
         eps = {}
@@ -184,7 +238,6 @@ class TcpSrv(object):
             m = rip.match(ln)
             if m and dev:
                 eps[m.group(1)] = dev
-                dev = None
 
         return eps
 
@@ -192,8 +245,11 @@ class TcpSrv(object):
         if MACOS:
             eps = self.ips_macos()
         elif ANYWIN:
-            eps = self.ips_windows_ipconfig()  # sees more interfaces
+            eps, off = self.ips_windows_ipconfig()  # sees more interfaces + link state
             eps.update(self.ips_windows_netsh())  # has better names
+            for k, v in eps.items():
+                if v in off:
+                    eps[k] += ", \033[31mLINK-DOWN"
         else:
             eps = self.ips_linux()
 
@@ -232,3 +288,26 @@ class TcpSrv(object):
                     eps[default_route] = desc
 
         return eps
+
+    def _set_wintitle(self, vars):
+        vars["all"] = vars.get("all", {"Local-Only": 1})
+        vars["pub"] = vars.get("pub", vars["all"])
+
+        vars2 = {}
+        for k, eps in vars.items():
+            vars2[k] = {
+                ep: 1
+                for ep in eps.keys()
+                if ":" not in ep or ep.split(":")[0] not in eps
+            }
+
+        title = ""
+        vars = vars2
+        for p in self.args.wintitle.split(" "):
+            if p.startswith("$"):
+                p = " and ".join(sorted(vars.get(p[1:], {"(None)": 1}).keys()))
+
+            title += "{} ".format(p)
+
+        print("\033]0;{}\033\\".format(title), file=sys.stderr, end="")
+        sys.stderr.flush()

copyparty/u2idx.py

@@ -51,11 +51,11 @@ class U2idx(object):
         fhash = body["hash"]
         wark = up2k_wark_from_hashlist(self.args.salt, fsize, fhash)
 
-        uq = "where substr(w,1,16) = ? and w = ?"
+        uq = "substr(w,1,16) = ? and w = ?"
         uv = [wark[:16], wark]
 
         try:
-            return self.run_query(vols, uq, uv)[0]
+            return self.run_query(vols, uq, uv, True, False)[0]
         except:
             raise Pebkac(500, min_ex())
 
@@ -87,17 +87,16 @@ class U2idx(object):
 
         q = ""
         va = []
-        joins = ""
+        have_up = False  # query has up.* operands
+        have_mt = False
         is_key = True
         is_size = False
         is_date = False
+        field_end = ""  # closing parenthesis or whatever
         kw_key = ["(", ")", "and ", "or ", "not "]
         kw_val = ["==", "=", "!=", ">", ">=", "<", "<=", "like "]
         ptn_mt = re.compile(r"^\.?[a-z_-]+$")
-        mt_ctr = 0
-        mt_keycmp = "substr(up.w,1,16)"
-        mt_keycmp2 = None
-        ptn_lc = re.compile(r" (mt[0-9]+\.v) ([=<!>]+) \? $")
+        ptn_lc = re.compile(r" (mt\.v) ([=<!>]+) \? \) $")
         ptn_lcv = re.compile(r"[a-zA-Z]")
 
         while True:
@@ -117,35 +116,47 @@ class U2idx(object):
             if ok:
                 continue
 
-            v, uq = (uq + " ").split(" ", 1)
+            if uq.startswith('"'):
+                v, uq = uq[1:].split('"', 1)
+                while v.endswith("\\"):
+                    v2, uq = uq.split('"', 1)
+                    v = v[:-1] + '"' + v2
+                uq = uq.strip()
+            else:
+                v, uq = (uq + " ").split(" ", 1)
+                v = v.replace('\\"', '"')
+
             if is_key:
                 is_key = False
 
                 if v == "size":
                     v = "up.sz"
                     is_size = True
+                    have_up = True
 
                 elif v == "date":
                     v = "up.mt"
                     is_date = True
+                    have_up = True
 
                 elif v == "path":
-                    v = "up.rd"
+                    v = "trim(?||up.rd,'/')"
+                    va.append("\nrd")
+                    have_up = True
 
                 elif v == "name":
                     v = "up.fn"
+                    have_up = True
 
                 elif v == "tags" or ptn_mt.match(v):
-                    mt_ctr += 1
-                    mt_keycmp2 = "mt{}.w".format(mt_ctr)
-                    joins += "inner join mt mt{} on {} = {} ".format(
-                        mt_ctr, mt_keycmp, mt_keycmp2
-                    )
-                    mt_keycmp = mt_keycmp2
+                    have_mt = True
+                    field_end = ") "
                     if v == "tags":
-                        v = "mt{0}.v".format(mt_ctr)
+                        vq = "mt.v"
                     else:
-                        v = "+mt{0}.k = '{1}' and mt{0}.v".format(mt_ctr, v)
+                        vq = "+mt.k = '{}' and mt.v".format(v)
+
+                    v = "exists(select 1 from mt where mt.w = mtw and " + vq
 
                 else:
                     raise Pebkac(400, "invalid key [" + v + "]")
@@ -191,6 +202,10 @@ class U2idx(object):
             va.append(v)
             is_key = True
 
+            if field_end:
+                q += field_end
+                field_end = ""
+
             # lowercase tag searches
             m = ptn_lc.search(q)
             if not m or not ptn_lcv.search(unicode(v)):
@@ -202,16 +217,16 @@ class U2idx(object):
 
             field, oper = m.groups()
             if oper in ["=", "=="]:
-                q += " {} like ? ".format(field)
+                q += " {} like ? ) ".format(field)
             else:
-                q += " lower({}) {} ? ".format(field, oper)
+                q += " lower({}) {} ? ) ".format(field, oper)
 
         try:
-            return self.run_query(vols, joins + "where " + q, va)
+            return self.run_query(vols, q, va, have_up, have_mt)
         except Exception as ex:
             raise Pebkac(500, repr(ex))
 
-    def run_query(self, vols, uq, uv):
+    def run_query(self, vols, uq, uv, have_up, have_mt):
         done_flag = []
         self.active_id = "{:.6f}_{}".format(
             time.time(), threading.current_thread().ident
@@ -228,16 +243,19 @@ class U2idx(object):
         thr.start()
 
         if not uq or not uv:
-            q = "select * from up"
-            v = ()
+            uq = "select * from up"
+            uv = ()
+        elif have_mt:
+            uq = "select up.*, substr(up.w,1,16) mtw from up where " + uq
+            uv = tuple(uv)
         else:
-            q = "select up.* from up " + uq
-            v = tuple(uv)
+            uq = "select up.* from up where " + uq
+            uv = tuple(uv)
 
-        self.log("qs: {!r} {!r}".format(q, v))
+        self.log("qs: {!r} {!r}".format(uq, uv))
 
         ret = []
-        lim = 1000
+        lim = int(self.args.srch_hits)
         taglist = {}
         for (vtop, ptop, flags) in vols:
             cur = self.get_cur(ptop)
@@ -246,11 +264,19 @@ class U2idx(object):
 
             self.active_cur = cur
 
+            vuv = []
+            for v in uv:
+                if v == "\nrd":
+                    v = vtop + "/"
+
+                vuv.append(v)
+            vuv = tuple(vuv)
+
             sret = []
             fk = flags.get("fk")
-            c = cur.execute(q, v)
+            c = cur.execute(uq, vuv)
             for hit in c:
-                w, ts, sz, rd, fn, ip, at = hit
+                w, ts, sz, rd, fn, ip, at = hit[:7]
                 lim -= 1
                 if lim <= 0:
                     break

copyparty/up2k.py

@@ -21,6 +21,7 @@ from .util import (
     Pebkac,
     Queue,
     ProgressPrinter,
+    SYMTIME,
     fsdec,
     fsenc,
     absreal,
@@ -1136,9 +1137,9 @@ class Up2k(object):
             m = "database is version {}, this copyparty only supports versions <= {}"
             raise Exception(m.format(ver, DB_VER))
 
-        msg = "creating new DB (old is bad); backup: {}"
+        msg = "creating new DB (old is bad); backup: "
         if ver:
-            msg = "creating new DB (too old to upgrade); backup: {}"
+            msg = "creating new DB (too old to upgrade); backup: "
 
         cur = self._backup_db(db_path, cur, ver, msg)
         db = cur.connection
@@ -1307,11 +1308,14 @@ class Up2k(object):
                         err = "partial upload exists at a different location; please resume uploading here instead:\n"
                         err += "/" + quotep(vsrc) + " "
 
-                        dupe = [cj["prel"], cj["name"]]
-                        try:
-                            self.dupesched[src].append(dupe)
-                        except:
-                            self.dupesched[src] = [dupe]
+                        # registry is size-constrained + can only contain one unique wark;
+                        # let want_recheck trigger symlink (if still in reg) or reupload
+                        if cur:
+                            dupe = [cj["prel"], cj["name"], cj["lmod"]]
+                            try:
+                                self.dupesched[src].append(dupe)
+                            except:
+                                self.dupesched[src] = [dupe]
 
                         raise Pebkac(400, err)
 
@@ -1332,7 +1336,7 @@ class Up2k(object):
                         dst = os.path.join(job["ptop"], job["prel"], job["name"])
                         if not self.args.nw:
                             bos.unlink(dst)  # TODO ed pls
-                            self._symlink(src, dst)
+                            self._symlink(src, dst, lmod=cj["lmod"])
 
                         if cur:
                             a = [cj[x] for x in "prel name lmod size addr".split()]
@@ -1404,13 +1408,14 @@ class Up2k(object):
         with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as f:
             return f["orz"][1]
 
-    def _symlink(self, src, dst, verbose=True):
+    def _symlink(self, src, dst, verbose=True, lmod=None):
         if verbose:
             self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
 
         if self.args.nw:
             return
 
+        linked = False
         try:
             if self.args.no_symlink:
                 raise Exception("disabled in config")
@@ -1441,10 +1446,18 @@ class Up2k(object):
                     hops = len(ndst[nc:]) - 1
                     lsrc = "../" * hops + "/".join(lsrc)
             os.symlink(fsenc(lsrc), fsenc(ldst))
+            linked = True
         except Exception as ex:
             self.log("cannot symlink; creating copy: " + repr(ex))
             shutil.copy2(fsenc(src), fsenc(dst))
 
+        if lmod and (not linked or SYMTIME):
+            times = (int(time.time()), int(lmod))
+            if ANYWIN:
+                self.lastmod_q.put([dst, 0, times])
+            else:
+                bos.utime(dst, times, False)
+
     def handle_chunk(self, ptop, wark, chash):
         with self.mutex:
             job = self.registry[ptop].get(wark)
@@ -1551,12 +1564,12 @@ class Up2k(object):
             return
 
         cur = self.cur.get(ptop)
-        for rd, fn in dupes:
+        for rd, fn, lmod in dupes:
             d2 = os.path.join(ptop, rd, fn)
             if os.path.exists(d2):
                 continue
 
-            self._symlink(dst, d2)
+            self._symlink(dst, d2, lmod=lmod)
             if cur:
                 self.db_rm(cur, rd, fn)
                 self.db_add(cur, wark, rd, fn, *a[-4:])
@@ -1773,8 +1786,9 @@ class Up2k(object):
             dlabs = absreal(sabs)
             m = "moving symlink from [{}] to [{}], target [{}]"
             self.log(m.format(sabs, dabs, dlabs))
-            os.unlink(sabs)
-            self._symlink(dlabs, dabs, False)
+            mt = bos.path.getmtime(sabs, False)
+            bos.unlink(sabs)
+            self._symlink(dlabs, dabs, False, lmod=mt)
 
             # folders are too scary, schedule rescan of both vols
             self.need_rescan[svn.vpath] = 1
@@ -1904,25 +1918,30 @@ class Up2k(object):
             slabs = list(sorted(links.keys()))[0]
             ptop, rem = links.pop(slabs)
             self.log("linkswap [{}] and [{}]".format(sabs, slabs))
+            mt = bos.path.getmtime(slabs, False)
             bos.unlink(slabs)
             bos.rename(sabs, slabs)
+            bos.utime(slabs, (int(time.time()), int(mt)), False)
             self._symlink(slabs, sabs, False)
             full[slabs] = [ptop, rem]
+            sabs = slabs
 
         if not dabs:
             dabs = list(sorted(full.keys()))[0]
 
         for alink in links.keys():
+            lmod = None
             try:
                 if alink != sabs and absreal(alink) != sabs:
                     continue
 
                 self.log("relinking [{}] to [{}]".format(alink, dabs))
+                lmod = bos.path.getmtime(alink, False)
                 bos.unlink(alink)
             except:
                 pass
 
-            self._symlink(dabs, alink, False)
+            self._symlink(dabs, alink, False, lmod=lmod)
 
         return len(full) + len(links)
 
@@ -2028,7 +2047,7 @@ class Up2k(object):
             for path, sz, times in ready:
                 self.log("lmod: setting times {} on {}".format(times, path))
                 try:
-                    bos.utime(path, times)
+                    bos.utime(path, times, False)
                 except:
                     self.log("lmod: failed to utime ({}, {})".format(path, times))
 

copyparty/util.py

@@ -67,8 +67,11 @@ if WINDOWS and PY2:
     FS_ENCODING = "utf-8"
 
 
+SYMTIME = sys.version_info >= (3, 6) and os.supports_follow_symlinks
+
 HTTP_TS_FMT = "%a, %d %b %Y %H:%M:%S GMT"
 
+META_NOBOTS = '<meta name="robots" content="noindex, nofollow">'
 
 HTTPCODE = {
     200: "OK",
@@ -482,13 +485,13 @@ def vol_san(vols, txt):
     return txt
 
 
-def min_ex():
+def min_ex(max_lines=8, reverse=False):
     et, ev, tb = sys.exc_info()
     tb = traceback.extract_tb(tb)
     fmt = "{} @ {} <{}>: {}"
     ex = [fmt.format(fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in tb]
     ex.append("[{}] {}".format(et.__name__, ev))
-    return "\n".join(ex[-8:])
+    return "\n".join(ex[-max_lines:][:: -1 if reverse else 1])
 
 
 @contextlib.contextmanager
@@ -1096,7 +1099,8 @@ def read_socket(sr, total_size):
 
         buf = sr.recv(bufsz)
         if not buf:
-            raise Pebkac(400, "client d/c during binary post")
+            m = "client d/c during binary post after {} bytes, {} bytes remaining"
+            raise Pebkac(400, m.format(total_size - remains, remains))
 
         remains -= len(buf)
         yield buf
@@ -1162,13 +1166,15 @@ def yieldfile(fn):
             yield buf
 
 
-def hashcopy(fin, fout):
+def hashcopy(fin, fout, slp=0):
     hashobj = hashlib.sha512()
     tlen = 0
     for buf in fin:
         tlen += len(buf)
         hashobj.update(buf)
         fout.write(buf)
+        if slp:
+            time.sleep(slp)
 
     digest = hashobj.digest()[:33]
     digest_b64 = base64.urlsafe_b64encode(digest).decode("utf-8")
@@ -1176,7 +1182,7 @@ def hashcopy(fin, fout):
     return tlen, hashobj.hexdigest(), digest_b64
 
 
-def sendfile_py(lower, upper, f, s, bufsz, slp):
+def sendfile_py(log, lower, upper, f, s, bufsz, slp):
     remains = upper - lower
     f.seek(lower)
     while remains > 0:
@@ -1196,17 +1202,24 @@ def sendfile_py(lower, upper, f, s, bufsz, slp):
     return 0
 
 
-def sendfile_kern(lower, upper, f, s, bufsz, slp):
+def sendfile_kern(log, lower, upper, f, s, bufsz, slp):
     out_fd = s.fileno()
     in_fd = f.fileno()
     ofs = lower
+    stuck = None
     while ofs < upper:
+        stuck = stuck or time.time()
         try:
             req = min(2 ** 30, upper - ofs)
             select.select([], [out_fd], [], 10)
             n = os.sendfile(out_fd, in_fd, ofs, req)
+            stuck = None
         except Exception as ex:
-            # print("sendfile: " + repr(ex))
+            d = time.time() - stuck
+            log("sendfile stuck for {:.3f} sec: {!r}".format(d, ex))
+            if d < 3600 and ex.errno == 11:  # eagain
+                continue
+
             n = 0
 
         if n <= 0:

copyparty/web/browser.css

@@ -37,7 +37,7 @@ pre, code, tt, #doc, #doc>code {
 	display: inline-block;
 	padding: .35em .5em .2em .5em;
 	border-radius: 0 .3em .3em 0;
-	margin: 1.3em 0 0 0;
+	margin: 1.3em 0 -.2em 0;
 	font-size: 1.4em;
 }
 #path #entree {
@@ -51,7 +51,8 @@ pre, code, tt, #doc, #doc>code {
 }
 #files tbody a {
 	display: block;
-	padding: .3em 0;
+	padding: .5em 0;
+	margin: -.3em 0;
 	scroll-margin-top: 45vh;
 }
 #files tr {
@@ -79,6 +80,27 @@ a, #files tbody div a:last-child {
 	color: #999;
 	font-weight: normal;
 }
+.s0:after,
+.s1:after {
+	content: '⌄';
+	margin-left: -.1em;
+}
+.s0r:after,
+.s1r:after {
+	content: '⌃';
+	margin-left: -.1em;
+}
+.s0:after,
+.s0r:after {
+	color: #fb0;
+}
+.s1:after,
+.s1r:after {
+	color: #d09;
+}
+#files thead th:after {
+	margin-right: -.7em;
+}
 #files tbody tr:hover td {
 	background: #1c1c1c;
 }
@@ -89,7 +111,7 @@ a, #files tbody div a:last-child {
 }
 #files td {
 	margin: 0;
-	padding: .1em .5em;
+	padding: .3em .5em;
 	border-left: 1px solid #3c3c3c;
 }
 #files td+td+td {
@@ -213,8 +235,8 @@ a, #files tbody div a:last-child {
 }
 #files tbody a.play {
 	color: #e70;
-	padding: .2em;
-	margin: -.2em;
+	padding: .3em;
+	margin: -.3em;
 }
 #files tbody a.play.act {
 	color: #720;
@@ -240,6 +262,8 @@ html.light #ggrid>a[tt].sel {
 #files tbody tr.sel:hover td,
 #files tbody tr.sel:focus td,
 #ggrid>a.sel:hover,
+#ggrid>a.sel:focus,
+html.light #ggrid>a.sel:focus,
 html.light #ggrid>a.sel:hover {
 	color: #fff;
 	background: #d39;
@@ -295,6 +319,8 @@ html.light #ggrid>a.sel {
 	width: 100%;
 	z-index: 3;
 	touch-action: none;
+}
+#widget.anim {
 	transition: bottom 0.15s;
 }
 #widget.open {
@@ -401,6 +427,9 @@ html.light #ggrid>a.sel {
 	opacity: .3;
 	color: #f6c;
 }
+#wfm a.hide {
+	display: none;
+}
 html.light #wfm a:not(.en) {
 	color: #c4a;
 }
@@ -462,7 +491,7 @@ html.light #wfm a:not(.en) {
 	width: calc(100% - 10.5em);
 	background: rgba(0,0,0,0.2);
 }
-@media (min-width: 80em) {
+@media (min-width: 70em) {
 	#barpos,
 	#barbuf {
 		width: calc(100% - 21em);
@@ -654,7 +683,7 @@ input.eq_gain {
 #wrap {
 	margin: 1.8em 1.5em 0 1.5em;
 	min-height: 70vh;
-	padding-bottom: 5em;
+	padding-bottom: 7em;
 }
 #tree {
 	display: none;
@@ -948,6 +977,12 @@ html.light .ghead {
 #ggrid>a.dir:before {
 	content: '📂';
 }
+#ggrid>a.au:before {
+	content: '💾';
+}
+html.np_open #ggrid>a.au:before {
+	content: '▶';
+}
 #ggrid>a:before {
 	display: block;
 	position: absolute;
@@ -957,6 +992,12 @@ html.light .ghead {
 	background: linear-gradient(135deg,rgba(255,255,255,0) 50%,rgba(255,255,255,0.2));
 	border-radius: .3em;
 	font-size: 2em;
+	transition: font-size .15s, margin .15s;
+}
+#ggrid>a:focus:before,
+#ggrid>a:hover:before {
+	font-size: 2.5em;
+	margin: -.2em;
 }
 #op_unpost {
 	padding: 1em;
@@ -1026,7 +1067,6 @@ html.light #rui {
 	font-size: 1.5em;
 }
 #doc {
-	background: none;
 	overflow: visible;
 	margin: -1em 0 .5em 0;
 	padding: 1em 0 1em 0;
@@ -1065,7 +1105,7 @@ html.light #doc .line-highlight {
 #docul li {
 	margin: 0;
 }
-#tree #docul a {
+#tree #docul li+li a {
 	display: block;
 }
 #seldoc.sel {
@@ -1114,6 +1154,7 @@ a.btn,
 
 
 html,
+#doc,
 #rui,
 #files td,
 #files thead th,
@@ -1161,6 +1202,7 @@ html,
 #ggrid>a[tt] {
 	background: linear-gradient(135deg, #2c2c2c 95%, #444 95%);
 }
+#ggrid>a:focus,
 #ggrid>a:hover {
 	background: #383838;
 	border-color: #555;
@@ -1174,6 +1216,7 @@ html.light #ggrid>a {
 html.light #ggrid>a[tt] {
 	background: linear-gradient(135deg, #f7f7f7 95%, #ccc 95%);
 }
+html.light #ggrid>a:focus,
 html.light #ggrid>a:hover {
 	background: #fff;
 	border-color: #ccc;
@@ -1193,6 +1236,7 @@ html.light {
 html.light #ops,
 html.light .opbox,
 html.light #path,
+html.light #doc,
 html.light #srch_form,
 html.light .ghead,
 html.light #u2etas {
@@ -1270,6 +1314,14 @@ html.light #ops a,
 html.light #files tbody div a:last-child {
 	color: #06a;
 }
+html.light .s0:after,
+html.light .s0r:after {
+	color: #059;
+}
+html.light .s1:after,
+html.light .s1r:after {
+	color: #f5d;
+}
 html.light #files thead th {
 	background: #eaeaea;
 	border-color: #ccc;
@@ -1376,6 +1428,7 @@ html.light .opview input[type="text"] {
 	border-color: #38d;
 }
 html.light #u2tab a>span,
+html.light #docul .bn a>span,
 html.light #files td div span {
 	color: #000;
 }
@@ -1833,12 +1886,13 @@ html.light #u2err.err {
 #u2tabw {
 	min-height: 0;
 	transition: min-height .2s;
-	margin: 3em auto;
+	margin: 3em 0;
 }
 #u2tab {
 	border-collapse: collapse;
 	width: calc(100% - 2em);
 	max-width: 100em;
+	margin: 0 auto;
 }
 #op_up2k.srch #u2tabf {
 	max-width: none;
@@ -2100,6 +2154,7 @@ html.light #u2foot .warn span {
 	border-color: #d06;
 }
 #u2tab a>span,
+#docul .bn a>span,
 #unpost a>span {
 	font-weight: bold;
 	font-style: italic;

copyparty/web/browser.html

@@ -6,6 +6,7 @@
 	<title>⇆🎉 {{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+{{ html_head }}
 	<link rel="stylesheet" media="screen" href="/.cpr/ui.css?_={{ ts }}">
 	<link rel="stylesheet" media="screen" href="/.cpr/browser.css?_={{ ts }}">
 	{%- if css %}
@@ -143,7 +144,8 @@
 			have_zip = {{ have_zip|tojson }},
 			txt_ext = "{{ txt_ext }}",
 			{% if no_prism %}no_prism = 1,{% endif %}
-			readme = {{ readme|tojson }};
+			readme = {{ readme|tojson }},
+			ls0 = {{ ls0|tojson }};
 
 		document.documentElement.setAttribute("class", localStorage.lightmode == 1 ? "light" : "dark");
 	</script>

copyparty/web/browser.js

@@ -8,13 +8,9 @@ function dbg(msg) {
 // toolbar
 ebi('ops').innerHTML = (
 	'<a href="#" data-dest="" tt="close submenu">--</a>\n' +
-	(have_up2k_idx ? (
-		'<a href="#" data-perm="read" data-dest="search" tt="search for files by attributes, path/name, music tags, or any combination of those$N$N&lt;code&gt;foo bar&lt;/code&gt; = must contain both foo and bar,$N&lt;code&gt;foo -bar&lt;/code&gt; = must contain foo but not bar,$N&lt;code&gt;^yana .opus$&lt;/code&gt; = must start with yana and have the opus extension">🔎</a>\n' +
-		(have_del && have_unpost ? '<a href="#" data-dest="unpost" tt="unpost: delete your recent uploads">🧯</a>\n' : '') +
-		'<a href="#" data-dest="up2k" tt="up2k: upload files (if you have write-access) or toggle into the search-mode to see if they exist somewhere on the server">🚀</a>\n'
-	) : (
-		'<a href="#" data-perm="write" data-dest="up2k" tt="up2k: upload files with resume support (close your browser and drop the same files in later)">🚀</a>\n'
-	)) +
+	'<a href="#" data-perm="read" data-dep="idx" data-dest="search" tt="search for files by attributes, path/name, music tags, or any combination of those$N$N&lt;code&gt;foo bar&lt;/code&gt; = must contain both foo and bar,$N&lt;code&gt;foo -bar&lt;/code&gt; = must contain foo but not bar,$N&lt;code&gt;^yana .opus$&lt;/code&gt; = start with yana and be an opus file$N&lt;code&gt;&quot;try unite&quot;&lt;/code&gt; = contain exactly «try unite»">🔎</a>\n' +
+	(have_del && have_unpost ? '<a href="#" data-dest="unpost" data-dep="idx" tt="unpost: delete your recent uploads">🧯</a>\n' : '') +
+	'<a href="#" data-dest="up2k">🚀</a>\n' +
 	'<a href="#" data-perm="write" data-dest="bup" tt="bup: basic uploader, even supports netscape 4.0">🎈</a>\n' +
 	'<a href="#" data-perm="write" data-dest="mkdir" tt="mkdir: create a new directory">📂</a>\n' +
 	'<a href="#" data-perm="read write" data-dest="new_md" tt="new-md: create a new markdown document">📝</a>\n' +
@@ -68,12 +64,10 @@ ebi('op_up2k').innerHTML = (
 	'			<input type="checkbox" id="ask_up" />\n' +
 	'			<label for="ask_up" tt="ask for confirmation before upload starts">💭</label>\n' +
 	'		</td>\n' +
-	(have_up2k_idx ? (
-		'		<td class="c" data-perm="read" rowspan="2">\n' +
-		'			<input type="checkbox" id="fsearch" />\n' +
-		'			<label for="fsearch" tt="don\'t actually upload, instead check if the files already $N exist on the server (will scan all folders you can read)">🔎</label>\n' +
-		'		</td>\n'
-	) : '') +
+	'		<td class="c" data-perm="read" data-dep="idx" rowspan="2">\n' +
+	'			<input type="checkbox" id="fsearch" />\n' +
+	'			<label for="fsearch" tt="don\'t actually upload, instead check if the files already $N exist on the server (will scan all folders you can read)">🔎</label>\n' +
+	'		</td>\n' +
 	'		<td data-perm="read" rowspan="2" id="u2btn_cw"></td>\n' +
 	'		<td data-perm="read" rowspan="2" id="u2c3w"></td>\n' +
 	'	</tr>\n' +
@@ -157,6 +151,7 @@ ebi('op_cfg').innerHTML = (
 	'		<a id="thumbs" class="tgl btn" href="#" tt="in icon view, toggle icons or thumbnails$NHotkey: T">🖼️ thumbs</a>\n' +
 	'		<a id="dotfiles" class="tgl btn" href="#" tt="show hidden files (if server permits)">dotfiles</a>\n' +
 	'		<a id="ireadme" class="tgl btn" href="#" tt="show README.md in folder listings">📜 readme</a>\n' +
+	'		<a id="spafiles" class="tgl btn" href="#" tt="speedboost when not using the navpane;$Nturn it off if things arent loading somehow">spa</a>\n' +
 	'	</div>\n' +
 	'</div>\n' +
 	(have_zip ? (
@@ -209,6 +204,9 @@ ebi('tree').innerHTML = (
 	var ops = QSA('#ops>a');
 	for (var a = 0; a < ops.length; a++) {
 		ops[a].onclick = opclick;
+		var v = ops[a].getAttribute('data-dest');
+		if (v)
+			ops[a].href = '#v=' + v;
 	}
 })();
 
@@ -306,7 +304,9 @@ function set_files_html(html) {
 
 
 var ACtx = window.AudioContext || window.webkitAudioContext,
-	actx = ACtx && new ACtx();
+	actx = ACtx && new ACtx(),
+	hash0 = location.hash,
+	mp;
 
 
 var mpl = (function () {
@@ -539,6 +539,7 @@ function MPlayer() {
 			r.tracks[tid] = url;
 			tds[0].innerHTML = '<a id="a' + tid + '" href="#a' + tid + '" class="play">play</a></td>';
 			ebi('a' + tid).onclick = ev_play;
+			clmod(trs[a], 'au', 1);
 		}
 	}
 
@@ -651,14 +652,11 @@ function MPlayer() {
 	};
 }
 
-addcrc();
-var mp = new MPlayer();
-makeSortable(ebi('files'), mp.read_order.bind(mp));
-
 
 function ft2dict(tr) {
 	var th = ebi('files').tHead.rows[0].cells,
 		rv = [],
+		rh = [],
 		ra = [],
 		rt = {};
 
@@ -670,10 +668,11 @@ function ft2dict(tr) {
 		if (!tv)
 			continue;
 
-		(vis ? rv : ra).push(tk);
+		(vis ? rv : rh).push(tk);
+		ra.push(tk);
 		rt[tk] = tv;
 	}
-	return [rt, rv, ra];
+	return [rt, rv, rh, ra];
 }
 
 
@@ -693,24 +692,19 @@ var widget = (function () {
 		touchmode = false,
 		was_paused = true;
 
-	r.is_open = false;
-
 	r.open = function () {
-		if (r.is_open)
-			return false;
-
-		clmod(document.documentElement, 'np_open', 1);
-		widget.className = 'open';
-		r.is_open = true;
-		return true;
+		return r.set(true);
 	};
 	r.close = function () {
-		if (!r.is_open)
+		return r.set(false);
+	};
+	r.set = function (is_open) {
+		if (r.is_open == is_open)
 			return false;
 
-		clmod(document.documentElement, 'np_open');
-		widget.className = '';
-		r.is_open = false;
+		clmod(document.documentElement, 'np_open', is_open);
+		widget.className = is_open ? 'open' : '';
+		bcfg_set('au_open', r.is_open = is_open);
 		return true;
 	};
 	r.toggle = function (e) {
@@ -757,6 +751,10 @@ var widget = (function () {
 			document.body.removeChild(o);
 		}, 500);
 	};
+	r.set(sread('au_open') == 1);
+	setTimeout(function () {
+		clmod(ebi('widget'), 'anim', 1);
+	}, 10);
 	return r;
 })();
 
@@ -808,7 +806,7 @@ var pbar = (function () {
 
 		bctx.clearRect(0, 0, bc.w, bc.h);
 
-		if (!mp.au)
+		if (!mp || !mp.au)
 			return;
 
 		var sm = bc.w * 1.0 / mp.au.duration,
@@ -835,7 +833,7 @@ var pbar = (function () {
 
 		pctx.clearRect(0, 0, pc.w, pc.h);
 
-		if (!mp.au || isNaN(adur = mp.au.duration) || isNaN(apos = mp.au.currentTime) || apos < 0 || adur < apos)
+		if (!mp || !mp.au || isNaN(adur = mp.au.duration) || isNaN(apos = mp.au.currentTime) || apos < 0 || adur < apos)
 			return;  // not-init || unsupp-codec
 
 		var sm = bc.w * 1.0 / adur;
@@ -906,6 +904,9 @@ var vbar = (function () {
 	}
 
 	r.draw = function () {
+		if (!mp)
+			return;
+
 		var gh = h + '' + light;
 		if (gradh != gh) {
 			gradh = gh;
@@ -1132,7 +1133,6 @@ var mpui = (function () {
 		if (mp.au.paused)
 			timer.rm(updater_impl);
 	}
-	r.progress_updater();
 	return r;
 })();
 
@@ -1343,7 +1343,7 @@ var audio_eq = (function () {
 	}
 
 	var html = ['<table><tr><td rowspan="4">',
-		'<a id="au_eq" class="tgl btn" href="#" tt="enables the equalizer and gain control;$Nboost 0 = unmodified 100% volume">enable</a></td>'],
+		'<a id="au_eq" class="tgl btn" href="#" tt="enables the equalizer and gain control;$Nboost 0 = unmodified 100% volume$N$Nenabling the equalizer makes gapless albums fully gapless, so leave it on with all the values at zero if you care about that">enable</a></td>'],
 		h2 = [], h3 = [], h4 = [];
 
 	var vs = [];
@@ -1484,7 +1484,7 @@ function play(tid, is_ev, seek, call_depth) {
 			seek_au_sec(seek);
 		}
 
-		if (!seek) {
+		if (!seek && !ebi('unsearch')) {
 			var o = ebi(oid);
 			o.setAttribute('id', 'thx_js');
 			sethash(oid);
@@ -1492,7 +1492,8 @@ function play(tid, is_ev, seek, call_depth) {
 		}
 
 		mpui.progress_updater();
-		pbar.drawbuf();
+		pbar.onresize();
+		vbar.onresize();
 		mpl.announce();
 		return true;
 	}
@@ -1567,7 +1568,8 @@ function autoplay_blocked(seek) {
 
 
 function eval_hash() {
-	var v = location.hash;
+	var v = hash0;
+	hash0 = null;
 	if (!v)
 		return;
 
@@ -1592,6 +1594,11 @@ function eval_hash() {
 		i.value = uricom_dec(v.slice(3))[0];
 		return i.oninput();
 	}
+
+	if (v.indexOf('#v=') === 0) {
+		goto(v.slice(3));
+		return;
+	}
 }
 
 
@@ -1803,17 +1810,19 @@ var fileman = (function () {
 		clmod(bdel, 'en', nsel);
 		clmod(bcut, 'en', nsel);
 		clmod(bpst, 'en', r.clip && r.clip.length);
-		bren.style.display = have_mv && has(perms, 'write') && has(perms, 'move') ? '' : 'none';
-		bdel.style.display = have_del && has(perms, 'delete') ? '' : 'none';
-		bcut.style.display = have_mv && has(perms, 'move') ? '' : 'none';
-		bpst.style.display = have_mv && has(perms, 'write') ? '' : 'none';
+
+		clmod(bren, 'hide', !(have_mv && has(perms, 'write') && has(perms, 'move')));
+		clmod(bdel, 'hide', !(have_del && has(perms, 'delete')));
+		clmod(bcut, 'hide', !(have_mv && has(perms, 'move')));
+		clmod(bpst, 'hide', !(have_mv && has(perms, 'write')));
+		clmod(ebi('wfm'), 'act', QS('#wfm a.en:not(.hide)'));
+
 		bpst.setAttribute('tt', 'paste ' + r.clip.length + ' items$NHotkey: ctrl-V');
-		clmod(ebi('wfm'), 'act', QS('#wfm a.en:not([style])'));
 	};
 
 	r.rename = function (e) {
 		ev(e);
-		if (bren.style.display)
+		if (clgot(bren, 'hide'))
 			return toast.err(3, 'cannot rename:\nyou do not have “move” permission in this folder');
 
 		var sel = msel.getsel();
@@ -1929,10 +1938,6 @@ var fileman = (function () {
 				(function (a) {
 					f[a].inew.onkeydown = function (e) {
 						rn_ok(a, true);
-
-						if (e.key == 'Escape')
-							return rn_cancel();
-
 						if (e.key == 'Enter')
 							return rn_apply();
 					};
@@ -2104,7 +2109,7 @@ var fileman = (function () {
 
 	r.delete = function (e) {
 		ev(e);
-		if (bdel.style.display)
+		if (clgot(bdel, 'hide'))
 			return toast.err(3, 'cannot delete:\nyou do not have “delete” permission in this folder');
 
 		var sel = msel.getsel(),
@@ -2150,7 +2155,7 @@ var fileman = (function () {
 
 	r.cut = function (e) {
 		ev(e);
-		if (bcut.style.display)
+		if (clgot(bcut, 'hide'))
 			return toast.err(3, 'cannot cut:\nyou do not have “move” permission in this folder');
 
 		var sel = msel.getsel(),
@@ -2191,7 +2196,7 @@ var fileman = (function () {
 
 	r.paste = function (e) {
 		ev(e);
-		if (bpst.style.display)
+		if (clgot(bpst, 'hide'))
 			return toast.err(3, 'cannot paste:\nyou do not have “write” permission in this folder');
 
 		if (!r.clip.length)
@@ -2384,6 +2389,7 @@ var showfile = (function () {
 				continue;
 
 			td.innerHTML = '<a href="#" class="doc bri" hl="' + link.id + '">-txt-</a>';
+			td.getElementsByTagName('a')[0].setAttribute('href', '?doc=' + fn);
 		}
 		r.mktree();
 		if (em) {
@@ -2420,6 +2426,7 @@ var showfile = (function () {
 			lnh = doc[1],
 			txt = doc[2],
 			name = url.split('/').pop(),
+			tname = uricom_dec(name)[0],
 			lang = r.getlang(name),
 			is_md = lang == 'md';
 
@@ -2466,13 +2473,14 @@ var showfile = (function () {
 		wr.style.display = '';
 		set_tabindex();
 
+		wintitle(tname + ' \u2014 ');
 		document.documentElement.scrollTop = 0;
 		var hfun = no_push ? hist_replace : hist_push;
 		hfun(get_evpath() + '?doc=' + url.split('/').pop());
 
 		qsr('#docname');
 		el = mknod('span');
-		el.textContent = uricom_dec(name)[0];
+		el.textContent = tname;
 		el.setAttribute('id', 'docname');
 		ebi('path').appendChild(el);
 
@@ -2482,7 +2490,7 @@ var showfile = (function () {
 	}
 
 	r.mktree = function () {
-		var html = ['<li class="bn">list of textfiles in<br />' + esc(get_vpath()) + '</li>'];
+		var html = ['<li class="bn">list of textfiles in<br />' + linksplit(get_vpath()).join('') + '</li>'];
 		for (var a = 0; a < r.files.length; a++) {
 			var file = r.files[a];
 			html.push('<li><a href="#" hl="' + file.id +
@@ -2559,9 +2567,9 @@ var thegrid = (function () {
 		'<a href="#" class="btn" z="1.2" tt="Hotkey: shift-D">+</a></span> <span>chop: ' +
 		'<a href="#" class="btn" l="-1" tt="truncate filenames more (show less)">&ndash;</a> ' +
 		'<a href="#" class="btn" l="1" tt="truncate filenames less (show more)">+</a></span> <span>sort by: ' +
-		'<a href="#" s="href">name</a>, ' +
-		'<a href="#" s="sz">size</a>, ' +
-		'<a href="#" s="ts">date</a>, ' +
+		'<a href="#" s="href">name</a> ' +
+		'<a href="#" s="sz">size</a> ' +
+		'<a href="#" s="ts">date</a> ' +
 		'<a href="#" s="ext">type</a>' +
 		'</span></div>' +
 		'<div id="ggrid"></div>'
@@ -2607,6 +2615,7 @@ var thegrid = (function () {
 				return;
 
 			hist_push(get_evpath());
+			wintitle();
 		}
 
 		var vis = has(perms, "read");
@@ -2675,14 +2684,12 @@ var thegrid = (function () {
 			href = noq_href(this),
 			aplay = ebi('a' + oth.getAttribute('id')),
 			is_img = /\.(gif|jpe?g|png|webp|webm|mp4)(\?|$)/i.test(href),
-			in_tree = null,
+			is_dir = href.endsWith('/'),
+			in_tree = is_dir && treectl.find(oth.textContent.slice(0, -1)),
 			have_sel = QS('#files tr.sel'),
 			td = oth.closest('td').nextSibling,
 			tr = td.parentNode;
 
-		if (href.endsWith('/'))
-			in_tree = treectl.find(oth.textContent.slice(0, -1));
-
 		if (r.sel && !dbl) {
 			td.click();
 			clmod(this, 'sel', clgot(tr, 'sel'));
@@ -2693,6 +2700,9 @@ var thegrid = (function () {
 		else if (in_tree && !have_sel)
 			in_tree.click();
 
+		else if (is_dir && !have_sel && treectl.spa)
+			treectl.reqls(href, true, true);
+
 		else if (!is_img && have_sel)
 			window.open(href, '_blank');
 
@@ -2867,10 +2877,6 @@ var thegrid = (function () {
 		import_js('/.cpr/baguettebox.js', r.bagit);
 	}, 1);
 
-	if (r.en) {
-		loadgrid();
-	}
-
 	return r;
 })();
 
@@ -2968,12 +2974,12 @@ document.onkeydown = function (e) {
 	if (k == 'Escape') {
 		ae && ae.blur();
 
+		if (ebi('rn_cancel'))
+			return ebi('rn_cancel').click();
+
 		if (QS('.opview.act'))
 			return QS('#ops>a').click();
 
-		if (QS('#unsearch'))
-			return QS('#unsearch').click();
-
 		if (widget.is_open)
 			return widget.close();
 
@@ -2983,6 +2989,9 @@ document.onkeydown = function (e) {
 		if (!treectl.hidden)
 			return treectl.detree();
 
+		if (QS('#unsearch'))
+			return QS('#unsearch').click();
+
 		if (thegrid.en)
 			return ebi('griden').click();
 	}
@@ -3020,7 +3029,7 @@ document.onkeydown = function (e) {
 		}
 	}
 
-	if (ae.closest('pre')) {
+	if (ae && ae.closest('pre')) {
 		if (k == 'KeyA' && ctrl(e)) {
 			var sel = document.getSelection(),
 				ran = document.createRange();
@@ -3214,7 +3223,7 @@ document.onkeydown = function (e) {
 			clearTimeout(defer_timeout);
 			clearTimeout(search_timeout);
 			search_timeout = setTimeout(do_search,
-				v && v.length < (is_touch ? 4 : 3) ? 600 : 200);
+				v && v.length < (is_touch ? 4 : 3) ? 1000 : 500);
 		}
 	}
 
@@ -3234,7 +3243,34 @@ document.onkeydown = function (e) {
 			for (var b = 1; b < sconf[a].length; b++) {
 				var k = sconf[a][b][0],
 					chk = 'srch_' + k + 'c',
-					tvs = ebi('srch_' + k + 'v').value.split(/ +/g);
+					vs = ebi('srch_' + k + 'v').value,
+					tvs = [];
+
+				if (k == 'name')
+					console.log('a');
+				while (vs) {
+					vs = vs.trim();
+					if (!vs)
+						break;
+
+					var v = '';
+					if (vs.startsWith('"')) {
+						var vp = vs.slice(1).split(/"(.*)/);
+						v = vp[0];
+						vs = vp[1] || '';
+						while (v.endsWith('\\')) {
+							vp = vs.split(/"(.*)/);
+							v = v.slice(0, -1) + '"' + vp[0];
+							vs = vp[1] || '';
+						}
+					}
+					else {
+						var vp = vs.split(/ +(.*)/);
+						v = vp[0].replace(/\\"/g, '"');
+						vs = vp[1] || '';
+					}
+					tvs.push(v);
+				}
 
 				if (!ebi(chk).checked)
 					continue;
@@ -3257,10 +3293,10 @@ document.onkeydown = function (e) {
 					}
 
 					if (k == 'path' || k == 'name' || k == 'tags') {
-						var not = ' ';
+						var not = '';
 						if (tv.slice(0, 1) == '-') {
 							tv = tv.slice(1);
-							not = ' not ';
+							not = 'not ';
 						}
 
 						if (tv.slice(0, 1) == '^') {
@@ -3277,7 +3313,11 @@ document.onkeydown = function (e) {
 							tv += '*';
 						}
 
-						q += k + not + 'like ' + tv;
+						if (tv.indexOf(' ') + 1) {
+							tv = '"' + tv + '"';
+						}
+
+						q += not + k + ' like ' + tv;
 					}
 				}
 			}
@@ -3325,8 +3365,9 @@ document.onkeydown = function (e) {
 			return;
 
 		treectl.hide();
+		thegrid.setvis(true);
 
-		var html = mk_files_header(tagord);
+		var html = mk_files_header(tagord), seen = {};
 		html.push('<tbody>');
 		html.push('<tr><td>-</td><td colspan="42"><a href="#" id="unsearch"><big style="font-weight:bold">[❌] close search results</big></a></td></tr>');
 		for (var a = 0; a < res.hits.length; a++) {
@@ -3334,14 +3375,19 @@ document.onkeydown = function (e) {
 				ts = parseInt(r.ts),
 				sz = esc(r.sz + ''),
 				rp = esc(uricom_dec(r.rp + '')[0]),
-				ext = rp.lastIndexOf('.') > 0 ? rp.split('.').pop() : '%',
-				links = linksplit(r.rp + '');
+				ext = rp.lastIndexOf('.') > 0 ? rp.split('.').pop().split('?')[0] : '%',
+				id = 'f-' + ('00000000' + crc32(rp)).slice(-8);
+
+			while (seen[id])
+				id += 'a';
+			seen[id] = 1;
 
 			if (ext.length > 8)
 				ext = '%';
 
-			links = links.join('');
-			var nodes = ['<tr><td>-</td><td><div>' + links + '</div>', sz];
+			var links = linksplit(r.rp + '', id).join(''),
+				nodes = ['<tr><td>-</td><td><div>' + links + '</div>', sz];
+
 			for (var b = 0; b < tagord.length; b++) {
 				var k = tagord[b],
 					v = r.tags[k] || "";
@@ -3403,6 +3449,7 @@ var treectl = (function () {
 		mentered = null,
 		treesz = clamp(icfg_get('treesz', 16), 10, 50);
 
+	bcfg_bind(r, 'spa', 'spafiles', true);
 	bcfg_bind(r, 'ireadme', 'ireadme', true);
 	bcfg_bind(r, 'dyn', 'dyntree', true, onresize);
 	bcfg_bind(r, 'dots', 'dotfiles', false, function (v) {
@@ -3410,7 +3457,7 @@ var treectl = (function () {
 	});
 	setwrap(bcfg_bind(r, 'wtree', 'wraptree', true, setwrap));
 	setwrap(bcfg_bind(r, 'parpane', 'parpane', true, onscroll));
-	bcfg_bind(r, 'htree', 'hovertree', true, reload_tree);
+	bcfg_bind(r, 'htree', 'hovertree', false, reload_tree);
 
 	function setwrap(v) {
 		clmod(ebi('tree'), 'nowrap', !v);
@@ -3583,6 +3630,7 @@ var treectl = (function () {
 			if (!QS(q))
 				break;
 		}
+		nq = Math.max(nq, get_evpath().split('/').length - 2);
 		var iw = (treesz + Math.max(0, nq)),
 			w = iw + 'em',
 			w2 = (iw + 2) + 'em';
@@ -3605,7 +3653,7 @@ var treectl = (function () {
 
 	r.goto = function (url, push) {
 		get_tree("", url, true);
-		reqls(url, push, true);
+		r.reqls(url, push, true);
 	};
 
 	function get_tree(top, dst, rst) {
@@ -3774,12 +3822,12 @@ var treectl = (function () {
 			treegrow.call(this.previousSibling, e);
 			return;
 		}
-		reqls(this.getAttribute('href'), true);
+		r.reqls(this.getAttribute('href'), true);
 		r.dir_cb = tree_scrollto;
 		thegrid.setvis(true);
 	}
 
-	function reqls(url, hpush, no_tree) {
+	r.reqls = function (url, hpush, no_tree) {
 		var xhr = new XMLHttpRequest();
 		xhr.top = url;
 		xhr.hpush = hpush;
@@ -3834,8 +3882,31 @@ var treectl = (function () {
 
 		ebi('srv_info').innerHTML = '<span>' + res.srvinf + '</span>';
 
-		var top = this.top,
-			nodes = res.dirs.concat(res.files),
+		if (this.hpush && !showfile.active())
+			hist_push(this.top);
+
+		r.gentab(this.top, res);
+		despin('#files');
+		despin('#gfiles');
+
+		ebi('pro').innerHTML = res.logues ? res.logues[0] || "" : "";
+		ebi('epi').innerHTML = res.logues ? res.logues[1] || "" : "";
+
+		clmod(ebi('epi'), 'mdo');
+		if (res.readme)
+			show_readme(res.readme);
+
+		wintitle();
+		var fun = r.ls_cb;
+		if (fun) {
+			r.ls_cb = null;
+			fun();
+		}
+		eval_hash();
+	}
+
+	r.gentab = function (top, res) {
+		var nodes = res.dirs.concat(res.files),
 			html = mk_files_header(res.taglist),
 			seen = {};
 
@@ -3851,7 +3922,7 @@ var treectl = (function () {
 				id = 'f-' + ('00000000' + crc32(fname)).slice(-8),
 				lang = showfile.getlang(fname);
 
-			while (seen[id])
+			while (seen[id])  // ejyefs ev69gg y9j8sg .opus
 				id += 'a';
 			seen[id] = 1;
 
@@ -3859,7 +3930,7 @@ var treectl = (function () {
 				showfile.files.push({ 'id': id, 'name': fname });
 
 			if (tn.lead == '-')
-				tn.lead = '<a href="#" class="doc' + (lang ? ' bri' : '') +
+				tn.lead = '<a href="?doc=' + tn.href + '" class="doc' + (lang ? ' bri' : '') +
 					'" hl="' + id + '" name="' + hname + '">-txt-</a>';
 
 			var ln = ['<tr><td>' + tn.lead + '</td><td sortv="' + sortv +
@@ -3883,37 +3954,39 @@ var treectl = (function () {
 		html = html.join('\n');
 		set_files_html(html);
 
-		if (this.hpush && !showfile.active())
-			hist_push(this.top);
-
-		acct = res.acct;
-		apply_perms(res.perms);
-		despin('#files');
-		despin('#gfiles');
-
-		ebi('pro').innerHTML = res.logues ? res.logues[0] || "" : "";
-		ebi('epi').innerHTML = res.logues ? res.logues[1] || "" : "";
-
-		clmod(ebi('epi'), 'mdo');
-		if (res.readme)
-			show_readme(res.readme);
-
-		document.title = '⇆🎉 ' + uricom_dec(document.location.pathname.slice(1, -1))[0];
-
 		filecols.set_style();
 		showfile.mktree();
 		mukey.render();
 		reload_tree();
 		reload_browser();
 		tree_scrollto();
-
-		var fun = r.ls_cb;
-		if (fun) {
-			r.ls_cb = null;
-			fun();
+		if (res.acct) {
+			acct = res.acct;
+			have_up2k_idx = res.idx;
+			apply_perms(res.perms);
+			fileman.render();
 		}
 	}
 
+	r.hydrate = function () {
+		if (ls0 === null) {
+			var xhr = new XMLHttpRequest();
+			xhr.open('GET', '/?am_js', true);
+			xhr.send();
+
+			return r.reqls(get_evpath(), false, true);
+		}
+
+		r.gentab(get_evpath(), ls0);
+		reload_browser();
+		pbar.onresize();
+		vbar.onresize();
+		mukey.render();
+		showfile.addlinks();
+		thegrid.setdirty();
+		setTimeout(eval_hash, 1);
+	};
+
 	function parsetree(res, top) {
 		var ret = '';
 		for (var a = 0; a < res.a.length; a++) {
@@ -4008,6 +4081,17 @@ function despin(sel) {
 function apply_perms(newperms) {
 	perms = newperms || [];
 
+	var a = QS('#ops a[data-dest="up2k"]');
+	if (have_up2k_idx) {
+		a.removeAttribute('data-perm');
+		a.setAttribute('tt', 'up2k: upload files (if you have write-access) or toggle into the search-mode to see if they exist somewhere on the server');
+	}
+	else {
+		a.setAttribute('data-perm', 'write');
+		a.setAttribute('tt', 'up2k: upload files with resume support (close your browser and drop the same files in later)');
+	}
+	tt.att(QS('#ops'));
+
 	var axs = [],
 		aclass = '>',
 		chk = ['read', 'write', 'move', 'delete', 'get'];
@@ -4037,6 +4121,12 @@ function apply_perms(newperms) {
 		o[a].style.display = display;
 	}
 
+	var o = QSA('#ops>a[data-dep], #u2conf td[data-dep]');
+	for (var a = 0; a < o.length; a++)
+		o[a].style.display = (
+			o[a].getAttribute('data-dep') != 'idx' || have_up2k_idx
+		) ? '' : 'none';
+
 	var act = QS('#ops>a.act');
 	if (act && act.style.display === 'none')
 		goto();
@@ -4374,27 +4464,6 @@ var mukey = (function () {
 })();
 
 
-function addcrc() {
-	var links = QSA(
-		'#files>tbody>tr>td:first-child+td>' + (
-			ebi('unsearch') ? 'div>a:last-child' : 'a'));
-
-	var seen = {};  // ejyefs ev69gg y9j8sg .opus
-	for (var a = 0, aa = links.length; a < aa; a++) {
-		var id = links[a].getAttribute('id');
-		if (!id) {
-			var crc = crc32(links[a].textContent || links[a].innerText);
-			id = 'f-' + ('00000000' + crc).slice(-8);
-			while (seen[id])
-				id += 'a';
-
-			links[a].setAttribute('id', id);
-		}
-		seen[id] = 1;
-	}
-}
-
-
 var light;
 (function () {
 	function freshen() {
@@ -4726,7 +4795,7 @@ function show_md(md, name, div, url, depth) {
 
 	try {
 		clmod(div, 'mdo', 1);
-		div.innerHTML = marked(md, {
+		div.innerHTML = marked.parse(md, {
 			headerPrefix: 'md-',
 			breaks: true,
 			gfm: true
@@ -4948,15 +5017,38 @@ function goto_unpost(e) {
 }
 
 
+function wintitle(txt) {
+	document.title = (txt ? txt : '') + get_vpath().slice(1, -1).split('/').pop();
+}
+
+
+ebi('path').onclick = function (e) {
+	var a = e.target.closest('a[href]');
+	if (!treectl.spa || !a || !(a = a.getAttribute('href') + '') || !a.endsWith('/'))
+		return;
+
+	thegrid.setvis(true);
+	treectl.reqls(a, true, true);
+	return ev(e);
+};
+
+
 ebi('files').onclick = ebi('docul').onclick = function (e) {
+	if (ctrl(e))
+		return true;
+
 	var tgt = e.target.closest('a[id]');
 	if (tgt && tgt.getAttribute('id').indexOf('f-') === 0 && tgt.textContent.endsWith('/')) {
 		var el = treectl.find(tgt.textContent.slice(0, -1));
-		if (!el)
-			return;
-
-		el.click();
-		return ev(e);
+		if (el) {
+			el.click();
+			return ev(e);
+		}
+		if (treectl.spa) {
+			treectl.reqls(tgt.getAttribute('href'), true, true);
+			return ev(e);
+		}
+		return;
 	}
 
 	tgt = e.target.closest('a[hl]');
@@ -4964,7 +5056,14 @@ ebi('files').onclick = ebi('docul').onclick = function (e) {
 		showfile.show(noq_href(ebi(tgt.getAttribute('hl'))), tgt.getAttribute('lang'));
 		return ev(e);
 	}
-}
+
+	tgt = e.target.closest('a');
+	if (tgt && tgt.closest('li.bn')) {
+		thegrid.setvis(true);
+		treectl.goto(tgt.getAttribute('href'), true);
+		return ev(e);
+	}
+};
 
 
 function reload_mp() {
@@ -4972,33 +5071,33 @@ function reload_mp() {
 		audio_eq.stop();
 		mp.au.pause();
 		mp.au = null;
+		mpl.unbuffer();
 	}
 	mpl.stop();
-	widget.close();
 	var plays = QSA('tr>td:first-child>a.play');
 	for (var a = plays.length - 1; a >= 0; a--)
 		plays[a].parentNode.innerHTML = '-';
 
-	mpl.unbuffer();
 	mp = new MPlayer();
 	audio_eq.acst = {};
 	setTimeout(pbar.onresize, 1);
 }
 
 
-function reload_browser(not_mp) {
+function reload_browser() {
 	filecols.set_style();
 
 	var parts = get_evpath().split('/'),
-		rm = QSA('#path>a+a+a');
+		rm = QSA('#path>a+a+a'),
+		ftab = ebi('files'),
+		link = '/', o;
 
 	for (a = rm.length - 1; a >= 0; a--)
 		rm[a].parentNode.removeChild(rm[a]);
 
-	var link = '/';
 	for (var a = 1; a < parts.length - 1; a++) {
 		link += parts[a] + '/';
-		var o = mknod('a');
+		o = mknod('a');
 		o.setAttribute('href', link);
 		o.textContent = uricom_dec(parts[a])[0];
 		ebi('path').appendChild(o);
@@ -5012,11 +5111,12 @@ function reload_browser(not_mp) {
 		oo[a].textContent = hsz;
 	}
 
-	if (!not_mp) {
-		addcrc();
-		reload_mp();
-		makeSortable(ebi('files'), mp.read_order.bind(mp));
-	}
+	reload_mp();
+	try { showsort(ftab); } catch (ex) { }
+	makeSortable(ftab, function () {
+		thegrid.setdirty();
+		mp.read_order();
+	});
 
 	for (var a = 0; a < 2; a++)
 		clmod(ebi(a ? 'pro' : 'epi'), 'hidden', ebi('unsearch'));
@@ -5027,7 +5127,4 @@ function reload_browser(not_mp) {
 	thegrid.setdirty();
 	msel.render();
 }
-reload_browser(true);
-showfile.addlinks();
-mukey.render();
-setTimeout(eval_hash, 1);
+treectl.hydrate();

copyparty/web/browser2.html

@@ -6,6 +6,7 @@
 	<title>{{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.8">
+{{ html_head }}
 	<style>
 		html{font-family:sans-serif}
 		td{border:1px solid #999;border-width:1px 1px 0 0;padding:0 5px}

copyparty/web/md.html

@@ -3,6 +3,7 @@
 	<title>📝🎉 {{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
+{{ html_head }}
 	<link rel="stylesheet" href="/.cpr/ui.css?_={{ ts }}">
 	<link rel="stylesheet" href="/.cpr/md.css?_={{ ts }}">
 	{%- if edit %}
@@ -10,7 +11,7 @@
 	{%- endif %}
 </head>
 <body>
-	<div id="mn">navbar</div>
+	<div id="mn"></div>
 	<div id="mh">
 		<a id="lightswitch" href="#">go dark</a>
 		<a id="navtoggle" href="#">hide nav</a>

copyparty/web/md.js

@@ -39,20 +39,14 @@ var md_plug = {};
 
 // add navbar
 (function () {
-    var n = document.location + '';
-    n = n.substr(n.indexOf('//') + 2).split('?')[0].split('/');
-    n[0] = 'top';
-    var loc = [];
-    var nav = [];
-    for (var a = 0; a < n.length; a++) {
-        if (a > 0)
-            loc.push(n[a]);
-
-        var dec = esc(uricom_dec(n[a])[0]);
-
-        nav.push('<a href="/' + loc.join('/') + '">' + dec + '</a>');
+    var parts = get_evpath().split('/'), link = '', o;
+    for (var a = 0, aa = parts.length - 2; a <= aa; a++) {
+        link += parts[a] + (a < aa ? '/' : '');
+        o = mknod('a');
+        o.setAttribute('href', link);
+        o.textContent = uricom_dec(parts[a])[0] || 'top';
+        dom_nav.appendChild(o);
     }
-    dom_nav.innerHTML = nav.join('');
 })();
 
 
@@ -91,13 +85,13 @@ function copydom(src, dst, lv) {
 
     var rpl = [];
     for (var a = sc.length - 1; a >= 0; a--) {
-        var st = sc[a].tagName,
-            dt = dc[a].tagName;
+        var st = sc[a].tagName || sc[a].nodeType,
+            dt = dc[a].tagName || dc[a].nodeType;
 
         if (st !== dt) {
             dbg("replace L%d (%d/%d) type %s/%s", lv, a, sc.length, st, dt);
-            rpl.push(a);
-            continue;
+            dst.innerHTML = src.innerHTML;
+            return;
         }
 
         var sa = sc[a].attributes || [],
@@ -146,8 +140,11 @@ function copydom(src, dst, lv) {
     // repl is reversed; build top-down
     var nbytes = 0;
     for (var a = rpl.length - 1; a >= 0; a--) {
-        var html = sc[rpl[a]].outerHTML;
-        dc[rpl[a]].outerHTML = html;
+        var i = rpl[a],
+            prop = sc[i].nodeType == 1 ? 'outerHTML' : 'nodeValue';
+
+        var html = sc[i][prop];
+        dc[i][prop] = html;
         nbytes += html.length;
     }
     if (nbytes > 0)
@@ -256,7 +253,7 @@ function convert_markdown(md_text, dest_dom) {
         Object.assign(marked_opts, ext[0]);
 
     try {
-        var md_html = marked(md_text, marked_opts);
+        var md_html = marked.parse(md_text, marked_opts);
     }
     catch (ex) {
         if (ext)

copyparty/web/mde.html

@@ -3,6 +3,7 @@
 	<title>📝🎉 {{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
+{{ html_head }}
 	<link rel="stylesheet" href="/.cpr/ui.css?_={{ ts }}">
 	<link rel="stylesheet" href="/.cpr/mde.css?_={{ ts }}">
 	<link rel="stylesheet" href="/.cpr/deps/mini-fa.css?_={{ ts }}">

copyparty/web/msg.html

@@ -3,9 +3,10 @@
 
 <head>
     <meta charset="utf-8">
-    <title>copyparty</title>
+    <title>{{ svcname }}</title>
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width, initial-scale=0.8">
+	{{ html_head }}
     <link rel="stylesheet" media="screen" href="/.cpr/msg.css?_={{ ts }}">
 </head>
 

copyparty/web/splash.css

@@ -81,9 +81,10 @@ table {
 	text-align: right;
 }
 blockquote {
-	margin: 0 0 0 .6em;
-	padding: .7em 1em;
+	margin: 0 0 1.6em .6em;
+	padding: .7em 1em 0 1em;
 	border-left: .3em solid rgba(128,128,128,0.5);
+	border-radius: 0 0 0 .25em;
 }
 
 

copyparty/web/splash.html

@@ -3,9 +3,10 @@
 
 <head>
     <meta charset="utf-8">
-    <title>copyparty</title>
+    <title>{{ svcname }}</title>
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width, initial-scale=0.8">
+{{ html_head }}
     <link rel="stylesheet" media="screen" href="/.cpr/splash.css?_={{ ts }}">
     <link rel="stylesheet" media="screen" href="/.cpr/ui.css?_={{ ts }}">
 </head>
@@ -75,11 +76,13 @@
         <h1 id="cc">client config:</h1>
         <ul>
             {% if k304 %}
-            <li><a href="/?k304=n" class="r">disable k304</a> (currently enabled)
+            <li><a href="/?k304=n">disable k304</a> (currently enabled)
             {%- else %}
-            <li><a href="/?k304=y">enable k304</a> (currently disabled)
+            <li><a href="/?k304=y" class="r">enable k304</a> (currently disabled)
             {% endif %}
             <blockquote>enabling this will disconnect your client on every HTTP 304, which can prevent some buggy browsers/proxies from getting stuck (suddenly not being able to load pages), <em>but</em> it will also make things slower in general</blockquote></li>
+            
+            <li><a href="/?reset" class="r" onclick="localStorage.clear();return true">reset client settings</a></li>
         </ul>
 
         <h1>login for more:</h1>

copyparty/web/ui.css

@@ -128,6 +128,7 @@ html {
 }
 #tth.act {
 	display: block;
+	z-index: 9001;
 }
 #tt.b {
 	padding: 0 2em;
@@ -353,6 +354,13 @@ html.light textarea:focus {
 }
 .mdo ul,
 .mdo ol {
+	padding-left: 1em;
+}
+.mdo ul ul,
+.mdo ul ol,
+.mdo ol ul,
+.mdo ol ol {
+	padding-left: 2em;
 	border-left: .3em solid #ddd;
 }
 .mdo ul>li,

copyparty/web/up2k.js

@@ -525,13 +525,15 @@ function Donut(uc, st) {
     }
 
     r.on = function (ya) {
-        r.fc = 99;
+        r.fc = r.tc = 99;
         r.eta = null;
         r.base = pos();
         optab.innerHTML = ya ? svg() : optab.getAttribute('ico');
         el = QS('#ops a .donut');
-        if (!ya)
+        if (!ya) {
             favico.upd();
+            wintitle();
+        }
     };
     r.do = function () {
         if (!el)
@@ -541,6 +543,11 @@ function Donut(uc, st) {
             v = pos() - r.base,
             ofs = el.style.strokeDashoffset = o - o * v / t;
 
+        if (++r.tc >= 10) {
+            wintitle(f2f(v * 100 / t, 1) + '%, ' + r.eta + 's, ', true);
+            r.tc = 0;
+        }
+
         if (favico.txt) {
             if (++r.fc < 10 && r.eta && r.eta > 99)
                 return;
@@ -728,7 +735,6 @@ function up2k_init(subtle) {
         if (++nenters <= 0)
             nenters = 1;
 
-        //console.log(nenters, Date.now(), 'enter', this, e.target);
         if (onover.bind(this)(e))
             return true;
 
@@ -750,12 +756,19 @@ function up2k_init(subtle) {
         ebi('up_dz').setAttribute('err', mup || '');
         ebi('srch_dz').setAttribute('err', msr || '');
     }
+    function onoverb(e) {
+        // zones are alive; disable cuo2duo branch
+        document.body.ondragover = document.body.ondrop = null;
+        return onover.bind(this)(e);
+    }
     function onover(e) {
         try {
             var ok = false, dt = e.dataTransfer.types;
             for (var a = 0; a < dt.length; a++)
                 if (dt[a] == 'Files')
                     ok = true;
+                else if (dt[a] == 'text/uri-list')
+                    return true;
 
             if (!ok)
                 return true;
@@ -781,17 +794,20 @@ function up2k_init(subtle) {
             clmod(ebi('drops'), 'vis');
             clmod(ebi('up_dz'), 'hl');
             clmod(ebi('srch_dz'), 'hl');
+            // cuo2duo:
+            document.body.ondragover = onover;
+            document.body.ondrop = gotfile;
         }
-
-        //console.log(nenters, Date.now(), 'leave', this, e && e.target);
     }
     document.body.ondragenter = ondrag;
     document.body.ondragleave = offdrag;
+    document.body.ondragover = onover;
+    document.body.ondrop = gotfile;
 
     var drops = [ebi('up_dz'), ebi('srch_dz')];
     for (var a = 0; a < 2; a++) {
         drops[a].ondragenter = ondrag;
-        drops[a].ondragover = onover;
+        drops[a].ondragover = onoverb;
         drops[a].ondragleave = offdrag;
         drops[a].ondrop = gotfile;
     }
@@ -801,7 +817,10 @@ function up2k_init(subtle) {
         ev(e);
         nenters = 0;
         offdrag.bind(this)();
-        var dz = (this && this.getAttribute('id'));
+        var dz = this && this.getAttribute('id');
+        if (!dz && e && e.clientY)
+            // cuo2duo fallback
+            dz = e.clientY < window.innerHeight / 2 ? 'up_dz' : 'srch_dz';
 
         var err = this.getAttribute('err');
         if (err)
@@ -1154,7 +1173,7 @@ function up2k_init(subtle) {
         var t = st.todo.handshake[0],
             cd = t.cooldown;
 
-        if (cd && cd - Date.now() > 0)
+        if (cd && cd > Date.now())
             return false;
 
         // keepalive or verify
@@ -1351,6 +1370,14 @@ function up2k_init(subtle) {
         return taskerd;
     })();
 
+    function chill(t) {
+        var now = Date.now();
+        if ((t.coolmul || 0) < 2 || now - t.cooldown < t.coolmul * 700)
+            t.coolmul = Math.min((t.coolmul || 0.5) * 2, 32);
+
+        t.cooldown = Math.max(t.cooldown || 1, Date.now() + t.coolmul * 1000);
+    }
+
     /////
     ////
     ///   hashing
@@ -1466,7 +1493,8 @@ function up2k_init(subtle) {
                     err.indexOf('NotFoundError') !== -1  // macos-firefox permissions
                 ) {
                     pvis.seth(t.n, 1, 'OS-error');
-                    pvis.seth(t.n, 2, err);
+                    pvis.seth(t.n, 2, err + ' @ ' + car);
+                    console.log('OS-error', reader.error, '@', car);
                     handled = true;
                 }
 
@@ -1736,8 +1764,12 @@ function up2k_init(subtle) {
 
                     pvis.move(t.n, 'ok');
                 }
-                else t.t_uploaded = undefined;
+                else {
+                    if (t.t_uploaded)
+                        chill(t);
 
+                    t.t_uploaded = undefined;
+                }
                 tasker();
             }
             else {
@@ -1849,7 +1881,8 @@ function up2k_init(subtle) {
             else {
                 toast.err(0, "server broke; cu-err {0} on file [{1}]:\n".format(
                     xhr.status, t.name) + (txt || "no further information"));
-                return;
+
+                chill(t);
             }
             orz2(xhr);
         }
@@ -2019,7 +2052,7 @@ function up2k_init(subtle) {
                 new_state = true;
                 fixed = true;
             }
-            if (!has(perms, 'read')) {
+            if (!has(perms, 'read') || !have_up2k_idx) {
                 new_state = false;
                 fixed = true;
             }
@@ -2094,7 +2127,7 @@ function up2k_init(subtle) {
     if (parallel_uploads < 1)
         bumpthread(1);
 
-    return { "init_deps": init_deps, "set_fsearch": set_fsearch, "ui": pvis }
+    return { "init_deps": init_deps, "set_fsearch": set_fsearch, "ui": pvis, "st": st, "uc": uc }
 }
 
 

copyparty/web/util.js

@@ -86,6 +86,9 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
     if ((msg + '').indexOf('ResizeObserver') !== -1)
         return;  // chrome issue 809574 (benign, from <video>)
 
+    if ((msg + '').indexOf('l2d.js') !== -1)
+        return;  // `t` undefined in tapEvent -> hitTestSimpleCustom
+
     var ekey = url + '\n' + lineNo + '\n' + msg;
     if (ignexd[ekey] || crashed)
         return;
@@ -219,15 +222,15 @@ if (!String.prototype.endsWith)
         return this.substring(this_len - search.length, this_len) === search;
     };
 
-if (!String.startsWith)
+if (!String.prototype.startsWith)
     String.prototype.startsWith = function (s, i) {
         i = i > 0 ? i | 0 : 0;
         return this.substring(i, i + s.length) === s;
     };
 
-if (!String.trimEnd)
+if (!String.prototype.trimEnd)
     String.prototype.trimEnd = String.prototype.trimRight = function () {
-        return this.replace(/[ \t\r\n]+$/m, '');
+        return this.replace(/[ \t\r\n]+$/, '');
     };
 
 if (!Element.prototype.matches)
@@ -329,14 +332,45 @@ function clgot(el, cls) {
 }
 
 
+function showsort(tab) {
+    var v, vn, v1, v2, th = tab.tHead,
+        sopts = jread('fsort', [["href", 1, ""]]);
+
+    th && (th = th.rows[0]) && (th = th.cells);
+
+    for (var a = sopts.length - 1; a >= 0; a--) {
+        if (!sopts[a][0])
+            continue;
+
+        v2 = v1;
+        v1 = sopts[a];
+    }
+
+    v = [v1, v2];
+    vn = [v1 ? v1[0] : '', v2 ? v2[0] : ''];
+
+    var ga = QSA('#ghead a[s]');
+    for (var a = 0; a < ga.length; a++)
+        ga[a].className = '';
+
+    for (var a = 0; a < th.length; a++) {
+        var n = vn.indexOf(th[a].getAttribute('name')),
+            cl = n < 0 ? ' ' : ' s' + n + (v[n][1] > 0 ? ' ' : 'r ');
+
+        th[a].className = th[a].className.replace(/ *s[01]r? */, ' ') + cl;
+        if (n + 1) {
+            ga = QS('#ghead a[s="' + vn[n] + '"]');
+            if (ga)
+                ga.className = cl;
+        }
+    }
+}
 function sortTable(table, col, cb) {
     var tb = table.tBodies[0],
         th = table.tHead.rows[0].cells,
         tr = Array.prototype.slice.call(tb.rows, 0),
-        i, reverse = th[col].className.indexOf('sort1') !== -1 ? -1 : 1;
-    for (var a = 0, thl = th.length; a < thl; a++)
-        th[a].className = th[a].className.replace(/ *sort-?1 */, " ");
-    th[col].className += ' sort' + reverse;
+        i, reverse = /s0[^r]/.exec(th[col].className + ' ') ? -1 : 1;
+
     var stype = th[col].getAttribute('sort');
     try {
         var nrules = [], rules = jread("fsort", []);
@@ -354,6 +388,7 @@ function sortTable(table, col, cb) {
                 break;
         }
         jwrite("fsort", nrules);
+        try { showsort(table); } catch (ex) { }
     }
     catch (ex) {
         console.log("failed to persist sort rules, resetting: " + ex);
@@ -402,7 +437,7 @@ function makeSortable(table, cb) {
 }
 
 
-function linksplit(rp) {
+function linksplit(rp, id) {
     var ret = [],
         apath = '/',
         q = null;
@@ -432,8 +467,13 @@ function linksplit(rp) {
             vlink = vlink.slice(0, -1) + '<span>/</span>';
         }
 
-        if (!rp && q)
-            link += q;
+        if (!rp) {
+            if (q)
+                link += q;
+
+            if (id)
+                link += '" id="' + id;
+        }
 
         ret.push('<a href="' + apath + link + '">' + vlink + '</a>');
         apath += link;

docs/README.md

@@ -2,24 +2,12 @@
 
 
 
-# example resource files
+# utilities
 
-can be provided to copyparty to tweak things
-
-
-
-## example `.epilogue.html`
-save one of these as `.epilogue.html` inside a folder to customize it:
-
-* [`minimal-up2k.html`](minimal-up2k.html) will [simplify the upload ui](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
-
-
-
-## example browser-css
-point `--css-browser` to one of these by URL:
-
-* [`browser.css`](browser.css) changes the background
-* [`browser-icons.css`](browser-icons.css) adds filetype icons
+## [`multisearch.html`](multisearch.html)
+* takes a list of filenames of youtube rips, grabs the youtube-id of each file, and does a search on the server for those
+* use it by putting it somewhere on the server and opening it as an html page
+* also serves as an extendable template for other specific search behaviors
 
 
 

docs/multisearch.html

@@ -0,0 +1,124 @@
+<!DOCTYPE html><html lang="en"><head>
+	<meta charset="utf-8">
+	<title>multisearch</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <style>
+
+html, body {
+    margin: 0;
+    padding: 0;
+    color: #ddd;
+    background: #222;
+    font-family: sans-serif;
+}
+body {
+    padding: 1em;
+}
+a {
+    color: #fc5;
+}
+ul {
+    line-height: 1.5em;
+}
+code {
+    color: #fc5;
+    border: 1px solid #444;
+    padding: .1em .2em;
+    font-family: sans-serif, sans-serif;
+}
+#src {
+    display: block;
+    width: calc(100% - 1em);
+    padding: .5em;
+    margin: 0;
+}
+td {
+    padding-left: 1em;
+}
+.hit,
+.miss {
+    font-weight: bold;
+    padding-left: 0;
+    padding-top: 1em;
+}
+.hit {color: #af0;}
+.miss {color: #f0c;}
+.hit:before {content: '✅';}
+.miss:before {content: '❌';}
+
+</style></head><body>
+    <ul>
+        <li>paste a list of filenames (youtube rips) below and hit search</li>
+        <li>it will grab the youtube-id from the filenames and search for each id</li>
+        <li>filenames must be like <code>-YTID.webm</code> (youtube-dl style) or <code>[YTID].webm</code> (ytdlp style)</li>
+    </ul>
+    <textarea id="src"></textarea>
+    <button id="go">search</button>
+    <div id="res"></div>
+    <script>
+
+var ebi = document.getElementById.bind(document);
+function esc(txt) {
+    return txt.replace(/[&"<>]/g, function (c) {
+        return {
+            '&': '&amp;',
+            '"': '&quot;',
+            '<': '&lt;',
+            '>': '&gt;'
+        }[c];
+    });
+}
+
+ebi('go').onclick = async function() {
+    var queries = [];
+    for (var ln of ebi('src').value.split(/\n/g)) {
+        // filter the list of input files,
+        // only keeping youtube videos,
+        // meaning the filename ends with either
+        //   [YOUTUBEID].EXTENSION or
+        //   -YOUTUBEID.EXTENSION
+        var m = /[[-]([0-9a-zA-Z_-]{11})\]?\.(mp4|webm|mkv)$/.exec(ln);
+        if (!m || !(m = m[1]))
+            continue;
+
+        // create a search query for each line: name like *youtubeid*
+        queries.push([ln, `name like *${m}*`]);
+    }
+
+    var a = 0, html = ['<table>'], hits = [], misses = [];
+    for (var [fn, q] of queries) {
+        var r = await fetch('/?srch', {
+            method: 'POST',
+            body: JSON.stringify({'q': q})
+        });
+        r = await r.json();
+        
+        var cl, tab2;
+        if (r.hits.length) {
+            tab2 = hits;
+            cl = 'hit';
+        }
+        else {
+            tab2 = misses;
+            cl = 'miss';
+        }
+        var h = `<tr><td class="${cl}" colspan="9">${esc(fn)}</td></tr>`;
+        tab2.push(h);
+        html.push(h);
+        for (var h of r.hits) {
+            var link = `<a href="/${h.rp}">${esc(decodeURIComponent(h.rp))}</a>`;
+            html.push(`<tr><td>${h.sz}</td><td>${link}</td></tr>`);
+        }
+        ebi('res').innerHTML = `searching, ${++a} / ${queries.length} done, ${hits.length} hits, ${misses.length} miss`;
+    }
+    html.push('<tr><td><h1>hits:</h1></td></tr>');
+    html = html.concat(hits);
+
+    html.push('<tr><td><h1>miss:</h1></td></tr>');
+    html = html.concat(misses);
+
+    html.push('</table>');
+    ebi('res').innerHTML = html.join('\n');
+};
+
+</script></body></html>

docs/notes.sh

@@ -3,6 +3,12 @@ echo not a script
 exit 1
 
 
+##
+## add index.html banners
+
+find -name index.html | sed -r 's/index.html$//' | while IFS= read -r dir; do f="$dir/.prologue.html"; [ -e "$f" ] || echo '<h1><a href="index.html">open index.html</a></h1>' >"$f"; done
+
+
 ##
 ## delete all partial uploads
 ##  (supports linux/macos, probably windows+msys2)
@@ -95,6 +101,7 @@ var t=[]; var b=document.location.href.split('#')[0].slice(0, -1); document.quer
 # debug md-editor line tracking
 var s=mknod('style');s.innerHTML='*[data-ln]:before {content:attr(data-ln)!important;color:#f0c;background:#000;position:absolute;left:-1.5em;font-size:1rem}';document.head.appendChild(s);
 
+
 ##
 ## bash oneliners
 
@@ -193,6 +200,12 @@ git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --
 # download all sfx versions
 curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | .tag_name + " " + .name' | tr -d '\r' | while read v t; do fn="$(printf '%s\n' "copyparty $v $t.py" | tr / -)"; [ -e "$fn" ] || curl https://github.com/9001/copyparty/releases/download/$v/copyparty-sfx.py -Lo "$fn"; done
 
+# push to multiple git remotes
+git config -l | grep '^remote'
+git remote add all git@github.com:9001/copyparty.git
+git remote set-url --add --push all git@gitlab.com:9001/copyparty.git
+git remote set-url --add --push all git@github.com:9001/copyparty.git
+
 
 ##
 ## http 206

scripts/copyparty-repack.sh

@@ -12,21 +12,18 @@ set -e
 #
 # output summary (filesizes and contents):
 #
-# 535672  copyparty-extras/sfx-full/copyparty-sfx.sh
 # 550760  copyparty-extras/sfx-full/copyparty-sfx.py
 #           `- original unmodified sfx from github
 #
 # 572923  copyparty-extras/sfx-full/copyparty-sfx-gz.py
 #           `- unmodified but recompressed from bzip2 to gzip
 #
-# 341792  copyparty-extras/sfx-ent/copyparty-sfx.sh
 # 353975  copyparty-extras/sfx-ent/copyparty-sfx.py
 # 376934  copyparty-extras/sfx-ent/copyparty-sfx-gz.py
 #           `- removed iOS ogg/opus/vorbis audio decoder,
 #              removed the audio tray mouse cursor,
 #              "enterprise edition"
 #
-# 259288  copyparty-extras/sfx-lite/copyparty-sfx.sh
 # 270004  copyparty-extras/sfx-lite/copyparty-sfx.py
 # 293159  copyparty-extras/sfx-lite/copyparty-sfx-gz.py
 #           `- also removed the codemirror markdown editor
@@ -81,7 +78,7 @@ cache="$od/.copyparty-repack.cache"
 		# fallback to awk (sorry)
 		awk -F\" '/"browser_download_url".*(\.tar\.gz|-sfx\.)/ {print$4}'
 	) |
-	grep -E '(sfx\.(sh|py)|tar\.gz)$' |
+	grep -E '(sfx\.py|tar\.gz)$' |
 	tee /dev/stderr |
 	tr -d '\r' | tr '\n' '\0' |
 	xargs -0 bash -c 'dl_files "$@"' _
@@ -139,11 +136,11 @@ repack() {
 	)
 }
 
-repack sfx-full "re gz no-sh"
+repack sfx-full "re gz"
 repack sfx-ent  "re no-dd"
-repack sfx-ent  "re no-dd gz no-sh"
+repack sfx-ent  "re no-dd gz"
 repack sfx-lite "re no-dd no-cm no-hl"
-repack sfx-lite "re no-dd no-cm no-hl gz no-sh"
+repack sfx-lite "re no-dd no-cm no-hl gz"
 
 
 # move fuse and up2k clients into copyparty-extras/,

scripts/deps-docker/Dockerfile

@@ -1,16 +1,16 @@
-FROM    alpine:3.14
+FROM    alpine:3.15
 WORKDIR /z
 ENV     ver_asmcrypto=5b994303a9d3e27e0915f72a10b6c2c51535a4dc \
         ver_hashwasm=4.9.0 \
-        ver_marked=3.0.4 \
-        ver_mde=2.15.0 \
-        ver_codemirror=5.62.3 \
+        ver_marked=4.0.12 \
+        ver_mde=2.16.1 \
+        ver_codemirror=5.65.2 \
         ver_fontawesome=5.13.0 \
         ver_zopfli=1.0.3
 
 
 # download;
-# the scp url is latin from https://fonts.googleapis.com/css2?family=Source+Code+Pro&display=swap
+# the scp url is regular latin from https://fonts.googleapis.com/css2?family=Source+Code+Pro&display=swap
 RUN     mkdir -p /z/dist/no-pk \
         && wget https://fonts.gstatic.com/s/sourcecodepro/v11/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2 -O scp.woff2 \
         && apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzip python3 python3-dev brotli py3-brotli \
@@ -82,7 +82,6 @@ RUN     cd marked-$ver_marked \
         && patch -p1 < /z/marked.patch \
         && npm run build \
         && cp -pv marked.min.js /z/dist/marked.js \
-        && cp -pv lib/marked.js /z/dist/marked.full.js \
         && mkdir -p /z/nodepkgs \
         && ln -s $(pwd) /z/nodepkgs/marked
 #        && npm run test \
@@ -102,9 +101,10 @@ COPY    easymde.patch /z/
 RUN     cd easy-markdown-editor-$ver_mde \
         && patch -p1 < /z/easymde.patch \
         && sed -ri 's`https://registry.npmjs.org/marked/-/marked-[0-9\.]+.tgz`file:/z/nodepkgs/marked`' package-lock.json \
+        && sed -ri 's`https://registry.npmjs.org/codemirror/-/codemirror-[0-9\.]+.tgz`file:/z/nodepkgs/codemirror`' package-lock.json \
         && sed -ri 's`("marked": ")[^"]+`\1file:/z/nodepkgs/marked`' ./package.json \
         && sed -ri 's`("codemirror": ")[^"]+`\1file:/z/nodepkgs/codemirror`' ./package.json \
-        && sed -ri 's`^var marked = require\(.marked/lib/marked.\);$`var marked = window.marked;`' src/js/easymde.js \
+        && sed -ri 's`^var marked = require\(.marked.\).marked;$`var marked = window.marked;`' src/js/easymde.js \
         && npm install
 
 COPY    easymde-ln.patch /z/
@@ -118,6 +118,7 @@ RUN     cd easy-markdown-editor-$ver_mde \
 # build fontawesome and scp
 COPY    mini-fa.sh /z
 COPY    mini-fa.css /z
+COPY    shiftbase.py /z
 RUN     /bin/ash /z/mini-fa.sh
 
 

scripts/deps-docker/codemirror.patch

@@ -1,6 +1,6 @@
-diff -NarU2 codemirror-5.59.3-orig/mode/gfm/gfm.js codemirror-5.59.3/mode/gfm/gfm.js
---- codemirror-5.59.3-orig/mode/gfm/gfm.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/mode/gfm/gfm.js	2021-02-21 20:42:02.166174775 +0000
+diff -wNarU2 codemirror-5.65.1-orig/mode/gfm/gfm.js codemirror-5.65.1/mode/gfm/gfm.js
+--- codemirror-5.65.1-orig/mode/gfm/gfm.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/mode/gfm/gfm.js	2022-02-09 22:50:18.145862052 +0100
 @@ -97,5 +97,5 @@
          }
        }
@@ -15,9 +15,9 @@ diff -NarU2 codemirror-5.59.3-orig/mode/gfm/gfm.js codemirror-5.59.3/mode/gfm/gf
 +      }*/
        stream.next();
        return null;
-diff -NarU2 codemirror-5.59.3-orig/mode/meta.js codemirror-5.59.3/mode/meta.js
---- codemirror-5.59.3-orig/mode/meta.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/mode/meta.js	2021-02-21 20:42:54.798742821 +0000
+diff -wNarU2 codemirror-5.65.1-orig/mode/meta.js codemirror-5.65.1/mode/meta.js
+--- codemirror-5.65.1-orig/mode/meta.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/mode/meta.js	2022-02-09 22:50:18.145862052 +0100
 @@ -13,4 +13,5 @@
  
    CodeMirror.modeInfo = [
@@ -62,10 +62,10 @@ diff -NarU2 codemirror-5.59.3-orig/mode/meta.js codemirror-5.59.3/mode/meta.js
 +    */
    ];
    // Ensure all modes have a mime property for backwards compatibility
-diff -NarU2 codemirror-5.59.3-orig/src/display/selection.js codemirror-5.59.3/src/display/selection.js
---- codemirror-5.59.3-orig/src/display/selection.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/display/selection.js	2021-02-21 20:44:14.860894328 +0000
-@@ -84,29 +84,21 @@
+diff -wNarU2 codemirror-5.65.1-orig/src/display/selection.js codemirror-5.65.1/src/display/selection.js
+--- codemirror-5.65.1-orig/src/display/selection.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/display/selection.js	2022-02-09 22:50:18.145862052 +0100
+@@ -96,29 +96,21 @@
      let order = getOrder(lineObj, doc.direction)
      iterateBidiSections(order, fromArg || 0, toArg == null ? lineLen : toArg, (from, to, dir, i) => {
 -      let ltr = dir == "ltr"
@@ -105,24 +105,24 @@ diff -NarU2 codemirror-5.59.3-orig/src/display/selection.js codemirror-5.59.3/sr
 +          botRight = openEnd && last ? rightSide : toPos.right
          add(topLeft, fromPos.top, topRight - topLeft, fromPos.bottom)
          if (fromPos.bottom < toPos.top) add(leftSide, fromPos.bottom, null, toPos.top)
-diff -NarU2 codemirror-5.59.3-orig/src/input/ContentEditableInput.js codemirror-5.59.3/src/input/ContentEditableInput.js
---- codemirror-5.59.3-orig/src/input/ContentEditableInput.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/input/ContentEditableInput.js	2021-02-21 20:44:33.273953867 +0000
-@@ -399,4 +399,5 @@
+diff -wNarU2 codemirror-5.65.1-orig/src/input/ContentEditableInput.js codemirror-5.65.1/src/input/ContentEditableInput.js
+--- codemirror-5.65.1-orig/src/input/ContentEditableInput.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/input/ContentEditableInput.js	2022-02-09 22:50:18.145862052 +0100
+@@ -400,4 +400,5 @@
    let info = mapFromLineView(view, line, pos.line)
  
 +  /*
    let order = getOrder(line, cm.doc.direction), side = "left"
    if (order) {
-@@ -404,4 +405,5 @@
+@@ -405,4 +406,5 @@
      side = partPos % 2 ? "right" : "left"
    }
 +  */
    let result = nodeAndOffsetInLineMap(info.map, pos.ch, side)
    result.offset = result.collapse == "right" ? result.end : result.start
-diff -NarU2 codemirror-5.59.3-orig/src/input/movement.js codemirror-5.59.3/src/input/movement.js
---- codemirror-5.59.3-orig/src/input/movement.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/input/movement.js	2021-02-21 20:45:12.763093671 +0000
+diff -wNarU2 codemirror-5.65.1-orig/src/input/movement.js codemirror-5.65.1/src/input/movement.js
+--- codemirror-5.65.1-orig/src/input/movement.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/input/movement.js	2022-02-09 22:50:18.145862052 +0100
 @@ -15,4 +15,5 @@
  
  export function endOfLine(visually, cm, lineObj, lineNo, dir) {
@@ -146,9 +146,16 @@ diff -NarU2 codemirror-5.59.3-orig/src/input/movement.js codemirror-5.59.3/src/i
    return null
 +  */
  }
-diff -NarU2 codemirror-5.59.3-orig/src/line/line_data.js codemirror-5.59.3/src/line/line_data.js
---- codemirror-5.59.3-orig/src/line/line_data.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/line/line_data.js	2021-02-21 20:45:36.472549599 +0000
+diff -wNarU2 codemirror-5.65.1-orig/src/line/line_data.js codemirror-5.65.1/src/line/line_data.js
+--- codemirror-5.65.1-orig/src/line/line_data.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/line/line_data.js	2022-02-09 22:54:11.542722046 +0100
+@@ -3,5 +3,5 @@
+ import { elt, eltP, joinClasses } from "../util/dom.js"
+ import { eventMixin, signal } from "../util/event.js"
+-import { hasBadBidiRects, zeroWidthElement } from "../util/feature_detection.js"
++import { zeroWidthElement } from "../util/feature_detection.js"
+ import { lst, spaceStr } from "../util/misc.js"
+ 
 @@ -79,6 +79,6 @@
      // Optionally wire in some hacks into the token-rendering
      // algorithm, to deal with browser quirks.
@@ -158,10 +165,10 @@ diff -NarU2 codemirror-5.59.3-orig/src/line/line_data.js codemirror-5.59.3/src/l
 +    //  builder.addToken = buildTokenBadBidi(builder.addToken, order)
      builder.map = []
      let allowFrontierUpdate = lineView != cm.display.externalMeasured && lineNo(line)
-diff -NarU2 codemirror-5.59.3-orig/src/measurement/position_measurement.js codemirror-5.59.3/src/measurement/position_measurement.js
---- codemirror-5.59.3-orig/src/measurement/position_measurement.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/measurement/position_measurement.js	2021-02-21 20:50:52.372945293 +0000
-@@ -380,5 +380,6 @@
+diff -wNarU2 codemirror-5.65.1-orig/src/measurement/position_measurement.js codemirror-5.65.1/src/measurement/position_measurement.js
+--- codemirror-5.65.1-orig/src/measurement/position_measurement.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/measurement/position_measurement.js	2022-02-09 22:50:18.145862052 +0100
+@@ -382,5 +382,6 @@
      sticky = "after"
    }
 -  if (!order) return get(sticky == "before" ? ch - 1 : ch, sticky == "before")
@@ -169,39 +176,39 @@ diff -NarU2 codemirror-5.59.3-orig/src/measurement/position_measurement.js codem
 +  /*
  
    function getBidi(ch, partPos, invert) {
-@@ -391,4 +392,5 @@
+@@ -393,4 +394,5 @@
    if (other != null) val.other = getBidi(ch, other, sticky != "before")
    return val
 +  */
  }
  
-@@ -468,4 +470,5 @@
+@@ -470,4 +472,5 @@
    let begin = 0, end = lineObj.text.length, ltr = true
  
 +  /*
    let order = getOrder(lineObj, cm.doc.direction)
    // If the line isn't plain left-to-right text, first figure out
-@@ -482,4 +485,5 @@
+@@ -484,4 +487,5 @@
      end = ltr ? part.to : part.from - 1
    }
 +  */
  
    // A binary search to find the first character whose bounding box
-@@ -526,4 +530,5 @@
+@@ -528,4 +532,5 @@
  }
  
 +/*
  function coordsBidiPart(cm, lineObj, lineNo, preparedMeasure, order, x, y) {
    // Bidi parts are sorted left-to-right, and in a non-line-wrapping
-@@ -580,4 +585,5 @@
+@@ -582,4 +587,5 @@
    return part
  }
 +*/
  
  let measureText
-diff -NarU2 codemirror-5.59.3-orig/src/util/bidi.js codemirror-5.59.3/src/util/bidi.js
---- codemirror-5.59.3-orig/src/util/bidi.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/util/bidi.js	2021-02-21 20:52:18.168092225 +0000
+diff -wNarU2 codemirror-5.65.1-orig/src/util/bidi.js codemirror-5.65.1/src/util/bidi.js
+--- codemirror-5.65.1-orig/src/util/bidi.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/util/bidi.js	2022-02-09 22:50:18.145862052 +0100
 @@ -4,5 +4,5 @@
  
  export function iterateBidiSections(order, from, to, f) {
@@ -259,9 +266,9 @@ diff -NarU2 codemirror-5.59.3-orig/src/util/bidi.js codemirror-5.59.3/src/util/b
 -  return order
 +  return false;
  }
-diff -NarU2 codemirror-5.59.3-orig/src/util/feature_detection.js codemirror-5.59.3/src/util/feature_detection.js
---- codemirror-5.59.3-orig/src/util/feature_detection.js	2021-02-20 21:24:57.000000000 +0000
-+++ codemirror-5.59.3/src/util/feature_detection.js	2021-02-21 20:49:22.191269270 +0000
+diff -wNarU2 codemirror-5.65.1-orig/src/util/feature_detection.js codemirror-5.65.1/src/util/feature_detection.js
+--- codemirror-5.65.1-orig/src/util/feature_detection.js	2022-01-20 13:06:23.000000000 +0100
++++ codemirror-5.65.1/src/util/feature_detection.js	2022-02-09 22:50:18.145862052 +0100
 @@ -25,4 +25,5 @@
  }
  

scripts/deps-docker/easymde-marked6.patch

@@ -0,0 +1,12 @@
+diff --git a/src/js/easymde.js b/src/js/easymde.js
+--- a/src/js/easymde.js
++++ b/src/js/easymde.js
+@@ -1962,7 +1962,7 @@ EasyMDE.prototype.markdown = function (text) {
+         marked.setOptions(markedOptions);
+ 
+         // Convert the markdown to HTML
+-        var htmlText = marked(text);
++        var htmlText = marked.parse(text);
+ 
+         // Sanitize HTML
+         if (this.options.renderingConfig && typeof this.options.renderingConfig.sanitizerFunction === 'function') {

scripts/deps-docker/easymde.patch

@@ -1,52 +1,52 @@
-diff -NarU2 easy-markdown-editor-2.14.0-orig/gulpfile.js easy-markdown-editor-2.14.0/gulpfile.js
---- easy-markdown-editor-2.14.0-orig/gulpfile.js	2021-02-14 12:11:48.000000000 +0000
-+++ easy-markdown-editor-2.14.0/gulpfile.js	2021-02-21 20:55:37.134701007 +0000
+diff -wNarU2 easy-markdown-editor-2.16.1-orig/gulpfile.js easy-markdown-editor-2.16.1/gulpfile.js
+--- easy-markdown-editor-2.16.1-orig/gulpfile.js	2022-01-14 23:27:44.000000000 +0100
++++ easy-markdown-editor-2.16.1/gulpfile.js	2022-02-09 23:06:01.694592535 +0100
 @@ -25,5 +25,4 @@
      './node_modules/codemirror/lib/codemirror.css',
      './src/css/*.css',
 -    './node_modules/codemirror-spell-checker/src/css/spell-checker.css',
  ];
  
-diff -NarU2 easy-markdown-editor-2.14.0-orig/package.json easy-markdown-editor-2.14.0/package.json
---- easy-markdown-editor-2.14.0-orig/package.json	2021-02-14 12:11:48.000000000 +0000
-+++ easy-markdown-editor-2.14.0/package.json	2021-02-21 20:55:47.761190082 +0000
-@@ -21,5 +21,4 @@
-     "dependencies": {
-         "codemirror": "^5.59.2",
+diff -wNarU2 easy-markdown-editor-2.16.1-orig/package.json easy-markdown-editor-2.16.1/package.json
+--- easy-markdown-editor-2.16.1-orig/package.json	2022-01-14 23:27:44.000000000 +0100
++++ easy-markdown-editor-2.16.1/package.json	2022-02-09 23:06:24.778501888 +0100
+@@ -23,5 +23,4 @@
+         "@types/marked": "^4.0.1",
+         "codemirror": "^5.63.1",
 -        "codemirror-spell-checker": "1.1.2",
-         "marked": "^2.0.0"
+         "marked": "^4.0.10"
      },
-diff -NarU2 easy-markdown-editor-2.14.0-orig/src/js/easymde.js easy-markdown-editor-2.14.0/src/js/easymde.js
---- easy-markdown-editor-2.14.0-orig/src/js/easymde.js	2021-02-14 12:11:48.000000000 +0000
-+++ easy-markdown-editor-2.14.0/src/js/easymde.js	2021-02-21 20:57:09.143171536 +0000
+diff -wNarU2 easy-markdown-editor-2.16.1-orig/src/js/easymde.js easy-markdown-editor-2.16.1/src/js/easymde.js
+--- easy-markdown-editor-2.16.1-orig/src/js/easymde.js	2022-01-14 23:27:44.000000000 +0100
++++ easy-markdown-editor-2.16.1/src/js/easymde.js	2022-02-09 23:07:21.203131415 +0100
 @@ -12,5 +12,4 @@
  require('codemirror/mode/gfm/gfm.js');
  require('codemirror/mode/xml/xml.js');
 -var CodeMirrorSpellChecker = require('codemirror-spell-checker');
- var marked = require('marked/lib/marked');
+ var marked = require('marked').marked;
  
-@@ -1762,9 +1761,4 @@
+@@ -1816,9 +1815,4 @@
          options.autosave.uniqueId = options.autosave.unique_id;
  
 -    // If overlay mode is specified and combine is not provided, default it to true
 -    if (options.overlayMode && options.overlayMode.combine === undefined) {
--      options.overlayMode.combine = true;
+-        options.overlayMode.combine = true;
 -    }
 -
      // Update this options
      this.options = options;
-@@ -2003,28 +1997,7 @@
+@@ -2057,34 +2051,7 @@
      var mode, backdrop;
  
 -    // CodeMirror overlay mode
 -    if (options.overlayMode) {
--      CodeMirror.defineMode('overlay-mode', function(config) {
--        return CodeMirror.overlayMode(CodeMirror.getMode(config, options.spellChecker !== false ? 'spell-checker' : 'gfm'), options.overlayMode.mode, options.overlayMode.combine);
--      });
+-        CodeMirror.defineMode('overlay-mode', function (config) {
+-            return CodeMirror.overlayMode(CodeMirror.getMode(config, options.spellChecker !== false ? 'spell-checker' : 'gfm'), options.overlayMode.mode, options.overlayMode.combine);
+-        });
 -
--      mode = 'overlay-mode';
--      backdrop = options.parsingConfig;
--      backdrop.gitHubSpice = false;
+-        mode = 'overlay-mode';
+-        backdrop = options.parsingConfig;
+-        backdrop.gitHubSpice = false;
 -    } else {
          mode = options.parsingConfig;
          mode.name = 'gfm';
@@ -58,31 +58,35 @@ diff -NarU2 easy-markdown-editor-2.14.0-orig/src/js/easymde.js easy-markdown-edi
 -        backdrop.name = 'gfm';
 -        backdrop.gitHubSpice = false;
 -
--        CodeMirrorSpellChecker({
--            codeMirrorInstance: CodeMirror,
--        });
+-        if (typeof options.spellChecker === 'function') {
+-            options.spellChecker({
+-                codeMirrorInstance: CodeMirror,
+-            });
+-        } else {
+-            CodeMirrorSpellChecker({
+-                codeMirrorInstance: CodeMirror,
+-            });
+-        }
 -    }
  
      // eslint-disable-next-line no-unused-vars
-diff -NarU2 easy-markdown-editor-2.14.0-orig/types/easymde.d.ts easy-markdown-editor-2.14.0/types/easymde.d.ts
---- easy-markdown-editor-2.14.0-orig/types/easymde.d.ts	2021-02-14 12:11:48.000000000 +0000
-+++ easy-markdown-editor-2.14.0/types/easymde.d.ts	2021-02-21 20:57:42.492620979 +0000
-@@ -160,9 +160,4 @@
+diff -wNarU2 easy-markdown-editor-2.16.1-orig/types/easymde.d.ts easy-markdown-editor-2.16.1/types/easymde.d.ts
+--- easy-markdown-editor-2.16.1-orig/types/easymde.d.ts	2022-01-14 23:27:44.000000000 +0100
++++ easy-markdown-editor-2.16.1/types/easymde.d.ts	2022-02-09 23:07:55.427605243 +0100
+@@ -167,9 +167,4 @@
      }
  
 -    interface OverlayModeOptions {
--      mode: CodeMirror.Mode<any>
--      combine?: boolean
+-        mode: CodeMirror.Mode<any>;
+-        combine?: boolean;
 -    }
 -
-     interface Options {
-         autoDownloadFontAwesome?: boolean;
-@@ -214,7 +209,5 @@
+     interface SpellCheckerOptions {
+         codeMirrorInstance: CodeMirror.Editor;
+@@ -229,6 +224,4 @@
+         syncSideBySidePreviewScroll?: boolean;
  
-         promptTexts?: PromptTexts;
--        syncSideBySidePreviewScroll?: boolean;
+-        overlayMode?: OverlayModeOptions;
 -
--        overlayMode?: OverlayModeOptions
-+        syncSideBySidePreviewScroll?: boolean
+         direction?: 'ltr' | 'rtl';
      }
- }

scripts/deps-docker/marked-ln.patch

@@ -1,15 +1,15 @@
 diff --git a/src/Lexer.js b/src/Lexer.js
-adds linetracking to marked.js v3.0.4;
+adds linetracking to marked.js v4.0.6;
 add data-ln="%d" to most tags, %d is the source markdown line
 --- a/src/Lexer.js
 +++ b/src/Lexer.js
 @@ -50,4 +50,5 @@ function mangle(text) {
- module.exports = class Lexer {
+ export class Lexer {
    constructor(options) {
 +    this.ln = 1;  // like most editors, start couting from 1
      this.tokens = [];
      this.tokens.links = Object.create(null);
-@@ -127,4 +128,15 @@ module.exports = class Lexer {
+@@ -127,4 +128,15 @@ export class Lexer {
    }
  
 +  set_ln(token, ln = this.ln) {
@@ -25,7 +25,7 @@ add data-ln="%d" to most tags, %d is the source markdown line
 +
    /**
     * Lexing
-@@ -134,7 +146,11 @@ module.exports = class Lexer {
+@@ -134,7 +146,11 @@ export class Lexer {
        src = src.replace(/^ +$/gm, '');
      }
 -    let token, lastToken, cutSrc, lastParagraphClipped;
@@ -38,105 +38,105 @@ add data-ln="%d" to most tags, %d is the source markdown line
 +
        if (this.options.extensions
          && this.options.extensions.block
-@@ -142,4 +158,5 @@ module.exports = class Lexer {
+@@ -142,4 +158,5 @@ export class Lexer {
            if (token = extTokenizer.call({ lexer: this }, src, tokens)) {
              src = src.substring(token.raw.length);
 +            this.set_ln(token, ln);
              tokens.push(token);
              return true;
-@@ -153,4 +170,5 @@ module.exports = class Lexer {
+@@ -153,4 +170,5 @@ export class Lexer {
        if (token = this.tokenizer.space(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln); // is \n if not type
          if (token.type) {
            tokens.push(token);
-@@ -162,4 +180,5 @@ module.exports = class Lexer {
+@@ -162,4 +180,5 @@ export class Lexer {
        if (token = this.tokenizer.code(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          lastToken = tokens[tokens.length - 1];
          // An indented code block cannot interrupt a paragraph.
-@@ -177,4 +196,5 @@ module.exports = class Lexer {
+@@ -177,4 +196,5 @@ export class Lexer {
        if (token = this.tokenizer.fences(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -184,4 +204,5 @@ module.exports = class Lexer {
+@@ -184,4 +204,5 @@ export class Lexer {
        if (token = this.tokenizer.heading(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -191,4 +212,5 @@ module.exports = class Lexer {
+@@ -191,4 +212,5 @@ export class Lexer {
        if (token = this.tokenizer.hr(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -198,4 +220,5 @@ module.exports = class Lexer {
+@@ -198,4 +220,5 @@ export class Lexer {
        if (token = this.tokenizer.blockquote(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -205,4 +228,5 @@ module.exports = class Lexer {
+@@ -205,4 +228,5 @@ export class Lexer {
        if (token = this.tokenizer.list(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -212,4 +236,5 @@ module.exports = class Lexer {
+@@ -212,4 +236,5 @@ export class Lexer {
        if (token = this.tokenizer.html(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -219,4 +244,5 @@ module.exports = class Lexer {
+@@ -219,4 +244,5 @@ export class Lexer {
        if (token = this.tokenizer.def(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          lastToken = tokens[tokens.length - 1];
          if (lastToken && (lastToken.type === 'paragraph' || lastToken.type === 'text')) {
-@@ -236,4 +262,5 @@ module.exports = class Lexer {
+@@ -236,4 +262,5 @@ export class Lexer {
        if (token = this.tokenizer.table(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -243,4 +270,5 @@ module.exports = class Lexer {
+@@ -243,4 +270,5 @@ export class Lexer {
        if (token = this.tokenizer.lheading(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
          tokens.push(token);
          continue;
-@@ -263,4 +291,5 @@ module.exports = class Lexer {
+@@ -263,4 +291,5 @@ export class Lexer {
        }
        if (this.state.top && (token = this.tokenizer.paragraph(cutSrc))) {
 +        this.set_ln(token, ln);
          lastToken = tokens[tokens.length - 1];
          if (lastParagraphClipped && lastToken.type === 'paragraph') {
-@@ -280,4 +309,6 @@ module.exports = class Lexer {
+@@ -280,4 +309,6 @@ export class Lexer {
        if (token = this.tokenizer.text(src)) {
          src = src.substring(token.raw.length);
 +        this.set_ln(token, ln);
 +        this.ln++;
          lastToken = tokens[tokens.length - 1];
          if (lastToken && lastToken.type === 'text') {
-@@ -355,4 +386,5 @@ module.exports = class Lexer {
+@@ -355,4 +386,5 @@ export class Lexer {
            if (token = extTokenizer.call({ lexer: this }, src, tokens)) {
              src = src.substring(token.raw.length);
 +            this.ln = token.ln || this.ln;
              tokens.push(token);
              return true;
-@@ -420,4 +452,6 @@ module.exports = class Lexer {
+@@ -420,4 +452,6 @@ export class Lexer {
        if (token = this.tokenizer.br(src)) {
          src = src.substring(token.raw.length);
 +        // no need to reset (no more blockTokens anyways)
 +        token.ln = this.ln++;
          tokens.push(token);
          continue;
-@@ -462,4 +496,5 @@ module.exports = class Lexer {
+@@ -462,4 +496,5 @@ export class Lexer {
        if (token = this.tokenizer.inlineText(cutSrc, smartypants)) {
          src = src.substring(token.raw.length);
 +        this.ln = token.ln || this.ln;
@@ -145,13 +145,13 @@ add data-ln="%d" to most tags, %d is the source markdown line
 diff --git a/src/Parser.js b/src/Parser.js
 --- a/src/Parser.js
 +++ b/src/Parser.js
-@@ -18,4 +18,5 @@ module.exports = class Parser {
+@@ -18,4 +18,5 @@ export class Parser {
      this.textRenderer = new TextRenderer();
      this.slugger = new Slugger();
 +    this.ln = 0; // error indicator; should always be set >=1 from tokens
    }
  
-@@ -64,4 +65,8 @@ module.exports = class Parser {
+@@ -64,4 +65,8 @@ export class Parser {
      for (i = 0; i < l; i++) {
        token = tokens[i];
 +      // take line-numbers from tokens whenever possible
@@ -160,7 +160,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 +      this.renderer.tag_ln(this.ln);
  
        // Run any renderer extensions
-@@ -124,7 +129,10 @@ module.exports = class Parser {
+@@ -124,7 +129,10 @@ export class Parser {
              }
  
 -            body += this.renderer.tablerow(cell);
@@ -173,7 +173,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 +          out += this.renderer.tag_ln(token.ln).table(header, body);
            continue;
          }
-@@ -167,8 +175,12 @@ module.exports = class Parser {
+@@ -167,8 +175,12 @@ export class Parser {
  
              itemBody += this.parse(item.tokens, loose);
 -            body += this.renderer.listitem(itemBody, task, checked);
@@ -188,7 +188,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 +          out += this.renderer.tag_ln(token.ln).list(body, ordered, start);
            continue;
          }
-@@ -179,5 +191,6 @@ module.exports = class Parser {
+@@ -179,5 +191,6 @@ export class Parser {
          }
          case 'paragraph': {
 -          out += this.renderer.paragraph(this.parseInline(token.tokens));
@@ -196,7 +196,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 +          out += this.renderer.tag_ln(token.ln).paragraph(t);
            continue;
          }
-@@ -221,4 +234,7 @@ module.exports = class Parser {
+@@ -221,4 +234,7 @@ export class Parser {
        token = tokens[i];
  
 +      // another thing that only affects <br/> and other inlines
@@ -207,7 +207,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 diff --git a/src/Renderer.js b/src/Renderer.js
 --- a/src/Renderer.js
 +++ b/src/Renderer.js
-@@ -11,6 +11,12 @@ module.exports = class Renderer {
+@@ -11,6 +11,12 @@ export class Renderer {
    constructor(options) {
      this.options = options || defaults;
 +    this.ln = "";
@@ -220,7 +220,7 @@ diff --git a/src/Renderer.js b/src/Renderer.js
 +  
    code(code, infostring, escaped) {
      const lang = (infostring || '').match(/\S*/)[0];
-@@ -26,10 +32,10 @@ module.exports = class Renderer {
+@@ -26,10 +32,10 @@ export class Renderer {
  
      if (!lang) {
 -      return '<pre><code>'
@@ -233,55 +233,55 @@ diff --git a/src/Renderer.js b/src/Renderer.js
 +    return '<pre' + this.ln + '><code class="'
        + this.options.langPrefix
        + escape(lang, true)
-@@ -40,5 +46,5 @@ module.exports = class Renderer {
+@@ -40,5 +46,5 @@ export class Renderer {
  
    blockquote(quote) {
 -    return '<blockquote>\n' + quote + '</blockquote>\n';
 +    return '<blockquote' + this.ln + '>\n' + quote + '</blockquote>\n';
    }
  
-@@ -51,4 +57,5 @@ module.exports = class Renderer {
+@@ -51,4 +57,5 @@ export class Renderer {
        return '<h'
          + level
 +        + this.ln
          + ' id="'
          + this.options.headerPrefix
-@@ -61,5 +68,5 @@ module.exports = class Renderer {
+@@ -61,5 +68,5 @@ export class Renderer {
      }
      // ignore IDs
 -    return '<h' + level + '>' + text + '</h' + level + '>\n';
 +    return '<h' + level + this.ln + '>' + text + '</h' + level + '>\n';
    }
  
-@@ -75,5 +82,5 @@ module.exports = class Renderer {
+@@ -75,5 +82,5 @@ export class Renderer {
  
    listitem(text) {
 -    return '<li>' + text + '</li>\n';
 +    return '<li' + this.ln + '>' + text + '</li>\n';
    }
  
-@@ -87,5 +94,5 @@ module.exports = class Renderer {
+@@ -87,5 +94,5 @@ export class Renderer {
  
    paragraph(text) {
 -    return '<p>' + text + '</p>\n';
 +    return '<p' + this.ln + '>' + text + '</p>\n';
    }
  
-@@ -102,5 +109,5 @@ module.exports = class Renderer {
+@@ -102,5 +109,5 @@ export class Renderer {
  
    tablerow(content) {
 -    return '<tr>\n' + content + '</tr>\n';
 +    return '<tr' + this.ln + '>\n' + content + '</tr>\n';
    }
  
-@@ -127,5 +134,5 @@ module.exports = class Renderer {
+@@ -127,5 +134,5 @@ export class Renderer {
  
    br() {
 -    return this.options.xhtml ? '<br/>' : '<br>';
 +    return this.options.xhtml ? '<br' + this.ln + '/>' : '<br' + this.ln + '>';
    }
  
-@@ -153,5 +160,5 @@ module.exports = class Renderer {
+@@ -153,5 +160,5 @@ export class Renderer {
      }
  
 -    let out = '<img src="' + href + '" alt="' + text + '"';
@@ -291,7 +291,7 @@ diff --git a/src/Renderer.js b/src/Renderer.js
 diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 --- a/src/Tokenizer.js
 +++ b/src/Tokenizer.js
-@@ -301,4 +301,7 @@ module.exports = class Tokenizer {
+@@ -297,4 +297,7 @@ export class Tokenizer {
        const l = list.items.length;
  
 +      // each nested list gets +1 ahead; this hack makes every listgroup -1 but atleast it doesn't get infinitely bad

scripts/deps-docker/marked.patch

@@ -1,7 +1,7 @@
 diff --git a/src/Lexer.js b/src/Lexer.js
 --- a/src/Lexer.js
 +++ b/src/Lexer.js
-@@ -6,5 +6,5 @@ const { repeatString } = require('./helpers.js');
+@@ -6,5 +6,5 @@ import { repeatString } from './helpers.js';
  /**
   * smartypants text replacement
 - */
@@ -15,21 +15,21 @@ diff --git a/src/Lexer.js b/src/Lexer.js
 + *
  function mangle(text) {
    let out = '',
-@@ -465,5 +465,5 @@ module.exports = class Lexer {
+@@ -466,5 +466,5 @@ export class Lexer {
  
        // autolink
 -      if (token = this.tokenizer.autolink(src, mangle)) {
 +      if (token = this.tokenizer.autolink(src)) {
          src = src.substring(token.raw.length);
          tokens.push(token);
-@@ -472,5 +472,5 @@ module.exports = class Lexer {
+@@ -473,5 +473,5 @@ export class Lexer {
  
        // url (gfm)
 -      if (!this.state.inLink && (token = this.tokenizer.url(src, mangle))) {
 +      if (!this.state.inLink && (token = this.tokenizer.url(src))) {
          src = src.substring(token.raw.length);
          tokens.push(token);
-@@ -493,5 +493,5 @@ module.exports = class Lexer {
+@@ -494,5 +494,5 @@ export class Lexer {
          }
        }
 -      if (token = this.tokenizer.inlineText(cutSrc, smartypants)) {
@@ -39,14 +39,14 @@ diff --git a/src/Lexer.js b/src/Lexer.js
 diff --git a/src/Renderer.js b/src/Renderer.js
 --- a/src/Renderer.js
 +++ b/src/Renderer.js
-@@ -142,5 +142,5 @@ module.exports = class Renderer {
+@@ -142,5 +142,5 @@ export class Renderer {
  
    link(href, title, text) {
 -    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
 +    href = cleanUrl(this.options.baseUrl, href);
      if (href === null) {
        return text;
-@@ -155,5 +155,5 @@ module.exports = class Renderer {
+@@ -155,5 +155,5 @@ export class Renderer {
  
    image(href, title, text) {
 -    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
@@ -56,7 +56,7 @@ diff --git a/src/Renderer.js b/src/Renderer.js
 diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 --- a/src/Tokenizer.js
 +++ b/src/Tokenizer.js
-@@ -321,14 +321,7 @@ module.exports = class Tokenizer {
+@@ -320,14 +320,7 @@ export class Tokenizer {
          type: 'html',
          raw: cap[0],
 -        pre: !this.options.sanitizer
@@ -72,7 +72,7 @@ diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 -      }
        return token;
      }
-@@ -477,15 +470,9 @@ module.exports = class Tokenizer {
+@@ -476,15 +469,9 @@ export class Tokenizer {
  
        return {
 -        type: this.options.sanitize
@@ -90,7 +90,7 @@ diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 +        text: cap[0]
        };
      }
-@@ -672,10 +659,10 @@ module.exports = class Tokenizer {
+@@ -671,10 +658,10 @@ export class Tokenizer {
    }
  
 -  autolink(src, mangle) {
@@ -103,7 +103,7 @@ diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 +        text = escape(cap[1]);
          href = 'mailto:' + text;
        } else {
-@@ -700,10 +687,10 @@ module.exports = class Tokenizer {
+@@ -699,10 +686,10 @@ export class Tokenizer {
    }
  
 -  url(src, mangle) {
@@ -116,7 +116,7 @@ diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 +        text = escape(cap[0]);
          href = 'mailto:' + text;
        } else {
-@@ -737,12 +724,12 @@ module.exports = class Tokenizer {
+@@ -736,12 +723,12 @@ export class Tokenizer {
    }
  
 -  inlineText(src, smartypants) {
@@ -135,7 +135,7 @@ diff --git a/src/Tokenizer.js b/src/Tokenizer.js
 diff --git a/src/defaults.js b/src/defaults.js
 --- a/src/defaults.js
 +++ b/src/defaults.js
-@@ -9,12 +9,8 @@ function getDefaults() {
+@@ -9,12 +9,8 @@ export function getDefaults() {
      highlight: null,
      langPrefix: 'language-',
 -    mangle: true,
@@ -151,10 +151,10 @@ diff --git a/src/defaults.js b/src/defaults.js
 diff --git a/src/helpers.js b/src/helpers.js
 --- a/src/helpers.js
 +++ b/src/helpers.js
-@@ -64,18 +64,5 @@ function edit(regex, opt) {
+@@ -64,18 +64,5 @@ export function edit(regex, opt) {
  const nonWordAndColonTest = /[^\w:]/g;
  const originIndependentUrl = /^$|^[a-z][a-z0-9+.-]*:|^[?#]/i;
--function cleanUrl(sanitize, base, href) {
+-export function cleanUrl(sanitize, base, href) {
 -  if (sanitize) {
 -    let prot;
 -    try {
@@ -168,36 +168,30 @@ diff --git a/src/helpers.js b/src/helpers.js
 -      return null;
 -    }
 -  }
-+function cleanUrl(base, href) {
++export function cleanUrl(base, href) {
    if (base && !originIndependentUrl.test(href)) {
      href = resolveUrl(base, href);
-@@ -227,10 +214,4 @@ function findClosingBracket(str, b) {
+@@ -227,10 +214,4 @@ export function findClosingBracket(str, b) {
  }
  
--function checkSanitizeDeprecation(opt) {
+-export function checkSanitizeDeprecation(opt) {
 -  if (opt && opt.sanitize && !opt.silent) {
 -    console.warn('marked(): sanitize and sanitizer parameters are deprecated since version 0.7.0, should not be used and will be removed in the future. Read more here: https://marked.js.org/#/USING_ADVANCED.md#options');
 -  }
 -}
 -
  // copied from https://stackoverflow.com/a/5450113/806777
- function repeatString(pattern, count) {
-@@ -260,5 +241,4 @@ module.exports = {
-   rtrim,
-   findClosingBracket,
--  checkSanitizeDeprecation,
-   repeatString
- };
+ export function repeatString(pattern, count) {
 diff --git a/src/marked.js b/src/marked.js
 --- a/src/marked.js
 +++ b/src/marked.js
-@@ -7,5 +7,4 @@ const Slugger = require('./Slugger.js');
- const {
+@@ -7,5 +7,4 @@ import { Slugger } from './Slugger.js';
+ import {
    merge,
 -  checkSanitizeDeprecation,
    escape
- } = require('./helpers.js');
-@@ -35,5 +34,4 @@ function marked(src, opt, callback) {
+ } from './helpers.js';
+@@ -35,5 +34,4 @@ export function marked(src, opt, callback) {
  
    opt = merge({}, marked.defaults, opt || {});
 -  checkSanitizeDeprecation(opt);
@@ -219,37 +213,37 @@ diff --git a/src/marked.js b/src/marked.js
 diff --git a/test/bench.js b/test/bench.js
 --- a/test/bench.js
 +++ b/test/bench.js
-@@ -33,5 +33,4 @@ async function runBench(options) {
+@@ -37,5 +37,4 @@ export async function runBench(options) {
      breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
    });
-@@ -45,5 +44,4 @@ async function runBench(options) {
+@@ -49,5 +48,4 @@ export async function runBench(options) {
      breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
    });
-@@ -58,5 +56,4 @@ async function runBench(options) {
+@@ -62,5 +60,4 @@ export async function runBench(options) {
      breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
    });
-@@ -70,5 +67,4 @@ async function runBench(options) {
+@@ -74,5 +71,4 @@ export async function runBench(options) {
      breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
    });
-@@ -83,5 +79,4 @@ async function runBench(options) {
+@@ -87,5 +83,4 @@ export async function runBench(options) {
      breaks: false,
      pedantic: true,
 -    sanitize: false,
      smartLists: false
    });
-@@ -95,5 +90,4 @@ async function runBench(options) {
+@@ -99,5 +94,4 @@ export async function runBench(options) {
      breaks: false,
      pedantic: true,
 -    sanitize: false,
@@ -258,7 +252,7 @@ diff --git a/test/bench.js b/test/bench.js
 diff --git a/test/specs/run-spec.js b/test/specs/run-spec.js
 --- a/test/specs/run-spec.js
 +++ b/test/specs/run-spec.js
-@@ -22,9 +22,4 @@ function runSpecs(title, dir, showCompletionTable, options) {
+@@ -25,9 +25,4 @@ function runSpecs(title, dir, showCompletionTable, options) {
            }
  
 -          if (spec.options.sanitizer) {
@@ -268,77 +262,77 @@ diff --git a/test/specs/run-spec.js b/test/specs/run-spec.js
 -
            (spec.only ? fit : (spec.skip ? xit : it))('should ' + passFail + example, async() => {
              const before = process.hrtime();
-@@ -53,3 +48,2 @@ runSpecs('Original', './original', false, { gfm: false, pedantic: true });
+@@ -56,3 +51,2 @@ runSpecs('Original', './original', false, { gfm: false, pedantic: true });
  runSpecs('New', './new');
  runSpecs('ReDOS', './redos');
 -runSpecs('Security', './security', false, { silent: true }); // silent - do not show deprecation warning
 diff --git a/test/unit/Lexer-spec.js b/test/unit/Lexer-spec.js
 --- a/test/unit/Lexer-spec.js
 +++ b/test/unit/Lexer-spec.js
-@@ -589,5 +589,5 @@ paragraph
+@@ -635,5 +635,5 @@ paragraph
      });
  
 -    it('sanitize', () => {
 +    /*it('sanitize', () => {
        expectTokens({
          md: '<div>html</div>',
-@@ -607,5 +607,5 @@ paragraph
+@@ -653,5 +653,5 @@ paragraph
          ]
        });
 -    });
 +    });*/
    });
  
-@@ -652,5 +652,5 @@ paragraph
+@@ -698,5 +698,5 @@ paragraph
        });
  
 -      it('html sanitize', () => {
 +      /*it('html sanitize', () => {
          expectInlineTokens({
            md: '<div>html</div>',
-@@ -660,5 +660,5 @@ paragraph
+@@ -706,5 +706,5 @@ paragraph
            ]
          });
 -      });
 +      });*/
  
        it('link', () => {
-@@ -971,5 +971,5 @@ paragraph
+@@ -1017,5 +1017,5 @@ paragraph
          });
  
 -        it('autolink mangle email', () => {
 +        /*it('autolink mangle email', () => {
            expectInlineTokens({
              md: '<test@example.com>',
-@@ -991,5 +991,5 @@ paragraph
+@@ -1037,5 +1037,5 @@ paragraph
              ]
            });
 -        });
 +        });*/
  
          it('url', () => {
-@@ -1028,5 +1028,5 @@ paragraph
+@@ -1074,5 +1074,5 @@ paragraph
          });
  
 -        it('url mangle email', () => {
 +        /*it('url mangle email', () => {
            expectInlineTokens({
              md: 'test@example.com',
-@@ -1048,5 +1048,5 @@ paragraph
+@@ -1094,5 +1094,5 @@ paragraph
              ]
            });
 -        });
 +        });*/
        });
  
-@@ -1064,5 +1064,5 @@ paragraph
+@@ -1110,5 +1110,5 @@ paragraph
        });
  
 -      describe('smartypants', () => {
 +      /*describe('smartypants', () => {
          it('single quotes', () => {
            expectInlineTokens({
-@@ -1134,5 +1134,5 @@ paragraph
+@@ -1180,5 +1180,5 @@ paragraph
            });
          });
 -      });

scripts/deps-docker/mini-fa.sh

@@ -29,3 +29,10 @@ pyftsubset "$orig_woff" --unicodes-file=/z/icon.list --no-ignore-missing-unicode
 
 # scp is easier, just want basic latin
 pyftsubset /z/scp.woff2 --unicodes="20-7e,ab,b7,bb,2022" --no-ignore-missing-unicodes --flavor=woff2 --output-file=/z/dist/no-pk/scp.woff2 --verbose
+
+exit 0
+
+# kinda works but ruins hinting on windows, just use the old version of the font which has correct baseline
+python3 shiftbase.py /z/dist/no-pk/scp.woff2
+cd /z/dist/no-pk/
+mv scp.woff2.woff2 scp.woff2

scripts/deps-docker/shiftbase.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+
+import sys
+from fontTools.ttLib import TTFont, newTable
+
+
+def main():
+    woff = sys.argv[1]
+    font = TTFont(woff)
+    print(repr(font["hhea"].__dict__))
+    print(repr(font["OS/2"].__dict__))
+    # font["hhea"].ascent = round(base_asc * mul)
+    # font["hhea"].descent = round(base_desc * mul)
+    # font["OS/2"].usWinAscent = round(base_asc * mul)
+    font["OS/2"].usWinDescent = round(font["OS/2"].usWinDescent * 1.1)
+    font["OS/2"].sTypoDescender = round(font["OS/2"].sTypoDescender * 1.1)
+
+    try:
+        del font["post"].mapping["Delta#1"]
+    except:
+        pass
+
+    font.save(woff + ".woff2")
+
+
+if __name__ == "__main__":
+    main()

scripts/make-pypi-release.sh

@@ -86,8 +86,6 @@ function have() {
 	python -c "import $1; $1; $1.__version__"
 }
 
-mv copyparty/web/deps/marked.full.js.gz srv/ || true
-
 . buildenv/bin/activate
 have setuptools
 have wheel

scripts/make-sfx.sh

@@ -14,8 +14,6 @@ help() { exec cat <<'EOF'
 #
 # `gz` creates a gzip-compressed python sfx instead of bzip2
 #
-# `no-sh` makes just the python sfx, skips the sh/unix sfx
-#
 # `no-cm` saves ~82k by removing easymde/codemirror
 #   (the fancy markdown editor)
 #
@@ -64,8 +62,6 @@ pybin=$(command -v python3 || command -v python) || {
 }
 
 use_gz=
-do_sh=1
-do_py=1
 zopf=2560
 while [ ! -z "$1" ]; do
 	case $1 in
@@ -76,8 +72,6 @@ while [ ! -z "$1" ]; do
 		no-hl)  no_hl=1  ; ;;
 		no-dd)  no_dd=1  ; ;;
 		no-cm)  no_cm=1  ; ;;
-		no-sh)  do_sh=   ; ;;
-		no-py)  do_py=   ; ;;
 		fast)   zopf=100 ; ;;
 		*)      help     ; ;;
 	esac
@@ -107,7 +101,7 @@ tmpdir="$(
 [ $repack ] && {
 	old="$tmpdir/pe-copyparty"
 	echo "repack of files in $old"
-	cp -pR "$old/"*{dep-j2,copyparty} .
+	cp -pR "$old/"*{dep-j2,dep-ftp,copyparty} .
 }
 
 [ $repack ] || {
@@ -134,6 +128,27 @@ tmpdir="$(
 	mkdir dep-j2/
 	mv {markupsafe,jinja2} dep-j2/
 
+	echo collecting pyftpdlib
+	f="../build/pyftpdlib-1.5.6.tar.gz"
+	[ -e "$f" ] ||
+		(url=https://github.com/giampaolo/pyftpdlib/archive/refs/tags/release-1.5.6.tar.gz;
+		wget -O$f "$url" || curl -L "$url" >$f)
+
+	tar -zxf $f
+	mv pyftpdlib-release-*/pyftpdlib .
+	rm -rf pyftpdlib-release-* pyftpdlib/test
+
+	mkdir dep-ftp/
+	mv pyftpdlib dep-ftp/
+
+	echo collecting asyncore, asynchat
+	for n in asyncore.py asynchat.py; do
+		f=../build/$n
+		[ -e "$f" ] ||
+			(url=https://raw.githubusercontent.com/python/cpython/c4d45ee670c09d4f6da709df072ec80cb7dfad22/Lib/$n;
+			wget -O$f "$url" || curl -L "$url" >$f)
+	done
+
 	# msys2 tar is bad, make the best of it
 	echo collecting source
 	[ $clean ] && {
@@ -144,6 +159,12 @@ tmpdir="$(
 		(cd .. && tar -cf tar copyparty) && tar -xf ../tar
 	}
 	rm -f ../tar
+
+	# insert asynchat
+	mkdir copyparty/vend
+	for n in asyncore.py asynchat.py; do
+		awk 'NR<4||NR>27;NR==4{print"# license: https://opensource.org/licenses/ISC\n"}' ../build/$n >copyparty/vend/$n
+	done
 }
 
 ver=
@@ -245,7 +266,7 @@ rm have
 find | grep -E '\.py$' |
   grep -vE '__version__' |
   tr '\n' '\0' |
-  xargs -0 $pybin ../scripts/uncomment.py
+  xargs -0 "$pybin" ../scripts/uncomment.py
 
 f=dep-j2/jinja2/constants.py
 awk '/^LOREM_IPSUM_WORDS/{o=1;print "LOREM_IPSUM_WORDS = u\"a\"";next} !o; /"""/{o=0}' <$f >t
@@ -331,11 +352,18 @@ nf=$(ls -1 "$zdir"/arc.* | wc -l)
 
 
 echo gen tarlist
-for d in copyparty dep-j2; do find $d -type f; done |
+for d in copyparty dep-j2 dep-ftp; do find $d -type f; done |
 sed -r 's/(.*)\.(.*)/\2 \1/' | LC_ALL=C sort |
 sed -r 's/([^ ]*) (.*)/\2.\1/' | grep -vE '/list1?$' > list1
 
-(grep -vE '\.(gz|br)$' list1; grep -E '\.(gz|br)$' list1 | shuf) >list || true
+for n in {1..50}; do
+	(grep -vE '\.(gz|br)$' list1; grep -E '\.(gz|br)$' list1 | shuf) >list || true
+	s=$(md5sum list | cut -c-16)
+	grep -q $s "$zdir/h" && continue
+	echo $s >> "$zdir/h"
+	break
+done
+[ $n -eq 50 ] && exit
 
 echo creating tar
 args=(--owner=1000 --group=1000)
@@ -350,41 +378,27 @@ pe=bz2
 
 echo compressing tar
 # detect best level; bzip2 -7 is usually better than -9
-[ $do_py ] && { for n in {2..9}; do cp tar t.$n; $pc  -$n t.$n & done; wait; mv -v $(ls -1S t.*.$pe | tail -n 1) tar.bz2; }
-[ $do_sh ] && { for n in {2..9}; do cp tar t.$n; xz -ze$n t.$n & done; wait; mv -v $(ls -1S t.*.xz  | tail -n 1) tar.xz; }
+for n in {2..9}; do cp tar t.$n; $pc  -$n t.$n & done; wait; mv -v $(ls -1S t.*.$pe | tail -n 1) tar.bz2
 rm t.* || true
 exts=()
 
 
-[ $do_sh ] && {
-exts+=(.sh)
-echo creating unix sfx
-(
-	sed "s/PACK_TS/$ts/; s/PACK_HTS/$hts/; s/CPP_VER/$ver/" <../scripts/sfx.sh |
-	grep -E '^sfx_eof$' -B 9001;
-	cat tar.xz
-) >$sfx_out.sh
+echo creating sfx
+
+py=../scripts/sfx.py
+suf=
+[ $use_gz ] && {
+	sed -r 's/"r:bz2"/"r:gz"/' <$py >$py.t
+	py=$py.t
+	suf=-gz
 }
 
+"$pybin" $py --sfx-make tar.bz2 $ver $ts
+mv sfx.out $sfx_out$suf.py
 
-[ $do_py ] && {
-	echo creating generic sfx
-
-	py=../scripts/sfx.py
-	suf=
-	[ $use_gz ] && {
-		sed -r 's/"r:bz2"/"r:gz"/' <$py >$py.t
-		py=$py.t
-		suf=-gz
-	}
-
-	$pybin $py --sfx-make tar.bz2 $ver $ts
-	mv sfx.out $sfx_out$suf.py
-	
-	exts+=($suf.py)
-	[ $use_gz ] &&
-		rm $py
-}
+exts+=($suf.py)
+[ $use_gz ] &&
+	rm $py
 
 
 chmod 755 $sfx_out*
@@ -395,4 +409,4 @@ for ext in ${exts[@]}; do
 done
 
 # apk add bash python3 tar xz bzip2
-# while true; do ./make-sfx.sh; for f in ..//dist/copyparty-sfx.{sh,py}; do mv $f $f.$(wc -c <$f | awk '{print$1}'); done; done
+# while true; do ./make-sfx.sh; f=../dist/copyparty-sfx.py; mv $f $f.$(wc -c <$f | awk '{print$1}'); done

scripts/make-tgz-release.sh

@@ -35,8 +35,6 @@ ver="$1"
 	exit 1
 }
 
-mv copyparty/web/deps/marked.full.js.gz srv/ || true
-
 mkdir -p dist
 zip_path="$(pwd)/dist/copyparty-$ver.zip"
 tgz_path="$(pwd)/dist/copyparty-$ver.tar.gz"

scripts/rls.sh

@@ -4,33 +4,31 @@ set -e
 cd ~/dev/copyparty/scripts
 
 v=$1
-printf '%s\n' "$v" | grep -qE '^[0-9\.]+$' || exit 1
-grep -E "(${v//./, })" ../copyparty/__version__.py || exit 1
 
-git tag v$v
-git push origin --tags
+[ "$v" = sfx ] || {
+    printf '%s\n' "$v" | grep -qE '^[0-9\.]+$' || exit 1
+    grep -E "(${v//./, })" ../copyparty/__version__.py || exit 1
 
-rm -rf ../dist
+    git push all
+    git tag v$v
+    git push all --tags
 
-./make-pypi-release.sh u
-(cd .. && python3 ./setup.py clean2)
+    rm -rf ../dist
 
-./make-tgz-release.sh $v
+    ./make-pypi-release.sh u
+    (cd .. && python3 ./setup.py clean2)
+
+    ./make-tgz-release.sh $v
+}
 
 rm -f ../dist/copyparty-sfx.*
-./make-sfx.sh no-sh
-../dist/copyparty-sfx.py -h
+f=../dist/copyparty-sfx.py
+./make-sfx.sh
+$f -h
 
-ar=
 while true; do
-    for ((a=0; a<100; a++)); do
-        for f in ../dist/copyparty-sfx.{py,sh}; do
-            [ -e $f ] || continue;
-            mv $f $f.$(wc -c <$f | awk '{print$1}')
-        done
-        ./make-sfx.sh re $ar
-    done
-    ar=no-sh
+    mv $f $f.$(wc -c <$f | awk '{print$1}')
+    ./make-sfx.sh re $ar
 done
 
 # git tag -d v$v; git push --delete origin v$v

scripts/sfx.ls

@@ -11,6 +11,7 @@ copyparty/broker_mp.py,
 copyparty/broker_mpw.py,
 copyparty/broker_thr.py,
 copyparty/broker_util.py,
+copyparty/ftpd.py,
 copyparty/httpcli.py,
 copyparty/httpconn.py,
 copyparty/httpsrv.py,
@@ -31,6 +32,9 @@ copyparty/th_srv.py,
 copyparty/u2idx.py,
 copyparty/up2k.py,
 copyparty/util.py,
+copyparty/vend,
+copyparty/vend/asynchat.py,
+copyparty/vend/asyncore.py,
 copyparty/web,
 copyparty/web/baguettebox.js,
 copyparty/web/browser.css,

scripts/sfx.py

@@ -1,10 +1,10 @@
 #!/usr/bin/env python3
 # coding: latin-1
 from __future__ import print_function, unicode_literals
-
 import re, os, sys, time, shutil, signal, threading, tarfile, hashlib, platform, tempfile, traceback
 import subprocess as sp
 
+
 """
 to edit this file, use HxD or "vim -b"
   (there is compressed stuff at the end)
@@ -20,6 +20,7 @@ the archive data is attached after the b"\n# eof\n" archive marker,
   b"\n# " decodes to b""
 """
 
+
 # set by make-sfx.sh
 VER = None
 SIZE = None
@@ -367,8 +368,9 @@ def confirm(rv):
     sys.exit(rv or 1)
 
 
-def run(tmp, j2):
+def run(tmp, j2, ftp):
     msg("jinja2:", j2 or "bundled")
+    msg("pyftpd:", ftp or "bundled")
     msg("sfxdir:", tmp)
     msg()
 
@@ -386,9 +388,8 @@ def run(tmp, j2):
     t.daemon = True
     t.start()
 
-    ld = [tmp, os.path.join(tmp, "dep-j2")]
-    if j2:
-        del ld[-1]
+    ld = (("", ""), (j2, "dep-j2"), (ftp, "dep-ftp"))
+    ld = [os.path.join(tmp, b) for a, b in ld if not a]
 
     if any([re.match(r"^-.*j[0-9]", x) for x in sys.argv]):
         run_s(ld)
@@ -461,7 +462,12 @@ def main():
         j2 = None
 
     try:
-        run(tmp, j2)
+        from pyftpdlib.__init__ import __ver__ as ftp
+    except:
+        ftp = None
+
+    try:
+        run(tmp, j2, ftp)
     except SystemExit as ex:
         c = ex.code
         if c not in [0, -15]:

setup.py

@@ -112,7 +112,12 @@ args = {
     "data_files": data_files,
     "packages": find_packages(),
     "install_requires": ["jinja2"],
-    "extras_require": {"thumbnails": ["Pillow"], "audiotags": ["mutagen"]},
+    "extras_require": {
+        "thumbnails": ["Pillow"],
+        "audiotags": ["mutagen"],
+        "ftpd": ["pyftpdlib"],
+        "ftps": ["pyopenssl"],
+    },
     "entry_points": {"console_scripts": ["copyparty = copyparty.__main__:main"]},
     "scripts": ["bin/copyparty-fuse.py", "bin/up2k.py"],
     "cmdclass": {"clean2": clean2},

tests/test_httpcli.py

@@ -29,6 +29,9 @@ class Cfg(Namespace):
             v=v or [],
             c=c,
             rproxy=0,
+            rsp_slp=0,
+            s_wr_slp=0,
+            s_wr_sz=512 * 1024,
             ed=False,
             nw=False,
             unpost=600,
@@ -48,9 +51,13 @@ class Cfg(Namespace):
             mte="a",
             mth="",
             textfiles="",
+            doctitle="",
+            html_head="",
             hist=None,
             no_idx=None,
             no_hash=None,
+            force_js=False,
+            no_robots=False,
             js_browser=None,
             css_browser=None,
             **{k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr no_acode".split()}

tests/test_vfs.py

@@ -17,12 +17,14 @@ from copyparty import util
 
 class Cfg(Namespace):
     def __init__(self, a=None, v=None, c=None):
-        ex = "nw e2d e2ds e2dsa e2t e2ts e2tsr no_logues no_readme no_acode"
+        ex = "nw e2d e2ds e2dsa e2t e2ts e2tsr no_logues no_readme no_acode force_js no_robots"
         ex = {k: False for k in ex.split()}
         ex2 = {
             "mtp": [],
             "mte": "a",
             "mth": "",
+            "doctitle": "",
+            "html_head": "",
             "hist": None,
             "no_idx": None,
             "no_hash": None,
@@ -31,6 +33,9 @@ class Cfg(Namespace):
             "no_voldump": True,
             "re_maxage": 0,
             "rproxy": 0,
+            "rsp_slp": 0,
+            "s_wr_slp": 0,
+            "s_wr_sz": 512 * 1024,
         }
         ex.update(ex2)
         super(Cfg, self).__init__(a=a or [], v=v or [], c=c, **ex)

tests/util.py

@@ -109,6 +109,9 @@ class VSock(object):
         self._reply += buf
         return len(buf)
 
+    def getsockname(self):
+        return ("a", 1)
+
 
 class VHttpSrv(object):
     def __init__(self):