v1.5.5

posted from warzaw airport otw to japan

.vscode/launch.json

@@ -8,6 +8,7 @@
             "module": "copyparty",
             "console": "integratedTerminal",
             "cwd": "${workspaceFolder}",
+            "justMyCode": false,
             "args": [
                 //"-nw",
                 "-ed",

.vscode/settings.json

@@ -52,9 +52,11 @@
         "--disable=missing-module-docstring",
         "--disable=missing-class-docstring",
         "--disable=missing-function-docstring",
+        "--disable=import-outside-toplevel",
         "--disable=wrong-import-position",
         "--disable=raise-missing-from",
         "--disable=bare-except",
+        "--disable=broad-except",
         "--disable=invalid-name",
         "--disable=line-too-long",
         "--disable=consider-using-f-string"
@@ -64,6 +66,7 @@
     "editor.formatOnSave": true,
     "[html]": {
         "editor.formatOnSave": false,
+        "editor.autoIndent": "keep",
     },
     "[css]": {
         "editor.formatOnSave": false,

README.md

@@ -230,7 +230,7 @@ browser-specific:
 * Android-Chrome: increase "parallel uploads" for higher speed (android bug)
 * Android-Firefox: takes a while to select files (their fix for ☝️)
 * Desktop-Firefox: ~~may use gigabytes of RAM if your files are massive~~ *seems to be OK now*
-* Desktop-Firefox: may stop you from deleting files you've uploaded until you visit `about:memory` and click `Minimize memory usage`
+* Desktop-Firefox: [may stop you from unplugging USB flashdrives](https://bugzilla.mozilla.org/show_bug.cgi?id=1792598) until you visit `about:memory` and click `Minimize memory usage`
 
 server-os-specific:
 * RHEL8 / Rocky8: you can run copyparty using `/usr/libexec/platform-python`
@@ -248,23 +248,15 @@ server-os-specific:
 * Windows: if the `up2k.db` (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
   * use `--hist` or the `hist` volflag (`-v [...]:c,hist=/tmp/foo`) to place the db on a local disk instead
 * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
-* [the database can get stuck](https://github.com/9001/copyparty/issues/10)
-  * has only happened once but that is once too many
-  * luckily not dangerous for file integrity and doesn't really stop uploads or anything like that
-  * but would really appreciate some logs if anyone ever runs into it again
 * probably more, pls let me know
 
 ## not my bugs
 
 * [Chrome issue 1317069](https://bugs.chromium.org/p/chromium/issues/detail?id=1317069) -- if you try to upload a folder which contains symlinks by dragging it into the browser, the symlinked files will not get uploaded
 
-* [Chrome issue 1354816](https://bugs.chromium.org/p/chromium/issues/detail?id=1354816) -- chrome may eat all RAM uploading over plaintext http with `mt` enabled
+* [Chrome issue 1352210](https://bugs.chromium.org/p/chromium/issues/detail?id=1352210) -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive)
 
-  * more amusingly, [Chrome issue 1354800](https://bugs.chromium.org/p/chromium/issues/detail?id=1354800) -- chrome may eat all RAM uploading in general (altho you probably won't run into this one)
-
-* [Chrome issue 1352210](https://bugs.chromium.org/p/chromium/issues/detail?id=1352210) -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive and likely to run into the above gc bugs)
-
-* [Firefox issue 1790500](https://bugzilla.mozilla.org/show_bug.cgi?id=1790500) -- sometimes forgets to close filedescriptors during upload so the browser can crash after ~4000 files
+* [Firefox issue 1790500](https://bugzilla.mozilla.org/show_bug.cgi?id=1790500) -- entire browser can crash after uploading ~4000 small files
 
 * iPhones: the volume control doesn't work because [apple doesn't want it to](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html#//apple_ref/doc/uid/TP40009523-CH5-SW11)
   * *future workaround:* enable the equalizer, make it all-zero, and set a negative boost to reduce the volume

SECURITY.md

@@ -0,0 +1,9 @@
+# Security Policy
+
+if you hit something extra juicy pls let me know on either of the following
+* email -- `copyparty@ocv.ze` except `ze` should be `me`
+* [mastodon dm](https://layer8.space/@tripflag) -- `@tripflag@layer8.space`
+* [github private vulnerability report](https://github.com/9001/copyparty/security/advisories/new), wow that form is complicated
+* [twitter dm](https://twitter.com/tripflag) (if im somehow not banned yet)
+
+no bug bounties sorry! all i can offer is greetz in the release notes

bin/partyfuse.py

@@ -997,7 +997,7 @@ def main():
     ap.add_argument(
         "-cf", metavar="NUM_BLOCKS", type=int, default=nf, help="file cache"
     )
-    ap.add_argument("-a", metavar="PASSWORD", help="password")
+    ap.add_argument("-a", metavar="PASSWORD", help="password or $filepath")
     ap.add_argument("-d", action="store_true", help="enable debug")
     ap.add_argument("-te", metavar="PEM_FILE", help="certificate to expect/verify")
     ap.add_argument("-td", action="store_true", help="disable certificate check")

bin/up2k.py

@@ -994,7 +994,7 @@ source file/folder selection uses rsync syntax, meaning that:
     ap.add_argument("url", type=unicode, help="server url, including destination folder")
     ap.add_argument("files", type=unicode, nargs="+", help="files and/or folders to process")
     ap.add_argument("-v", action="store_true", help="verbose")
-    ap.add_argument("-a", metavar="PASSWORD", help="password")
+    ap.add_argument("-a", metavar="PASSWORD", help="password or $filepath")
     ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
     ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
     
@@ -1041,6 +1041,12 @@ source file/folder selection uses rsync syntax, meaning that:
     if "://" not in ar.url:
         ar.url = "http://" + ar.url
 
+    if ar.a and ar.a.startswith("$"):
+        fn = ar.a[1:]
+        print("reading password from file [{}]".format(fn))
+        with open(fn, "rb") as f:
+            ar.a = f.read().decode("utf-8").strip()
+
     if ar.cls:
         print("\x1b\x5b\x48\x1b\x5b\x32\x4a\x1b\x5b\x33\x4a", end="")
 

contrib/openrc/copyparty

@@ -14,5 +14,5 @@ name="$SVCNAME"
 command_background=true
 pidfile="/var/run/$SVCNAME.pid"
 
-command="/usr/bin/python /usr/local/bin/copyparty-sfx.py"
+command="/usr/bin/python3 /usr/local/bin/copyparty-sfx.py"
 command_args="-q -v /mnt::rw"

copyparty/__main__.py

@@ -253,7 +253,7 @@ def ensure_locale() -> None:
         except:
             continue
 
-    t = "setlocale {} failed,\n  sorting and dates will be funky"
+    t = "setlocale {} failed,\n  sorting and dates might get funky\n"
     warn(t.format(safe))
 
 

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 5, 3)
+VERSION = (1, 5, 5)
 CODENAME = "babel"
-BUILD_DT = (2022, 12, 13)
+BUILD_DT = (2022, 12, 30)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/mdns.py

@@ -25,7 +25,7 @@ from .stolen.dnslib import (
     DNSQuestion,
     DNSRecord,
 )
-from .util import CachedSet, Daemon, Netdev, min_ex
+from .util import CachedSet, Daemon, Netdev, list_ips, min_ex
 
 if TYPE_CHECKING:
     from .svchub import SvcHub
@@ -55,6 +55,7 @@ class MDNS_Sck(MC_Sck):
         self.bp_bye = b""
 
         self.last_tx = 0.0
+        self.tx_ex = False
 
 
 class MDNS(MCast):
@@ -374,6 +375,14 @@ class MDNS(MCast):
                 # avahi broadcasting 127.0.0.1-only packets
                 return
 
+            # check if we've been given additional IPs
+            for ip in list_ips():
+                if ip in cips:
+                    self.sips.add(ip)
+
+            if not self.sips.isdisjoint(cips):
+                return
+
             t = "mdns zeroconf: "
             if self.probing:
                 t += "Cannot start; hostname '{}' is occupied"
@@ -507,6 +516,15 @@ class MDNS(MCast):
         if now < srv.last_tx + cooldown:
             return False
 
-        srv.sck.sendto(msg, (srv.grp, 5353))
-        srv.last_tx = now
+        try:
+            srv.sck.sendto(msg, (srv.grp, 5353))
+            srv.last_tx = now
+        except Exception as ex:
+            if srv.tx_ex:
+                return True
+
+            srv.tx_ex = True
+            t = "tx({},|{}|,{}): {}"
+            self.log(t.format(srv.ip, len(msg), cooldown, ex), 3)
+
         return True

copyparty/ssdp.py

@@ -75,6 +75,7 @@ class SSDPr(object):
 
         c = html_escape
         sip, sport = hc.s.getsockname()[:2]
+        sip = sip.replace("::ffff:", "")
         proto = "https" if self.args.https_only else "http"
         ubase = "{}://{}:{}".format(proto, sip, sport)
         zsl = self.args.zsl
@@ -160,7 +161,7 @@ class SSDPd(MCast):
 
         self.rxc.add(buf)
         if not buf.startswith(b"M-SEARCH * HTTP/1."):
-            raise Exception("not an ssdp message")
+            return
 
         if not self.ptn_st.search(buf):
             return
@@ -184,7 +185,8 @@ BOOTID.UPNP.ORG: 0
 CONFIGID.UPNP.ORG: 1
 
 """
-        zs = zs.format(formatdate(usegmt=True), srv.ip, srv.hport, self.args.zsid)
+        v4 = srv.ip.replace("::ffff:", "")
+        zs = zs.format(formatdate(usegmt=True), v4, srv.hport, self.args.zsid)
         zb = zs[1:].replace("\n", "\r\n").encode("utf-8", "replace")
         srv.sck.sendto(zb, addr[:2])
 

copyparty/th_srv.py

@@ -3,6 +3,7 @@ from __future__ import print_function, unicode_literals
 
 import base64
 import hashlib
+import logging
 import os
 import shutil
 import subprocess as sp
@@ -61,12 +62,16 @@ try:
         HAVE_AVIF = True
     except:
         pass
+
+    logging.getLogger("PIL").setLevel(logging.WARNING)
 except:
     pass
 
 try:
     HAVE_VIPS = True
     import pyvips
+
+    logging.getLogger("pyvips").setLevel(logging.WARNING)
 except:
     HAVE_VIPS = False
 
@@ -242,40 +247,40 @@ class ThumbSrv(object):
             abspath, tpath = task
             ext = abspath.split(".")[-1].lower()
             png_ok = False
-            fun = None
+            funs = []
             if not bos.path.exists(tpath):
                 for lib in self.args.th_dec:
-                    if fun:
-                        break
-                    elif lib == "pil" and ext in self.fmt_pil:
-                        fun = self.conv_pil
+                    if lib == "pil" and ext in self.fmt_pil:
+                        funs.append(self.conv_pil)
                     elif lib == "vips" and ext in self.fmt_vips:
-                        fun = self.conv_vips
+                        funs.append(self.conv_vips)
                     elif lib == "ff" and ext in self.fmt_ffi or ext in self.fmt_ffv:
-                        fun = self.conv_ffmpeg
+                        funs.append(self.conv_ffmpeg)
                     elif lib == "ff" and ext in self.fmt_ffa:
                         if tpath.endswith(".opus") or tpath.endswith(".caf"):
-                            fun = self.conv_opus
+                            funs.append(self.conv_opus)
                         elif tpath.endswith(".png"):
-                            fun = self.conv_waves
+                            funs.append(self.conv_waves)
                             png_ok = True
                         else:
-                            fun = self.conv_spec
+                            funs.append(self.conv_spec)
 
             if not png_ok and tpath.endswith(".png"):
                 raise Pebkac(400, "png only allowed for waveforms")
 
-            if fun:
+            for fun in funs:
                 try:
                     fun(abspath, tpath)
+                    break
                 except Exception as ex:
                     msg = "{} could not create thumbnail of {}\n{}"
                     msg = msg.format(fun.__name__, abspath, min_ex())
                     c: Union[str, int] = 1 if "<Signals.SIG" in msg else "90"
                     self.log(msg, c)
                     if getattr(ex, "returncode", 0) != 321:
-                        with open(tpath, "wb") as _:
-                            pass
+                        if fun == funs[-1]:
+                            with open(tpath, "wb") as _:
+                                pass
                     else:
                         # ffmpeg may spawn empty files on windows
                         try:
@@ -363,7 +368,8 @@ class ThumbSrv(object):
                 img = pyvips.Image.thumbnail(abspath, w, **kw)
                 break
             except:
-                pass
+                if c == crops[-1]:
+                    raise
 
         img.write_to_file(tpath, Q=40)
 

copyparty/up2k.py

@@ -642,9 +642,15 @@ class Up2k(object):
         ff = "\033[0;35m{}{:.0}"
         fv = "\033[0;36m{}:\033[90m{}"
         fx = set(("html_head",))
+        fd = {"dbd": "dbd"}
+        fl = {
+            k: v
+            for k, v in flags.items()
+            if k not in fd or v != getattr(self.args, fd[k])
+        }
         a = [
             (ft if v is True else ff if v is False else fv).format(k, str(v))
-            for k, v in flags.items()
+            for k, v in fl.items()
             if k not in fx
         ]
         if a:
@@ -842,6 +848,7 @@ class Up2k(object):
         seen = seen + [rcdir]
         unreg: list[str] = []
         files: list[tuple[int, int, str]] = []
+        fat32 = True
 
         assert self.pp and self.mem_cur
         self.pp.msg = "a{} {}".format(self.pp.n, cdir)
@@ -866,6 +873,9 @@ class Up2k(object):
 
             lmod = int(inf.st_mtime)
             sz = inf.st_size
+            if fat32 and inf.st_mtime % 2:
+                fat32 = False
+
             if stat.S_ISDIR(inf.st_mode):
                 rap = absreal(abspath)
                 if dev and inf.st_dev != dev:
@@ -953,6 +963,9 @@ class Up2k(object):
                         self.log(t.format(top, rp, len(in_db), rep_db))
                         dts = -1
 
+                    if fat32 and abs(dts - lmod) == 1:
+                        dts = lmod
+
                     if dts == lmod and dsz == sz and (nohash or dw[0] != "#" or not sz):
                         continue
 

copyparty/util.py

@@ -437,9 +437,7 @@ class HLog(logging.Handler):
         else:
             c = 1
 
-        if record.name.startswith("PIL") and lv < logging.WARNING:
-            return
-        elif record.name == "pyftpdlib":
+        if record.name == "pyftpdlib":
             m = self.ptn_ftp.match(msg)
             if m:
                 ip = m.group(1)
@@ -2008,6 +2006,20 @@ def read_socket_chunked(
             raise Pebkac(400, t.format(x))
 
 
+def list_ips() -> list[str]:
+    from .stolen.ifaddr import get_adapters
+
+    ret: set[str] = set()
+    for nic in get_adapters():
+        for ipo in nic.ips:
+            if len(ipo.ip) < 7:
+                ret.add(ipo.ip[0])  # ipv6 is (ip,0,0)
+            else:
+                ret.add(ipo.ip)
+
+    return list(ret)
+
+
 def yieldfile(fn: str) -> Generator[bytes, None, None]:
     with open(fsenc(fn), "rb", 512 * 1024) as f:
         while True:

copyparty/web/baguettebox.js

@@ -27,7 +27,7 @@ window.baguetteBox = (function () {
         isOverlayVisible = false,
         touch = {},  // start-pos
         touchFlag = false,  // busy
-        re_i = /.+\.(gif|jpe?g|png|webp)(\?|$)/i,
+        re_i = /.+\.(a?png|avif|bmp|gif|heif|jpe?g|jfif|svg|webp)(\?|$)/i,
         re_v = /.+\.(webm|mkv|mp4)(\?|$)/i,
         anims = ['slideIn', 'fadeIn', 'none'],
         data = {},  // all galleries

copyparty/web/browser.js

@@ -4068,7 +4068,7 @@ var thegrid = (function () {
 		var oth = ebi(this.getAttribute('ref')),
 			href = noq_href(this),
 			aplay = ebi('a' + oth.getAttribute('id')),
-			is_img = /\.(gif|jpe?g|png|webp|webm|mkv|mp4)(\?|$)/i.test(href),
+			is_img = /\.(a?png|avif|bmp|gif|heif|jpe?g|jfif|svg|webp|webm|mkv|mp4)(\?|$)/i.test(href),
 			is_dir = href.endsWith('/'),
 			in_tree = is_dir && treectl.find(oth.textContent.slice(0, -1)),
 			have_sel = QS('#files tr.sel'),

docs/changelog.md

@@ -1,3 +1,34 @@
+▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
+# 2022-1213-1956  `v1.5.3`  folder-sync + turbo-rust
+
+* read-only demo server at https://a.ocv.me/pub/demo/
+
+## new features
+* one-way folder sync (client to server) using [up2k.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/README.md#up2kpy) `-z --dr`
+  * great rsync alternative when combined with `-e2ds --hardlink` deduplication on the server
+* **50x faster** when uploading small files to HDD, especially SMR
+  * by switching sqlite to WAL which carries a small chance of temporarily forgetting the ~200 most recent uploads if you have a power outage or your OS crashes; see `--help-dbd` if you have `-mtp` plugins which produces metadata you can't afford to lose
+* location-based [reverse-proxying](https://github.com/9001/copyparty/#reverse-proxy) (but it's still recommended to use a dedicated domain/subdomain instead)
+* IPv6 link-local automatically enabled for TCP and zeroconf on NICs without a routable IPv6
+* zeroconf network filters now accept subnets too, for example `--z-on 192.168.0.0/16`
+* `.hist` folders are hidden on windows
+* ux:
+  * more accurate total ETA on upload
+  * sorting of batch-unpost links was unintuitive / dangerous
+  * hotkey `Y` turns files into download links if nothing's selected
+  * option to replace or disable the mediaplayer-toggle mouse cursor with `--mpmc`
+
+## bugfixes
+* WAL probably/hopefully fixes #10 (we'll know in 6 months roughly)
+* repair db inconsistencies (which can happen if terminated during startup)
+* [davfs2](https://wiki.archlinux.org/title/Davfs2) did not approve of the authentication prompt
+* the `connect` button on the control-panel didn't work on phones
+* couldn't specify windows NICs in arguments `--z-on` / `--z-off` and friends
+* ssdp xml escaping for `--zsl` URL
+* no longer possible to accidentally launch multiple copyparty instances on the same port on windows
+
+
+
 ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
 # 2022-1203-2048  `v1.5.1`  babel
 

scripts/deps-docker/Dockerfile

@@ -3,9 +3,9 @@ FROM    alpine:3.16
 WORKDIR /z
 ENV     ver_asmcrypto=c72492f4a66e17a0e5dd8ad7874de354f3ccdaa5 \
         ver_hashwasm=4.9.0 \
-        ver_marked=4.2.4 \
+        ver_marked=4.2.5 \
         ver_mde=2.18.0 \
-        ver_codemirror=5.65.10 \
+        ver_codemirror=5.65.11 \
         ver_fontawesome=5.13.0 \
         ver_zopfli=1.0.3