Compare commits

..

24 Commits

Author SHA1 Message Date
ed
b0cc396bca v1.6.8 2023-03-12 16:10:07 +00:00
ed
ae463518f6 u2cli: send upload stats to server + fix py2.6 support 2023-03-11 21:39:56 +00:00
ed
2be2e9a0d8 index folder thumbs in db 2023-03-11 11:43:29 +00:00
ed
e405fddf74 specify that only up2k clients will resume uploads 2023-03-09 22:47:37 +00:00
ed
c269b0dd91 show an error (instead of crashing) if a pic is 404 2023-03-09 22:37:12 +00:00
ed
8c3211263a keep scanning folders for more music to play 2023-03-09 22:26:41 +00:00
ed
bf04e7c089 update some docs 2023-03-09 22:11:39 +00:00
ed
c7c6e48b1a didn't compress numbered logfiles 2023-03-09 21:59:59 +00:00
ed
974ca773be just to be extra sure 2023-03-09 21:49:29 +00:00
ed
9270c2df19 evict basic-browser from crawlers 2023-03-09 21:35:07 +00:00
ed
b39ff92f34 u2cli: support long paths on win7 2023-03-08 22:27:13 +00:00
ed
7454167f78 add DCO PR template 2023-03-08 08:27:17 +01:00
ed
5ceb3a962f build up2k.exe 2023-03-07 22:58:14 +00:00
ed
52bd5642da update archpkg to 1.6.7 2023-03-05 20:20:15 +00:00
ed
c39c93725f v1.6.7 2023-03-05 20:18:16 +00:00
ed
d00f0b9fa7 ftp: support filezilla mkdir 2023-03-05 20:18:02 +00:00
ed
01cfc70982 add example for webdav automount 2023-03-05 19:52:45 +00:00
ed
e6aec189bd fix flickering toast on upload finish 2023-03-05 19:49:54 +00:00
ed
c98fff1647 fix chunkpost-handshake race (affects --no-dedup only);
a handshake arriving in the middle of the final chunk could cause
dupes to become empty -- worst case leading to loss of data
2023-03-05 19:45:50 +00:00
ed
0009e31bd3 heavy webworker load can park the main thread of a
background chrome tab for 10sec; piggyback some pokes off postmessage
2023-03-02 22:35:32 +00:00
ed
db95e880b2 thats not how it works 2023-02-28 22:19:06 +00:00
ed
e69fea4a59 exe: update screenshots 2023-02-26 22:26:40 +00:00
ed
4360800a6e update archpkg to 1.6.6 2023-02-26 22:11:56 +00:00
ed
b179e2b031 prisonparty: ignore unresolvable mount paths;
allows startup even if some locations are missing,
for example if a server rebooted and some disks aren't up yet
2023-02-26 22:11:15 +00:00
32 changed files with 630 additions and 141 deletions

2
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,2 @@
Please include the following text somewhere in this PR description:
This PR complies with the DCO; https://developercertificate.org/

View File

@@ -1,4 +1,4 @@
# 🎉 copyparty
# 💾🎉 copyparty
* portable file sharing hub (py2/py3) [(on PyPI)](https://pypi.org/project/copyparty/)
* MIT-Licensed, 2019-05-26, ed @ irc.rizon.net
@@ -240,6 +240,9 @@ browser-specific:
server-os-specific:
* RHEL8 / Rocky8: you can run copyparty using `/usr/libexec/platform-python`
server notes:
* pypy is supported but regular cpython is faster if you enable the database
# bugs
@@ -513,11 +516,14 @@ up2k has several advantages:
* much higher speeds than ftp/scp/tarpipe on some internet connections (mainly american ones) thanks to parallel connections
* the last-modified timestamp of the file is preserved
> it is perfectly safe to restart / upgrade copyparty while someone is uploading to it!
> all known up2k clients will resume just fine 💪
see [up2k](#up2k) for details on how it works, or watch a [demo video](https://a.ocv.me/pub/demo/pics-vids/#gf-0f6f5c0d)
![copyparty-upload-fs8](https://user-images.githubusercontent.com/241032/129635371-48fc54ca-fa91-48e3-9b1d-ba413e4b68cb.png)
**protip:** you can avoid scaring away users with [contrib/plugins/minimal-up2k.html](contrib/plugins/minimal-up2k.html) which makes it look [much simpler](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
**protip:** you can avoid scaring away users with [contrib/plugins/minimal-up2k.js](contrib/plugins/minimal-up2k.js) which makes it look [much simpler](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
**protip:** if you enable `favicon` in the `[⚙️] settings` tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress -- also, the `[🔔]` and/or `[🔊]` switches enable visible and/or audible notifications on upload completion
@@ -1378,7 +1384,7 @@ you can reduce the sfx size by repacking it; see [./docs/devnotes.md#sfx-repack]
download [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) (win8+) or [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) (win7+)
![copyparty-exe-fs8](https://user-images.githubusercontent.com/241032/194707422-cb7f66c9-41a2-4cb9-8dbc-2ab866cd4338.png)
![copyparty-exe-fs8](https://user-images.githubusercontent.com/241032/221445946-1e328e56-8c5b-44a9-8b9f-dee84d942535.png)
can be convenient on machines where installing python is problematic, however is **not recommended** -- if possible, please use **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py)** instead
@@ -1386,9 +1392,9 @@ can be convenient on machines where installing python is problematic, however is
* on win8 it needs [vc redist 2015](https://www.microsoft.com/en-us/download/details.aspx?id=48145), on win10 it just works
* dangerous: [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is compatible with windows7, which means it uses an ancient copy of python (3.7.9) which cannot be upgraded and should never be exposed to the internet (LAN is fine)
* dangerous: [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is compatible with [windows7](https://user-images.githubusercontent.com/241032/221445944-ae85d1f4-d351-4837-b130-82cab57d6cca.png), which means it uses an ancient copy of python (3.7.9) which cannot be upgraded and should never be exposed to the internet (LAN is fine)
* dangerous and deprecated: [copyparty64.exe](https://github.com/9001/copyparty/releases/download/v1.6.5/copyparty64.exe) lets you [run copyparty in WinPE](https://user-images.githubusercontent.com/241032/205454984-e6b550df-3c49-486d-9267-1614078dd0dd.png) and is otherwise completely useless
* dangerous and deprecated: [copyparty-winpe64.exe](https://github.com/9001/copyparty/releases/download/v1.6.8/copyparty-winpe64.exe) lets you [run copyparty in WinPE](https://user-images.githubusercontent.com/241032/205454984-e6b550df-3c49-486d-9267-1614078dd0dd.png) and is otherwise completely useless
meanwhile [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) instead relies on your system python which gives better performance and will stay safe as long as you keep your python install up-to-date

View File

@@ -31,7 +31,7 @@ run [`install-deps.sh`](install-deps.sh) to build/install most dependencies requ
*alternatively* (or preferably) use packages from your distro instead, then you'll need at least these:
* from distro: `numpy vamp-plugin-sdk beatroot-vamp mixxx-keyfinder ffmpeg`
* from pypy: `keyfinder vamp`
* from pip: `keyfinder vamp`
# usage from copyparty

View File

@@ -4,8 +4,9 @@ set -e
# runs copyparty (or any other program really) in a chroot
#
# assumption: these directories, and everything within, are owned by root
sysdirs=( /bin /lib /lib32 /lib64 /sbin /usr /etc/alternatives )
sysdirs=(); for v in /bin /lib /lib32 /lib64 /sbin /usr /etc/alternatives ; do
[ -e $v ] && sysdirs+=($v)
done
# error-handler
help() { cat <<'EOF'
@@ -38,7 +39,7 @@ while true; do
v="$1"; shift
[ "$v" = -- ] && break # end of volumes
[ "$#" -eq 0 ] && break # invalid usage
vols+=( "$(realpath "$v")" )
vols+=( "$(realpath "$v" || echo "$v")" )
done
pybin="$1"; shift
pybin="$(command -v "$pybin")"
@@ -82,7 +83,7 @@ jail="${jail%/}"
printf '%s\n' "${sysdirs[@]}" "${vols[@]}" | sed -r 's`/$``' | LC_ALL=C sort | uniq |
while IFS= read -r v; do
[ -e "$v" ] || {
# printf '\033[1;31mfolder does not exist:\033[0m %s\n' "/$v"
printf '\033[1;31mfolder does not exist:\033[0m %s\n' "$v"
continue
}
i1=$(stat -c%D.%i "$v" 2>/dev/null || echo a)

View File

@@ -1,9 +1,12 @@
#!/usr/bin/env python3
from __future__ import print_function, unicode_literals
S_VERSION = "1.5"
S_BUILD_DT = "2023-03-12"
"""
up2k.py: upload to copyparty
2023-01-13, v1.2, ed <irc.rizon.net>, MIT-Licensed
2021, ed <irc.rizon.net>, MIT-Licensed
https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py
- dependencies: requests
@@ -24,6 +27,8 @@ import platform
import threading
import datetime
EXE = sys.executable.endswith("exe")
try:
import argparse
except:
@@ -34,7 +39,9 @@ except:
try:
import requests
except ImportError:
if sys.version_info > (2, 7):
if EXE:
raise
elif sys.version_info > (2, 7):
m = "\nERROR: need 'requests'; please run this command:\n {0} -m pip install --user requests\n"
else:
m = "requests/2.18.4 urllib3/1.23 chardet/3.0.4 certifi/2020.4.5.1 idna/2.7"
@@ -245,7 +252,13 @@ def eprint(*a, **ka):
def flushing_print(*a, **ka):
_print(*a, **ka)
try:
_print(*a, **ka)
except:
v = " ".join(str(x) for x in a)
v = v.encode("ascii", "replace").decode("ascii")
_print(v, **ka)
if "flush" not in ka:
sys.stdout.flush()
@@ -372,6 +385,23 @@ def walkdir(err, top, seen):
def walkdirs(err, tops):
"""recursive statdir for a list of tops, yields [top, relpath, stat]"""
sep = "{0}".format(os.sep).encode("ascii")
if not VT100:
za = []
for td in tops:
try:
ap = os.path.abspath(os.path.realpath(td))
if td[-1:] in (b"\\", b"/"):
ap += sep
except:
# maybe cpython #88013 (ok)
ap = td
za.append(ap)
za = [x if x.startswith(b"\\\\") else b"\\\\?\\" + x for x in za]
za = [x.replace(b"/", b"\\") for x in za]
tops = za
for top in tops:
isdir = os.path.isdir(top)
if top[-1:] == sep:
@@ -520,7 +550,11 @@ def handshake(ar, file, search):
except Exception as ex:
em = str(ex).split("SSLError(")[-1].split("\nURL: ")[0].strip()
if sc == 422 or "<pre>partial upload exists at a different" in txt:
if (
sc == 422
or "<pre>partial upload exists at a different" in txt
or "<pre>source file busy; please try again" in txt
):
file.recheck = True
return [], False
elif sc == 409 or "<pre>upload rejected, file already exists" in txt:
@@ -552,8 +586,8 @@ def handshake(ar, file, search):
return r["hash"], r["sprs"]
def upload(file, cid, pw):
# type: (File, str, str) -> None
def upload(file, cid, pw, stats):
# type: (File, str, str, str) -> None
"""upload one specific chunk, `cid` (a chunk-hash)"""
headers = {
@@ -561,6 +595,10 @@ def upload(file, cid, pw):
"X-Up2k-Wark": file.wark,
"Content-Type": "application/octet-stream",
}
if stats:
headers["X-Up2k-Stat"] = stats
if pw:
headers["Cookie"] = "=".join(["cppwd", pw])
@@ -629,6 +667,8 @@ class Ctl(object):
req_ses.verify = ar.te
self.filegen = walkdirs([], ar.files)
self.recheck = [] # type: list[File]
if ar.safe:
self._safe()
else:
@@ -647,11 +687,11 @@ class Ctl(object):
self.t0 = time.time()
self.t0_up = None
self.spd = None
self.eta = "99:99:99"
self.mutex = threading.Lock()
self.q_handshake = Queue() # type: Queue[File]
self.q_upload = Queue() # type: Queue[tuple[File, str]]
self.recheck = [] # type: list[File]
self.st_hash = [None, "(idle, starting...)"] # type: tuple[File, int]
self.st_up = [None, "(idle, starting...)"] # type: tuple[File, int]
@@ -693,7 +733,8 @@ class Ctl(object):
ncs = len(hs)
for nc, cid in enumerate(hs):
print(" {0} up {1}".format(ncs - nc, cid))
upload(file, cid, self.ar.a)
stats = "{0}/0/0/{1}".format(nf, self.nfiles - nf)
upload(file, cid, self.ar.a, stats)
print(" ok!")
if file.recheck:
@@ -768,12 +809,12 @@ class Ctl(object):
eta = (self.nbytes - self.up_b) / (spd + 1)
spd = humansize(spd)
eta = str(datetime.timedelta(seconds=int(eta)))
self.eta = str(datetime.timedelta(seconds=int(eta)))
sleft = humansize(self.nbytes - self.up_b)
nleft = self.nfiles - self.up_f
tail = "\033[K\033[u" if VT100 and not self.ar.ns else "\r"
t = "{0} eta @ {1}/s, {2}, {3}# left".format(eta, spd, sleft, nleft)
t = "{0} eta @ {1}/s, {2}, {3}# left".format(self.eta, spd, sleft, nleft)
eprint(txt + "\033]0;{0}\033\\\r{0}{1}".format(t, tail))
if not self.recheck:
@@ -811,7 +852,7 @@ class Ctl(object):
zb += quotep(rd.replace(b"\\", b"/"))
r = req_ses.get(zb + b"?ls&dots", headers=headers)
if not r:
raise Exception("HTTP {}".format(r.status_code))
raise Exception("HTTP {0}".format(r.status_code))
j = r.json()
for f in j["dirs"] + j["files"]:
@@ -886,6 +927,9 @@ class Ctl(object):
self.handshaker_busy += 1
upath = file.abs.decode("utf-8", "replace")
if not VT100:
upath = upath[4:]
hs, sprs = handshake(self.ar, file, search)
if search:
if hs:
@@ -951,9 +995,20 @@ class Ctl(object):
self.uploader_busy += 1
self.t0_up = self.t0_up or time.time()
zs = "{0}/{1}/{2}/{3} {4}/{5} {6}"
stats = zs.format(
self.up_f,
len(self.recheck),
self.uploader_busy,
self.nfiles - self.up_f,
int(self.nbytes / (1024 * 1024)),
int((self.nbytes - self.up_b) / (1024 * 1024)),
self.eta,
)
file, cid = task
try:
upload(file, cid, self.ar.a)
upload(file, cid, self.ar.a, stats)
except:
eprint("upload failed, retrying: {0} #{1}\n".format(file.name, cid[:8]))
# handshake will fix it
@@ -989,8 +1044,13 @@ def main():
cores = (os.cpu_count() if hasattr(os, "cpu_count") else 0) or 2
hcores = min(cores, 3) # 4% faster than 4+ on py3.9 @ r5-4500U
ver = "{0}, v{1}".format(S_BUILD_DT, S_VERSION)
if "--version" in sys.argv:
print(ver)
return
# fmt: off
ap = app = argparse.ArgumentParser(formatter_class=APF, epilog="""
ap = app = argparse.ArgumentParser(formatter_class=APF, description="copyparty up2k uploader / filesearch tool, " + ver, epilog="""
NOTE:
source file/folder selection uses rsync syntax, meaning that:
"foo" uploads the entire folder to URL/foo/
@@ -1003,6 +1063,7 @@ source file/folder selection uses rsync syntax, meaning that:
ap.add_argument("-a", metavar="PASSWORD", help="password or $filepath")
ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
ap.add_argument("--version", action="store_true", help="show version and exit")
ap = app.add_argument_group("compatibility")
ap.add_argument("--cls", action="store_true", help="clear screen before start")
@@ -1026,7 +1087,16 @@ source file/folder selection uses rsync syntax, meaning that:
ap.add_argument("-td", action="store_true", help="disable certificate check")
# fmt: on
ar = app.parse_args()
try:
ar = app.parse_args()
finally:
if EXE and not sys.argv[1:]:
print("*** hit enter to exit ***")
try:
input()
except:
pass
if ar.drd:
ar.dr = True
@@ -1040,7 +1110,7 @@ source file/folder selection uses rsync syntax, meaning that:
ar.files = [
os.path.abspath(os.path.realpath(x.encode("utf-8")))
+ (x[-1:] if x[-1:] == os.sep else "").encode("utf-8")
+ (x[-1:] if x[-1:] in ("\\", "/") else "").encode("utf-8")
for x in ar.files
]
@@ -1050,7 +1120,7 @@ source file/folder selection uses rsync syntax, meaning that:
if ar.a and ar.a.startswith("$"):
fn = ar.a[1:]
print("reading password from file [{}]".format(fn))
print("reading password from file [{0}]".format(fn))
with open(fn, "rb") as f:
ar.a = f.read().decode("utf-8").strip()

View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8">
<title>🎉 redirect</title>
<title>💾🎉 redirect</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<style>

View File

@@ -1,6 +1,6 @@
# Maintainer: icxes <dev.null@need.moe>
pkgname=copyparty
pkgver="1.6.5"
pkgver="1.6.7"
pkgrel=1
pkgdesc="Portable file sharing hub"
arch=("any")
@@ -26,12 +26,12 @@ source=("${url}/releases/download/v${pkgver}/${pkgname}-sfx.py"
"https://raw.githubusercontent.com/9001/${pkgname}/v${pkgver}/LICENSE"
)
backup=("etc/${pkgname}.d/init" )
sha256sums=("947d3f191f96f6a9e451bbcb35c5582ba210d81cfdc92dfa9ab0390dbecf26ee"
sha256sums=("3fb40a631e9decf0073db06aab6fd8d743de91f4ddb82a65164d39d53e0b413f"
"b8565eba5e64dedba1cf6c7aac7e31c5a731ed7153d6810288a28f00a36c28b2"
"f65c207e0670f9d78ad2e399bda18d5502ff30d2ac79e0e7fc48e7fbdc39afdc"
"c4f396b083c9ec02ad50b52412c84d2a82be7f079b2d016e1c9fad22d68285ff"
"dba701de9fd584405917e923ea1e59dbb249b96ef23bad479cf4e42740b774c8"
"746971e95817c54445ce7f9c8406822dffc814cd5eb8113abd36dd472fd677d7"
"23054bb206153a1ed34038accaf490b8068f9c856e423c2f2595b148b40c0a0c"
"cb2ce3d6277bf2f5a82ecf336cc44963bc6490bcf496ffbd75fc9e21abaa75f3"
)

View File

@@ -874,7 +874,7 @@ def add_thumbnail(ap):
ap2.add_argument("--th-poke", metavar="SEC", type=int, default=300, help="activity labeling cooldown -- avoids doing keepalive pokes (updating the mtime) on thumbnail folders more often than SEC seconds")
ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval; 0=disabled")
ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age -- folders which haven't been poked for longer than --th-poke seconds will get deleted every --th-clean seconds")
ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat/look for")
ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat/look for; case-insensitive if -e2d")
# https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
# https://github.com/libvips/libvips
# ffmpeg -hide_banner -demuxers | awk '/^ D /{print$2}' | while IFS= read -r x; do ffmpeg -hide_banner -h demuxer=$x; done | grep -E '^Demuxer |extensions:'

View File

@@ -1,8 +1,8 @@
# coding: utf-8
VERSION = (1, 6, 6)
VERSION = (1, 6, 8)
CODENAME = "cors k"
BUILD_DT = (2023, 2, 26)
BUILD_DT = (2023, 3, 12)
S_VERSION = ".".join(map(str, VERSION))
S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

View File

@@ -218,7 +218,7 @@ class FtpFs(AbstractedFS):
def mkdir(self, path: str) -> None:
ap = self.rv2a(path, w=True)[0]
bos.mkdir(ap)
bos.makedirs(ap) # filezilla expects this
def listdir(self, path: str) -> list[str]:
vpath = join(self.cwd, path).lstrip("/")

View File

@@ -1714,7 +1714,7 @@ class HttpCli(object):
except:
raise Pebkac(500, min_ex())
x = self.conn.hsrv.broker.ask("up2k.handle_json", body)
x = self.conn.hsrv.broker.ask("up2k.handle_json", body, self.u2fh.aps)
ret = x.get()
if self.is_vproxied:
if "purl" in ret:
@@ -1884,17 +1884,10 @@ class HttpCli(object):
with self.mutex:
self.u2fh.close(path)
# windows cant rename open files
if ANYWIN and path != fin_path and not self.args.nw:
self.conn.hsrv.broker.ask("up2k.finish_upload", ptop, wark).get()
if not ANYWIN and not num_left:
times = (int(time.time()), int(lastmod))
self.log("no more chunks, setting times {}".format(times))
try:
bos.utime(fin_path, times)
except:
self.log("failed to utime ({}, {})".format(fin_path, times))
if not num_left and not self.args.nw:
self.conn.hsrv.broker.ask(
"up2k.finish_upload", ptop, wark, self.u2fh.aps
).get()
cinf = self.headers.get("x-up2k-stat", "")
@@ -3255,23 +3248,47 @@ class HttpCli(object):
):
raise Pebkac(403)
e2d = "e2d" in vn.flags
e2t = "e2t" in vn.flags
self.html_head = vn.flags.get("html_head", "")
if vn.flags.get("norobots"):
if vn.flags.get("norobots") or "b" in self.uparam:
self.out_headers["X-Robots-Tag"] = "noindex, nofollow"
else:
self.out_headers.pop("X-Robots-Tag", None)
is_dir = stat.S_ISDIR(st.st_mode)
icur = None
if e2t or (e2d and is_dir):
idx = self.conn.get_u2idx()
icur = idx.get_cur(dbv.realpath)
if self.can_read:
th_fmt = self.uparam.get("th")
if th_fmt is not None:
if is_dir:
for fn in self.args.th_covers.split(","):
fp = os.path.join(abspath, fn)
if bos.path.exists(fp):
vrem = "{}/{}".format(vrem.rstrip("/"), fn).strip("/")
is_dir = False
break
vrem = vrem.rstrip("/")
if icur and vrem:
q = "select fn from cv where rd=? and dn=?"
crd, cdn = vrem.rsplit("/", 1) if "/" in vrem else ("", vrem)
# no mojibake support:
try:
cfn = icur.execute(q, (crd, cdn)).fetchone()
if cfn:
fn = cfn[0]
fp = os.path.join(abspath, fn)
if bos.path.exists(fp):
vrem = "{}/{}".format(vrem, fn).strip("/")
is_dir = False
except:
pass
else:
for fn in self.args.th_covers:
fp = os.path.join(abspath, fn)
if bos.path.exists(fp):
vrem = "{}/{}".format(vrem, fn).strip("/")
is_dir = False
break
if is_dir:
return self.tx_ico("a.folder")
@@ -3378,8 +3395,8 @@ class HttpCli(object):
"taglist": [],
"srvinf": srv_infot,
"acct": self.uname,
"idx": ("e2d" in vn.flags),
"itag": ("e2t" in vn.flags),
"idx": e2d,
"itag": e2t,
"lifetime": vn.flags.get("lifetime") or 0,
"frand": bool(vn.flags.get("rand")),
"perms": perms,
@@ -3398,8 +3415,8 @@ class HttpCli(object):
"taglist": [],
"def_hcols": [],
"have_emp": self.args.emp,
"have_up2k_idx": ("e2d" in vn.flags),
"have_tags_idx": ("e2t" in vn.flags),
"have_up2k_idx": e2d,
"have_tags_idx": e2t,
"have_acode": (not self.args.no_acode),
"have_mv": (not self.args.no_mv),
"have_del": (not self.args.no_del),
@@ -3411,7 +3428,7 @@ class HttpCli(object):
"url_suf": url_suf,
"logues": logues,
"readme": readme,
"title": html_escape(self.vpath, crlf=True) or "🎉",
"title": html_escape(self.vpath, crlf=True) or "💾🎉",
"srv_info": srv_infot,
"dtheme": self.args.theme,
"themes": self.args.themes,
@@ -3475,11 +3492,6 @@ class HttpCli(object):
if not self.args.ed or "dots" not in self.uparam:
ls_names = exclude_dotfiles(ls_names)
icur = None
if "e2t" in vn.flags:
idx = self.conn.get_u2idx()
icur = idx.get_cur(dbv.realpath)
add_fk = vn.flags.get("fk")
dirs = []

View File

@@ -149,12 +149,9 @@ class SvcHub(object):
self.log("root", t.format(args.j))
if not args.no_fpool and args.j != 1:
t = "WARNING: --use-fpool combined with multithreading is untested and can probably cause undefined behavior"
if ANYWIN:
t = 'windows cannot do multithreading without --no-fpool, so enabling that -- note that upload performance will suffer if you have microsoft defender "real-time protection" enabled, so you probably want to use -j 1 instead'
args.no_fpool = True
self.log("root", t, c=3)
t = "WARNING: ignoring --use-fpool because multithreading (-j{}) is enabled"
self.log("root", t.format(args.j), c=3)
args.no_fpool = True
bri = "zy"[args.theme % 2 :][:1]
ch = "abcdefghijklmnopqrstuvwx"[int(args.theme / 2)]
@@ -351,6 +348,8 @@ class SvcHub(object):
if al.rsp_jtr:
al.rsp_slp = 0.000001
al.th_covers = set(al.th_covers.split(","))
return True
def _setlimits(self) -> None:
@@ -405,6 +404,7 @@ class SvcHub(object):
def _setup_logfile(self, printed: str) -> None:
base_fn = fn = sel_fn = self._logname()
do_xz = fn.lower().endswith(".xz")
if fn != self.args.lo:
ctr = 0
# yup this is a race; if started sufficiently concurrently, two
@@ -416,7 +416,7 @@ class SvcHub(object):
fn = sel_fn
try:
if fn.lower().endswith(".xz"):
if do_xz:
import lzma
lh = lzma.open(fn, "wt", encoding="utf-8", errors="replace", preset=0)

View File

@@ -73,6 +73,9 @@ if True: # pylint: disable=using-constant-test
if TYPE_CHECKING:
from .svchub import SvcHub
zs = "avif,avifs,bmp,gif,heic,heics,heif,heifs,ico,j2p,j2k,jp2,jpeg,jpg,jpx,png,tga,tif,tiff,webp"
CV_EXTS = set(zs.split(","))
class Dbw(object):
def __init__(self, c: "sqlite3.Cursor", n: int, t: float) -> None:
@@ -124,6 +127,7 @@ class Up2k(object):
self.droppable: dict[str, list[str]] = {}
self.volstate: dict[str, str] = {}
self.vol_act: dict[str, float] = {}
self.busy_aps: set[str] = set()
self.dupesched: dict[str, list[tuple[str, str, float]]] = {}
self.snap_persist_interval = 300 # persist unfinished index every 5 min
self.snap_discard_interval = 21600 # drop unfinished after 6 hours inactivity
@@ -161,12 +165,6 @@ class Up2k(object):
t = "could not initialize sqlite3, will use in-memory registry only"
self.log(t, 3)
if ANYWIN:
# usually fails to set lastmod too quickly
self.lastmod_q: list[tuple[str, int, tuple[int, int], bool]] = []
self.lastmod_q2 = self.lastmod_q[:]
Daemon(self._lastmodder, "up2k-lastmod")
self.fstab = Fstab(self.log_func)
self.gen_fk = self._gen_fk if self.args.log_fk else gen_filekey
@@ -463,11 +461,9 @@ class Up2k(object):
q = "select * from up where substr(w,1,16)=? and +rd=? and +fn=?"
ups = []
for wrf in wrfs:
try:
# almost definitely exists; don't care if it doesn't
ups.append(cur.execute(q, wrf).fetchone())
except:
pass
up = cur.execute(q, wrf).fetchone()
if up:
ups.append(up)
# t1 = time.time()
# self.log("mapped {} warks in {:.3f} sec".format(len(wrfs), t1 - t0))
@@ -952,6 +948,7 @@ class Up2k(object):
unreg: list[str] = []
files: list[tuple[int, int, str]] = []
fat32 = True
cv = ""
assert self.pp and self.mem_cur
self.pp.msg = "a{} {}".format(self.pp.n, cdir)
@@ -1014,6 +1011,12 @@ class Up2k(object):
continue
files.append((sz, lmod, iname))
liname = iname.lower()
if sz and (
iname in self.args.th_covers
or (not cv and liname.rsplit(".", 1)[-1] in CV_EXTS)
):
cv = iname
# folder of 1000 files = ~1 MiB RAM best-case (tiny filenames);
# free up stuff we're done with before dhashing
@@ -1026,6 +1029,7 @@ class Up2k(object):
zh = hashlib.sha1()
_ = [zh.update(str(x).encode("utf-8", "replace")) for x in files]
zh.update(cv.encode("utf-8", "replace"))
zh.update(spack(b"<d", cst.st_mtime))
dhash = base64.urlsafe_b64encode(zh.digest()[:12]).decode("ascii")
sql = "select d from dh where d = ? and h = ?"
@@ -1039,6 +1043,18 @@ class Up2k(object):
if c.fetchone():
return ret
if cv and rd:
# mojibake not supported (for performance / simplicity):
try:
q = "select * from cv where rd=? and dn=? and +fn=?"
crd, cdn = rd.rsplit("/", 1) if "/" in rd else ("", rd)
if not db.c.execute(q, (crd, cdn, cv)).fetchone():
db.c.execute("delete from cv where rd=? and dn=?", (crd, cdn))
db.c.execute("insert into cv values (?,?,?)", (crd, cdn, cv))
db.n += 1
except Exception as ex:
self.log("cover {}/{} failed: {}".format(rd, cv, ex), 6)
seen_files = set([x[2] for x in files]) # for dropcheck
for sz, lmod, fn in files:
if self.stop:
@@ -1235,6 +1251,19 @@ class Up2k(object):
if n_rm2:
self.log("forgetting {} shadowed deleted files".format(n_rm2))
# then covers
n_rm3 = 0
q = "delete from cv where rd=? and dn=? and +fn=?"
for crd, cdn, fn in cur.execute("select * from cv"):
ap = os.path.join(top, crd, cdn, fn)
if not bos.path.exists(ap):
c2.execute(q, (crd, cdn, fn))
n_rm3 += 1
if n_rm3:
self.log("forgetting {} deleted covers".format(n_rm3))
c2.connection.commit()
c2.close()
return n_rm + n_rm2
@@ -1387,6 +1416,7 @@ class Up2k(object):
cur, _ = reg
self._set_tagscan(cur, True)
cur.execute("delete from dh")
cur.execute("delete from cv")
cur.connection.commit()
def _set_tagscan(self, cur: "sqlite3.Cursor", need: bool) -> bool:
@@ -1967,6 +1997,7 @@ class Up2k(object):
if ver == DB_VER:
try:
self._add_cv_tab(cur)
self._add_xiu_tab(cur)
self._add_dhash_tab(cur)
except:
@@ -2062,6 +2093,7 @@ class Up2k(object):
self._add_dhash_tab(cur)
self._add_xiu_tab(cur)
self._add_cv_tab(cur)
self.log("created DB at {}".format(db_path))
return cur
@@ -2110,12 +2142,34 @@ class Up2k(object):
cur.connection.commit()
def _add_cv_tab(self, cur: "sqlite3.Cursor") -> None:
# v5b -> v5c
try:
cur.execute("select rd, dn, fn from cv limit 1").fetchone()
return
except:
pass
for cmd in [
r"create table cv (rd text, dn text, fn text)",
r"create index cv_i on cv(rd, dn)",
]:
cur.execute(cmd)
try:
cur.execute("delete from dh")
except:
pass
cur.connection.commit()
def _job_volchk(self, cj: dict[str, Any]) -> None:
if not self.register_vpath(cj["ptop"], cj["vcfg"]):
if cj["ptop"] not in self.registry:
raise Pebkac(410, "location unavailable")
def handle_json(self, cj: dict[str, Any]) -> dict[str, Any]:
def handle_json(self, cj: dict[str, Any], busy_aps: set[str]) -> dict[str, Any]:
self.busy_aps = busy_aps
try:
# bit expensive; 3.9=10x 3.11=2x
if self.mutex.acquire(timeout=10):
@@ -2289,6 +2343,22 @@ class Up2k(object):
else:
# symlink to the client-provided name,
# returning the previous upload info
psrc = src + ".PARTIAL"
if self.args.dotpart:
m = re.match(r"(.*[\\/])(.*)", psrc)
if m: # always true but...
zs1, zs2 = m.groups()
psrc = zs1 + "." + zs2
if (
src in self.busy_aps
or psrc in self.busy_aps
or (wark in reg and "done" not in reg[wark])
):
raise Pebkac(
422, "source file busy; please try again later"
)
job = deepcopy(job)
job["wark"] = wark
job["at"] = cj.get("at") or time.time()
@@ -2332,7 +2402,7 @@ class Up2k(object):
if not n4g:
raise
if cur:
if cur and not self.args.nw:
zs = "prel name lmod size ptop vtop wark host user addr at"
a = [job[x] for x in zs.split()]
self.db_add(cur, vfs.flags, *a)
@@ -2507,10 +2577,7 @@ class Up2k(object):
if lmod and (not linked or SYMTIME):
times = (int(time.time()), int(lmod))
if ANYWIN:
self.lastmod_q.append((dst, 0, times, False))
else:
bos.utime(dst, times, False)
bos.utime(dst, times, False)
def handle_chunk(
self, ptop: str, wark: str, chash: str
@@ -2591,13 +2658,10 @@ class Up2k(object):
self.regdrop(ptop, wark)
return ret, dst
# windows cant rename open files
if not ANYWIN or src == dst:
self._finish_upload(ptop, wark)
return ret, dst
def finish_upload(self, ptop: str, wark: str) -> None:
def finish_upload(self, ptop: str, wark: str, busy_aps: set[str]) -> None:
self.busy_aps = busy_aps
with self.mutex:
self._finish_upload(ptop, wark)
@@ -2610,6 +2674,10 @@ class Up2k(object):
except Exception as ex:
raise Pebkac(500, "finish_upload, wark, " + repr(ex))
if job["need"]:
t = "finish_upload {} with remaining chunks {}"
raise Pebkac(500, t.format(wark, job["need"]))
# self.log("--- " + wark + " " + dst + " finish_upload atomic " + dst, 4)
atomic_move(src, dst)
@@ -2617,14 +2685,15 @@ class Up2k(object):
vflags = self.flags[ptop]
times = (int(time.time()), int(job["lmod"]))
if ANYWIN:
z1 = (dst, job["size"], times, job["sprs"])
self.lastmod_q.append(z1)
elif not job["hash"]:
try:
bos.utime(dst, times)
except:
pass
self.log(
"no more chunks, setting times {} ({}) on {}".format(
times, bos.path.getsize(dst), dst
)
)
try:
bos.utime(dst, times)
except:
self.log("failed to utime ({}, {})".format(dst, times))
zs = "prel name lmod size ptop vtop wark host user addr"
z2 = [job[x] for x in zs.split()]
@@ -2645,6 +2714,7 @@ class Up2k(object):
if self.idx_wark(vflags, *z2):
del self.registry[ptop][wark]
else:
self.registry[ptop][wark]["done"] = 1
self.regdrop(ptop, wark)
if wake_sr:
@@ -2814,6 +2884,16 @@ class Up2k(object):
with self.rescan_cond:
self.rescan_cond.notify_all()
if rd and sz and fn.lower() in self.args.th_covers:
# wasteful; db_add will re-index actual covers
# but that won't catch existing files
crd, cdn = rd.rsplit("/", 1) if "/" in rd else ("", rd)
try:
db.execute("delete from cv where rd=? and dn=?", (crd, cdn))
db.execute("insert into cv values (?,?,?)", (crd, cdn, fn))
except:
pass
def handle_rm(self, uname: str, ip: str, vpaths: list[str], lim: list[int]) -> str:
n_files = 0
ok = {}
@@ -3428,27 +3508,6 @@ class Up2k(object):
if not job["hash"]:
self._finish_upload(job["ptop"], job["wark"])
def _lastmodder(self) -> None:
while True:
ready = self.lastmod_q2
self.lastmod_q2 = self.lastmod_q
self.lastmod_q = []
time.sleep(1)
for path, sz, times, sparse in ready:
self.log("lmod: setting times {} on {}".format(times, path))
try:
bos.utime(path, times, False)
except:
t = "lmod: failed to utime ({}, {}):\n{}"
self.log(t.format(path, times, min_ex()))
if sparse and self.args.sparse and self.args.sparse * 1024 * 1024 <= sz:
try:
sp.check_call(["fsutil", "sparse", "setflag", path, "0"])
except:
self.log("could not unsparse [{}]".format(path), 3)
def _snapshot(self) -> None:
slp = self.snap_persist_interval
while True:

View File

@@ -668,6 +668,7 @@ class FHC(object):
def __init__(self) -> None:
self.cache: dict[str, FHC.CE] = {}
self.aps: set[str] = set()
def close(self, path: str) -> None:
try:
@@ -679,6 +680,7 @@ class FHC(object):
fh.close()
del self.cache[path]
self.aps.remove(path)
def clean(self) -> None:
if not self.cache:
@@ -699,6 +701,7 @@ class FHC(object):
return self.cache[path].fhs.pop()
def put(self, path: str, fh: typing.BinaryIO) -> None:
self.aps.add(path)
try:
ce = self.cache[path]
ce.fhs.append(fh)

View File

@@ -93,6 +93,7 @@
--g-fsel-bg: #d39;
--g-fsel-b1: #f4a;
--g-fsel-ts: #804;
--g-dfg: var(--srv-3);
--g-fg: var(--a-hil);
--g-bg: var(--bg-u2);
--g-b1: var(--bg-u4);
@@ -327,6 +328,7 @@ html.c {
}
html.cz {
--bgg: var(--bg-u2);
--srv-3: #fff;
}
html.cy {
--fg: #fff;
@@ -354,6 +356,7 @@ html.cy {
--chk-fg: #fd0;
--srv-1: #f00;
--srv-3: #fff;
--op-aa-bg: #fff;
--u2-b1-bg: #f00;
@@ -964,6 +967,9 @@ html.y #path a:hover {
#ggrid>a.dir:before {
content: '📂';
}
#ggrid>a.dir>span {
color: var(--g-dfg);
}
#ggrid>a.au:before {
content: '💾';
}
@@ -1010,6 +1016,9 @@ html.np_open #ggrid>a.au:before {
background: var(--g-sel-bg);
border-color: var(--g-sel-b1);
}
#ggrid>a.sel>span {
color: var(--g-sel-fg);
}
#ggrid>a.sel,
#ggrid>a[tt].sel {
border-top: 1px solid var(--g-fsel-b1);

View File

@@ -259,6 +259,10 @@ var Ls = {
"mm_e404": "Could not play audio; error 404: File not found.",
"mm_e403": "Could not play audio; error 403: Access denied.\n\nTry pressing F5 to reload, maybe you got logged out",
"mm_e5xx": "Could not play audio; server error ",
"mm_nof": "not finding any more audio files nearby",
"mm_hnf": "that song no longer exists",
"im_hnf": "that image no longer exists",
"f_chide": 'this will hide the column «{0}»\n\nyou can unhide columns in the settings tab',
"f_bigtxt": "this file is {0} MiB large -- really view as text?",
@@ -714,6 +718,10 @@ var Ls = {
"mm_e404": "Avspilling feilet: Fil ikke funnet.",
"mm_e403": "Avspilling feilet: Tilgang nektet.\n\nKanskje du ble logget ut?\nPrøv å trykk F5 for å laste siden på nytt.",
"mm_e5xx": "Avspilling feilet: ",
"mm_nof": "finner ikke flere sanger i nærheten",
"mm_hnf": "sangen finnes ikke lenger",
"im_hnf": "bildet finnes ikke lenger",
"f_chide": 'dette vil skjule kolonnen «{0}»\n\nfanen for "andre innstillinger" lar deg vise kolonnen igjen',
"f_bigtxt": "denne filen er hele {0} MiB -- vis som tekst?",
@@ -1312,6 +1320,7 @@ var mpl = (function () {
var r = {
"pb_mode": (sread('pb_mode') || 'next').split('-')[0],
"os_ctl": bcfg_get('au_os_ctl', have_mctl) && have_mctl,
'traversals': 0,
};
bcfg_bind(r, 'preload', 'au_preload', true);
bcfg_bind(r, 'fullpre', 'au_fullpre', false);
@@ -2093,7 +2102,15 @@ function song_skip(n) {
}
function next_song(e) {
ev(e);
return song_skip(1);
if (mp.order.length) {
mpl.traversals = 0;
return song_skip(1);
}
if (mpl.traversals++ < 5) {
treectl.ls_cb = next_song;
return tree_neigh(1);
}
toast.inf(10, L.mm_nof);
}
function prev_song(e) {
ev(e);
@@ -2582,7 +2599,7 @@ function play(tid, is_ev, seek) {
if ((tn + '').indexOf('f-') === 0) {
tn = mp.order.indexOf(tn);
if (tn < 0)
return;
return toast.warn(10, L.mm_hnf);
}
if (tn >= mp.order.length) {
@@ -2850,6 +2867,9 @@ function eval_hash() {
clearInterval(t);
baguetteBox.urltime(ts);
var im = QS('#ggrid a[ref="' + id + '"]');
if (!im)
return toast.warn(10, L.im_hnf);
im.click();
im.scrollIntoView();
}, 50);

View File

@@ -64,6 +64,11 @@
yum install davfs2
{% if accs %}printf '%s\n' <b>{{ pw }}</b> k | {% endif %}mount -t davfs -ouid=1000 http{{ s }}://{{ ep }}/{{ rvp }} <b>mp</b>
</pre>
<p>make it automount on boot:</p>
<pre>
printf '%s\n' "http{{ s }}://{{ ep }}/{{ rvp }} <b>{{ pw }}</b> k" >> /etc/davfs2/secrets
printf '%s\n' "http{{ s }}://{{ ep }}/{{ rvp }} <b>mp</b> davfs rw,user,uid=1000,noauto 0 0" >> /etc/fstab
</pre>
<p>or you can use rclone instead, which is much slower but doesn't require root:</p>
<pre>
rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=other{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}

View File

@@ -114,10 +114,10 @@ function up2k_flagbus() {
do_take(now);
return;
}
if (flag.owner && now - flag.owner[1] > 5000) {
if (flag.owner && now - flag.owner[1] > 12000) {
flag.owner = null;
}
if (flag.wants && now - flag.wants[1] > 5000) {
if (flag.wants && now - flag.wants[1] > 12000) {
flag.wants = null;
}
if (!flag.owner && !flag.wants) {
@@ -772,6 +772,7 @@ function fsearch_explain(n) {
function up2k_init(subtle) {
var r = {
"tact": Date.now(),
"init_deps": init_deps,
"set_fsearch": set_fsearch,
"gotallfiles": [gotallfiles] // hooks
@@ -1647,8 +1648,14 @@ function up2k_init(subtle) {
running = true;
while (true) {
var now = Date.now(),
blocktime = now - r.tact,
is_busy = st.car < st.files.length;
if (blocktime > 2500)
console.log('main thread blocked for ' + blocktime);
r.tact = now;
if (was_busy && !is_busy) {
for (var a = 0; a < st.files.length; a++) {
var t = st.files[a];
@@ -1788,6 +1795,15 @@ function up2k_init(subtle) {
})();
function uptoast() {
if (st.busy.handshake.length)
return;
for (var a = 0; a < st.files.length; a++) {
var t = st.files[a];
if (t.want_recheck && !t.rechecks)
return;
}
var sr = uc.fsearch,
ok = pvis.ctr.ok,
ng = pvis.ctr.ng,
@@ -2043,6 +2059,8 @@ function up2k_init(subtle) {
nbusy++;
reading++;
nchunk++;
if (Date.now() - up2k.tact > 1500)
tasker();
}
function onmsg(d) {
@@ -2373,16 +2391,17 @@ function up2k_init(subtle) {
}
var err_pend = rsp.indexOf('partial upload exists at a different') + 1,
err_srcb = rsp.indexOf('source file busy; please try again') + 1,
err_plug = rsp.indexOf('upload blocked by x') + 1,
err_dupe = rsp.indexOf('upload rejected, file already exists') + 1;
if (err_pend || err_plug || err_dupe) {
if (err_pend || err_srcb || err_plug || err_dupe) {
err = rsp;
ofs = err.indexOf('\n/');
if (ofs !== -1) {
err = err.slice(0, ofs + 1) + linksplit(err.slice(ofs + 2).trimEnd()).join(' ');
}
if (!t.rechecks && err_pend) {
if (!t.rechecks && (err_pend || err_srcb)) {
t.rechecks = 0;
t.want_recheck = true;
}

View File

@@ -112,12 +112,13 @@ if ((document.location + '').indexOf(',rej,') + 1)
try {
console.hist = [];
var CMAXHIST = 100;
var hook = function (t) {
var orig = console[t].bind(console),
cfun = function () {
console.hist.push(Date.now() + ' ' + t + ': ' + Array.from(arguments).join(', '));
if (console.hist.length > 100)
console.hist = console.hist.slice(50);
if (console.hist.length > CMAXHIST)
console.hist = console.hist.slice(CMAXHIST / 2);
orig.apply(console, arguments);
};

View File

@@ -1,3 +1,80 @@
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
# 2023-0305-2018 `v1.6.7` fix no-dedup + add up2k.exe
## new features
* controlpanel-connect: add example for webdav automount
## bugfixes
* fix a race which, in worst case (but unlikely on linux), **could cause data loss**
* could only happen if `--no-dedup` or volflag `copydupes` was set (**not** default)
* if two identical files were uploaded at the same time, there was a small chance that one of the files would become empty
* check if you were affected by doing a search for zero-byte files using either of the following:
* https://127.0.0.1:3923/#q=size%20%3D%200
* `find -type f -size 0`
* let me know if you lost something important and had logging enabled!
* ftp: mkdir can do multiple levels at once (support filezilla)
* fix flickering toast on upload finish
* `[💤]` (upload-baton) could disengage if chrome decides to pause the background tab for 10sec (which it sometimes does)
----
## introducing [up2k.exe](https://github.com/9001/copyparty/releases/latest/download/up2k.exe)
the commandline up2k upload / filesearch client, now as a standalone windows exe
* based on python 3.7 so it runs on 32bit windows7 or anything newer
* *no https support* (saves space + the python3.7 openssl is getting old)
* built from b39ff92f34e3fca389c78109d20d5454af761f8e so it can do long filepaths and mojibake
----
⭐️ **you probably want [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) below;**
the exe is [not recommended](https://github.com/9001/copyparty#copypartyexe) for longterm use
and the zip and tar.gz files are source code
(python packages are available at [PyPI](https://pypi.org/project/copyparty/#files))
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
# 2023-0226-2030 `v1.6.6` r 2 0 0
two hundred releases wow
* read-only demo server at https://a.ocv.me/pub/demo/
* [docker image](https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker) [similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) [client testbed](https://cd.ocv.me/b/)
* currently fighting a ground fault so the demo server will be unreliable for a while
## new features
* more docker containers! now runs on x64, x32, aarch64, armhf, ppc64, s390x
* pls let me know if you actually run copyparty on an IBM mainframe 👍
* new [event hook](https://github.com/9001/copyparty/tree/hovudstraum/bin/hooks) type `xiu` runs just once for all recent uploads
* example hook [xiu-sha.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/xiu-sha.py) generates sha512 checksum files
* new arg `--rsp-jtr` simulates connection jitter
* copyparty.exe integrity selftest
* ux:
* return to previous page after logging in
* show a warning on the login page if you're not using https
* freebsd: detect `fetch` and return the [colorful sortable plaintext](https://user-images.githubusercontent.com/241032/215322619-ea5fd606-3654-40ad-94ee-2bc058647bb2.png) listing
## bugfixes
* permit replacing empty files only during a `--blank-wt` grace period
* lifetimes: keep upload-time when a size/mtime change triggers a reindex
* during cleanup after an unlink, never rmdir the entire volume
* rescan button in the controlpanel required volumes to be e2ds
* dupes could get indexed with the wrong mtime
* only affected the search index; the filesystem got the right one
* ux: search results could include the same hit twice in case of overlapping volumes
* ux: upload UI would remain expanded permanently after visiting a huge tab
* ftp: return proper error messages when client does something illegal
* ie11: support the back button
## other changes
* [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) replaces copyparty64.exe -- now built for 64-bit windows 10
* **on win10 it just works** -- on win8 it needs [vc redist 2015](https://www.microsoft.com/en-us/download/details.aspx?id=48145) -- no win7 support
* has the latest security patches, but sfx.py is still better for long-term use
* has pillow and mutagen; can make thumbnails and parse/index media
* [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is the old win7-compatible, dangerously-insecure edition
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
# 2023-0212-1411 `v1.6.5` windows smb fix + win10.exe

View File

@@ -43,11 +43,11 @@ filt=
[ $purge ] && filt='NR>1{print$3}'
[ $filt ] && {
[ $purge ] && {
podman kill $(podman ps -q)
podman rm $(podman ps -qa)
podman kill $(podman ps -q) || true
podman rm $(podman ps -qa) || true
}
podman rmi -f $(podman images -a --history | awk "$filt") || true
podman rmi $(podman images -a --history | awk '/^<none>.*<none>.*-tmp:/{print$3}')
podman rmi $(podman images -a --history | awk '/^<none>.*<none>.*-tmp:/{print$3}') || true
}
[ $pull ] && {

View File

@@ -1,7 +1,9 @@
builds a fully standalone copyparty.exe compatible with 32bit win7-sp1 and later
builds copyparty32.exe, fully standalone, compatible with 32bit win7-sp1 and later
requires a win7 vm which has never been connected to the internet and a host-only network with the linux host at 192.168.123.1
copyparty.exe is built by a win10-ltsc-2021 vm with similar setup
first-time setup steps in notes.txt
run build.sh in the vm to fetch src + compile + push a new exe to the linux host for manual publishing

View File

@@ -9,6 +9,8 @@ tee build2.sh | cmp build.sh && rm build2.sh || {
[[ $r =~ [yY] ]] && mv build{2,}.sh && exec ./build.sh
}
./up2k.sh
uname -s | grep WOW64 && m= || m=32
uname -s | grep NT-10 && w10=1 || w7=1
[ $w7 ] && pyv=37 || pyv=311
@@ -57,6 +59,8 @@ read a b c d _ < <(
sed -r 's/[^0-9]+//;s/[" )]//g;s/[-,]/ /g;s/$/ 0/'
)
sed -r 's/1,2,3,0/'$a,$b,$c,$d'/;s/1\.2\.3/'$a.$b.$c/ <loader.rc >loader.rc2
[ $m ] &&
sed -ri 's/copyparty.exe/copyparty32.exe/' loader.rc2
excl=(
copyparty.broker_mp

View File

@@ -7,6 +7,12 @@ adf0d23a98da38056de25e07e68921739173efc70fb9bf3f68d8c7c3d0d092e09efa69d35c0c9ecc
132a5380f33a245f2e744413a0e1090bc42b7356376de5121397cec5976b04b79f7c9ebe28af222c9c7b01461f7d7920810d220e337694727e0d7cd9e91fa667 pywin32_ctypes-0.2.0-py2.py3-none-any.whl
3c5adf0a36516d284a2ede363051edc1bcc9df925c5a8a9fa2e03cab579dd8d847fdad42f7fd5ba35992e08234c97d2dbfec40a9d12eec61c8dc03758f2bd88e typing_extensions-4.4.0-py3-none-any.whl
4b6e9ae967a769fe32be8cf0bc0d5a213b138d1e0344e97656d08a3d15578d81c06c45b334c872009db2db8f39db0c77c94ff6c35168d5e13801917667c08678 upx-4.0.2-win32.zip
# up2k (win7)
a7d259277af4948bf960682bc9fb45a44b9ae9a19763c8a7c313cef4aa9ec2d447d843e4a7c409e9312c8c8f863a24487a8ee4ffa6891e9b1c4e111bb4723861 certifi-2022.12.7-py3-none-any.whl
2822c0dae180b1c8cfb7a70c8c00bad62af9afdbb18b656236680def9d3f1fcdcb8ef5eb64fc3b4c934385cd175ad5992a2284bcba78a243130de75b2d1650db charset_normalizer-3.1.0-cp37-cp37m-win32.whl
ffdd45326f4e91c02714f7a944cbcc2fdd09299f709cfa8aec0892053eef0134fb80d9ba3790afd319538a86feb619037cbf533e2f5939cb56b35bb17f56c858 idna-3.4-py3-none-any.whl
220e0e122d5851aaccf633224dd7fbd3ba8c8d2720944d8019d6a276ed818d83e3426fe21807f22d673b5428f19fcf9a6b4e645f69bbecd967c568bb6aeb7c8d requests-2.28.2-py3-none-any.whl
8770011f4ad1fe40a3062e6cdf1fda431530c59ee7de3fc5f8c57db54bfdb71c3aa220ca0e0bb1874fc6700e9ebb57defbae54ac84938bc9ad8f074910106681 urllib3-1.26.14-py2.py3-none-any.whl
# win7
91c025f7d94bcdf93df838fab67053165a414fc84e8496f92ecbb910dd55f6b6af5e360bbd051444066880c5a6877e75157bd95e150ead46e5c605930dfc50f2 future-0.18.2.tar.gz
c06b3295d1d0b0f0a6f9a6cd0be861b9b643b4a5ea37857f0bd41c45deaf27bb927b71922dab74e633e43d75d04a9bd0d1c4ad875569740b0f2a98dd2bfa5113 importlib_metadata-5.0.0-py3-none-any.whl

View File

@@ -13,6 +13,13 @@ https://pypi.org/project/MarkupSafe/#files
https://pypi.org/project/mutagen/#files
https://pypi.org/project/Pillow/#files
# up2k (win7) additionals
https://pypi.org/project/certifi/#files
https://pypi.org/project/charset-normalizer/#files # cp37-cp37m-win32.whl
https://pypi.org/project/idna/#files
https://pypi.org/project/requests/#files
https://pypi.org/project/urllib3/#files
# win7 additionals
https://pypi.org/project/future/#files
https://pypi.org/project/importlib-metadata/#files

View File

@@ -1,8 +1,10 @@
#!/bin/bash
set -e
genico() {
# imagemagick png compression is broken, use pillow instead
convert ~/AndroidStudioProjects/PartyUP/metadata/en-US/images/icon.png a.bmp
convert $1 a.bmp
#convert a.bmp -trim -resize '48x48!' -strip a.png
python3 <<'EOF'
@@ -17,11 +19,15 @@ EOF
pngquant --strip --quality 30 a.png
mv a-*.png a.png
python3 <<'EOF'
python3 <<EOF
from PIL import Image
Image.open('a.png').save('loader.ico',sizes=[(48,48)])
Image.open('a.png').save('$2',sizes=[(48,48)])
EOF
rm a.{bmp,png}
}
genico ~/AndroidStudioProjects/PartyUP/metadata/en-US/images/icon.png loader.ico
genico https://raw.githubusercontent.com/googlefonts/noto-emoji/main/png/512/emoji_u1f680.png up2k.ico
ls -al
exit 0

View File

@@ -16,7 +16,7 @@ VSVersionInfo(
StringTable(
'000004b0',
[StringStruct('CompanyName', 'ocv.me'),
StringStruct('FileDescription', 'copyparty'),
StringStruct('FileDescription', 'copyparty file server'),
StringStruct('FileVersion', '1.2.3'),
StringStruct('InternalName', 'copyparty'),
StringStruct('LegalCopyright', '2019, ed'),

View File

@@ -27,6 +27,13 @@ fns=(
Pillow-9.4.0-cp311-cp311-win_amd64.whl
python-3.11.2-amd64.exe
}
[ $w7 ] && fns+=(
certifi-2022.12.7-py3-none-any.whl
chardet-5.1.0-py3-none-any.whl
idna-3.4-py3-none-any.whl
requests-2.28.2-py3-none-any.whl
urllib3-1.26.14-py2.py3-none-any.whl
)
[ $w7 ] && fns+=(
future-0.18.2.tar.gz
importlib_metadata-5.0.0-py3-none-any.whl
@@ -58,28 +65,32 @@ manually install:
===[ copy-paste into git-bash ]================================
uname -s | grep NT-10 && w10=1 || w7=1
[ $w7 ] && pyv=37 || pyv=311
appd=$(cygpath.exe "$APPDATA")
cd ~/Downloads &&
unzip upx-*-win32.zip &&
mv upx-*/upx.exe . &&
python -m ensurepip &&
python -m pip install --user -U pip-*.whl &&
{ [ $w7 ] || python -m pip install --user -U mutagen-*.whl Pillow-*.whl; } &&
{ [ $w10 ] || python -m pip install --user -U {requests,urllib3,charset_normalizer,certifi,idna}-*.whl; } &&
{ [ $w10 ] || python -m pip install --user -U future-*.tar.gz importlib_metadata-*.whl typing_extensions-*.whl zipp-*.whl; } &&
python -m pip install --user -U pyinstaller-*.whl pefile-*.whl pywin32_ctypes-*.whl pyinstaller_hooks_contrib-*.whl altgraph-*.whl &&
sed -ri 's/--lzma/--best/' $appd/Python/Python$pyv/site-packages/pyinstaller/building/utils.py &&
curl -fkLO https://192.168.123.1:3923/cpp/scripts/uncomment.py &&
python uncomment.py $(for d in $appd/Python/Python$pyv/site-packages/{requests,urllib3,charset_normalizer,certifi,idna}; do find $d -name \*.py; done) &&
cd &&
rm -f build.sh &&
curl -fkLO https://192.168.123.1:3923/cpp/scripts/pyinstaller/build.sh &&
curl -fkLO https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.sh &&
echo ok
# python -m pip install --user -U Pillow-9.2.0-cp37-cp37m-win32.whl
# sed -ri 's/, bestopt, /]+bestopt+[/' $APPDATA/Python/Python37/site-packages/pyinstaller/building/utils.py
# sed -ri 's/(^\s+bestopt = ).*/\1["--best","--lzma","--ultra-brute"]/' $APPDATA/Python/Python37/site-packages/pyinstaller/building/utils.py
===[ win10: copy-paste into git-bash ]=========================
appd=$(cygpath.exe "$APPDATA")
curl -fkLO https://192.168.123.1:3923/cpp/scripts/uncomment.py &&
#for f in $appd/Python/Python311/site-packages/mutagen/*.py; do awk -i inplace '/^\s*def _?(save|write)/{sub(/d.*/," ");s=$0;ns=length(s)} ns&&/[^ ]/&&substr($0,0,ns)!=s{ns=0} !ns' "$f"; done &&
python uncomment.py $appd/Python/Python311/site-packages/{mutagen,PIL,jinja2,markupsafe}/*.py &&
sed -ri 's/--lzma/--best/' $APPDATA/Python/Python311/site-packages/pyinstaller/building/utils.py &&
echo ok

View File

@@ -0,0 +1,29 @@
# UTF-8
VSVersionInfo(
ffi=FixedFileInfo(
filevers=(1,2,3,0),
prodvers=(1,2,3,0),
mask=0x3f,
flags=0x0,
OS=0x4,
fileType=0x1,
subtype=0x0,
date=(0, 0)
),
kids=[
StringFileInfo(
[
StringTable(
'000004b0',
[StringStruct('CompanyName', 'ocv.me'),
StringStruct('FileDescription', 'copyparty uploader / filesearch command'),
StringStruct('FileVersion', '1.2.3'),
StringStruct('InternalName', 'up2k'),
StringStruct('LegalCopyright', '2019, ed'),
StringStruct('OriginalFilename', 'up2k.exe'),
StringStruct('ProductName', 'copyparty up2k client'),
StringStruct('ProductVersion', '1.2.3')])
]),
VarFileInfo([VarStruct('Translation', [0, 1200])])
]
)

View File

@@ -0,0 +1,48 @@
#!/bin/bash
set -e
curl -k https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.sh |
tee up2k2.sh | cmp up2k.sh && rm up2k2.sh || {
[ -s up2k2.sh ] || exit 1
echo "new up2k script; upgrade y/n:"
while true; do read -u1 -n1 -r r; [[ $r =~ [yYnN] ]] && break; done
[[ $r =~ [yY] ]] && mv up2k{2,}.sh && exec ./up2k.sh
}
uname -s | grep -E 'WOW64|NT-10' && echo need win7-32 && exit 1
dl() { curl -fkLO "$1"; }
cd ~/Downloads
dl https://192.168.123.1:3923/cpp/bin/up2k.py
dl https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.ico
dl https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.rc
dl https://192.168.123.1:3923/cpp/scripts/pyinstaller/up2k.spec
# $LOCALAPPDATA/programs/python/python37-32/python -m pip install --user -U pyinstaller requests
grep -E '^from .ssl_ import' $APPDATA/python/python37/site-packages/urllib3/util/proxy.py && {
echo golfing
echo > $APPDATA/python/python37/site-packages/requests/certs.py
sed -ri 's/^(DEFAULT_CA_BUNDLE_PATH = ).*/\1""/' $APPDATA/python/python37/site-packages/requests/utils.py
sed -ri '/^import zipfile$/d' $APPDATA/python/python37/site-packages/requests/utils.py
sed -ri 's/"idna"//' $APPDATA/python/python37/site-packages/requests/packages.py
sed -ri 's/import charset_normalizer.*/pass/' $APPDATA/python/python37/site-packages/requests/compat.py
sed -ri 's/raise.*charset_normalizer.*/pass/' $APPDATA/python/python37/site-packages/requests/__init__.py
sed -ri 's/import charset_normalizer.*//' $APPDATA/python/python37/site-packages/requests/packages.py
sed -ri 's/chardet.__name__/"\\roll\\tide"/' $APPDATA/python/python37/site-packages/requests/packages.py
sed -ri 's/chardet,//' $APPDATA/python/python37/site-packages/requests/models.py
for n in util/__init__.py connection.py; do awk -i inplace '/^from (\.util)?\.ssl_ /{s=1} !s; /^\)/{s=0}' $APPDATA/python/python37/site-packages/urllib3/$n; done
sed -ri 's/^from .ssl_ import .*//' $APPDATA/python/python37/site-packages/urllib3/util/proxy.py
echo golfed
}
read a b _ < <(awk -F\" '/^S_VERSION =/{$0=$2;sub(/\./," ");print}' < up2k.py)
sed -r 's/1,2,3,0/'$a,$b,0,0'/;s/1\.2\.3/'$a.$b.0/ <up2k.rc >up2k.rc2
#python uncomment.py up2k.py
$APPDATA/python/python37/scripts/pyinstaller -y --clean --upx-dir=. up2k.spec
./dist/up2k.exe --version
curl -fkT dist/up2k.exe -HPW:wark https://192.168.123.1:3923/

View File

@@ -0,0 +1,78 @@
# -*- mode: python ; coding: utf-8 -*-
block_cipher = None
a = Analysis(
['up2k.py'],
pathex=[],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[
'ftplib',
'lzma',
'pickle',
'ssl',
'tarfile',
'bz2',
'zipfile',
'tracemalloc',
'zlib',
'urllib3.util.ssl_',
'urllib3.contrib.pyopenssl',
'urllib3.contrib.socks',
'certifi',
'idna',
'chardet',
'charset_normalizer',
'email.contentmanager',
'email.policy',
'encodings.zlib_codec',
'encodings.base64_codec',
'encodings.bz2_codec',
'encodings.charmap',
'encodings.hex_codec',
'encodings.palmos',
'encodings.punycode',
'encodings.rot_13',
],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False,
)
# this is the only change to the autogenerated specfile:
xdll = ["libcrypto-1_1.dll"]
a.binaries = TOC([x for x in a.binaries if x[0] not in xdll])
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='up2k',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
version='up2k.rc2',
icon=['up2k.ico'],
)

View File

@@ -0,0 +1,14 @@
#!/bin/bash
set -e
# grep '">encodings.cp' C:/Users/ed/dev/copyparty/bin/dist/xref-up2k.html | sed -r 's/.*encodings.cp//;s/<.*//' | sort -n | uniq | tr '\n' ,
# grep -i encodings -A1 build/up2k/xref-up2k.html | sed -r 's/.*(Missing|Excluded)Module.*//' | grep moduletype -B1 | grep -v moduletype
ex=(
ftplib lzma pickle ssl tarfile bz2 zipfile tracemalloc zlib
urllib3.util.ssl_ urllib3.contrib.pyopenssl urllib3.contrib.socks certifi idna chardet charset_normalizer
email.contentmanager email.policy
encodings.{zlib_codec,base64_codec,bz2_codec,charmap,hex_codec,palmos,punycode,rot_13}
);
cex=(); for a in "${ex[@]}"; do cex+=(--exclude "$a"); done
$APPDATA/python/python37/scripts/pyi-makespec --version-file up2k.rc2 -i up2k.ico -n up2k -c -F up2k.py "${cex[@]}"