Compare commits

..

10 Commits

Author SHA1 Message Date
ed
c39c93725f v1.6.7 2023-03-05 20:18:16 +00:00
ed
d00f0b9fa7 ftp: support filezilla mkdir 2023-03-05 20:18:02 +00:00
ed
01cfc70982 add example for webdav automount 2023-03-05 19:52:45 +00:00
ed
e6aec189bd fix flickering toast on upload finish 2023-03-05 19:49:54 +00:00
ed
c98fff1647 fix chunkpost-handshake race (affects --no-dedup only);
a handshake arriving in the middle of the final chunk could cause
dupes to become empty -- worst case leading to loss of data
2023-03-05 19:45:50 +00:00
ed
0009e31bd3 heavy webworker load can park the main thread of a
background chrome tab for 10sec; piggyback some pokes off postmessage
2023-03-02 22:35:32 +00:00
ed
db95e880b2 thats not how it works 2023-02-28 22:19:06 +00:00
ed
e69fea4a59 exe: update screenshots 2023-02-26 22:26:40 +00:00
ed
4360800a6e update archpkg to 1.6.6 2023-02-26 22:11:56 +00:00
ed
b179e2b031 prisonparty: ignore unresolvable mount paths;
allows startup even if some locations are missing,
for example if a server rebooted and some disks aren't up yet
2023-02-26 22:11:15 +00:00
15 changed files with 136 additions and 92 deletions

View File

@@ -1378,7 +1378,7 @@ you can reduce the sfx size by repacking it; see [./docs/devnotes.md#sfx-repack]
download [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) (win8+) or [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) (win7+)
![copyparty-exe-fs8](https://user-images.githubusercontent.com/241032/194707422-cb7f66c9-41a2-4cb9-8dbc-2ab866cd4338.png)
![copyparty-exe-fs8](https://user-images.githubusercontent.com/241032/221445946-1e328e56-8c5b-44a9-8b9f-dee84d942535.png)
can be convenient on machines where installing python is problematic, however is **not recommended** -- if possible, please use **[copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py)** instead
@@ -1386,7 +1386,7 @@ can be convenient on machines where installing python is problematic, however is
* on win8 it needs [vc redist 2015](https://www.microsoft.com/en-us/download/details.aspx?id=48145), on win10 it just works
* dangerous: [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is compatible with windows7, which means it uses an ancient copy of python (3.7.9) which cannot be upgraded and should never be exposed to the internet (LAN is fine)
* dangerous: [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is compatible with [windows7](https://user-images.githubusercontent.com/241032/221445944-ae85d1f4-d351-4837-b130-82cab57d6cca.png), which means it uses an ancient copy of python (3.7.9) which cannot be upgraded and should never be exposed to the internet (LAN is fine)
* dangerous and deprecated: [copyparty64.exe](https://github.com/9001/copyparty/releases/download/v1.6.5/copyparty64.exe) lets you [run copyparty in WinPE](https://user-images.githubusercontent.com/241032/205454984-e6b550df-3c49-486d-9267-1614078dd0dd.png) and is otherwise completely useless

View File

@@ -4,8 +4,9 @@ set -e
# runs copyparty (or any other program really) in a chroot
#
# assumption: these directories, and everything within, are owned by root
sysdirs=( /bin /lib /lib32 /lib64 /sbin /usr /etc/alternatives )
sysdirs=(); for v in /bin /lib /lib32 /lib64 /sbin /usr /etc/alternatives ; do
[ -e $v ] && sysdirs+=($v)
done
# error-handler
help() { cat <<'EOF'
@@ -38,7 +39,7 @@ while true; do
v="$1"; shift
[ "$v" = -- ] && break # end of volumes
[ "$#" -eq 0 ] && break # invalid usage
vols+=( "$(realpath "$v")" )
vols+=( "$(realpath "$v" || echo "$v")" )
done
pybin="$1"; shift
pybin="$(command -v "$pybin")"
@@ -82,7 +83,7 @@ jail="${jail%/}"
printf '%s\n' "${sysdirs[@]}" "${vols[@]}" | sed -r 's`/$``' | LC_ALL=C sort | uniq |
while IFS= read -r v; do
[ -e "$v" ] || {
# printf '\033[1;31mfolder does not exist:\033[0m %s\n' "/$v"
printf '\033[1;31mfolder does not exist:\033[0m %s\n' "$v"
continue
}
i1=$(stat -c%D.%i "$v" 2>/dev/null || echo a)

View File

@@ -3,7 +3,7 @@ from __future__ import print_function, unicode_literals
"""
up2k.py: upload to copyparty
2023-01-13, v1.2, ed <irc.rizon.net>, MIT-Licensed
2023-03-05, v1.3, ed <irc.rizon.net>, MIT-Licensed
https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py
- dependencies: requests
@@ -520,7 +520,11 @@ def handshake(ar, file, search):
except Exception as ex:
em = str(ex).split("SSLError(")[-1].split("\nURL: ")[0].strip()
if sc == 422 or "<pre>partial upload exists at a different" in txt:
if (
sc == 422
or "<pre>partial upload exists at a different" in txt
or "<pre>source file busy; please try again" in txt
):
file.recheck = True
return [], False
elif sc == 409 or "<pre>upload rejected, file already exists" in txt:

View File

@@ -1,6 +1,6 @@
# Maintainer: icxes <dev.null@need.moe>
pkgname=copyparty
pkgver="1.6.5"
pkgver="1.6.6"
pkgrel=1
pkgdesc="Portable file sharing hub"
arch=("any")
@@ -26,12 +26,12 @@ source=("${url}/releases/download/v${pkgver}/${pkgname}-sfx.py"
"https://raw.githubusercontent.com/9001/${pkgname}/v${pkgver}/LICENSE"
)
backup=("etc/${pkgname}.d/init" )
sha256sums=("947d3f191f96f6a9e451bbcb35c5582ba210d81cfdc92dfa9ab0390dbecf26ee"
sha256sums=("bf8b8a630589db47f70391ed5bf71210bcb2d52d34a9360bd8776123854077c0"
"b8565eba5e64dedba1cf6c7aac7e31c5a731ed7153d6810288a28f00a36c28b2"
"f65c207e0670f9d78ad2e399bda18d5502ff30d2ac79e0e7fc48e7fbdc39afdc"
"c4f396b083c9ec02ad50b52412c84d2a82be7f079b2d016e1c9fad22d68285ff"
"dba701de9fd584405917e923ea1e59dbb249b96ef23bad479cf4e42740b774c8"
"746971e95817c54445ce7f9c8406822dffc814cd5eb8113abd36dd472fd677d7"
"23054bb206153a1ed34038accaf490b8068f9c856e423c2f2595b148b40c0a0c"
"cb2ce3d6277bf2f5a82ecf336cc44963bc6490bcf496ffbd75fc9e21abaa75f3"
)

View File

@@ -1,8 +1,8 @@
# coding: utf-8
VERSION = (1, 6, 6)
VERSION = (1, 6, 7)
CODENAME = "cors k"
BUILD_DT = (2023, 2, 26)
BUILD_DT = (2023, 3, 5)
S_VERSION = ".".join(map(str, VERSION))
S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

View File

@@ -218,7 +218,7 @@ class FtpFs(AbstractedFS):
def mkdir(self, path: str) -> None:
ap = self.rv2a(path, w=True)[0]
bos.mkdir(ap)
bos.makedirs(ap) # filezilla expects this
def listdir(self, path: str) -> list[str]:
vpath = join(self.cwd, path).lstrip("/")

View File

@@ -1714,7 +1714,7 @@ class HttpCli(object):
except:
raise Pebkac(500, min_ex())
x = self.conn.hsrv.broker.ask("up2k.handle_json", body)
x = self.conn.hsrv.broker.ask("up2k.handle_json", body, self.u2fh.aps)
ret = x.get()
if self.is_vproxied:
if "purl" in ret:
@@ -1884,17 +1884,10 @@ class HttpCli(object):
with self.mutex:
self.u2fh.close(path)
# windows cant rename open files
if ANYWIN and path != fin_path and not self.args.nw:
self.conn.hsrv.broker.ask("up2k.finish_upload", ptop, wark).get()
if not ANYWIN and not num_left:
times = (int(time.time()), int(lastmod))
self.log("no more chunks, setting times {}".format(times))
try:
bos.utime(fin_path, times)
except:
self.log("failed to utime ({}, {})".format(fin_path, times))
if not num_left and not self.args.nw:
self.conn.hsrv.broker.ask(
"up2k.finish_upload", ptop, wark, self.u2fh.aps
).get()
cinf = self.headers.get("x-up2k-stat", "")

View File

@@ -149,12 +149,9 @@ class SvcHub(object):
self.log("root", t.format(args.j))
if not args.no_fpool and args.j != 1:
t = "WARNING: --use-fpool combined with multithreading is untested and can probably cause undefined behavior"
if ANYWIN:
t = 'windows cannot do multithreading without --no-fpool, so enabling that -- note that upload performance will suffer if you have microsoft defender "real-time protection" enabled, so you probably want to use -j 1 instead'
args.no_fpool = True
self.log("root", t, c=3)
t = "WARNING: ignoring --use-fpool because multithreading (-j{}) is enabled"
self.log("root", t.format(args.j), c=3)
args.no_fpool = True
bri = "zy"[args.theme % 2 :][:1]
ch = "abcdefghijklmnopqrstuvwx"[int(args.theme / 2)]

View File

@@ -124,6 +124,7 @@ class Up2k(object):
self.droppable: dict[str, list[str]] = {}
self.volstate: dict[str, str] = {}
self.vol_act: dict[str, float] = {}
self.busy_aps: set[str] = set()
self.dupesched: dict[str, list[tuple[str, str, float]]] = {}
self.snap_persist_interval = 300 # persist unfinished index every 5 min
self.snap_discard_interval = 21600 # drop unfinished after 6 hours inactivity
@@ -161,12 +162,6 @@ class Up2k(object):
t = "could not initialize sqlite3, will use in-memory registry only"
self.log(t, 3)
if ANYWIN:
# usually fails to set lastmod too quickly
self.lastmod_q: list[tuple[str, int, tuple[int, int], bool]] = []
self.lastmod_q2 = self.lastmod_q[:]
Daemon(self._lastmodder, "up2k-lastmod")
self.fstab = Fstab(self.log_func)
self.gen_fk = self._gen_fk if self.args.log_fk else gen_filekey
@@ -463,11 +458,9 @@ class Up2k(object):
q = "select * from up where substr(w,1,16)=? and +rd=? and +fn=?"
ups = []
for wrf in wrfs:
try:
# almost definitely exists; don't care if it doesn't
ups.append(cur.execute(q, wrf).fetchone())
except:
pass
up = cur.execute(q, wrf).fetchone()
if up:
ups.append(up)
# t1 = time.time()
# self.log("mapped {} warks in {:.3f} sec".format(len(wrfs), t1 - t0))
@@ -2115,7 +2108,8 @@ class Up2k(object):
if cj["ptop"] not in self.registry:
raise Pebkac(410, "location unavailable")
def handle_json(self, cj: dict[str, Any]) -> dict[str, Any]:
def handle_json(self, cj: dict[str, Any], busy_aps: set[str]) -> dict[str, Any]:
self.busy_aps = busy_aps
try:
# bit expensive; 3.9=10x 3.11=2x
if self.mutex.acquire(timeout=10):
@@ -2289,6 +2283,13 @@ class Up2k(object):
else:
# symlink to the client-provided name,
# returning the previous upload info
if src in self.busy_aps or (
wark in reg and "done" not in reg[wark]
):
raise Pebkac(
422, "source file busy; please try again later"
)
job = deepcopy(job)
job["wark"] = wark
job["at"] = cj.get("at") or time.time()
@@ -2332,7 +2333,7 @@ class Up2k(object):
if not n4g:
raise
if cur:
if cur and not self.args.nw:
zs = "prel name lmod size ptop vtop wark host user addr at"
a = [job[x] for x in zs.split()]
self.db_add(cur, vfs.flags, *a)
@@ -2507,10 +2508,7 @@ class Up2k(object):
if lmod and (not linked or SYMTIME):
times = (int(time.time()), int(lmod))
if ANYWIN:
self.lastmod_q.append((dst, 0, times, False))
else:
bos.utime(dst, times, False)
bos.utime(dst, times, False)
def handle_chunk(
self, ptop: str, wark: str, chash: str
@@ -2591,13 +2589,10 @@ class Up2k(object):
self.regdrop(ptop, wark)
return ret, dst
# windows cant rename open files
if not ANYWIN or src == dst:
self._finish_upload(ptop, wark)
return ret, dst
def finish_upload(self, ptop: str, wark: str) -> None:
def finish_upload(self, ptop: str, wark: str, busy_aps: set[str]) -> None:
self.busy_aps = busy_aps
with self.mutex:
self._finish_upload(ptop, wark)
@@ -2610,6 +2605,10 @@ class Up2k(object):
except Exception as ex:
raise Pebkac(500, "finish_upload, wark, " + repr(ex))
if job["need"]:
t = "finish_upload {} with remaining chunks {}"
raise Pebkac(500, t.format(wark, job["need"]))
# self.log("--- " + wark + " " + dst + " finish_upload atomic " + dst, 4)
atomic_move(src, dst)
@@ -2617,14 +2616,15 @@ class Up2k(object):
vflags = self.flags[ptop]
times = (int(time.time()), int(job["lmod"]))
if ANYWIN:
z1 = (dst, job["size"], times, job["sprs"])
self.lastmod_q.append(z1)
elif not job["hash"]:
try:
bos.utime(dst, times)
except:
pass
self.log(
"no more chunks, setting times {} ({}) on {}".format(
times, bos.path.getsize(dst), dst
)
)
try:
bos.utime(dst, times)
except:
self.log("failed to utime ({}, {})".format(dst, times))
zs = "prel name lmod size ptop vtop wark host user addr"
z2 = [job[x] for x in zs.split()]
@@ -2645,6 +2645,7 @@ class Up2k(object):
if self.idx_wark(vflags, *z2):
del self.registry[ptop][wark]
else:
self.registry[ptop][wark]["done"] = 1
self.regdrop(ptop, wark)
if wake_sr:
@@ -3428,27 +3429,6 @@ class Up2k(object):
if not job["hash"]:
self._finish_upload(job["ptop"], job["wark"])
def _lastmodder(self) -> None:
while True:
ready = self.lastmod_q2
self.lastmod_q2 = self.lastmod_q
self.lastmod_q = []
time.sleep(1)
for path, sz, times, sparse in ready:
self.log("lmod: setting times {} on {}".format(times, path))
try:
bos.utime(path, times, False)
except:
t = "lmod: failed to utime ({}, {}):\n{}"
self.log(t.format(path, times, min_ex()))
if sparse and self.args.sparse and self.args.sparse * 1024 * 1024 <= sz:
try:
sp.check_call(["fsutil", "sparse", "setflag", path, "0"])
except:
self.log("could not unsparse [{}]".format(path), 3)
def _snapshot(self) -> None:
slp = self.snap_persist_interval
while True:

View File

@@ -668,6 +668,7 @@ class FHC(object):
def __init__(self) -> None:
self.cache: dict[str, FHC.CE] = {}
self.aps: set[str] = set()
def close(self, path: str) -> None:
try:
@@ -679,6 +680,7 @@ class FHC(object):
fh.close()
del self.cache[path]
self.aps.remove(path)
def clean(self) -> None:
if not self.cache:
@@ -699,6 +701,7 @@ class FHC(object):
return self.cache[path].fhs.pop()
def put(self, path: str, fh: typing.BinaryIO) -> None:
self.aps.add(path)
try:
ce = self.cache[path]
ce.fhs.append(fh)

View File

@@ -64,6 +64,11 @@
yum install davfs2
{% if accs %}printf '%s\n' <b>{{ pw }}</b> k | {% endif %}mount -t davfs -ouid=1000 http{{ s }}://{{ ep }}/{{ rvp }} <b>mp</b>
</pre>
<p>make it automount on boot:</p>
<pre>
printf '%s\n' "http{{ s }}://{{ ep }}/{{ rvp }} <b>{{ pw }}</b> k" >> /etc/davfs2/secrets
printf '%s\n' "http{{ s }}://{{ ep }}/{{ rvp }} <b>mp</b> davfs rw,user,uid=1000,noauto 0 0" >> /etc/fstab
</pre>
<p>or you can use rclone instead, which is much slower but doesn't require root:</p>
<pre>
rclone config create {{ aname }}-dav webdav url=http{{ s }}://{{ rip }}{{ hport }} vendor=other{% if accs %} user=k pass=<b>{{ pw }}</b>{% endif %}

View File

@@ -114,10 +114,10 @@ function up2k_flagbus() {
do_take(now);
return;
}
if (flag.owner && now - flag.owner[1] > 5000) {
if (flag.owner && now - flag.owner[1] > 12000) {
flag.owner = null;
}
if (flag.wants && now - flag.wants[1] > 5000) {
if (flag.wants && now - flag.wants[1] > 12000) {
flag.wants = null;
}
if (!flag.owner && !flag.wants) {
@@ -772,6 +772,7 @@ function fsearch_explain(n) {
function up2k_init(subtle) {
var r = {
"tact": Date.now(),
"init_deps": init_deps,
"set_fsearch": set_fsearch,
"gotallfiles": [gotallfiles] // hooks
@@ -1647,8 +1648,14 @@ function up2k_init(subtle) {
running = true;
while (true) {
var now = Date.now(),
blocktime = now - r.tact,
is_busy = st.car < st.files.length;
if (blocktime > 2500)
console.log('main thread blocked for ' + blocktime);
r.tact = now;
if (was_busy && !is_busy) {
for (var a = 0; a < st.files.length; a++) {
var t = st.files[a];
@@ -1788,6 +1795,15 @@ function up2k_init(subtle) {
})();
function uptoast() {
if (st.busy.handshake.length)
return;
for (var a = 0; a < st.files.length; a++) {
var t = st.files[a];
if (t.want_recheck && !t.rechecks)
return;
}
var sr = uc.fsearch,
ok = pvis.ctr.ok,
ng = pvis.ctr.ng,
@@ -2043,6 +2059,8 @@ function up2k_init(subtle) {
nbusy++;
reading++;
nchunk++;
if (Date.now() - up2k.tact > 1500)
tasker();
}
function onmsg(d) {
@@ -2373,16 +2391,17 @@ function up2k_init(subtle) {
}
var err_pend = rsp.indexOf('partial upload exists at a different') + 1,
err_srcb = rsp.indexOf('source file busy; please try again') + 1,
err_plug = rsp.indexOf('upload blocked by x') + 1,
err_dupe = rsp.indexOf('upload rejected, file already exists') + 1;
if (err_pend || err_plug || err_dupe) {
if (err_pend || err_srcb || err_plug || err_dupe) {
err = rsp;
ofs = err.indexOf('\n/');
if (ofs !== -1) {
err = err.slice(0, ofs + 1) + linksplit(err.slice(ofs + 2).trimEnd()).join(' ');
}
if (!t.rechecks && err_pend) {
if (!t.rechecks && (err_pend || err_srcb)) {
t.rechecks = 0;
t.want_recheck = true;
}

View File

@@ -112,12 +112,13 @@ if ((document.location + '').indexOf(',rej,') + 1)
try {
console.hist = [];
var CMAXHIST = 100;
var hook = function (t) {
var orig = console[t].bind(console),
cfun = function () {
console.hist.push(Date.now() + ' ' + t + ': ' + Array.from(arguments).join(', '));
if (console.hist.length > 100)
console.hist = console.hist.slice(50);
if (console.hist.length > CMAXHIST)
console.hist = console.hist.slice(CMAXHIST / 2);
orig.apply(console, arguments);
};

View File

@@ -1,3 +1,44 @@
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
# 2023-0226-2030 `v1.6.6` r 2 0 0
two hundred releases wow
* read-only demo server at https://a.ocv.me/pub/demo/
* [docker image](https://github.com/9001/copyparty/tree/hovudstraum/scripts/docker) [similar software](https://github.com/9001/copyparty/blob/hovudstraum/docs/versus.md) [client testbed](https://cd.ocv.me/b/)
* currently fighting a ground fault so the demo server will be unreliable for a while
## new features
* more docker containers! now runs on x64, x32, aarch64, armhf, ppc64, s390x
* pls let me know if you actually run copyparty on an IBM mainframe 👍
* new [event hook](https://github.com/9001/copyparty/tree/hovudstraum/bin/hooks) type `xiu` runs just once for all recent uploads
* example hook [xiu-sha.py](https://github.com/9001/copyparty/blob/hovudstraum/bin/hooks/xiu-sha.py) generates sha512 checksum files
* new arg `--rsp-jtr` simulates connection jitter
* copyparty.exe integrity selftest
* ux:
* return to previous page after logging in
* show a warning on the login page if you're not using https
* freebsd: detect `fetch` and return the [colorful sortable plaintext](https://user-images.githubusercontent.com/241032/215322619-ea5fd606-3654-40ad-94ee-2bc058647bb2.png) listing
## bugfixes
* permit replacing empty files only during a `--blank-wt` grace period
* lifetimes: keep upload-time when a size/mtime change triggers a reindex
* during cleanup after an unlink, never rmdir the entire volume
* rescan button in the controlpanel required volumes to be e2ds
* dupes could get indexed with the wrong mtime
* only affected the search index; the filesystem got the right one
* ux: search results could include the same hit twice in case of overlapping volumes
* ux: upload UI would remain expanded permanently after visiting a huge tab
* ftp: return proper error messages when client does something illegal
* ie11: support the back button
## other changes
* [copyparty.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty.exe) replaces copyparty64.exe -- now built for 64-bit windows 10
* **on win10 it just works** -- on win8 it needs [vc redist 2015](https://www.microsoft.com/en-us/download/details.aspx?id=48145) -- no win7 support
* has the latest security patches, but sfx.py is still better for long-term use
* has pillow and mutagen; can make thumbnails and parse/index media
* [copyparty32.exe](https://github.com/9001/copyparty/releases/latest/download/copyparty32.exe) is the old win7-compatible, dangerously-insecure edition
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
# 2023-0212-1411 `v1.6.5` windows smb fix + win10.exe

View File

@@ -43,11 +43,11 @@ filt=
[ $purge ] && filt='NR>1{print$3}'
[ $filt ] && {
[ $purge ] && {
podman kill $(podman ps -q)
podman rm $(podman ps -qa)
podman kill $(podman ps -q) || true
podman rm $(podman ps -qa) || true
}
podman rmi -f $(podman images -a --history | awk "$filt") || true
podman rmi $(podman images -a --history | awk '/^<none>.*<none>.*-tmp:/{print$3}')
podman rmi $(podman images -a --history | awk '/^<none>.*<none>.*-tmp:/{print$3}') || true
}
[ $pull ] && {