Compare commits
62 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dd122111e6 | ||
|
|
00c177fa74 | ||
|
|
f6c7e49eb8 | ||
|
|
1a8dc3d18a | ||
|
|
38a163a09a | ||
|
|
8f031246d2 | ||
|
|
8f3d97dde7 | ||
|
|
4acaf24d65 | ||
|
|
9a8dbbbcf8 | ||
|
|
a3efc4c726 | ||
|
|
0278bf328f | ||
|
|
17ddd96cc6 | ||
|
|
0e82e79aea | ||
|
|
30f124c061 | ||
|
|
e19d90fcfc | ||
|
|
184bbdd23d | ||
|
|
30b50aec95 | ||
|
|
c3c3d81db1 | ||
|
|
49b7231283 | ||
|
|
edbedcdad3 | ||
|
|
e4ae5f74e6 | ||
|
|
2c7ffe08d7 | ||
|
|
3ca46bae46 | ||
|
|
7e82aaf843 | ||
|
|
315bd71adf | ||
|
|
2c612c9aeb | ||
|
|
36aee085f7 | ||
|
|
d01bb69a9c | ||
|
|
c9b1c48c72 | ||
|
|
aea3843cf2 | ||
|
|
131b6f4b9a | ||
|
|
6efb8b735a | ||
|
|
223b7af2ce | ||
|
|
e72c2a6982 | ||
|
|
dd9b93970e | ||
|
|
e4c7cd81a9 | ||
|
|
12b3a62586 | ||
|
|
2da3bdcd47 | ||
|
|
c1dccbe0ba | ||
|
|
9629fcde68 | ||
|
|
cae436b566 | ||
|
|
01714700ae | ||
|
|
51e6c4852b | ||
|
|
b206c5d64e | ||
|
|
62c3272351 | ||
|
|
c5d822c70a | ||
|
|
9c09b4061a | ||
|
|
c26fb43ced | ||
|
|
deb8f20db6 | ||
|
|
50e18ed8ff | ||
|
|
31f3895f40 | ||
|
|
615929268a | ||
|
|
b8b15814cf | ||
|
|
7766fffe83 | ||
|
|
2a16c150d1 | ||
|
|
418c2166cc | ||
|
|
a4dd44f648 | ||
|
|
5352f7cda7 | ||
|
|
5533b47099 | ||
|
|
e9b14464ee | ||
|
|
4e986e5cd1 | ||
|
|
8a59b40c53 |
58
README.md
58
README.md
@@ -52,7 +52,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
|
||||
* [compress uploads](#compress-uploads) - files can be autocompressed on upload
|
||||
* [database location](#database-location) - in-volume (`.hist/up2k.db`, default) or somewhere else
|
||||
* [metadata from audio files](#metadata-from-audio-files) - set `-e2t` to index tags on upload
|
||||
* [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags
|
||||
* [file parser plugins](#file-parser-plugins) - provide custom parsers to index additional tags, also see [./bin/mtag/README.md](./bin/mtag/README.md)
|
||||
* [upload events](#upload-events) - trigger a script/program on each upload
|
||||
* [complete examples](#complete-examples)
|
||||
* [browser support](#browser-support) - TLDR: yes
|
||||
@@ -78,6 +78,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
|
||||
* [sfx](#sfx) - there are two self-contained "binaries"
|
||||
* [sfx repack](#sfx-repack) - reduce the size of an sfx by removing features
|
||||
* [install on android](#install-on-android)
|
||||
* [reporting bugs](#reporting-bugs) - ideas for context to include in bug reports
|
||||
* [building](#building)
|
||||
* [dev env setup](#dev-env-setup)
|
||||
* [just the sfx](#just-the-sfx)
|
||||
@@ -161,9 +162,11 @@ feature summary
|
||||
* ☑ file manager (cut/paste, delete, [batch-rename](#batch-rename))
|
||||
* ☑ audio player (with OS media controls and opus transcoding)
|
||||
* ☑ image gallery with webm player
|
||||
* ☑ textfile browser with syntax hilighting
|
||||
* ☑ [thumbnails](#thumbnails)
|
||||
* ☑ ...of images using Pillow
|
||||
* ☑ ...of videos using FFmpeg
|
||||
* ☑ ...of audio (spectrograms) using FFmpeg
|
||||
* ☑ cache eviction (max-age; maybe max-size eventually)
|
||||
* ☑ SPA (browse while uploading)
|
||||
* if you use the navpane to navigate, not folders in the file list
|
||||
@@ -233,6 +236,10 @@ some improvement ideas
|
||||
|
||||
## not my bugs
|
||||
|
||||
* iPhones: the volume control doesn't work because [apple doesn't want it to](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html#//apple_ref/doc/uid/TP40009523-CH5-SW11)
|
||||
* *future workaround:* enable the equalizer, make it all-zero, and set a negative boost to reduce the volume
|
||||
* "future" because `AudioContext` is broken in the current iOS version (15.1), maybe one day...
|
||||
|
||||
* Windows: folders cannot be accessed if the name ends with `.`
|
||||
* python or windows bug
|
||||
|
||||
@@ -249,6 +256,7 @@ some improvement ideas
|
||||
|
||||
* is it possible to block read-access to folders unless you know the exact URL for a particular file inside?
|
||||
* yes, using the [`g` permission](#accounts-and-volumes), see the examples there
|
||||
* you can also do this with linux filesystem permissions; `chmod 111 music` will make it possible to access files and folders inside the `music` folder but not list the immediate contents -- also works with other software, not just copyparty
|
||||
|
||||
* can I make copyparty download a file to my server if I give it a URL?
|
||||
* not officially, but there is a [terrible hack](https://github.com/9001/copyparty/blob/hovudstraum/bin/mtag/wget.py) which makes it possible
|
||||
@@ -256,7 +264,10 @@ some improvement ideas
|
||||
|
||||
# accounts and volumes
|
||||
|
||||
per-folder, per-user permissions
|
||||
per-folder, per-user permissions - if your setup is getting complex, consider making a [config file](./docs/example.conf) instead of using arguments
|
||||
* much easier to manage, and you can modify the config at runtime with `systemctl reload copyparty` or more conveniently using the `[reload cfg]` button in the control-panel (if logged in as admin)
|
||||
|
||||
configuring accounts/volumes with arguments:
|
||||
* `-a usr:pwd` adds account `usr` with password `pwd`
|
||||
* `-v .::r` adds current-folder `.` as the webroot, `r`eadable by anyone
|
||||
* the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
|
||||
@@ -473,8 +484,6 @@ the files will be hashed on the client-side, and each hash is sent to the server
|
||||
files go into `[ok]` if they exist (and you get a link to where it is), otherwise they land in `[ng]`
|
||||
* the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else, this is no longer the case but now i've warmed up to the idea too much
|
||||
|
||||
adding the same file multiple times is blocked, so if you first search for a file and then decide to upload it, you have to click the `[cleanup]` button to discard `[done]` files (or just refresh the page)
|
||||
|
||||
|
||||
### unpost
|
||||
|
||||
@@ -562,6 +571,8 @@ and there are *two* editors
|
||||
|
||||
* you can link a particular timestamp in an audio file by adding it to the URL, such as `&20` / `&20s` / `&1m20` / `&t=1:20` after the `.../#af-c8960dab`
|
||||
|
||||
* enabling the audio equalizer can help make gapless albums fully gapless in some browsers (chrome), so consider leaving it on with all the values at zero
|
||||
|
||||
* get a plaintext file listing by adding `?ls=t` to a URL, or a compact colored one with `?ls=v` (for unix terminals)
|
||||
|
||||
* if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider https://ocv.me/dev/?media-osd-bgone.ps1
|
||||
@@ -594,6 +605,8 @@ add the argument `-e2ts` to also scan/index tags from music files, which brings
|
||||
|
||||
using arguments or config files, or a mix of both:
|
||||
* config files (`-c some.conf`) can set additional commandline arguments; see [./docs/example.conf](docs/example.conf)
|
||||
* `kill -s USR1` (same as `systemctl reload copyparty`) to reload accounts and volumes from config files without restarting
|
||||
* or click the `[reload cfg]` button in the control-panel when logged in as admin
|
||||
|
||||
|
||||
## file indexing
|
||||
@@ -632,7 +645,7 @@ if you set `--no-hash [...]` globally, you can enable hashing for specific volum
|
||||
|
||||
set upload rules using volume flags, some examples:
|
||||
|
||||
* `:c,sz=1k-3m` sets allowed filesize between 1 KiB and 3 MiB inclusive (suffixes: b, k, m, g)
|
||||
* `:c,sz=1k-3m` sets allowed filesize between 1 KiB and 3 MiB inclusive (suffixes: `b`, `k`, `m`, `g`)
|
||||
* `:c,nosub` disallow uploading into subdirectories; goes well with `rotn` and `rotf`:
|
||||
* `:c,rotn=1000,2` moves uploads into subfolders, up to 1000 files in each folder before making a new one, two levels deep (must be at least 1)
|
||||
* `:c,rotf=%Y/%m/%d/%H` enforces files to be uploaded into a structure of subfolders according to that date format
|
||||
@@ -666,6 +679,12 @@ things to note,
|
||||
* the files will be indexed after compression, so dupe-detection and file-search will not work as expected
|
||||
|
||||
some examples,
|
||||
* `-v inc:inc:w:c,pk=xz,0`
|
||||
folder named inc, shared at inc, write-only for everyone, forces xz compression at level 0
|
||||
* `-v inc:inc:w:c,pk`
|
||||
same write-only inc, but forces gz compression (default) instead of xz
|
||||
* `-v inc:inc:w:c,gz`
|
||||
allows (but does not force) gz compression if client uploads to `/inc?pk` or `/inc?gz` or `/inc?gz=4`
|
||||
|
||||
|
||||
## database location
|
||||
@@ -710,7 +729,7 @@ see the beautiful mess of a dictionary in [mtag.py](https://github.com/9001/copy
|
||||
|
||||
## file parser plugins
|
||||
|
||||
provide custom parsers to index additional tags
|
||||
provide custom parsers to index additional tags, also see [./bin/mtag/README.md](./bin/mtag/README.md)
|
||||
|
||||
copyparty can invoke external programs to collect additional metadata for files using `mtp` (either as argument or volume flag), there is a default timeout of 30sec
|
||||
|
||||
@@ -781,7 +800,7 @@ TLDR: yes
|
||||
* internet explorer 6 to 8 behave the same
|
||||
* firefox 52 and chrome 49 are the final winxp versions
|
||||
* `*1` yes, but extremely slow (ie10: `1 MiB/s`, ie11: `270 KiB/s`)
|
||||
* `*3` using a wasm decoder which consumes a bit more power
|
||||
* `*3` iOS 11 and newer, opus only, and requires FFmpeg on the server
|
||||
|
||||
quick summary of more eccentric web-browsers trying to view a directory index:
|
||||
|
||||
@@ -801,8 +820,8 @@ quick summary of more eccentric web-browsers trying to view a directory index:
|
||||
interact with copyparty using non-browser clients
|
||||
|
||||
* javascript: dump some state into a file (two separate examples)
|
||||
* `await fetch('https://127.0.0.1:3923/', {method:"PUT", body: JSON.stringify(foo)});`
|
||||
* `var xhr = new XMLHttpRequest(); xhr.open('POST', 'https://127.0.0.1:3923/msgs?raw'); xhr.send('foo');`
|
||||
* `await fetch('//127.0.0.1:3923/', {method:"PUT", body: JSON.stringify(foo)});`
|
||||
* `var xhr = new XMLHttpRequest(); xhr.open('POST', '//127.0.0.1:3923/msgs?raw'); xhr.send('foo');`
|
||||
|
||||
* curl/wget: upload some files (post=file, chunk=stdin)
|
||||
* `post(){ curl -b cppwd=wark -F act=bput -F f=@"$1" http://127.0.0.1:3923/;}`
|
||||
@@ -970,6 +989,7 @@ authenticate using header `Cookie: cppwd=foo` or url param `&pw=foo`
|
||||
| GET | `?txt=iso-8859-1` | ...with specific charset |
|
||||
| GET | `?th` | get image/video at URL as thumbnail |
|
||||
| GET | `?th=opus` | convert audio file to 128kbps opus |
|
||||
| GET | `?th=caf` | ...in the iOS-proprietary container |
|
||||
|
||||
| method | body | result |
|
||||
|--|--|--|
|
||||
@@ -1065,13 +1085,11 @@ pls note that `copyparty-sfx.sh` will fail if you rename `copyparty-sfx.py` to `
|
||||
reduce the size of an sfx by removing features
|
||||
|
||||
if you don't need all the features, you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except if you're on windows then you need msys2 or WSL)
|
||||
* `584k` size of original sfx.py as of v1.1.0
|
||||
* `392k` after `./scripts/make-sfx.sh re no-ogv`
|
||||
* `310k` after `./scripts/make-sfx.sh re no-ogv no-cm`
|
||||
* `269k` after `./scripts/make-sfx.sh re no-ogv no-cm no-hl`
|
||||
* `393k` size of original sfx.py as of v1.1.3
|
||||
* `310k` after `./scripts/make-sfx.sh re no-cm`
|
||||
* `269k` after `./scripts/make-sfx.sh re no-cm no-hl`
|
||||
|
||||
the features you can opt to drop are
|
||||
* `ogv`.js, the opus/vorbis decoder which is needed by apple devices to play foss audio files, saves ~192k
|
||||
* `cm`/easymde, the "fancy" markdown editor, saves ~82k
|
||||
* `hl`, prism, the syntax hilighter, saves ~41k
|
||||
* `fnt`, source-code-pro, the monospace font, saves ~9k
|
||||
@@ -1079,7 +1097,7 @@ the features you can opt to drop are
|
||||
|
||||
for the `re`pack to work, first run one of the sfx'es once to unpack it
|
||||
|
||||
**note:** you can also just download and run [scripts/copyparty-repack.sh](scripts/copyparty-repack.sh) -- this will grab the latest copyparty release from github and do a `no-ogv no-cm` repack; works on linux/macos (and windows with msys2 or WSL)
|
||||
**note:** you can also just download and run [scripts/copyparty-repack.sh](scripts/copyparty-repack.sh) -- this will grab the latest copyparty release from github and do a few repacks; works on linux/macos (and windows with msys2 or WSL)
|
||||
|
||||
|
||||
# install on android
|
||||
@@ -1093,6 +1111,16 @@ echo $?
|
||||
after the initial setup, you can launch copyparty at any time by running `copyparty` anywhere in Termux
|
||||
|
||||
|
||||
# reporting bugs
|
||||
|
||||
ideas for context to include in bug reports
|
||||
|
||||
if something broke during an upload (replacing FILENAME with a part of the filename that broke):
|
||||
```
|
||||
journalctl -aS '48 hour ago' -u copyparty | grep -C10 FILENAME | tee bug.log
|
||||
```
|
||||
|
||||
|
||||
# building
|
||||
|
||||
## dev env setup
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* command-line up2k client [(webm)](https://ocv.me/stuff/u2cli.webm)
|
||||
* file uploads, file-search, autoresume of aborted/broken uploads
|
||||
* faster than browsers
|
||||
* early beta, if something breaks just restart it
|
||||
* if something breaks just restart it
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -6,9 +6,13 @@ some of these rely on libraries which are not MIT-compatible
|
||||
|
||||
* [audio-bpm.py](./audio-bpm.py) detects the BPM of music using the BeatRoot Vamp Plugin; imports GPL2
|
||||
* [audio-key.py](./audio-key.py) detects the melodic key of music using the Mixxx fork of keyfinder; imports GPL3
|
||||
* [media-hash.py](./media-hash.py) generates checksums for audio and video streams; uses FFmpeg (LGPL or GPL)
|
||||
|
||||
these do not have any problematic dependencies:
|
||||
these invoke standalone programs which are GPL or similar, so is legally fine for most purposes:
|
||||
|
||||
* [media-hash.py](./media-hash.py) generates checksums for audio and video streams; uses FFmpeg (LGPL or GPL)
|
||||
* [image-noexif.py](./image-noexif.py) removes exif tags from images; uses exiftool (GPLv1 or artistic-license)
|
||||
|
||||
these do not have any problematic dependencies at all:
|
||||
|
||||
* [cksum.py](./cksum.py) computes various checksums
|
||||
* [exe.py](./exe.py) grabs metadata from .exe and .dll files (example for retrieving multiple tags with one parser)
|
||||
|
||||
@@ -19,18 +19,18 @@ dep: ffmpeg
|
||||
def det(tf):
|
||||
# fmt: off
|
||||
sp.check_call([
|
||||
"ffmpeg",
|
||||
"-nostdin",
|
||||
"-hide_banner",
|
||||
"-v", "fatal",
|
||||
"-ss", "13",
|
||||
"-y", "-i", fsenc(sys.argv[1]),
|
||||
"-map", "0:a:0",
|
||||
"-ac", "1",
|
||||
"-ar", "22050",
|
||||
"-t", "300",
|
||||
"-f", "f32le",
|
||||
tf
|
||||
b"ffmpeg",
|
||||
b"-nostdin",
|
||||
b"-hide_banner",
|
||||
b"-v", b"fatal",
|
||||
b"-ss", b"13",
|
||||
b"-y", b"-i", fsenc(sys.argv[1]),
|
||||
b"-map", b"0:a:0",
|
||||
b"-ac", b"1",
|
||||
b"-ar", b"22050",
|
||||
b"-t", b"300",
|
||||
b"-f", b"f32le",
|
||||
fsenc(tf)
|
||||
])
|
||||
# fmt: on
|
||||
|
||||
|
||||
@@ -23,15 +23,15 @@ dep: ffmpeg
|
||||
def det(tf):
|
||||
# fmt: off
|
||||
sp.check_call([
|
||||
"ffmpeg",
|
||||
"-nostdin",
|
||||
"-hide_banner",
|
||||
"-v", "fatal",
|
||||
"-y", "-i", fsenc(sys.argv[1]),
|
||||
"-map", "0:a:0",
|
||||
"-t", "300",
|
||||
"-sample_fmt", "s16",
|
||||
tf
|
||||
b"ffmpeg",
|
||||
b"-nostdin",
|
||||
b"-hide_banner",
|
||||
b"-v", b"fatal",
|
||||
b"-y", b"-i", fsenc(sys.argv[1]),
|
||||
b"-map", b"0:a:0",
|
||||
b"-t", b"300",
|
||||
b"-sample_fmt", b"s16",
|
||||
fsenc(tf)
|
||||
])
|
||||
# fmt: on
|
||||
|
||||
|
||||
93
bin/mtag/image-noexif.py
Normal file
93
bin/mtag/image-noexif.py
Normal file
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
"""
|
||||
remove exif tags from uploaded images
|
||||
|
||||
dependencies:
|
||||
exiftool
|
||||
|
||||
about:
|
||||
creates a "noexif" subfolder and puts exif-stripped copies of each image there,
|
||||
the reason for the subfolder is to avoid issues with the up2k.db / deduplication:
|
||||
|
||||
if the original image is modified in-place, then copyparty will keep the original
|
||||
hash in up2k.db for a while (until the next volume rescan), so if the image is
|
||||
reuploaded after a rescan then the upload will be renamed and kept as a dupe
|
||||
|
||||
alternatively you could switch the logic around, making a copy of the original
|
||||
image into a subfolder named "exif" and modify the original in-place, but then
|
||||
up2k.db will be out of sync until the next rescan, so any additional uploads
|
||||
of the same image will get symlinked (deduplicated) to the modified copy
|
||||
instead of the original in "exif"
|
||||
|
||||
or maybe delete the original image after processing, that would kinda work too
|
||||
|
||||
example copyparty config to use this:
|
||||
-v/mnt/nas/pics:pics:rwmd,ed:c,e2ts,mte=+noexif:c,mtp=noexif=ejpg,ejpeg,ad,bin/mtag/image-noexif.py
|
||||
|
||||
explained:
|
||||
for realpath /mnt/nas/pics (served at /pics) with read-write-modify-delete for ed,
|
||||
enable file analysis on upload (e2ts),
|
||||
append "noexif" to the list of known tags (mtp),
|
||||
and use mtp plugin "bin/mtag/image-noexif.py" to provide that tag,
|
||||
do this on all uploads with the file extension "jpg" or "jpeg",
|
||||
ad = parse file regardless if FFmpeg thinks it is audio or not
|
||||
|
||||
PS: this requires e2ts to be functional,
|
||||
meaning you need to do at least one of these:
|
||||
* apt install ffmpeg
|
||||
* pip3 install mutagen
|
||||
and your python must have sqlite3 support compiled in
|
||||
"""
|
||||
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import filecmp
|
||||
import subprocess as sp
|
||||
|
||||
try:
|
||||
from copyparty.util import fsenc
|
||||
except:
|
||||
|
||||
def fsenc(p):
|
||||
return p.encode("utf-8")
|
||||
|
||||
|
||||
def main():
|
||||
cwd, fn = os.path.split(sys.argv[1])
|
||||
if os.path.basename(cwd) == "noexif":
|
||||
return
|
||||
|
||||
os.chdir(cwd)
|
||||
f1 = fsenc(fn)
|
||||
f2 = os.path.join(b"noexif", f1)
|
||||
cmd = [
|
||||
b"exiftool",
|
||||
b"-exif:all=",
|
||||
b"-iptc:all=",
|
||||
b"-xmp:all=",
|
||||
b"-P",
|
||||
b"-o",
|
||||
b"noexif/",
|
||||
b"--",
|
||||
f1,
|
||||
]
|
||||
sp.check_output(cmd)
|
||||
if not os.path.exists(f2):
|
||||
print("failed")
|
||||
return
|
||||
|
||||
if filecmp.cmp(f1, f2, shallow=False):
|
||||
print("clean")
|
||||
else:
|
||||
print("exif")
|
||||
|
||||
# lastmod = os.path.getmtime(f1)
|
||||
# times = (int(time.time()), int(lastmod))
|
||||
# os.utime(f2, times)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -13,7 +13,7 @@ try:
|
||||
except:
|
||||
|
||||
def fsenc(p):
|
||||
return p
|
||||
return p.encode("utf-8")
|
||||
|
||||
|
||||
"""
|
||||
@@ -24,13 +24,13 @@ dep: ffmpeg
|
||||
def det():
|
||||
# fmt: off
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-nostdin",
|
||||
"-hide_banner",
|
||||
"-v", "fatal",
|
||||
"-i", fsenc(sys.argv[1]),
|
||||
"-f", "framemd5",
|
||||
"-"
|
||||
b"ffmpeg",
|
||||
b"-nostdin",
|
||||
b"-hide_banner",
|
||||
b"-v", b"fatal",
|
||||
b"-i", fsenc(sys.argv[1]),
|
||||
b"-f", b"framemd5",
|
||||
b"-"
|
||||
]
|
||||
# fmt: on
|
||||
|
||||
|
||||
70
bin/up2k.py
70
bin/up2k.py
@@ -3,7 +3,7 @@ from __future__ import print_function, unicode_literals
|
||||
|
||||
"""
|
||||
up2k.py: upload to copyparty
|
||||
2021-10-31, v0.11, ed <irc.rizon.net>, MIT-Licensed
|
||||
2021-11-28, v0.13, ed <irc.rizon.net>, MIT-Licensed
|
||||
https://github.com/9001/copyparty/blob/hovudstraum/bin/up2k.py
|
||||
|
||||
- dependencies: requests
|
||||
@@ -224,29 +224,47 @@ class CTermsize(object):
|
||||
ss = CTermsize()
|
||||
|
||||
|
||||
def statdir(top):
|
||||
def _scd(err, top):
|
||||
"""non-recursive listing of directory contents, along with stat() info"""
|
||||
if hasattr(os, "scandir"):
|
||||
with os.scandir(top) as dh:
|
||||
for fh in dh:
|
||||
yield [os.path.join(top, fh.name), fh.stat()]
|
||||
else:
|
||||
for name in os.listdir(top):
|
||||
abspath = os.path.join(top, name)
|
||||
with os.scandir(top) as dh:
|
||||
for fh in dh:
|
||||
abspath = os.path.join(top, fh.name)
|
||||
try:
|
||||
yield [abspath, fh.stat()]
|
||||
except:
|
||||
err.append(abspath)
|
||||
|
||||
|
||||
def _lsd(err, top):
|
||||
"""non-recursive listing of directory contents, along with stat() info"""
|
||||
for name in os.listdir(top):
|
||||
abspath = os.path.join(top, name)
|
||||
try:
|
||||
yield [abspath, os.stat(abspath)]
|
||||
except:
|
||||
err.append(abspath)
|
||||
|
||||
|
||||
def walkdir(top):
|
||||
if hasattr(os, "scandir"):
|
||||
statdir = _scd
|
||||
else:
|
||||
statdir = _lsd
|
||||
|
||||
|
||||
def walkdir(err, top):
|
||||
"""recursive statdir"""
|
||||
for ap, inf in sorted(statdir(top)):
|
||||
for ap, inf in sorted(statdir(err, top)):
|
||||
if stat.S_ISDIR(inf.st_mode):
|
||||
for x in walkdir(ap):
|
||||
yield x
|
||||
try:
|
||||
for x in walkdir(err, ap):
|
||||
yield x
|
||||
except:
|
||||
err.append(ap)
|
||||
else:
|
||||
yield ap, inf
|
||||
|
||||
|
||||
def walkdirs(tops):
|
||||
def walkdirs(err, tops):
|
||||
"""recursive statdir for a list of tops, yields [top, relpath, stat]"""
|
||||
sep = "{0}".format(os.sep).encode("ascii")
|
||||
for top in tops:
|
||||
@@ -256,7 +274,7 @@ def walkdirs(tops):
|
||||
stop = os.path.dirname(top)
|
||||
|
||||
if os.path.isdir(top):
|
||||
for ap, inf in walkdir(top):
|
||||
for ap, inf in walkdir(err, top):
|
||||
yield stop, ap[len(stop) :].lstrip(sep), inf
|
||||
else:
|
||||
d, n = top.rsplit(sep, 1)
|
||||
@@ -372,7 +390,7 @@ def handshake(req_ses, url, file, pw, search):
|
||||
r = req_ses.post(url, headers=headers, json=req)
|
||||
break
|
||||
except:
|
||||
eprint("handshake failed, retry...\n")
|
||||
eprint("handshake failed, retrying: {0}\n".format(file.name))
|
||||
time.sleep(1)
|
||||
|
||||
try:
|
||||
@@ -446,10 +464,21 @@ class Ctl(object):
|
||||
|
||||
nfiles = 0
|
||||
nbytes = 0
|
||||
for _, _, inf in walkdirs(ar.files):
|
||||
err = []
|
||||
for _, _, inf in walkdirs(err, ar.files):
|
||||
nfiles += 1
|
||||
nbytes += inf.st_size
|
||||
|
||||
if err:
|
||||
eprint("\n# failed to access {0} paths:\n".format(len(err)))
|
||||
for x in err:
|
||||
eprint(x.decode("utf-8", "replace") + "\n")
|
||||
|
||||
eprint("^ failed to access those {0} paths ^\n\n".format(len(err)))
|
||||
if not ar.ok:
|
||||
eprint("aborting because --ok is not set\n")
|
||||
return
|
||||
|
||||
eprint("found {0} files, {1}\n\n".format(nfiles, humansize(nbytes)))
|
||||
self.nfiles = nfiles
|
||||
self.nbytes = nbytes
|
||||
@@ -460,7 +489,7 @@ class Ctl(object):
|
||||
if ar.te:
|
||||
req_ses.verify = ar.te
|
||||
|
||||
self.filegen = walkdirs(ar.files)
|
||||
self.filegen = walkdirs([], ar.files)
|
||||
if ar.safe:
|
||||
self.safe()
|
||||
else:
|
||||
@@ -476,7 +505,7 @@ class Ctl(object):
|
||||
print("{0} {1}\n hash...".format(self.nfiles - nf, upath))
|
||||
get_hashlist(file, None)
|
||||
|
||||
burl = self.ar.url[:8] + self.ar.url[8:].split("/")[0] + "/"
|
||||
burl = self.ar.url[:12] + self.ar.url[8:].split("/")[0] + "/"
|
||||
while True:
|
||||
print(" hs...")
|
||||
hs = handshake(req_ses, self.ar.url, file, self.ar.a, search)
|
||||
@@ -744,7 +773,7 @@ class Ctl(object):
|
||||
try:
|
||||
upload(req_ses, file, cid, self.ar.a)
|
||||
except:
|
||||
eprint("upload failed, retry...\n")
|
||||
eprint("upload failed, retrying: {0} #{1}\n".format(file.name, cid[:8]))
|
||||
pass # handshake will fix it
|
||||
|
||||
with self.mutex:
|
||||
@@ -783,6 +812,7 @@ source file/folder selection uses rsync syntax, meaning that:
|
||||
ap.add_argument("files", type=unicode, nargs="+", help="files and/or folders to process")
|
||||
ap.add_argument("-a", metavar="PASSWORD", help="password")
|
||||
ap.add_argument("-s", action="store_true", help="file-search (disables upload)")
|
||||
ap.add_argument("--ok", action="store_true", help="continue even if some local files are inaccessible")
|
||||
ap = app.add_argument_group("performance tweaks")
|
||||
ap.add_argument("-j", type=int, metavar="THREADS", default=4, help="parallel connections")
|
||||
ap.add_argument("-nh", action="store_true", help="disable hashing while uploading")
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
# change '/mnt::rw' to another location or permission-set
|
||||
# remove '-p 80,443,3923' to only listen on port 3923
|
||||
# add '-i 127.0.0.1' to only allow local connections
|
||||
# add '--use-fpool' if uploading into nfs locations
|
||||
#
|
||||
# with `Type=notify`, copyparty will signal systemd when it is ready to
|
||||
# accept connections; correctly delaying units depending on copyparty.
|
||||
|
||||
@@ -23,7 +23,7 @@ from textwrap import dedent
|
||||
from .__init__ import E, WINDOWS, ANYWIN, VT100, PY2, unicode
|
||||
from .__version__ import S_VERSION, S_BUILD_DT, CODENAME
|
||||
from .svchub import SvcHub
|
||||
from .util import py_desc, align_tab, IMPLICATIONS, ansi_re
|
||||
from .util import py_desc, align_tab, IMPLICATIONS, ansi_re, min_ex
|
||||
from .authsrv import re_vol
|
||||
|
||||
HAVE_SSL = True
|
||||
@@ -222,6 +222,54 @@ def sighandler(sig=None, frame=None):
|
||||
print("\n".join(msg))
|
||||
|
||||
|
||||
def disable_quickedit():
|
||||
import ctypes
|
||||
import atexit
|
||||
from ctypes import wintypes
|
||||
|
||||
def ecb(ok, fun, args):
|
||||
if not ok:
|
||||
err = ctypes.get_last_error()
|
||||
if err:
|
||||
raise ctypes.WinError(err)
|
||||
return args
|
||||
|
||||
k32 = ctypes.WinDLL("kernel32", use_last_error=True)
|
||||
if PY2:
|
||||
wintypes.LPDWORD = ctypes.POINTER(wintypes.DWORD)
|
||||
|
||||
k32.GetStdHandle.errcheck = ecb
|
||||
k32.GetConsoleMode.errcheck = ecb
|
||||
k32.SetConsoleMode.errcheck = ecb
|
||||
k32.GetConsoleMode.argtypes = (wintypes.HANDLE, wintypes.LPDWORD)
|
||||
k32.SetConsoleMode.argtypes = (wintypes.HANDLE, wintypes.DWORD)
|
||||
|
||||
def cmode(out, mode=None):
|
||||
h = k32.GetStdHandle(-11 if out else -10)
|
||||
if mode:
|
||||
return k32.SetConsoleMode(h, mode)
|
||||
|
||||
mode = wintypes.DWORD()
|
||||
k32.GetConsoleMode(h, ctypes.byref(mode))
|
||||
return mode.value
|
||||
|
||||
# disable quickedit
|
||||
mode = orig_in = cmode(False)
|
||||
quickedit = 0x40
|
||||
extended = 0x80
|
||||
mask = quickedit + extended
|
||||
if mode & mask != extended:
|
||||
atexit.register(cmode, False, orig_in)
|
||||
cmode(False, mode & ~mask | extended)
|
||||
|
||||
# enable colors in case the os.system("rem") trick ever stops working
|
||||
if VT100:
|
||||
mode = orig_out = cmode(True)
|
||||
if mode & 4 != 4:
|
||||
atexit.register(cmode, True, orig_out)
|
||||
cmode(True, mode | 4)
|
||||
|
||||
|
||||
def run_argparse(argv, formatter):
|
||||
ap = argparse.ArgumentParser(
|
||||
formatter_class=formatter,
|
||||
@@ -376,12 +424,16 @@ def run_argparse(argv, formatter):
|
||||
ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload")
|
||||
ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even if copyparty thinks you're better off without")
|
||||
ap2.add_argument("--no-symlink", action="store_true", help="duplicate file contents instead")
|
||||
ap2.add_argument("--reg-cap", metavar="N", type=int, default=9000, help="max number of uploads to keep in memory when running without -e2d")
|
||||
|
||||
ap2 = ap.add_argument_group('network options')
|
||||
ap2.add_argument("-i", metavar="IP", type=u, default="0.0.0.0", help="ip to bind (comma-sep.)")
|
||||
ap2.add_argument("-p", metavar="PORT", type=u, default="3923", help="ports to bind (comma/range)")
|
||||
ap2.add_argument("--rproxy", metavar="DEPTH", type=int, default=1, help="which ip to keep; 0 = tcp, 1 = origin (first x-fwd), 2 = cloudflare, 3 = nginx, -1 = closest proxy")
|
||||
|
||||
ap2.add_argument("--s-wr-sz", metavar="B", type=int, default=256*1024, help="socket write size in bytes")
|
||||
ap2.add_argument("--s-wr-slp", metavar="SEC", type=float, default=0, help="socket write delay in seconds")
|
||||
ap2.add_argument("--rsp-slp", metavar="SEC", type=float, default=0, help="response delay in seconds")
|
||||
|
||||
ap2 = ap.add_argument_group('SSL/TLS options')
|
||||
ap2.add_argument("--http-only", action="store_true", help="disable ssl/tls")
|
||||
ap2.add_argument("--https-only", action="store_true", help="disable plaintext")
|
||||
@@ -392,6 +444,7 @@ def run_argparse(argv, formatter):
|
||||
|
||||
ap2 = ap.add_argument_group('opt-outs')
|
||||
ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
|
||||
ap2.add_argument("--keep-qem", action="store_true", help="do not disable quick-edit-mode on windows")
|
||||
ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
|
||||
ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
|
||||
ap2.add_argument("-nih", action="store_true", help="no info hostname")
|
||||
@@ -433,6 +486,7 @@ def run_argparse(argv, formatter):
|
||||
ap2.add_argument("--no-vthumb", action="store_true", help="disable video thumbnails")
|
||||
ap2.add_argument("--th-size", metavar="WxH", default="320x256", help="thumbnail res")
|
||||
ap2.add_argument("--th-mt", metavar="CORES", type=int, default=cores, help="num cpu cores to use for generating thumbnails")
|
||||
ap2.add_argument("--th-convt", metavar="SEC", type=int, default=60, help="conversion timeout in seconds")
|
||||
ap2.add_argument("--th-no-crop", action="store_true", help="dynamic height; show full image")
|
||||
ap2.add_argument("--th-no-jpg", action="store_true", help="disable jpg output")
|
||||
ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output")
|
||||
@@ -517,7 +571,7 @@ def main(argv=None):
|
||||
if HAVE_SSL:
|
||||
ensure_cert()
|
||||
|
||||
for k, v in zip(argv, argv[1:]):
|
||||
for k, v in zip(argv[1:], argv[2:]):
|
||||
if k == "-c":
|
||||
supp = args_from_cfg(v)
|
||||
argv.extend(supp)
|
||||
@@ -545,6 +599,12 @@ def main(argv=None):
|
||||
except AssertionError:
|
||||
al = run_argparse(argv, Dodge11874)
|
||||
|
||||
if WINDOWS and not al.keep_qem:
|
||||
try:
|
||||
disable_quickedit()
|
||||
except:
|
||||
print("\nfailed to disable quick-edit-mode:\n" + min_ex() + "\n")
|
||||
|
||||
nstrs = []
|
||||
anymod = False
|
||||
for ostr in al.v or []:
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# coding: utf-8
|
||||
|
||||
VERSION = (1, 1, 1)
|
||||
VERSION = (1, 1, 4)
|
||||
CODENAME = "opus"
|
||||
BUILD_DT = (2021, 11, 8)
|
||||
BUILD_DT = (2021, 11, 28)
|
||||
|
||||
S_VERSION = ".".join(map(str, VERSION))
|
||||
S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)
|
||||
|
||||
@@ -126,7 +126,8 @@ class HttpCli(object):
|
||||
self.loud_reply(unicode(ex), status=ex.code, volsan=True)
|
||||
return self.keepalive
|
||||
|
||||
# time.sleep(0.4)
|
||||
if self.args.rsp_slp:
|
||||
time.sleep(self.args.rsp_slp)
|
||||
|
||||
# normalize incoming headers to lowercase;
|
||||
# outgoing headers however are Correct-Case
|
||||
@@ -227,8 +228,8 @@ class HttpCli(object):
|
||||
if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"):
|
||||
self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0]
|
||||
|
||||
ua = self.headers.get("user-agent", "")
|
||||
self.is_rclone = ua.startswith("rclone/")
|
||||
self.ua = self.headers.get("user-agent", "")
|
||||
self.is_rclone = self.ua.startswith("rclone/")
|
||||
if self.is_rclone:
|
||||
uparam["raw"] = False
|
||||
uparam["dots"] = False
|
||||
@@ -283,12 +284,19 @@ class HttpCli(object):
|
||||
n = "604800" if cache == "i" else cache or "69"
|
||||
self.out_headers["Cache-Control"] = "max-age=" + n
|
||||
|
||||
def k304(self):
|
||||
k304 = self.cookies.get("k304", "")
|
||||
return k304 == "y" or ("; Trident/" in self.ua and not k304)
|
||||
|
||||
def send_headers(self, length, status=200, mime=None, headers=None):
|
||||
response = ["{} {} {}".format(self.http_ver, status, HTTPCODE[status])]
|
||||
|
||||
if length is not None:
|
||||
response.append("Content-Length: " + unicode(length))
|
||||
|
||||
if status == 304 and self.k304():
|
||||
self.keepalive = False
|
||||
|
||||
# close if unknown length, otherwise take client's preference
|
||||
response.append("Connection: " + ("Keep-Alive" if self.keepalive else "Close"))
|
||||
|
||||
@@ -428,6 +436,9 @@ class HttpCli(object):
|
||||
if "ups" in self.uparam:
|
||||
return self.tx_ups()
|
||||
|
||||
if "k304" in self.uparam:
|
||||
return self.set_k304()
|
||||
|
||||
if "h" in self.uparam:
|
||||
return self.tx_mounts()
|
||||
|
||||
@@ -505,7 +516,7 @@ class HttpCli(object):
|
||||
return self.handle_stash()
|
||||
|
||||
if "save" in opt:
|
||||
post_sz, _, _, path = self.dump_to_file()
|
||||
post_sz, _, _, _, path = self.dump_to_file()
|
||||
self.log("urlform: {} bytes, {}".format(post_sz, path))
|
||||
elif "print" in opt:
|
||||
reader, _ = self.get_body_reader()
|
||||
@@ -532,11 +543,11 @@ class HttpCli(object):
|
||||
raise Pebkac(405, "don't know how to handle POST({})".format(ctype))
|
||||
|
||||
def get_body_reader(self):
|
||||
chunked = "chunked" in self.headers.get("transfer-encoding", "").lower()
|
||||
if "chunked" in self.headers.get("transfer-encoding", "").lower():
|
||||
return read_socket_chunked(self.sr), -1
|
||||
|
||||
remains = int(self.headers.get("content-length", -1))
|
||||
if chunked:
|
||||
return read_socket_chunked(self.sr), remains
|
||||
elif remains == -1:
|
||||
if remains == -1:
|
||||
self.keepalive = False
|
||||
return read_socket_unbounded(self.sr), remains
|
||||
else:
|
||||
@@ -590,8 +601,8 @@ class HttpCli(object):
|
||||
alg = alg or "gz" # def.pk
|
||||
try:
|
||||
# config-forced opts
|
||||
alg, lv = pk.split(",")
|
||||
lv[alg] = int(lv)
|
||||
alg, nlv = pk.split(",")
|
||||
lv[alg] = int(nlv)
|
||||
except:
|
||||
pass
|
||||
|
||||
@@ -621,7 +632,7 @@ class HttpCli(object):
|
||||
with ren_open(fn, *open_a, **params) as f:
|
||||
f, fn = f["orz"]
|
||||
path = os.path.join(fdir, fn)
|
||||
post_sz, _, sha_b64 = hashcopy(reader, f)
|
||||
post_sz, sha_hex, sha_b64 = hashcopy(reader, f)
|
||||
|
||||
if lim:
|
||||
lim.nup(self.ip)
|
||||
@@ -645,13 +656,14 @@ class HttpCli(object):
|
||||
time.time(),
|
||||
)
|
||||
|
||||
return post_sz, sha_b64, remains, path
|
||||
return post_sz, sha_hex, sha_b64, remains, path
|
||||
|
||||
def handle_stash(self):
|
||||
post_sz, sha_b64, remains, path = self.dump_to_file()
|
||||
post_sz, sha_hex, sha_b64, remains, path = self.dump_to_file()
|
||||
spd = self._spd(post_sz)
|
||||
self.log("{} wrote {}/{} bytes to {}".format(spd, post_sz, remains, path))
|
||||
self.reply("{}\n{}\n".format(post_sz, sha_b64).encode("utf-8"))
|
||||
m = "{}\n{}\n{}\n".format(post_sz, sha_b64, sha_hex[:56])
|
||||
self.reply(m.encode("utf-8"))
|
||||
return True
|
||||
|
||||
def _spd(self, nbytes, add=True):
|
||||
@@ -783,6 +795,10 @@ class HttpCli(object):
|
||||
return True
|
||||
|
||||
def handle_search(self, body):
|
||||
idx = self.conn.get_u2idx()
|
||||
if not hasattr(idx, "p_end"):
|
||||
raise Pebkac(500, "sqlite3 is not available on the server; cannot search")
|
||||
|
||||
vols = []
|
||||
seen = {}
|
||||
for vtop in self.rvol:
|
||||
@@ -794,7 +810,6 @@ class HttpCli(object):
|
||||
seen[vfs] = True
|
||||
vols.append([vfs.vpath, vfs.realpath, vfs.flags])
|
||||
|
||||
idx = self.conn.get_u2idx()
|
||||
t0 = time.time()
|
||||
if idx.p_end:
|
||||
penalty = 0.7
|
||||
@@ -854,63 +869,63 @@ class HttpCli(object):
|
||||
response = x.get()
|
||||
chunksize, cstart, path, lastmod = response
|
||||
|
||||
if self.args.nw:
|
||||
path = os.devnull
|
||||
|
||||
if remains > chunksize:
|
||||
raise Pebkac(400, "your chunk is too big to fit")
|
||||
|
||||
self.log("writing {} #{} @{} len {}".format(path, chash, cstart, remains))
|
||||
|
||||
reader = read_socket(self.sr, remains)
|
||||
|
||||
f = None
|
||||
fpool = not self.args.no_fpool
|
||||
if fpool:
|
||||
with self.mutex:
|
||||
try:
|
||||
f = self.u2fh.pop(path)
|
||||
except:
|
||||
pass
|
||||
|
||||
f = f or open(fsenc(path), "rb+", 512 * 1024)
|
||||
|
||||
try:
|
||||
f.seek(cstart[0])
|
||||
post_sz, _, sha_b64 = hashcopy(reader, f)
|
||||
if self.args.nw:
|
||||
path = os.devnull
|
||||
|
||||
if sha_b64 != chash:
|
||||
raise Pebkac(
|
||||
400,
|
||||
"your chunk got corrupted somehow (received {} bytes); expected vs received hash:\n{}\n{}".format(
|
||||
post_sz, chash, sha_b64
|
||||
),
|
||||
)
|
||||
if remains > chunksize:
|
||||
raise Pebkac(400, "your chunk is too big to fit")
|
||||
|
||||
if len(cstart) > 1 and path != os.devnull:
|
||||
self.log(
|
||||
"clone {} to {}".format(
|
||||
cstart[0], " & ".join(unicode(x) for x in cstart[1:])
|
||||
)
|
||||
)
|
||||
ofs = 0
|
||||
while ofs < chunksize:
|
||||
bufsz = min(chunksize - ofs, 4 * 1024 * 1024)
|
||||
f.seek(cstart[0] + ofs)
|
||||
buf = f.read(bufsz)
|
||||
for wofs in cstart[1:]:
|
||||
f.seek(wofs + ofs)
|
||||
f.write(buf)
|
||||
self.log("writing {} #{} @{} len {}".format(path, chash, cstart, remains))
|
||||
|
||||
ofs += len(buf)
|
||||
reader = read_socket(self.sr, remains)
|
||||
|
||||
self.log("clone {} done".format(cstart[0]))
|
||||
finally:
|
||||
if not fpool:
|
||||
f.close()
|
||||
else:
|
||||
f = None
|
||||
fpool = not self.args.no_fpool
|
||||
if fpool:
|
||||
with self.mutex:
|
||||
self.u2fh.put(path, f)
|
||||
try:
|
||||
f = self.u2fh.pop(path)
|
||||
except:
|
||||
pass
|
||||
|
||||
f = f or open(fsenc(path), "rb+", 512 * 1024)
|
||||
|
||||
try:
|
||||
f.seek(cstart[0])
|
||||
post_sz, _, sha_b64 = hashcopy(reader, f)
|
||||
|
||||
if sha_b64 != chash:
|
||||
m = "your chunk got corrupted somehow (received {} bytes); expected vs received hash:\n{}\n{}"
|
||||
raise Pebkac(400, m.format(post_sz, chash, sha_b64))
|
||||
|
||||
if len(cstart) > 1 and path != os.devnull:
|
||||
self.log(
|
||||
"clone {} to {}".format(
|
||||
cstart[0], " & ".join(unicode(x) for x in cstart[1:])
|
||||
)
|
||||
)
|
||||
ofs = 0
|
||||
while ofs < chunksize:
|
||||
bufsz = min(chunksize - ofs, 4 * 1024 * 1024)
|
||||
f.seek(cstart[0] + ofs)
|
||||
buf = f.read(bufsz)
|
||||
for wofs in cstart[1:]:
|
||||
f.seek(wofs + ofs)
|
||||
f.write(buf)
|
||||
|
||||
ofs += len(buf)
|
||||
|
||||
self.log("clone {} done".format(cstart[0]))
|
||||
finally:
|
||||
if not fpool:
|
||||
f.close()
|
||||
else:
|
||||
with self.mutex:
|
||||
self.u2fh.put(path, f)
|
||||
finally:
|
||||
x = self.conn.hsrv.broker.put(True, "up2k.release_chunk", ptop, wark, chash)
|
||||
x.get() # block client until released
|
||||
|
||||
x = self.conn.hsrv.broker.put(True, "up2k.confirm_chunk", ptop, wark, chash)
|
||||
x = x.get()
|
||||
@@ -957,15 +972,13 @@ class HttpCli(object):
|
||||
def get_pwd_cookie(self, pwd):
|
||||
if pwd in self.asrv.iacct:
|
||||
msg = "login ok"
|
||||
dt = datetime.utcfromtimestamp(time.time() + 60 * 60 * 24 * 365)
|
||||
exp = dt.strftime("%a, %d %b %Y %H:%M:%S GMT")
|
||||
dur = 60 * 60 * 24 * 365
|
||||
else:
|
||||
msg = "naw dude"
|
||||
pwd = "x" # nosec
|
||||
exp = "Fri, 15 Aug 1997 01:00:00 GMT"
|
||||
dur = None
|
||||
|
||||
ck = "cppwd={}; Path=/; Expires={}; SameSite=Lax".format(pwd, exp)
|
||||
return [ck, msg]
|
||||
return [gencookie("cppwd", pwd, dur), msg]
|
||||
|
||||
def handle_mkdir(self):
|
||||
new_dir = self.parser.require("name", 512)
|
||||
@@ -1073,7 +1086,7 @@ class HttpCli(object):
|
||||
f, fname = f["orz"]
|
||||
abspath = os.path.join(fdir, fname)
|
||||
self.log("writing to {}".format(abspath))
|
||||
sz, sha512_hex, _ = hashcopy(p_data, f)
|
||||
sz, sha_hex, sha_b64 = hashcopy(p_data, f)
|
||||
if sz == 0:
|
||||
raise Pebkac(400, "empty files in post")
|
||||
|
||||
@@ -1086,7 +1099,7 @@ class HttpCli(object):
|
||||
bos.unlink(abspath)
|
||||
raise
|
||||
|
||||
files.append([sz, sha512_hex, p_file, fname, abspath])
|
||||
files.append([sz, sha_hex, sha_b64, p_file, fname, abspath])
|
||||
dbv, vrem = vfs.get_dbv(rem)
|
||||
self.conn.hsrv.broker.put(
|
||||
False,
|
||||
@@ -1138,7 +1151,7 @@ class HttpCli(object):
|
||||
jmsg["error"] = errmsg
|
||||
errmsg = "ERROR: " + errmsg
|
||||
|
||||
for sz, sha512, ofn, lfn, ap in files:
|
||||
for sz, sha_hex, sha_b64, ofn, lfn, ap in files:
|
||||
vsuf = ""
|
||||
if self.can_read and "fk" in vfs.flags:
|
||||
vsuf = "?k=" + gen_filekey(
|
||||
@@ -1149,8 +1162,13 @@ class HttpCli(object):
|
||||
)[: vfs.flags["fk"]]
|
||||
|
||||
vpath = "{}/{}".format(upload_vpath, lfn).strip("/")
|
||||
msg += 'sha512: {} // {} bytes // <a href="/{}">{}</a> {}\n'.format(
|
||||
sha512[:56], sz, quotep(vpath) + vsuf, html_escape(ofn, crlf=True), vsuf
|
||||
msg += 'sha512: {} // {} // {} bytes // <a href="/{}">{}</a> {}\n'.format(
|
||||
sha_hex[:56],
|
||||
sha_b64,
|
||||
sz,
|
||||
quotep(vpath) + vsuf,
|
||||
html_escape(ofn, crlf=True),
|
||||
vsuf,
|
||||
)
|
||||
# truncated SHA-512 prevents length extension attacks;
|
||||
# using SHA-512/224, optionally SHA-512/256 = :64
|
||||
@@ -1160,7 +1178,8 @@ class HttpCli(object):
|
||||
self.headers.get("host", "copyparty"),
|
||||
vpath + vsuf,
|
||||
),
|
||||
"sha512": sha512[:56],
|
||||
"sha512": sha_hex[:56],
|
||||
"sha_b64": sha_b64,
|
||||
"sz": sz,
|
||||
"fn": lfn,
|
||||
"fn_orig": ofn,
|
||||
@@ -1378,8 +1397,7 @@ class HttpCli(object):
|
||||
if "gzip" not in supported_editions:
|
||||
decompress = True
|
||||
else:
|
||||
ua = self.headers.get("user-agent", "")
|
||||
if re.match(r"MSIE [4-6]\.", ua) and " SV1" not in ua:
|
||||
if re.match(r"MSIE [4-6]\.", self.ua) and " SV1" not in self.ua:
|
||||
decompress = True
|
||||
|
||||
if not decompress:
|
||||
@@ -1484,10 +1502,10 @@ class HttpCli(object):
|
||||
|
||||
ret = True
|
||||
with open_func(*open_args) as f:
|
||||
if use_sendfile:
|
||||
remains = sendfile_kern(lower, upper, f, self.s)
|
||||
else:
|
||||
remains = sendfile_py(lower, upper, f, self.s)
|
||||
sendfun = sendfile_kern if use_sendfile else sendfile_py
|
||||
remains = sendfun(
|
||||
lower, upper, f, self.s, self.args.s_wr_sz, self.args.s_wr_slp
|
||||
)
|
||||
|
||||
if remains > 0:
|
||||
logmsg += " \033[31m" + unicode(upper - remains) + "\033[0m"
|
||||
@@ -1692,10 +1710,16 @@ class HttpCli(object):
|
||||
tagq=vs["tagq"],
|
||||
mtpq=vs["mtpq"],
|
||||
url_suf=suf,
|
||||
k304=self.k304(),
|
||||
)
|
||||
self.reply(html.encode("utf-8"))
|
||||
return True
|
||||
|
||||
def set_k304(self):
|
||||
ck = gencookie("k304", self.uparam["k304"], 60 * 60 * 24 * 365)
|
||||
self.out_headers["Set-Cookie"] = ck
|
||||
self.redirect("", "?h#cc")
|
||||
|
||||
def tx_404(self, is_403=False):
|
||||
if self.args.vague_403:
|
||||
m = '<h1>404 not found ┐( ´ -`)┌</h1><p>or maybe you don\'t have access -- try logging in or <a href="/?h">go home</a></p>'
|
||||
@@ -1812,13 +1836,16 @@ class HttpCli(object):
|
||||
if not self.args.unpost:
|
||||
raise Pebkac(400, "the unpost feature is disabled in server config")
|
||||
|
||||
idx = self.conn.get_u2idx()
|
||||
if not hasattr(idx, "p_end"):
|
||||
raise Pebkac(500, "sqlite3 is not available on the server; cannot unpost")
|
||||
|
||||
filt = self.uparam.get("filter")
|
||||
lm = "ups [{}]".format(filt)
|
||||
self.log(lm)
|
||||
|
||||
ret = []
|
||||
t0 = time.time()
|
||||
idx = self.conn.get_u2idx()
|
||||
lim = time.time() - self.args.unpost
|
||||
for vol in self.asrv.vfs.all_vols.values():
|
||||
cur = idx.get_cur(vol.realpath)
|
||||
|
||||
@@ -8,7 +8,7 @@ import shutil
|
||||
import subprocess as sp
|
||||
|
||||
from .__init__ import PY2, WINDOWS, unicode
|
||||
from .util import fsenc, fsdec, uncyg, REKOBO_LKEY
|
||||
from .util import fsenc, fsdec, uncyg, runcmd, REKOBO_LKEY
|
||||
from .bos import bos
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ class MParser(object):
|
||||
raise Exception()
|
||||
|
||||
|
||||
def ffprobe(abspath):
|
||||
def ffprobe(abspath, timeout=10):
|
||||
cmd = [
|
||||
b"ffprobe",
|
||||
b"-hide_banner",
|
||||
@@ -82,10 +82,8 @@ def ffprobe(abspath):
|
||||
b"--",
|
||||
fsenc(abspath),
|
||||
]
|
||||
p = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
|
||||
r = p.communicate()
|
||||
txt = r[0].decode("utf-8", "replace")
|
||||
return parse_ffprobe(txt)
|
||||
rc = runcmd(cmd, timeout=timeout)
|
||||
return parse_ffprobe(rc[1])
|
||||
|
||||
|
||||
def parse_ffprobe(txt):
|
||||
|
||||
@@ -57,8 +57,10 @@ class SvcHub(object):
|
||||
if args.log_thrs:
|
||||
start_log_thrs(self.log, args.log_thrs, 0)
|
||||
|
||||
if not ANYWIN and not args.use_fpool:
|
||||
if not args.use_fpool and args.j != 1:
|
||||
args.no_fpool = True
|
||||
m = "multithreading enabled with -j {}, so disabling fpool -- this can reduce upload performance on some filesystems"
|
||||
self.log("root", m.format(args.j))
|
||||
|
||||
if not args.no_fpool and args.j != 1:
|
||||
m = "WARNING: --use-fpool combined with multithreading is untested and can probably cause undefined behavior"
|
||||
@@ -395,7 +397,6 @@ class SvcHub(object):
|
||||
|
||||
def check_mp_enable(self):
|
||||
if self.args.j == 1:
|
||||
self.log("svchub", "multiprocessing disabled by argument -j 1")
|
||||
return False
|
||||
|
||||
if mp.cpu_count() <= 1:
|
||||
|
||||
@@ -30,7 +30,7 @@ class ThumbCli(object):
|
||||
if is_vid and self.args.no_vthumb:
|
||||
return None
|
||||
|
||||
want_opus = fmt == "opus"
|
||||
want_opus = fmt in ("opus", "caf")
|
||||
is_au = ext in FMT_FFA
|
||||
if is_au:
|
||||
if want_opus:
|
||||
|
||||
@@ -90,7 +90,7 @@ def thumb_path(histpath, rem, mtime, fmt):
|
||||
h = hashlib.sha512(fsenc(fn)).digest()
|
||||
fn = base64.urlsafe_b64encode(h).decode("ascii")[:24]
|
||||
|
||||
if fmt == "opus":
|
||||
if fmt in ("opus", "caf"):
|
||||
cat = "ac"
|
||||
else:
|
||||
fmt = "webp" if fmt == "w" else "jpg"
|
||||
@@ -216,7 +216,7 @@ class ThumbSrv(object):
|
||||
elif ext in FMT_FFV:
|
||||
fun = self.conv_ffmpeg
|
||||
elif ext in FMT_FFA:
|
||||
if tpath.endswith(".opus"):
|
||||
if tpath.endswith(".opus") or tpath.endswith(".caf"):
|
||||
fun = self.conv_opus
|
||||
else:
|
||||
fun = self.conv_spec
|
||||
@@ -349,7 +349,7 @@ class ThumbSrv(object):
|
||||
|
||||
def _run_ff(self, cmd):
|
||||
# self.log((b" ".join(cmd)).decode("utf-8"))
|
||||
ret, sout, serr = runcmd(cmd)
|
||||
ret, sout, serr = runcmd(cmd, timeout=self.args.th_convt)
|
||||
if ret != 0:
|
||||
m = "FFmpeg failed (probably a corrupt video file):\n"
|
||||
m += "\n".join(["ff: {}".format(x) for x in serr.split("\n")])
|
||||
@@ -406,21 +406,45 @@ class ThumbSrv(object):
|
||||
if "ac" not in ret:
|
||||
raise Exception("not audio")
|
||||
|
||||
# fmt: off
|
||||
cmd = [
|
||||
b"ffmpeg",
|
||||
b"-nostdin",
|
||||
b"-v", b"error",
|
||||
b"-hide_banner",
|
||||
b"-i", fsenc(abspath),
|
||||
b"-map", b"0:a:0",
|
||||
b"-c:a", b"libopus",
|
||||
b"-b:a", b"128k",
|
||||
fsenc(tpath)
|
||||
]
|
||||
# fmt: on
|
||||
src_opus = abspath.lower().endswith(".opus") or ret["ac"][1] == "opus"
|
||||
want_caf = tpath.endswith(".caf")
|
||||
tmp_opus = tpath
|
||||
if want_caf:
|
||||
tmp_opus = tpath.rsplit(".", 1)[0] + ".opus"
|
||||
|
||||
self._run_ff(cmd)
|
||||
if not want_caf or (not src_opus and not bos.path.isfile(tmp_opus)):
|
||||
# fmt: off
|
||||
cmd = [
|
||||
b"ffmpeg",
|
||||
b"-nostdin",
|
||||
b"-v", b"error",
|
||||
b"-hide_banner",
|
||||
b"-i", fsenc(abspath),
|
||||
b"-map_metadata", b"-1",
|
||||
b"-map", b"0:a:0",
|
||||
b"-c:a", b"libopus",
|
||||
b"-b:a", b"128k",
|
||||
fsenc(tmp_opus)
|
||||
]
|
||||
# fmt: on
|
||||
self._run_ff(cmd)
|
||||
|
||||
if want_caf:
|
||||
# fmt: off
|
||||
cmd = [
|
||||
b"ffmpeg",
|
||||
b"-nostdin",
|
||||
b"-v", b"error",
|
||||
b"-hide_banner",
|
||||
b"-i", fsenc(abspath if src_opus else tmp_opus),
|
||||
b"-map_metadata", b"-1",
|
||||
b"-map", b"0:a:0",
|
||||
b"-c:a", b"copy",
|
||||
b"-f", b"caf",
|
||||
fsenc(tpath)
|
||||
]
|
||||
# fmt: on
|
||||
self._run_ff(cmd)
|
||||
|
||||
def poke(self, tdir):
|
||||
if not self.poke_cd.poke(tdir):
|
||||
@@ -461,7 +485,7 @@ class ThumbSrv(object):
|
||||
thumbpath = os.path.join(histpath, cat)
|
||||
|
||||
# self.log("cln {}".format(thumbpath))
|
||||
exts = ["jpg", "webp"] if cat == "th" else ["opus"]
|
||||
exts = ["jpg", "webp"] if cat == "th" else ["opus", "caf"]
|
||||
maxage = getattr(self.args, cat + "_maxage")
|
||||
now = time.time()
|
||||
prev_b64 = None
|
||||
|
||||
@@ -117,7 +117,16 @@ class U2idx(object):
|
||||
if ok:
|
||||
continue
|
||||
|
||||
v, uq = (uq + " ").split(" ", 1)
|
||||
if uq.startswith('"'):
|
||||
v, uq = uq[1:].split('"', 1)
|
||||
while v.endswith("\\"):
|
||||
v2, uq = uq.split('"', 1)
|
||||
v = v[:-1] + '"' + v2
|
||||
uq = uq.strip()
|
||||
else:
|
||||
v, uq = (uq + " ").split(" ", 1)
|
||||
v = v.replace('\\"', '"')
|
||||
|
||||
if is_key:
|
||||
is_key = False
|
||||
|
||||
|
||||
@@ -73,6 +73,7 @@ class Up2k(object):
|
||||
self.need_rescan = {}
|
||||
self.dupesched = {}
|
||||
self.registry = {}
|
||||
self.droppable = {}
|
||||
self.entags = {}
|
||||
self.flags = {}
|
||||
self.cur = {}
|
||||
@@ -125,11 +126,11 @@ class Up2k(object):
|
||||
all_vols = self.asrv.vfs.all_vols
|
||||
have_e2d = self.init_indexes(all_vols)
|
||||
|
||||
if have_e2d:
|
||||
thr = threading.Thread(target=self._snapshot, name="up2k-snapshot")
|
||||
thr.daemon = True
|
||||
thr.start()
|
||||
thr = threading.Thread(target=self._snapshot, name="up2k-snapshot")
|
||||
thr.daemon = True
|
||||
thr.start()
|
||||
|
||||
if have_e2d:
|
||||
thr = threading.Thread(target=self._hasher, name="up2k-hasher")
|
||||
thr.daemon = True
|
||||
thr.start()
|
||||
@@ -295,7 +296,8 @@ class Up2k(object):
|
||||
def _vis_reg_progress(self, reg):
|
||||
ret = []
|
||||
for _, job in reg.items():
|
||||
ret.append(self._vis_job_progress(job))
|
||||
if job["need"]:
|
||||
ret.append(self._vis_job_progress(job))
|
||||
|
||||
return ret
|
||||
|
||||
@@ -483,26 +485,41 @@ class Up2k(object):
|
||||
self.log("/{} {}".format(vpath, " ".join(sorted(a))), "35")
|
||||
|
||||
reg = {}
|
||||
drp = None
|
||||
path = os.path.join(histpath, "up2k.snap")
|
||||
if "e2d" in flags and bos.path.exists(path):
|
||||
if bos.path.exists(path):
|
||||
with gzip.GzipFile(path, "rb") as f:
|
||||
j = f.read().decode("utf-8")
|
||||
|
||||
reg2 = json.loads(j)
|
||||
try:
|
||||
drp = reg2["droppable"]
|
||||
reg2 = reg2["registry"]
|
||||
except:
|
||||
pass
|
||||
|
||||
for k, job in reg2.items():
|
||||
path = os.path.join(job["ptop"], job["prel"], job["name"])
|
||||
if bos.path.exists(path):
|
||||
reg[k] = job
|
||||
job["poke"] = time.time()
|
||||
job["busy"] = {}
|
||||
else:
|
||||
self.log("ign deleted file in snap: [{}]".format(path))
|
||||
|
||||
m = "loaded snap {} |{}|".format(path, len(reg.keys()))
|
||||
if drp is None:
|
||||
drp = [k for k, v in reg.items() if not v.get("need", [])]
|
||||
else:
|
||||
drp = [x for x in drp if x in reg]
|
||||
|
||||
m = "loaded snap {} |{}| ({})".format(path, len(reg.keys()), len(drp or []))
|
||||
m = [m] + self._vis_reg_progress(reg)
|
||||
self.log("\n".join(m))
|
||||
|
||||
self.flags[ptop] = flags
|
||||
self.registry[ptop] = reg
|
||||
self.droppable[ptop] = drp or []
|
||||
self.regdrop(ptop, None)
|
||||
if not HAVE_SQLITE3 or "e2d" not in flags or "d2d" in flags:
|
||||
return None
|
||||
|
||||
@@ -1256,6 +1273,7 @@ class Up2k(object):
|
||||
"at": at,
|
||||
"hash": [],
|
||||
"need": [],
|
||||
"busy": {},
|
||||
}
|
||||
|
||||
if job and wark in reg:
|
||||
@@ -1338,6 +1356,7 @@ class Up2k(object):
|
||||
"t0": now,
|
||||
"hash": deepcopy(cj["hash"]),
|
||||
"need": [],
|
||||
"busy": {},
|
||||
}
|
||||
# client-provided, sanitized by _get_wark: name, size, lmod
|
||||
for k in [
|
||||
@@ -1444,6 +1463,14 @@ class Up2k(object):
|
||||
if not nchunk:
|
||||
raise Pebkac(400, "unknown chunk")
|
||||
|
||||
if chash in job["busy"]:
|
||||
nh = len(job["hash"])
|
||||
idx = job["hash"].index(chash)
|
||||
m = "that chunk is already being written to:\n {}\n {} {}/{}\n {}"
|
||||
raise Pebkac(400, m.format(wark, chash, idx, nh, job["name"]))
|
||||
|
||||
job["busy"][chash] = 1
|
||||
|
||||
job["poke"] = time.time()
|
||||
|
||||
chunksize = up2k_chunksize(job["size"])
|
||||
@@ -1453,6 +1480,14 @@ class Up2k(object):
|
||||
|
||||
return [chunksize, ofs, path, job["lmod"]]
|
||||
|
||||
def release_chunk(self, ptop, wark, chash):
|
||||
with self.mutex:
|
||||
job = self.registry[ptop].get(wark)
|
||||
if job:
|
||||
job["busy"].pop(chash, None)
|
||||
|
||||
return [True]
|
||||
|
||||
def confirm_chunk(self, ptop, wark, chash):
|
||||
with self.mutex:
|
||||
try:
|
||||
@@ -1463,6 +1498,8 @@ class Up2k(object):
|
||||
except Exception as ex:
|
||||
return "confirm_chunk, wark, " + repr(ex)
|
||||
|
||||
job["busy"].pop(chash, None)
|
||||
|
||||
try:
|
||||
job["need"].remove(chash)
|
||||
except Exception as ex:
|
||||
@@ -1473,7 +1510,7 @@ class Up2k(object):
|
||||
return ret, src
|
||||
|
||||
if self.args.nw:
|
||||
# del self.registry[ptop][wark]
|
||||
self.regdrop(ptop, wark)
|
||||
return ret, dst
|
||||
|
||||
# windows cant rename open files
|
||||
@@ -1505,9 +1542,9 @@ class Up2k(object):
|
||||
a = [job[x] for x in "ptop wark prel name lmod size addr".split()]
|
||||
a += [job.get("at") or time.time()]
|
||||
if self.idx_wark(*a):
|
||||
# self.log("pop " + wark + " " + dst + " finish_upload idx_wark", 4)
|
||||
del self.registry[ptop][wark]
|
||||
# in-memory registry is reserved for unfinished uploads
|
||||
else:
|
||||
self.regdrop(ptop, wark)
|
||||
|
||||
dupes = self.dupesched.pop(dst, [])
|
||||
if not dupes:
|
||||
@@ -1527,6 +1564,21 @@ class Up2k(object):
|
||||
if cur:
|
||||
cur.connection.commit()
|
||||
|
||||
def regdrop(self, ptop, wark):
|
||||
t = self.droppable[ptop]
|
||||
if wark:
|
||||
t.append(wark)
|
||||
|
||||
if len(t) <= self.args.reg_cap:
|
||||
return
|
||||
|
||||
n = len(t) - int(self.args.reg_cap / 2)
|
||||
m = "up2k-registry [{}] has {} droppables; discarding {}"
|
||||
self.log(m.format(ptop, len(t), n))
|
||||
for k in t[:n]:
|
||||
self.registry[ptop].pop(k, None)
|
||||
self.droppable[ptop] = t[n:]
|
||||
|
||||
def idx_wark(self, ptop, wark, rd, fn, lmod, sz, ip, at):
|
||||
cur = self.cur.get(ptop)
|
||||
if not cur:
|
||||
@@ -2042,7 +2094,8 @@ class Up2k(object):
|
||||
bos.makedirs(histpath)
|
||||
|
||||
path2 = "{}.{}".format(path, os.getpid())
|
||||
j = json.dumps(reg, indent=2, sort_keys=True).encode("utf-8")
|
||||
body = {"droppable": self.droppable[ptop], "registry": reg}
|
||||
j = json.dumps(body, indent=2, sort_keys=True).encode("utf-8")
|
||||
with gzip.GzipFile(path2, "wb") as f:
|
||||
f.write(j)
|
||||
|
||||
|
||||
@@ -104,6 +104,7 @@ MIMES = {
|
||||
"txt": "text/plain",
|
||||
"js": "text/javascript",
|
||||
"opus": "audio/ogg; codecs=opus",
|
||||
"caf": "audio/x-caf",
|
||||
"mp3": "audio/mpeg",
|
||||
"m4a": "audio/mp4",
|
||||
"jpg": "image/jpeg",
|
||||
@@ -821,6 +822,17 @@ def gen_filekey(salt, fspath, fsize, inode):
|
||||
).decode("ascii")
|
||||
|
||||
|
||||
def gencookie(k, v, dur):
|
||||
v = v.replace(";", "")
|
||||
if dur:
|
||||
dt = datetime.utcfromtimestamp(time.time() + dur)
|
||||
exp = dt.strftime("%a, %d %b %Y %H:%M:%S GMT")
|
||||
else:
|
||||
exp = "Fri, 15 Aug 1997 01:00:00 GMT"
|
||||
|
||||
return "{}={}; Path=/; Expires={}; SameSite=Lax".format(k, v, exp)
|
||||
|
||||
|
||||
def humansize(sz, terse=False):
|
||||
for unit in ["B", "KiB", "MiB", "GiB", "TiB"]:
|
||||
if sz < 1024:
|
||||
@@ -1164,12 +1176,14 @@ def hashcopy(fin, fout):
|
||||
return tlen, hashobj.hexdigest(), digest_b64
|
||||
|
||||
|
||||
def sendfile_py(lower, upper, f, s):
|
||||
def sendfile_py(lower, upper, f, s, bufsz, slp):
|
||||
remains = upper - lower
|
||||
f.seek(lower)
|
||||
while remains > 0:
|
||||
# time.sleep(0.01)
|
||||
buf = f.read(min(1024 * 32, remains))
|
||||
if slp:
|
||||
time.sleep(slp)
|
||||
|
||||
buf = f.read(min(bufsz, remains))
|
||||
if not buf:
|
||||
return remains
|
||||
|
||||
@@ -1182,7 +1196,7 @@ def sendfile_py(lower, upper, f, s):
|
||||
return 0
|
||||
|
||||
|
||||
def sendfile_kern(lower, upper, f, s):
|
||||
def sendfile_kern(lower, upper, f, s, bufsz, slp):
|
||||
out_fd = s.fileno()
|
||||
in_fd = f.fileno()
|
||||
ofs = lower
|
||||
@@ -1310,9 +1324,17 @@ def guess_mime(url, fallback="application/octet-stream"):
|
||||
return ret
|
||||
|
||||
|
||||
def runcmd(argv):
|
||||
def runcmd(argv, timeout=None):
|
||||
p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
|
||||
stdout, stderr = p.communicate()
|
||||
if not timeout or PY2:
|
||||
stdout, stderr = p.communicate()
|
||||
else:
|
||||
try:
|
||||
stdout, stderr = p.communicate(timeout=timeout)
|
||||
except sp.TimeoutExpired:
|
||||
p.kill()
|
||||
stdout, stderr = p.communicate()
|
||||
|
||||
stdout = stdout.decode("utf-8", "replace")
|
||||
stderr = stderr.decode("utf-8", "replace")
|
||||
return [p.returncode, stdout, stderr]
|
||||
|
||||
@@ -237,7 +237,7 @@ window.baguetteBox = (function () {
|
||||
}
|
||||
|
||||
function keyDownHandler(e) {
|
||||
if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
|
||||
if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing || modal.busy)
|
||||
return;
|
||||
|
||||
var k = e.code + '', v = vid();
|
||||
|
||||
@@ -44,8 +44,9 @@ pre, code, tt, #doc, #doc>code {
|
||||
margin-left: -.7em;
|
||||
}
|
||||
#files {
|
||||
border-spacing: 0;
|
||||
z-index: 1;
|
||||
top: -.3em;
|
||||
border-spacing: 0;
|
||||
position: relative;
|
||||
}
|
||||
#files tbody a {
|
||||
@@ -72,7 +73,7 @@ a, #files tbody div a:last-child {
|
||||
}
|
||||
#files thead {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
top: -1px;
|
||||
}
|
||||
#files thead a {
|
||||
color: #999;
|
||||
@@ -82,7 +83,7 @@ a, #files tbody div a:last-child {
|
||||
background: #1c1c1c;
|
||||
}
|
||||
#files thead th {
|
||||
padding: 0 .3em .3em .3em;
|
||||
padding: .3em;
|
||||
border-bottom: 1px solid #444;
|
||||
cursor: pointer;
|
||||
}
|
||||
@@ -239,6 +240,8 @@ html.light #ggrid>a[tt].sel {
|
||||
#files tbody tr.sel:hover td,
|
||||
#files tbody tr.sel:focus td,
|
||||
#ggrid>a.sel:hover,
|
||||
#ggrid>a.sel:focus,
|
||||
html.light #ggrid>a.sel:focus,
|
||||
html.light #ggrid>a.sel:hover {
|
||||
color: #fff;
|
||||
background: #d39;
|
||||
@@ -294,6 +297,8 @@ html.light #ggrid>a.sel {
|
||||
width: 100%;
|
||||
z-index: 3;
|
||||
touch-action: none;
|
||||
}
|
||||
#widget.anim {
|
||||
transition: bottom 0.15s;
|
||||
}
|
||||
#widget.open {
|
||||
@@ -559,6 +564,11 @@ input[type="radio"]:checked+label,
|
||||
input[type="checkbox"]:checked+label {
|
||||
color: #fc5;
|
||||
}
|
||||
.opwide div>span>input+label {
|
||||
padding: .3em 0 .3em .3em;
|
||||
margin: 0 0 0 -.3em;
|
||||
cursor: pointer;
|
||||
}
|
||||
.opview input.i {
|
||||
width: calc(100% - 16.2em);
|
||||
}
|
||||
@@ -871,13 +881,15 @@ html.light #tree.nowrap .ntree a+a:hover {
|
||||
}
|
||||
#thumbs,
|
||||
#au_fullpre,
|
||||
#au_os_seek,
|
||||
#au_osd_cv,
|
||||
#u2tdate {
|
||||
opacity: .3;
|
||||
}
|
||||
#griden.on+#thumbs,
|
||||
#au_preload.on+#au_fullpre,
|
||||
#au_os_ctl.on+#au_osd_cv,
|
||||
#au_os_ctl.on+#au_os_seek,
|
||||
#au_os_ctl.on+#au_os_seek+#au_osd_cv,
|
||||
#u2turbo.on+#u2tdate {
|
||||
opacity: 1;
|
||||
}
|
||||
@@ -940,6 +952,12 @@ html.light .ghead {
|
||||
#ggrid>a.dir:before {
|
||||
content: '📂';
|
||||
}
|
||||
#ggrid>a.au:before {
|
||||
content: '💾';
|
||||
}
|
||||
html.np_open #ggrid>a.au:before {
|
||||
content: '▶';
|
||||
}
|
||||
#ggrid>a:before {
|
||||
display: block;
|
||||
position: absolute;
|
||||
@@ -949,6 +967,12 @@ html.light .ghead {
|
||||
background: linear-gradient(135deg,rgba(255,255,255,0) 50%,rgba(255,255,255,0.2));
|
||||
border-radius: .3em;
|
||||
font-size: 2em;
|
||||
transition: font-size .15s, margin .15s;
|
||||
}
|
||||
#ggrid>a:focus:before,
|
||||
#ggrid>a:hover:before {
|
||||
font-size: 2.5em;
|
||||
margin: -.2em;
|
||||
}
|
||||
#op_unpost {
|
||||
padding: 1em;
|
||||
@@ -1153,6 +1177,7 @@ html,
|
||||
#ggrid>a[tt] {
|
||||
background: linear-gradient(135deg, #2c2c2c 95%, #444 95%);
|
||||
}
|
||||
#ggrid>a:focus,
|
||||
#ggrid>a:hover {
|
||||
background: #383838;
|
||||
border-color: #555;
|
||||
@@ -1166,6 +1191,7 @@ html.light #ggrid>a {
|
||||
html.light #ggrid>a[tt] {
|
||||
background: linear-gradient(135deg, #f7f7f7 95%, #ccc 95%);
|
||||
}
|
||||
html.light #ggrid>a:focus,
|
||||
html.light #ggrid>a:hover {
|
||||
background: #fff;
|
||||
border-color: #ccc;
|
||||
@@ -1658,8 +1684,6 @@ html.light #bbox-overlay figcaption a {
|
||||
|
||||
#op_up2k {
|
||||
padding: 0 1em 1em 1em;
|
||||
min-height: 0;
|
||||
transition: min-height .2s;
|
||||
}
|
||||
#drops {
|
||||
display: none;
|
||||
@@ -1824,13 +1848,18 @@ html.light #u2err.err {
|
||||
#u2notbtn * {
|
||||
line-height: 1.3em;
|
||||
}
|
||||
#u2tabw {
|
||||
min-height: 0;
|
||||
transition: min-height .2s;
|
||||
margin: 3em 0;
|
||||
}
|
||||
#u2tab {
|
||||
border-collapse: collapse;
|
||||
margin: 3em auto;
|
||||
width: calc(100% - 2em);
|
||||
max-width: 100em;
|
||||
margin: 0 auto;
|
||||
}
|
||||
#op_up2k.srch #u2tab {
|
||||
#op_up2k.srch #u2tabf {
|
||||
max-width: none;
|
||||
}
|
||||
#u2tab td {
|
||||
@@ -2096,10 +2125,6 @@ html.light #u2foot .warn span {
|
||||
color: #fff;
|
||||
padding-left: .2em;
|
||||
}
|
||||
#u2cleanup {
|
||||
float: right;
|
||||
margin-bottom: -.3em;
|
||||
}
|
||||
.fsearch_explain {
|
||||
padding-left: .7em;
|
||||
font-size: 1.1em;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -37,7 +37,9 @@ a+a {
|
||||
float: right;
|
||||
margin: -.2em 0 0 .5em;
|
||||
}
|
||||
.logout {
|
||||
.logout,
|
||||
.btns a,
|
||||
a.r {
|
||||
color: #c04;
|
||||
border-color: #c7a;
|
||||
}
|
||||
@@ -78,6 +80,11 @@ table {
|
||||
margin-top: .3em;
|
||||
text-align: right;
|
||||
}
|
||||
blockquote {
|
||||
margin: 0 0 0 .6em;
|
||||
padding: .7em 1em;
|
||||
border-left: .3em solid rgba(128,128,128,0.5);
|
||||
}
|
||||
|
||||
|
||||
html.dark,
|
||||
@@ -94,7 +101,9 @@ html.dark a {
|
||||
background: #057;
|
||||
border-color: #37a;
|
||||
}
|
||||
html.dark .logout {
|
||||
html.dark .logout,
|
||||
html.dark .btns a,
|
||||
html.dark a.r {
|
||||
background: #804;
|
||||
border-color: #c28;
|
||||
}
|
||||
|
||||
@@ -72,6 +72,16 @@
|
||||
</ul>
|
||||
{%- endif %}
|
||||
|
||||
<h1 id="cc">client config:</h1>
|
||||
<ul>
|
||||
{% if k304 %}
|
||||
<li><a href="/?k304=n" class="r">disable k304</a> (currently enabled)
|
||||
{%- else %}
|
||||
<li><a href="/?k304=y">enable k304</a> (currently disabled)
|
||||
{% endif %}
|
||||
<blockquote>enabling this will disconnect your client on every HTTP 304, which can prevent some buggy browsers/proxies from getting stuck (suddenly not being able to load pages), <em>but</em> it will also make things slower in general</blockquote></li>
|
||||
</ul>
|
||||
|
||||
<h1>login for more:</h1>
|
||||
<ul>
|
||||
<form method="post" enctype="multipart/form-data" action="/{{ qvpath }}">
|
||||
@@ -84,8 +94,7 @@
|
||||
<a href="#" id="repl">π</a>
|
||||
<script>
|
||||
|
||||
if (localStorage.lightmode != 1)
|
||||
document.documentElement.setAttribute("class", "dark");
|
||||
document.documentElement.setAttribute("class", localStorage.lightmode == 1 ? "light" : "dark");
|
||||
|
||||
</script>
|
||||
<script src="/.cpr/util.js?_={{ ts }}"></script>
|
||||
|
||||
@@ -116,6 +116,20 @@ html {
|
||||
#toast.err #toastc {
|
||||
background: #d06;
|
||||
}
|
||||
#tth {
|
||||
color: #fff;
|
||||
background: #111;
|
||||
font-size: .9em;
|
||||
padding: 0 .26em;
|
||||
line-height: .97em;
|
||||
border-radius: 1em;
|
||||
position: absolute;
|
||||
display: none;
|
||||
}
|
||||
#tth.act {
|
||||
display: block;
|
||||
z-index: 9001;
|
||||
}
|
||||
#tt.b {
|
||||
padding: 0 2em;
|
||||
border-radius: .5em;
|
||||
@@ -133,7 +147,8 @@ html {
|
||||
}
|
||||
#modalc code,
|
||||
#tt code {
|
||||
background: #3c3c3c;
|
||||
color: #eee;
|
||||
background: #444;
|
||||
padding: .1em .3em;
|
||||
border-top: 1px solid #777;
|
||||
border-radius: .3em;
|
||||
@@ -158,6 +173,10 @@ html.light #tt code {
|
||||
html.light #tt em {
|
||||
color: #d38;
|
||||
}
|
||||
html.light #tth {
|
||||
color: #000;
|
||||
background: #fff;
|
||||
}
|
||||
#modal {
|
||||
position: fixed;
|
||||
overflow: auto;
|
||||
|
||||
@@ -525,13 +525,15 @@ function Donut(uc, st) {
|
||||
}
|
||||
|
||||
r.on = function (ya) {
|
||||
r.fc = 99;
|
||||
r.fc = r.tc = 99;
|
||||
r.eta = null;
|
||||
r.base = pos();
|
||||
optab.innerHTML = ya ? svg() : optab.getAttribute('ico');
|
||||
el = QS('#ops a .donut');
|
||||
if (!ya)
|
||||
if (!ya) {
|
||||
favico.upd();
|
||||
wintitle();
|
||||
}
|
||||
};
|
||||
r.do = function () {
|
||||
if (!el)
|
||||
@@ -541,6 +543,11 @@ function Donut(uc, st) {
|
||||
v = pos() - r.base,
|
||||
ofs = el.style.strokeDashoffset = o - o * v / t;
|
||||
|
||||
if (++r.tc >= 10) {
|
||||
wintitle(f2f(v * 100 / t, 1) + '%, ' + r.eta + 's, ', true);
|
||||
r.tc = 0;
|
||||
}
|
||||
|
||||
if (favico.txt) {
|
||||
if (++r.fc < 10 && r.eta && r.eta > 99)
|
||||
return;
|
||||
@@ -562,9 +569,9 @@ function fsearch_explain(n) {
|
||||
return toast.inf(60, 'your access to this folder is Read-Only\n\n' + (acct == '*' ? 'you are currently not logged in' : 'you are currently logged in as "' + acct + '"'));
|
||||
|
||||
if (bcfg_get('fsearch', false))
|
||||
return toast.inf(60, 'you are currently in file-search mode\n\nswitch to upload-mode by clicking the green magnifying glass (next to the big yellow search button), and then refresh\n\nsorry');
|
||||
return toast.inf(60, 'you are currently in file-search mode\n\nswitch to upload-mode by clicking the green magnifying glass (next to the big yellow search button), and try uploading again\n\nsorry');
|
||||
|
||||
return toast.inf(60, 'refresh the page and try again, it should work now');
|
||||
return toast.inf(60, 'try again, it should work now');
|
||||
}
|
||||
|
||||
|
||||
@@ -669,6 +676,7 @@ function up2k_init(subtle) {
|
||||
|
||||
var st = {
|
||||
"files": [],
|
||||
"seen": {},
|
||||
"todo": {
|
||||
"head": [],
|
||||
"hash": [],
|
||||
@@ -727,7 +735,6 @@ function up2k_init(subtle) {
|
||||
if (++nenters <= 0)
|
||||
nenters = 1;
|
||||
|
||||
//console.log(nenters, Date.now(), 'enter', this, e.target);
|
||||
if (onover.bind(this)(e))
|
||||
return true;
|
||||
|
||||
@@ -749,12 +756,19 @@ function up2k_init(subtle) {
|
||||
ebi('up_dz').setAttribute('err', mup || '');
|
||||
ebi('srch_dz').setAttribute('err', msr || '');
|
||||
}
|
||||
function onoverb(e) {
|
||||
// zones are alive; disable cuo2duo branch
|
||||
document.body.ondragover = document.body.ondrop = null;
|
||||
return onover.bind(this)(e);
|
||||
}
|
||||
function onover(e) {
|
||||
try {
|
||||
var ok = false, dt = e.dataTransfer.types;
|
||||
for (var a = 0; a < dt.length; a++)
|
||||
if (dt[a] == 'Files')
|
||||
ok = true;
|
||||
else if (dt[a] == 'text/uri-list')
|
||||
return true;
|
||||
|
||||
if (!ok)
|
||||
return true;
|
||||
@@ -780,17 +794,20 @@ function up2k_init(subtle) {
|
||||
clmod(ebi('drops'), 'vis');
|
||||
clmod(ebi('up_dz'), 'hl');
|
||||
clmod(ebi('srch_dz'), 'hl');
|
||||
// cuo2duo:
|
||||
document.body.ondragover = onover;
|
||||
document.body.ondrop = gotfile;
|
||||
}
|
||||
|
||||
//console.log(nenters, Date.now(), 'leave', this, e && e.target);
|
||||
}
|
||||
document.body.ondragenter = ondrag;
|
||||
document.body.ondragleave = offdrag;
|
||||
document.body.ondragover = onover;
|
||||
document.body.ondrop = gotfile;
|
||||
|
||||
var drops = [ebi('up_dz'), ebi('srch_dz')];
|
||||
for (var a = 0; a < 2; a++) {
|
||||
drops[a].ondragenter = ondrag;
|
||||
drops[a].ondragover = onover;
|
||||
drops[a].ondragover = onoverb;
|
||||
drops[a].ondragleave = offdrag;
|
||||
drops[a].ondrop = gotfile;
|
||||
}
|
||||
@@ -800,7 +817,10 @@ function up2k_init(subtle) {
|
||||
ev(e);
|
||||
nenters = 0;
|
||||
offdrag.bind(this)();
|
||||
var dz = (this && this.getAttribute('id'));
|
||||
var dz = this && this.getAttribute('id');
|
||||
if (!dz && e && e.clientY)
|
||||
// cuo2duo fallback
|
||||
dz = e.clientY < window.innerHeight / 2 ? 'up_dz' : 'srch_dz';
|
||||
|
||||
var err = this.getAttribute('err');
|
||||
if (err)
|
||||
@@ -994,13 +1014,9 @@ function up2k_init(subtle) {
|
||||
}
|
||||
|
||||
function up_them(good_files) {
|
||||
var seen = {},
|
||||
evpath = get_evpath(),
|
||||
var evpath = get_evpath(),
|
||||
draw_each = good_files.length < 50;
|
||||
|
||||
for (var a = 0; a < st.files.length; a++)
|
||||
seen[st.files[a].name + '\n' + st.files[a].size] = 1;
|
||||
|
||||
for (var a = 0; a < good_files.length; a++) {
|
||||
var fobj = good_files[a][0],
|
||||
name = good_files[a][1],
|
||||
@@ -1026,15 +1042,20 @@ function up2k_init(subtle) {
|
||||
"bytes_uploaded": 0,
|
||||
"hash": []
|
||||
},
|
||||
key = entry.name + '\n' + entry.size;
|
||||
key = name + '\n' + entry.size + '\n' + lmod + '\n' + uc.fsearch;
|
||||
|
||||
if (uc.fsearch)
|
||||
entry.srch = 1;
|
||||
|
||||
if (seen[key])
|
||||
continue;
|
||||
try {
|
||||
if (st.seen[fdir][key])
|
||||
continue;
|
||||
}
|
||||
catch (ex) {
|
||||
st.seen[fdir] = {};
|
||||
}
|
||||
|
||||
seen[key] = 1;
|
||||
st.seen[fdir][key] = 1;
|
||||
|
||||
pvis.addfile([
|
||||
uc.fsearch ? esc(entry.name) : linksplit(
|
||||
@@ -1067,21 +1088,7 @@ function up2k_init(subtle) {
|
||||
}
|
||||
more_one_file();
|
||||
|
||||
function u2cleanup(e) {
|
||||
ev(e);
|
||||
for (var a = 0; a < st.files.length; a++) {
|
||||
var t = st.files[a];
|
||||
if (t.done && t.name) {
|
||||
if (!qsr('#f' + t.n))
|
||||
continue;
|
||||
|
||||
t.name = undefined;
|
||||
}
|
||||
}
|
||||
}
|
||||
ebi('u2cleanup').onclick = u2cleanup;
|
||||
|
||||
var etaref = 0, etaskip = 0, op_minh = 0;
|
||||
var etaref = 0, etaskip = 0, utw_minh = 0;
|
||||
function etafun() {
|
||||
var nhash = st.busy.head.length + st.busy.hash.length + st.todo.head.length + st.todo.hash.length,
|
||||
nsend = st.busy.upload.length + st.todo.upload.length,
|
||||
@@ -1094,13 +1101,10 @@ function up2k_init(subtle) {
|
||||
|
||||
//ebi('acc_info').innerHTML = humantime(st.time.busy) + ' ' + f2f(now / 1000, 1);
|
||||
|
||||
var op = ebi('op_up2k'),
|
||||
uff = ebi('u2footfoot'),
|
||||
minh = QS('#op_up2k.act') ? Math.max(op_minh, uff.offsetTop + uff.offsetHeight - op.offsetTop + 32) : 0;
|
||||
|
||||
if (minh > op_minh || !op_minh) {
|
||||
op_minh = minh;
|
||||
op.style.minHeight = op_minh + 'px';
|
||||
var minh = QS('#op_up2k.act') && st.is_busy ? Math.max(utw_minh, ebi('u2tab').offsetHeight + 32) : 0;
|
||||
if (utw_minh < minh || !utw_minh) {
|
||||
utw_minh = minh;
|
||||
ebi('u2tabw').style.minHeight = utw_minh + 'px';
|
||||
}
|
||||
|
||||
if (!nhash)
|
||||
@@ -1223,15 +1227,16 @@ function up2k_init(subtle) {
|
||||
running = true;
|
||||
while (true) {
|
||||
var now = Date.now(),
|
||||
is_busy = 0 !=
|
||||
st.todo.head.length +
|
||||
st.todo.hash.length +
|
||||
st.todo.handshake.length +
|
||||
st.todo.upload.length +
|
||||
st.busy.head.length +
|
||||
st.busy.hash.length +
|
||||
st.busy.handshake.length +
|
||||
st.busy.upload.length;
|
||||
oldest_active = Math.min( // gzip take the wheel
|
||||
st.todo.head.length ? st.todo.head[0].n : st.files.length,
|
||||
st.todo.hash.length ? st.todo.hash[0].n : st.files.length,
|
||||
st.todo.upload.length ? st.todo.upload[0].nfile : st.files.length,
|
||||
st.todo.handshake.length ? st.todo.handshake[0].n : st.files.length,
|
||||
st.busy.head.length ? st.busy.head[0].n : st.files.length,
|
||||
st.busy.hash.length ? st.busy.hash[0].n : st.files.length,
|
||||
st.busy.upload.length ? st.busy.upload[0].nfile : st.files.length,
|
||||
st.busy.handshake.length ? st.busy.handshake[0].n : st.files.length),
|
||||
is_busy = oldest_active < st.files.length;
|
||||
|
||||
if (was_busy && !is_busy) {
|
||||
for (var a = 0; a < st.files.length; a++) {
|
||||
@@ -1251,7 +1256,7 @@ function up2k_init(subtle) {
|
||||
}
|
||||
|
||||
if (was_busy != is_busy) {
|
||||
was_busy = is_busy;
|
||||
st.is_busy = was_busy = is_busy;
|
||||
|
||||
window[(is_busy ? "add" : "remove") +
|
||||
"EventListener"]("beforeunload", warn_uploader_busy);
|
||||
@@ -1280,7 +1285,7 @@ function up2k_init(subtle) {
|
||||
|
||||
timer.rm(etafun);
|
||||
timer.rm(donut.do);
|
||||
op_minh = 0;
|
||||
utw_minh = 0;
|
||||
}
|
||||
else {
|
||||
timer.add(donut.do);
|
||||
@@ -1332,7 +1337,8 @@ function up2k_init(subtle) {
|
||||
}
|
||||
|
||||
if (st.todo.head.length &&
|
||||
st.busy.head.length < parallel_uploads) {
|
||||
st.busy.head.length < parallel_uploads &&
|
||||
(!is_busy || st.todo.head[0].n - oldest_active < parallel_uploads * 2)) {
|
||||
exec_head();
|
||||
mou_ikkai = true;
|
||||
}
|
||||
@@ -1855,7 +1861,8 @@ function up2k_init(subtle) {
|
||||
st.bytes.uploaded += cdr - car;
|
||||
t.bytes_uploaded += cdr - car;
|
||||
}
|
||||
else if (txt.indexOf('already got that') !== -1) {
|
||||
else if (txt.indexOf('already got that') + 1 ||
|
||||
txt.indexOf('already being written') + 1) {
|
||||
console.log("ignoring dupe-segment error", t);
|
||||
}
|
||||
else {
|
||||
@@ -1863,6 +1870,9 @@ function up2k_init(subtle) {
|
||||
xhr.status, t.name) + (txt || "no further information"));
|
||||
return;
|
||||
}
|
||||
orz2(xhr);
|
||||
}
|
||||
function orz2(xhr) {
|
||||
apop(st.busy.upload, upt);
|
||||
apop(t.postlist, npart);
|
||||
if (!t.postlist.length) {
|
||||
@@ -1884,9 +1894,11 @@ function up2k_init(subtle) {
|
||||
if (crashed)
|
||||
return;
|
||||
|
||||
toast.err(9.98, "failed to upload a chunk,\n" + tries + " retries so far -- retrying in 10sec\n\n" + t.name);
|
||||
if (!toast.visible)
|
||||
toast.warn(9.98, "failed to upload a chunk;\nprobably harmless, continuing\n\n" + t.name);
|
||||
|
||||
console.log('chunkpit onerror,', ++tries, t);
|
||||
setTimeout(do_send, 10 * 1000);
|
||||
orz2(xhr);
|
||||
};
|
||||
xhr.open('POST', t.purl, true);
|
||||
xhr.setRequestHeader("X-Up2k-Hash", t.hash[npart]);
|
||||
@@ -1911,8 +1923,8 @@ function up2k_init(subtle) {
|
||||
wpx = window.innerWidth,
|
||||
fpx = parseInt(getComputedStyle(bar)['font-size']),
|
||||
wem = wpx * 1.0 / fpx,
|
||||
wide = wem > 54 ? 'w' : '',
|
||||
write = has(perms, 'write'),
|
||||
wide = write && wem > 54 ? 'w' : '',
|
||||
parent = ebi(wide && write ? 'u2btn_cw' : 'u2btn_ct'),
|
||||
btn = ebi('u2btn');
|
||||
|
||||
@@ -1924,7 +1936,7 @@ function up2k_init(subtle) {
|
||||
ebi('u2etaw').setAttribute('class', wide);
|
||||
}
|
||||
|
||||
wide = wem > 78 ? 'ww' : wide;
|
||||
wide = write && wem > 78 ? 'ww' : wide;
|
||||
parent = ebi(wide == 'ww' && write ? 'u2c3w' : 'u2c3t');
|
||||
var its = [ebi('u2etaw'), ebi('u2cards')];
|
||||
if (its[0].parentNode !== parent) {
|
||||
|
||||
@@ -7,8 +7,7 @@ if (!window['console'])
|
||||
|
||||
|
||||
var is_touch = 'ontouchstart' in window,
|
||||
IPHONE = /iPhone|iPad|iPod/i.test(navigator.userAgent),
|
||||
ANDROID = /android/i.test(navigator.userAgent),
|
||||
IPHONE = is_touch && /iPhone|iPad|iPod/i.test(navigator.userAgent),
|
||||
WINDOWS = navigator.platform ? navigator.platform == 'Win32' : /Windows/.test(navigator.userAgent);
|
||||
|
||||
|
||||
@@ -181,6 +180,7 @@ function ignex(all) {
|
||||
if (!all)
|
||||
window.onerror = vis_exh;
|
||||
}
|
||||
window.onerror = vis_exh;
|
||||
|
||||
|
||||
function noop() { }
|
||||
@@ -286,15 +286,19 @@ function crc32(str) {
|
||||
|
||||
|
||||
function clmod(el, cls, add) {
|
||||
if (!el)
|
||||
return false;
|
||||
|
||||
if (el.classList) {
|
||||
var have = el.classList.contains(cls);
|
||||
if (add == 't')
|
||||
add = !have;
|
||||
|
||||
if (add != have)
|
||||
el.classList[add ? 'add' : 'remove'](cls);
|
||||
if (!add == !have)
|
||||
return false;
|
||||
|
||||
return;
|
||||
el.classList[add ? 'add' : 'remove'](cls);
|
||||
return true;
|
||||
}
|
||||
|
||||
var re = new RegExp('\\s*\\b' + cls + '\\s*\\b', 'g'),
|
||||
@@ -305,12 +309,18 @@ function clmod(el, cls, add) {
|
||||
|
||||
var n2 = n1.replace(re, ' ') + (add ? ' ' + cls : '');
|
||||
|
||||
if (n1 != n2)
|
||||
el.className = n2;
|
||||
if (!n1 == !n2)
|
||||
return false;
|
||||
|
||||
el.className = n2;
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
function clgot(el, cls) {
|
||||
if (!el)
|
||||
return;
|
||||
|
||||
if (el.classList)
|
||||
return el.classList.contains(cls);
|
||||
|
||||
@@ -782,13 +792,18 @@ var timer = (function () {
|
||||
var tt = (function () {
|
||||
var r = {
|
||||
"tt": mknod("div"),
|
||||
"th": mknod("div"),
|
||||
"en": true,
|
||||
"el": null,
|
||||
"skip": false
|
||||
"skip": false,
|
||||
"lvis": 0
|
||||
};
|
||||
|
||||
r.th.innerHTML = '?';
|
||||
r.tt.setAttribute('id', 'tt');
|
||||
r.th.setAttribute('id', 'tth');
|
||||
document.body.appendChild(r.tt);
|
||||
document.body.appendChild(r.th);
|
||||
|
||||
var prev = null;
|
||||
r.cshow = function () {
|
||||
@@ -798,11 +813,25 @@ var tt = (function () {
|
||||
prev = this;
|
||||
};
|
||||
|
||||
r.show = function () {
|
||||
if (r.skip) {
|
||||
r.skip = false;
|
||||
var tev;
|
||||
r.dshow = function (e) {
|
||||
clearTimeout(tev);
|
||||
if (!r.getmsg(this))
|
||||
return;
|
||||
}
|
||||
|
||||
if (Date.now() - r.lvis < 400)
|
||||
return r.show.bind(this)();
|
||||
|
||||
tev = setTimeout(r.show.bind(this), 800);
|
||||
if (is_touch)
|
||||
return;
|
||||
|
||||
this.addEventListener('mousemove', r.move);
|
||||
clmod(r.th, 'act', 1);
|
||||
r.move(e);
|
||||
};
|
||||
|
||||
r.getmsg = function (el) {
|
||||
if (QS('body.bbox-open'))
|
||||
return;
|
||||
|
||||
@@ -810,7 +839,16 @@ var tt = (function () {
|
||||
if (cfg !== null && cfg != '1')
|
||||
return;
|
||||
|
||||
var msg = this.getAttribute('tt');
|
||||
return el.getAttribute('tt');
|
||||
};
|
||||
|
||||
r.show = function () {
|
||||
clearTimeout(tev);
|
||||
if (r.skip) {
|
||||
r.skip = false;
|
||||
return;
|
||||
}
|
||||
var msg = r.getmsg(this);
|
||||
if (!msg)
|
||||
return;
|
||||
|
||||
@@ -824,6 +862,7 @@ var tt = (function () {
|
||||
if (dir.indexOf('u') + 1) top = false;
|
||||
if (dir.indexOf('d') + 1) top = true;
|
||||
|
||||
clmod(r.th, 'act');
|
||||
clmod(r.tt, 'b', big);
|
||||
r.tt.style.left = '0';
|
||||
r.tt.style.top = '0';
|
||||
@@ -849,14 +888,27 @@ var tt = (function () {
|
||||
|
||||
r.hide = function (e) {
|
||||
ev(e);
|
||||
clearTimeout(tev);
|
||||
window.removeEventListener('scroll', r.hide);
|
||||
clmod(r.tt, 'show');
|
||||
|
||||
clmod(r.tt, 'b');
|
||||
clmod(r.th, 'act');
|
||||
if (clmod(r.tt, 'show'))
|
||||
r.lvis = Date.now();
|
||||
|
||||
if (r.el)
|
||||
r.el.removeEventListener('mouseleave', r.hide);
|
||||
|
||||
if (e && e.target)
|
||||
e.target.removeEventListener('mousemove', r.move);
|
||||
};
|
||||
|
||||
if (is_touch && IPHONE) {
|
||||
r.move = function (e) {
|
||||
r.th.style.left = (e.pageX + 12) + 'px';
|
||||
r.th.style.top = (e.pageY + 12) + 'px';
|
||||
};
|
||||
|
||||
if (IPHONE) {
|
||||
var f1 = r.show,
|
||||
f2 = r.hide,
|
||||
q = [];
|
||||
@@ -882,14 +934,14 @@ var tt = (function () {
|
||||
|
||||
r.att = function (ctr) {
|
||||
var _cshow = r.en ? r.cshow : null,
|
||||
_show = r.en ? r.show : null,
|
||||
_dshow = r.en ? r.dshow : null,
|
||||
_hide = r.en ? r.hide : null,
|
||||
o = ctr.querySelectorAll('*[tt]');
|
||||
|
||||
for (var a = o.length - 1; a >= 0; a--) {
|
||||
o[a].onfocus = _cshow;
|
||||
o[a].onblur = _hide;
|
||||
o[a].onmouseenter = _show;
|
||||
o[a].onmouseenter = _dshow;
|
||||
o[a].onmouseleave = _hide;
|
||||
}
|
||||
r.hide();
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
|
||||
#ops, #tree, #path, #wrap>h2:last-child, /* main tabs and navigators (tree/breadcrumbs) */
|
||||
|
||||
#u2cleanup, #u2conf tr:first-child>td[rowspan]:not(#u2btn_cw), /* most of the config options */
|
||||
#u2conf tr:first-child>td[rowspan]:not(#u2btn_cw), /* most of the config options */
|
||||
|
||||
#srch_dz, #srch_zd, /* the filesearch dropzone */
|
||||
|
||||
|
||||
@@ -80,6 +80,12 @@ shab64() { sp=$1; f="$2"; v=0; sz=$(stat -c%s "$f"); while true; do w=$((v+sp*10
|
||||
command -v gdate && date() { gdate "$@"; }; while true; do t=$(date +%s.%N); (time wget http://127.0.0.1:3923/?ls -qO- | jq -C '.files[]|{sz:.sz,ta:.tags.artist,tb:.tags.".bpm"}|del(.[]|select(.==null))' | awk -F\" '/"/{t[$2]++} END {for (k in t){v=t[k];p=sprintf("%" (v+1) "s",v);gsub(/ /,"#",p);printf "\033[36m%s\033[33m%s ",k,p}}') 2>&1 | awk -v ts=$t 'NR==1{t1=$0} NR==2{sub(/.*0m/,"");sub(/s$/,"");t2=$0;c=2; if(t2>0.3){c=3} if(t2>0.8){c=1} } END{sub(/[0-9]{6}$/,"",ts);printf "%s \033[3%dm%s %s\033[0m\n",ts,c,t2,t1}'; sleep 0.1 || break; done
|
||||
|
||||
|
||||
##
|
||||
## track an up2k upload and print all chunks in file-order
|
||||
|
||||
grep '"name": "2021-07-18 02-17-59.mkv"' fug.log | head -n 1 | sed -r 's/.*"hash": \[//; s/\].*//' | tr '"' '\n' | grep -E '^[a-zA-Z0-9_-]{44}$' | while IFS= read -r cid; do cat -n fug.log | grep -vF '"purl": "' | grep -- "$cid"; echo; done | stdbuf -oL tr '\t' ' ' | while IFS=' ' read -r ln _ _ _ _ _ ts ip port msg; do [ -z "$msg" ] && echo && continue; printf '%6s [%s] [%s] %s\n' $ln "$ts" "$ip $port" "$msg"; read -r ln _ _ _ _ _ ts ip port msg < <(cat -n fug.log | tail -n +$((ln+1)) | grep -F "$ip $port" | head -n 1); printf '%6s [%s] [%s] %s\n' $ln "$ts" "$ip $port" "$msg"; done
|
||||
|
||||
|
||||
##
|
||||
## js oneliners
|
||||
|
||||
@@ -169,7 +175,7 @@ brew install python@2
|
||||
pip install virtualenv
|
||||
|
||||
# readme toc
|
||||
cat README.md | awk 'function pr() { if (!h) {return}; if (/^ *[*!#]/||!s) {printf "%s\n",h;h=0;return}; if (/.../) {printf "%s - %s\n",h,$0;h=0}; }; /^#/{s=1;pr()} /^#* *(file indexing|install on android|dev env setup|just the sfx|complete release|optional gpl stuff)|`$/{s=0} /^#/{lv=length($1);sub(/[^ ]+ /,"");bab=$0;gsub(/ /,"-",bab); h=sprintf("%" ((lv-1)*4+1) "s [%s](#%s)", "*",$0,bab);next} !h{next} {sub(/ .*/,"");sub(/[:,]$/,"")} {pr()}' > toc; grep -E '^## readme toc' -B1000 -A2 <README.md >p1; grep -E '^## quickstart' -B2 -A999999 <README.md >p2; (cat p1; grep quickstart -A1000 <toc; cat p2) >README.md; rm p1 p2 toc
|
||||
cat README.md | awk 'function pr() { if (!h) {return}; if (/^ *[*!#|]/||!s) {printf "%s\n",h;h=0;return}; if (/.../) {printf "%s - %s\n",h,$0;h=0}; }; /^#/{s=1;pr()} /^#* *(file indexing|install on android|dev env setup|just the sfx|complete release|optional gpl stuff)|`$/{s=0} /^#/{lv=length($1);sub(/[^ ]+ /,"");bab=$0;gsub(/ /,"-",bab); h=sprintf("%" ((lv-1)*4+1) "s [%s](#%s)", "*",$0,bab);next} !h{next} {sub(/ .*/,"");sub(/[:,]$/,"")} {pr()}' > toc; grep -E '^## readme toc' -B1000 -A2 <README.md >p1; grep -E '^## quickstart' -B2 -A999999 <README.md >p2; (cat p1; grep quickstart -A1000 <toc; cat p2) >README.md; rm p1 p2 toc
|
||||
|
||||
# fix firefox phantom breakpoints,
|
||||
# suggestions from bugtracker, doesnt work (debugger is not attachable)
|
||||
@@ -185,8 +191,13 @@ about:config >> devtools.debugger.prefs-schema-version = -1
|
||||
git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > ../revs && cat ../{util,browser,up2k}.js >../vr && cat ../revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser,up2k}.js >../vg 2>/dev/null && diff -wNarU0 ../{vg,vr} | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done
|
||||
|
||||
# download all sfx versions
|
||||
curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | .tag_name + " " + .name' | tr -d '\r' | while read v t; do fn="copyparty $v $t.py"; [ -e "$fn" ] || curl https://github.com/9001/copyparty/releases/download/$v/copyparty-sfx.py -Lo "$fn"; done
|
||||
curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | .tag_name + " " + .name' | tr -d '\r' | while read v t; do fn="$(printf '%s\n' "copyparty $v $t.py" | tr / -)"; [ -e "$fn" ] || curl https://github.com/9001/copyparty/releases/download/$v/copyparty-sfx.py -Lo "$fn"; done
|
||||
|
||||
# push to multiple git remotes
|
||||
git config -l | grep '^remote'
|
||||
git remote add all git@github.com:9001/copyparty.git
|
||||
git remote set-url --add --push all git@gitlab.com:9001/copyparty.git
|
||||
git remote set-url --add --push all git@github.com:9001/copyparty.git
|
||||
|
||||
##
|
||||
## http 206
|
||||
|
||||
@@ -140,10 +140,10 @@ repack() {
|
||||
}
|
||||
|
||||
repack sfx-full "re gz no-sh"
|
||||
repack sfx-ent "re no-dd no-ogv"
|
||||
repack sfx-ent "re no-dd no-ogv gz no-sh"
|
||||
repack sfx-lite "re no-dd no-ogv no-cm no-hl"
|
||||
repack sfx-lite "re no-dd no-ogv no-cm no-hl gz no-sh"
|
||||
repack sfx-ent "re no-dd"
|
||||
repack sfx-ent "re no-dd gz no-sh"
|
||||
repack sfx-lite "re no-dd no-cm no-hl"
|
||||
repack sfx-lite "re no-dd no-cm no-hl gz no-sh"
|
||||
|
||||
|
||||
# move fuse and up2k clients into copyparty-extras/,
|
||||
|
||||
@@ -3,7 +3,6 @@ WORKDIR /z
|
||||
ENV ver_asmcrypto=5b994303a9d3e27e0915f72a10b6c2c51535a4dc \
|
||||
ver_hashwasm=4.9.0 \
|
||||
ver_marked=3.0.4 \
|
||||
ver_ogvjs=1.8.4 \
|
||||
ver_mde=2.15.0 \
|
||||
ver_codemirror=5.62.3 \
|
||||
ver_fontawesome=5.13.0 \
|
||||
@@ -15,7 +14,6 @@ ENV ver_asmcrypto=5b994303a9d3e27e0915f72a10b6c2c51535a4dc \
|
||||
RUN mkdir -p /z/dist/no-pk \
|
||||
&& wget https://fonts.gstatic.com/s/sourcecodepro/v11/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2 -O scp.woff2 \
|
||||
&& apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzip python3 python3-dev brotli py3-brotli \
|
||||
&& wget https://github.com/brion/ogv.js/releases/download/$ver_ogvjs/ogvjs-$ver_ogvjs.zip -O ogvjs.zip \
|
||||
&& wget https://github.com/openpgpjs/asmcrypto.js/archive/$ver_asmcrypto.tar.gz -O asmcrypto.tgz \
|
||||
&& wget https://github.com/markedjs/marked/archive/v$ver_marked.tar.gz -O marked.tgz \
|
||||
&& wget https://github.com/Ionaru/easy-markdown-editor/archive/$ver_mde.tar.gz -O mde.tgz \
|
||||
@@ -23,7 +21,6 @@ RUN mkdir -p /z/dist/no-pk \
|
||||
&& wget https://github.com/FortAwesome/Font-Awesome/releases/download/$ver_fontawesome/fontawesome-free-$ver_fontawesome-web.zip -O fontawesome.zip \
|
||||
&& wget https://github.com/google/zopfli/archive/zopfli-$ver_zopfli.tar.gz -O zopfli.tgz \
|
||||
&& wget https://github.com/Daninet/hash-wasm/releases/download/v$ver_hashwasm/hash-wasm@$ver_hashwasm.zip -O hash-wasm.zip \
|
||||
&& unzip ogvjs.zip \
|
||||
&& (mkdir hash-wasm \
|
||||
&& cd hash-wasm \
|
||||
&& unzip ../hash-wasm.zip) \
|
||||
@@ -77,21 +74,6 @@ RUN cd hash-wasm \
|
||||
&& mv sha512.umd.min.js /z/dist/sha512.hw.js
|
||||
|
||||
|
||||
# build ogvjs
|
||||
RUN cd ogvjs-$ver_ogvjs \
|
||||
&& cp -pv \
|
||||
ogv-worker-audio.js \
|
||||
ogv-demuxer-ogg-wasm.js \
|
||||
ogv-demuxer-ogg-wasm.wasm \
|
||||
ogv-decoder-audio-opus-wasm.js \
|
||||
ogv-decoder-audio-opus-wasm.wasm \
|
||||
ogv-decoder-audio-vorbis-wasm.js \
|
||||
ogv-decoder-audio-vorbis-wasm.wasm \
|
||||
/z/dist \
|
||||
&& cp -pv \
|
||||
ogv-es2017.js /z/dist/ogv.js
|
||||
|
||||
|
||||
# build marked
|
||||
COPY marked.patch /z/
|
||||
COPY marked-ln.patch /z/
|
||||
|
||||
@@ -16,9 +16,6 @@ help() { exec cat <<'EOF'
|
||||
#
|
||||
# `no-sh` makes just the python sfx, skips the sh/unix sfx
|
||||
#
|
||||
# `no-ogv` saves ~192k by removing the opus/vorbis audio codecs
|
||||
# (only affects apple devices; everything else has native support)
|
||||
#
|
||||
# `no-cm` saves ~82k by removing easymde/codemirror
|
||||
# (the fancy markdown editor)
|
||||
#
|
||||
@@ -75,7 +72,6 @@ while [ ! -z "$1" ]; do
|
||||
clean) clean=1 ; ;;
|
||||
re) repack=1 ; ;;
|
||||
gz) use_gz=1 ; ;;
|
||||
no-ogv) no_ogv=1 ; ;;
|
||||
no-fnt) no_fnt=1 ; ;;
|
||||
no-hl) no_hl=1 ; ;;
|
||||
no-dd) no_dd=1 ; ;;
|
||||
@@ -218,9 +214,6 @@ cat have | while IFS= read -r x; do
|
||||
done
|
||||
rm have
|
||||
|
||||
[ $no_ogv ] &&
|
||||
rm -rf copyparty/web/deps/{dynamicaudio,ogv}*
|
||||
|
||||
[ $no_cm ] && {
|
||||
rm -rf copyparty/web/mde.* copyparty/web/deps/easymde*
|
||||
echo h > copyparty/web/mde.html
|
||||
|
||||
@@ -7,8 +7,9 @@ v=$1
|
||||
printf '%s\n' "$v" | grep -qE '^[0-9\.]+$' || exit 1
|
||||
grep -E "(${v//./, })" ../copyparty/__version__.py || exit 1
|
||||
|
||||
git push all
|
||||
git tag v$v
|
||||
git push origin --tags
|
||||
git push all --tags
|
||||
|
||||
rm -rf ../dist
|
||||
|
||||
|
||||
@@ -49,14 +49,6 @@ copyparty/web/deps/easymde.js,
|
||||
copyparty/web/deps/marked.js,
|
||||
copyparty/web/deps/mini-fa.css,
|
||||
copyparty/web/deps/mini-fa.woff,
|
||||
copyparty/web/deps/ogv-decoder-audio-opus-wasm.js,
|
||||
copyparty/web/deps/ogv-decoder-audio-opus-wasm.wasm,
|
||||
copyparty/web/deps/ogv-decoder-audio-vorbis-wasm.js,
|
||||
copyparty/web/deps/ogv-decoder-audio-vorbis-wasm.wasm,
|
||||
copyparty/web/deps/ogv-demuxer-ogg-wasm.js,
|
||||
copyparty/web/deps/ogv-demuxer-ogg-wasm.wasm,
|
||||
copyparty/web/deps/ogv-worker-audio.js,
|
||||
copyparty/web/deps/ogv.js,
|
||||
copyparty/web/deps/prism.js,
|
||||
copyparty/web/deps/prism.css,
|
||||
copyparty/web/deps/prismd.css,
|
||||
|
||||
Reference in New Issue
Block a user